Language & intelligence

NLP & linguistics

Deterministic, fully embedded classic pipeline; beats NLTK on simplicity, a tier below spaCy's neural models.

3 Strong · 5 Solid · 3 Partial · 1 Emerging

Measured against: spaCy · NLTK · Stanza · Folders: base/linguistic(opens in a new tab) · base/natural(opens in a new tab) · base/learning(opens in a new tab) · For programmers: this area's reference

**the words of one paragraph, sized by how often they occur.** drawn on 2026-10-01 by the site's script tools/areas-figurative.ring, run inside the library at commit 0e72e2e2c · the source(opens in a new tab)

Rethought from first principles

What Softanza kept from what works, and what it rethought, as the library wrote it down in its design documents and narrations.

The Softanza wayA string is characters; a text is a string raised to meaning, and every language capability must work locally, with nothing to install or download.

Kept from the best practice

Published reference algorithms, not home-made ones: Snowball stemming in 25 languages, VADER sentiment, an averaged-perceptron part-of-speech tagger, and TextRank summaries. source(opens in a new tab)
Words and sentences are split by the Unicode segmentation rules (UAX #29), with a few documented deviations chosen for good language tokens. source(opens in a new tab)
Classic, explainable machine learning — nearest neighbours, naive Bayes, decision trees, association rules — is the learning floor, running locally and explaining its answers. source(opens in a new tab)

Rethought

Everything is embedded: one line works with no install, no download and no internet, instead of fetching data packages feature by feature. source(opens in a new tab)
Verdicts carry their confidence — certainly, probably or apparently — and Why() explains them, because language reports how surely something is known, not only what. source(opens in a new tab)
The same call sharpens when a local model is present — rule-based entities become transformer entities — and never requires anything paid or remote. source(opens in a new tab)
In near-natural code the programmer is the parser: method chains follow English grammar (articles, agreement, 'which'), so the machine never has to guess. source(opens in a new tab)

A deterministic, fully-embedded classic NLP pipeline (UAX#29 tokenization, 25-language Snowball stemming, an averaged-perceptron POS tagger reimplemented in Zig, VADER sentiment, RAKE + TextRank + TF-IDF keywords, WordNet, phonetics) with zero downloads and a one-line first contact. Its stated target is to beat NLTK on simplicity, and against NLTK it largely does. But it sits a full tier below spaCy and Stanza: NER is gazetteer-rule not neural, there is no dependency parsing, and the neural upgrades are wired but no-op without a local model, gated on the still-unstarted ggml/GGUF foundry.

What a crafter does with it, and what stands out

Zero downloads: every model and lexicon is @embedFile'd, one-line first contact
RAKE, TextRank and TF-IDF plus extractive summary in one engine
NLTK's own averaged-perceptron POS model reimplemented in Zig with no ML runtime
25-language Snowball stemming at reference parity

What is owed

The ggml bridge + GGUF foundry that turns every neural delegator real
Statistical or neural NER above the gazetteer floor
Restore the full WordNet graph (hyponyms, meronyms, antonyms, glosses)
Dependency parsing, entirely absent

The Softanza way, run

What Softanza does differently in this area, shown by code taken from its narrations and guards, and run for this page.

With nothing downloaded, a sentence is tagged and its noun phrases are found by a pattern over word classes.

Softanza

o = new stzText("The quick brown fox jumps over the lazy dog")
aNP = o.NounPhrases()
? "  NounPhrases: " + @@(aNP)
chk("the classic NP grammar finds both phrases",
	len(aNP) = 2 and aNP[1] = "The quick brown fox" and aNP[2] = "the lazy dog")
chk("any tag grammar works (VBZ IN)", o.Chunks("VBZ IN")[1] = "jumps over")

Output

  NounPhrases: [ "The quick brown fox", "the lazy dog" ]
  [OK] the classic NP grammar finds both phrases
  [OK] any tag grammar works (VBZ IN)

run on 2026-10-01 21:17 inside the library at commit 0e72e2e2c; taken from base/test/linguistic/chunker_narrated.ring(opens in a new tab)

A verdict is spoken with its confidence: strong, good or weak evidence becomes certainly, probably or apparently.

Softanza

Then("strong evidence", StzEvidentialVerdict(1, 0.95), "certainly yes")
Then("good evidence", StzEvidentialVerdict(1, 0.7), "probably yes")
Then("weak evidence", StzEvidentialVerdict(1, 0.3), "apparently yes")
Then("strong evidence AGAINST", StzEvidentialVerdict(0, 0.9), "certainly not")

Output

  THEN  strong evidence  [PASS]
  THEN  good evidence  [PASS]
  THEN  weak evidence  [PASS]
  THEN  strong evidence AGAINST  [PASS]

run on 2026-10-01 21:17 inside the library at commit 0e72e2e2c; taken from base/test/natural/evidentiality_narrated.ring(opens in a new tab)

Code that reads like English checks a claim, refuses it, and explains why in words.

Softanza

Then("'at most 2 vowels' refuses on 3", Q("AnnIE").AtMost(2).VowelNB(), FALSE)
Then("...and explains itself", Why(), "no: expected atmost 2, found 3")

Output

  THEN  'at most 2 vowels' refuses on 3  [PASS]
  THEN  ...and explains itself  [PASS]

run on 2026-10-01 21:17 inside the library at commit 0e72e2e2c; taken from base/test/natural/nnl_narrated.ring(opens in a new tab)

The articles of this area (10)

The lanes, one by one

Lanes and their notes are quoted as they were read at the source, on Read from the source on 2026-09-16.

Tokenization and sentence segmentation

Strong

word_break.zig implements the full UAX#29 state machine; WordIter / SentenceIter are the shared seam every lane routes through; CJK and Thai break per-codepointengine/src/string/word_break.zig

Stemming

Strong

Vendored Snowball (stem.zig) across 25 languages, engine-backed, with per-language and auto-detect entry points; at reference parity because it is Snowballstem.zig

Keyword and keyphrase extraction

Strong

RAKE (textutil.zig), TextRank keywords + extractive summary (textrank.zig), TF-IDF (textstats.zig, corpus IDF via stzCorpus), all engine-backedtextutil.zig / textrank.zig / textstats.zig

POS tagging

Solid

pos.zig averaged perceptron, a faithful reimplementation of NLTK's tagger (Penn Treebank), ~75k weights @embedFile'd; English only, below neural taggerspos.zig

Sentiment (VADER)

Solid

sentiment.zig faithful VADER: 7,500-term lexicon plus five rule heuristics with an explainable breakdown; English only, rule-basedsentiment.zig; build-time tally over 1,000 reviews

Lemmatization

Solid

lemma.zig dictionary lookup for en / fr / ar (fr table 4.5 MB) plus English suffix rules; POS-blind, ambiguous forms resolve by an alphabetical tie-breaklemma.zig

Statistical text (collocations, readability, lang-detect)

Solid

PMI collocations, n-grams + frequency, cosine similarity, Flesch readability, stopwords, lexical diversity, language detection; mostly English-tunedtextstats.zig / textutil.zig

Phonetics

Solid

phonetics.zig: Soundex (4-char) and Metaphone (6-char), pure algorithm with no data file; no Double Metaphone, NYSIIS or Caverphonephonetics.zig

Named-entity recognition

Partial

ner.zig deterministic gazetteer + POS-chunk + rules into PERSON / ORGANIZATION / LOCATION / ENTITY; four types, English gazetteers, no learned disambiguationner.zig

WordNet

Partial

wordnet.zig over a compacted Princeton WordNet 3.1 (8.6 MB, ~140k words): synonyms and hypernyms only; hyponyms, meronyms, antonyms and glosses droppedwordnet.zig

Parsing (chunk and constituency)

Partial

Haro-side Chunks, NounPhrases and a cascaded constituency parse tree (stzParseTree, Penn bracket notation); no dependency parsing at allstzParseTree

Neural NLP and embeddings

Emerging

stzText exposes Embedding(), SemanticSimilarityWith(), Classify(labels), SummarizedAbstractively() gated on HasSemanticModel(); no-op without a GGUF modelstzText HasSemanticModel()