Language resists computation
Images are grids of numbers. Audio is a waveform. Text is a sequence of arbitrary symbols whose meaning depends on other symbols, on the speaker, on the world, and on things nobody wrote down. Five properties cause most of the difficulty.
Ambiguity, at every level
One word, many senses. One sentence, many parses. One pronoun, many referents. I saw the man with the telescope has two readings and no amount of grammar resolves which. Humans disambiguate with context and world knowledge, mostly unconsciously.
Discreteness
You cannot take a small step from "cat" to "dog". There is no gradient through the space of words, which is why the numbers-and-vectors detour exists at all — continuous space is where optimization works.
Compositionality with exceptions
Meaning is usually built from parts, until it isn't. Kick the bucket composes into nothing like its parts. Idiom, sarcasm, and implicature all break the rule the rest of the language follows.
An unbounded, unbalanced vocabulary
New words appear constantly, and word frequency follows a brutal power law — a few hundred words cover most of any text while the long tail runs forever. Whatever you train on, most of what you meet later will be rare.
It presumes the world
The trophy didn't fit in the suitcase because it was too big. Resolving it requires knowing something about trophies and suitcases. Language is written by people who assume you already know things.
Zipf's law, and why the tail never ends
Rank words by frequency and the nth most common appears roughly proportional to 1/n. The most frequent word is about twice as common as the second, three times the third, and onward. On log-log axes this is a straight line.
The practical consequence: about 150 words cover half of ordinary English, while roughly half of the distinct words in any large corpus appear exactly once. Vocabulary can never be complete, which is precisely the problem subword tokenization was invented to dissolve.
How the field got here
Worth knowing, because each era's tools are still in production somewhere, and because the current one inherits its vocabulary from all four before it.
Rules and symbols
Hand-written grammars, parsers, ontologies, expert systems. Linguists encoded knowledge directly. Precise where it worked, catastrophically brittle everywhere else — every exception needed its own rule, and language is mostly exceptions. Survives today in regex, grammar checkers, and rule layers wrapped around statistical systems.
Statistics and counting
Learn from corpora instead of legislating. N-gram language models, hidden Markov models for tagging, naive Bayes and SVMs over bag-of-words features, phrase-based statistical translation. Robust and interpretable, but blind to word similarity and unable to hold context beyond a few words.
Embeddings and recurrence
word2vec and GloVe give words dense vectors where similar words sit near each other — the first real notion of semantic distance. RNNs, LSTMs and GRUs process sequences with memory; sequence-to-sequence with attention makes neural translation work. Still fundamentally sequential, so still slow and still forgetful over long spans.
Transformers and pretraining
Attention replaces recurrence entirely, which makes training parallel across the sequence and therefore makes scale affordable. BERT establishes pretrain-then-fine-tune as the default recipe; GPT-2 shows that a plain next-token objective at scale produces surprisingly general capability.
Scale, instruction, and assembly
Models grow until few-shot prompting substitutes for fine-tuning. Instruction tuning and preference optimization turn text predictors into things that follow requests. The engineering centre of gravity shifts from training models to assembling systems around them — retrieval, tools, structured output, evaluation harnesses.
Choosing the atoms
Before anything else, text has to be cut into units the model has a fixed vocabulary for. This choice is more consequential than it looks — it sets what the model can perceive, how much fits in context, and what it costs to run.
| Granularity | How it works | Trade-off |
|---|---|---|
| Character | Every character is a token | Tiny vocabulary, nothing unknown, but sequences get very long and each unit carries almost no meaning |
| Word | Split on whitespace and punctuation | Meaningful units, but the vocabulary is unbounded, misspellings become unknowns, and it fails on languages without spaces |
| Subword | Frequent sequences stay whole, rare ones split into pieces | The universal answer. Fixed vocabulary, no unknown token ever, sensible lengths |
BPE
Start from characters, repeatedly merge the most frequent adjacent pair, stop at the target vocabulary size. Used by GPT models. Simple, fast, entirely frequency-driven.
WordPiece
Same idea, but merges are chosen to maximize training-corpus likelihood rather than raw count. BERT's tokenizer. Marks word-internal pieces explicitly.
SentencePiece / Unigram
Treats the raw byte stream as-is, so it needs no pre-splitting on whitespace and handles any script. Unigram prunes a large candidate vocabulary down probabilistically. The multilingual default.
The classical pipeline, and what's left of it
For twenty years, preprocessing was most of the work. Modern neural pipelines discard nearly all of it — but the steps remain correct and necessary for classical models, search systems, and any pipeline where you control the features.
| Step | What it does | Still worth doing? |
|---|---|---|
| Unicode normalize | NFC/NFKC so visually identical strings compare equal | Always. Silent, real bugs otherwise |
| Whitespace & control | Strip zero-width and control characters, collapse runs of space | Always. Invisible characters are a genuine attack surface |
| Lowercasing | Collapse case | Classical models yes; neural models no — case carries real signal |
| Stopword removal | Drop "the", "of", "and" | Bag-of-words and search yes; neural models never — function words carry the syntax |
| Stemming | Chop suffixes crudely: running → run | Search and retrieval only. Fast, lossy, produces non-words |
| Lemmatization | Map to dictionary form using POS: better → good | Linguistic analysis, some classical pipelines. Slower, correct |
| Sentence splitting | Find real sentence boundaries | Yes — still needed for chunking documents for retrieval |
| Deduplication | Remove near-duplicate documents | Critically yes at training scale; also prevents train/test leakage |
From counting words to placing them in space
Tokens are integers, and integers are meaningless — id 4,502 is not "greater" than id 12. Everything depends on what you turn those ids into.
Sparse — count-based
- one-hot
- A vector as long as the vocabulary, all zeros but one. Every pair of words is equally distant. No notion of similarity whatsoever.
- bag of words
- Count each vocabulary word in the document. Loses all order — "dog bites man" and "man bites dog" are identical.
- n-grams
- Count short sequences instead of single words, recovering a little local order at the cost of a much larger feature space.
- TF-IDF
- Weight each count by how rare the word is across the corpus, so common words stop dominating. Still the backbone of classical search.
Sparse methods remain genuinely competitive for text classification on modest data, and BM25 — TF-IDF's refined descendant — is still a strong retrieval baseline that hybrid search systems deliberately keep.
Dense — learned
- word2vec
- Predict a word from its neighbours, or the reverse. The learned weights become vectors where similar words cluster. 2013's breakthrough.
- GloVe
- Factorize a global co-occurrence matrix instead. Different route, comparable destination.
- fastText
- Build word vectors from character n-grams, so unseen and misspelled words still get a sensible representation.
- contextual
- ELMo, BERT and everything since: the vector for a word depends on the sentence it's in. "Bank" by a river and "bank" holding money finally get different representations.
Geometry that means something
In a well-trained embedding space, direction carries meaning. The offset from king to queen runs roughly parallel to the offset from man to woman, which is why vector arithmetic on words produces the famous analogies. Related words cluster; unrelated ones don't.
Similarity is measured by the cosine of the angle between vectors rather than the distance between them, because length tends to track word frequency rather than meaning.
Predict the next token. That's the whole objective.
A language model assigns probabilities to sequences. Factor a sequence by the chain rule and the task becomes: given everything so far, what comes next? Every generative system in this document is that one operation, run repeatedly.
The n-gram approximation truncates that history to the last few words, then estimates the probabilities by counting. It is the oldest working language model, and building one takes about fifteen lines. The model below is real — it counts over a corpus embedded in this page and samples from what it finds.
What you should notice
- Local fluency, global incoherence. Every three-word window looks like English. The paragraph goes nowhere. That gap is the entire motivation for neural language models.
- Trigram is more fluent and more plagiaristic. With longer context there are fewer options, so it increasingly recites the corpus verbatim. More context with fixed data means memorization.
- Temperature is the confidence dial. Low values always take the likeliest continuation and loop; high values flatten the distribution into noise. The same control sits on every model you use today.
- Unseen context is fatal. A count-based model assigns probability zero to anything it never observed — the sparsity problem that smoothing techniques spent two decades patching.
Perplexity
The standard intrinsic metric: the exponentiated average negative log-likelihood per token. Read it as how many options the model was effectively choosing between at each step. Lower is better; a perplexity of 20 means the model was about as uncertain as if picking uniformly among 20 words.
From recurrence to attention
Counting can't generalize across similar contexts, so the field moved to neural networks that read sequences. The path from there to the transformer took about four years and one decisive idea.
Recurrent networks
Read one token at a time, carrying a hidden state forward. In principle the state summarizes everything seen so far. In practice, gradients vanish across long spans and early information fades.
LSTM and GRU add gates that decide what to keep, forget and expose, which genuinely extended usable memory to hundreds of tokens and powered a decade of production systems.
Seq2seq pairs an encoder with a decoder for translation and summarization — but forcing an entire input sentence through one fixed-size vector was a hard bottleneck.
Attention
Instead of one summary vector, let the decoder look back at every input position and weight them by relevance at each step. Translation quality jumped immediately and the alignments turned out to be interpretable.
Then the removal. In 2017 the recurrence was deleted and only attention was kept. Because nothing depends on the previous step's output, the whole sequence processes in parallel — which is what made training on internet-scale corpora economically possible.
Self-attention, structurally
Each token is projected three ways. Its query asks what it needs; every token's key advertises what it offers; the dot product of query against key scores the match. Softmax turns those scores into weights that sum to one, and the output is the weighted blend of every token's value.
Multi-head runs several of these in parallel with separate projections, so different heads can specialize — some track syntax, some track coreference. Positional encoding is added because attention alone is order-blind: without it, a sentence and its shuffle are identical inputs.
The block, and the three shapes it comes in
What each part does
- attention
- Mixes information between tokens. The only place positions talk to each other.
- feed-forward
- Transforms each position independently. Holds most of the parameters, and appears to store much of the factual knowledge.
- residual
- Adds the input back to the output, so gradients reach the bottom of a very deep stack.
- layer norm
- Rescales activations to keep training stable.
- positions
- Injects order — learned, sinusoidal, or rotary (RoPE), which is now the common choice.
Three architectures, three jobs
Encoder-only
BERT, RoBERTa, DeBERTa. Every token sees every other in both directions. Trained by masking words and predicting them. Produces rich representations, cannot generate. Best for classification, tagging, extraction, retrieval embeddings — and still smaller, faster and cheaper than a generative model for those jobs.
Decoder-only
GPT, Llama, Claude, Mistral. Each token sees only what came before — the triangular mask. Trained purely to predict the next token. Generates text, and at scale turns out to handle nearly every task if you phrase it as text continuation. The dominant shape today.
Encoder–decoder
T5, BART, mT5. Bidirectional encoder reads the input; causal decoder writes the output while attending back to it. The natural fit for transformation tasks with a clear input and output — translation, summarization, structured rewriting.
Pretrain, then align
Modern language models are built in stages. The first is enormous and generic; everything after it is comparatively small and shapes behaviour rather than knowledge.
| Stage | What happens | Scale |
|---|---|---|
| Pretraining | Next-token prediction (or masked prediction) over a very large text corpus. Where language competence and world knowledge come from | Trillions of tokens; the overwhelming majority of total compute |
| Supervised fine-tuning | Train on curated instruction–response pairs so the model answers rather than merely continues | Thousands to millions of examples |
| Preference tuning | RLHF, DPO and relatives. Humans rank outputs; the model is pushed toward the preferred ones. Shapes helpfulness, tone and refusal behaviour | Tens of thousands of comparisons |
| Task fine-tuning | Adapt to one narrow domain or format with your own labelled data | Hundreds to thousands of examples |
| PEFT / LoRA | Freeze the base model, train small low-rank adapters. Nearly all the benefit at a fraction of the memory and storage | Often under 1% of parameters |
The standard problem set
| Task | Input → output | Typical approach today |
|---|---|---|
| Classification | Document → label | Fine-tuned encoder, or an LLM prompt when labels are scarce |
| Sentiment | Text → polarity or aspect scores | Same. Aspect-level is much harder than document-level |
| NER | Text → typed spans | Token classification with an encoder; spaCy for speed at volume |
| POS / parsing | Text → tags or a syntax tree | Largely solved for major languages; mostly a linguistic-analysis tool now |
| Coreference | Text → which mentions refer to the same thing | Still genuinely hard; needs world knowledge |
| Question answering | Question (+ context) → answer | Extractive from a passage, or generative with retrieval |
| Summarization | Long text → short text | Generative. Abstractive is fluent and prone to inventing details |
| Translation | Text → text, another language | Encoder–decoder or a large multilingual LLM |
| Retrieval | Query → ranked documents | Hybrid: BM25 plus dense embeddings, then a cross-encoder rerank |
| Generation | Prompt → text | Decoder-only LLM with sampling controls |
| Structured extraction | Text → JSON or table rows | Constrained decoding or schema-enforced LLM output. Very common in production |
The patterns in production right now
Prompting
Zero-shot, few-shot examples, explicit reasoning steps, role framing, output-format specification. Cheap to iterate, hard to make reliable — treat prompts as versioned artifacts with a regression suite, not as text you edit in a text box.
Structured output
Force responses into a schema so downstream code can consume them. Constrained decoding, function/tool schemas, JSON mode. The single highest-leverage technique for putting a language model inside a real system.
Tool use and agents
Give the model functions it can call — search, calculators, databases, APIs — and let it decide when. Powerful and fragile; error compounds across steps, so bound the loop and log every call.
Retrieval-augmented generation
Embed your documents into a vector store. At query time, embed the question, fetch the nearest chunks, paste them into the prompt, and ask the model to answer from them.
It grounds answers in sources you control, updates without retraining, and gives citations. Retrieval quality is the whole game — most disappointing RAG systems are retrieval failures wearing a generation costume. Chunking strategy, hybrid search and reranking matter far more than which model writes the final paragraph.
The hardest part of the field
Classification has clean metrics. Generation does not — there are many acceptable outputs and no reference list of them, which makes automatic scoring approximate at best.
| Metric | Measures | Where it misleads |
|---|---|---|
| Accuracy / F1 | Classification correctness | Fine — just watch class imbalance |
| Exact match | Extractive QA | Punishes correct answers phrased differently |
| BLEU | N-gram overlap with a reference translation | Rewards surface similarity. A perfect paraphrase can score badly |
| ROUGE | Overlap with a reference summary | Same flaw. Correlates weakly with human judgement of quality |
| BERTScore | Embedding similarity to a reference | Better on paraphrase; inherits the encoder's own blind spots |
| Perplexity | Prediction quality | Not comparable across tokenizers, and unrelated to truthfulness |
| LLM-as-judge | A model scoring another model's output | Scalable and increasingly standard, but biased toward length, fluency, and its own family's style |
| Human evaluation | What you actually care about | Slow, costly, needs real annotation guidelines and inter-rater checks. Still the ground truth |
What goes wrong, and why it's structural
What to reach for
Classical & linguistic
- spaCy — fast production pipelines, NER, POS
- NLTK — teaching, linguistic resources
- scikit-learn — TF-IDF and classical classifiers
- gensim — word2vec, topic models
Neural
- transformers — models and pipelines
- tokenizers / tiktoken — the real thing, fast
- datasets — loading and streaming corpora
- sentence-transformers — embeddings for retrieval
- peft — LoRA and adapters
Systems
- Vector stores — FAISS, Qdrant, pgvector
- rank_bm25 — sparse retrieval baseline
- Orchestration — LangChain, LlamaIndex, or plain code
- Serving — vLLM, TGI
- Eval — your own harness, first
# 1 — look at what the tokenizer actually did. do this before anything else. from transformers import AutoTokenizer tok = AutoTokenizer.from_pretrained("bert-base-uncased") ids = tok("Tokenization decides what the model can see.")["input_ids"] print(tok.convert_ids_to_tokens(ids)) # 2 — the strong classical baseline. fit this before you reach for a GPU. from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.linear_model import LogisticRegression from sklearn.pipeline import make_pipeline baseline = make_pipeline( TfidfVectorizer(ngram_range=(1, 2), min_df=2, sublinear_tf=True), LogisticRegression(max_iter=1000, class_weight="balanced"), ).fit(train_texts, train_labels) # 3 — dense embeddings + cosine similarity: the core of every retrieval system from sentence_transformers import SentenceTransformer import numpy as np enc = SentenceTransformer("all-MiniLM-L6-v2") docs = ["the gripper closed on the part", "pasta boils in salted water"] D = enc.encode(docs, normalize_embeddings=True) q = enc.encode(["robot arm picked up the component"], normalize_embeddings=True) print(np.argsort(-(D @ q.T).ravel())) # normalized ⇒ dot product IS cosine # 4 — generation, with the two knobs that matter from transformers import pipeline gen = pipeline("text-generation", model="gpt2") gen("The robot moved through the corridor", max_new_tokens=40, temperature=0.8, top_p=0.9, do_sample=True)
An order that works
| Step | Do this | You'll know it when |
|---|---|---|
| 1 | Tokenize a paragraph with three different tokenizers and diff the output | You can predict which words will split before you run it |
| 2 | Build TF-IDF + logistic regression on a real classification set | You have a number that later work has to beat |
| 3 | Implement an n-gram language model and sample from it | You can articulate exactly what it can't do and why |
| 4 | Train word2vec on a modest corpus; inspect neighbours and analogies | You can find both a delightful analogy and a biased one |
| 5 | Fine-tune a small BERT for classification | You know whether it beat step 2, and by how much |
| 6 | Implement single-head self-attention in NumPy from the equations | Q, K and V stop being letters and become operations |
| 7 | Build a RAG pipeline over your own documents | You've debugged a bad answer down to a retrieval failure |
| 8 | Write an eval set of 100 examples for a task you care about | You can tell whether a prompt change helped or hurt |
| 9 | Fine-tune with LoRA, and compare against prompting honestly | You can say when fine-tuning is worth it — and usually conclude it isn't |
The one-paragraph summary
NLP turns text into numbers, computes over them, and turns numbers back into text. Text is split into subword tokens from a fixed vocabulary, which determines what the model can perceive and what it costs to run. Tokens become vectors — once static and count-based, now contextual and learned — where geometric closeness stands in for semantic similarity. The dominant objective is next-token prediction, which at sufficient scale produces general language competence as a side effect, and the transformer is the architecture that made that scale affordable by replacing sequential recurrence with parallel attention. Systems are built by pretraining broadly, aligning with instruction and preference data, and then assembling retrieval, tools and structured output around the result. The persistent difficulties are not architectural: they are evaluation, hallucination, bias, injection, and the fact that language quietly assumes a world the model has only ever read about.