How an LLM actually works.

00 · The big picture

One simple objective, run in a loop.

Here's the thing that I think is beautiful, and that most explanations bury: a large language model does exactly one thing. Given a sequence of tokens, it outputs a probability distribution over what token comes next. That's the whole job. The poetry, the code, the reasoning — all of it falls out of doing that one thing extremely well, on an enormous amount of text.

The full pipeline, which the rest of this page unpacks step by step:

How we got here — the seq2seq lineage

"Sequence-to-sequence" is the family name: models that read one sequence and produce another. The history matters because each step fixes a real bottleneck:

• 2014RNN encoder–decoder. Read the input word by word, squash everything into one fixed vector, decode from it. Problem: a whole paragraph through a single vector is a memory bottleneck — early words get crushed.
• 2015Attention (Bahdanau). Let the decoder look back at every encoder state and take a weighted average, with learned weights. The bottleneck disappears. Attention is born as a patch on RNNs.
• 2017"Attention Is All You Need." Throw away the recurrence entirely; keep only attention plus small MLPs. Now every position processes in parallel — perfect for GPUs — and the path between any two words is one step, not n steps.
• 2018+Decoder-only (GPT line). Drop the separate encoder. Prompt and completion live in one stream; the model just keeps predicting the next token. Simpler, and it scales. Almost every modern LLM — GPT, Llama, Claude, DeepSeek — is this shape.

01 · Tokens

The model has never seen a letter.

Before anything touches the network, text becomes tokens — integer ids from a fixed vocabulary of 50,257 entries. Not characters (sequences would be brutally long), not whole words (you'd need millions of entries and still miss "unfriendliness"). The compromise is subwords, built by Byte-Pair Encoding.

BPE in one paragraph: start with raw bytes as your alphabet. Scan a huge corpus, find the most frequent adjacent pair, merge it into a new symbol, add it to the vocabulary. Repeat ~50,000 times. Frequent words ("the", " and") end up as single tokens; rare words split into recognizable pieces — something like un+believ+able. Every possible string is representable, because worst case it falls back to bytes.

• A token is ≈ ¾ of an English word on average — that's the ratio behind every "context window" number you see.
• The space is part of the token:  the (with leading space) and the are different ids. Tokenizers are full of sharp edges like this.
• This is why models are bad at counting letters — they never see letters. "strawberry" arrives as a couple of opaque ids, not as r's you can count.
• Numbers get sliced inconsistently ("1234" might be 12+34), which is part of why arithmetic is hard for pure LLMs.

Output of this stage: an integer array of shape [n], where n = sequence length. No floats yet, no meaning yet. Just ids.

02 · Embeddings & position

From ids to geometry.

Now each id becomes a vector. There's a big lookup table — the embedding matrix W_E of shape 50,257 × 768 — and token id 464 simply selects row 464. That's it. (Mathematically it's a one-hot vector times the matrix, but implementation-wise it's an array index.)

After this lookup, your sequence of n ids is a matrix [n × 768] — n tokens, each a point in 768-dimensional space. Nobody designed those dimensions. They start random and training molds them until geometry encodes meaning: related concepts drift close together, and directions start meaning things (the classic king − man + woman ≈ queen arithmetic falls out, unplanned).

Why position must be injected

Here's a fact that surprises people: attention is permutation-invariant. Shuffle the input tokens and the attention math produces the same set of outputs, shuffled. The architecture literally cannot see word order unless you tell it. So we inject position:

• Learned absolute (GPT-2): a second table of shape 1,024 × 768, one row per position, added to the token embedding. Simple; caps the context at the table size.
• Sinusoidal (original 2017 paper): fixed sin/cos waves of different frequencies. Zero parameters, extrapolates a little.
• RoPE (Llama, most modern models): no addition at all — instead, rotate each query/key pair by an angle proportional to its position. Relative distances fall out of the dot product naturally. Zero parameters, plays well with long context. Currently active: learned table, 1,024 × 768.

03 · The transformer block

A conveyor belt with 12 stations.

The best mental model I know: think of a residual stream — a [n × 768] conveyor belt carrying each token's current representation. Each of the 12 blocks doesn't replace what's on the belt. It reads the belt, computes something, and adds its contribution back:

Two sub-modules, cleanly divided by role:

• Attention is the only place tokens talk to each other — it moves information between positions.
• The MLP processes each token independently — it transforms information in place.

Why the residual connections matter so much

Without them, gradients in training would have to survive multiplication through every layer, and deep stacks would be untrainable — signals vanish or explode. The addition gives gradients an identity highway straight through all 12 blocks. Residuals are arguably the trick that made depth work, here and in the CNNs of 2015.

LayerNorm (LayerNorm here; Llama uses the leaner RMSNorm) rescales each token's vector to a stable magnitude before each sub-module — a volume knob that keeps activations in a healthy range. Modern models normalize before the sub-module ("pre-LN"), which trains far more stably than the original post-LN arrangement.

04 · Attention, in full detail

Every token asks, every token answers.

Slow down here — this is the heart, and it deserves the full shape-by-shape walk. The intuition first: each token emits a query ("what am I looking for?"), a key ("what do I contain?"), and a value ("here's my payload if you want it"). Attention scores every query against every key, and each token collects a weighted blend of everyone's values.

The shape walk, one head

Start from the stream [n × 768]. Three learned projection matrices W_Q, W_K, W_V — each 768 × 768 overall, but sliced per head into 12 heads of width 64:

Why divide by √d_k? A dot product of two random 64-dim vectors has variance that grows with the dimension. Un-scaled, the scores get large, softmax saturates into near one-hot, and gradients die. Dividing by √d_k keeps the scores in softmax's sensitive zone. Tiny detail, load-bearing.

The causal mask

For generation, token 5 must not peek at token 6 — we're trying to predict it. So before the softmax, every future position gets −∞, which softmax turns into exactly zero weight. Each row of the [n × n] matrix can only look left:

Multi-head: 12 conversations at once

One attention pattern per layer would be a bottleneck — position 7 might need syntax from position 6 and a name from position 2. So the 768 dims split into 12 independent heads of 64, each with its own Q/K/V slice, each free to learn a different relationship (heads that track syntax, coreference, induction patterns like "A B … A → predict B"). The 12 outputs of [n × 64] concatenate back to [n × 768], pass through output projection W_O 768 × 768, and get added to the residual stream.

05 · The MLP

Where two-thirds of the model lives.

After attention has moved information around, each token — independently, no cross-talk — passes through a two-layer MLP. It looks almost too simple:

Expand into a wider space, apply a nonlinearity, come back. But don't let the simplicity fool you — count the parameters and the MLPs hold roughly two-thirds of the entire model (4.72M per block vs 2.36M for attention).

The cleanest current understanding of the division of labor: attention routes, MLPs compute and store. Interpretability work keeps finding MLP neurons and directions that behave like stored associations — fire on "Eiffel", push "Paris" into the residual stream. Treat that as a useful working picture rather than settled law; what's certain is that when you scale a model up, most of the new capacity goes here.

06 · Logits & sampling

The model doesn't pick a word. You do.

After block 12, one final normalization, then the unembedding (LM head): a matrix of shape 768 × 50,257 maps each token's vector to one raw score per vocabulary entry. Those scores are logits — for generation you only need them at the last position: a vector of 50,257 numbers. (GPT-2 ties this matrix to the embedding table — same weights transposed, a free 38M-parameter saving.)

Softmax with a temperature turns logits into probabilities:

• Greedy (T→0): always the argmax. Deterministic, and weirdly repetitive in practice — it loops.
• Top-k: keep only the k highest-probability tokens, renormalize, sample.
• Top-p / nucleus: keep the smallest set whose probabilities sum to p (say 0.9) — adaptive, tight when the model is confident, wide when it isn't.

07 · Generation & the KV cache

Why the first token is slow and the rest are fast.

Generation is a serial loop: forward pass → distribution → sample one token → append → forward pass again. Done naively, step n re-processes all n previous tokens from scratch — O(n²) work that repeats itself. The fix is the single most important systems idea in LLM inference:

The KV cache. Notice what step n+1 actually needs from the past: only the keys and values of previous tokens (queries are only needed by the token doing the asking). And those K,V vectors don't change. So: compute them once, keep them in GPU memory, and each new token only computes its own Q/K/V and attends over the stored keys.

• Prefill (your prompt): all prompt tokens process in parallel — one big matrix multiply. Fast per token; this is the initial pause.
• Decode (the reply): one token at a time, each attending over the growing cache. This serial phase is why replies stream.

The memory bill, in fp16 (2 bytes), is worth memorizing as a formula:

This is why long context is expensive even after the model fits in memory: the cache grows linearly with every token, and each new token's attention still scans all of it. And at prefill, the [n × n] score matrices exist per head, per layer — sequence length hits you quadratically there.

08 · Training

The text is its own teacher.

Here's the elegant part: next-token prediction needs no human labels. Take a chunk of text, and every position is simultaneously an input and a test — tokens 1..k are context, token k+1 is the answer. One forward pass over a sequence of n tokens yields n training examples at once (this parallel scoring against shifted targets is "teacher forcing": each position predicts against the real next token, not its own guess).

The loss at each position is cross-entropy — exactly what the MLP Lab on the home page minimizes, at 10⁶× the width:

Then it's gradient descent, industrialized:

• Backprop computes ∂L/∂w for every one of the 124.4M weights — same chain rule as any two-layer net, just deeper and wider.
• AdamW is the optimizer of choice: per-parameter adaptive step sizes from running gradient moments, with weight decay applied separately (the "W"). Its two moment buffers are why optimizer state costs ~2× the model in memory during training.
• Schedule: a short linear warmup (an untrained model produces violent gradients), then a long cosine decay. Plus gradient clipping and mixed precision (bf16 math, fp32 master weights) to survive at scale.
• Batches are measured in tokens — frontier-scale runs push millions of tokens per optimizer step, spread across thousands of GPUs.

Scaling laws — the reason for the "large" in LLM

The empirical discovery that changed everything: loss falls as a smooth, predictable power law in both parameter count N and training tokens D. You can fit the curve on small models and forecast the big one before spending the money. The Chinchilla analysis (2022) added the budget split: compute-optimal training wants roughly D ≈ 20N — 20 tokens per parameter. Then practice moved past it: Llama-2-7B trained on ~2T tokens, nearly 300 tokens per parameter — deliberately "overtrained", because a smaller model trained longer is cheaper to serve forever after.

09 · Base model → assistant

Capability first, behavior second.

A freshly pretrained "base" model is not a chatbot. It's a document-completer. Ask it "What is the capital of Nepal?" and it may answer — or continue with nine more quiz questions, because that's a likely document shape. The knowledge is in there; the interface isn't. Post-training installs it:

• SFT — supervised fine-tuning. Continue training (same cross-entropy) on curated conversations: prompt → good response. Tens of thousands to millions of examples. The model learns the assistant format: answer the question, stop, be helpful.
• RLHF — learning from preference. For subtle qualities ("which of these two answers is more honest/helpful?") humans can't write the perfect answer but can rank pairs. Train a reward model on those rankings, then optimize the LLM against it with reinforcement learning (PPO), with a KL leash so it can't drift too far from the SFT model while reward-hacking.
• DPO and friends: a 2023 simplification that skips the explicit reward model and RL loop, turning preference pairs directly into a classification-style loss. Much simpler pipeline, comparable results in many setups.

Two framings worth keeping: pretraining is where capability comes from (trillions of tokens, ~all the compute); post-training is where behavior comes from (a comparatively tiny nudge). And the chat interface itself is a formatting trick — special tokens template the conversation, and a system prompt is just privileged text prepended at inference.

10 · Where the parameters live

Counting all 124.4M, by hand.

If you can reproduce this table from memory, you understand the architecture. Every row below is computed live from the config — switch models in the corner and audit the arithmetic (V=50,257 · d=768 · L=12 · H=12 · ff=3,072 · ctx=1,024):

Two patterns to internalize. First, the MLPs dominate — about two-thirds of every block. Second, the embedding share collapses with scale: 31% of this model is the embedding table; for the 7B it's ~2%. Scaling up buys compute layers, not vocabulary.

11 · Beginner → expert

The revision checklist.

Fourteen sentences. If every one feels obvious — congratulations, you have the full picture most practitioners carry.

• An LLM is a next-token predictor run in a loop — training builds the predictor, generation is the loop.
• Tokens ≈ ¾ of a word, produced by BPE; the model never sees letters, and leading spaces are part of tokens.
• Embedding = row lookup in a [V × d] table; attention is permutation-invariant, so position must be injected (learned / sinusoidal / RoPE).
• A block is two residual edits: x += Attn(LN(x)), then x += MLP(LN(x)) — the residual stream is a conveyor belt blocks write onto.
• Attention = softmax(QKᵀ/√d_k)·V; the √d_k keeps softmax out of saturation so gradients survive.
• The causal mask sets future scores to −∞ before softmax → exactly zero weight on tokens that don't exist yet.
• Multi-head = split d into H subspaces that attend independently, concat, project with W_O.
• Attention is the only cross-token operation; the per-token MLP (expand ~4×, nonlinearity, project back) holds ~⅔ of all parameters.
• Output = logits over V; temperature/top-k/top-p shape the sampling distribution — decoding is a choice, not knowledge.
• KV cache: store K,V per layer per token (2·L·d·2 bytes in fp16) so decode is O(n) per token; prefill is parallel, decode is serial.
• Training = cross-entropy on shifted targets; every position is an example; perplexity = e^loss.
• AdamW + warmup→cosine schedule + gradient clipping + mixed precision; batch sizes are measured in tokens.
• Scaling laws: loss is a power law in N and D; Chinchilla-optimal is D ≈ 20N; modern models overtrain far past it for cheap inference.
• SFT installs the assistant format; RLHF/DPO installs preferences; pretraining = capability, post-training = behavior.

Where to go from here

The fastest route from "read about it" to "know it": build the small version. Train a tiny MLP in the MLP Lab and watch loss fall; slide kernels in the Conv Lab; probe real QKᵀ softmax in the Attention Lab. Then read "Attention Is All You Need" with this page open, and try reproducing the §10 parameter table for any model on its config sheet. When the numbers match, you're not a beginner anymore.