Before a text model runs, a tokenizer converts the input string into a sequence of vocabulary entries. Each entry is represented by an integer ID; the exact pieces depend on that model's tokenizer.

A tokenizer takes a string and produces a sequence of integers. Each integer points to one entry in a fixed vocabulary: the tokenizer's frozen list of known pieces. Modern vocabularies hold anywhere from tens of thousands to a few hundred thousand entries.

The pieces usually aren't whole words. They're subword pieces. Type anything below and watch it split. The colored chips are the tokens, and the small number under each is its ID.

The number under each chip is that piece's fixed vocabulary ID. The model does not receive a separate character array, so operations that humans solve by visually inspecting letters can be awkward when several letters are packed into one token. That is one contributor—not the only contributor—to mistakes on spelling and character-counting tasks.

Tokenization helps explain these failures, but it does not determine them. A model can learn spellings from data, internally recover character patterns, or use a tool. Separating letters can sometimes help by making them explicit units. Number tokenization can likewise make digit-level algorithms harder to learn, although training data, architecture, and tool use also matter.

Different tokenizer vocabularies often draw different boundaries because they were trained with different algorithms, data, normalization rules, and target sizes. Models can also share a tokenizer, so the boundary belongs to the tokenizer—not inherently to the model family.

The practical upshot is compute, context usage, and sometimes price. The same text can occupy different numbers of tokens across APIs. Efficiency also varies by language and script: a tokenizer trained with weak coverage of a script may need more tokens for comparable text, but this must be measured for the specific tokenizer.

Why subwords, and not whole words or single letters? It's a trade-off, and subwords sit in the middle of two bad extremes:

The classic example is asking an LLM how many times a letter appears in blueberry. The model may receive a few multi-character pieces rather than nine explicit character units. A correct answer therefore requires learned spelling knowledge, an internal decomposition, or a tool; the answer is not directly present as a character array in the input.

Capable models often pass this test because training can teach spelling patterns and decomposition strategies, and some products can call code or other tools. Success does not mean tokenization stopped mattering; it means the larger system learned or supplied a way around the mismatch.

Tokenizer implementations vary. GPT-style systems have commonly used byte-level BPE variants; several Llama releases use SentencePiece-based tokenizers. SentencePiece is a tokenizer framework, not a single segmentation algorithm: it supports both BPE and Unigram models. In every case, normalization, pre-tokenization, special-token handling, and fallback rules are part of the specification.

So what is a tokenizer, on disk?

The explainer above describes what tokenization does. A common BPE tokenizer distribution contains a vocabulary, ranked merge rules, and configuration for normalization, pre-tokenization, and special tokens. Other tokenizer families store different artifacts, and some bundle everything into one file.

"the":    464,
"Ġkeys": 8251,
"Ġswim": 9422,
"ming":  2229,
"Ġexploit":14561,
"ation": 341

# ranked, in order:
s  w        → sw
sw im       → swim
m i         → mi
mi ng       → ming
Ġ swim      → Ġswim
at ion      → ation

"bos_token": "<s>",
"eos_token": "</s>",
"model_type":"BPE",
"add_prefix_space":
        true,
"pre_tokenizer":
        "ByteLevel"

The key insight people miss: BPE does not store “swimming → swim + ming” directly. It stores an ordered list of merge rules, and applying them in rank order is the algorithm. Watch it build a word from single characters, one merge at a time:

The prompt is now a sequence of token IDs. The numeric value 8251 carries no ordinal meaning: it selects a learned row in the embedding table. The next stage shows that lookup.

A token ID is discrete: ID 8251 is not numerically “closer” to 8252 than to ID 10. An embedding table maps each ID to a learned vector that the network can transform.

A text decoder typically has an input embedding table with one row per vocabulary entry. Each row has the model's hidden width. The output projection may reuse those same weights—called weight tying—or use a separate matrix, so “every model has exactly one table” would be too strong.

The ID therefore acts as an address. Its selected row is the token's input embedding, learned during training rather than written by a person. Calling it “the meaning” is tempting but incomplete: before attention, that row is the same wherever the token appears and cannot yet encode which sense the surrounding sentence requires.

Pick a sentence, then run the target token through a conceptual stack. The starting point is identical; context changes the trajectory:

The next stage is where tokens start comparing themselves to each other, a step called attention that we'll open up in a moment. That comparison, on its own, is completely blind to order. Without a position signal it can't tell that lock came before picks rather than after, and “lock picks” and “picks locks” are not the same sentence.

The original 2017 Transformer added a deterministic pattern of sine and cosine values to each token embedding. Different dimension pairs use different frequencies, so the same token at positions 1 and 5 enters the first layer differently.

The authors chose sinusoids partly because their values can be computed at unseen positions and relative offsets may be learnable from them. Later decoder models explored learned absolute embeddings, relative biases, RoPE, ALiBi, and other schemes. There is no single transition caused by one proven defect.

So modern models (LLaMA, Mistral, Gemma, Qwen…) mostly switched to Rotary Position Embeddings, RoPE. Instead of adding a position vector, RoPE rotates each token's Query and Key vectors (two per-token vectors that attention compares; proper introductions in the next section) by an angle that grows with position. Drag the slider to move a token along the sequence and watch its vector turn:

RoPE became common in open decoder models because it is parameter-free and gives attention scores an explicit dependence on relative position. Its base frequencies and scaling rules still matter: extending a RoPE model beyond its training window often requires interpolation, frequency rescaling, additional training, or a combination of them.

This is the mechanism the architecture is named after. Inside every layer, attention lets each token look at the other tokens it's allowed to see and decide which ones matter.

For each head, learned projections turn every input hidden state into a Query, Key, and Value. Queries and Keys determine the mixing weights; Values supply the vectors that are mixed. These are useful functional labels, not three symbolic objects stored inside a token.

Each Query is compared with the permitted Keys using a scaled dot product. These learned compatibility scores need not correspond to human semantic similarity. Softmax converts the scores into non-negative weights that sum to 1, and the head outputs their weighted sum of Value vectors.

Here is a deliberately constructed sentence-level example. Click a query token to inspect an illustrative attention pattern:

In a standard causal decoder, position 5 may attend only to positions 1 through 5. During training the later tokens are present in the batch but masked; during generation they do not exist yet. Implementations apply a mask before softmax so disallowed positions receive zero attention probability.

The cost: full self-attention forms a quadratic number of Query–Key comparisons during training or prompt prefill. During cached autoregressive decoding, one new token compares with the existing prefix, so that step grows linearly with current context length. FlashAttention reduces memory traffic and avoids materializing the full score matrix, while exact attention still performs quadratic arithmetic over a full sequence.

Multi-head attention runs several learned Query–Key–Value projections in parallel. Each head can produce a different mixing pattern, and a learned output projection combines their results.

Each parallel pass is a head. The part tutorials often get wrong: a head does not get a literal slice of the token vector. Each head has its own learned projection that maps the full vector down to its own smaller Q, K, V. With 4,096 numbers and 32 heads, each head works in 128 dimensions. But those 128 are a learned projection of all 4,096, not a fixed chunk.

The flow is: project every position into each head's Q/K/V spaces, run attention per head, concatenate the resulting head outputs, and apply a learned output projection. Implementations often compute all projections in fused matrices and reshape them into heads, but the mathematics is equivalent.

Training does not assign human-readable jobs to heads. Researchers can sometimes identify consistent behaviors—copying, positional patterns, or syntactic correlations—but heads may be redundant, polysemantic, or difficult to interpret. A useful attention pattern is evidence about one computation, not a complete explanation of model behavior.

Autoregressive serving normally caches the Keys and Values already computed for the prefix. The KV cache grows with sequence length, layer count, batch size, element precision, and the number of KV heads. Depending on those values, it can become a major serving-memory constraint.

Many decoder models reduce this cost with Grouped-Query Attention (GQA): multiple query heads share fewer Key/Value heads. Multi-query attention (MQA) is the extreme case with one KV head per layer. Toggle between MHA and one example GQA configuration:

After attention mixes information between tokens, every layer has a second step that gets far less attention itself. The feed-forward network (FFN) processes each token on its own: no cross-token mixing at all.

It does three things in order: expand the vector to a larger size, apply a non-linearity, then compress it back down. Press play:

That middle step is the whole reason the FFN exists. A non-linearity is a function that bends its input. The simplest one, ReLU, is almost comically plain: it outputs zero for any negative number and passes positive numbers straight through, unchanged.

With common expansion ratios, dense FFNs hold a large share—and often the majority—of a decoder's block parameters. Probing and intervention studies show that FFN computations participate in factual recall and concept processing, but knowledge is distributed across layers and components rather than living in a tidy database of one-neuron facts.

Some models replace each dense FFN with a sparse Mixture of Experts (MoE): multiple expert FFNs plus a learned router that selects a subset for each token. The whole layer still pays memory and communication costs even though each token activates only part of the expert weights.

Two unglamorous pieces hold the whole tower together. The first makes the model additive instead of replacing: when attention or the FFN runs, its output usually doesn't overwrite the token's vector. It gets added to it.

New vector = old vector + the sub-block's output. Across thirty, fifty, a hundred layers, each contribution accumulates. That running sum is the residual stream, and it has a strange property: the original input embedding still has a direct additive path all the way into the final layers.

The “running sum” framing is useful in mechanistic interpretability because branch outputs can be studied as additions to a shared representation. It is still a coordinate-dependent description of a neural computation, not a literal message bus with independently readable facts.

Residual connections were popularized by ResNet. Identity shortcuts create direct paths for activations and gradients and made much deeper image networks easier to optimize. Transformers adopted the same residual pattern around attention and FFN sublayers.

The second piece is normalization. In a pre-norm decoder, the model normalizes the input to each attention or FFN branch; the branch's output is then added to the otherwise unnormalized residual stream. In a post-norm architecture, normalization happens after the addition. That placement changes optimization and signal propagation.

The 2017 Transformer used post-norm. Many later decoder-only models use pre-norm, which generally improves optimization stability for deep stacks, though both variants and hybrids remain in use. Several open models use RMSNorm, which normalizes by root-mean-square magnitude without centering by the mean.

The output projection can turn every final hidden state into one score per vocabulary entry. During next-token generation, only the logits at the newest position are needed to choose the following token.

If the vocabulary has 100,000 entries, that's 100,000 numbers. They're called logits: raw scores, any size, positive or negative, not yet probabilities. A softmax (the same operation from attention, in a new place) turns them into a probability distribution over what comes next.

A decoder can choose the highest-scoring token greedily or sample from a distribution. Temperature rescales logits before softmax; top-k and top-p restrict which candidates remain eligible. These are inference procedures, not changes to the model's learned weights.

Once a token is chosen, its ID is appended and the decoder computes the next position. A KV cache avoids recomputing earlier Keys and Values, but each new step still runs the model's layers for the new token and attends over the cached prefix.

A common chat interface serializes roles and messages into a token sequence using a chat template. Reserved tokens or structured control tokens mark boundaries. Post-training teaches the model how those markers relate to user, assistant, system, or tool turns; the markers are useful because of that learned behavior, not merely because they appear in the string.

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
why is the sky blue?<|im_end|>
<|im_start|>assistant

Role markers do not create operating-system permissions, but they are more than arbitrary early words: the model is trained to respond differently to role-labelled content. Across API calls, continuity comes from context assembled by the application—conversation history, summaries, or retrieved memory. Providers may reuse cached prefix computation, so “included in the logical context” does not necessarily mean “recomputed from scratch.”

One efficiency technique is speculative decoding. A cheaper draft model proposes multiple tokens, then the target model evaluates them together. With the exact acceptance-and-resampling algorithm, the resulting samples follow the target distribution; approximate variants may trade fidelity for speed. The speedup depends on draft quality and hardware utilization.

Many current language models belong to the Transformer family, but “Transformer” names a design space rather than one fixed blueprint. Public open-weight models expose these choices; proprietary systems disclose only part of their architecture and training recipe.

A common decoder skeleton is tokenization → embeddings → repeated attention and FFN blocks with residual paths and normalization → output logits. Models can differ in attention layout, position method, normalization, dense versus sparse layers, vocabulary, modality encoders, context strategy, and decoding implementation—before training data or post-training even enter the picture.

1. Representation and architecture: tokenizer, width and depth, attention variant, position encoding, FFN or MoE design, modalities.
2. Learned parameters: weights shaped by the data mixture, ordering, objectives, optimization, and compute budget.
3. Post-training and inference: supervised tuning, preference or RL objectives, tool formats, decoding, quantization, and serving choices.
4. The product around the model: system instructions, retrieval, memory, tools, permissions, and safety enforcement.

Several motifs recur across successful open decoder models—pre-norm blocks, rotary positions, gated FFNs, and fewer KV heads—but there is substantial variation and no permanent consensus:

Transformers were preceded by recurrent and convolutional sequence models, and those ideas never disappeared. Comparing them reveals a central trade-off: what information is retained, how much work scales with sequence length, and which computations can run in parallel.

A basic RNN repeatedly updates a hidden state from the current input and previous state. Backpropagation through time multiplies a chain of Jacobians; depending on their singular values, gradients can shrink or grow exponentially with distance. The slider reduces that matrix problem to one scalar factor so the mechanism is visible:

An LSTM adds a gated cell-state path whose update uses elementwise additions and multiplications. The gates themselves still require learned affine transforms and nonlinearities. When the forget gate stays near 1 and other terms are well behaved, gradients can propagate more effectively; this mitigates rather than abolishes long-range optimization problems.

An exact step-by-step recurrence has a sequential dependency: state 1,000 depends on state 999. Standard Transformer training exposes much more parallel work across token positions, while full attention uses quadratic pairwise comparisons and cached decoding stores a growing prefix. Structured state-space models seek linear sequence work with parallelizable training algorithms and fixed-size recurrent inference state.

Transformer blocks can also process representations derived from images, audio, or video. A multimodal system still needs modality-specific preprocessing, encoders or adapters, training data, and objectives; “turn it into tokens” is the interface, not the entire architecture.

Transformer-style blocks now appear across language, vision, audio, and multimodal systems, often alongside convolutional, recurrent, state-space, or modality-specific components. That is broad reuse of an architectural pattern, not proof that one architecture has absorbed every problem.

The active design space includes dense Transformers, MoE variants, state-space models, recurrent hybrids, and multimodal compositions. Rather than predict one winner, it is more useful to recognize the recurring engineering questions:

If you made it here, the next model card or architecture paper you open should read like a parts list you already recognize: every line maps to a mechanism you just poked at. That was the point of building this.

Dense linear projections and matrix multiplications dominate much Transformer compute, alongside attention, normalization, elementwise operations, memory movement, and communication. Matmuls map well to accelerator hardware; “everything is a matmul” is a useful performance shorthand, not a literal architecture description.

Recurrent networks process sequence state serially across time, which limits parallelism during training. Transformers replace that recurrence with attention and position-wise blocks that can process training positions in parallel. Autoregressive generation remains serial across newly generated tokens.

Recurrence creates a sequential dependency across positions; attention creates a direct path between permitted token pairs within a layer. That shorter path can help optimization, but attention also introduces quadratic score structure and does not guarantee that information is retained or used.

For a square N×N matmul, arithmetic grows as N³ while the three matrices contain O(N²) elements, so idealized arithmetic intensity grows with N. Actual traffic depends on tiling, cache reuse, dtype, layout, and matrix shape; larger operations can also exceed fast memory or require distribution.

A first performance model separates compute throughput, data movement, and memory capacity. Real latency also includes kernel launch, synchronization, networking, scheduling, queueing, and software overhead, so these are three core constraints rather than the only three causes.

The first is compute. Peak FLOP/s depends on dtype, sparsity mode, and operation shape; sustained application throughput is lower. Dividing work by an achieved—not marketing peak—rate gives a useful lower-level estimate.

The second is memory bandwidth: data movement through a particular level of the memory hierarchy. HBM bandwidth is one ceiling; caches, interconnects, access patterns, and reuse determine actual bytes transferred and achieved bandwidth.

The roofline lower bound uses the larger of ideal compute time and ideal data-movement time, assuming useful overlap. Dependencies and overhead can prevent full overlap, so measured time may be higher.

Here's the single most useful idea in this whole page. Take any operation and ask: how much math does it do per byte it has to move? That one ratio predicts whether your expensive chip is working hard or sitting idle waiting for data.

That ratio is arithmetic intensity, FLOPs per byte transferred at a stated memory level. A device's ideal crossover is peak FLOP/s divided by peak bytes/s for the chosen dtype and mode. Achieved crossovers differ with kernels and reuse, so no single threshold applies to “an H100” or “a TPU” without more qualifiers.

For a weight-dominated matrix-vector or thin matrix multiply where weights are fetched once for a batch, intensity can scale roughly with batch tokens. The exact formula depends on all matrix dimensions and whether weights, inputs, and outputs are reused from cache.

You can estimate almost everything about a model's cost from a handful of numbers on a napkin. Let's build the calculator, then read off the two rules everyone in the field has memorized.

The calculator assumes a dense decoder layer with a three-projection gated MLP (≈3DF) and full-size Q/K/V/O projections (≈4D²). A two-matrix FFN, GQA/MQA, MoE, biases, embeddings, tied output heads, and architecture-specific blocks change the count. In common gated configurations the MLP is often larger, not universally so.

A common dense-Transformer estimate is 6ND training FLOPs for N parameters and D tokens. It approximates multiply-add work for parameterized linear layers and their backward passes. Attention's sequence-dependent work, embeddings, normalization, optimizer math, recomputation, sparsity, and hardware utilization require separate accounting.

Full self-attention scores all permitted query–key pairs, giving a quadratic term in sequence length during training or prefill. Whether that term dominates depends on architecture, phase, kernel, and hardware; the calculator makes one set of assumptions explicit.

For bidirectional full attention the score grid is T×T; causal attention uses roughly half its entries but has the same asymptotic order. Projection and feed-forward arithmetic grows linearly with T at fixed width. Which is larger at a given length follows from the actual block dimensions.

A naive implementation materializes the score and probability matrices in high-bandwidth memory. FlashAttention changes the tiling and online softmax schedule to reduce that traffic without changing full attention's asymptotic arithmetic.

Autoregressive serving commonly stores past attention keys and values so they do not have to be recomputed at every decode step. This KV cache grows with retained sequence length and can limit concurrency, though weights and other buffers may remain the dominant memory cost in other regimes.

With dense causal self-attention, a new token can attend to retained prior positions. The cache stores keys and values per layer and per KV head; query heads do not each require a distinct cache under GQA or MQA. Sliding windows, recurrent state, cache eviction, and compression change the growth pattern.

It is worth slowing down on what the cache actually replaces, because the mechanics explain half of modern serving. Generation has two phases. Prefill runs the whole prompt through the model in one parallel pass and stores every token's Key and Value. Decode then produces one token at a time, and each new token only computes its own Query, Key and Value: the fresh Q looks back across the stored K and V, and the new K and V get appended for the next step. Without the cache, every single step would re-run the full forward pass on the entire prefix, just to regenerate numbers the model already produced.

Identical token prefixes under the same model configuration produce reusable prefix state. Serving systems can exploit that state within an explicitly scoped cache. PagedAttention primarily addresses KV-cache memory fragmentation and sharing through paged blocks; prefix caching is an additional policy built on such storage. Radix-tree systems provide another lookup and reuse strategy. A commercial “cached input” feature may use related ideas, but its internal implementation cannot be inferred from pricing alone.

A valid prefix-cache hit can save both KV storage and repeated prefill computation. Actual savings depend on block granularity, cache residency, isolation scope, model version, token-exact matching, and whether the provider exposes the benefit in latency or price.

Some serving architectures disaggregate prefill and decode to tune and scale them separately, transferring KV state between workers. This is a workload-dependent systems choice: transfer latency, bandwidth, locality, queueing, and failure handling can outweigh the benefit at smaller scales.

Cross-tenant reuse can create timing or state side channels if cache hits are externally distinguishable. Isolation by tenant, access-controlled cache keys, conservative reuse rules, and constant-time or noise-aware designs reduce risk. Whether a particular commercial service is vulnerable or partitioned must be established from its documentation and testing, not assumed.

A sparse Mixture of Experts can increase total parameter capacity while activating only a subset of feed-forward experts for each token. Active expert FLOPs may grow more slowly than total parameters, but routing, communication, memory traffic, and shared layers remain part of the bill.

In a common sparse MoE layer, multiple feed-forward “experts” replace or augment a dense FFN. A learned router selects a small subset per token. Expert matmul work follows the selected subset; total token cost still includes routing, shared components, dispatch, and combining outputs.

Large training and serving jobs distribute parameters, activations, optimizer state, experts, or examples across accelerators. Data, tensor, pipeline, and expert parallelism are important patterns—not an exhaustive list, and not required for every model called “frontier.”

The goal has a name, strong scaling: add twice the chips, get twice the throughput. In a perfect world that's a straight line. The real world bends it, because every split adds a tax: chips now have to talk to each other, passing partial results back and forth, and that communication isn't free.

Distributed systems build many communication patterns from collectives. Broadcast, all-gather, reduce-scatter, and all-reduce cover much of the vocabulary used in model-parallel diagrams, alongside all-to-all, point-to-point sends, and topology-specific operations.

Think of an array split into four chunks across four GPUs. Each collective is a different answer to "who has what, before and after":

An all-reduce can be implemented as reduce-scatter followed by all-gather; rings are one common algorithm. Trees, hierarchical collectives, topology-aware variants, and hardware offload may be better for other message sizes and networks. The animation shows a four-device ring.

Conventional replicated data parallelism keeps a copy of parameters, gradients, and optimizer state on each worker. The displayed 16 bytes per parameter is one mixed-precision Adam accounting example—not a property of all training stacks.

ZeRO's observation: if the gradients get summed across GPUs anyway, why does everyone need to store everything? Shard the state instead, with each GPU keeping 1/M of it, in three escalating stages:

ZeRO shards storage, but every GPU still computes every layer. When a single layer's matmul is too big, or you want more chips on one token, you split the math. And where you cut the matrix decides how much the chips have to talk.

Split the up-projection by columns and each GPU owns complete hidden units: the nonlinearity applies locally, no talking needed. Split the down-projection by rows and each GPU produces a partial sum of the full output: one all-reduce at the end and you're done. Two matmuls, one collective. (With SwiGLU, Wgate and Wup both get column-sharded so the elementwise gate stays local.) Any other cut forces a collective in the middle of the MLP, because you can't apply a nonlinearity to a partial sum.

Pipeline parallelism cuts along the other axis: different GPUs hold different layers. The catch: layer 2 can't start until layer 1 finishes, so a naive pipeline is just four GPUs taking turns being awake. The fix is to chop the batch into micro-batches and stream them through:

Numerical formats trade range, precision, storage, bandwidth, and hardware throughput. Halving storage bits can halve a tensor's raw bytes, but end-to-end memory and speed do not automatically improve by 2×: kernels, scaling metadata, packing, conversions, alignment, and hardware support matter.

Why the range/precision split matters is best seen with one concrete number. A typical gradient update: weight 1.0 plus gradient 0.0001.

The rounding example motivates higher-precision accumulation, but it does not prescribe one universal layout. Mixed-precision systems may keep master parameters or optimizer states in FP32, use BF16/FP16/FP8 compute inputs, and accumulate selected operations at higher precision:

GPTQ and AWQ are post-training weight-quantization methods. GGUF is a file format and quantization ecosystem, not itself the same algorithm. Raw 4-bit payload is half a byte per parameter, but scales, zero-points, higher-precision tensors, metadata, runtime buffers, and KV cache add overhead. Whether a model fits or runs well on a laptop or phone depends on memory, kernels, context, architecture, and acceptable quality; “4-bit is free” is not a safe general claim.

Serving repeatedly schedules requests under latency, throughput, memory, reliability, and cost constraints. Small-batch decode is often bandwidth-bound, while continuous batching and cache management improve utilization. The optimal policy depends on service-level objectives and workload.

A slot is a useful scheduler abstraction for an in-flight sequence and its state. Batched decode can reuse weights across active sequences, but advancing B sequences does not cost the same as one: activation, attention, KV traffic, sampling, and communication grow with batch, while weight traffic is better amortized.

So the scheduling question is what happens when a slot frees up. Static batching, the naive way, loads four requests, runs until all four finish, then swaps in the next four: whoever finishes early leaves a slot idling while the longest request grinds on. Continuous batching refills a freed slot on the very next decode step. Same eight requests, same hardware, one clock. Watch them race:

Batching targets aggregate throughput. Speculative decoding can reduce target-model decode steps by drafting multiple tokens and verifying them in parallel. Verification is not literally the cost of one token; speedup depends on draft cost, acceptance, verification shape, hardware, and batching.

Every number on this page is a version of one question: is my expensive hardware doing math, or is it waiting? Hold onto that and the rest is bookkeeping.

Quality, latency, throughput, memory, energy, reliability, and total cost are separate axes. A quality gain is not automatically irrelevant when efficiency falls; the right tradeoff depends on the application. These estimators help form hypotheses that must be checked with profiles and end-to-end benchmarks.

A model learns the distribution represented by its training mixture, not “the internet” in the abstract. Before optimization begins, builders extract text, filter it, remove duplicates, reduce contamination, and choose how often each source appears. Those choices shape both capability and failure modes.

FineWeb, one published example, was produced from 96 Common Crawl snapshots and contains roughly 15 trillion GPT-2-tokenizer tokens after processing. That is one documented pipeline—not a template every lab follows. The interactive below is a conceptual pipeline; its percentages are illustrative, not FineWeb measurements.

Different stages target different problems. Extraction separates main text from HTML and boilerplate. Language identification estimates a document's language. Quality filters combine inexpensive heuristics or learned classifiers to reject spam, malformed text, and low-value material. These filters encode preferences and can remove useful dialects, languages, or domains, so their error rates matter as much as their throughput.

Deduplication handles exact and near copies. One common approach represents word shingles with MinHash signatures so similar documents can be found without comparing every pair. Published experiments show that deduplication can reduce memorized emission and evaluation contamination, but the effect depends on the corpus, matching threshold, model, and prompting setup.

Filtering is not a monotonic “more is better” knob. Repeated material may be boilerplate, but it may also be a widely mirrored high-quality reference. A near-duplicate threshold can erase legitimate variants or leave templated spam. Builders therefore evaluate filters downstream instead of treating removal rate as a quality score.

Finally comes the mixture. Web pages, code, books, math, papers, and multilingual text can be sampled at different rates. FineWeb-Edu is a concrete result: a learned educational-quality filter improved several benchmark results relative to other FineWeb subsets in the authors' experiments. That does not establish a universal recipe; mixture effects depend on the tokenizer, model, schedule, and evaluation.

A decoder-only language model is commonly pretrained with next-token cross-entropy. Given earlier tokens, it assigns a probability to each vocabulary item. The observed next token supplies the target. All positions in a training sequence can be scored in parallel under a causal mask; training is not literally a hide-and-guess game run one token at a time.

How wrong a guess is gets turned into a single number, the loss. If the model put probability p on the correct token, the loss for that token is −ln(p). Nail it with p≈1 and the loss is near zero. Put p≈0 on the right answer and the loss explodes. Drag the confidence and watch:

Cross-entropy has a useful population-level decomposition: unavoidable uncertainty plus avoidable model mismatch. It explains why even a model matching the true conditional distribution can have positive expected loss. It does not tell us that every observed training curve has reached that theoretical limit.

For a specified context, imagine a true next-token distribution p and a model distribution q. The entropy H(p)=−Σp·ln p measures uncertainty in that chosen distribution; cross-entropy scores predictions from q when outcomes follow p. The demo uses a made-up five-token distribution so the identity is visible.

KL(p‖q) is the excess expected log loss caused by using q instead of p; it is zero only when the distributions match wherever p has support. H(p) is irreducible for that data-generating distribution and context representation. In practice, the distribution is unknown, corpora are finite and changing, tokenizers differ, and training loss can approach zero by memorization. A fitted scaling-law constant should therefore not be casually identified with “the entropy of human language.”

At each scored position, the language-model head produces one logit per vocabulary item. A numerically stable cross-entropy implementation combines log-softmax with the negative log-likelihood; it need not materialize a probability vector and then take its logarithm as separate operations.

Start with what actually comes out. The final layer emits a logit for every possible next token: an unbounded score, positive or negative, that says how strongly the model favors that token right now. Bigger means more favored, but the values have no fixed scale and don't sum to anything. You can't call the top one a "63% chance" until you normalize.

Softmax maps the logits to a categorical distribution: every value is positive and the set sums to 1. A logit difference is a log probability ratio, so a 1.1 lead corresponds to odds of about e^1.1≈3 to 1 between those two tokens, independent of the other logits. These probabilities are model predictions, not guaranteed real-world confidence.

Sequence likelihoods multiply conditional probabilities; log-likelihoods turn those products into sums and are numerically easier to work with. Stable implementations use the log-sum-exp identity rather than computing tiny probabilities first. Log space greatly reduces underflow risk, though finite-precision computation is never literally immune to numerical error.

The negative log-probability assigned to the observed next token is its loss contribution. Contributions are reduced across non-masked tokens and examples, then differentiated through the network. Some training systems add other losses, masks, weights, or regularizers, so next-token cross-entropy is central rather than universally the whole signal.

The loss depends on every parameter, so it can be pictured as a surface in a very high-dimensional space. The picture is useful, but real training uses a noisy gradient estimated from a minibatch, optimizer state, regularization, and a schedule—not a ball following one smooth global landscape.

What you can measure is the slope: for each weight, which way is downhill and how steep. That's the gradient. Step every weight a little in the downhill direction, recompute, step again. The size of each step is the learning rate. Too small and you crawl; too big and you fly past the valley and bounce up the far side. Drag it and drop the ball:

One number, the loss, has to improve billions of weights. The trick is to ask, for every weight: if I nudged you a hair, which way would the loss move, and how much? That answer is the weight's gradient, and backpropagation computes all of them in a single backward sweep using the chain rule.

First a forward pass runs the input through the layers to produce a prediction and its loss. Then the backward pass walks the same path in reverse, handing each layer its share of the blame. Press play:

Gradients estimate a local descent direction; the optimizer turns them into updates. Plain minibatch SGD can oscillate in an ill-conditioned quadratic. Adam-family optimizers are common in language-model training, though implementation details and alternatives vary.

Adam tracks exponential moving averages of gradients and squared gradients. The first smooths direction; the second rescales coordinates based on recent gradient magnitude. Toggle between two toy optimizer paths:

The learning rate scales optimizer updates. Values that are too large can destabilize training; values that are too small may waste a fixed compute budget. The usable range depends on optimizer, batch size, parameterization, model scale, and schedule.

Warmup followed by cosine decay is one common schedule, not “the standard.” Other successful runs use linear decay, inverse-square-root schedules, plateaus, or warmup-stable-decay. The chart shows the shape of one cosine schedule; its verdict labels are illustrative.

Finite floating-point formats have limited range and precision. Overflow, underflow, and accumulated rounding can produce infinities or NaNs and derail optimization. Softmax is a compact place to see the issue, though unstable runs can have many causes.

A naive softmax can overflow when it directly exponentiates a large positive logit. Subtracting the maximum logit leaves the mathematical result unchanged and makes every exponent's argument non-positive. This prevents overflow in that exponentiation; it does not, by itself, guarantee an entire training run is stable.

Initialization is chosen so signals and gradients begin at workable scales. Gaussian assumptions make several calculations easy, but real models also use uniform, truncated-normal, orthogonal, constant, and architecture-specific initializers. Treat this section as a set of conditional calculations, not a claim that every tensor is bell-shaped.

For a linear layer with independent zero-mean inputs and weights, output variance depends on fan-in and weight variance. Xavier and He initialization preserve selected forward or backward variances under different activation assumptions; they do not make gain “exactly 1” through an arbitrary Transformer. Residual branches, normalization, nonlinearities, and depth motivate additional scaling choices.

Why use a Gaussian approximation at all? Sums of many sufficiently independent, well-behaved contributions can approach a normal distribution. Neural activations violate those assumptions in important ways: coordinates share inputs and weights, attention couples positions, and nonlinearities reshape distributions. Mean-field or Gaussian calculations can still be useful approximations at initialization, but “the entire network is Gaussian” is false.

For a vector with d independent standard-normal coordinates, its norm concentrates near √d. Two independent isotropic random directions have mean cosine zero with typical scale about 1/√d. This is a statement about that random model—not every pair of learned representations.

High-dimensional spaces can contain many nearly orthogonal directions. Mechanistic-interpretability work uses superposition to study how more features than dimensions may be represented, but high dimensionality alone does not prove that a particular model stores a particular number of concepts or that interference is negligible.

If every vocabulary logit is equal, softmax is uniform and each target has loss ln V: about 11.51 for 100k tokens or 10.37 for 32k. A carefully initialized model may start near this value, making it a useful sanity check. Unequal logits, tied embeddings, biases, masking, and nonuniform target frequencies can shift the initial loss, so a mismatch is a debugging clue—not proof of a bug.

Training changes parameter and activation distributions, and some Transformers develop heavy-tailed or unusually large activation channels that complicate quantization. Non-Gaussianity is not a map of “where learning happened”: nonlinearities make distributions non-Gaussian even before useful learning, and different parameterizations produce different statistics.

Across controlled families of language-model experiments, validation loss has often been fit well by power laws in parameters, data, or compute over the measured range. Teams use smaller runs to forecast larger ones—but extrapolation is only as trustworthy as the shared architecture, tokenizer, data distribution, optimizer, and range.

The interface draws one hypothetical saturating power law. Its constants are authored for illustration and are not a forecast for any model family:

Natural data occupies a highly structured subset of all mathematically possible inputs. Researchers often study that structure with the manifold hypothesis. It is a useful lens for representation learning, not proof that “everything a model knows” lies on one literal smooth sheet.

The combinatorics motivate the idea. A 1,000-token string over a 100,000-token vocabulary has 100,000¹⁰⁰⁰ possible forms, while only a tiny subset resembles language people produce. “Low-dimensional,” however, depends on representation and scale, and discrete token sequences are not automatically a smooth Euclidean manifold.

Structure lets models generalize from finite samples: nearby or related observations can share statistical regularities. The animation is a geometric analogy, not a derivation of why language models learn, and it is not an assumption required by the empirical scaling-law equation in the previous section.

Real data may have multiple modes, intersections, discrete factors, noise, and changing dimensionality. Whether a dataset satisfies a tidy manifold definition is an empirical question. “Off-distribution” also cannot be reduced to perpendicular distance from a known sheet, because in practice neither the true distribution nor its support is known.

Models often perform worse under distribution shift, but distance, familiarity, and confidence are not interchangeable. A model can fail on an in-distribution rare case, generalize to a novel combination, or be uncertain on familiar-looking text. The safe operational lesson is to evaluate the actual domains and perturbations that matter.

Different models can learn partially aligned representations, and adversarial examples sometimes transfer across models. These are observations with competing explanations—not proof that decision boundaries coincide because every model recovered one underlying manifold. Representation similarity also depends on the layer, metric, task, data, and allowed transformation.

Adversarial examples show that small changes under one metric can cause large prediction changes. They need not be “perpendicular to the manifold,” and the manifold hypothesis alone does not guarantee the exact failure pattern. Robustness requires an explicit threat model: which changes are allowed, what must remain invariant, and what the attacker knows.

In REINFORCE, the model samples an action or sequence, receives a return, and estimates a policy gradient with ∇log π weighted by an advantage. This resembles weighted log-likelihood algebraically, but the targets are sampled from a policy and the estimator has on-policy and credit-assignment concerns that ordinary supervised fine-tuning does not.

A baseline can reduce estimator variance without changing its expectation when it is independent of the sampled action under the relevant conditioning. In this single-prompt toy batch, centering four positive rewards makes below-baseline samples contribute negative advantages:

Reusing samples after the policy changes introduces an off-policy mismatch. PPO uses probability ratios and a clipped surrogate objective to limit incentives for some overly large changes. Clipping does not universally “turn the gradient off”: which branch is active depends on the advantage sign and ratio.

Pretraining produces a conditional sequence model, not a product-ready assistant. Post-training changes its behavior with demonstrations, preference data, rewards, safety data, tool-use trajectories, and other objectives. It can teach formats and policies and can improve task performance; it does not cleanly separate “behavior” from “knowledge.”

A common pipeline begins with supervised fine-tuning (SFT) on curated responses, then optimizes human preferences, model-generated preferences, verifiable outcomes, or a mixture. The buttons compare families of methods, not mandatory sequential stages:

Outcome-based training can reinforce visible behaviors such as decomposing a problem, checking work, or trying another approach. Calling every generated rationale the model's private “thought” is risky: a trace can be useful, unfaithful, strategically shaped by training, or simply wrong.

Inference systems can spend additional compute by generating longer traces, sampling multiple candidates, using search, checking answers, or calling tools. Accuracy often improves on suitable tasks when the method and budget are chosen well, but more tokens alone are not guaranteed to help. The curve below is illustrative, not benchmark data.

Most gradient-based training can be organized around a recurring loop: compute an objective, differentiate it, update parameters, and repeat. Different stages vary substantially in how data is generated, what objective is optimized, how samples are weighted, and when reference or reward models run.

A forward pass turns a batch of text into a loss; backprop turns that loss into a gradient for every weight; the optimizer turns the gradient into an update; and then it happens again with the next batch. Watch one turn of the crank:

A deployed assistant combines one or more trained models with software for context construction, retrieval, tool execution, state, safety checks, permissions, and user experience. Product details differ and are partly undisclosed. When a product remembers a preference or fetches a current page, that state is commonly outside the model weights and supplied at inference time.

Failures must be located before they can be fixed. Disclosure can come from context assembly, retrieval, logs, tool output, cross-tenant state, or memorized training data. An external action requires some execution path, but the model may influence it. The model is a component inside a larger threat model.

At inference time, a model's output depends on its parameters and supplied inputs. Retrieval-augmented generation finds external material and conditions generation on selected results. This can provide fresh or private evidence without changing weights, though retrieval is not always the right substitute for fine-tuning, tools, or structured queries.

Two things follow, and both matter for the security section later:

A standalone language-model inference call has no external side effect merely because it emits a tool-shaped sequence. In a tool-enabled system, the model proposes a structured call and an orchestrator validates, authorizes, executes, and returns a result. Some end-to-end products hide that boundary, but it still exists somewhere in the system.

This gap between proposing and executing is the single most important thing about agents. It's where permissions live, where confirmation prompts live, and where a well-built system stops a bad idea. A model that "tried to delete the database" only did so if the software on the other side of the call agreed to.

“Agent” is an overloaded systems label, not a distinct neural architecture. A common pattern iterates between model inference, state updates, tool calls, and stopping logic. Some agents plan first, run parallel workers, use deterministic workflow nodes, or require approvals, so the displayed loop is a minimal reference pattern.

The blast radius is strongly shaped by granted capabilities, credential scope, data access, network reach, approvals, and rollback. A text-only response can still cause harm if a person or downstream system trusts it, while a well-sandboxed shell may have bounded effects. Risk belongs to the whole use context.

An application can label content by role and provenance: platform policy, developer instructions, user requests, quoted text, retrieved documents, and tool results. Some chat models are trained to follow a documented priority order when instructions conflict. The exact hierarchy is model- and platform-specific; it is not an inherent law of all LLMs.

Prompt injection occurs when untrusted content changes model behavior contrary to the application's intent—for example, when instructions embedded in a retrieved page influence a tool-using assistant. It may exploit ambiguity in roles, parsing, transformations, or model behavior; source markers help but do not create a deterministic trust boundary.

The danger is a chain, and the model is only one link in it. Watch where an injected instruction goes — and flip the application's controls to see what actually stops it:

Leave "model follows the injection" switched on — assume the model can be fooled — and notice that the application controls on the right can still stop the damage cold. That's the durable lesson:

A standard language-model objective rewards probability assigned to observed continuations, not direct agreement with an external database of facts. Post-training can reward factuality, citation, abstention, or verified outcomes, yet unsupported or incorrect generations remain possible. “Hallucination” groups together several failure modes and does not reveal intent.

A few things worth correcting, because they're the usual over-simple story: