At a glance
Most introductions to large language models are either marketing or a firehose of architecture diagrams. Sasha Rush, associate professor at Cornell Tech and researcher at Hugging Face, takes a different route in this tutorial, originally delivered for the Harvard Data Science Initiative. His organizing question is not "how do they work" but "which parts of this do we genuinely understand well enough to write down."
His answer is five formulas, each owning one aspect of the technology: perplexity for generation, attention for memory, the matrix multiply for efficiency, Chinchilla for scaling, and RASP for reasoning. Each comes with its history, the intuition behind it, and, importantly, an honest statement of what it fails to explain. The framing he keeps returning to is humility: these are the five places where we can measure, bound, or forecast behavior, and the enormous remainder is still hard.
This page rebuilds all five in the tutorial's order.
The deep explanation
Formula one: perplexity, or how generation is scored
Everything a language model does at generation time comes from a probability distribution over the next token given everything before it. The natural way to score such a model is to ask how surprised it is by real text, and the standard measure is perplexity: the exponential of the average negative log probability the model assigns to each token of a held out corpus. Low perplexity means the text was unsurprising to the model.
Rush anchors this historically in Claude Shannon, whose 1951 experiments on the entropy of printed English had human subjects guess the next letter of a text to estimate how much information English actually carries. That is exactly what a language model is measured against, decades before there were any. The connection between compression and prediction is not a metaphor: a model that predicts text well is a good compressor of text, and the improvements in perplexity across generations of models are improvements in compression.
The honest caveats matter. Perplexity is corpus dependent and tokenizer dependent, so two numbers are only comparable under identical conditions. It is a proxy that correlates with almost everything we care about and measures none of it directly. And it says nothing about whether the model is helpful, truthful, or safe. Still, this is the loss the whole industry optimizes, so it is the right place to start.
Formula two: attention, or how memory works
The second formula is the attention equation from Attention Is All You Need: softmax of queries times keys transposed, divided by the square root of the key dimension, applied to values.
Rush's framing here is the one that makes it click for people with a programming background: attention is a soft dictionary lookup. A hard lookup takes a key and returns exactly one value. Attention takes a query, scores it against every key, normalizes the scores, and returns a weighted blend of all values. It is differentiable, which is the whole point, because it means the lookup can be learned by gradient descent.
That gives the model something the fixed weights cannot: content addressable memory over its own context. Anything in the prompt can be retrieved by matching against it. This is the mechanism underneath in context learning, few shot prompting, and retrieval augmented generation. When people say a model "looked up" a fact from the document you pasted, this equation is what they mean.
He connects it to induction heads, the specific circuit interpretability researchers found that implements copying: if a pattern A followed by B occurred earlier in the context, then on seeing A again, attend back to it and predict B. That single behavior, discoverable inside real models, explains a surprising fraction of few shot learning.
Formula three: the matrix multiply, or where the money goes
The third formula is the least glamorous and possibly the most useful: general matrix multiplication, the GEMM. Essentially every operation in a transformer is a matrix multiply, so the economics of these systems is the economics of that one primitive.
Rush walks through the reasoning that decides what is expensive. A modern accelerator can perform arithmetic vastly faster than it can move data between memory and compute. So the metric that matters is arithmetic intensity: how many operations you perform per byte you move. Multiply a big matrix by a big matrix and you are compute bound, using the hardware well. Multiply a big matrix by a single vector, which is exactly what generating one token for one user looks like, and you are memory bound: you drag the entire weight matrix through memory to do a comparatively tiny amount of arithmetic.
This single fact explains most of the operational strangeness of serving these models:
- Batching is not an optimization, it is the business model. Serving many users at once turns vector multiplies into matrix multiplies and recovers the hardware's actual throughput.
- Training is compute bound and inference is often memory bound, which is why the two have completely different hardware economics.
- Quantization helps more than the arithmetic savings suggest, because it shrinks the bytes moved.
- Key value caching exists so that generating token n does not redo the work for tokens 1 through n minus 1.
If you only remember one thing from the tutorial for practical purposes, Rush suggests it should be this one, because it is the formula that governs cost.
Formula four: Chinchilla, or how to spend a training budget
The fourth formula is the scaling law from the Chinchilla paper, Training Compute-Optimal Large Language Models, by Hoffmann and colleagues at DeepMind in 2022.
The setup: you have a fixed compute budget. You can spend it on a bigger model trained on less data, or a smaller model trained on more data. Which gives the lower loss? Kaplan and colleagues at OpenAI had published scaling laws in 2020 that were read as favoring bigger models, and the industry duly built enormous, undertrained ones. The Chinchilla work fit the loss surface more carefully across many training runs and reached a different answer: model size and training tokens should be scaled roughly in proportion, with a rule of thumb around 20 tokens per parameter.
They then proved it the convincing way, by training a 70 billion parameter model, Chinchilla, on far more data than the 280 billion parameter Gopher and beating it across benchmarks with a quarter of the parameters. A smaller model trained longer wins, and it is also much cheaper to serve, which compounds the benefit.
Rush is careful about the limits. The law describes the pretraining loss, not downstream capability. It assumes a single pass over data of a given quality, and data quality is the variable that has moved most since. And in production, inference cost dominates over the model's lifetime, which pushes practitioners past the compute optimal point toward even smaller models trained on even more tokens than Chinchilla would recommend. That is exactly what the Llama line of models did.
| Formula | Governs | What it lets you actually do | What it does not explain |
|---|---|---|---|
| Perplexity | Generation | Compare models on a fixed corpus, track progress across generations | Whether the model is useful, honest, or safe |
| Attention | Memory | Retrieve anything from the context by content, learn in context | Which head learns what, and why |
| Matrix multiply | Efficiency | Predict cost, size batches, choose quantization | Nothing much, this one is solid engineering |
| Chinchilla | Scaling | Split a compute budget between parameters and tokens | Capability, data quality, inference economics |
| RASP | Reasoning | Say what a transformer can express in principle | Whether training will ever find it |
Formula five: RASP, or what a transformer can express at all
The fifth is the least known and the most interesting: RASP, the Restricted Access Sequence Processing language from Weiss, Goldberg, and Yahav's paper "Thinking Like Transformers." RASP is a small programming language whose primitives correspond to what a transformer layer can do: elementwise operations over a sequence, and a select and aggregate pair that is exactly attention. Write a program in RASP and you have a claim about what some transformer could compute.
This gives a real handle on reasoning. Some algorithms are easy to write in RASP, such as counting occurrences of a token or reversing a sequence. Others are awkward or impossible within a fixed number of layers, because a transformer's depth bounds the number of sequentially dependent steps it can perform in a single forward pass. A task requiring more sequential steps than the model has layers simply cannot be done in one pass.
That reframes several practical mysteries. Chain of thought is not a prompting trick, it is a workaround for a depth limit: by writing intermediate results into the output and reading them back on the next pass, the model converts unavailable depth into available sequence length. It also predicts the failure modes correctly. Problems that need a long chain of dependent steps, such as multi digit arithmetic done in one shot, are exactly the ones that fail without scratch space.
Rush closes on the humility point. RASP tells you what is expressible, not what gradient descent will actually discover, and the gap between those two is where most of the open research questions live.
The shape of the argument
Taken together the five formulas cover the technology's four dimensions and one honest gap. Generation is measured. Memory is understood mechanically. Efficiency is a solved engineering problem with well known trade offs. Scaling is empirically characterized within its assumptions. Reasoning has a theory of what is possible and almost no theory of what will be learned. Anyone claiming more certainty than that about why a model does what it does is, on this account, ahead of the evidence.
Key takeaways
- Perplexity is the objective everything else is downstream of, and it descends directly from Shannon's information theory. It correlates with quality without measuring it.
- Attention is a differentiable soft lookup. It gives models content addressable memory over their context, which is what in context learning and retrieval are built on.
- The matrix multiply is where the money goes. Memory bandwidth, not arithmetic, is the binding constraint at inference, which is why batching, caching, and quantization matter as much as they do.
- Chinchilla says parameters and training tokens should scale together, roughly 20 tokens per parameter, and proved it by beating a model four times its size.
- Inference economics push practitioners past the compute optimal point toward smaller models trained even longer.
- RASP gives a language for what a transformer can express. Fixed depth bounds sequential reasoning, which is why chain of thought works: it trades depth for length.
- The gap between what is expressible and what training discovers is the honest frontier.
Where this sits in the LLM Learning track
This is the third stop, after the full stack tour and the attention chapter, and it exists to install a quantitative frame. Every later part of the track sits on one of these five: the build from scratch videos are an exercise in formula three, the reinforcement learning talk is about what happens once formula one stops being the objective, and the evaluation talk is what you do when you accept that formula one does not measure whether your product works.
Resources mentioned
- Sasha Rush's site, where the slides and written version of this tutorial live, and his open teaching materials on GitHub
- Attention Is All You Need
- Training Compute-Optimal Large Language Models, the Chinchilla paper
- Scaling Laws for Neural Language Models, the earlier Kaplan paper
- Thinking Like Transformers, the RASP paper
- In-context Learning and Induction Heads from Transformer Circuits
- Claude Shannon's Prediction and Entropy of Printed English
- Harvard Data Science Initiative, where the tutorial was given
About this page
This reconstruction was built from the public record of the tutorial rather than from its caption track, because the machine that assembled it had no network route to YouTube. The five formulas, their history, and their limits are here; the verbatim transcript, the timestamped chapter map, and pull quotes are not, and those get backfilled on the next run from a machine that can reach the source. Until then, watch it on YouTube alongside this page.
