At a glance
Four hours, one empty file, one trained model. Andrej Karpathy writes the GPT-2 124 million parameter architecture from scratch in PyTorch, loads the released OpenAI weights into it to prove the implementation is correct, then throws those weights away and trains the thing himself. The companion repository is build-nanogpt, which has one commit per step of the video.
What makes it more than an architecture tutorial is the middle two hours. Writing a correct transformer is the easy part and takes about forty minutes. Making it train fast enough to be worth running is the actual craft, and the video walks through every lever in order: numerical precision, kernel fusion with torch.compile, flash attention, choosing dimensions that are nice powers of two, then the full optimization recipe lifted from the GPT-3 paper, gradient accumulation to reach a half million token batch, and finally distributed training across eight GPUs.
It ends with a real run on FineWeb-Edu that beats the published GPT-2 124M numbers, for roughly the price of a nice dinner.
The deep explanation
Writing the network so it matches the real thing
The build starts from the GPT-2 paper and the Attention Is All You Need architecture, with the differences called out: GPT-2 is decoder only, so the cross attention block is gone, and the layer normalizations moved to the input of each sub block with an additional normalization after the final block. Karpathy loads the Hugging Face GPT-2 checkpoint and inspects the tensor shapes to derive the module names and dimensions, then writes his own module tree to match: token embeddings, position embeddings, a stack of blocks each containing causal self attention and a multilayer perceptron with residual connections around both, then the final layer norm and the language model head.
Several details in this section are the kind you only learn by reading real code:
- Weight tying. The token embedding matrix and the output projection share the same weights. That is roughly 38 million parameters saved in a 124 million parameter model, and it reflects a real symmetry: tokens that are similar going in should produce similar logits going out.
- Initialization. The original code uses a normal distribution with standard deviation 0.02, and scales the initialization of layers that write into the residual stream by one over the square root of twice the number of layers. The reason is that residual contributions accumulate, so without the scaling the variance of activations grows with depth.
- The residual stream should be a clean highway. He points out that the residual path adds contributions without passing them through a normalization, which is what lets gradients flow to the earliest layers unimpeded.
Then the correctness check: load the released weights into his implementation, generate from it, and confirm it produces the same sensible completions. Only after that does he delete the weights and start training from random initialization.
Making it fast, in order of return on effort
This is the heart of the video, and the ordering is deliberate: each step is measured, so you see the milliseconds per iteration fall.
Precision. By default PyTorch uses float32 for everything, but the tensor cores in modern GPUs are far faster at lower precision. Turning on TF32 for matrix multiplies with one line gives a large speedup for a small loss of mantissa bits. Going further to bfloat16 with autocast gives another jump. Karpathy explains why bfloat16 rather than float16: it keeps float32's exponent range and sacrifices mantissa bits instead, so no gradient scaler is needed. He is also careful to say what stays in higher precision, notably the accumulations and the parameters themselves.
torch.compile. One line that costs a minute of compile time and delivers a large speedup. His explanation of why is the clearest part of the section: the GPU is fast at arithmetic and slow at moving data between high bandwidth memory and the compute units, and eager mode PyTorch does a round trip to memory for every elementwise operation. The compiler sees the whole graph, fuses those operations into single kernels, and the intermediate values never leave the chip. The example he uses is GELU, where the naive version costs several memory round trips and the fused version costs one.
Flash attention. torch.compile cannot find this one, because it is an algorithmic restructuring rather than a fusion. Flash attention computes the same result while never materializing the full attention matrix in memory, using an online softmax to accumulate results tile by tile. It performs more floating point operations and is dramatically faster, which is a memory hierarchy lesson rather than an arithmetic one. In PyTorch you get it by calling scaled dot product attention.
Nice numbers. The one that feels like a joke and is not. GPT-2's vocabulary size is 50,257, an ugly number with an odd factor. Padding it to 50,304, which is divisible by 128, makes the kernels line up with the hardware's preferred tile sizes and gives a measurable speedup, even though the model is now doing slightly more work. The extra tokens are never used and the network learns to drive their logits to negative infinity. His general rule: look for ugly numbers in your shapes and round them up to powers of two.
| Lever | What it changes | Why it works |
|---|---|---|
| TF32 and bfloat16 | Numerical precision of matmuls | Tensor cores are far faster at lower precision, and bfloat16 keeps float32's exponent range |
| torch.compile | How kernels are scheduled | Fuses elementwise ops so intermediates never round trip to memory |
| Flash attention | The attention algorithm itself | Never materializes the attention matrix; more FLOPs, far less memory traffic |
| Powers of two | Tensor shapes | Ugly dimensions fall off the fast path in CUDA kernels |
| Gradient accumulation | Effective batch size | Reaches a half million token batch on hardware that cannot hold it |
| DDP across 8 GPUs | Data parallelism | Each rank runs its own shard, gradients are averaged before the step |
The optimization recipe, borrowed from the GPT-3 paper
GPT-2's paper is thin on hyperparameters, so Karpathy takes them from the GPT-3 paper, which documents its setup properly and uses a very similar architecture at several scales. The recipe:
- AdamW with betas of 0.9 and 0.95 and epsilon of 1e-8. The lower second beta relative to the default is a small but real detail.
- Gradient clipping at a global norm of 1.0, which stops a single bad batch from producing an enormous update. He recommends logging the norm, because a spike in it is an early warning that something is wrong with the data or the schedule.
- A cosine learning rate schedule with linear warmup, decaying to about ten percent of the peak.
- Weight decay of 0.1, applied only to the two dimensional parameters, which is to say the matrices and embeddings, not the biases and layer norm gains. Decaying one dimensional parameters is a common mistake.
- The fused AdamW implementation, another single flag that avoids launching one kernel per parameter tensor.
Batch size by gradient accumulation. GPT-3's small models used a batch size of half a million tokens, which does not fit on a single GPU. The fix is to run several micro batches, accumulate their gradients, and step once. The subtlety he flags, and which trips people up constantly, is the normalization: because the loss is a mean over tokens, accumulating raw gradients over micro batches gives you a sum rather than a mean, so each micro batch loss must be divided by the number of accumulation steps.
Distributed data parallel. The final step runs eight processes, one per GPU, each with its own shard of the data. PyTorch's DistributedDataParallel averages the gradients across ranks before each optimizer step, overlapping the communication with the backward pass. Only rank zero prints and checkpoints. Combined with gradient accumulation, the accumulation happens locally and the synchronization only fires on the last micro step.
The dataset and the evaluation
Karpathy switches from tiny Shakespeare to FineWeb-Edu, a filtered educational subset of the FineWeb crawl, using the 10 billion token sample. The point he makes is that the dataset improved more than the architecture did since 2019: modern filtered data is far better per token than what GPT-2 trained on, which is why a from scratch run can beat the original.
For evaluation he uses HellaSwag, a sentence completion benchmark with four candidate endings, scored by comparing the average token probability of each completion. He likes it for this purpose because it gives a smooth signal even for small models, where many benchmarks sit at chance until a model is large enough to do anything at all.
The result: the run passes the published GPT-2 124M HellaSwag number partway through, and by the end of a single epoch over the 10 billion token sample it is comfortably above both the GPT-2 124M and GPT-3 124M reference points. The whole run costs on the order of tens of dollars of rented eight GPU time, for a model that was a serious research artifact five years earlier.
He also shows the failure that a long run inevitably produces: a loss spike and a plateau, traced to the data loader's handling of shard boundaries, plus a discussion of whether to permute the shards between epochs. Watching him debug that is arguably the most useful ten minutes of the video, because it is what actual training work looks like.
The closing point about llm.c
Karpathy ends by pointing at llm.c, his implementation of the same training run in raw C and CUDA with no PyTorch at all, which at the time ran the identical training faster than the Python version. The point is not that everyone should write CUDA. It is that a transformer is a fixed and finite set of operations, so a from scratch implementation is a weekend of work rather than a mystery, and knowing that changes how you read the entire field.
Key takeaways
- Writing a correct GPT-2 takes under an hour. Making it train efficiently takes the rest of the day, and that is the real skill.
- Verify a from scratch implementation by loading the released weights into it and checking generation before you train anything.
- Precision, kernel fusion, and flash attention are memory bandwidth wins, not arithmetic wins. The GPU is starved for data, not for math.
- Ugly tensor dimensions cost real speed. Round the vocabulary up to a multiple of 128.
- Take hyperparameters from the GPT-3 paper: AdamW with betas 0.9 and 0.95, clip at 1.0, cosine schedule with warmup, weight decay 0.1 on matrices only.
- Gradient accumulation lets you hit the intended half million token batch on hardware that cannot hold it, and the per micro batch loss must be divided by the number of accumulation steps.
- Better data, not a better architecture, is why a 2024 run beats the 2019 model. FineWeb-Edu is filtered far more aggressively than GPT-2's WebText.
- The economics changed completely. What was a serious research budget in 2019 is a few tens of dollars of rented GPUs now.
Where this sits in the LLM Learning track
This is the second half of the build part of the track, after the tokenizer. Everything the earlier videos describe abstractly, the attention block, the parameter counts, the scaling relationships, appears here as code you can run. It is also the video that makes the efficiency formula from the Five Formulas tutorial concrete: every optimization in the middle section is an arithmetic intensity argument.
Resources mentioned
- build-nanogpt, the repo built in this video, one commit per step
- nanoGPT, the cleaned up successor
- llm.c, the same training in C and CUDA
- GPT-2 paper and GPT-3 paper
- Attention Is All You Need
- FlashAttention
- FineWeb and FineWeb-Edu
- HellaSwag
- PyTorch, torch.compile, and DistributedDataParallel
About this page
This reconstruction was built from the public record of the video rather than from its caption track, because the machine that assembled it had no network route to YouTube. The build order, the optimizations, and the recipe are here; the verbatim transcript, the timestamped chapter map, and pull quotes are not, and those get backfilled on the next run from a machine that can reach the source. Until then, watch it on YouTube with build-nanogpt open beside it.
