Creator

Andrej Karpathy

Founding member of OpenAI and former director of AI at Tesla, teaching the whole LLM stack from first principles, in code, with nothing hidden.

3videos

← All videos

Andrej Karpathy

Deep Dive into LLMs like ChatGPT

Andrej Karpathy's general audience deep dive into the full training stack behind ChatGPT, in one sitting: pretraining on a filtered crawl of the internet, tokenization, what the base model actually is (an internet document simulator), supervised fine tuning that turns it into an assistant, and reinforcement learning as the third and least mature stage. The second half is LLM psychology: why hallucinations happen and how they get patched, why models need tokens to think, why they cannot spell or compare 9.11 to 9.9, and what emergent chains of thought in DeepSeek R1 mean. It ends with where to find and run these models yourself.

AIDeep LearningFeb 5, 2025

Andrej Karpathy

Let's reproduce GPT-2 (124M)

Four hours, empty file to trained model. Karpathy writes the GPT-2 124M architecture to match the released weights exactly, then spends most of the video making it fast: TF32 and bfloat16, torch.compile, flash attention, choosing vocabulary sizes with nice powers of two, then the optimization recipe from the GPT-3 paper (AdamW settings, gradient clipping, cosine schedule with warmup, weight decay, gradient accumulation to reach a half million token batch) and finally distributed training across eight GPUs. It ends with a real run on FineWeb-Edu that beats the published GPT-2 124M numbers.

AIDeep LearningJun 9, 2024

Andrej Karpathy

Let's build the GPT Tokenizer

Karpathy builds a byte pair encoding tokenizer from scratch and argues that tokenization is the root of a startling amount of LLM weirdness: bad spelling, failed string reversal, worse performance in non English languages, arithmetic errors, the Python indentation problem in GPT-2, and the unspeakable SolidGoldMagikarp tokens. It starts at Unicode and UTF-8, works up through the BPE merge algorithm, the GPT-2 and GPT-4 splitting patterns, special tokens, and SentencePiece, then closes on how to choose a vocabulary size.

AIDeep LearningFeb 20, 2024