Topic

Deep Learning

Everything tagged Deep Learning.

9videos

Deep Dive into LLMs like ChatGPT

Andrej Karpathy's general audience deep dive into the full training stack behind ChatGPT, in one sitting: pretraining on a filtered crawl of the internet, tokenization, what the base model actually is (an internet document simulator), supervised fine tuning that turns it into an assistant, and reinforcement learning as the third and least mature stage. The second half is LLM psychology: why hallucinations happen and how they get patched, why models need tokens to think, why they cannot spell or compare 9.11 to 9.9, and what emergent chains of thought in DeepSeek R1 mean. It ends with where to find and run these models yourself.

AIDeep LearningFeb 5, 2025

Ilya Sutskever

Ilya Sutskever: Sequence to sequence learning with neural networks: what a decade

Accepting a NeurIPS Test of Time award for the 2014 sequence to sequence paper, Ilya Sutskever revisits what that work bet on, autoregression plus scale plus connectionism, and what held up. Then the line the talk is remembered for: pretraining as we know it will end, because compute keeps growing while data does not. Data is the fossil fuel of AI and there is only one internet. He sketches what comes after, agents, synthetic data and inference time compute, and uses hominid brain scaling as evidence that a different scaling regime can exist at all.

AIDeep LearningScienceDec 14, 2024

Andrej Karpathy

Let's reproduce GPT-2 (124M)

Four hours, empty file to trained model. Karpathy writes the GPT-2 124M architecture to match the released weights exactly, then spends most of the video making it fast: TF32 and bfloat16, torch.compile, flash attention, choosing vocabulary sizes with nice powers of two, then the optimization recipe from the GPT-3 paper (AdamW settings, gradient clipping, cosine schedule with warmup, weight decay, gradient accumulation to reach a half million token batch) and finally distributed training across eight GPUs. It ends with a real run on FineWeb-Edu that beats the published GPT-2 124M numbers.

AIDeep LearningJun 9, 2024

3Blue1Brown

Attention in transformers, step-by-step | Deep Learning Chapter 6

Chapter 6 of the 3Blue1Brown deep learning series opens the attention block and walks through it one matrix at a time: queries, keys, the attention pattern and its softmax, masking so tokens cannot see the future, the value matrix and its low rank factorization, and finally multi headed attention running many of these in parallel. Grant Sanderson keeps a running parameter tally against GPT-3 the whole way, so the abstractions stay attached to real numbers. This is the page to read when you want the mechanism itself rather than a metaphor for it.

AIDeep LearningApr 7, 2024

Andrej Karpathy

Let's build the GPT Tokenizer

Karpathy builds a byte pair encoding tokenizer from scratch and argues that tokenization is the root of a startling amount of LLM weirdness: bad spelling, failed string reversal, worse performance in non English languages, arithmetic errors, the Python indentation problem in GPT-2, and the unspeakable SolidGoldMagikarp tokens. It starts at Unicode and UTF-8, works up through the BPE merge algorithm, the GPT-2 and GPT-4 splitting patterns, special tokens, and SentencePiece, then closes on how to choose a vocabulary size.

AIDeep LearningFeb 20, 2024

Sasha Rush

Large Language Models in Five Formulas

Sasha Rush's tutorial takes the opposite approach to hype: pick the five places where large language models can actually be measured, bounded, or forecast, and be honest that the rest is hard. Perplexity for generation, attention for memory, matrix multiplication for efficiency, Chinchilla for scaling, and RASP for reasoning. Each formula gets its history, its intuition, and the limits of what it explains. The result is a compact quantitative frame you can hang everything else on.

AIDeep LearningScienceJan 30, 2024

Jeremy Howard

A Hackers' Guide to Language Models

Jeremy Howard's code first tour of language models for people who want to build with them today. It starts with what a language model is and the three stage recipe he helped popularize with ULMFiT, moves to using the strongest hosted models well, then to the practical engineering: the API and function calling, running open models locally with quantization, retrieval augmented generation, and fine tuning a small model on a narrow task. The through line is that you learn this by running notebooks, not by reading about it.

AIDeep LearningDevOpsSep 24, 2023

FAR.AI

Chris Olah - Looking Inside Neural Networks with Mechanistic Interpretability

Chris Olah lays out the case for treating a neural network as an object to be studied under a microscope rather than a black box to be probed from outside: features as the basic unit, circuits as the connections between them, and the awkward discovery that neurons are usually polysemantic. His answer is superposition, the hypothesis that models pack far more features than they have dimensions by using nearly orthogonal directions for sparse features, and that dictionary learning can pull those features back out. The safety payoff is auditing: finding out what a model is doing rather than asking it.

AIDeep LearningScienceSep 1, 2023

Berkeley EECS

John Schulman - Reinforcement Learning from Human Feedback: Progress and Challenges

John Schulman, co-founder of OpenAI and lead of the ChatGPT RLHF work, gives the clearest available argument for why supervised fine tuning alone teaches a model to hallucinate and why reinforcement learning is the lever that can teach it to hedge instead. The core idea: imitation makes the model assert things it has no internal evidence for, while a reward that scores confident wrong answers below honest uncertainty makes calibration the optimal policy. He then walks through reward models, retrieval and citation as the route to verifiability, and the open problems he had not solved.

AIDeep LearningScienceApr 19, 2023