Let's build the GPT Tokenizer

At a glance

Andrej Karpathy opens this one with a claim that sounds like an exaggeration and turns out to be an understatement: tokenization is at the heart of much of the strange behavior people attribute to large language models being dumb. Not the model. The step before the model.

Over two and a bit hours he builds a byte pair encoding tokenizer from an empty file, starting at Unicode code points and UTF-8, working through the merge algorithm, then through the details that production tokenizers add: the regex splitting patterns that GPT-2 and GPT-4 use to stop merges from crossing category boundaries, special tokens, and SentencePiece, the different library that Llama and much of the open ecosystem use. The companion code is minbpe.

The payoff is diagnostic. Once you can see what the tokenizer did, a long list of famous LLM failures stops being mysterious.

The deep explanation

The list of crimes

Karpathy front loads the motivation with a catalogue of failures that all trace back to this one preprocessing step:

• Spelling. Models are bad at character level tasks because they never see characters. A long word may be a single token, and asking how many letters it contains is asking about the contents of an opaque symbol.
• String reversal. Same cause. Reversing a string is trivial character work and awkward token work. The reliable fix is to make the model split the string into characters first, then reverse the list.
• Non English performance. More tokens are spent per unit of meaning in languages underrepresented in the tokenizer's training data. That is a triple penalty: the text eats more of the context window, costs more per call, and gives the model less well trained representations.
• Arithmetic. Numbers are split into tokens in ways that have nothing to do with place value. Whether 677 arrives as one token or as 6 and 77 depends on frequency in the tokenizer's corpus, and a model doing digit arithmetic on inconsistent chunks is being set up to fail.
• Python indentation. GPT-2 was noticeably bad at Python, and a large part of the reason is that it tokenized each space individually, so an indented block wasted an enormous number of tokens on whitespace and shredded the context. GPT-4 grouped whitespace and got much better at code partly for that reason.
• Trailing whitespace. Ending your prompt with a space puts the model out of distribution, because in training that space would have been part of the next token rather than a token of its own.
• SolidGoldMagikarp. The most entertaining case. Certain tokens exist in the vocabulary because they were frequent in the tokenizer's training data, notably some Reddit usernames, but appeared almost never in the language model's training data. Their embeddings were therefore never trained and remain near their random initialization. Feed one in and the model behaves erratically, evading, insulting, or producing nonsense. These are unspeakable tokens, and they exist because the tokenizer and the model were trained on different data.
• YAML versus JSON. Structured formats do not tokenize equally. The same data in YAML often costs meaningfully fewer tokens than in JSON, which is a real cost difference at scale.

Starting at the bottom: Unicode and UTF-8

Text in Python is a sequence of Unicode code points, of which there are around 150,000 covering every script and emoji. The naive approach would be to make each code point a token. Two problems: the vocabulary is enormous, and the standard keeps changing, so the vocabulary would not be stable.

The alternative is to encode the text into bytes. UTF-8 is the right choice, being the dominant encoding on the web and backwards compatible with ASCII, whereas UTF-16 and UTF-32 waste space with padding for the common case. Now the vocabulary is a tidy 256 symbols and every possible string can be represented.

But a byte level sequence is far too long. The context window is a fixed number of tokens, and if every character costs one or more, the model sees a very short piece of text. You want a middle ground: a vocabulary bigger than 256 that compresses common sequences into single symbols. That is precisely what byte pair encoding provides.

The byte pair encoding algorithm

The algorithm is simple enough to state in three lines and Karpathy implements it live:

1. Count every adjacent pair of symbols in the training text.
2. Take the most frequent pair and mint a new symbol for it, appending to the vocabulary. Replace every occurrence.
3. Repeat until the vocabulary reaches the size you want.

Each merge shortens the sequence and grows the vocabulary by one. Run it 50,000 times starting from 256 bytes and you get a GPT-2 sized vocabulary. Encoding new text means applying the learned merges in the order they were learned. Decoding means walking the merges backwards to bytes and then decoding the bytes as UTF-8, with a note that you have to handle invalid byte sequences, usually by replacing them rather than crashing.

Karpathy is careful about a distinction people blur: the tokenizer is a completely separate object from the language model, with its own training set and its own training run. It is a translation layer between text and token sequences, and the model never sees text at all.

What production tokenizers add

A pure BPE run over raw text produces merges nobody wants. Karpathy walks through the GPT-2 tokenizer's regex splitting pattern, published with the GPT-2 code, which pre splits the text into chunks before any merging happens. Merges are only ever allowed within a chunk. The pattern separates letters, numbers, punctuation, and whitespace, and it deliberately attaches a leading space to the following word, which is why " hello" is the common token and "hello" is the rarer one.

Two consequences worth noting. The pattern stops merges like "dog." with the period attached, which would otherwise create hundreds of near duplicate tokens for every word followed by every punctuation mark. And it caps numeric merges, though GPT-2 did this imperfectly, which is part of why arithmetic was so uneven.

GPT-4's tokenizer, shipped in tiktoken, changes the pattern in visible ways: the case handling in the regex is made explicit rather than relying on case insensitivity, numbers are limited to at most three digits per token, and whitespace is grouped rather than being one token per space. The vocabulary roughly doubles to about 100,000. Karpathy notes that the exact reasoning behind these choices was never published, so parts of it are reverse engineered from behavior.

Then there are special tokens, which are not produced by merging at all but inserted by the surrounding code: an end of text marker to separate documents in training, and in chat models a set of markers that delimit turns and roles. These get added to the vocabulary with their own ids and are handled by string matching before encoding. He points out the security dimension: if user text containing a special token string is not escaped, a user can inject a turn boundary into a conversation, which is a real class of prompt injection.

SentencePiece, and why the open models differ

SentencePiece is the other library in wide use, chosen by Llama and many open models. Karpathy is direct about it: it does the same job with a much larger and more confusing configuration surface, and its defaults come from a machine translation lineage rather than a language modeling one.

The essential difference is order of operations. tiktoken encodes text to UTF-8 bytes and then merges bytes. SentencePiece merges Unicode code points directly and has a byte fallback option for characters it has never seen, which can be turned off, in which case rare characters map to an unknown token. It also has a mess of normalization and dummy prefix options inherited from translation systems, plus a character coverage setting. His practical advice is to copy the settings that a known good model used rather than reasoning from the documentation.

Choosing a vocabulary size

The final architectural question is how big the vocabulary should be, and Karpathy treats it as a genuine trade off with pressure in both directions.

Bigger vocabulary means shorter sequences, so more text fits in the context window and each forward pass covers more material. But the output layer has one row per token, so the parameter count and the softmax cost grow with vocabulary size, and each individual token's embedding is trained on fewer examples because occurrences are spread thinner. Push it far enough and rare tokens are undertrained, which is the same failure mode as SolidGoldMagikarp on a smaller scale.

He also covers extending a vocabulary after the fact, which is what happens when you add special tokens for a fine tune or introduce gist tokens for prompt compression. You resize the embedding table and the output layer, initialize the new rows, and typically freeze everything else while training only the new parameters. This is routine, not exotic, and it is how a lot of multimodal work attaches new modalities: new token ids with new embeddings, the same transformer underneath.

Key takeaways

• The tokenizer is a separate model with its own training data and its own training run. Mismatches between its corpus and the language model's corpus produce untrained tokens that break the model.
• Text becomes UTF-8 bytes, then byte pair encoding repeatedly merges the most frequent adjacent pair, trading sequence length for vocabulary size.
• Real tokenizers pre split text with a regex so merges never cross category boundaries. That pattern is why leading spaces are attached to words.
• GPT-4 improved on GPT-2 by grouping whitespace, capping digit runs, and doubling the vocabulary, which is a large part of why it writes better code.
• Special tokens are inserted by code, not learned by merging, and unescaped user text containing them is an injection vector.
• SentencePiece merges code points rather than bytes and carries a confusing configuration surface. Copy known good settings.
• Vocabulary size is a real trade off: shorter sequences against a fatter output layer and thinner training per token.
• Karpathy's closing position is that tokenization is a wart everyone would like to eliminate, and until someone does, understanding it is the cheapest way to explain most odd model behavior.

Where this sits in the LLM Learning track

This opens the build part of the track. It comes before the GPT-2 reproduction because tokenization is the first thing in the pipeline and the easiest to get quietly wrong, and because it retroactively explains several of the model behaviors described in the deep dive at the start of the track.

Resources mentioned

• minbpe, the clean reference implementation built in this video
• tiktoken, OpenAI's tokenizer library
• SentencePiece, the tokenizer used by Llama and many open models
• Tiktokenizer for inspecting tokenization interactively
• GPT-2 code and paper
• Llama 2 paper, a worked example of SentencePiece settings
• SolidGoldMagikarp, the original writeup on unspeakable tokens
• Unicode and UTF-8

About this page

This reconstruction was built from the public record of the video rather than from its caption track, because the machine that assembled it had no network route to YouTube. The build order, the algorithm, and the failure catalogue are here; the verbatim transcript, the timestamped chapter map, and pull quotes are not, and those get backfilled on the next run from a machine that can reach the source. Until then, watch it on YouTube alongside this page, with minbpe open in the other window.