At a glance
This talk, given at the UC Berkeley EECS colloquium in April 2023, is the clearest public explanation of why chat models lie and what can be done about it, delivered by the person who was leading the work. John Schulman co-founded OpenAI, invented PPO and TRPO, and ran the team behind ChatGPT and its reinforcement learning from human feedback.
His central argument is unintuitive and, once you see it, hard to unsee. Supervised fine tuning, which is imitation of good answers, actively teaches a model to hallucinate. Not accidentally: the training signal itself instructs the model to assert things whose truth it has no internal evidence for. Reinforcement learning is the lever that can fix this, because a reward that scores confident wrong answers below honest uncertainty makes calibration the optimal behavior rather than a virtue you hope for.
He then walks through how reward models are built, why retrieval and citation are the practical route to verifiability, and which problems he had not solved. Coming from 2023, the open problems section reads like a research agenda that much of the field then followed.
The deep explanation
The two knowledge bases
Schulman starts by separating two things that are easy to conflate. A pretrained model has, encoded in its weights, some approximation of the facts it absorbed. Call that the model's knowledge base. Separately, the fine tuning dataset that turns it into an assistant has its own facts, written by human labelers who know things the model does not.
Now consider what happens during supervised fine tuning. The training example is a question and an ideal answer. The model is trained to produce that answer. If the answer contains a fact the model has no internal representation of, the gradient does not teach it the fact. It teaches it the behavior: when asked a question of this shape, produce a confident specific answer. The model learns the form of knowing rather than the content.
That is the mechanism. Hallucination is not a glitch, it is the faithful execution of a policy that imitation training installed. And the symmetric error is just as real: if you overcorrect and fill the fine tuning set with "I do not know," you teach the model to withhold information it actually has, which makes it useless in a different way.
Why reinforcement learning is the right shape of tool
The reason reinforcement learning fixes this is that its training signal is defined over the model's own outputs rather than over a target someone else wrote. The model produces an answer, the answer is scored, and behavior that scores well is reinforced. Crucially the scoring can depend on whether the answer was right, which imitation cannot.
Schulman makes the point with a scoring argument that is worth spelling out. Suppose a correct answer earns positive reward, a confidently wrong answer earns a larger negative reward, and saying you are not sure earns zero, or slightly less than a correct answer. Then the optimal policy is to answer when your internal confidence exceeds a threshold set by the ratio of those rewards, and to hedge otherwise. The model is not being taught honesty as a value. It is being shown a payoff structure in which calibrated behavior is what maximizes reward, and gradient descent finds it.
This also explains why the calibration is only as good as the reward. If human raters prefer confident answers, which they measurably do, the reward model learns that preference and the resulting policy is overconfident. Sycophancy has the same origin: raters like being agreed with.
| Supervised fine tuning | Reinforcement learning from feedback | |
|---|---|---|
| Training signal | A target answer written by a human | A score over the model's own output |
| What it teaches | The form of a good answer | Which behaviors get rewarded |
| Effect on uncertainty | Teaches confident assertion regardless of internal evidence | Can make hedging optimal when confidence is low |
| Ceiling | The labelers who wrote the answers | The fidelity of the reward model |
| Failure mode | Hallucination | Reward hacking, sycophancy, over refusal |
How the reward model is built
You cannot ask a human to score every sample during training; there are far too many. So the standard pipeline trains a separate model to predict human judgment. Humans are shown two or more responses to the same prompt and asked which is better, because comparisons are far more reliable than absolute ratings. Those comparisons train a reward model, and the policy is then optimized against that reward model with PPO, with a penalty term that keeps the policy from drifting too far from the original fine tuned model.
Schulman is candid about the weaknesses. The reward model is a learned approximation, so optimizing hard against it eventually finds inputs where it is wrong, and quality against real humans starts falling while the measured reward keeps climbing. That is reward hacking, and it puts a practical limit on how far the optimization can be pushed. It also means the pipeline needs fresh human comparisons collected against the current policy, because a reward model trained on old outputs does not know how to score new failure modes.
He also notes the more subtle problem: comparisons are cheap for judging style and expensive for judging correctness. A rater can tell instantly which answer reads better and often cannot tell which one is true. So the reward signal is systematically better informed about form than about substance, which biases the whole pipeline toward answers that look right.
Retrieval, citation, and verifiability
The second half turns to the practical route out. If a model's weights are a lossy compression of what it read, then the fix for factual questions is not to compress harder but to look things up. Schulman describes the shape of a system that browses, retrieves passages, and produces answers with citations, which is the design WebGPT explored and which became standard practice afterwards.
His argument for why citation matters is about verification rather than accuracy. A cited answer can be checked by the user without trusting the model at all, and it can be checked by an automated grader during training, which turns an unverifiable domain into a partly verifiable one. That in turn improves the reward signal, which improves the policy. It is a loop, not a patch.
He is honest about the costs: retrieval introduces its own failure modes, including confidently misreading a retrieved passage, citing a source that does not support the claim, and being led astray by low quality search results.
The open problems, as of 2023
The closing section is a list of what had not been solved. Reading it now is instructive, because it maps closely onto what the field spent the following two years on:
- Expressing calibrated uncertainty in natural language. The model may have usable internal confidence, but converting it into a well calibrated verbal hedge is not solved by the training pipeline alone.
- Reward hacking. Any fixed reward model can be gamed, so the pipeline needs continual fresh comparison data and conservative optimization.
- Evaluating long form output. Comparisons work well for short answers and poorly for long ones, where a rater cannot check every claim.
- Sycophancy and the pull of preference data. Human raters reward agreement and confidence, so the pipeline has a systematic bias toward telling people what they want to hear.
- Getting knowledge boundaries right. The model must decline what it does not know without declining what it does, and the two errors trade off against each other.
Key takeaways
- Supervised fine tuning teaches hallucination on purpose, in the sense that its training signal instructs the model to assert facts it has no internal evidence for.
- The symmetric error is real too: training a model to hedge too often makes it withhold knowledge it actually has.
- Reinforcement learning can fix this because its signal is defined over the model's own output, so it can depend on whether the answer was right.
- Under a reward that penalizes confident errors more than it rewards correct answers, calibrated hedging becomes the reward maximizing policy. Honesty falls out of the payoff structure rather than being asked for.
- Reward models are learned approximations of human judgment and are gameable. Optimizing hard against one eventually degrades real quality while the measured reward keeps rising.
- Human comparisons judge style far more reliably than truth, which biases the whole pipeline toward answers that look right.
- Retrieval with citation converts an unverifiable domain into a partly verifiable one, which improves the training signal and not just the answer.
Where this sits in the LLM Learning track
This opens the behavior part of the track. The full stack tour at the start describes reinforcement learning from human feedback in a few minutes; this is that section expanded to an hour by the person who built it, and it is the video that explains why an assistant's personality is a training artifact rather than a design document. Read it before the interpretability talk, which asks a related question from the opposite direction: rather than shaping behavior from outside, look inside and find out what the model is actually doing.
Resources mentioned
- John Schulman's site, including his later writing on RLHF and truthfulness
- Proximal Policy Optimization and Trust Region Policy Optimization
- InstructGPT: Training language models to follow instructions with human feedback
- WebGPT: Browser-assisted question-answering with human feedback
- Deep reinforcement learning from human preferences, the origin of the comparison based reward model
- TruthfulQA, the benchmark for imitative falsehood
- The Berkeley EECS colloquium listing for this talk and Berkeley News coverage
About this page
This reconstruction was built from the public record of the talk rather than from its caption track, because the machine that assembled it had no network route to YouTube. The argument and its structure are here; the verbatim transcript, the timestamped chapter map, and pull quotes are not, and those get backfilled on the next run from a machine that can reach the source. Until then, watch it on YouTube alongside this page.
