Ilya Sutskever: Sequence to sequence learning with neural networks: what a decade

At a glance

Ilya Sutskever came to NeurIPS 2024 in Vancouver to accept a Test of Time award for Sequence to Sequence Learning with Neural Networks, the 2014 paper he wrote with Oriol Vinyals and Quoc Le. He used the slot to do two things: look back honestly at what that paper bet on and which bets held, and then say something about the future that made the talk the most quoted twenty minutes of the conference.

The line people remember: pretraining as we know it will end. His reasoning is arithmetic rather than mystical. Compute keeps growing through better hardware, better algorithms, and larger clusters. Data does not. Data is the fossil fuel of AI, and there is only one internet. When one input to a scaling recipe stops scaling, the recipe stops.

The rest of the talk sketches what comes after, agents, synthetic data, and inference time compute, and offers an unexpected piece of evidence that a different scaling regime is even possible: the way brain size scaled in hominids compared with every other mammal.

The deep explanation

What the 2014 paper bet on

Sutskever rewinds to the state of things a decade earlier and states the belief that the work rested on, which he calls the deep learning hypothesis. If a large neural network with ten layers can be trained on a task, it can do anything a human can do in a fraction of a second. The reasoning is a comparison to biological neurons: a human does a great deal in a tenth of a second, biological neurons are slow, so whatever the brain does in that window is a shallow computation, and an artificial network of comparable depth ought to be able to represent it.

From that hypothesis the design follows. Take a large LSTM, read a source sentence into a fixed vector, and generate the target sentence autoregressively from that vector. No alignment machinery, no phrase tables, none of the apparatus that statistical machine translation had accumulated. The engineering was as brute as the idea: the model was spread across multiple GPUs by pipelining layers, which was an unusual thing to do at the time.

He also names the third ingredient, connectionism: the belief that intelligence emerges from many simple units connected together and trained on data, rather than from designed symbolic structure. Sequence to sequence was a bet on all three, autoregression, scale, and connectionism, and Sutskever's assessment is that the bet was right, and that the transformer replaced the LSTM while leaving those three commitments intact.

There is a note of self awareness in this section. He shows the old slide and observes that the scaling hypothesis was stated plainly in 2014: a big enough dataset and a big enough neural network trained on it will succeed. What the field then did was execute that sentence for ten years.

• 2012AlexNet. Deep learning on GPUs wins ImageNet by a margin that ends the argument about whether it works at all.
• 2014Sequence to sequence. An LSTM encoder and decoder replace the hand built machinery of statistical machine translation. The scaling hypothesis is written down.
• 2017The transformer. Recurrence is dropped for attention, which makes training parallel and unlocks the scale the hypothesis called for.
• 2020Scaling laws and GPT-3. The relationship between compute, data, and loss is characterized, and the recipe becomes an industrial process.
• 2022Chinchilla. The split between parameters and tokens is corrected, and data becomes the visibly scarce input.
• 2024This talk. Compute still grows, data does not, so the pretraining era has a horizon. Reasoning, agents, and synthetic data are what comes next.

The claim: pretraining as we know it will end

The argument is short and hard to dodge. Pretraining consumes two inputs: compute and data. Compute keeps growing along several axes at once, better chips, better parallelism, larger capital budgets, better algorithms. Data does not grow that way. The public internet is a fixed artifact, growing far more slowly than compute and already substantially consumed by the frontier runs.

His phrase for it is that data is the fossil fuel of AI. It was laid down once, we are burning through it, and there is not another one. So the specific recipe that got us here, throw more compute and more freshly scraped text at the same objective, has an end date, even though nothing about the underlying trend in compute is slowing.

He is careful about the scope of the claim. It is not that models stop improving, or that scaling stops mattering. It is that the particular thing called pretraining, in the form the field has practiced it, cannot be the growth engine forever, and the field will have to find what replaces it.

His candidates, offered without much confidence in any one of them:

• Agents. Systems that take actions over time, which generate their own experience rather than consuming a fixed corpus.
• Synthetic data. Generating training material rather than harvesting it, which he flags as a genuinely hard problem rather than an obvious solution.
• Inference time compute. Spending more computation at the moment of answering rather than at training time. In December 2024 this was the freshest of the three, and it is the direction the reasoning models then took.

The biological argument for a different scaling regime

The most striking section is his answer to an implicit objection: if the current recipe runs out, why believe another one exists.

He shows the relationship between brain mass and body mass across species. Mammals fall along a clean line, larger bodies with proportionally larger brains, and the line is remarkably consistent. Then the hominid line departs from it. Human ancestors scale brain to body mass along a visibly different and steeper slope than every other mammal group on the chart.

The point is not about brains specifically. It is an existence proof about scaling itself. Biology found a different exponent. Something changed in the setup and the whole relationship shifted, and that happened in a system nobody designed. If evolution can stumble into a new scaling regime, it is not exotic to expect that a field which has been running one recipe for a decade might find another.

What superintelligence would look like

The closing section is his sketch of where this heads, stated as prediction rather than plan. The systems he expects eventually will be qualitatively different from today's, and he names the properties:

• Genuinely agentic, in the strong sense, rather than in the sense of current tool loops.
• Reasoning, and here he makes a point that gets less attention than it should: a system that truly reasons becomes unpredictable, because the whole value of reasoning is arriving at conclusions that were not obvious in advance. His example is chess engines, which are unpredictable to the best human players. Predictability and reasoning power pull against each other.
• Learning from limited data, rather than requiring an internet's worth of examples to acquire a concept.
• Self aware, which he mentions as a straightforward consequence rather than a mystical one, since a model of the world that includes the system itself is a useful model.

He declines to give a timeline and says so directly, and the talk ends on an open note about what such systems should be to us and what we should be to them, including a Q&A exchange on whether such systems would want rights. It is deliberately unresolved.

Key takeaways

• The 2014 sequence to sequence paper bet on three things: autoregression, scale, and connectionism. The transformer replaced the architecture while keeping all three, which is why the bet is judged as having held.
• The deep learning hypothesis behind it: if a ten layer network can be trained on a task, it can do anything a human can do in a fraction of a second, because biological neurons are slow and that window is a shallow computation.
• Pretraining consumes compute and data. Compute keeps growing and data does not, so the current recipe has a horizon.
• Data is the fossil fuel of AI. There is only one internet, and it is being burned once.
• The candidates for what comes next are agents, synthetic data, and inference time compute, offered as directions rather than answers.
• Hominid brain to body scaling departs from the mammalian line, which is an existence proof that different scaling regimes exist and can be found rather than designed.
• Real reasoning implies unpredictability. A system whose conclusions you can anticipate is not doing much reasoning, which is why chess engines surprise grandmasters.

Where this sits in the LLM Learning track

This opens the final part of the track, the part about where all of it goes. Everything earlier describes a recipe that works. This is the person who wrote the recipe's opening chapter saying that its main ingredient is finite. Read it against the Chinchilla section of the Five Formulas tutorial, which quantifies the same data constraint from the other direction, and then read the Karpathy keynote that closes the track for the view of what gets built while the research question is open.

Resources mentioned

• Sequence to Sequence Learning with Neural Networks, the 2014 paper receiving the award
• NeurIPS 2024 and its Test of Time award
• AlexNet, Sutskever's earlier work with Alex Krizhevsky and Geoffrey Hinton
• Long short-term memory, the architecture the paper used
• Attention Is All You Need, the transformer paper that replaced it
• Scaling Laws for Neural Language Models and the Chinchilla paper
• Safe Superintelligence, the company Sutskever founded in 2024
• Connectionism, the intellectual tradition he places the work in

About this page

This reconstruction was built from the public record of the talk rather than from its caption track, because the machine that assembled it had no network route to YouTube. The argument, the structure, and the claims are here; the verbatim transcript, the timestamped chapter map, and pull quotes are not, and those get backfilled on the next run from a machine that can reach the source. Until then, watch it on YouTube alongside this page.