Ilya Sutskever

Ilya Sutskever

Ilya Sutskever: Sequence to sequence learning with neural networks: what a decade

Accepting a NeurIPS Test of Time award for the 2014 sequence to sequence paper, Ilya Sutskever revisits what that work bet on, autoregression plus scale plus connectionism, and what held up. Then the line the talk is remembered for: pretraining as we know it will end, because compute keeps growing while data does not. Data is the fossil fuel of AI and there is only one internet. He sketches what comes after, agents, synthetic data and inference time compute, and uses hominid brain scaling as evidence that a different scaling regime can exist at all.