From the release of GPT-4 to the end of the METR measure, the amount of work AI could do doubled eight times. In 2024, an AI could do a four minute task. At the moment I turned in my draft of Co-Existence in April of 2026, an AI could do 17 hours of human work.
More agents are useful for speed: they can solve many problems faster. They're less useful for boosting AI intelligence. Multi-agent gains in solving harder problems often diminish incredibly quickly and may even saturate.
According to Anthropic's internal index, Claude led 26% of measured model R&D work in August, up from under 1% in February. How fast will internal model RSI teams accelerate new more powerful models? Researchers set the tasks and supervise execution, and Anthropic reports that this is speeding up development.
I'm getting increasingly alarmed by the compute Anthropic will have at its disposal to put its world-leading models at the intersection of biotechnology in the 2030s.
But one reason that I think it quite underrated, and also which reveals the canyon walls against which the river of AI progress will only slowly chip away at, is that it is not enough for a domain to be verifiable. It also has to be very grindable—in the sense that you can run lots of parallel rollouts against a deterministic and replayable simulator.
Imagine if it took a couple decades worth of courses with hundreds of concurrent professors and millions of practice tasks for you to learn how to polish a word file. Even the task count difference understates the gap - the models have to grind their far more numerous tasks each far harder. Whereas a human student might practice a textbook problem once or twice, GRPO has the model generate hundreds to thousands of rollouts per task.
It is increasingly hard to quickly demonstrate each generational change as AI has gotten better, since a lot of the old things AI was bad at, like math or counting letters in words, are now trivial for AI to do.
6mo ago
Underscored — save the words that stop you in your tracks.
