LLM-as-judge evals are too generous and cluster toward the middle of the scale. Claire had both GPT-5.5 and Opus 4.8 judge the outputs, and neither was spiky enough. They missed things she flagged immediately on a visual pass, like broken prototypes and ignored wireframe constraints. Models can't yet see what the human eye catches in the first screenshot.
"These models are so amazing, it creates this illusion that it will be able to do everything, because you can experience GPT-3, you can experience GPT-3.5, right? And then you extrapolate what that's going to mean for size and scale in a way that's very logical, but it's not rooted in the reality of what a transformer is, right?"
But one reason that I think it quite underrated, and also which reveals the canyon walls against which the river of AI progress will only slowly chip away at, is that it is not enough for a domain to be verifiable. It also has to be very grindable—in the sense that you can run lots of parallel rollouts against a deterministic and replayable simulator.
It can't have feelings. It doesn't have taste, instinct or chemistry. Those things remain profoundly human, and they're essential to good writing. But AI is brilliant at structure and organisation.
If you or I go and vibe-code something, we think we've replaced the engineer, replaced the accountant, replaced the lawyer. But then you actually look — that was the first 80% of the job. The extra 20%, it turns out, is all the value creation of that profession. All the expertise and domain knowledge is in that last 20%, not the text that got generated.
2mo ago
Underscored — save the words that stop you in your tracks.
