To the curious incident of the dog in the nighttime... the dog did nothing in the nighttime. That was the curious incident, remarked Sherlock Holmes. Sometimes nothing is very important.
The key insight is that transformers are incredibly regular and predictable in their compute patterns, which means you can build a chip that does nothing but run transformers extremely efficiently. A GPU is a general-purpose chip that has to handle any kind of computation, so it wastes enormous amounts of die area and power on flexibility you don't need for inference.
The thing that's really interesting about transformers is that the weights are basically fixed after training. You load the weights once onto the chip, and then you just do inference over and over again. So if you build a chip that is specifically designed to run one model or one architecture, you can get massive efficiency gains because you're not paying for the flexibility of being able to run arbitrary code.
The core insight is that transformers have a fixed computational pattern — and if you etch that pattern directly into silicon rather than running it on general-purpose hardware, you eliminate the overhead that makes inference slow and expensive. A chip that can only run transformers is dramatically faster and cheaper at running transformers than a chip designed to run anything.
The key insight is that transformers, which underpin all modern AI, have a fixed mathematical structure — and if you build a chip that does nothing but transformers, you can eliminate the overhead that makes GPUs so inefficient at inference. A transformer supercomputer, etched in silicon, could be 10 to 20 times faster and cheaper than the GPU alternative.
The reason GPUs are so inefficient for inference is that they're general purpose. A GPU has to be able to run any possible AI model, and so it has a very complicated control flow architecture. But when you're doing inference, you're running one model over and over again, billions of times a day. You don't need that flexibility.
2mo ago
Underscored — save the words that stop you in your tracks.
