The thing that's interesting about transformers is that, unlike previous neural network architectures, the operations they do are basically fixed. A transformer from 2024 does the same operations as a transformer from 2017. So you can build a chip that is essentially a transformer, etched in silicon, and it will always be doing the most important computation in AI.
The thing that's so special about transformers is they can be trained end to end with gradient descent. You don't have to come up with the algorithm yourself. You just show the network examples of inputs and outputs and it figures out the algorithm. And because the algorithm is learned rather than programmed, it can be much more sophisticated than anything a human could come up with.
The thing that's so special about a transformer ASIC is that you can literally etch the weights of a model into the chip itself. The weights never need to move — they're just there, on the chip, permanently. That means you eliminate the memory bandwidth bottleneck that makes GPUs so inefficient at inference.
It was basically sort of thought to be career suicide. Like of course we know AI doesn't work. You know, we tried it in the 90s, places like MIT and it was a dead end. You know, and that was the prevailing view. But we just felt the small band of us felt that that actually with the right ideas and using learning systems, reinforcement learning and betting on neural networks that a lot of fast progress could be made.
1mo ago
Underscored — save the words that stop you in your tracks.
