The thing that's really interesting about transformers is that actually, when you look at what all these models are doing, they're all basically running the same algorithm, which is the transformer architecture. And so if you build a chip that's specifically designed to run transformers as fast as possible, you can get enormous speedups compared to a GPU, which is a general-purpose chip that has to be able to run any algorithm.
The thing that's really special about Sohu is that it's an ASIC, meaning it can only run transformers. It cannot run anything else. And that might sound like a huge limitation, but we think it's actually the killer feature because by hardcoding the transformer architecture into silicon, we're able to get massive gains in efficiency over a GPU, which has to be flexible enough to run any workload.
The interesting thing about transformers is that they're actually not that computationally intensive if you look at the raw math. The reason GPUs are inefficient for inference is that GPUs are optimized for training, where you do massive parallel matrix multiplications, but inference is fundamentally a different computational problem — you're doing sequential token generation where memory bandwidth is the bottleneck, not compute.
2w ago
Underscored — save the words that stop you in your tracks.
