The interesting thing about transformers is that they're actually not that computationally intensive if you look at the raw math. The reason GPUs are inefficient for inference is that GPUs are optimized for training, where you do massive parallel matrix multiplications, but inference is fundamentally a different computational problem — you're doing sequential token generation where memory bandwidth is the bottleneck, not compute.
2w ago
Underscored — save the words that stop you in your tracks.
