The thing that's so special about a transformer ASIC is that you can literally etch the weights of a model into the chip itself. The weights never need to move — they're just there, on the chip, permanently. That means you eliminate the memory bandwidth bottleneck that makes GPUs so inefficient at inference.
4d ago
Underscored — save the words that stop you in your tracks.
