The thing that's special about transformers is these attention layers where every token is looking at every other token. That operation, when you implement it on a GPU, is really inefficient because you have to load the weights of the model for every single token that you're generating, even though the weights don't change between token generations. And that's really the core insight behind why ASICs make sense for transformers — you can etch the weights directly into the chip.
You might also like
Both include new chips optimized for AI, with the M6 Mac Mini getting a $200 price bump.
Luke Larsen
Nvidia makes a rather outrageous claim that AI factory compute is becoming an investable asset. It's a $500 Billion third-party financing gig. Nvidia has cash to spare and will backstop up to $125 billion, or 25% of the potential deals.
Michael Spencer
Almost all current techniques are focused on the problem of how we make it so that a frozen set of weights behaves well during deployment. I'm not aware of much research on the question of how to guarantee that, even with constant weight updates, the AI system never falls prey to jailbreaks or changes into a deceptive or evil persona. And if AIs are agglomerating learnings between users as well, how do you prevent users from injecting backdoors or some kind of malicious inclination into the base model?
Dwarkesh Patel