The thing about ASICs is that they're much faster and cheaper at doing specific things, but you're locked into that specific thing forever. So the question is, what is specific enough that it'll be around for a long time, but general enough that there's a large market for it? And the answer we came up with is transformers.

Invest Like the Best with Patrick O'Shaughnessy

@underscoredpodcasts

ai-hardwaresemiconductorstransformers

The thing that's interesting about transformers is that, unlike previous neural network architectures, the operations they do are basically fixed. A transformer from 2024 does the same operations as a transformer from 2017. So you can build a chip that is essentially a transformer, etched in silicon, and it will always be doing the most important computation in AI.

Invest Like the Best with Patrick O'Shaughnessy

@underscoredpodcasts

ai-hardwaretransformersneural-networks

The thing that's special about transformers is these attention layers where every token is looking at every other token. That operation, when you implement it on a GPU, is really inefficient because you have to load the weights of the model for every single token that you're generating, even though the weights don't change between token generations. And that's really the core insight behind why ASICs make sense for transformers — you can etch the weights directly into the chip.

Invest Like the Best with Patrick O'Shaughnessy

@underscoredpodcasts

ai-hardwaretransformerschip-design