Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that I find most exciting is that transformers have essentially won. We know that this is the architecture. And if you know that, then you can build a chip that only runs transformers, and you can get massive efficiency gains by doing that, because you're not building a general purpose chip.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers specifically is that it's not obvious that the architecture is going to change. Usually in the history of chip design, you build a chip for a specific algorithm and then the algorithm changes and your chip is worthless. But transformers have been around for seven years and they've just gotten bigger and bigger and more and more capable, and there's no real sign that they're going to go away.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really different about a transformer ASIC versus a GPU is that a GPU is a general-purpose chip. It has to be able to run any kind of workload. A transformer ASIC, by contrast, can be completely specialized for one thing, which means you can make very different design decisions.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's so special about a transformer ASIC is that you can literally etch the weights of a model into the chip itself. The weights never need to move — they're just there, on the chip, permanently. That means you eliminate the memory bandwidth bottleneck that makes GPUs so inefficient at inference.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's special about transformers specifically is that the weights are so large that you can't fit them all on chip. And so what happens is that you spend most of your time just loading weights from memory rather than doing computation. And so the key insight behind Etched is that if you hardcode the transformer architecture into silicon, you can do a lot of tricks to basically never have to load the weights from memory.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that the weights are basically fixed after training. You load the weights once onto the chip, and then you just do inference over and over again. So if you build a chip that is specifically designed to run one model or one architecture, you can get massive efficiency gains because you're not paying for the flexibility of being able to run arbitrary code.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The core insight is that transformers have a fixed architecture, and if that architecture is going to be with us for a long time, there's a massive opportunity to build a chip that does nothing but run transformers — and does it faster and cheaper than anything else. A general-purpose chip has to be flexible, and flexibility costs you in efficiency.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The core insight is that transformers have a fixed architecture, and if that architecture is going to be with us for a long time, then building a chip that is essentially a transformer in silicon — where the dataflow is hardwired — means you get rid of all the overhead that a GPU has to carry around to be flexible.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that they've kind of eaten AI. So many of the workloads that people care about are transformers. And because the transformer is a fixed algorithm, you can etch it into silicon — you can build a chip that only runs transformers, and it will be orders of magnitude faster and cheaper than a chip that can run any algorithm.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The bet that Etched is making is that transformers are here to stay, and that by building a chip that only runs transformers, they can be so much more efficient than a general purpose chip that it's worth the trade-off of only being able to run one type of model.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The core insight is that transformers have a fixed computational pattern — and if you etch that pattern directly into silicon rather than running it on general-purpose hardware, you eliminate the overhead that makes inference slow and expensive. A chip that can only run transformers is dramatically faster and cheaper at running transformers than a chip designed to run anything.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The bet that Etched is making is that transformers are so dominant and so unlikely to be displaced that it makes sense to build a chip that does nothing but run transformers — sacrificing all flexibility for maximum efficiency on that one task.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The key insight is that transformers, which underpin all modern AI, have a fixed mathematical structure — and if you build a chip that does nothing but transformers, you can eliminate the overhead that makes GPUs so inefficient at inference. A transformer supercomputer, etched in silicon, could be 10 to 20 times faster and cheaper than the GPU alternative.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that I find most stunning about this is that transformers were invented in 2017, and the reason that they've taken over everything is that they're extremely good at using parallel compute. And so the question is, are transformers the last great neural network architecture? And our bet is yes, or at least that they will be dominant for the foreseeable future, which means that if you build a chip that is perfect for transformers, you will have a chip that is perfect for AI.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's special about transformers specifically is that you can make an ASIC, an application specific integrated circuit, specifically for transformers, and that chip will be so much faster and cheaper than a GPU at running transformers that it's almost incomprehensible. GPUs are a general-purpose chip that has to do everything, whereas an ASIC can be laser-focused on one task.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's interesting about transformers specifically is that you have this operation called attention, which requires you to look at all of the previous tokens every single time you generate a new token. So if you have a very long context, meaning a very long conversation or document, you have to do an enormous amount of memory reads every single time you want to generate even one new word. And that's actually a really hard problem.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that actually, when you look at what all these models are doing, they're all basically running the same algorithm, which is the transformer architecture. And so if you build a chip that's specifically designed to run transformers as fast as possible, you can get enormous speedups compared to a GPU, which is a general-purpose chip that has to be able to run any algorithm.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really special about Sohu is that it's an ASIC, meaning it can only run transformers. It cannot run anything else. And that might sound like a huge limitation, but we think it's actually the killer feature because by hardcoding the transformer architecture into silicon, we're able to get massive gains in efficiency over a GPU, which has to be flexible enough to run any workload.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The core insight is that transformers are actually very predictable in their memory access patterns — you know exactly what weights you're going to need before you need them. And so if you build a chip that's specifically designed around that access pattern, you can be much more efficient than a general-purpose chip that has to handle arbitrary memory access patterns.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that I think people underestimate about ASICs in general is that when you build something that's an ASIC, you can't change it. And that seems like a huge weakness, but actually it becomes a strength because you can optimize the hardware so specifically for one thing that you get much better performance and power efficiency than you would with something general purpose like a GPU.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that I find most compelling about the transformer is that, unlike prior neural network architectures, the transformer's compute requirements are almost entirely matrix multiplications. And matrix multiplication is a very structured computation that you can build hardware specifically to do very efficiently.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The reason GPUs are so inefficient for inference is that they're general purpose. A GPU has to be able to run any possible AI model, and so it has a very complicated control flow architecture. But when you're doing inference, you're running one model over and over again, billions of times a day. You don't need that flexibility.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The interesting thing about transformers is that they're actually not that computationally intensive if you look at the raw math. The reason GPUs are inefficient for inference is that GPUs are optimized for training, where you do massive parallel matrix multiplications, but inference is fundamentally a different computational problem — you're doing sequential token generation where memory bandwidth is the bottleneck, not compute.