Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's interesting about transformers is that, unlike previous neural network architectures, the operations they do are basically fixed. A transformer from 2024 does the same operations as a transformer from 2017. So you can build a chip that is essentially a transformer, etched in silicon, and it will always be doing the most important computation in AI.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's special about transformers is these attention layers where every token is looking at every other token. That operation, when you implement it on a GPU, is really inefficient because you have to load the weights of the model for every single token that you're generating, even though the weights don't change between token generations. And that's really the core insight behind why ASICs make sense for transformers — you can etch the weights directly into the chip.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that the architecture has been so stable. If you look at what people are running in production today, it's the same basic architecture that was published in 2017. And the bet that we're making is that this architecture is going to be around for a long time, which means you can build hardware that's specifically optimized for it.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that I think is underappreciated is that transformers are probably here to stay. People have been trying to beat transformers for years and years and years, and nobody has been able to do it at scale. And so if you believe that transformers are here to stay, then you can build a chip that only runs transformers.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The key insight is that transformers are incredibly regular and predictable in their compute patterns, which means you can build a chip that does nothing but run transformers extremely efficiently. A GPU is a general-purpose chip that has to handle any kind of computation, so it wastes enormous amounts of die area and power on flexibility you don't need for inference.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that the model architecture is essentially fixed. The operations that transformers do are the same whether you're running GPT-4 or any other large language model — it's always matrix multiplications and attention. And if you build a chip that only does those operations, you can make it much faster and much cheaper than a general-purpose GPU.

Decoder with Nilay Patel

Yes, even Nvidia's head of automotive is fighting for compute

There's actually been startling progress in reinventing the fundamental nature of the car itself — something the industry calls the 'software-defined vehicle,' controlled by just a handful of powerful computers instead of dozens or even hundreds of independent electronic control units, or ECUs. Xinzhou says that moment is basically here.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that I find most exciting is that transformers have essentially won. We know that this is the architecture. And if you know that, then you can build a chip that only runs transformers, and you can get massive efficiency gains by doing that, because you're not building a general purpose chip.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers specifically is that it's not obvious that the architecture is going to change. Usually in the history of chip design, you build a chip for a specific algorithm and then the algorithm changes and your chip is worthless. But transformers have been around for seven years and they've just gotten bigger and bigger and more and more capable, and there's no real sign that they're going to go away.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really different about a transformer ASIC versus a GPU is that a GPU is a general-purpose chip. It has to be able to run any kind of workload. A transformer ASIC, by contrast, can be completely specialized for one thing, which means you can make very different design decisions.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's so special about a transformer ASIC is that you can literally etch the weights of a model into the chip itself. The weights never need to move — they're just there, on the chip, permanently. That means you eliminate the memory bandwidth bottleneck that makes GPUs so inefficient at inference.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's special about transformers specifically is that the weights are so large that you can't fit them all on chip. And so what happens is that you spend most of your time just loading weights from memory rather than doing computation. And so the key insight behind Etched is that if you hardcode the transformer architecture into silicon, you can do a lot of tricks to basically never have to load the weights from memory.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that the weights are basically fixed after training. You load the weights once onto the chip, and then you just do inference over and over again. So if you build a chip that is specifically designed to run one model or one architecture, you can get massive efficiency gains because you're not paying for the flexibility of being able to run arbitrary code.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The core insight is that transformers have a fixed architecture, and if that architecture is going to be with us for a long time, there's a massive opportunity to build a chip that does nothing but run transformers — and does it faster and cheaper than anything else. A general-purpose chip has to be flexible, and flexibility costs you in efficiency.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The core insight is that transformers have a fixed architecture, and if that architecture is going to be with us for a long time, then building a chip that is essentially a transformer in silicon — where the dataflow is hardwired — means you get rid of all the overhead that a GPU has to carry around to be flexible.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that they've kind of eaten AI. So many of the workloads that people care about are transformers. And because the transformer is a fixed algorithm, you can etch it into silicon — you can build a chip that only runs transformers, and it will be orders of magnitude faster and cheaper than a chip that can run any algorithm.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The bet that Etched is making is that transformers are here to stay, and that by building a chip that only runs transformers, they can be so much more efficient than a general purpose chip that it's worth the trade-off of only being able to run one type of model.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The core insight is that transformers have a fixed computational pattern — and if you etch that pattern directly into silicon rather than running it on general-purpose hardware, you eliminate the overhead that makes inference slow and expensive. A chip that can only run transformers is dramatically faster and cheaper at running transformers than a chip designed to run anything.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The bet that Etched is making is that transformers are so dominant and so unlikely to be displaced that it makes sense to build a chip that does nothing but run transformers — sacrificing all flexibility for maximum efficiency on that one task.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The key insight is that transformers, which underpin all modern AI, have a fixed mathematical structure — and if you build a chip that does nothing but transformers, you can eliminate the overhead that makes GPUs so inefficient at inference. A transformer supercomputer, etched in silicon, could be 10 to 20 times faster and cheaper than the GPU alternative.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that I find most stunning about this is that transformers were invented in 2017, and the reason that they've taken over everything is that they're extremely good at using parallel compute. And so the question is, are transformers the last great neural network architecture? And our bet is yes, or at least that they will be dominant for the foreseeable future, which means that if you build a chip that is perfect for transformers, you will have a chip that is perfect for AI.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's special about transformers specifically is that you can make an ASIC, an application specific integrated circuit, specifically for transformers, and that chip will be so much faster and cheaper than a GPU at running transformers that it's almost incomprehensible. GPUs are a general-purpose chip that has to do everything, whereas an ASIC can be laser-focused on one task.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's interesting about transformers specifically is that you have this operation called attention, which requires you to look at all of the previous tokens every single time you generate a new token. So if you have a very long context, meaning a very long conversation or document, you have to do an enormous amount of memory reads every single time you want to generate even one new word. And that's actually a really hard problem.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that actually, when you look at what all these models are doing, they're all basically running the same algorithm, which is the transformer architecture. And so if you build a chip that's specifically designed to run transformers as fast as possible, you can get enormous speedups compared to a GPU, which is a general-purpose chip that has to be able to run any algorithm.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really special about Sohu is that it's an ASIC, meaning it can only run transformers. It cannot run anything else. And that might sound like a huge limitation, but we think it's actually the killer feature because by hardcoding the transformer architecture into silicon, we're able to get massive gains in efficiency over a GPU, which has to be flexible enough to run any workload.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The core insight is that transformers are actually very predictable in their memory access patterns — you know exactly what weights you're going to need before you need them. And so if you build a chip that's specifically designed around that access pattern, you can be much more efficient than a general-purpose chip that has to handle arbitrary memory access patterns.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that I think people underestimate about ASICs in general is that when you build something that's an ASIC, you can't change it. And that seems like a huge weakness, but actually it becomes a strength because you can optimize the hardware so specifically for one thing that you get much better performance and power efficiency than you would with something general purpose like a GPU.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that I find most compelling about the transformer is that, unlike prior neural network architectures, the transformer's compute requirements are almost entirely matrix multiplications. And matrix multiplication is a very structured computation that you can build hardware specifically to do very efficiently.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The reason GPUs are so inefficient for inference is that they're general purpose. A GPU has to be able to run any possible AI model, and so it has a very complicated control flow architecture. But when you're doing inference, you're running one model over and over again, billions of times a day. You don't need that flexibility.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The interesting thing about transformers is that they're actually not that computationally intensive if you look at the raw math. The reason GPUs are inefficient for inference is that GPUs are optimized for training, where you do massive parallel matrix multiplications, but inference is fundamentally a different computational problem — you're doing sequential token generation where memory bandwidth is the bottleneck, not compute.