Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The key insight is that transformers are incredibly regular and predictable in their compute patterns, which means you can build a chip that does nothing but run transformers extremely efficiently. A GPU is a general-purpose chip that has to handle any kind of computation, so it wastes enormous amounts of die area and power on flexibility you don't need for inference.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that I find most exciting is that transformers have essentially won. We know that this is the architecture. And if you know that, then you can build a chip that only runs transformers, and you can get massive efficiency gains by doing that, because you're not building a general purpose chip.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers specifically is that it's not obvious that the architecture is going to change. Usually in the history of chip design, you build a chip for a specific algorithm and then the algorithm changes and your chip is worthless. But transformers have been around for seven years and they've just gotten bigger and bigger and more and more capable, and there's no real sign that they're going to go away.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really different about a transformer ASIC versus a GPU is that a GPU is a general-purpose chip. It has to be able to run any kind of workload. A transformer ASIC, by contrast, can be completely specialized for one thing, which means you can make very different design decisions.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's so special about a transformer ASIC is that you can literally etch the weights of a model into the chip itself. The weights never need to move — they're just there, on the chip, permanently. That means you eliminate the memory bandwidth bottleneck that makes GPUs so inefficient at inference.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's special about transformers specifically is that the weights are so large that you can't fit them all on chip. And so what happens is that you spend most of your time just loading weights from memory rather than doing computation. And so the key insight behind Etched is that if you hardcode the transformer architecture into silicon, you can do a lot of tricks to basically never have to load the weights from memory.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that the weights are basically fixed after training. You load the weights once onto the chip, and then you just do inference over and over again. So if you build a chip that is specifically designed to run one model or one architecture, you can get massive efficiency gains because you're not paying for the flexibility of being able to run arbitrary code.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The core insight is that transformers have a fixed architecture, and if that architecture is going to be with us for a long time, then building a chip that is essentially a transformer in silicon — where the dataflow is hardwired — means you get rid of all the overhead that a GPU has to carry around to be flexible.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that they've kind of eaten AI. So many of the workloads that people care about are transformers. And because the transformer is a fixed algorithm, you can etch it into silicon — you can build a chip that only runs transformers, and it will be orders of magnitude faster and cheaper than a chip that can run any algorithm.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that actually, when you look at what all these models are doing, they're all basically running the same algorithm, which is the transformer architecture. And so if you build a chip that's specifically designed to run transformers as fast as possible, you can get enormous speedups compared to a GPU, which is a general-purpose chip that has to be able to run any algorithm.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The core insight is that transformers are actually very predictable in their memory access patterns — you know exactly what weights you're going to need before you need them. And so if you build a chip that's specifically designed around that access pattern, you can be much more efficient than a general-purpose chip that has to handle arbitrary memory access patterns.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that I find most compelling about the transformer is that, unlike prior neural network architectures, the transformer's compute requirements are almost entirely matrix multiplications. And matrix multiplication is a very structured computation that you can build hardware specifically to do very efficiently.

China Decode

Can China REPLACE Nvidia? | China Decode

Chip companies in the future might stop obsessing so much as they do right now about how small the transistors in a semiconductor should be and they might focus more instead on how fast the data moves through a semiconductor.