Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers specifically is that it's not obvious that the architecture is going to change. Usually in the history of chip design, you build a chip for a specific algorithm and then the algorithm changes and your chip is worthless. But transformers have been around for seven years and they've just gotten bigger and bigger and more and more capable, and there's no real sign that they're going to go away.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's so special about transformers is they can be trained end to end with gradient descent. You don't have to come up with the algorithm yourself. You just show the network examples of inputs and outputs and it figures out the algorithm. And because the algorithm is learned rather than programmed, it can be much more sophisticated than anything a human could come up with.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's special about transformers specifically is that the weights are so large that you can't fit them all on chip. And so what happens is that you spend most of your time just loading weights from memory rather than doing computation. And so the key insight behind Etched is that if you hardcode the transformer architecture into silicon, you can do a lot of tricks to basically never have to load the weights from memory.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The core insight is that transformers have a fixed architecture, and if that architecture is going to be with us for a long time, there's a massive opportunity to build a chip that does nothing but run transformers — and does it faster and cheaper than anything else. A general-purpose chip has to be flexible, and flexibility costs you in efficiency.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The core insight is that transformers have a fixed architecture, and if that architecture is going to be with us for a long time, then building a chip that is essentially a transformer in silicon — where the dataflow is hardwired — means you get rid of all the overhead that a GPU has to carry around to be flexible.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that they've kind of eaten AI. So many of the workloads that people care about are transformers. And because the transformer is a fixed algorithm, you can etch it into silicon — you can build a chip that only runs transformers, and it will be orders of magnitude faster and cheaper than a chip that can run any algorithm.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The bet that Etched is making is that transformers are here to stay, and that by building a chip that only runs transformers, they can be so much more efficient than a general purpose chip that it's worth the trade-off of only being able to run one type of model.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The core insight is that transformers have a fixed computational pattern — and if you etch that pattern directly into silicon rather than running it on general-purpose hardware, you eliminate the overhead that makes inference slow and expensive. A chip that can only run transformers is dramatically faster and cheaper at running transformers than a chip designed to run anything.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The bet that Etched is making is that transformers are so dominant and so unlikely to be displaced that it makes sense to build a chip that does nothing but run transformers — sacrificing all flexibility for maximum efficiency on that one task.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The key insight is that transformers, which underpin all modern AI, have a fixed mathematical structure — and if you build a chip that does nothing but transformers, you can eliminate the overhead that makes GPUs so inefficient at inference. A transformer supercomputer, etched in silicon, could be 10 to 20 times faster and cheaper than the GPU alternative.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that I find most stunning about this is that transformers were invented in 2017, and the reason that they've taken over everything is that they're extremely good at using parallel compute. And so the question is, are transformers the last great neural network architecture? And our bet is yes, or at least that they will be dominant for the foreseeable future, which means that if you build a chip that is perfect for transformers, you will have a chip that is perfect for AI.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's special about transformers specifically is that you can make an ASIC, an application specific integrated circuit, specifically for transformers, and that chip will be so much faster and cheaper than a GPU at running transformers that it's almost incomprehensible. GPUs are a general-purpose chip that has to do everything, whereas an ASIC can be laser-focused on one task.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's interesting about transformers specifically is that you have this operation called attention, which requires you to look at all of the previous tokens every single time you generate a new token. So if you have a very long context, meaning a very long conversation or document, you have to do an enormous amount of memory reads every single time you want to generate even one new word. And that's actually a really hard problem.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that the attention mechanism, which is the key innovation, is actually a fixed algorithm. It's not like the weights change the algorithm; the weights just parameterize the algorithm. And so if you know that the algorithm is fixed, you can build hardware that's specifically designed to run that algorithm as fast as possible.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's special about transformers is the attention mechanism, and the attention mechanism has this property that every single token has to interact with every other token. The amount of computation required scales with the square of the sequence length. So as you try to process longer and longer contexts, it gets quadratically more expensive.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The core insight is that transformers are actually very predictable in their memory access patterns — you know exactly what weights you're going to need before you need them. And so if you build a chip that's specifically designed around that access pattern, you can be much more efficient than a general-purpose chip that has to handle arbitrary memory access patterns.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that I find most compelling about the transformer is that, unlike prior neural network architectures, the transformer's compute requirements are almost entirely matrix multiplications. And matrix multiplication is a very structured computation that you can build hardware specifically to do very efficiently.