Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing about ASICs is that they're much faster and cheaper at doing specific things, but you're locked into that specific thing forever. So the question is, what is specific enough that it'll be around for a long time, but general enough that there's a large market for it? And the answer we came up with is transformers.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's interesting about transformers is that, unlike previous neural network architectures, the operations they do are basically fixed. A transformer from 2024 does the same operations as a transformer from 2017. So you can build a chip that is essentially a transformer, etched in silicon, and it will always be doing the most important computation in AI.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's special about transformers is these attention layers where every token is looking at every other token. That operation, when you implement it on a GPU, is really inefficient because you have to load the weights of the model for every single token that you're generating, even though the weights don't change between token generations. And that's really the core insight behind why ASICs make sense for transformers — you can etch the weights directly into the chip.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that the architecture has been so stable. If you look at what people are running in production today, it's the same basic architecture that was published in 2017. And the bet that we're making is that this architecture is going to be around for a long time, which means you can build hardware that's specifically optimized for it.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that I think is underappreciated is that transformers are probably here to stay. People have been trying to beat transformers for years and years and years, and nobody has been able to do it at scale. And so if you believe that transformers are here to stay, then you can build a chip that only runs transformers.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that the model architecture is essentially fixed. The operations that transformers do are the same whether you're running GPT-4 or any other large language model — it's always matrix multiplications and attention. And if you build a chip that only does those operations, you can make it much faster and much cheaper than a general-purpose GPU.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really different about a transformer ASIC versus a GPU is that a GPU is a general-purpose chip. It has to be able to run any kind of workload. A transformer ASIC, by contrast, can be completely specialized for one thing, which means you can make very different design decisions.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's so special about a transformer ASIC is that you can literally etch the weights of a model into the chip itself. The weights never need to move — they're just there, on the chip, permanently. That means you eliminate the memory bandwidth bottleneck that makes GPUs so inefficient at inference.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's special about transformers specifically is that the weights are so large that you can't fit them all on chip. And so what happens is that you spend most of your time just loading weights from memory rather than doing computation. And so the key insight behind Etched is that if you hardcode the transformer architecture into silicon, you can do a lot of tricks to basically never have to load the weights from memory.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The core insight is that transformers have a fixed architecture, and if that architecture is going to be with us for a long time, there's a massive opportunity to build a chip that does nothing but run transformers — and does it faster and cheaper than anything else. A general-purpose chip has to be flexible, and flexibility costs you in efficiency.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The core insight is that transformers have a fixed architecture, and if that architecture is going to be with us for a long time, then building a chip that is essentially a transformer in silicon — where the dataflow is hardwired — means you get rid of all the overhead that a GPU has to carry around to be flexible.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that they've kind of eaten AI. So many of the workloads that people care about are transformers. And because the transformer is a fixed algorithm, you can etch it into silicon — you can build a chip that only runs transformers, and it will be orders of magnitude faster and cheaper than a chip that can run any algorithm.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The bet that Etched is making is that transformers are here to stay, and that by building a chip that only runs transformers, they can be so much more efficient than a general purpose chip that it's worth the trade-off of only being able to run one type of model.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The bet that Etched is making is that transformers are so dominant and so unlikely to be displaced that it makes sense to build a chip that does nothing but run transformers — sacrificing all flexibility for maximum efficiency on that one task.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The key insight is that transformers, which underpin all modern AI, have a fixed mathematical structure — and if you build a chip that does nothing but transformers, you can eliminate the overhead that makes GPUs so inefficient at inference. A transformer supercomputer, etched in silicon, could be 10 to 20 times faster and cheaper than the GPU alternative.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that I find most interesting about Etched is that they're making a bet that transformers are not going away. And the reason that's a interesting bet is that every other chip company has to hedge. Nvidia has to build a chip that runs all possible AI models, because they don't know what's going to win. Etched is saying transformers have won, and we're going to build a chip that only runs transformers, and by doing that, we can be so much faster and cheaper that it doesn't matter if we don't run other things.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that I find most stunning about this is that transformers were invented in 2017, and the reason that they've taken over everything is that they're extremely good at using parallel compute. And so the question is, are transformers the last great neural network architecture? And our bet is yes, or at least that they will be dominant for the foreseeable future, which means that if you build a chip that is perfect for transformers, you will have a chip that is perfect for AI.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's special about transformers specifically is that you can make an ASIC, an application specific integrated circuit, specifically for transformers, and that chip will be so much faster and cheaper than a GPU at running transformers that it's almost incomprehensible. GPUs are a general-purpose chip that has to do everything, whereas an ASIC can be laser-focused on one task.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that the attention mechanism, which is the key innovation, is actually a fixed algorithm. It's not like the weights change the algorithm; the weights just parameterize the algorithm. And so if you know that the algorithm is fixed, you can build hardware that's specifically designed to run that algorithm as fast as possible.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that actually, when you look at what all these models are doing, they're all basically running the same algorithm, which is the transformer architecture. And so if you build a chip that's specifically designed to run transformers as fast as possible, you can get enormous speedups compared to a GPU, which is a general-purpose chip that has to be able to run any algorithm.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really special about Sohu is that it's an ASIC, meaning it can only run transformers. It cannot run anything else. And that might sound like a huge limitation, but we think it's actually the killer feature because by hardcoding the transformer architecture into silicon, we're able to get massive gains in efficiency over a GPU, which has to be flexible enough to run any workload.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that I think people underestimate about ASICs in general is that when you build something that's an ASIC, you can't change it. And that seems like a huge weakness, but actually it becomes a strength because you can optimize the hardware so specifically for one thing that you get much better performance and power efficiency than you would with something general purpose like a GPU.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

A few years ago, when they set out to build a better AI chip than the largest companies in the world, almost everyone I called told me it could not be done. They have since done it, taping out a working chip on their first attempt and becoming the first hardware company founded after ChatGPT to do so.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The reason GPUs are so inefficient for inference is that they're general purpose. A GPU has to be able to run any possible AI model, and so it has a very complicated control flow architecture. But when you're doing inference, you're running one model over and over again, billions of times a day. You don't need that flexibility.

Decoder with Nilay Patel

Rewind: CEO Jim Farley on Ford's EV gamble

The M1 chip in a new Mac or iPad Pro is made using a 5nm process, but a lot of the chips in cars are made using a 55nm or even 90nm process, and as demand has skyrocketed, the factories using that older tech just can't keep up.

China Decode

China’s AI Has Apple PANICKING, DeepSeek's Billions, and the Californication of Food | China Decode

Allowing one of America's most impressive tech companies to buy Chinese chips made by uh this CXMT company which is a very proficient maker of uh memory chips and does so at a pretty low price uh will be a huge leg up to China and therefore it will be countering America's broad strategy of trying to keep Chinese tech contained trying to keep its advance contained.

Decoder with Nilay Patel

Rewind: CEO Jim Farley on Ford's EV gamble

The chips in cars are made using much older technologies: the M1 chip in a new Mac or iPad Pro is made using a 5nm process, but a lot of the chips in cars are made using a 55nm or even 90nm process, and as demand has skyrocketed, the factories using that older tech just can't keep up.

Decoder with Nilay Patel

Rewind: CEO Jim Farley on Ford's EV gamble

The M1 chip in a new Mac or iPad Pro is made using a 5nm process, but a lot of the chips in cars are made using a 55nm or even 90nm process, and as demand has skyrocketed, the factories using that older tech just can't keep up. As more and more things turn into computers, the more problems across the business landscape look like the problems of the computer industry.

China Decode

Hong Kong's AI Crackdown, Lululemon’s Marketing Backlash, and World Cup Fever | China Decode

No other company, no other country in the world can make these so-called EUV uh machines. And without them, you simply can't make the world's most advanced semiconductors.

China Decode

Can China REPLACE Nvidia? | China Decode

Chip companies in the future might stop obsessing so much as they do right now about how small the transistors in a semiconductor should be and they might focus more instead on how fast the data moves through a semiconductor.

Invest Like the Best with Patrick O'Shaughnessy

Gavin Baker - Watts and Wafers - [Invest Like the Best, EP.473]

the two physical constraints that in Gavin's view will dictate the next phase of AI. On power, he thinks the near-term shortage starts to ease in 2027 and 2028 as new sources of energy come online, and that orbital compute solves it in the long term. On wafers, he explains what is different this time from the dotcom bubble and why TSMC's capacity decisions may be the single most important variable to watch.

TBPN

Nvidia Restarts China Sales, Vibe Coding Backlash, Peptide Craze | Diet TBPN

The more the economies are interlinked, the less likely there is a conflict. So all of this underscores the importance of TSMC Arizona Samsung Intel broadly as well as startup fab projects.