Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's special about transformers specifically is that you can make an ASIC, an application specific integrated circuit, specifically for transformers, and that chip will be so much faster and cheaper than a GPU at running transformers that it's almost incomprehensible. GPUs are a general-purpose chip that has to do everything, whereas an ASIC can be laser-focused on one task.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The reason GPUs are so inefficient for inference is that they're general purpose. A GPU has to be able to run any possible AI model, and so it has a very complicated control flow architecture. But when you're doing inference, you're running one model over and over again, billions of times a day. You don't need that flexibility.