Lex Fridman Podcast

#503 – AI Crisis in Math: Navier-Stokes, Riemann Hypothesis & Complexity Theory

AI solved some of its hardest problems: Navier-Stokes, the quasi-Riemann hypothesis, the unique games conjecture and hundreds of other problems.

The a16z Show

The Race to Recursive Self-Improving AI

Their goal is a closed loop where the system being improved is also doing the improving, across components including models, harnesses, and eventually more of the AI stack.

Future Proof

Alex Stamos on The Growing AI Hacking Risk, & The 'Almost Religious' Safety Debate | Future Proof

I think one of the criticisms that has been somewhat fairly leveled at the big L labs has been that there's been a lot of focus on alignment that the way you protect against these models is you train them to not want to do bad things when maybe just putting a firewall in place could do it, right? Like we don't have to spend gigajoules of energy changing the model weights when you can just actually apply tried and tested cyber security ideas that we've had for decades to contain these models during training.

Diet TBPN

AI Video Levels Up, Amazon Blocks Muse, Paramount Stays in CA | Diet TBPN

Gone are the days of uh face swaps... this guy who's just like a hobbyist machine learning engineer figured out how to get Dream Booth running on like basically commodity gaming hardware. So then you could go and you could take your face, fine-tune it, and then be able to generate new photos in the style of whatever.

The a16z Show

Who Grades the AI Models? | Ben Horowitz & Rayan Krishnan

As public benchmarks saturate and models get better at optimizing for the tests themselves, Rayan makes the case for independent, continuously evolving evaluations.

TED

How AI Is Breaking the Internet (and What to Do About It) | Matthew Prince | TED

For the first time in human history, we have a mathematical model of all of human knowledge. It's not perfect, but that's essentially what all of the LLMs are.

The a16z Show

Can Open Source Keep AI Power From Concentrating?

today's concentration may be a feature of the current technological paradigm rather than a permanent feature of AI. Transformers reward enormous amounts of data and compute, but future breakthroughs could make smaller, more specialized models far more capable.

The a16z Show

Fei Fei Li: The Race to Build World Models For AI

where Fei-Fei argues that one of today's biggest constraints is access to real-world training data

The a16z Show

Daniel Litt: The Mathematician's Guide to AI

Today's models can grind through calculations, combine known techniques, and search enormous spaces, but still struggle with intuition, theory building, identifying the right questions, and developing the kind of big-picture understanding that drives much of mathematical progress.

Decoder with Nilay Patel

Welcome to the AI crisis in math

It's funny that AI systems are all still pretty bad at elementary school arithmetic, but getting increasingly good at very high-end abstract math. That raises some big questions for the field of advanced math.

TBPN

AMD CEO Lisa Su Live on TBPN

What I believe is that driverless cars today and lots of automated systems look very much like NLP systems in the 2010s. They are these very complex very handtuned systems. They're intelligent but they're intelligent in sort of isolated ways and that those systems hold back what these technologies can do.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's special about transformers is these attention layers where every token is looking at every other token. That operation, when you implement it on a GPU, is really inefficient because you have to load the weights of the model for every single token that you're generating, even though the weights don't change between token generations. And that's really the core insight behind why ASICs make sense for transformers — you can etch the weights directly into the chip.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that the model architecture is essentially fixed. The operations that transformers do are the same whether you're running GPT-4 or any other large language model — it's always matrix multiplications and attention. And if you build a chip that only does those operations, you can make it much faster and much cheaper than a general-purpose GPU.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's so special about transformers is they can be trained end to end with gradient descent. You don't have to come up with the algorithm yourself. You just show the network examples of inputs and outputs and it figures out the algorithm. And because the algorithm is learned rather than programmed, it can be much more sophisticated than anything a human could come up with.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that the weights are basically fixed after training. You load the weights once onto the chip, and then you just do inference over and over again. So if you build a chip that is specifically designed to run one model or one architecture, you can get massive efficiency gains because you're not paying for the flexibility of being able to run arbitrary code.

TBPN

5.6 Sol + Superapp Release, AI 2040, Tibothy Joins, New Robot Hand Alert, Meta Releases Muse 1.1

Already, SOL has been transforming our research program. As one example, GPT-5.6 SOL autonomously post-trained 5.6 Luna.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that I find most stunning about this is that transformers were invented in 2017, and the reason that they've taken over everything is that they're extremely good at using parallel compute. And so the question is, are transformers the last great neural network architecture? And our bet is yes, or at least that they will be dominant for the foreseeable future, which means that if you build a chip that is perfect for transformers, you will have a chip that is perfect for AI.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that the attention mechanism, which is the key innovation, is actually a fixed algorithm. It's not like the weights change the algorithm; the weights just parameterize the algorithm. And so if you know that the algorithm is fixed, you can build hardware that's specifically designed to run that algorithm as fast as possible.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's special about transformers is the attention mechanism, and the attention mechanism has this property that every single token has to interact with every other token. The amount of computation required scales with the square of the sequence length. So as you try to process longer and longer contexts, it gets quadratically more expensive.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that's really interesting about transformers is that actually, when you look at what all these models are doing, they're all basically running the same algorithm, which is the transformer architecture. And so if you build a chip that's specifically designed to run transformers as fast as possible, you can get enormous speedups compared to a GPU, which is a general-purpose chip that has to be able to run any algorithm.

Invest Like the Best with Patrick O'Shaughnessy

Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]

The thing that I find most compelling about the transformer is that, unlike prior neural network architectures, the transformer's compute requirements are almost entirely matrix multiplications. And matrix multiplication is a very structured computation that you can build hardware specifically to do very efficiently.

The Business of Tech

Kiwi co-founded world‑builder hits $2.5 billion valuation

Odyssey AI is not trying to build yet another large language model. Instead, Jeff Hawk, the company's co founder, is focused on world models AI systems that learn from site and sound to understand and simulate how the real world works over time, predicting what happens next and letting us interact with those simulated futures.

Semafor

DeepMind Chief Demis Hassabis Says Google’s Still Winning AI Talent | Semafor Tech

It was basically sort of thought to be career suicide. Like of course we know AI doesn't work. You know, we tried it in the 90s, places like MIT and it was a dead end. You know, and that was the prevailing view. But we just felt the small band of us felt that that actually with the right ideas and using learning systems, reinforcement learning and betting on neural networks that a lot of fast progress could be made.

The Business of Tech

Kiwi co-founded world‑builder hits $2.5 billion valuation

world models AI systems that learn from site and sound to understand and simulate how the real world works over time, predicting what happens next and letting us interact with those simulated futures.

Decoder with Nilay Patel

Microsoft AI chief thinks superintelligence is near, but won't take your job

We covered everything from his approach to training new models to his criticisms of Anthropic talking about Claude as though it is conscious.

The a16z Show

From Models to Mobility: Building Waymo with Dmitri Dolgov

getting there required more than better models. It meant building a complete system for training, evaluating, and deploying a driver in the real world.

The a16z Show

What's Missing Between LLMs and AGI - Vishal Misra & Martin Casado

what's actually required for AGI: the ability to keep learning after training and the move from pattern matching to understanding cause and effect