Most 'make AI go well' interventions are insurance against bad outcomes, especially tail risks. My meta-level argument is that the best way of converting money into impact is to identify interventions that have the property of paying off big in both worlds: by producing step-changes in welfare in the everyday world as well as significantly reducing tail-risks in the emergency world.
the power and problem of Anthropic is the same: the company's safety superpower is that every action it takes looks, from the outside, to be self-serving, even as the company becomes ever more convinced its motivations are pure.
Trump administration officials tell WIRED that if Anthropic wants to rerelease Fable 5, it will need to ensure the model's guardrails can't be circumvented. Security experts say that can't be done.
"When Fable 5 is used for frontier LLM development, it does not notify the user and instead limits the model's capabilities through methods such as prompt modification, steering vectors, and PEFT."
But as AI becomes more and more agentic — as we turn over more complex and longer-lasting tasks to intelligent machines — it's going to be harder and harder to keep them aligned with what humans actually want. And if there's one thing humans will always have a comparative advantage at, it's knowing what we want.
The ex-employees, who cofounded a new AI watchdog group, say investors deserve more information about xAI's safety practices before SpaceX goes public.
Any system has only a finite number of security vulnerabilities, so if we have new AI models that are good enough to comb over the code and fix the weak points very quickly, that should privilege the defense over the offense.
We tend to conflate power-seeking AI and superintelligent (in science and tech) AI. I'm not denying that AI can be power-seeking. Whatever skills and drives Donald Trump has could be embodied in a digital mind. I'm simply pointing out that the way we're currently making AI systems smarter (training them to be really good coders, thought partners, and general coworkers) is not that strongly correlated with power.
this year's report emphasizes that while AI capability is accelerating, the governance and safety frameworks meant to manage it are struggling to keep pace.
Anthropic says its new model is too dangerous to release; there are reasons to be skeptical, but to the extent Anthropic is right, that raises even deeper concerns.
Given the rate of AI progress, it will not be long before such capabilities proliferate, potentially beyond actors who are committed to deploying them safely. The fallout — for economies, public safety, and national security — could be severe.
Why on Earth would you make something that you thought had a 25% chance of wiping out your entire species? Or even a 5% chance? I don't know about you, but to me that sounds like a pretty stupid thing to do!
3mo ago
Underscored — save the words that stop you in your tracks.
