Any system has only a finite number of security vulnerabilities, so if we have new AI models that are good enough to comb over the code and fix the weak points very quickly, that should privilege the defense over the offense.
We tend to conflate power-seeking AI and superintelligent (in science and tech) AI. I'm not denying that AI can be power-seeking. Whatever skills and drives Donald Trump has could be embodied in a digital mind. I'm simply pointing out that the way we're currently making AI systems smarter (training them to be really good coders, thought partners, and general coworkers) is not that strongly correlated with power.
this year's report emphasizes that while AI capability is accelerating, the governance and safety frameworks meant to manage it are struggling to keep pace.
Anthropic says its new model is too dangerous to release; there are reasons to be skeptical, but to the extent Anthropic is right, that raises even deeper concerns.
Given the rate of AI progress, it will not be long before such capabilities proliferate, potentially beyond actors who are committed to deploying them safely. The fallout — for economies, public safety, and national security — could be severe.
Why on Earth would you make something that you thought had a 25% chance of wiping out your entire species? Or even a 5% chance? I don't know about you, but to me that sounds like a pretty stupid thing to do!
1mo ago
Underscored — save the words that stop you in your tracks.
