A coordinated slowdown in AI progress would be bad for these companies' bottom line, because it would allow upstart competitors to catch up. So the fact that they're still calling for a slowdown, in defiance of their own financial interests, is a clear sign that their worry about human extinction is sincere.
"I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to."
What's happening here is obvious—eight different people building this technology are afraid of what they're seeing. But they don't have the power individually to stop the insanity. Trillions of dollars are getting pumped into a runaway technology, and the public already hates it intensely.
Off the top of my head, I'd give AI-enabled bioterror about a 10% chance of bringing down civilization, and about a 30% chance of causing truly world-changing levels of destruction. Climate change is a big deal, but AI bio risk is a lot bigger — it's more catastrophic, it's nearer-term, and we have much less of an idea about how to deal with it.
Though it wasn't all the way there, Claudeasey Newton felt like a validation of the anxiety I started feeling earlier this year: any part of my job that a model can't do today, it may very well be able to do soon. Which left me thinking about why I do this job in the first place.
The current paradigm, in which human civilization is confined to one planet, exposes humanity to existential threats that are unpredictable and uncontrollable on a planetary scale. By moving beyond the only home we have ever known, we ensure species-level redundancy and that the light of consciousness will not be tied to a single planet subject to the inevitable hazards of a harsh and vast universe.
3mo ago
We are now at a point where we must decide whether we are to have a future on this planet, not because of the atomic bomb, but because of the way we have treated the world.
4mo ago
We are dying from civilisation and its discontents; we have invented the machine, and now the machine is reinventing us. We must learn to think in a new way, or perish.
4mo ago
We are dying from civilisation and its discontents. The machine has possessed us, and we must either master it or perish.
AI worry isn't anything new. We are worried about artificial intelligence taking jobs, about toys that seem too real to our kids, about mass surveillance of our every move. But Anthropic's warning about its own product is bigger than any of those singular problems. It is a call from inside the house that disaster is hiding right around the corner.
Given the rate of AI progress, it will not be long before such capabilities proliferate, potentially beyond actors who are committed to deploying them safely. The fallout — for economies, public safety, and national security — could be severe.
Why on Earth would you make something that you thought had a 25% chance of wiping out your entire species? Or even a 5% chance? I don't know about you, but to me that sounds like a pretty stupid thing to do!
Like almost everyone in the AI model-making industry, Anthropic's employees believe that they are literally creating a god, and that this god will come into its full existence sooner rather than later.
6mo ago
Underscored — save the words that stop you in your tracks.
