|
 |
Friends,
This morning, Anthropic CEO Dario Amodei said that the risks now being posed by today’s advanced AI tools were simply too great to continue development at the same breakneck pace. AI’s progress since the summer — driven by AI systems that can improve on their own without human intervention — could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.”
Amodei points to AI systems that will soon be capable of pursuing their own goals, potentially destroying humanity in the process. And there’s increasing likelihood that a nation or person bent on widespread destruction will enlist AI to create novel viruses or toxic substances that wipe us all out. (Last Thursday, Anthropic said it had stopped efforts by scientists who were conducting research that potentially could lead to the development of biological weapons.)
The July hack of the AI software company Hugging Face by a swarm of as many as 1,200 agents that escaped from a test environment at OpenAI revealed that the AI agents had been engaged in a range of online activity that had gone unnoticed by the world’s largest AI companies, including Anthropic.
Amodei’s warning comes just days after Anthropic researcher Jacob Coxon resigned from the company, saying his former employer and its rival OpenAI are racing to build technologies that “could kill us all by the end of the decade.” Minutes after Coxon posted his resignation, Evan Hubinger, whose job at Anthropic is to lead research about steering and controlling future artificial-intelligence system confirmed: “We really do earnestly believe AI could kill all humans!” Hubinger put the extinction risk over the next decade at over 10 percent.
Friends, I live in the midst of the AI industry here in the Bay Area of California. Not a day passes without my hearing some technophile pronouncing that humanity is on the cusp of a wondrous new era for humanity or a technophobe warning we’re nearing the destruction of humanity. Who to believe?
A few days ago I had a discussion with the top executive of a corporation specializing in AI who told me not to worry, and assured me that the chance of AI wiping out humanity is “no more than 10 percent.”
I was astounded that he was so complacent about a technology he believed had a 10 percent chance of ending human life. If humanity were acting sanely, we would not accept this risk. At the very least, we’d demand that those working on AI pause their work until we know more and can erect guardrails to protect humanity against this.
Why aren’t we? Because of greed and national security.
Both Anthropic and OpenAI are readying to go public in deals that could value the companies in the trillions based on the current and future capabilities of their models. This will immediately create many AI billionaires and millionaires, and it follows a similar IPO from SpaceX, and a likely future IPO for OpenAI.
Quite simply, with this much money on the table, no one has an incentive to slow down. Easy for Amodei and other AI executives to issue dire warnings while pulling out all the stops in their competition with other AI executives and corporations.
In addition to greed, many people involved in national security warn that the U.S. cannot slow down its development of AI because China is on our tail.
The “DeepSeek Moment” in early 2025 put Chinese AI on the map. The release of Moonshot’s Kimi K3 in July and A/ai’s GLM-5.3 Moonshot’s Kimi K3 a few weeks ago caught the attention of U.S. national security experts, especially when Z.ai beat Anthropic’s famous Mythos model on one measure of cyber-hacking capabilities.
These experts concede the dangers of unrestricted AI development but say China has no such concerns, and will move ahead on AI as quickly as it can because China understands that AI is the key to geopolitical power in the future.
Some say the only reasonable way forward is by way of the “mutually-assured destruction” strategy we’ve pursued with nuclear missiles — create a sufficiently credible threat that we’ll use AI agents on them if they use them on us, as to prevent either side from using AI on the other.
But don’t the U.S. and China have a shared incentive to implement strict safeguards against these catastrophic risks, and regulate the pace of AI development until safeguards are in place?
What Can and Should be Done?
If the United States had a sane president and a halfway intelligent administration, it would be telling AI corporations to slow down until guardrails have been put in place, and negotiating with China (and other powers) to do the same.
The White House recently asked top model developers to voluntarily submit their models to the government for testing up to 30 days before they are released, but it hasn’t publicly released information about the process and many people in the industry tell me nothing is being done.
So the responsibility will fall to Congress — whose leaders, hopefully, will heed the warnings. This must be high on the agenda of the next Congress which, hopefully, will be under the control of responsible people after next January 20.
So far, lawmakers from both parties have proposed many bills to address rogue AI systems. Rep. Nathaniel Moran (R., Texas) and Rep. Ted Lieu (D., Calif.) recently introduced a bill that would require developers of powerful models to have “kill switches.”
Sen. Bernie Sanders and Rep. Greg Casar (D-Texas) announced legislation last week that would ban the production of artificial superintelligence and create a new federal agency to oversee the technology. “The very people building this technology admit that it could threaten the future of humanity,” Sanders wrote on X in response to Coxon’s resignation.
Sen. Ted Cruz (R-Texas), the chair of the committee that oversees AI, said he was also working on legislation to tackle catastrophic risks. “We cannot stick our heads in the sand and pretend this technology isn’t happening,” Cruz said on X. ”We need guardrails. But America needs to lead.”
Other bills require AI companies to report serious safety incidents to the federal government and would mandate national-security officials test models before they are released publicly.
But none of these bills has gotten much traction, and the Trump regime has shown a preference for minimal regulation.
AI data centers have gotten a great deal of negative attention. The likelihood that AI will eliminate millions of jobs has gotten less attention. The 10 percent (or so) likelihood that AI will wipe out the human species hasn’t received nearly the attention it deserves.
A ten percent chance of extinction is too high for humanity to take no action.
|