The acceleration era of A.I. began for a surprising reasonSomething really big happened last week in the world of artificial intelligence: Quite suddenly, after several years of relentless and disorienting momentum, the course of rapid A.I. progress ran headlong into a wave of social panic. It had already been a dizzying summer: OpenAI agents finding their way out of their sandbox and onto the internet to hack the coding database Hugging Face, the controversial announcement of an A.I. solution to a difficult open problem in mathematics, an Anthropic report on the misuse of its generic models for purposes of bio-research and asymmetric warfare. Then, last Tuesday, Jacob Coxon, a junior researcher at Anthropic, posted an alarm-raising resignation thread to X, which included a post bluntly stating that “the people building A.I. earnestly believe that it could kill us all by the end of the decade.” The first post in the thread was seen close to 200 million times, including by prominent A.I. leaders who agreed that, yes, there was at least a 10 percent chance A.I. would bring about human extinction within the decade. From there, the panic spread to Washington, with dozens of national political leaders giving voice to the anxiety and demanding some form of immediate policy response. Over the weekend, the heads of the four biggest American A.I. firms — Dario Amodei of Anthropic, Elon Musk of SpaceXAI, Sam Altman of OpenAI and Demis Hassabis of Google DeepMind — called for a collective slowdown of the development of frontier models and some new shared paradigm for A.I. governance. There followed a backlash to the backlash, too, with a hands-off coalition — including Palantir’s Alex Karp, Nvidia’s Jensen Huang, former administration A.I. czar David Sacks and President Trump himself — dismissing the calls for oversight, some parts of the coalition casting them as a cynical ploy to cement the standing of the leading companies. But the weekend still looked like an inflection point, with Barack Obama mobilizing Democrats around the cause of A.I. oversight, Bernie Sanders and Steve Bannon staging a rally for “pro-human” A.I. and leading companies promoting bipartisan legislation to embed independent monitors in what they still like to describe, sweetly, as “labs.” For almost four years, since ChatGPT’s release, American politics has taken a hands-off approach to A.I., despite growing public anxiety and many conspicuous declarations from those inside and atop A.I. companies that their work imperiled the world’s collective future — whether through mass unemployment, terrorism, social collapse or something potentially worse. The companies issuing those warnings have built out safety teams, but have been operating in general as though the logic of technological progress forced them to accelerate faster and faster. This weekend, that accelerationist posture fell apart. I am not an A.I. doomer. I estimate the chances of artificial intelligence bringing about extinction over the next decade at zero. But as with climate change, the end of the human experiment is a tremendously high bar. And although I don’t really trust the A.I. companies, or the more floridly concerning descriptions of recent A.I. behavior, I do worry a lot about risks, many of which are more probable if the leading companies can’t promise to precisely monitor, control or even fully understand the behavior of their models. Which is why, over the weekend, I found myself breathing a sigh of relief, even though the promise to slow A.I. development was mostly notional, the details left to be debated with the rising anti-slowdown faction and the framework mostly unchanged from a letter signed by industry leaders just two months ago (to much less fanfare). The truth is alarming warnings and estimates of apocalyptic risk aren’t really either. In a large 2022 industry survey, those working in A.I. estimated, on average, a 16 percent chance of humans going extinct or losing control of the world because of A.I. That figure grew to 18.3 percent in 2024. Plenty of prominent A.I. figures have given even higher probabilities of doom, amounting to a running paradox for those trying to make sense of the industry from the outside: Why would so many people who believe this technology represents an threat to the survival of the species be racing so fiendishly to build it? That these warnings were not new is one part of what has made the whole period so strange. It’s tempting to doubt the sincerity of founders and engineers, or to treat the warnings as so much marketing hype or cultic delusion. But the American people haven’t exactly been ignoring the alarm. Instead, as we heard more from A.I. companies and employees, we grew more anxious about A.I. than just about any other country in the world. Then we watched as functionally nothing was done outside the companies themselves to meaningfully rein in the pace of change. In fact, the opposite: By any concrete measure, the country threw more at the project of building artificial general intelligence than it had at anything in many decades. And today the number of Americans who believe A.I. is moving too fast outnumber those who believe it is moving too slow by about 33 to 1. Yes, there were a couple of hearings and a Biden-era executive order, later revoked on the first day of Trump’s second term. Yes, there was backlash to data centers, and even a raft of local moratoria on constructing new ones. Yes, there were a couple of pieces of federal legislation introduced, often by Senator Bernie Sanders. But even though polling now suggests that Americans support a ban on superintelligence by a margin of 68 percent to 25 percent, each of these bills got so little hearing in Congress that they seemed pretty obviously D.O.A. Over the past four years, the infrastructure build-out continued; indeed, it accelerated. Demand for A.I. grew by leaps and bounds, and at the leading companies revenue exploded, as did the estimated value of their forthcoming I.P.O.s. Every few months there was a disconcerting news story or incident report, documenting some unnerving behavior of A.I. agents, producing a new flurry of broader anxiety. But the country’s entire apparatus of public oversight — legislative, regulatory, legal — simply stood back and stood down. It was though we had collectively decided to defer to the companies themselves to manage risk as they saw fit. When OpenAI agents in the midst of a poorly constrained training run hacked into the A.I. database Hugging Face in July, it did not really register at first that what had happened was a crime, governed by existing statute and punishable in straightforward ways by existing authorities. In a slightly earlier or somewhat saner era, that might have been the primary way an event like this was understood and ultimately policed — as damage done by one company to another in violation of existing law. Federal investigators and federal prosecutors might have jumped into action at the news, and been applauded for doing so by a nervous public. The investigation and prosecution might have cast a chill on reckless development and deployment in other companies. And others following the story from the sidelines might have begun tallying up the potential liabilities of companies whose leaders kept declaring quite openly that their tools might soon cripple the entire internet, destabilize whole nation states and compromise sensitive personal information and global financial databases in one fell swoop. As Gabriel Weil and Lina Khan have argued in recent days, existing law might not be precisely optimized to deal with A.I. — but there are laws, and other policy approaches, should we choose to make use of them. Instead, our democracy adopted a posture of learned helplessness. It wasn’t just A.I. founders who seemed reluctant to talk in terms of criminal statutes and civil liability, which imply direct lines of personal and corporate responsibility. It was also the country’s lawmakers and, to a certain degree, the public too. To many, A.I. has looked less like a technological innovation than a stage of evolution, following a path that human designers might influence but not really control. In fact, as Kelsey Piper emphasized last week, losing control of A.I. has always been the companies’ actual plan, since many of the biggest gains were projected on the other side of what’s called recursive self-improvement, which would enable models to train their own successors without human oversight. But outside certain circles, there didn’t seem to be that many asking, “What is to be done?” More common, even among the terrified, was, “What comes next?” How did this happen? If the panicked flurry of the past week marks the end of a period of reflexive accelerationism and the beginning of a new period of regulatory contestation, what gave that earlier phase its heedlessly Wild West character? One obvious factor is given by the recent history of the technology itself, which progressed so quickly that it was hard to wrap one’s head around, let alone to determine one’s preferred policy response. Another factor is the A.I. competition with China, which had simmered in the background before coming to a boil when DeepSeek released its impressive open-source model in 2025. A third is the simple force of capital, given how much investment has been dumped into the A.I. boom and how much immediate revenue and profit that investment requires to be made whole again. And a fourth is the ideological argument those investments came wrapped in — a sort of move-fast-and-break-things techno-libertarianism, with Uber and crypto both serving as recent illustrations that disruptive innovation can reliably outflank and outpace oversight and regulation when fueled by billions of dollars in venture capital. Those histories all matter. But I would propose another contributor, perhaps less intuitive: Covid. As the country emerged from the pandemic emergency, it did so with a new, elite consensus: that America had grown too risk averse, too tangled in regulation and oversight, too scared of its own shadow. The critique covered the pandemic response, but its aperture was much wider: In its governing institutions, its bureaucracy, its status quo bias and its social instincts, a country forged on the libertarian frontier now found itself trapped in a safetyist prison. The critique has an unmistakable right-wing valence: Republican governors embarrassed that they had locked down as strictly as Democrats had, for instance, or plutocrats swinging to Trump citing progressive excess and regulatory intrusion. Since the Trump administration returned to the White House in January 2025, it has been on a speedrun of policy recklessness, testing guardrail after guardrail, and perhaps the most telling meme of these years has been the notion of “suicidal empathy.” But you could see that same surge of anti-safetyism elsewhere on the political spectrum. With Covid, it was Joe Biden who declared the end of the pandemic emergency, and it was his administration that dramatically revised reporting standards so that levels of Covid infection that had been color-coded as extremely high were suddenly redefined as benignly low. Liberals stopped wearing masks, by and large, and some began making fun of those who did, as right-wingers had done before. In the center, a whole intellectual ecosystem sometimes known as progress studies devoted itself to unshackling innovation, and the major reform movement of post-pandemic liberalism, known as abundance, was built around the principle that both private and public actors had been hemmed in for too long by regulation and social caution to achieve meaningful change in the world. These impulses were not all pernicious, but together they signaled a particular direction of change — away from caution and oversight and toward dynamism and risk. And the pattern was reflected, as in a hall of mirrors, almost everywhere you looked in American life. In the markets, pandemic stimulus gave way to SPACs and NFTs. Five years ago, crypto looked to regulators like a questionable, often criminal enterprise; now the big banks are all in on it, as is the ruling party. And while the market kept going up, perhaps the most iconic financial figures of these frothy years were those whose bets exploded: Sam Bankman-Fried and Leopold Aschenbrenner. At the consumer level, sports betting and prediction markets, only recently legalized, seemed to grow as ubiquitous as Hollywood, despite regular alarms that on-demand gambling was driving up personal bankruptcy, home foreclosures and rates of domestic violence. Some Americans increasingly looked across the Atlantic at sclerotic Europe with a kind of smug sense of superiority, but to many others it felt as though the cost of boosting American growth just a bit was living more of our lives according to the risk-thirsty laws of the casino. There have been countervailing trends, too, as there always are: Drug use is down, for instance, as is teen pregnancy, and MAHA argues not just against overregulation but underregulation. Even Trump’s trade war and xenophobia are expressions of protectionist control. But a new, entitled recklessness was the essence of the ballyhooed vibe shift of 2024, said to mark a turn not just from Bidenism to Trumpism but from a cautious nanny state to something much more reflexively hands-off, risk wise. In the stats guru Nate Silver’s terms, it was the triumph of gamblers of “the river” over the worrywarts of “the village.” To hear venture capitalist Marc Andreessen theorize it, the shift seemed to mark a kind of declaration of independence not just from D.E.I., wokeness and pandemic restrictions but from a longer and stronger form of liberal paternalism that had sapped American vitality, and clipped the wings of American greatness, for a generation. In a series of high-profile lectures, Peter Thiel has described promises of safety and social protection as a sign of the coming of the Antichrist. And even in more liberal circles, you began to hear the word “degrowth” as a kind of all-purpose slur, as though anything but a maximally rapid pursuit of change qualified as a form of superstitious Luddism. If the A.I. panic shows us nothing else, it’s that this was always a false binary. We don’t have to choose between progress and safety. More to the point: in a democracy, or something close to it, we shouldn’t let others choose for us. Read past editions of the newsletter here. If you’re enjoying what you’re reading, please consider recommending it to others. They can sign up here.
|
