Roundup #89: It isn’t X, it’s Y

I’m having a lot of fun writing shorter posts and aggregating interesting items these days. A lot of people are putting out an astonishing amount of good content these days, and sometimes I just want to sit there absorbing it all.

1. AI risk has gone mainstream

Americans have been deeply pessimistic about AI for a while now, but their reasons for pessimism — or at least, the reasons they tell pollsters — have changed in recent months. Earlier this year, Americans were mostly concerned that AI would take their jobs. Now, they’re more concerned that AI is getting too powerful for humans to control:

Echelon didn’t poll people about the idea that autonomous AI would destroy humans on purpose. But they did ask about bioterror risk, which I’ve been yelling about for a while now. And it turns out that Americans are pretty worried about that:

Surprisingly, the issue hasn’t fallen victim to partisan polarization yet. As you can see in the chart above, Trump voters and Harris voters are about equally as concerned about humanity losing control of AI. And despite Trump’s staunch stance in favor of acceleration, a lot more Republicans want to slow AI development down:

Meanwhile, economists have gotten in on the AI risk debate — which is how you know it’s really gone mainstream. Drew Fudenberg and Andrew Koh have a new game theory paper about “pacing the frontier” — i.e., about whether it makes sense for top AI companies like Anthropic and OpenAI to slow down AI development in order to allow “alignment” research time to catch up.

It’s a very cool model. Basically, the idea is that every company has a competitive incentive to make its AI more powerful as fast as possible, in order to stay ahead of the other companies. But if you’re comfortably ahead, you can afford to slow down a little bit, for safety’s sake — because in this model, if AI gets too powerful before safety research can catch up, everyone could die. So you can get a sort of stop-start pattern where the leading company voluntarily slows down for a while, until its competitors start nipping at its heels again. You can almost sort of see this happening, with Anthropic refusing to allow the public to access Mythos earlier this year, and OpenAI recently pausing development of its top models after some of them hacked the government.

There’s also the case where there’s no clear market leader, in which case companies have to basically agree to all slow down together. In this case, what you really need is transparency — the companies have to all see that the others aren’t secretly racing ahead behind their backs. That might be easier said than done — it’s not clear how to monitor all the AI labs in the world to each other’s satisfaction.

But one positive result is that if the risk of “doom” is high enough, slowing down becomes the only rational option. That’s cool! Unfortunately, this is just one model, which might not hold in reality. Game theory was famously unreliable when people applied it to the Cold War — a small change in assumptions could flip the optimal strategy from “the only winning move is not to play” to “nuke em all and let God sort em out.” Game theory yielded important conceptual insights — especially the importance of “second strike” capability in preventing conflict — but it rarely gave definitive answers.

So where might this current model break down? I don’t think it’s clear how fast AI safety research is really advancing. If “pacing the frontier” only works by giving safety research time to catch up, and safety research isn’t really advancing, then we’re all in big trouble.

Anyway, Andrew Koh has a great thread summarizing the paper, and some folks made