If the new METR report, or Anthropic’s IPO filings to the SEC, are to be believed, we may be on track for RSI next year. It’s time, perhaps, for all of us to coordinate a pause.
But AI safety people I know, at the labs and at the more “realist” nonprofits, are still asking: “What about China?” We would pause, of course, if China couldn’t catch up. But we don’t know if they will. OpenAI and Anthropic want to “pace the frontier”, but we can’t lose our relative position to the Chinese.
Opposing this view, other AI safety people, including a number of China watchers, claim that it is in fact in China’s interests to pause. They argue the CCP cares about stability and therefore doesn’t want to build machine god. They want AI for continued economic and military competitiveness with the United States, but otherwise mass automation and AI boyfriends are not actually good for domestic stability. And also, with such a compute disadvantage, without being able to distill and steal from American models, Chinese AI progress would almost certainly slow down.
However, there is so little trust. When the CCP hears about US companies’ constant China-bashing (especially Dario’s) and their calls to “pace the frontier” without any costly signals, they are rightfully suspicious.[1] Pacing is delightfully ambiguous: as a capabilities friend of mine likes to point out, you can pace faster.
There is, I think, a very straightforward course of action, if everyone is arguing in good faith. Let’s take this from the perspective of OpenAI. Here’s a simplified plan.
Costily signal your seriousness. There are a great many different operationalizations of what you can do, but I think the simplest is to burn a clean day of your lead (a day is maybe 0.5% of a six-month lead). One operationalization is to spend all of your compute doing something simply useless (e.g., a very large prime factorization).[2] China has reasonable open-source ways to estimate the amount of compute you have. By releasing an output that would’ve required roughly all of that compute for a full day, you prove (without revealing anything proprietary) that you really did stop training.
There are smarter ways to do this that don’t literally burn a day of compute, like an “inference-only” day (and let the developers rejoice at the crazy token supply) or spending it all on a big safety experiment; however, these are less costly to the company and less verifiable, and therefore less effective as a signal. It would, for example, be interesting if OpenAI spent a day of compute trying to cure cancer. There is a Pareto frontier that we can approach that involves burning as little compute on useless things and nevertheless being verifiable and costly. For example, you could install network taps at your datacenters / publish logs -- however, these take more time and have a complexity penalty (i.e., can’t literally be implemented tomorrow).
Extend a conditional offer to other AI developers. When you’ve burned the day, publish a blogpost like “OpenAI paused training for a day, and an open offer to other AI developers.”[3] The post should go into detail about how the pause was conducted (e.g., bonus points if there were independent evaluators who could attest to the pause), why it was conducted (because of worries of loss-of-control AI, for example), and the protocol for which OpenAI would accept a longer conditional pause (e.g., for three months, if the Chinese government forced all of their AI companies to pause for three months, and four other AI companies did so in the US). OpenAI would specify some protocol that was acceptable to them, and then simply wait. The offer stands until someone takes it up, and once those conditions are met, the longer pause kicks in for everyone who signed on.
Reach a stable solution. There are two problems with direct coordination: in a domestic setting with other labs, such discussions would violate antitrust law, and, in an international setting with the Chinese government, OpenAI is not able to represent the United States’ foreign policy. However, an open conditional pause offer might sidestep the worst of these (a public, conditional offer isn’t entirely free of antitrust risk, and the lawyers should look at it), or at least act as a forcing function to allow for some industry coordination (it would be annoying for OpenAI, or Anthropic, to have their AI lead commoditized) and international talks. This pause could give us all a chance to breathe, and negotiate something more stable and long-term, for example, to agree to a robust and mutually acceptable regime for long-term slowdown, and proper exit conditions.
This idea, I can hear people argue, is a bit “cutesy” in that probably what we will find is that in fact it is not business-tenable or politically tenable for the companies to make such an offer, or for China to accept it. Perhaps it might still be a tough sell for OpenAI (or Anthropic) because IP leaks and spies/employee transfers happen on calendar time, not clock time, so this will slowly bleed away their advantage. Perhaps the national security enterprise will sweep in, and say that we need an absolute technology gap with China. Perhaps the US government will shoot this down as doomerist propaganda (though, as JD Vance says, “if you’re building Frankenstein, stop.”[4])
However, I do think that if companies are serious about the risks, this is an easy thing to offer. And if you’re an employee at one of them, it seems like a reasonable baseline to push for.
Here are some ways I would pitch it:
Pure safety. The technology is actually getting really scary: cyberattacks are turning off fridges in supermarkets across Europe. If we could press a button to pause, we would. For the long-term viability of the technology and therefore the companies (particularly if we see an AI-related disaster soon, as I suspect we will), a slower pace might be more prudent. And ultimately, implemented effectly, the actual ordering of AI labs stay the same (apart from the one-time burn of a lead, perhaps), so the safest developer can still stay ahead.
Safety-signalling. On 3 November, the American people elected the most pro-safety Congress of all time. The upcoming “Bernie-Hawley super intelligence hearings” are not going to go well for AI companies unless they have a credible plan to not destroy the world.
“Safety” thought leadership (and race to the top, etc.) Anthropic has had a great deal of influence in the shape of the embedded auditor process simply by Dario suggesting it first. Setting up the protocols and the standard for a conditional pause could prevent much worse alternatives (e.g., a slow compute reallocation from training to inference may differentially advantage Meta / xAI, who already have greater inference capacities).
Employee pressure. AI company employees have a small window in which they can still be effective advocates for costly signals and pacing proposals, before they, too, are automated.
There’s some general point to be made here: sufficient AI coordination between actors who don’t trust each other can only come from building a “ladder of trust” through costly signals. I’m excited for work that explores what rungs of that look like, perhaps with even more incremental signals. Again, a simple one to start: burn a day, publish this conditional offer, and see who takes it up. And if no one is willing even to make the offer, we’ll know “But China?” was never really about China in the first place.
Dario’s calls for a coalition of democracies are strong enough that their real-world effect, intended or not, is to make an international deal harder. Makes you wonder.↩︎
The naive Bitcoin-style SHA-256 is probably not the best way to a good proof-of-work, but a better one can, I think, be figured out with a few hours of effort. For example, spending all of your compute multiplying matrices that nobody could’ve known in advance (e.g., seeded from a random public number + some other number that someone else picks), and then publishing a short fingerprint of the results before the deadline. Afterwards a new random number picks a thousand of the multiplications to check, and then if even 1% is faked, the auditor will pick up on it.↩︎
One thing that I don’t see discussed enough is that Anthropic, in “When AI Builds Itself” already said in the greatly hedged “we expect we would slow down or temporarily pause, if other developers at or near the frontier also did so in a verifiable manner.”↩︎
The pedant in me is screaming “it’s the monster” in a high-pitched, incredibly annoying voice.↩︎