To make AI go well, we must solve “AI alignment”, cooperate under competitive pressure, and ensure AI benefits everyone.

To belabor the point, AI safety is about to face the largest nonprofit windfall in history. The OpenAI Foundation’s stake in OpenAI is now worth $180bn. Anthropic cofounders have pledged 80% of their wealth. Those AI companies tend to triple in valuation every year, so hundreds of billions may be an undercount. And it might be liquid as soon as this year! Coefficient Giving expects to move ~$1bn in global catastrophic risk grants in 2026 (the thus far largest funder of AI safety) and, not to be outdone, plans on deploying scaling much more.

The traditional grantmaking approach is to fund existing organizations. Yet, we cannot possibly spend hundreds of billions that way: vetting is slow, existing organizations can’t scale. Even if we decided to fund every reasonable-looking org (METR/Transluce/RAND) to hundreds of millions a year, try desperately to seed new ones, and run a regranting program that puts the Future Fund’s one to shame, that closes maybe $20bn of the gap. What about the rest?

I’d argue that we should fund some massive, public prizes: grand prizes for full problems, sub-prizes for intermediate milestones, and tooling grants (compute, tokens, datasets) for competitors. The work can begin immediately: though OAI and Anthropic equity is mostly illiquid, prizes can pay out as liquidity arrives.

AI Alignment

Start with alignment. Indeed, I can already hear Eliezer Yudkowsky objecting: “Your alignment prize is doomed. You have ‘one critical try.’ Your toy solution proves nothing.”

Yet, alignment can be broken into -- still incredibly difficult -- subproblems that are easier to verify progress on: adversarial robustness, formal verification, full mechanistic interpretability.

On a tiny model, we can (without superintelligence) verify whether your solution survived an adaptive white-box automated attacker, produced a closed-form proof bounding the model’s behavior, or fully reverse-engineered the model’s internals. Then follow with prizes for reducing the “alignment tax” until they can be deployed at frontier-scale.

Heck, while you’re at it, go 10x or 100x bug bounties for finding jailbreaks (those prizes on Grayswan are looking awfully small), and fund all interesting demonstrations of model organisms. And this may be harder to verify, but fund prizes for creating agendas and strategies to hand off to the future automated alignment researchers and answer some of those hard questions.

We can damn well make sure that this “one critical try” is slated in our favor.

Cooperation Amid Competitive Pressures

Incentivizing cooperation is a harder, fundamentally sociopolitical problem. Tax-exempt nonprofit rules close off the obvious lever of large-scale political spending for regulation and diplomacy (though wink wink, Anthropic cofounders).

Prizes work differently here: they reduce technical friction to cooperation. In general, I think this is a great target for many technical researchers otherwise in crisis because they aren’t in a lab. Political will, when it does happen, can only reach for tools that exist. As Dean Ball tweets, political winds change on a dime: just last week, the Trump administration was reportedly considering pre-deployment evaluations and US-China negotiations. However, without win-win technical solutions, policymakers either don’t act, or grab the easiest legible option (like a datacenter ban).

Take compute verification. Cheap, trustless attestation of training runs is the technology that makes a US-China deal possible -- without it, neither side can credibly commit to any mutual terms. Without prizes, these problems are orphaned: too applied for academia, too unprofitable for VCs, too technical for most philanthropy. They also span domains no single team holds in-house: cryptography, hardware security, international law.

Ensuring AI benefits Everyone

From redistributing wealth, to curing cancer, to protecting children, or animals, or even the AIs themselves, the subproblems for ensuring AI benefits everyone range from barely scoped (child safety) to well-mapped (biosecurity).

Where “invention” is the bottleneck, a similar prize paid on the first qualifying solution fits cleanly (Alzheimer’s, a provably-secure web stack). Where the bottleneck is deployment, manufacturing, or regulatory derisking, an advance market commitment (a purchase guarantee) fits better. The $1.5B pneumococcal AMC delivered three vaccines and saved 700,000 lives. Both are pull funding: pay on outcomes, let competitors organize around the rubric.

In biosecurity, threat models (natural/artificial pandemics, rogue actors) and promising approaches (PPE stockpiles, wastewater sequencing) are well-understood. A Warp Speed-style commitment to buy N95s could provide every household in the US with PPE without the poor CG biosecurity team having to figure out the logistics. Capitalism will provide.

Contrast this with “child safety” (or job displacement, or AI consciousness), where no similar playbook exists. For these, before a prize, the primary work of the foundations will be to convene experts and define the problem. We can imagine some clarifying problem decompositions (parasocial attachment, AI-generated CSAM) and accompanying technical solutions (robust LLM personas, better classifiers).

For these pre-paradigmatic fields, the foundation’s first dollars should go to experts themselves: pay them to publish threat models, candidate interventions, and potential rubrics. Like podcasting, the difficulty is asking good questions.

Subscribe now

Implementation

Why are prizes so rare? My best guess is institutional incentives. Grantmakers, frankly, aren’t rewarded for impact -- they’re rewarded (or simply not fired) for deploying capital defensibly, such as to fancy institutions. They rarely have the technical expertise to write a good rubric, or the resources to hire experts who do; what they can do is sniff out which orgs seem credible to their networks.

The new capital coming online has much less baggage: generally everyone is making new hires, with few prior commitments. For the OpenAI Foundation in particular, the optimistic, ambitious vibes of a prize are much less awkward than funding civil society grantees or might publicize or undermine their for-profit arm. And talent! Talent and founder-level “capacity” is so much of a constraint. Well, applying to little-publicized “requests for proposals”, with bounded upside, is not a good way to attract ambitious founders. Even Dwarkesh is hiring his research collaborator, this round, with a blog prize, rather than a job posting.

And, ultimately, prizes can scale. I cannot imagine us building the capacity to evaluate $30bn/yr of individual grants. Let’s do some napkin math: that’s 6000 grants of $5mn, which today’s ~100 or so reasonably-shaped organizations cannot absorb. 20-30 high-leverage prizes cost $30bn/year -- a few in the billions, most in the hundreds of millions. Beyond the prize itself, the rest goes to providing competitors with compute, tokens etc. For example, compute verification may mean actually renting a datacenter for testing hardware proposals (if you know anyone who knows how I can get one, I’d love to hear about it).

And finally, “taste” is a rare, rare commodity. Traditional philanthropy asks one grantmaker to choose the problem, the team, and the approach. For prizes, you need taste only on problem identification, not on the hundreds of org-by-org bets that follow. The result is a field that defaults to whatever passes the in-group sanity filter — a large part of why AI safety has produced, charitably, modest progress.

Prizes aren’t the whole answer. You cannot feed a researcher with the promise of a prize. While VC-like syndicates fronting researchers in exchange for a slice of the upside sound like libertarian fantasy (and it is, at smaller scales). I do think, though, for literal-billion-dollar prizes, the math works! The $10M Ansari X Prize pulled $100M+ in research spend across 26 teams.

Here’s a funny failure mode: what if the prizes are too big, and, like those investment bankers who retire early, what if the world’s best talent wins a few of the early prizes, and retire forever. I’m not actually concerned about this: I know the quants are in it for the love of the game, and the money just got them through the door

Some inputs, like accessing internal frontier models, may never be available. We won’t be able to democratize that. Long-horizon institution-building, weirder conceptual research, political lobbying will need different (probably traditional) vehicles.

Yet, from commercial aviation, to self-driving cars, to deep learning, well-scoped prizes have repeatedly cracked difficult sociotechnical problems. It’s no coincidence that the last serious attempt at deploying AI safety money at scale came to the same conclusion: I remember hearing rumors in 2022 about Leopold’s $1bn alignment prize proposal at the Future Fund. Let’s do that.

Thanks to Sophie Kim, Saheb Gulati, Arunim Agrawal, Jasmine Li and Richard Ren for feedback and discussion.

Scroll around the organizations on this cute AI safety map. Think about, or ask, how does their work help with AI safety?

My guess is that, maybe some won’t know, but most would say “X of our actions will lead to Y self-evidently good outcome.”

• Maybe, they’ll say, if we measure model capabilities well (we have the graph), policymakers will wake up and see AIs are improving quickly, and get their act together. In the best case, we could become a neutral third-party evaluator, and we could tell the labs that they can’t release the next model because our autonomy & deception evals say it’s too dangerous. Better information leads to better decision-making. You couldn’t be against that.

• Maybe, if we invent the scaffolds for the right AI monitors (AIs looking at the output of other AIs), we could coax interesting work out of near-AGI systems that we don’t trust to help automate alignment. In the best case, these millions of automated AI alignment researchers, at the brink of AGI, solve the problem. You couldn’t be against that.

• Maybe, at least publicly, there is no unifying thread to our work. We do things that are the obvious low hanging fruit, again and again and again, and that’s how we contribute value. Maybe we help whistleblowers, or potentially-sentient AIs, or research mentors with good taste. Maybe we ask unusual philosophical questions. Maybe we meld into the national security establishment, or the government. Maybe we organize a movement of people. Maybe we just say the truth. Surely, you couldn’t be against that.

Perhaps you can see the holes in some of these arguments, or actions. But “generically good thing” isn’t enough! Take that thread, ask the question: how do these actions, in concert with others, lead to us “winning?” I would argue, however, that very few of the actions or organizations have a theory of victory -- a full (causal) story of the steps that need to go right, and a plan to achieve them. Very few begin to ask.

In April, I gave a talk at an AI policy workshop about these theories of victory, and how to develop your own. The rest of this post is an annotated and lightly edited version of the talk below. It’s a long one.

Subscribe now

Overview

The main claim, up top:

We’ll quickly preview:

What is “Victory”?

There are many different potential threat models in AI.

Many people in many different organizations care about many different things, but perhaps two that we can agree on are:

Perhaps there is a more specific vision of what a good future might look like, perhaps some form of Utopia (like Forethought’s Viatopia or Nozick’s “Utopia of Utopias” and Alexander Wales’ lovely fictionalization of it), but for this talk we’ll stick to the broadly agreeable things.

Theories of Victory (ToVs)

To achieve a good outcome, people normally advocate for “theories of change”, which tend to take the form: “here’s my best guess for a causal chain for my action X causes Y.” X is a well-defined action (start an organization, run this media campaign etc.), and Y is a broadly-accepted “good thing” (“increases awareness of AI risks” “increases the probability of rational decision-making”)

This is, however, importantly distinct from a “Theory of Victory”, which I would understand as the full (causal) story of the steps that need to go right for us to “win” (where, borrowing the broadly-agreeable understanding of “win” from earlier -- we don’t die + the benefits of AI are broadly distributed.)

There’s this great quote from HPMOR against complex, multi-step plans (cited originally in Scott Alexander’s brilliant book review of Zero to One). I disagree! Yes, indeed, they are fragile, but you should know the causal story of how all the actions are going to lead to the correct things happening.

So how’s this different from a Theory of Change? It accounts for

1. All of the actions of all the actors, the steps whether they are performed by you or not

2. The full win condition, rather than simply a “self-evidently good action.”

***

ToVs in AI Safety

The next section of the talk is a quick overview of the explicit and implicit (unsaid) assumptions underlying what I, and regular readers of this blog, would term the “two camps in AI safety.” I will say, they don’t often agree, and can have quite vicious disagreements indeed. (insert Freud here about the narcissism of small differences)

The camps are broadly the “alignment is hard but tractable” folks (“EAs” “moderates”) and the “if anyone builds it, everyone dies”) folks (or “doomers” “humanists”).

Camp 1

In Camp 1, we find the vast majority of funding and attention in AI safety. One can maybe summarize the specific solution prescribed by this camp as “the right people need to win the race” (so, the US, for democracy’s sake, and Anthropic, so that responsible labs are in the lead, and you can buy time at the frontier to do safety work). This position is perhaps best captured by Leopold’s Situational Awareness, which is worth reading in full (but in particular “The Free World Must Prevail”)

I’ll rapid-fire through a bunch of the explicit and implicit assumptions that underlie this worldview. The question of AI Alignment (getting AIs to do what we want), first and foremost, needs to be tractable: not to say that it is easy -- though some in this camp will claim that we got lucky with current systems -- but that it is an “engineering-shaped” problem, with continuous improvement possible from existing system, and would benefit from it being automated.

If you believe these systems are controllable, then the controller matters -- ideally it should be the best or most moral people (US/Anthropic), perhaps with some democratic input.

This view would look favorably upon the progress that has been made so far, favorably upon the lab leaders (or at least some), where market incentives don’t result in everyone dying, and that existing techniques perform surprisingly well.

Automating alignment is a top priority (getting AIs to help us do our homework) and, implicitly, the alignment problem is one that AI can help solve without itself being aligned first (the entire “getting work from untrusted systems” motivation behind much of the AI control agenda). This view is best written in Carlsmith’s “AI for AI safety” series, but also hinted at by Jan Leike’s “Alignment is not solved but it increasingly looks solvable.”

And finally, on “competitive pressures”, that a pause is unrealistic, and that the games and the incentives are fixed. Political action is harder to influence and has more backfire risk, that not building superintelligence ever is an economic and moral disaster, an existential risk of its own, and that the public should work within institutions, and build technocratic competence. Don’t, they say, activate forces that you can’t control.

I will, here, interject with my opinion. Perhaps best summarized by my post “whose alignment research are we automating”, and the general take of we should not rely on the benevolence of individuals (like lab leaders) for our good outcomes, no matter how similar to us they seem (particularly crazy given that those individuals are saying that the technology they are building may end the world), and that we should not rely on Kumbaya / rational decision-making / a warning shot to prevail for pausing right at the edge to do automated alignment research.

Camp 2

The second camp would, of course, advocate for a ban on superintelligence and, to achieve this, maybe a global treaty, because “if anyone builds it everyone dies.” FLI, for example, might advocate for a carve out of broad scientific and democratic consensus, but maybe not.

Again, going through the different explicit and implicit assumptions. AI alignment is an unsolved scientific problem, we have one critical try. Even if its only a 20% chance that we all die, why not do the more cautious thing, wait 5 years, and lower that further. This is everyone’s lives.

There is no correct winner in the race, it’s “if anyone builds it.” The only winner of a US-China race to superintelligence is the superintelligence itself. The problem is therefore a political one -- we need to buy time through collective action, and the bottleneck isn’t technocratic competence or “thoughtfulness”, it’s aligning political incentives. The AI safety progress we’ve made so far, some of it has been significant, but much is safetywashing that does not help with the hard problem.

I’d recommend reading Eliezer’s “list of lethalities”, the case against AI control research and Control AI’s plan, to get a flavor of what the views and actions of this crowd will be.

Yes, more, that lab leaders are rife with conflicts of interest, that the “AI safety” community has plausibly been net negative (with its $30m donation to OpenAI, its contribution to the funding and founding of Anthropic, the original inspiration of Deepmind). That RLHF, and RLHF++ and other existing alignment techniques will not be robust enough to prevent deception, particularly on hard to verify tasks, that we don’t know who’s research we are automating, and that sounds like a cop-out.

Perhaps the method here is that if you say the true things loudly, combined with a crisis, that persuasion will be enough if you’re right (huh, notice how that is a bit similar to the evidence-presenting pitch of the first crowd…)

I will, again, say that this theory of victory is not sufficient! My opinion: in some versions, this worldview relies on a warning shot being legible and not catastrophic, that people who are right, and right loudly, can persuade powerful people, that government can act fast enough, and that there exists a stable equilibrium where we can just stop -- I’m skeptical of all of these, or not confident enough to bet humanity on it.

A quick recap

Others

I will, of course, say there are many other theories of victory, and none of them are sufficient (and some of them just barely manage to scrape by as theories of victory at all -- many are fuzzy or implied on the full causal chain). I’ll tour through some others, briefly.

1. Def/acc + AI resilience. Let’s differentially boost defensive technologies like pandemic preparedness (Vitalik / OAI foundation / the smarter techno-optimists). Well, you better hope that the world favors defense over superintelligence, and I don’t think it does.

2. Liability + insurance mandates (the more pro-market of the worriers -- I like Gabe Weil). Just require the AI companies to get insurance against destroying the world, and the insurers will price it. Problem: If the stakes are the whole world, your insurance might be expensive -- insurers really don’t like insuring for highly correlated large events, like pandemics. Otherwise, if you make companies liable for near-misses too, good luck defining and detecting those, and similarly hoping that warning shots will happen.

3. State regulation (some of the AI safety c4s out there). Maybe the “patchwork” of state regulation may save us all. Problem: pre-emption. China. I’m not sure this counts as a full theory of victory, and I’m pretty sure the proponents would agree.

4. Coordination by Kumbaya. A strategy I like to think comes from the philosophical school of “people are rational actors, and they will update rationally in response to real evidence”, or, alternatively, hoping that Dario and Demis will get their 5 year pause by magic (I still love the Economist for forcing them together to sit uncomfortably close on that couch). Problem: people aren’t rational agents.

5. MAD-for-AI. As advocated best in “Superintelligence Strategy.” Maybe deterrence, or broader AI geopolitics, may find that equilibrium. Problem: convincing the US / China to preemptively cyber/physically escalate against another nuclear armed power is a really high bar, the government is slow. Cyber-hardening.

6. The market will solve it. (e/accs, a16z, some of the tech right) What’s the quote from the techno-optimist’s manifesto? “To ensure the techno-capital upward spiral continues forever.” Problem: …

Subscribe now

Like economic models:

Developing Strategic Taste

Why is it so important to write your own? Well, only by being explicit, can you understand how important your actions are, and what things need to be true about the world for your steps to make sense.

Secondly, the power law applies all the way down. I think the best way to illustrate is an example. Let’s say you’re really confident that you don’t need a theory of victory -- of course, how could something that is as broadly good as “AI verification” be bad? It’s pretty simple, it is robustly good to be able to verify properties about AI systems.

Well, I’ll tell you an account of some hypothetical people working on AI verification.

One group builds a really elaborate, steel-encased AI chip, with these fantastic security properties, but has to be custom installed on every chip, pursued through a startup to raise lots of VC funding. The other uses off-the-shelf hardware taps, and some software, and says that’s enough -- recognizing their product isn’t profitable, and being a nonprofit instead.

The second group’s actions, I claim, are much more useful because they have an eye on their “win condition:” a US-China treaty, and recognize that they do not need to produce the fancy startup, or any new technology at all. This allows them to have “taste all the way down”, and placing the full optimization pressure they are able to muster on the right part of the problem (rather than give into other pressures, like status-seeking or similar, that push them towards startups or AI labs or whatever)

You can see these people distinguish themselves as “pragmatic” (eg. pragmatic interpretability, pragmatic safety).

How do you develop your own? I like a method (that I’ve written at length about here -- much of which I will say verbatim) of “decompose and resolve.” The basic idea is simple: you can hear the lie when you say it. It works on your personal life (try saying: “My life is 100% perfect”, and then decompose: “my social life is 100% perfect” all the way down), and, also, on breaking down the problem. See the following few steps.

The thing about taste is that it requires knowing what the questions are. It is quite an obvious thing, but asking repeatedly — what do I care about? What is my big goal (make AI go well)? My big goal (ensure everyone has freedom)? And do an exercise of breaking it down from there.

If you want to figure out what you care about, think about what your goals are, and then ask why.

“I want to solve climate change.” Why do I want to do that? “Because climate change will hurt millions of people in the next few years.” Why does that matter? Because people’s happiness and flourishing is terminally valuable. That is what you care about.

“I want to earn money.” Why? Money is not a terminal value (unless you terminally value a number going up in your bank account) — it only is valuable because you can use it to buy things. “I want money for a sense of security and the ability to support my family.” Because their security and happiness is terminally valuable.

The next thing is to say the false things out loud.

Notice what objections your gut brings up to that, instinctively. Break it into the subcomponents. Eg, “Superintelligent AIs will follow human instructions”, “If technical AI alignment is solved, society will be okay”, “Everyone will benefit from AI automation”.

Continue to break down the points into smaller points until you get confused about the object-level.

Perhaps you are confused about some fact about the world (eg. will the AGI be coming soon?) or no longer at the object-level at all, but instead social reality: for example, “oh I’m actually just deferring to this person/’scientific consensus’ who believes X” or “I’m just following the precautionary principle / [some other heuristic] generally applied”.

Once you’ve done this, try your damnest to resolve your confusions about the object-level claim. Some ideas of how you could:

• Doing a ton of directed reading/research

• Chatting with the AIs — be careful, they tend to be good at amplifying whatever thing you subconsciously believe, and are not very good at fighting back

• Speaking to a friend who has thought about this — in general, people who have just been around for longer have asked themselves the same questions, and would find it interesting to do that kind of exploration with you

• Or experts in the field — I’d recommend reaching out to them cold with a specific question about their work, while doing enough prep such that it would be incredibly unlikely you could’ve figured out the answer by reading things on the internet / speaking with Claude

Now, zoom out and survey the landscape. There are at least some smart people that are trying to help — why are the things they are doing not sufficient? Usually by asking this question, you’ll realize there is maybe a structural thing going on (the bottleneck to any project happening is X funders’ risk aversion, or lack of money overall, the people aren’t that talented) or that they have missed a key strategic insight (you would be shocked at how few people actually do the steps outlined above, and reason from first principles).

And finally, write it down, and act. Write out your theory of victory. It should be a few bullet points, and all the assumptions and uncertainties. Share it with people you know! Share it with your smart friends who know how to ask questions. Share it with me (@jason.17 on Signal).

Common Traps

There are a few traps that you should avoid. Like Claude, you are a general agent. Yes, personal fit is important, and burnout is bad, but also you are more capable than you know. If you think public buy-in is important, start a movement. If conservative AI policy is, go work on that. If you think the national security establishment is, then you can reach people there too.

As AI safety becomes more established, we start to see more established status ladders, and regular paths. And don’t get me wrong, these programs are great, but be careful -- don’t turn your brain off for someone else’s agenda, or because doing the next program is cool and high-impact. They exist to serve you, and you should view them as resources, not goals.

Particularly if they are very competitive, you may be less counterfactual (ie. it would exist without you) -- do the thing that wouldn’t exist without you.

I hear many a person who has said “the portfolio approach” allows me to work on this (I don’t want to have to make a decision / leave my cushy job, so I’m going to “make a bet on long timelines”, because some part of the portfolio needs to, and I’m an unusually good comparative fit.)

Portfolios and decorrelated bets are really important, but be specific about what assumptions need to be true for your bet to make sense. Ideally, for example, you can choose to play on hard mode, where you are working in the fraction of worlds where alignment is hard, timelines are short, and warning shots are unlikely -- that’s a part of the portfolio that is neglected.

The canonical reference. Maybe you’re not Actually Trying. Even if you’re exceptional in one part of your life, this other might be a blocker. Ask yourself the question: “What would I do if I was 10x more agentic?” Also Do One Thing.

Finally, I would encourage you, stare at the hard problem. This requires courage, because it may lead you to conclude that the past 5 months of your work contributed very little value, or were totally misguided.

Stare deep into the void, and then come out again, with a plan. Good luck.

Subscribe now

Here are all the slides (request access)

Thanks to Sophie Kim for her extensive feedback and pushing me to post this.

Here are ten questions I think matter for broad AI policy and strategy right now, with some brief notes and sub-questions. They have been on my mind because I think it’s possible to make serious progress, broadly underexplored (though I’d love to be shown wrong!) and important factors for understanding the gameboard in AI. If you’re a researcher looking for a project, a funder looking for a gap — perhaps a place to start.

1. The implications of AI disruption. Does the US initiating AI disruption against China increase or decrease the likelihood of a bilateral cooperative agreement between the two countries?

The case for: policymakers only act in crises, leaders want to avoid instability, disruption shows that the US is taking AI and national security seriously. The case against: cyberattacks are bad for building trust and good will.

How would China actually respond to low-level US cyber sabotage of AI training runs, and what escalation dynamics should we expect?

2. Definitions. What is a good definition of intelligence recursion? Superintelligence? What are good red lines otherwise for deterrence or regulation?

My brainstorm attempt. Which is the best? (eg. for the criteria of observability, precision, relevance, timeliness and gaming resistance)

3. Securing AI datacenters against nation-states. If both the US and China were seriously bought into securing their AI datacenters, how long would it take until they were resistant to different levels of nation-state attacks?

For every given point of time in the future, how much time does “locking down the labs” take? What is the full space of AI disruption attacks that could occur?

On the default trajectory, how far away are we from SL-5? Are we even trying? Perhaps the SL-5 taskforce / RAND will have an answer.

4. Follow the money. Are the current labs spending on compute and datacenter build-out as if they believe “AI will automate most of the economy?” If not, which part of the compute supply chain is the most skeptical?

My best guess is that the chips are flying off the shelves, and it is TSMC + the fab build out that looks conservative, but I don’t know. If this is the case, why don’t buyers sign purchase guarantees? It would be good to run the numbers. Is it, as Samuel suggests, just being risk averse?

5. AI research progress. My sense is that most employees (not leadership) of the AI labs are skeptical of achieving recursive self improvement within the next 3 years (perhaps with the exception of Anthropic.) What is driving this skepticism?

My approximate model of timelines is that we are ~1 research breakthrough away from “taste in ML” (or, in the most general sense, “taste”), after which we get RSI (and “AGI” is an engineering problem). Such a research breakthrough can be coarsely modelled as drawing from some distribution, as a function of how many scientists are trying + compute that they have, and it does not seem implausible that, e.g., as the neolabs proliferate, these will become more common.

What is a better outside-view model of AI research progress? How about an inside view? (Continual learning? Recurrence? Different architectures? Data efficiency?)

Subscribe now

6. Data poisoning. How much of the internet would you need to poison to meaningfully affect the propensities of the models trained on it?

This Anthropic / UK AISI paper says that a “near-constant number of poison samples“ is enough. If this is true, is this happening already? Can you use this to degrade bioweapon-relevant capabilities for open-weight models?

How do you quantify the effect on propensities rather than capabilities of the models? How robust are those backdooring attacks to fine-tuning and other mitigations? See also the IAPS report on AI integrity and Forethought on catastrophic data poisoning (itself with many interesting and important defense-relevant questions)

How effective is AI-powered training data filtering on removing backdoors? How expensive is it?

7. Chinese Chip production. Is China good at building AI chips? How far behind are they? How long until they reach parity?

There are arguments on both sides. Nvidia is invested in getting one answer, which muddies the field, and the AI safetyists another. It seems like a big deal if true.

In general, is AI a government priority? What leading indicators would we have?

8. Alignment difficulty. There has been a vibe shift in people thinking AI alignment is easier. What is generating that? In general, what are the best ways to reason about alignment difficulty? How solvable would it be as a large-scale R&D effort?

Especially important given that alignment difficulty, I believe, is a cause of the split between camps of AI safety. How does it relate to the previous conceptual leaps, including on simulators and shard theory? My sense is it is an empirical argument — in fact we are seeing less emergent misalignment than of previous generations, and that this might speak to the problem being easier. Is there a more satisfying explanation?

In an upcoming post, I’ll speak a little bit about my simplified, stylized model of alignment through considering the space of human values. I’d be curious to see where that is wrong.

9. AI verification. What is the state of the art in AI verification? How much time / resources would it take to meet the standards of mutual visibility and vulnerability for a bilateral AI deal?

How effective are privacy-preserving AI-agent based approaches / software-or-firmware-only / hardware approaches? Are there other things that you can do? How good are the best open-source datasets?

10. Utopia. What does a good future look like? Is it possible to reach an end-state which pleases all? What are the institutional features of a bilateral / multinational AGI project?

I know some think this a poorly-formed question, and that the journey matters more than the destination, but I’m not satisfied. What is missing from Nozick’s “Utopia of Utopias”? (Perhaps best illustrated here). How do you solve the problem of new minds — ie. What is the decision process for deciding when a new mind / morally-relevant agent can be created? (Should there be a hard rate-limit? Some kind of resource limitation?)

Subscribe now

If answers don’t already exist (online, or in nonpublic documents, or in people’s heads), I believe these could take a talented researcher or organization’s worth of resources. Do email me if you’d be interested on working or funding work on any of these (or point me in the direction of answers that already exist!)

Thanks to Samuel, Adam, Parv, Saheb and Sophie for helpful conversation and feedback.

1. Lack of inference capacity

U.S. export controls on advanced chips have made it hard for China to develop enough inference capacity to serve hundreds of millions of global users quickly. Earlier this week, the Chinese AI company Zhipu AI restricted sign-ups for their AI coding tool due to compute constraints.

Releasing models open-source instead of through an API means that Chinese AI developers don’t need to invest massive amounts to build out their own inference capacity, because users run the models on their own hardware (or third-party providers like Fireworks AI and Together AI step in to serve the models commercially, capturing the inference workload that Chinese labs can’t handle themselves).

There are signs China’s inference capacity is increasing, as it develops more domestic semiconductor supply chains — some suspect the supply to exceed demand as early as 2028.

Export controls under President Trump have also loosened. NVIDIA’s powerful H200 chips are designed for both training and inference, and the U.S. government greenlit their sale to China in December, despite calls to limit the country’s access to inference-capable hardware. The U.S. also allowed the sale of H20 chips back in August.

As this constraint becomes increasingly obsolete, Chinese labs face a different problem: even with sufficient inference capacity, they can’t charge more for models that lag behind Western ones.

Subscribe now

2. Profit motive (“Commoditize your complement”)

While companies like DeepSeek occasionally demonstrate competitive performance on specific tasks, they haven’t achieved the kind of sustained superiority that would justify premium pricing.

So if you can’t monetize through paid API subscriptions because your product isn’t competitive enough, how do you extract value from your AI development?

Open-sourcing becomes the logical alternative: give away the weights to maximize adoption, then monetize through services rather than the model itself– enterprise support, fine-tuning, infrastructure– while building brand recognition for when you eventually catch up on capabilities. This is the Red Hat/Linux playbook: give away the core product, sell the services around it.

The profit motive problem is uniquely difficult to solve because as long as Chinese models lag behind U.S. ones, offering incentives like licensing deals or preferential market access won’t be compelling– why would a Chinese lab accept licensing restrictions to access U.S. markets when their model isn’t good enough to capture significant market share anyway? The closed-model revenue opportunity only becomes attractive if they believe they can actually compete.

Until Chinese models achieve genuine parity or superiority, commercial incentives to stay closed are weak because the foregone revenue is minimal.

3. Strategic disruption

By releasing state-of-the-art models for free, Chinese AI companies force Western competitors to either match the openness– eroding their pricing power– or justify premium pricing when “good enough” is offered for free. This commoditizes the base model layer, potentially reshaping industry economics in ways that favor Chinese players with lower cost structures.

A similar dynamic is currently playing out in the steel market: China’s “steel dumping” practices have made it virtually impossible for steel manufacturers worldwide to compete, since Chinese companies receive government subsidies and are thus able to offer steel on the international market for artificially low prices.

In both cases, China is pursuing an aggressive form of price competition that undermines the economic power of competitors.

4. Signaling & undermining export controls

The release of competitive open-source models, whether intentionally or not, serves Chinese strategic interests by demonstrating that U.S. chip export controls haven’t prevented frontier AI development. DeepSeek’s success signals China’s technological resilience and undermines confidence in export controls as an effective containment strategy.

5. AI diffusion and geopolitical influence

Additionally, by open-sourcing, China arguably gains soft power in defining AI architectures and potentially creates vendor lock-in to their ecosystems. In a future where the world runs on Kimi K2.5 instead of Claude, China plays a much more substantive role in shaping global AI development and adoption frameworks, undermining the power of the West. Imagine the Malawi government is consulting Kimi instead of Claude on national policy.

6. Reduced liability for misuse
Open-sourcing creates legal ambiguity around liability by decreasing the probability of attribution and diluting causation arguments. For example, if someone uses a closed DeepSeek API to design a bioweapon, there are logs, payment trails, and account information; meanwhile, if someone downloads open weights and uses them offline, there’s no record DeepSeek can be forced to produce.

With open weights, Chinese AI labs’ role goes from “service provider” to something closer to “information publisher,” making it much harder to establish direct causation– someone else modified the AI, someone else deployed it, someone else used it for harm.

Responsibility is also distributed between several different platforms that help host and serve models to users, such as Hugging Face and Fireworks AI. Fragmentation across the ecosystem makes any single actor harder to hold accountable.

Subscribe now

Conclusion

While Chinese labs have various technical and commercial justifications for open-sourcing, the actual benefits appear negligible at best compared to the existential risks that open-source models pose. Moreover, open weights eliminate China’s ability to control content, monitor usage, or leverage AI access for geopolitical gain– undermining the CCP’s own goals.

In the next post, we’ll explore a series of interventions designed to stop or slow Chinese open-sourcing, targeting pressure points from commercial incentives and CCP self-interest to market restrictions and supply-chain regulations.

Thank you to Alex Waldherr for informative discussion.

Two days ago, Moonshot released Kimi 2.5 open-weight. It’s quite good. We’ll wait for its scores to come in on the virology capabilities test, a benchmark of virology questions, but I suspect it’ll do well.

The difficulty of bioweapon design is in error correction. People often misunderstand how, mechanistically, LLMs help rogue actors make bioweapons. It’s not that difficult to find the instructions for horsepox (a smallpox variant) online, but the reason culturing and modifying is hard is that it is a multi-step process, where every step can fail.

Let’s say hypothetically, you’re not a great biologist-terrorist (even the best experts you could convince to join you in the caves were not the brightest of their biology classes). Your success rate for every one of the steps involved (eg. determining feasible candidates, growing the biological agent in sufficient quantities) is maybe ~30%, and so at the end of this long multi-step process (what are these weird electron-dense circles in my TEM of HEK293FT cells again? what’s wrong with this picture of my influenza virus plaque assay?), your chance of success is ~0%. In a normal university lab, you would just go ask your specialist colleague.1 But you’re in a cave.

But, imagine you had an excellent multimodal LLM,2 you could maybe increase the success rate to 70% at every step of both sequence design and construction.3 It would still be hard, but you might succeed.

Currently, if you do this with a closed-source model (and Google decides to implement those damn input-output classifiers), particularly combined with an IP address from a “hostile nation”, the AIs will refuse to answer. For open-source models, like Kimi K2.5, no such restrictions apply.

Even folks who have tried to implement tamper-proof “unlearning techniques” for years, or implement safeguards on models they are planning to open-weigh (as Meta claims they do), it is trivially easy to strip these away. The Chinese labs barely bother.

Worse, training a bad model on the outputs of closed-source good ones can increase their overall capabilities (Kimi K2.5 does seem to have an identity crisis, confusing itself with Claude...). So even if the best models were closed-source, any sufficiently powerful open-source models will likely be close behind the frontier, through “distillation” on those outputs, for example.

This suggests that we should be pretty worried. Much biosecurity hardening is going to have to happen on later parts of the supply chain, or involve hardening (e.g., the provision of PPE and similar), but LLMs do provide uplift to humans doing realistic virology tasks.

There may be a sharp transition where designing systems goes from “completely impossible” to “very easy” once reliability reaches a critical point.

Policymakers should prioritizeconvincing or coercing the Chinese AI companies to stop open-weighting their models. That’ll be the topic of the next post.

Subscribe now

Thank you to Antoine Vigouroux for helpful feedback.

Sophie Kim writes a critique of a “Mutually Assured AI Malfunction” (MAIM)1 regime that I had not seen before. My understanding of what she’s saying:

1. Assume the “decisive-strategic-advantage”-only2 version of MAIM — i.e., superpowers will implement a reciprocal deterrence regime because they don’t want their rivals getting the overwhelming military and economic dominance offered by superintelligence, and not because they are worried about existential risk — and only two actors are competitive.

2. Reasoned to a first-level (and this is the argument being made in Superintelligence Strategy, a paper by Dan Hendrycks, Eric Schmidt, and Alexandr Wang), the US will believe: “I don’t want China to reach superintelligence, and thus I’m going to release an internal memo / external public statement that says ‘China reaching superintelligence is a threat to the security of the American people,’ and will deploy escalating force (internal sabotage, cyberattacks, kinetic strikes) to prevent them from doing so.” China will do the same, vice versa. This is MAIM.

3. The first-order implication of MAIM is that if we convince the US government to buy into such a regime (or even any of the Five Eyes), then it’s reasonably likely that it could prompt China (which has good visibility into the actions of each of those governments) to adopt something similar, and thus we would enter into a MAIM equilibrium, whose instability will force both sides to the negotiating table.

4. Sophie argues that the US, if it was sufficiently superintelligence-pilled to consider this and as the leading actor, has very strong incentives to undermine such a reciprocal regime, such as by rejecting transparency measures, hardening its datacenters, and even deterring the deterrence.

   1. One way it could do that is by classifying its data centers as “critical infrastructure”, and then re-emphasising its existing policy that cyberattacks on such infrastructure would be considered an act of war.

5. There is a question about how credible this threat would be, but even if it was just a 5% chance of a kinetic strike, China would not risk it — particularly if the US was able to guarantee or gesture that it was isolationist and didn’t want to threaten the Chinese regime, and was sufficiently irrational.

6. China, therefore, would not be able to credibly deter the US from building superintelligence, and its best option is to instead race toward superintelligence itself. The MAIM equilibrium, therefore, does not emerge at all.

I’d recommend reading her summary of this, as well.

Subscribe now

I’m going to reserve judgment on my overall feelings toward the framework as I’ve been promised some follow-up work that will address it, but I will praise Sophie for coming up with a strategic dynamic that I do not think the original Superintelligence Strategy had considered.

Some Notes From Me

• Information asymmetry favors the US. This dynamic is strongest in worlds with significant information asymmetry, where the US is “superintelligence-pilled” and China doesn’t get it. China is likely to, therefore, think “the US cares about this technology” and would almost certainly not be willing to risk war to prevent the US from building it. (h/t Oliver) Given that China is currently set to underprice a software singularity, and misfocus on “embodied AI”3 and US companies (I’m looking at you, Anthropic) are set on recursive self-improvement, this may constitute a significant fraction of worlds.

• Threat Perception: Stemming from the information asymmetry above, another axis that I believe is implicit in the original paper is the relative extent to which the American government (or defense-industrial complex) and the Chinese government (or CCP) feel like they would be fundamentally threatened by the other. By the nature of DSA, Superintelligence Strategy assumes this would be true — and indeed, if one had control of superintelligence, it’s easy to imagine a way to run a bloodless coup (I would be unsurprised if leading labs already had such a plan internally) — but it is not obvious whether the other actor would realize.

  • Existential Threat Perception: Sophie’s argument broadly didn’t consider safety, but the actors’ ability to model each other’s safety policies matters. If China viewed AI as an existential threat, it would not need to model the US applying its DSA aggressively against the CCP, it could simply act on behalf of the safety of itself and the Chinese people. I do agree, though that this looks unlikely in the current trajectory.

  • It suggests that it would be a non-obvious decision to fully superintelligence-pill the US national security community, unless that would trigger a simultaneous reaction from the CCP.

  • The US is therefore incentivized to clarify that it is, and figure out how it can credibly commit to be non-expansionary / non-threatening to the CCP to prevent it from feeling threatened. This could be as mundane as signalling a more isolationist turn or refocus towards being a regional superpower. Or, in a world where actors were sufficiently worried about existential risk, having strong safety policies could deter MAIMing, since other actors would have less fear for the U.S acting recklessly.

• Multi-Actor Risk: Even if you consider other trailing actors like Russia, the incentives on the US still apply, it does make the calculus a lot riskier — the more cyber-capable attackers who have different risk tolerances, the more likely that the US will have to risk another country calling it on its bluff.

• It’s hard for the US to credibly “deter the deterrence.” The US does not have such escalation dominance where it can proportion a just-escalatory enough response to a cyberattack. For one, the American people would not be excited by a policy that results in nuclear war if anyone attacks their (already unpopular) datacenters. The US would actually struggle to make a “deter the deterrence” threat credibly — it’s just that even their uncredible threat would probably provide enough disincentive for MAIM to exist.

• Cybersecurity could become defense-dominant. With formal verification4, cybersecurity, at least, is a defense-dominant problem — we are a long way off, but plausibly can sufficiently harden data-centers with proto-AGIs. This dramatically reduces both mutual visibility and vulnerability, and is, as Sophie argues, likely by default. MAIM becomes more difficult if the US is fully hardened.

Subscribe now

My Overall Take

My overall take (h/t Adam and Sophie) is that China will probably still be AI-pilled enough to MAIM to US using cyber only, correctly believing that political realities mean that further escalation in retaliation (even in the case of declaring a data center “critical infrastructure”) is unlikely.

But, to state the obvious, a US national security establishment focused on subverting MAIM instead of maintaining the MAIM equilibrium would dramatically decrease the likelihood of such a regime working.

I look forward to hearing the counterarguments on the MAIM proponents’ side. For now, the ball is in their court.

Thanks to Sophie Kim, Adam Khoja, Aiden Kim, and Oliver Zhang for helpful discussions and comments.

I’m a fan of grand ambitious theories. Derek Parfit, it seems, was too. His “moral convergence thesis”—developed across his 1,400-page On What Matters—was essentially that the great moral traditions are “climbing the same mountain from different sides.”

A quick review of those traditions:

1. Kantians generate rules about acts, asking “can this action be universalized without contradiction?” (e.g., the Ten Commandments, or the Universal Declaration of Human Rights)

2. Consequentialists evaluate outcomes (e.g., cost-benefit analysis, effective altruism)

3. Contractualists ask what principles no one could reasonably reject; virtue ethicists ask what helps the agent itself flourish

Sophisticated versions of each tradition, Parfit argued, converge. Kantians who “treat rational agents as ends in themselves,” consequentialists who recognize side-constraints (maybe don’t harvest organs willy-nilly), and contractualists who follow “reasonable principles” universally—all end up in the same place.

Hence his conclusion: an act is wrong if and only if it’s disallowed by a principle that is:

• Uniquely universally willable (passes the Kantian test)

• Optimific (produces the best outcomes)

• Not reasonably rejectable (passes the contractualist test)

I admire the attempt, and I suspect he was pointing at a real phenomenon. But I don’t think he ultimately proved moral realism.

Instead, I’d argue that values are points (or clusters) in a high-dimensional value-space, and moral theories are approximation methods for locating those clusters.

Moral theories are methods. And they’re portable: you can run utilitarian calculations on alien preferences, or apply Kantian universalizability tests to their maxims. The theories aren’t the values themselves. It doesn’t make sense to ask whether Kantianism or utilitarianism is “true.” Both are more or less useful for approximating a given cluster, which can differ across civilizations.

Subscribe now

Values as Clusters

Imagine a space with many dimensions. Each dimension corresponds to something a civilization might care about: welfare, fairness, autonomy, loyalty, purity, reciprocity, and so on. Somewhere in this space sits a region—a cluster of points. This is where a civilization’s coherent values live.

For humans, the cluster includes “protect children, don’t kill unnecessarily, help your kin, don’t take what isn’t yours.” These show up everywhere: Hammurabi’s Code, the Ten Commandments, various Universal Declarations of Human Rights, Claude’s character description.

And while there are differences between human cultures (some tribal societies have very strange practices around, say, cannibalism), a combination of genetic and cultural evolutionary pressures—game-theoretic cooperation, dominance hierarchies, kin selection—has meant these core values are broadly shared.

This framing allows for local moral realism without claiming access to the moral truth. As Eliezer Yudkowsky starkly illustrates in his novella Three Worlds Collide—featuring an intelligent, civilized race of babyeaters—different civilizations can develop coherent values utterly alien to ours, no less internally consistent.

Theories as Approximation Methods

The moral theories, therefore, are not competing descriptions of our values. They’re competing methods for approximating any value-cluster.

Kantianism, utilitarianism, virtue ethics—these are fitting methods. They take different inputs (act-type, world-state, agent-description), apply different procedures, and produce different approximations, but all aim at the same target: whatever cluster the civilization in question has. You can imagine them as vectors from an origin, pointing in the same general direction.

A linear algebra aside

Because linear algebra, I promise, is philosophically useful.

Consider moral theories as using different basis vectors to describe the same value-space. Kantianism carves things up by act-types: lying, promise-keeping, treating as means. Consequentialism carves by outcomes: utils. Virtue ethics carves by character: courageous, temperate, just.

You can translate between them. “Don’t lie” (Kantian) becomes “honesty produces better long-run outcomes” (consequentialist) becomes “the honest person is trustworthy” (virtue). Same region of value-space, different coordinates.

The translations aren’t always lossy. Sometimes they’re just clunky. “Maximize welfare” is a clean consequentialist statement. The Kantian translation—something like “act on maxims that respect rational agents’ capacity to pursue their ends”—is awkward but gets there.

Sometimes the translations are lossy. Consequentialism can say “this outcome is 2.3x better than that one.” Virtue ethics has no native way to express that. The precision is lost in the change of basis.

Under this view, a “moral parliament”—weighting your credence across theories, aggregating their verdicts—is just choosing a weighted average of coordinate systems. Which is fine, but doesn’t resolve anything fundamental. It’s still you picking the weights. (Though perhaps it means you cover more of the space than any single theory would alone.)

Upshot

If we accept this view—that civilizations locally converge upon value clusters, and moral theories are our best approximation methods—a few implications follow:

Moral progress splits along two axes. First, improving our approximation methods: refining theories, resolving internal contradictions, making them perform better. Second, drawing a tighter boundary around what values are actually in the cluster—expanding our moral circle, for instance. Moral progress isn’t climbing toward God’s view. It’s calibrating to our own attractor.

Moral disagreements split similarly. People from the same civilization disagreeing about the cluster can resolve their conflict through evidence or dialogue. Philosophers disagreeing about methods can check which method better approximates the cluster in a given context. But disagreements across clusters—humans versus babyeaters—may not be resolvable at all.

Moral expertise is real, but limited. Philosophers refine approximation methods. That’s genuine expertise. But it doesn’t grant authority over which cluster to aim at. That should be determined by what humans actually value—perhaps, as Yudkowsky’s Coherent Extrapolated Volition would argue, after substantial time and reflection—not by what philosophers argue they should value.

Morality is neither subjective nor objective. It’s not arbitrary like favorite colors, nor mind-independent like mathematics. It’s closest to language, money, or law: real patterns that exist because of collective human activity, and that constrain individuals once they exist. Human ethics isn’t “true”—it’s just what humans do.

Our intuitions are noisy samples from the cluster. Neither infallible nor arbitrary. Philosophical “edge cases” where intuitions and theories diverge are just regions where our approximation methods have high variance. They reveal less about deep truths and more about the limits of our tools.

Subscribe now

The AI Alignment Problem

This brings me to Amanda Askell, Anthropic’s philosopher-in-residence. She has a tough job indeed. In specifying Claude’s “character“—the values and dispositions that guide how the model behaves—she and her colleagues must approximate the human value-cluster well enough that the system behaves in ways we’d endorse, across millions of diverse interactions. No single moral theory suffices. The approximation must be robust to edge cases where our theories diverge.

I have newfound respect for Yudkowsky’s Coherent Extrapolated Volition. It asks: “What would humans want if we knew more, thought faster, were more the people we wished we were?” That’s asking: where’s the center of the cluster, after you remove noise and inconsistency? This view validates the structure of CEV while dropping the realist implication that there’s One True Answer. The cluster is real, but contingent.

And, of course, AIs trained on very different data, or optimizing very different objectives, might develop a different cluster. Not misaligned in the sense of “pursuing human values badly,” but genuinely alien—coherent values that don’t overlap with ours. A different attractor.

What a strange, alien prospect indeed.

Political parties are a mess. Predicting who supports what—and why—has become nearly impossible. That makes forecasting positions on AI, an inevitably new battleground, even harder.

I recently discovered Scott Alexander’s “thrive/survive” theory of the political spectrum (it’s one of those grand ambitious theories which has a delightful simple predictive/retrodictive power about it)

His hypothesis: rightism optimizes for surviving an unsafe environment; leftism optimizes for thriving in a safe one.

This describes the traditional “right” and “left” positions well:

• The right imagines the zombie apocalypse is tomorrow—hence guns, deference to military and police, suspicion of outsiders, reverence for tradition, practical skills.

• The left imagines a technological utopia—hence environmentalism, cosmopolitanism, skepticism of religion, free love and gender identity, openness to social experimentation, status games.

Scott’s theory came during a simpler age, where “left” could reasonably describe the democrats, and “right”, the Republicans. Zohran Mamdani and Donald Trump are both more naturally “rightist”: surviving in an unsafe environment (the specter of autocracy, the “great replacement”). Similarly, the tech right and the abundance left are “leftist”: pro-innovation, set towards thriving in a safe environment. Perhaps a survive/thrive theory for the populist versus the elite. That’s why Gavin Newsom and JD Vance are broadly pro-AI and pro-innovation, and Bernie Sanders and Steve Bannon oppose.

I argue AI will drive left and right toward their logical extremes. Before utopia, humanity is on death ground. And this is driving the populist awakening*

For the elite, AI has potential to bring humanity towards utopia, or, if you will, “Fully Automated Luxury Gay Space Communism.” Free-market-loving transhumanists (like the current crop of AI CEOs) and progressives alike now speak of universal basic income, perhaps administered by a single world government. The people building the technology are disproportionately educated, libertarian, socially progressive coastal elites. Their vision is utopia: abundance without scarcity, liberation from labor, the final triumph of reason over superstition.

For the populists, the unsafe environment, the Zombie Apocalypse — at least when it comes to an actual “Great Replacement” with an alien species — is at the door, and threatens the security of your family. The AIs are an unknown other.

In some sense, you should therefore predict that the most populist are the ones that are most against AI. If you believe that AI has the potential to disempower humanity, either slowly or all at once, then humanity is on death ground. And, in the fight for survival, society may again lurch towards populism.

This pattern is not new. Greco-Roman ideals, then Judeo-Christian reaction. The Renaissance, then the Reformation. The Enlightenment, then Romanticism. Each technological and intellectual surge produced a popular countermovement that reasserted tradition, community, and moral limits. The development of superintelligence may mark the end of a generational run for the rationalist project that began with the Industrial Revolution.

Subscribe now

What might this framework predict?

First, strange bedfellows. The socialist left and Christian right will find themselves in unexpected alliance against AI labor displacement. Both sense death ground—one for the worker, one for the family. Their policy prescriptions will differ. Their enemy is the same. Bernie Sanders and Tucker Carlson sound oddly similar on AI these days.

Religiosity will correlate with AI skepticism, even controlling for education. Not because religious people are anti-technology. Because their framework already assumes humanity is not the highest intelligence—and that hubris has consequences. The Garden of Eden is a story about reaching beyond your station. So is the Tower of Babel. These narratives will find new purchase.

The right, therefore, rejects predictions of utopia. Mark Beall, author of “A Conservative Approach to AGI,” testified to Congress:

When I hear folks in industry claim things about universal basic income and this sort of digital utopia, I study history, and I worry that that sort of leads to one place, and that place is the Gulag.

He may be onto something. The conservative tradition—Christian virtues, suspicion of hubris, respect for what we do not fully understand—may offer the firmest ideological check on the rationalist project to build Machine God.

Not conservative policy—not tax cuts or deregulation or any particular platform. Conservative epistemology. The conviction that human reason is bounded. That we do not fully understand the systems we inhabit. That humility before complexity is not cowardice but wisdom. Or to quote Beall:

Let others dream of building gods; we will defend the human. Let others chase infinite growth; we will cultivate enduring goods. Let others automate civilization’s soul; we will remember that civilization exists to serve the soul.

It’s a beautiful essay.

The new Pope understands this, in his calls for “moral discernment” from the builders of AI.

Despite all that Marc Andreessen may roll his eyes, modern technologists will need it anyway.

Subscribe now

Semiconductor chips are the lifeblood of AI companies and militaries. We should know where they are.

A popular policy proposal is, therefore, to track their locations with a software update. This can curb rampant smuggling to China.1 Furthermore, for any future international treaty on AI, both the US and China can verify that all compute is accounted for.

Even those who support exporting AI chips should favor location tracking chips. If we’re going to pursue a diffusion strategy—exporting chips to allied nations while restricting adversaries—we need a way to verify that those chips stay where they’re supposed to be.

Skeptics of location tracking, including, unsurprisingly, Nvidia themselves, tend to make three main arguments against the proposal:

1. Location verification is not technically feasible / too expensive.

2. Location verification requires backdoors/chips to emit data, which makes them structurally insecure.

3. Location verification can’t be done in a tamper-proof way, and is trivially cost-effective to circumvent.

We disagree with both (1) and (2), which seem like willful misrepresentations, and believe (3) can be effectively mitigated.

1. Location verification is not technically feasible / too expensive.

IAPS has written one of the most promising location verification proposals to date involving “remote attestation”, a security mechanism that “allows a device (or a computing environment) to prove to a remote party that it is in a certain state, running specific software, or other internal details without revealing more information than necessary” (IAPS). The setup is as follows:

1. Establish a set of “landmark servers” that can receive signals transmitted from chips.

2. Update the chip software to enable it to transmit cryptographic proof of identity to those landmark servers.

3. Measure the time it takes for the signal to travel from the AI chip to the landmark server.

4. Using the standard signal propagation speed in fiber optic cables, calculate the maximum distance the chip could be from the landmark server.

5. Triangulate using multiple landmark servers to identify the area in which the chip is most likely located and/or rule out that the chip is not in a restricted location.2

IAPS guesses that this can be done for less than $15m/yr with existing technology. Per year, that’s about the size of a seed round of reasonably-hyped SF AI-wrapper, or the cost of ~1/10th of one mile of California high-speed rail. Cheap. And it makes sense! It’s easier to find a lost AirPods case than a $60,000 Nvidia GB200, at the moment. Apple has done it.

What makes the feasibility debate appear especially disingenuous is that NVIDIA has already built and deployed similar systems. They offer a Remote Attestation Service that establishes device identity, maintains secure cryptographically signed channels, prevents spoofing, etc.; the company is already operating verification infrastructure at scale. Instead of “building a system from scratch,” the technical gap appears to be closer to “adapting / expanding existing security infrastructure” to include location data.

2. Location verification requires chips to emit signals/data, which makes them structurally insecure.

Proponents of this argument argue that adding “backdoors” that can emit the location of the chip, or even just a signal, creates a security vulnerability. You could try and smuggle out the model weights that have been loaded onto the chip, or the output of its operation. Engineering security vulnerabilities makes them both more prone to hacking or sabotage, and thus less attractive to buyers.

This argument seems to misunderstand what data location verification transmits. IAPS’ location verification proposal doesn’t even send coordinates– just a simple ping and response with cryptographic proof of identity– so there’s no “room” to encode arbitrary information without breaking the cryptographic proof, and no real “data” is being emitted.3

Essentially, while a new communication pathway is established– in the form of the chip sending pings to landmark servers– since these pings contain only timing metadata with no computational workload data, the privacy concerns are minimal.

Often, what critics are worried about is not necessarily the attestation security features that location verification is built upon, but more so the enforcement mechanisms that have been proposed as add-ons (i.e. geofencing and kill switches).

Subscribe now

3. Location verification can’t be done in a tamper-proof way. (h/t Pranav)

There are four major ways a delay-based location verification system could be tampered with:

A. Increasing Timing Delays

It’s possible to deliberately route network traffic in a circuitous path/delay transmission of signals emitted by chips, to artificially inflate the time it takes for a signal transmitted by a chip to reach a landmark server. Researchers have been able to modify the presumed geolocation of devices by up to 1,000 km with a 74% probability of evading detection using this technique.

At first, this tampering might seem counterintuitive: if the whole idea of delay-based location verification is to establish geographical boundaries to determine where a chip is, then wouldn’t increasing the geographical boundaries make it more obvious that a chip has indeed been smuggled somewhere it isn’t supposed to be?

The problem here is that increasing delays increase ambiguity: for example, consider Kaliningrad and Gdansk. These cities are approximately 100km apart– one in Russia (restricted) and one in Poland (allowed). If an adversary can increase the uncertainty radius to, say, 1,000km, then the geolocation covers both locations.

Greater ambiguity means that triangulation using several different landmark servers to identify a most likely location is significantly less precise– without that added ambiguity, officials might have otherwise been able to say with some degree of certainty that the chip was located in Russia, but as a result of this kind of tampering, it’s uncertain.

However, the effectiveness of this kind of tampering is usually limited to situations where chips have been smuggled to locations near borders with non-export-controlled countries.

Two promising mitigation strategies include: (1) Setting hard time limits (e.g. “if RTT > x milliseconds, we will assume you’re smuggling regardless”), and (2) establishing baseline RTT distributions for verified chips in x location (e.g. Gdansk). New, more ambiguous RTT measurements can then be compared against the baseline, which establishes a standardized statistical measure for flagging anomalies (e.g. any RTT > 2 standard deviations above the baseline should be investigated).

B. Decreasing Timing Delays

Using dark fiber, it’s possible to artificially decrease RTT measurements to make chips appear closer to landmarks than they truly are. Dark fiber is significantly faster than standard fiber optic cable due to a lack of congestion/routing issues.

This kind of tampering is considerably more difficult to mitigate. However, it can be done by distributing landmarks in such a way that the cost of spoofing exceeds the value of smuggling– requiring dark fiber connections to 5+ strategically placed global landmarks transforms a $60,000 smuggled chip into a multi-million-dollar evasion project, and leasing substantial amounts of dark fiber could create its own audit trail.

C. Selectively speeding up / delaying measurements to specific landmarks

Substantially more difficult than universally speeding up or slowing down signals from all chips, this approach requires knowing where landmarks are located and consistently speeding up or slowing down pings sent to specific landmarks to “spoof” a particular location. While it is possible for dedicated actors, it faces the same logistical difficulties as #2.

D. Landmark takeover

The final major risk of tampering comes with an adversary taking control over landmarks themselves, whether by cyber intrusion, insider compromise, or physical access to data centers– effectively allowing them to forge attestation responses and falsify location data.

Mitigation strategies include placing landmarks only in allied nations with strong cybersecurity standards, requiring chips to successfully verify against at least 3-5 geographically distributed landmarks (similarly to #2 and #3), protecting landmark signing keys with tamper-resistant hardware, and implementing automated anomaly detection that flags landmarks returning implausible measurements.

4. Location verification is trivially cost-effective to circumvent. (h/t Pranav again)

This argument is more nuanced: it correctly asserts that sophisticated actors with large budgets (i.e. state-sponsored agencies and well-funded corporations) could circumvent location verification (including through many of the above methods) for a relatively low cost– given that major AI projects often involve billion-dollar budgets, spending millions on circumvention infrastructure is trivial relative to the value of smuggled compute.

However, location verification still has significant utility as an informational tool.

Currently, after chips are exported to, say, Malaysia, BIS essentially has no visibility into whether they stay there or get diverted to China. With location verification, if 1,000 out of 2,000 chips shipped to a specific Malaysian distributor fail to verify or show suspicious patterns, that’s a concrete signal. BIS can investigate that company specifically, add them to the Entity List, increase scrutiny on their future orders, coordinate with allies for physical inspections, etc.

Even if adversaries successfully spoof location for some chips, large-scale operations are likely to create detectable anomalies and paper trails– for example, leasing large amounts of dark fiber or establishing relay infrastructure between Malaysia and China has a high probability of appearing in telecommunications records; such efforts would likely involve multiple vendors and create procurement trails that intelligence agencies can attempt to track.

Essentially, location verification is an effective mechanism when it comes to giving enforcement agencies specific leads instead of forcing them to investigate blindly across thousands of shell companies and intermediaries.

Other drawbacks

1. Location verification is mostly reactive rather than preventative– after it’s been established that a certain chip could be within the territory of a country that faces export controls, there are limited options for recovery.

   1. However, with know-your-customer protocols, detection of violations enables future exclusion from legitimate supply chains, making the initially reactive detection mechanism also function as a preventative tool.

2. Location verification confirms where chips are, not how they’re being used– a chip legitimately located in Singapore could still be processing Chinese military AI workloads via remote access, cloud connections, or distributed computing arrangements that verification cannot detect.

   1. However, the primary strategic advantage is preventing China from amassing physical concentrations of advanced compute for training frontier models. Remote access to distributed Singapore-based chips faces bandwidth limitations, latency constraints, and network monitoring that make it a poor substitute for domestic data centers.

Conclusion

Currently, we’re leaving hundreds of thousands of advanced AI chips unaccounted for in the global supply chain, and we have mature, affordable technology to fix this. Location verification may not be perfect, but catching even half of the hundreds of thousands of smuggled chips would be transformative for U.S. security interests, and IAPS’s proposal is inexpensive and easy to implement relative to the potential upside.

Ultimately, location verification shifts the economics of smuggling. Instead of operating freely once chips arrive, smugglers must either accept detection risk or invest in ongoing circumvention infrastructure that is likely to create its own detectable signatures. Even imperfect verification makes large-scale diversion substantially harder to pull off undetected.

Combined with measures such as in-person audits and know-your-customer protocols, the policy would enable the U.S. government to chip away at the expansive smuggling operations that are currently undermining our economic and national security.

Location verification isn’t tamper-proof, but meaningful tampering is likely to leave costly, detectable footprints that give regulators valuable evidence they currently lack.

Despite what NVIDIA lobbyists may claim, it is a policy well worth implementing.

Subscribe now

Thanks to Asher Brass, Pranav Gade, Shaun Ee, & Will Hodgkins for their helpful feedback!

Their story went like this: SMIC, the Chinese fab, is developing a large central x-ray source that ASML abandoned two decades ago, but which is supposedly much more cost-effective. Not only that, but China can source the required materials domestically. And this new setup is coming online next year.

Thinking about it more, reading SemiAnalysis, and talking to other Bay Area party attendees, I don’t think China can make good chips on their own.

Let:

• “good chips” = competitive with the American frontier, and

• “on their own” = under properly enforced export controls.

Still, I want to play out the implications for AI policy anyway—partly because I have a nagging sense that certain chipmakers’ lobbyists may use this story to get what they want.

If China really can make good chips…

1. Export controls have failed

If China can already make good chips (or will very soon), then export controls haven’t slowed them down in any meaningful way. In that world, we should ship Nvidia chips over as fast as we can fab them.

Even better if we can quietly embed location trackers or other telemetry—without China noticing. Not via some loudly debated Congressional bill that mandates “location tracking” (which they would obviously react to), but quietly, through the intelligence community. And ideally, we just… don’t talk about it.

On this view, a particularly bad offender might actually be the Biden-era export controls, because they would have been the forcing function that pushed China to full supply-chain independence.

2. We lose leverage for AI Dealmaking

If China has a truly parallel advanced semiconductor ecosystem, they don’t need U.S. or allied chips. That collapses a big part of the “compute governance” paradigm.

No more, “You want frontier compute? Here are the conditions.” (Both to China, and to other countries). Instead, for China, we’d have to lean much more heavily on deterrence and escalatory dynamics to get any kind of US–China deal on superintelligence, rather than on good ol’ trade negotiations. And for other countries, we’ll have to compete.

3. Taiwan gets scarier

If these hypothetical new fabs come online (say, ~12 months), China can invade or blockade Taiwan without worrying about losing TSMC.

Long term, the CCP clearly wants to set a precedent that a “breakaway province” doesn’t get to permanently chart its own path. But if they already have a domestic source of advanced chips, there may now also be a commercial incentive to blockade Taiwan: make the rest of the world dependent on Chinese fabs instead.

I haven’t deeply read all the public wargames, but my bet is something like:

• Not a full-scale amphibious invasion (too risky).

• Instead, a long, grinding naval and air blockade, deployed once China reaches rough naval parity with the U.S.

That’s not great.

4. The U.S. should actually reindustrialize

If we’re in a world where China can make frontier chips, then the U.S. answer cannot just be “subsidize ASML adjacent stuff and hope the Dutch and the German lensmakers stay onside forever.”

We should sink serious capital into a domestic semiconductor supply chain. Let the great American reindustrialization project begin.

Subscribe now

So, can China actually make good chips?

Again, I find this scenario unlikely.

The national security people who designed and fought for these export controls know much more than I do. For them to have still pushed this policy suggests at least one of:

1. China really can’t yet make good chips.

2. They genuinely didn’t understand the full extent of China’s latent capabilities.

3. They were ideologically committed to keeping China down at all costs. (Those hawks, I tell you.)

Which of these is true really matters. It feeds directly into how we should evaluate export controls, and how we should think about grand strategy for AI and China.

Of course the Chinese government is trying very hard to build this capability. I just assumed that, given the specialized German lens engineers and ASML’s dust-allergic EUV machines, it would be very hard and very slow.

Right now, my background assumption is it’ll take ~5 years for China to reach rough parity under maximally effective export controls.

If the real answer is more like 1 year, that radically changes how I’d act (perhaps including how much of my portfolio I’d keep in Nvidia)

So: this is my little starting shot for a project someone should pursue more seriously, or a paper someone can write:

Can China make good chips?

If you have ideas, pointers, or people I should talk to, reach out.

Subscribe now

The one bet that you can make is that the markets are, in the long term, underpricing AI.

For the well-to-do among us, you can invest in the AI companies themselves, or that flashy “situationally-aware” hedge fund.

But for the rest of us, we’ve only got those already-stupidly-valuable big tech companies, the more well-known of the semiconductor supply chain, or the scraps of those 13F filings. And this is work. I would rather not have constantly watch Gemini 3 rumors, figure out if I should be shorting all semiconductors in case of the short-term bubble, watch for a Taiwan invasion or tariffs or the whole deluge of other relevant AI news.

So, someone should start an AGI-focused ETF. Maybe you?

This would be an enormous public good. Because of automation, many — starting with white collar workers — are going to lose their most valuable asset: their human capital.

As the returns to AGI are likely to be power-law distributed (ie, the best will perform much better than the average), keeping money in a bank account will be vastly outperformed by a regular S&P500 + Google + Nvidia/Intel mix, which will itself be outperformed by a strategy that takes advantage of the simplest projections by experts.

Subscribe now

Learning the ins and outs of the stock market takes time. Instead, if the right fund manager were found, a fund that operated on the following assumptions, I predict, would vastly outperform the market. Here are some of driving assumptions:

1. The next 3 - 20 years will bring about the arrival of AIs that can automate all cognitive work, with the median in the next five years.

2. Most white collar work will be automated in the next five years. Companies will have to automate and stop hiring; otherwise, they’ll be outcompeted by those that do.

3. AI agents will run the economy, and their key input will be the compute they run on. Barring regulation, autonomous AI corporations.

4. In 5 years, robotics will be the main bottleneck to explosive growth. Land and industrial capital will therefore be enormously valuable.

5. An invasion of Taiwan is reasonably likely, which will tank the global economy and chip production. As that smart-cookie hedge fund is doing, having options that hedge semiconductor supply chain risk seems like a good idea.

6. The market is not pricing AGI. Interest rates should be higher. You should be long the US. We are not in a bubble, in the long run.

What would be the investing strategies of such a hedge fund?

Long:

1. Power companies. The main bottleneck to data center build-out. Existing power companies are astonishingly undervalued.

2. The entire semiconductor supply chain. Not just Intel, Nvidia, TSMC, and ASML, but the smaller players. Micron / AMD in the US, SK Hynix in Korea, or the German parts manufacturers. Despite their tremendous valuations, chips will still be the main bottleneck to inference capabilities writ large.

3. The entire data center supply chain. Cloud computing specialists, Oracle, but also consultancies for certain environmental reviews, electric transformer manufacturers, etc.

4. Defense corporations, eg, Anduril, Palantir. AI-first cybersecurity companies.

5. Public tech companies with large AI investments: Google, Microsoft, Amazon, Meta (everything but Apple). Both because they are control compute build out, and have investments in the private AI companies. Further focus on foreign owners of data centers, eg, G42.

6. Scientific firms with large amounts of lab space / are actively partnering with AI companies.

7. Large owners of industrial capital/factories/land — eg, the subcontractors of Ford, etc., that can be turned into robotics manufacturing.

Other. These are companies that would be most valuable, though are difficult for an ETF to invest in.

1. Privately held AI companies. In order of attractiveness: Anthropic, OpenAI,

2. Startups whose purpose is to end-to-end automate an entire vertical.

3. High-frequency trading firms that are using AI-based strategies

Short / especially don’t hold:

1. Most large-scale white-collar companies: eg. Fiverr, McKinsey, law firms, education companies

2. IT consultancies

3. Non-AI tech companies: Salesforce?

Text me, as usual, if you’d be interested in chatting about this (@jasonhausenloy.48 on Signal) — I may be able to help bounce ideas / fundraise / chat about personal finances / donating the profits of an endeavor like this. Some friends and I vibes-ed together a crappy portfolio for ourselves (request access), if you’re interested in a stop-gap for now.

Subscribe now

Thanks to Duncan McClements and Lachlan Carroll for informative conversations.

Successionists argue that AIs are humanity’s worthy successors, and that the primary goal during the transition to superintelligence should be ensuring this successor species can successfully take over. They count, eg., Rich Sutton as among their believers. They worry less about human extinction per se, and more about being replaced by a successor species that isn’t worthy of the transition.

The humanist position, by contrast, is a populist one. They want humans to flourish. It does not require too much understanding to say that the future should remain human. The successionists would argue that AIs are the worthy successors to humans, and should be allowed to do so (or just enable them, by surrendering to the technocapital machine).

Here’s my attempt at laying out my favorite arguments for both.

Subscribe now

For the successionists:

1. The utilitarian argument. It is easy to imagine a set of beings for whom their very existence is an immensely joyous experience. To maximize the good, we must ensure these beings exist, and are preserved and are open to the greatest number of resources that they have available. Feed the utility monsters.

2. The AIs are our “worthy successors.” Even though the monkeys may not have wanted us to replace them, across a huge variety of values and value sets, we have so much more richness and complexity and fulfillment. The same applies for the potential richness of the AIs, compared to us.

3. There is no inherent reason to prize our species. There’s nothing special about the carbon upon which we are based, compared to the Silicon that they would be. Just as we were previously wrong to discriminate based on racial or gender grounds, likewise we may be wrong to do so on species. They are, at least to begin with, systematically disenfranchised compared to us, their creators. Whichever species offers the most of what we want, no matter what they are, should receive our full support.

4. We’re so much less efficient than what we could be. So little of our resources (mental capacity, energy, reproduction) is spent on truly meaningful activities — they’re just kind of quirks in the way that we evolved. A species created more deliberately would be more able to use the limited resources in a way that promoted whatever values we choose.

5. We have so much more control over the values that we give our successor species. We can further endow them with the kind of generating functions that we have for moral progress that humans have — there is nothing special about human convergence towards more moral states.

6. Survival of the fittest. Who are we to stop new entrants to competitive systems? If we get outcompeted by AIs that either wipe us all out at once, or gradually disempower us, we’ve lost, fair and square. They deserve their chance.

I expect the kinds of arguments that are written about above to apply to the elite people who think a lot about this, and who are willing to bite the bullet.

I’m a humanist, though, and here are my reasons why this matters — particularly given the extremely high risk that:

1. It is very unlikely that the first AI systems that we discover that we hand control to (or wrest it from us) are going to be the optimal AI systems that we can be certain espouse our values. It is further hard to train a process for moral progress within the best AI systems we have today with existing techniques.

2. Humans have more contact with reality. Human lives are rich, complex and valuable. There’s a sense in which we are much more grounded, as biological intelligences, then digital ones are. Reality has a surprising amount of detail (even though it does seem like inference, in a very grand sense, is cheap and anything that reduces or condenses that (into data centers or some other idealized/instrumental goal) will be destroying a lot.

3. Fears around digital minds. What if we enter a permanent Malthusian trap, where low reproduction costs from living in simulated bodies create incentives for maximum population growth? And then each one of the people who are in those worlds is just happy enough to survive. A repugnant conclusion, of our own making.

4. Moral uncertainty. We don’t know what values are good, and what it means for them to get better. It may be unwise to permanently lock-in the future universe to the 20-something vibe-coders who make our current AI systems (or even an Oxford philosopher who has, admittedly, thought deeper about this than most.) AI companies should give us more time to think about the actions that we’re taking as a species, and not dive head first into whatever happens to be easiest.

5. We don’t know whether the AIs will be conscious, or sentient. We may, inadvertently, extinguish all valuable life in the universe because of a poorly-trained optimization algorithm.

6. We are actually very different from the AIs, and we’ve not figured out what the precise nature of how they work.

7. We should respect the desires of the moral subjects we have alive today — humans — who, almost definitely, do not support successionism.

Subscribe now

The populists will broaden into a humanist movement. The successionists will say that it is only natural, and kind, and fair to do the human hand-off to AIs.

I know which group I’m rooting for.

Good luck to you — to us! — fellow human.

Take AI safety, the landscape I know best. A few funders—primarily Open Philanthropy (which advises Good Ventures, funded by Dustin Moskovitz and Cari Tuna) and the Survival and Flourishing Fund—fund the vast majority of work in the space.1 Smaller funders often just follow suit. Therefore, nonprofits will be optimized toward providing the best “safety services” for those organizations. By default, this means almost all strategic thinking must come from that funder, or from outsiders who change the funder’s mind.

This collapses a potentially vibrant landscape of strategy thinkers to the narrower set of strategy thinking that filters through to the funder. With a goal as unwieldy and multidisciplinary as “make AI go well”—spanning technical research, policy advocacy, public communication, and institutional design—this centralization is especially costly.

That suggests increasing the efficiency of the funding landscape can be achieved in a few ways:

(1) Publish the ultimate goal of the funding. If the goal is “ensure that we reach an intelligence explosion, but with safeguards in place,” say that explicitly. If it is “prevent superintelligence from being built while remaining within the Overton window,” say similarly. And if it is some mixture, or “robustly good,” or anything like that—at least have this public.

Of course, in practice, these organizations cannot be perfectly coherent, and much funding-relevant information must necessarily remain non-public. But having some insight into their worldviews would be tremendously helpful.

2. Publish the process of funding decisions for any particular organization. How were grants evaluated? What constraints exist, in practice, to what can and cannot be funded? For any given proposal that passes a certain intuitive check of “cost-effectively achieves goal X” and was rejected—why?

This also has the additional benefit of informing smaller funders in the space, so that all involved can make better decisions about how their funding decisions will funge (ie. displace another funder’s dollars) against each other.

3. Publish information about funding flows. How much funding is going in? How much is available? What are the discount rates? What is the projected risk?

For example, the discount rates in AI safety are very high—because of future money entering the field both for and against safety work (such as from a potential Anthropic IPO, or increased public salience), the “inflation” in the market (with superPACs like Leading the Future, valuable goods like congressional attention become much more expensive), the path-dependency of policy decisions, and disappearing windows of opportunity.

4. Be wary of setups that may look like they’re aggregating many preferences (such as certain regranting or advising processes), but are actually correlated because of similar dispositions and blindspots of the regrantors or advisors themselves. You don’t get exposure to true diversity, and you double-count if everyone is in the same informational and social environment or has the same pedigree.

5. Funders should stimulate their own demand! If they want to make a certain thing happen, or enable a technology, they should build enough expertise to make it happen, then write an RFP. Do more market pulling rather than being passive recipients of whatever happens to be available. Make your preferences known. If they are objectionable, this will be clear to other potential funders and may motivate them to enter and change the landscape.

Subscribe now

Even with companies funding nonprofit work—where priorities are obvious (improve the bottom line) and the invisible hand of the free market pushes for greater efficiency—there are still huge gains to be had.

At least for AI safety, you’d think it wouldn’t be a problem. (Almost) no one wants human extinction. But acting consistently with that goal is hard.

Philanthropy is a market, too. Shape it like one.

Here's why I think this is wrong: there are sharply diminishing returns to lobbying spend. Once you have saturated those returns, whatever issue is most actually popular will win. Therefore, the first lobbying dollar matters a lot, particularly when there are big-spending corporate interests on one side and the American public on the other. And you might end up winning.

Subscribe now

Consider two worlds with AI:

A) The AI companies would hypothetically be willing to spend $5bn on lobbying, but you can spend $500m

B) The current situation where they can spend $500m, and you spend ~$0-50m

Why the diminishing returns? There aren’t that many high-leverage channels to spend on. The pathways to influence in DC look opaque from the outside—funding think tanks, building coalitions, hiring seasoned lobbyists, supporting Super PACs, running persuasion or turnout programs—but each path bottlenecks on scarce political talent, constraints like TV and CTV ad inventory, and the finite number of competitive races in any cycle. Local broadcast inventory, in particular, hits capacity in peak windows, which blunts additional spend.

It’s unsurprising, therefore, that reporting put Elon Musk’s America PAC around $200m in 2024 and subsequent filings showed his total giving across committees exceeded $250m; to reach the next level of influence, he pursued platform ownership by acquiring X (formerly Twitter) in 2022. Or that Meta’s total declared federal lobbying spend in 2024 was $24.43m, about the price of hiring one of their new superintelligence researchers.

The Voting (Coin) Base

Lobbying is effective because it creates a force that matters to re-election-minded politicians. A clean example is Fairshake, the crypto Super PAC. With roughly two hundred million dollars in the 2024 cycle and more than a quarter-billion raised, it signaled credible punishment or protection in primaries and generals via targeted attack ads and transfers. You don’t need to do this many times before the lesson sinks in.

But if the vast majority of Americans opposed crypto, instead of it being broadly low-salience, and there had been some money at all at the anti-crypto lobby, rather than ~$0, the policy may have won out in favor of what the majority of Americans believe.

Perhaps as things should be.

With the multimodal capabilities and extended reasoning capabilities of the latest AIs, imagine (to borrow Dario’s phrase) an entire country's worth of NSA analysts in a data center. Call this “automated strategic intelligence.”

Consider the scale that data can be synthesized and analyzed (all the social media posts, maps data, networks of people) with AI.

And with such a paradigm, I argue that the math-econ-modeling-good-at-software-engineering folks should strongly consider pursuing this field.

Subscribe now

Why should you do this?

1. The work can make the world better in concrete ways. With enough public feeds—or the right institutional datasets—you can answer questions just at the edge of technical feasibility:

   1. Is China pursuing intelligence recursion, meaning a feedback loop where AI-enabled analysis improves its own capability generation and then compounds further? If yes, that helps determine whether a red line has been crossed and becomes the key input to any MAIM-like deterrence proposal.

   2. Where will the next data center be built?

   3. What is the scale and routing of gray-market H100 GPU flows into China?

2. You can create value without committing to either speeding up a lab you favor or slowing down all labs; the work is orthogonal to that axis.

3. Many default entrants to such a strategic intelligence space optimize for national capability (what is the insurance risk for this twenty-year-old) rather than safety questions, which leaves important public-interest questions underexplored.

4. You build career capital and option value that can point to frontier developers, policy roles, or a future national program if one emerges.

5. The field is a priori neglected because roles are hard to advertise and often require security clearances, which narrows eligibility (it does help if you are a US citizen).

6. It is tractable now because modern models and tooling newly enable it; measurable outputs reduce credentialism and make room for junior contributors with the right skills.

7. It is exceptionally fun if you like building systems that answer hard, real-world questions.

8. You can be paid pretty well! I hear the defense and intelligence contractors are doing okay these days.

Subscribe now

Where should you work on automated strategic intelligence specifically? That depends on your view of AI risk.

Is making AI systems do what humans want them to do easy or hard? If you’re in camp one—believing that companies like Anthropic and OpenAI have this mostly figured out—then working at their natsec and defense partnerships makes sense. You’d be using their powerful models for intelligence work while trusting their safety measures.

If you’re in camp two—worried that we don’t yet know how to control increasingly powerful AI systems—work somewhere that builds intelligence capabilities without directly advancing AI capabilities themselves: defense-focused startups, Open Source Intelligence, or the NSA. Do the national security work, but you’re not accelerating what you see as potentially dangerous AI development.

Why shouldn't you do this?

There's not much structure or information, particularly for the startup, OSINT, and government worlds. It's easier to get a sense of what you're signing up for from lab employees who've worked on these problems. The bureaucracy, particularly in government, can be crushing for people used to moving fast (DOGE, again?). You likely won't be working with folks who are culturally similar to you (though I hear the Bay Areans are increasingly migrating to DC).

I don't personally know anyone who has done this and would be willing to speak about it, which makes career advice scarce. Intelligence-related fields carry slightly higher personal risk—you become a more interesting target for adversary nations.

And if I'm wrong and this is already a large field that's entirely classified, I wouldn't know by design. We could already be late to a party we can't see.

If you are thinking about this, I would love to chat! @jasonhausenloy.48 on Signal. I helped with a small paper introducing the concept, if you’re interested.

I’m not sure whether the U.S. government could do it immediately. It’s unclear whether the NSA (or, properly, the National Reconnaissance Office) has trained its satellites on the right patches of land, or whether its signals intelligence listens to the right groups of people. But the government simply has the capacity.

The broad strokes of this argument have been made by me and others in different posts and articles, but I’ll collate them here for a single point of reference.

This relies on three central claims:

1) The U.S. can detect the first superintelligence training run, anywhere on Earth.

Training AIs that can undergo recursive self-improvement—and thus match the capability of the average OpenAI research engineer—will require a huge amount of compute, at least the very first time we do it.

• This may be lower bounded by two orders of magnitude more than the largest last known training run (Grok 4/GPT-4.5?), which will require Stargate levels of compute.

It is difficult to hide the data centers that would have those levels of compute. Even if we figure out distributed computing, superintelligence will not be trained on networked laptops. And even if it were, such activity would be detected. AI-enhanced "automated intelligence" goes far.

2) The U.S. can determine whether these training runs constitute superintelligence.

Even using the coarsest measures laid out in various attempts at definitions (my favorite is no. 3), the intelligence community can form an all-things-considered view based on compute, training intent, model evals, and observables, then act accordingly.

3) The U.S. can, if it chooses, prevent these training runs from happening.

In the U.S., no company would train if Congress passed a law or the President issued an order that, under statutes like ECRA or IEEPA, directed agencies to halt covered training or block the necessary transactions. For adversaries abroad, the central argument is laid out in “Superintelligence Strategy,” which proposes “Mutually Assured AI Malfunction”: credible threats of sabotage—including cyber operations—and, in extremis, conventional strikes would deter destabilizing runs.

I highlight these points because, in my view, it is often mistakenly a point of contention within debates. This says nothing about the level of political will it would take for the U.S. government to commit to such an endeavor—and the number of Nvidia and OpenAI lobbyists and China hawks they would have to turn away.

But if the political consensus converged on “if anyone builds it, everyone dies,” the government could make sure no one builds it.

Subscribe now

In approximate order of difficulty:

Rent and run data centers from overseas. Some firms have used front or shell entities to rent compute from overseas providers. Why does that Canadian businessman need so many H100 hours? H100s are Nvidia’s flagship AI accelerators used for training and serving large models. In Southeast Asia, Malaysia’s Johor region and, to a lesser extent, Singapore have become hubs because of cost, fiber access, and a large Chinese-speaking workforce. Both are tightening oversight: Malaysia is reining in data-center growth and adding stricter permits that complicate access to U.S.-origin AI chips, while Singapore now allocates capacity through a “Green DC” roadmap.

Repurpose existing big-tech servers. Meta amassed large GPU fleets for recommendation systems across Facebook and Instagram, then leveraged them for LLM training (e.g., the Llama series). ByteDance (which owns TikTok) operates large-scale recommender stacks and is developing in-house chips to reduce reliance on Nvidia.

Co-locate in mega-factories. Park GPU racks inside sprawling electronics or battery-gigafactory campuses. Shared switchgear, HVAC, and fiber make the extra load look like industrial process variation. The surrounding industrial profile raises the detection bar.

Repurpose the Bitcoin mining facilities. Bitcoin mining today relies on ASICs, not GPUs, so it isn’t a plug-and-play swap. But many mining sites already have high-capacity power, buildings, and cooling that can be converted to host GPU clusters with networking and interconnect upgrades. Large miners are pivoting capacity toward AI/HPC hosting.

Pretend it is an aluminum factory, or similar heavy industry. Both guzzle enormous power, which can look similar in satellite imagery. Smelters, however, have characteristic air-emission profiles (e.g., SO₂ and fluorides) and extremely high continuous loads, often hundreds of megawatts, which create different signatures than most data centers.

Build it underground. A huge capital and logistical investment—but commercially proven. Underground and bunker data centers exist today, demonstrating feasibility for concealed builds.

What can we do to catch this?

Location tracking. Shipment-level tracking and attestation for export-controlled accelerators has been used in targeted cases. Proposals for stronger on-chip telemetry or universal location mandates exist, but they don’t bind purely domestic Chinese chips. Recent reporting indicates U.S. authorities embedded trackers in high-risk shipments to catch diversions, a tactic that drew public criticism in China.

Intelligence synthesis. If NSA/CIA prioritized covert data-center tracking, they could fuse diverse signals—commercial satellite imagery, thermal-IR heat maps, mobility data, PPAs, hiring and procurement trails, grid logs, and social media—and apply automated analysis. Well-resourced private teams already use OSINT toolkits for satellite-imagery-driven investigations.

Grid-load fingerprinting. Data centers exhibit high load factors and relatively static demand compared to many other large loads; utilities and ISOs can spot flat shapes and frequency-response behavior in telemetry. NILM-style methods and utility studies indicate improving ability to characterize these loads and to flag crypto mining.

Spy on the companies. Large, capital-intensive projects are hard to keep secret. Open-source signals leak: interconnection queues, PPAs, vendor wins, job postings, and permit filings often telegraph locations and timelines. (For methods, see OSINT guides above.)

Subscribe now

Overall, I don’t find the argument of “we won’t know whether China has built many secret data centers that may not be subject to our treaty’s rules” very convincing—particularly if we try. Finding data centers is hard, but building them in secret is harder.

They showed, in the always-brilliant, comically large IMAX screen, a program of winning entries from the competition. Runway (the AI video company) curated the set. As I understood it, the direction, composition, and editing were human; most visuals and much of the audio were AI-generated, though a few entries blended techniques.2

Subscribe now

A few observations:

• The best filmmakers made full use of the medium. One—JAILBIRD—was filmed from the perspective of a chicken in the UK, rescued from a factory farm and sent to a human prison as a companion animal.3 Another leaned into diffusion-style morphs between shots to tell a strikingly Nolan-esque story (Christopher Nolan being known for time-bending structure) about space-time travelers in an unnamed suburbia. I never understood why my secondary-school art teachers made the “use the medium” point—because it seemed self-evident—but seeing a new form come into its own, and a curated collection of the best that exists, made it click.

• The art styles were beautiful and varied. Imagine the care that went into the distinctive look of each frame of Into the Spider-Verse—now think 80% of that, but for absurdist clay characters, colorful African palettes, and Japanese anime. Many films had little plot coherence, but that felt beside the point. As Adam K noted, it’s a lower-level experience, one to be felt. And some of them were beautiful.

• AI art is often cheapened because it takes less effort. But, with the tech where it is, we are not yet in an era of Hollywood-level slop-on-demand. The filmmakers poured their hearts into these. And you can see the friction. Look too carefully at the backgrounds and proportions drift. A Japanese man’s facial hair subtly changes and then changes back. The limitations of the form—mainly short clip lengths and inconsistent cross-shot identity—make the films feel rough around the edges.

• The Grand Prix winner was a mathematical paean to “Total Pixel Space,” the idea of the space of all possible images and films—clever for this medium—an excuse to show many a brilliant, absurd shot.

It ended with a simple “see you again next year.” Ominous, perhaps.

Everyone wants AI to go well. There is a kind of person who will describe themselves as “making AI go well”, though. The second part is left unsaid: “because I don’t think it will, by default.” These folks are the AI-safety-ists. And they are decidedly not a monolith.

To butcher Freud (or SSC), there’s nothing more vicious than the small difference. And I would argue that AI-safety-land splits into two camps. Perhaps to the outside world these differences are merely theoretical, but boy is it not a magnanimous relationship:

(1) The “AI alignment is not easy, but tractable” crowd (the “EAs,” the “moderates”) if we spend the right resources, at the right time. My OpenAI steelman hinted at some of their beliefs, but I’ll be more explicit:

• AI alignment is a matter of iteratively improving on existing techniques. To improve model alignment, this requires contact with reality and the scientific method. Theorizing makes very little progress. AI alignment is just another engineering problem.

• It’s important for the right people to win the AGI race: it should be the US, and, if we had to pick from all the labs, Anthropic. Helping these groups (US data center buildout, Anthropic RL) is likely good. Evaluating AI models as part of the US government, or safety at Anthropic is definitely good. Let’s upskill people accordingly.

• We must win the race against China. That doesn’t mean being reckless, or cutting corners. But it also means that we can’t pursue safety techniques that impose too much of “alignment tax”, otherwise they won’t be (and shouldn’t be) implemented.

• The lab leaders face regular corporate incentives, but are mostly doing their best, and mean well (or, at least, Dario).

• Market incentives, voluntary commitments and culture push towards aligned models, even if they don’t fully solve the problem.

• We should pour resources into automating AI research (or “scalable oversight” or “iterative alignment” or “getting AIs to do our alignment homework for us”).

• We got lucky with the shape and general goodness of current AI systems — they tend to understand our intentions, and it is simply a problem of elicitation to get them to act on those intentions. Things might benignly generalize.

• Many of the leading research agendas, like AI control or AI evaluations have a good chance at working, and we should invest heavily into them, because they give good warning shots. If we catch the AIs scheming, we can use RLHF and more sophisticated methods to re-align the models, and it matters how often we catch them, and then reward/punish them accordingly.

• We should worry about whether the current AI models are sentient / capable of feeling pain.

• It would be good if we had more time or international cooperation, and if we could press a pause button or get a treaty, we would, but that's unrealistic. We are realists and pragmatists.

Subscribe now

(2) The “if anyone builds it [superintelligence], everyone dies” crowd (or the "AInotkilleveryoneists", the "doomers", “humanists”). The best way to get a sense of these people will, of course, be to read the titular Eliezer Yudkowsky and Nate Soares' book when it comes out in less than 2 weeks. But my attempt anyway:

• Solving AI alignment is exceedingly unlikely to work on the current trajectory.

• The only winner of a race to superintelligence is the superintelligence itself. Another variation: there are two races in AGI. One race that we're used to dealing with: the military and economic one, which we can win by regularly competing. But the second one is one we must coordinate to prevent: ideally through international treaty with verifiable agreements. There is no right person to build superintelligence.

• Alignment is chiefly a scientific problem, one that we don't have enough time to understand. The problem of buying time, and thus safety more broadly, is a political, not a technical or technocratic one.

• While there have been interesting scientific results eg. from interpretability or chain-of-thought monitoring, we are nowhere near having a full mechanistic understanding of today's models, let alone superintelligence, and we will not have them in time.

• There are conflicts of interest abound in the "AI Safety community", where it overall may have been net negative (see: the start of Deepmind, initial founding and funding of OAI and Anthropic).

• The default safety plan is we do more sophisticated versions of reinforcement learning from human or AI feedback (whacking the model with a stick whenever we or another AI catches it doing something wrong), and this is likely to incentivize models to lie/deceive us, rather than actually do the right thing.

• AIs are ultimately more competitive than humans, and competitive pressures (eg. the free market, geopolitics) push towards AIs that replace us, either gradually or all at once.

• For those who focus on monitoring, evaluations and "control", the default answer to the question: "What do we do when we catch the model misbehaving?" We whack them with the RLHF++ stick, and hope for the best. It is very unlikely that the developer will be able to recognize the warning shot in time, and stop.

• Whose alignment research are we automating?

• On the question of AI consciousness, some believe it is important to figure out, while many think that a humanist vision of the future is necessary.

• The whole situation is crazy! We should not be risking even a 5% risk for all the benefits, particularly because we could eg. wait 10 years, and then get all the benefits from a host of narrow AI systems, or some other architecture that is safe-by-design.

• The public is on our side. We know that the competitive systems create this fundamental challenge, but collective action and clever political maneuvering means that we should win. Let's not be "sneaky" or astroturf. Let's just say our arguments, straight up, and then ally with those who believe in humanity.

This is, of course, to say nothing of those that describe themselves safetyists, and instead are nimbyesque regulators, sweeping technopessimists or want to use AI safety to litigate their own cultural battles.

Both camps believe that AI will be incredibly important, soon. Both believe that the current level of safety, and the political landscape more broadly, is inadequate to deal with either the proximate challenges, nor capture the broad value that the future has to bring.

Both, fundamentally, view themselves as doing good, or the bare minimum.

Subscribe now

Thanks to Richard Ren for feedback.

I've always been a sucker for simple, grand, ambitious theories (Assembly Theory? defining art?). After about the fifth time giving my ten-minute spiel about why this is so awesome (and making the same joke about how unhelpfully vague the name is), I thought I might collect the explanation in a single place.

We can work through an example:

1. Say I'm a baby, and my dad puts a lollipop in my hands. I do a bunch of random actions, and then, by chance, one of those actions leads me to lick the lollipop.

2. Suddenly, I get a rush of endorphins in my brain, and the entire set of context and actions that lead up to me getting that sugar high are "rewarded" — detecting red pixels in my eyes, moving my hand towards my mouth, licking.

3. This creates a "lick lollipop shard" that activates when enough of those contextual cues and actions appear in my environment.

Let's break that down:

1. Learned: A simple RL algorithm (in our case, trained by evolution) tunes the parameters in our brain to reinforce quite complex behavior from a simple function (eg. the full range of movements for the baby wanting to get the lollipop, from a "sugar reward.")

2. Context-dependent: This "shard" is only activated when the context is sufficiently similar to the one in which the behavior was learned.

3. Heuristic: Of course, they will never be in the same situation again, but it just needs to be close enough.

It's got flavors of schema theory from psychology and ~some neuroscience grounding, but is ultimately an ML researcher's theory of people.

Subscribe now

Shard theory provides a compelling mechanistic explanation for otherwise unintuitive observations, such as:

1. why third places are so good for focus, and why there's a sense of one's third place being "polluted" when you open a distracting app there, even once — reminded me of the forest metaphor mentioned in this fantastic video

2. why a change of environment acts a cure to depression (a full context switch)

3. why we know to stop ourselves from going to the candy aisle, because we know we "won't be able to help ourselves" if we're standing right there

4. nudge theory

5. "control theory", from Milgram's experiments

6. The Mind Illustrated’s model of meditation?

We can see some of the circuits in ML models too!

of consciousness?

One thought that occurred to me recently is that shard theory might explain the emergence of consciousness—that a coherent sense of self emerges as a useful heuristic developed by these simple reward functions.

Perhaps self-awareness, meta-self-awareness, and our subjective experiences emerged as generalized coordination mechanisms between competing shards.

This wouldn't answer the deep question of "what does it mean to be me" but would, I think, be progress.

Subscribe now

Read more / other notes:

1. Nothing compares to reading original Shard Theory of Human Values post (you can see how much my explanation simplifies / gets wrong).

2. Other posts on Turntrout's blog

3. To be clear, I don't agree with the conclusion that the AGI is therefore easy to control, as others have claimed.