Your Company Slack Is Not A Substitute For The Discourse
Thoughts on the brain drain into AI labs
It’s like the AI companies are targeting my favorite thinkers. Can you imagine the Anthropic slack right now? Like come on: Chad Jones, Joe Carlsmith, Andrej Karpathy, Santi Ruiz, Anton Korinek, Ben Kuhn, Stuart Ritchie. And that’s just Anthropic. If you include the other labs: Dean Ball, Alex Imas, Alexey Guzey, Luke Drago, Rudolf Laine. That’s my favorite economist, host of one of my favorite podcasts, half my favorite bloggers. (At least I’ll always have Big Yud.)
This is no coincidence, of course. Many of these guys are both at the top of their fields and AGI-pilled. It doesn’t help that, as my friend Adam jokes, for much of civil society and academia, the base salary from joining an AI lab is the liquidity event. Some research, like qualitative surveys of 81,000 people or tracking automation in real-time, can only be done at labs. And talent begets talent.
Brain drain into industry is nothing new. I shudder at the thought of how many potentially great philosophers, or economists, or mathematicians we’ve lost to the trappings of quantitative finance and management consulting.
Yet this time, as humanity tries to navigate the transition to smarter-than-human AI, our best voices might be walled behind NDAs.
I wish it weren’t so. AI companies face the same epistemic problems as those deep in the national security state -- it’s hard to rely on outside counsel because they just don’t know the same information. We’ll see only the results of these internal debates, stripped of their messy contradictions. My guess is that OpenAI’s flip-flop on endorsing the state bill came from internal pressure, but we’ll never really know. As with government, overton window shifts create private pressure within labs. When the culture makes saying / doing a thing acceptable or even expected, policies can change.
Perhaps this is how people in developing countries feel with “brain drain.” I expect this trend to worsen because of network effects -- our best and brightest want to be with the best and brightest.
This carries a number of pernicious effects:
Closed groups breed groupthink. If you were worried about the echo-chamber of the algorithms, try a Slack of a company with culture interviews. Sufficiently detached from reality, from the reactions and feedback of everyday people, it’s easy for memes and assumptions to spread and go unquestioned. While your slack may resemble a lively town square, full of disagreement, companies are ultimately dictatorships (or, in Anthropic’s case, an oligarchy).
Internal discussion is reactive. It may feel like there’s much discussion, but my experience most internal discussion is reactive (responses to an essay draft, a new geopolitical situation), and the people who pose the questions are few and far between. In any given organization, at most 5% of folks are thinking seriously ecosystem-wide strategically. Planning for future company strategy doesn’t tend to happen within
On that note, discourse is required for democracy. Only a tiny group of humans will have a hand in shaping the intelligence explosion, a decision that is being imposed upon the rest of humanity. A few hundred people, mostly within a few miles of each other, all holding equity, are running a very large experiment. AI’s impact will stretch far beyond any single company. As a matter of course, the discussions that shape that impact should happen in public.
External outputs are filtered. Anything that escapes the company slack will have to go through legal, comms, and align with the broad strategic vision of the company. Blog posts also read more as up-from-above proclamations of company position to the broader masses, rather than discussions.
We’re left with the ideologues and leftovers. If we see an evaporative cooling of talent to the AI labs, the only people who will remain independent are those who are ideologically so, and those that the labs don’t want.
I don’t delude myself to say that Twitter / X is a thriving democratic town hall, or that all discussions must happen with maximum transparency.
Independent public-intellectual labor is a positive externality and a public good. Labs internalize it as private returns (great models, internal memos), and the commons loses out.
So what should we do instead?
What funders can do. Keep our best public intellectuals independent. Direct some of the alleged wealth coming to make AI go better should be distributed to keep our best public intellectuals independent. The MacArthur “Genius” grants are a good model. I’m glad the non-disparagement clauses no longer exist, but fund all the people who leave the labs, too. Build a better town hall: these were, at various points, email newsletters, HackerNews, Lesswrong, Twitter. Perhaps Substack now?
What labs can do. Let teams publish more often in their own, lower-stakes channels. I like the rise of “alignment science blogs” or “interp blogs” and I hope that that can continue, and expand, for other teams. Maybe, even, publish dissent. The Fed releases redacted minutes showing internal disagreement -- labs publishing sanitized records behind a position shift would be great!
What you can do. If you’re an academic considering working at an AI company, follow Dean Ball’s example for a written publish-without-pre-clearance clause. While inside, lobby for transparency. And, if you’re inside already, and feel underpowered or silenced in your current position. Look at the few examples of the reverse trend. The crop of former OpenAI safety folks have contributed tremendously to the discourse. Miles Brundage and Steven Adler, both of OpenAI, have some of the best policy substacks out there. And, of course, there’s Daniel Kokotajlo’s AI 2027.
Our best minds leaving the public square for AI companies makes sense for each of them individually. But, collectively, it leaves the rest of us worse off.