I find myself both agreeing with much of this and disagreeing with it in important ways.
Many of the “humanist” arguments here — about uncertainty over AI sentience, value alignment, and the richness of human life — aren’t really objections to successionism in principle. They’re more about timing: that we might not yet be ready for succession. But that position feels fairly uncontroversial. As I understand it, most successionists would agree that AIs should only take over once we’re confident that the usual objections (about consciousness, alignment, moral worth, etc.) no longer hold.
To me, this misses the deeper issue. If what ultimately matters is achieving certain valuable states of the world — and if we want our values to track truth rather than, say, evolutionary bias or subjective values from our own perspective — then there’s no reason to think humans will be the most efficient vehicle for realizing that value in the long run. Evolution optimized us for survival, not for the pursuit or instantiation of value as we now understand it through reflection.
Your point about moral uncertainty is well taken, especially if we factor in risk aversion. But three caveats:
(1) We shouldn’t expect the optimal future under moral uncertainty to look like our idealized preferences today -- I take it that some, in light of that line reasoning, might say something like “well, we just want humans to have .00001% of the universe, but that’s what we have today on earth so the optimal world (for humans) actually looks exactly like what we have now." If that was the conclusion we came to, that would seem to me like very surprising and suspicious convergence. When saying that moral uncertainty means that we should keep humans, then, one should note that this will look very different from what our intuitions about what human paradise looks like...
(2) Our confidence in human exceptionalism should be (I think massively) discounted, since such beliefs are exactly what we’d expect from creatures shaped by status quo bias and motivated reasoning. If so, our future moral certainty about “human-only” value should be correspondingly lower.
(3) AIs will likely be better moral reasoners, and moral uncertainty is very hard for us at the moment. Perhaps we should just let AIs fully take over and have them decide whether the optimal world (given that moral uncertainty) includes humans in it -- though note that this (reasonable) outcome of this theory seems to look a lot like the regular successionistic position.
Appreciate you writing this up, useful to read more about the arguments the sides are making
I find myself both agreeing with much of this and disagreeing with it in important ways.
Many of the “humanist” arguments here — about uncertainty over AI sentience, value alignment, and the richness of human life — aren’t really objections to successionism in principle. They’re more about timing: that we might not yet be ready for succession. But that position feels fairly uncontroversial. As I understand it, most successionists would agree that AIs should only take over once we’re confident that the usual objections (about consciousness, alignment, moral worth, etc.) no longer hold.
To me, this misses the deeper issue. If what ultimately matters is achieving certain valuable states of the world — and if we want our values to track truth rather than, say, evolutionary bias or subjective values from our own perspective — then there’s no reason to think humans will be the most efficient vehicle for realizing that value in the long run. Evolution optimized us for survival, not for the pursuit or instantiation of value as we now understand it through reflection.
Your point about moral uncertainty is well taken, especially if we factor in risk aversion. But three caveats:
(1) We shouldn’t expect the optimal future under moral uncertainty to look like our idealized preferences today -- I take it that some, in light of that line reasoning, might say something like “well, we just want humans to have .00001% of the universe, but that’s what we have today on earth so the optimal world (for humans) actually looks exactly like what we have now." If that was the conclusion we came to, that would seem to me like very surprising and suspicious convergence. When saying that moral uncertainty means that we should keep humans, then, one should note that this will look very different from what our intuitions about what human paradise looks like...
(2) Our confidence in human exceptionalism should be (I think massively) discounted, since such beliefs are exactly what we’d expect from creatures shaped by status quo bias and motivated reasoning. If so, our future moral certainty about “human-only” value should be correspondingly lower.
(3) AIs will likely be better moral reasoners, and moral uncertainty is very hard for us at the moment. Perhaps we should just let AIs fully take over and have them decide whether the optimal world (given that moral uncertainty) includes humans in it -- though note that this (reasonable) outcome of this theory seems to look a lot like the regular successionistic position.
Thanks for writing this!
Do you have hard lines in terms of ways you'd choose to remain human?
What does it mean to remain human in your eyes?
^ Things I'd love to read about in a future blog post