Miss Distance retweeted
People outside SF should know that a small group of people with real influence in AI are trying to *intentionally disempower humanity*. Glad Ruxandra wrote it up.
What are the visions of the post-AGI future in San Francisco? Many think we're all going to die, but even the "positive" visions seem rather dystopian. Here, I wrote about "handoverism," which involves handing control of the future to an AGI God, with humans outside the loop.
ruxandrabio.substack.com/p/h…
We are on the cusp of RSI or recursive self-improvement: this means that AI systems can self-improve, without the need for human intervention. This means our world stands to be transformed in what many call “the singularity”. I moved to SF around three weeks ago and one of the things I have been trying to understand is where we are heading, at least according to those developing the tech.
The worst scenarios are plainly bleak: all of us, or vast numbers of us, die. But what is striking to me is how little reassurance even some of the happiest scenarios offer. But one common and ostensibly optimistic vision is known as “handoverism”: humanity cedes control of its future to a godlike AI and we become the comfortably maintained inhabitants of a civilisation whose direction is no longer ours to decide. It is difficult to see what democracy would mean in such a world, or where the consent of most of humanity to the instantiation of such a regime enters the picture.
The permanent displacement of human labour is treated almost as a given. At the same time, there is scarcely any satisfying answer to the question of how we might remain useful to one another. Even our relationships seem uncertain. The promised utopia looks to me much closer to a kind of dystopia.
“It’s not my preferred outcome,” someone working at one of the big labs told me, “but once we achieve superintelligence, which we cannot stop at this point, I cannot see any other option for where humanity will go.” Then, after a pause, he added: “It is still much better than what I actually fear most, which is that we might all die.”
He sounded resigned, as if the question of what humans would be for (apparently not much?) had already been settled somewhere else and by someone else. All that remains, it seems, is to wait for this pre-destined fate to unfold before our eyes.
But I do not think that is the case. I believe we always have a choice and that we should try and steer this technology in a way that maintains our agency and control. And even if that looks hard and daunting, the immensity of the stakes here make it worth trying at least.
I do not know whether the handoverist vision will come to pass. That the people building this technology consider it plausible, and often desirable, guarantees nothing: proximity to a technology does not correlate that well with predicting its impact on society.
But they are the ones closest to it, and what they say deserves to be at least taken seriously. The rest of society, including some very intelligent people, has been far too quick to wave these scenarios away and not even get involved in the debate. Whether or not the belief is correct, it is sincerely held, and it is not “marketing hype.” I am writing this because I think it is high time that more people, inside San Francisco and out, engaged with the arguments about what our future will look like, and answered them.
Meltdown (1994) is basically the ur-text for all these AI [year] / Commanding Heights / Situational Awareness / etc wishcasted essays; the horror-thrill of the human project crushed and terminated under the silicon die of ahuman technocapital; written as historical inevitability.
1
59
that's why i have boiling contempt for acceptance of inevitability; because it's usually downstream of exposure of this sort of demoralization shock content. people in ai industry of all people should SHUT UP AND FIGHT and not goon to thrice-reheated Nick Land wishcasting slop
1
2
78
this poast expresses it better than I ever could
I strongly dislike this line of thought, and it's in bad taste. Just the posturing, not the forecasting itself, which may be logically reasonable. IMO just too much "xyz sad + bad inevitable thing will happen and we have no control over it, well here's how you can get rich from it tho lol". This applies to a lot of posts high on their own thonkism.
- A huge share of talk (not just this post) prophesies in absolute terms and assumes we have no agency over how things play out.
- "the line must go up", "it is unavoidable", etc. No, it is not.
- The tone itself is unnecessarily triumphant: "Rejoice," "there is deliverance for those who dare." Brother in thonkism, deliverance for whom? Why the hell would you say "the benefits of abundant intelligence will accrue FAR outside SF"?
- They describe the K-shape, the inequality, the displacement. Okay FINE. Do they show any remorse, any contempt for that future? Barely. A few flickers only.
- ending is also hellishly contemptible. hey does this sound too terrible? they say, well remember that our ancestors would call our present apocalyptic too. Is that it? Is that what you will have to say to Ravi?
- they may say “Oh hey, we’re just analysts, we don’t want to take a moral position on all of this.” Then why write so indulgently and relishingly? “Rejoice,” “deliverance for those who dare,” and so on.
It's grim, smug, almost indulgent awe at a dystopia. You may think that all of it is logically inevitable, but PLEASE, FOR GOD'S SAKE, don't adopt a language of helplessness and pretend we have no agency, or one that normalises/glorifies these dystopian forecasts.
1
2
67
and more specifically; there's this entire genre of document that dresses itself up as resigned acceptance hard-eyed forecasting but effectively functions as demoralization shock porn
1
1
65
loss of control to "aligned ai" is a form of successionism and is loathsome and incompatible with any notion (and particularly the Anglo-American notions) of self-rule
1
28
retroactively vindicated in my decision to drive to Pasadena to get my brows done less than a week after getting 20 of my ribs fractured
A classic study by MIT researchers (Sadr et al., 2003) found that the eyebrows are fundamental for face recognition, even more so than the eyes themselves.
What this points to is that changing your eyebrow aesthetics can completely transform your facial appearance.
10
482
Miss Distance retweeted
One of the many, many fucked up things about this is a very implicit view of the deliberate destruction of nations. They are not saying "let's let san fran be ruled by an AI and see if it goes well, then we let it rule California, then we let it rule the US and stop there". No, no, no. They want their AI to go from "being super intelligent" to "ruling humanity" without so much as pausing at national borders.
I wrote about "handoverism": a vision of permanent disempowerment of humanity, as we shall all be ruled by a (hopefully benevolent) AI God that is surprisingly common.
Even among those who want humanity to survive (which to be clear, is the majority), and among some who devote their working lives to this mission, I have encountered an alarmingly common vision of success that does not sound quite like success.
It is sometimes called “handoverism”: the gist here is that humanity survives, but entrusts the direction of its future entirely to AI. Perhaps its clearest articulation comes from a recent essay by Scott Alexander, the most prominent intellectual in Silicon Valley, who has influenced the thinking of many a tech CEO: “If we build a godlike AI, and determine to our own satisfaction that is fully aligned, I’d rather let it choose its own direction for itself and for humanity [...] So why not just prompt it with ‘please arrange the world as you see fit?’”
The premise is that, once sufficiently reassured of an AI’s benevolence (though it is unclear how we could ever determine that?), we should basically give it control of the workings of society. AI 2040, a proposal written by some of the most prominent members of the AI safety community, offers the most concrete account of handoverism. In this vision, responsibility gradually passes to aligned machines until humans can no longer have influence. As its authors put it: “There is no single moment where humanity relinquishes control. But in theory, there is a point of no return.” Others I have spoken to envisage a much swifter transfer.
This is not an organised movement. I would estimate that something like 40 per cent of the AI safety community outside the labs holds some variation of this view, with a higher proportion of those working inside the labs. Nor is handoverism a coherent doctrine with a clearly articulated political philosophy: it’s more of an aggregation of ideas, a makeshift philosophy so to speak. This is perhaps what one would expect given that humans outside the SF bubble have not really had time to react and come up with good and coherent ideas about what society should look like in the face of the extraordinary pace of AI progress.
The motivations behind handoverism are also extremely varied. Some outright welcome the prospect of a superior intelligence governing our future and consider it an actual positive vision. If I read Alexander correctly, he seems to be in that camp. The enthusiastic handoverist often starts from a very bleak view of what we already have achieved as a civilization. In an earlier, 2014 essay called “Meditations on Moloch,” Alexander argues that competition steadily grinds away what the competitors care about, creating spirals of misaligned incentives. Its ending suggests that the only way to stop this is to instantiate a sort of absolute rule by a benevolent superintelligence.
The second camp are the resigned: they do not necessarily want it, but think it cannot be stopped if we get superintelligence, or prefer it to outright extinction. In my conversations, this resignation has seemed particularly common among people in the AI safety community outside the labs. Inevitability is one very common theme in SF, where many see themselves as mere tools in the unrelenting march of technocapital, a march that can only go in one direction: towards superintelligence. It’s often seen as cool and based to see the starkness of reality for what it is and face it head-on, it is wimpy and loser-coded to try to fight for what many consider the good outcome (which is that humans retain some control and agency.)
Like many loosely defined ideologies, handoverism sits alongside ideas that share its intellectual roots without amounting to quite the same thing. One is the vision of humans as cherished pets, or “labradors,” cared for by intelligences. Yet another related assumption is that human labour will be permanently displaced. This would leave us without an obvious economic role and even perhaps no leverage over our own future or political rights.
To understand how my on-the-ground experience matches with what the lab leaders have said publicly, I asked my “agents” to survey all public statements from the founders of the main labs that are still relevant to the direction of their labs (Mustafa Suleyman is no longer at DeepMind, but is head of Microsoft AI.) You can browse all the findings here.
Some openly support or at least accept outright handoverism. Anthropic’s Tom Brown appears to be most explicitly reconciled to the prospect and he actually uses the term “handoff” in a 2025 interview at Lightcone: “There’s going to be a handoff where humanity will hand off control to transformative AI at some point. And hopefully they’ll be aligned with us and that’ll be a good transition that goes well.”