Search: #Options

What are the visions of the post-AGI future in San Francisco? Many think we're all going to die, but even the "positive" visions seem rather dystopian. Here, I wrote about "handoverism," which involves handing control of the future to an AGI God, with humans outside the loop. ruxandrabio.substack.com/p/h… We are on the cusp of RSI or recursive self-improvement: this means that AI systems can self-improve, without the need for human intervention. This means our world stands to be transformed in what many call “the singularity”. I moved to SF around three weeks ago and one of the things I have been trying to understand is where we are heading, at least according to those developing the tech. The worst scenarios are plainly bleak: all of us, or vast numbers of us, die. But what is striking to me is how little reassurance even some of the happiest scenarios offer. But one common and ostensibly optimistic vision is known as “handoverism”: humanity cedes control of its future to a godlike AI and we become the comfortably maintained inhabitants of a civilisation whose direction is no longer ours to decide. It is difficult to see what democracy would mean in such a world, or where the consent of most of humanity to the instantiation of such a regime enters the picture. The permanent displacement of human labour is treated almost as a given. At the same time, there is scarcely any satisfying answer to the question of how we might remain useful to one another. Even our relationships seem uncertain. The promised utopia looks to me much closer to a kind of dystopia. “It’s not my preferred outcome,” someone working at one of the big labs told me, “but once we achieve superintelligence, which we cannot stop at this point, I cannot see any other option for where humanity will go.” Then, after a pause, he added: “It is still much better than what I actually fear most, which is that we might all die.” He sounded resigned, as if the question of what humans would be for (apparently not much?) had already been settled somewhere else and by someone else. All that remains, it seems, is to wait for this pre-destined fate to unfold before our eyes. But I do not think that is the case. I believe we always have a choice and that we should try and steer this technology in a way that maintains our agency and control. And even if that looks hard and daunting, the immensity of the stakes here make it worth trying at least. I do not know whether the handoverist vision will come to pass. That the people building this technology consider it plausible, and often desirable, guarantees nothing: proximity to a technology does not correlate that well with predicting its impact on society. But they are the ones closest to it, and what they say deserves to be at least taken seriously. The rest of society, including some very intelligent people, has been far too quick to wave these scenarios away and not even get involved in the debate. Whether or not the belief is correct, it is sincerely held, and it is not “marketing hype.” I am writing this because I think it is high time that more people, inside San Francisco and out, engaged with the arguments about what our future will look like, and answered them.