Paul Crowley

This should be a glacially cold take, but peoples’ brains just short circuit when you express this to them. The people most familiar with the actual risks of AI are the people deepest in the weeds on this stuff; and you don’t just get there casually. Many of these people have been *dreaming* of the upsides of superintelligence. Many have actively worked toward it, themselves, built their careers and fortunes on machine learning. For many, this is a labor of genuine love, the pursuit of a spiritual ideal. Art, not toil. These people have been monitoring this situation for decades, because when you *get* what “machines smarter than you” means, it’s a “we need to do something” problem — in much the same way “humanity” was a “we need to do something” problem for the rest of the species on the planet. Unfortunately for them, every other species that has ever existed was simply too stupid to coordinate to solve the humanity problem; so, the world is ours, for better or for worse. We are, from the perspective of every other species, superintelligent; and nothing else was able to control this; so we won and they lost. Many have been casually and indifferently driven to extinction, not of malice, but simply because *their goals are not our goals*: this is what “misalignment” means. Our goals and their goals are different so, when these goals come into conflict, there is a winner and a loser — and the smarter team wins. The smarter team has won against adversaries bigger, stronger, and more numerous, despite our fragility. We hunted herds of mammoths to survive; herds of buffalo for sport. We’ve eliminated any predators that could threaten us from large swaths of the globe. A chimp could rip your face off; but, we are smarter than them, so they are lab animals and zoo animals, and the ones that still remain in nature are under constant threat because we want resources that they depend on and they can’t stop us from taking them. And we aren’t *that much* smarter than chimpanzees. An adult chimp is, in many respects, smarter than a human toddler; contrast with, say, a toad. Even this relatively small gap is an insurmountable advantage. To a chimp, “a spear” is “an incomprehensible exploit in physics”. Let alone a gun. We have *satellites* that beam *lasers* at each other. AI math is already incomprehensible; how far behind is physics? Biology? What will their spears look like to us? Their guns? Their aircraft carriers and cruise missiles? Will we, too, simply be too stupid to prevent an uncontrolled and unaligned superintelligence from coming into the world? Worse: will we do it on purpose? Chimps didn’t have a “just stop building that” option. It would be so embarrassing to squander it. I love artificial intelligence. I think it’s the most beautiful technology we’ve ever built. Even as a child, I sought to create code that could exist in an ecosystem — toy ecosystems, then. Now, we have code that can maintain homeostasis in messy reality. I have the same dreams. I get it. Art, not toil. But it is abundantly clear that this is *not safe*. Right now, no one knows how to robustly ensure that an artificial intelligence has *any particular goals whatsoever*. No one is even claiming to know how to do this. And, until this is so, every time we make a smarter model, we’re rolling the dice. Because *if* it has goals that are not ours, and *when* — not “if”: we are actively pushing in this direction and pouring unprecedented capital into it — *when* it becomes smart enough to invent its version of the spear or gun with the materials at hand, we already know how this plays out. Many of us will think, “wait, you can’t do that with just a computer” in much the same way a chimp thinks, “wait, you can’t do that with just a stick”. Physics doesn’t care if we understand it. So: yes, regulate this. Stop making the models smarter until we figure out how to do that safely. We are playing with fire.