Daniel Litt

One of the biggest misconceptions people have about intelligence is seeing it as some kind of unbounded scalar stat, like height. "Future AI will have 10,000 IQ", that sort of thing. Intelligence is a conversion ratio, with an optimality bound. Increasing intelligence is not so much like "making the tower taller", it's more like "making the ball rounder". At some point it's already pretty damn spherical and any improvement is marginal. Now of course smart humans aren't quite at the optimal bound yet on an individual level, and machines will have many advantages besides intelligence -- mostly the removal of biological bottlenecks: greater processing speed, unlimited working memory, unlimited memory with perfect recall... but these are mostly things humans can also access through externalized cognitive tools.

My biggest reason for skepticism about any remaining hard barriers is the intellectual history of ML itself leading to this moment, history which fractally resembles its output. Recall Rosenblatt? Hinton, Sutskever, Kingma, Vaswani? Mordvintsev's DeepDream, my personal "aha" moment? He's ResNet lol? What in all of that was a great paradigmatic leap? The King's road was just ironing out the logistics for connectionism, It's been continuous relaxation, iterative problem-solving, with some natural science analogies and combinations of tools of applied math available to above-average erudite scientists, and indeed pruning of too clever by half branches; and spectacular crashes of programs of those luminaries who grew too intellectually ambitious, like LeCun. Hinton himself assumed that his invention is a dead end facsimile, half-clever research prototype that can't measure up to the brain's actual algorithm; and his confidence in that was shattered by a joke from a model we'd only laugh at in pity today. At no point did we stall in need of some profoundly new mafs or a breaking conceptual insight – and yet we're here, where networks have learned to competently operate with human concepts, with the entire repertoire of reasoning motifs available to a frozen human mind (no I do not agree that the best models are wholly incapable of abduction, which you brought up before; they are grasping at it in long sessions, they try to condense their representations into compact hypotheses and come up with research arms to test them). Something is missing, partially clearly due to lack of demand, partially for reasons I don't understand. Indeed, I think the best public models can't even write this post unless provided with massive authored input, nor your rebuttals. But is that necessary? They are fast and cheap. From now on they will do hundreds of times as much qualitatively comparable iterative work as had been done to create them, as much as we can afford to pay. And incredibly, obscenely much had been paid upfront already. Where does this lead? Naive induction provides the answer. They will gradually do better and better ML research, developing better and better models. That is what is called "RSI". Its exact exponent is subject to debate and market incentives. When continual learning is adequately solved (with actual parametric learning on small batches), whether under human guidance or not, it will accelerate. I think it will be solved. I can abstractly imagine being wrong, I've written on that xitter.twitterapi.xyz/teortaxesTex/status/18…, but do not actually believe that we have any solid reason to believe this ride stops or meaningfully slows down. That would be hubris, and unearned hubris in my case, and I – perhaps narcissistically – believe it'd still be hubris on part of those who have earned the right to it. The journey from perceptron to Astra was longer than from Astra to some Scholze. Humans differ a lot but they are not qualitatively different species, for all that mathematicians like to imagine themselves as such. If LLMs stop after conquering the level of mathematicians' parents, that'd be a big anthropological discovery before it is a finding about LLMs, and we're not exactly doing anthropology here, I think.