flaw

btw I've started having doubts about near-term transformative AGI. Timelines up to 2035 or 2040 seem plausible (my mainline scenario is ≤2028 still). We have scant theory of human scientific genius and near zero reliable data on its inner operation. «A genius is just a scaled-up error-corrected human… with extra obsession», perhaps. Something something MCTS, neural noise, branching factor, iterative algorithms; there's no space in the genome for occasionally rolling a superior subspecies with very different mechanics and inductive biases; of course. But is even a properly trained LLM a scaled-anything human, or just an error-corrected interpolation of human reflections? What is the scaling law for the latter towards the genius? And our attempts at making genius obsolete with scale and error-correction procedures have been, I believe, floundering since 1950s-70s; we're coasting on applied science and providing opportunities to geniuses which we do discover in a frenetically expanded talent pool. All papers I've seen about LLMs exceeding humans in "creativity", proving complex theorems or whatnot, discovering this compound or that architecture – have been meh upon scrutiny. Scaling test time compute allows to approximate ever better the flawless work of a mediocre mind that is content to terminate itself in rabbitholes. We are not used to mediocre minds working flawlessly, as in flesh all faults tend to be conjoined. But flawless mediocrity is not genius. Since 2014 I've been a believer in artificial genius, and since 2015 all we've been getting in this regard were amazing demos that can't walk on their own legs towards the summit. Will this change by EoY? By EoY 2025? 2028, surely? On fundamentals, I have to keep predicting "yes, very likely so". But my faith wavers. Galkovsky had this metaphor about a rat being chased into a bloodied corner with a multitude of tools (the rat representing himself, a Soviet citizen somehow incompatible with the regime and society around). A normal rodent is crushed swiftly. A talented one fights back, jumps more adroitly, bites at the fingers that hold the prongs, and perhaps even escapes into some lateral hole to survive another day or week. But a genius rat, if there is such a thing – flies away. Breaks the rules of the rat gridworld, violates the whole theory of the game in which the adversary exists. (Later this was, I think, referenced in Pelevin's Hermit and Six-Toe). No doubt rats cannot fly, and certainly humans can't do super-Turing computations; whatever it is that we can do, our machines can do it too. Yet this is no guarantee that we won't need one more revolution before finally building a machine that is peer to the Gods among us – not merely a clearer reflection of their cast-off shells, still hopelessly beneath it. What this means in practical terms is: we might not get all that many technological breakthroughs within a decade or two even if LLMs and their successors supercharge economic growth. Doordash bots but no cryonics. Automated Walmart but not SpaceX or CERN. It's possible that the world is a tougher nut to crack. You'd probably be wise not to act as if the end is nigh. Do your taxes, raise your kids, study your craft, live – and allow that you will die the way all previous generations have, with much the same means to minimize regret.