OpenAI Developers

🚨 Analysis: "Can AI agents conduct open-ended AI research?" They gave frontier agents six days, $3,000, full virtual sandboxes, web access, and the central research questions from two unpublished NeurIPS 2026 papers. The agents handled many engineering tasks, yet the final papers were rejected. Recursive self-improvement (RSI) never materialized. [ What the study says ] Primary runs used Anthropic Claude Opus 4.8 and OpenClaw. Agents completed the full execution loop with zero human intervention, yet failed at the core research objectives. Original authors acting as expert reviewers rejected both outputs (scores of 1/6 and 2/6). Five recurring failure modes: • Poor judgment (misjudging the bar for publishable research) • Narrow exploration (uncreative responses to design flaws) • Ineffective backtracking (getting stuck on dead ends) • Poor resource and time management (>50% API budget unused; early exits) • Instruction and context drift Self-reviews systematically inflated scores. A robustness check with OpenAI GPT-5.6 Sol Ultra + Codex reproduced every failure mode. Paper: CRUX (arxiv.org/abs/2607.27191) [ What the transformer actually does ] It samples from a conditional distribution frozen in pre-training. A research paper is not a verifiable problem. It requires judgment: weighing evidence, tracking dead hypotheses, recognizing when a branch is wrong, and restarting. Transformers have no evaluation layer. They can only sample "research-shaped" text. Engineering tasks (debug, run experiment, compile) sit in dense regions of the probability landscape with clear feedback. Research does not. The five failure modes are architectural, not temporary scaffolding bugs. [ Exhibits ] • Both AI papers were unambiguously rejected by the original authors • Agents got repeatedly stuck following dead-end hypotheses • Automated AI reviewers surfaced flaws, but agents failed to act on them • Claude Opus 4.8 + OpenClaw and GPT-5.6 Sol Ultra + Codex hit the exact same failure patterns [ Conclusion ] This is not partial success. It is controlled evidence that current frontier agents + coding harnesses cannot conduct open-ended AI research. They execute tasks, but cannot evaluate novelty, significance, or validity. Recursive self-improvement under the present architecture is hype not science. CORRAL (Ríos-García et al., Apr 2026, 25,000+ scientific agent runs) previously exposed the same "agentic science" flaws: evidence ignored in 68% of traces, beliefs never updated in 71%, and refutation-driven revision in only 26%. CRUX extends this to full multi-day research projects with the strongest available models and scaffolds. The failures persist. [ The real problem ] We keep treating next-token samplers wrapped in agent harnesses as if they possess judgment or instantiate true agency. They do not. Sampling ≠ research. Scaffolds do not create an evaluation layer. Stronger base models will not create one until the underlying architecture changes. [ What actually works ] Use AI for well-specified, verifiable engineering loops. Treat every output as a suggestion, not a finding. External human verification remains mandatory. Stop confusing distribution-based generation with reasoning or true agency. ☕️ How many more papers will treat AI as a digital researcher before the industry admits these are not agents and the transformer architecture is the real constraint? - - - Open for commissions on research published in the last 30 days. Need a solid paper analysis or a 1:1 session to tear into the specifics? Become a subscriber. I have slots open for both.