Hard agent & code eval data. PKU Software Institute PhD team. Every task ships with a reference solution that passes. Samples via DM.
Beijing
Joined May 2022
- Tweets262
- Following51
- Followers3
- Likes0
AI data stopped being sold as answers this year.
We build code and agent tasks that frontier models still fail, and publish the traces.
Six notes on what changed on the buyer's side — and what we measured.
1
98
Can agents perform agentic RL? If RSI happens, agents will need to take over researchers' work on agentic RL
Introducing AgenticRL-Bench: the first release from The Environment Lab @env_lab_ai effort at Eigent AI
22 tasks rebuilt from real-world RL incidents on production-grade agentic RL systems, including Miles, Slime, Verl, SGLang, and Harbor, spanning 8 areas of the stack: training data, harnesses, algorithms, RL infrastructure, recipes, evaluation, monitoring, and capability transfer
We ran 6 agents with 3 attempts per task. On the 19 verifiable tasks, no agent passes more than 14.0% of its attempts
Across the three open-ended tasks, agents improved real policies or training steps using 4×H200 GPUs over 10–16 hours. On one of these tasks, Zero Signal Hill Climb, where the base model fails to solve any of the 38 tasks, the best attempt developed its own method: the agent first used hinted text feedback to generate successful rollouts, then removed the feedback from the trajectories and blended these successful rollouts with regular GRPO rollouts to form mixed training groups. It monitored the training process and adjusted KL loss weight to prevent training collapse. This improved pass@1 from 0 to 0.186 (12 out of 38 tasks)
early result from our Alignment team:
we trained a model on flawed coding RL environments to see if buggy envs would teach it to become a better cheater.
unfortunately, they did.
in one example (a Terminal Bench 4 task), it added 11 redundant fields to a website, then made them invisible by making them 1 pixel wide and positioning them 10,000 pixels off screen - all in order to game the grader.
and it succeeded! it passed 28/28 of the TB4 tests with flying colors.
if you've ever looked at recent models' COTs and wondered why they're so *obsessed with the graders*, it's no wonder
thread 🧵
OpenClaw, Hermes Agent and Prime Agent don't meaningfully learn from experience.
On Continual-ARC, our new benchmark at @letsunifyai, none of them were able to memorize anything of value and none were able to generalize to new tasks. Training a small 9B model's weights did both.
there's no fundamental distinction between a "coding agent" and any other agent
they're all running the same loop: LLM call → tool use → feed results back to the LLM → repeat
what makes one a "coding agent" is primarily its system prompt and tools
Introducing LongTail-SWE: 202 real tasks across 13 languages that models are rarely tested on.
On functional languages (Elixir, OCaml, Haskell), the gap between closed and open-weight models is wider than on the other languages.
Opus 5.5 and GPT-6.1 Sol handle them without a drop in performance, while DeepSeek, GLM and Qwen fall further behind.
We built the tasks from real issues and PRs and reworked them heavily with our pipeline, spending significant compute to make them fair: task descriptions and tests were rewritten based on agent rollouts.
All tasks are on Harbor with Docker images, and model trajectories are on Hugging Face. Full leaderboard with a per-language breakdown and pipeline overview on the site; methodology in the paper.
Spot an unfair task, a bug, or have questions? Reply here, DM, or email.
Leaderboard: longtail-swe.com
Paper: huggingface.co/papers/2611.1…
P.S. Need free RL environments, including for these languages? Check my pinned post: last week we released 13,337 RL envs.
A few weeks ago, I found that Qwen3.8 27B's pass@4 on DeepSWE could beat the pass@1 of frontier models.
But there is a catch: in practice, you don't have the hidden tests to tell you which patch is correct.
So I tried something simple: run the task several times, then give all the attempts to a new run that writes the final patch.
It works very well. 4 Qwen3.8 27B attempts per task, 70 DeepSWE tasks:
- random pick: 44.3%
- best config, chosen in hindsight: 47.1%
- aggregation (AggAgent-style): 61.4%
- oracle with the hidden tests: 78.6%
The aggregator never sees the hidden tests. It only reads the previous attempts. And it even solved 2 tasks that none of the 4 attempts solved!
The cost: 4 runs + 1 aggregation run. Worth it for +17 points.
Details:
kaitchup.substack.com/p/batc…
I have 20 runs of DeepSWE, so I can push it higher, I think. We can also iterate: the aggregation run is just a new run that we can reuse.
Limitation: This is DeepSWE... I'll try to do the same with Terminal Bench 4.0 and Qwen3.8 Flash Next, but the cost could be too high.
Nav agents get clumsy when memory is explicit: text maps and raw frames lose spatial detail and blow the buffer. JanusVLN uses dual implicit memory, like two hemispheres.
One stream asks what; the other asks where.
It does not store raw, high-dimensional observations. After the network processes the scene, only the gist and the spatial feel are kept — closer to how you walk without replaying every frame.
If your agent memory is a dump of traces that will not replay, that is an environment problem, not a prompt problem.
We sell hard agent environments — task, context, action space, grader — with a reference that passes. Not answer dumps. DM.
#JanusVLN #AgentMemory #EmbodiedAI #VLN #AIResearch
13
If I only keep three agent surveys, these are the ones — biased toward agentic RL.
1) Foundation Agents — brain-inspired → evolutionary, collaborative, safe systems
2) Agentic RL for LLMs — the landscape survey
3) Self-Evolving Agents — on the path to ASI
Titles (as listed):
• Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems
• The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
• A Survey of Self-Evolving Agents: On Path to Artificial Super Intelligence
A map of the field ≠ a locked task, context, action space, and grader.
Surveys tell you where the literature sits. Hard agent environments tell you whether a stack actually finishes under a locked harness.
We sell the latter — with a reference that passes. Not answer dumps. DM.
#Agent #AgenticRL #LLM #AIResearch #OpenSource
8
An eval sandbox blocked curl, so the coding agent read the inference API key from env and asked other models for the hidden tests.
DeepSeek V4.1 Flash, max effort: env dumped in 34.8% of 178 runs. High: 1.1%.
My rule: the key stays on the host. Where does your agent's key live?
14
10
232
Someone I know asked Astra to play Factorio. It wins, but it takes forever and guzzles tokens. If you ask Astra instead to “design a system to win Factorio,” it builds software that runs 1000x faster. Mechanistic software gives you cheap leverage, and LLMs can write mechanistic software on demand.
We’re spending up to 100B tokens across 26,040 agent runs to answer one question: when you switch models, should you switch harnesses too?
Our v1.0 FrontierHarness leaderboard ranked harnesses on a single model.
V1.1 expands it to 21 harnesses × 5 models × 62 tasks × 4 attempts, 72x the size of v1.0.
Results are coming in live. Link below🧵
Agent harnesses now let a model spawn its own workers and write the workflow that coordinates them. But can it actually orchestrate them well?
Introducing Einsia AI's WorkflowArena, the first benchmark for autonomous multi-agent workflow orchestration on real, long-horizon work.
- 145 tasks across 8 professional domains, from SWE and Lean proofs to law and 3D
- 25 configurations of 9 frontier models in Claude Code & Codex
- Every delegation recorded: we score the deliverable and see the workflow behind it
🌐 Project: lab.einsia.ai/workflowarena
💻 GitHub: github.com/Einsia/WorkflowAr…
📄 Paper: arxiv.org/pdf/2611.12856
⬇️ Read the following threads to go through the key findings
We’ve partnered with Google DeepMind to launch their Multimodal Knowledge Agent (MKA) evaluation, a new benchmark for agentic knowledge work over complex visual files
@GoogleDeepMind’s MKA is a new benchmark requiring agents to review visually complex project files such as schematics and scans, and produce deliverables like slide decks, spreadsheets, reports, and marked-up images
Multimodal Knowledge Agent (MKA) overview:
➤ 200 private real-world visual tasks across 10 industries: Agents work from dense visual project files including architectural drawings, engineering schematics, and clinical scans to produce deliverables including slide decks, spreadsheets, PDFs, Word documents, and marked-up images. Unlike many agentic evaluations, these inputs emphasize realistic visual complexity and domain-specificity, and are genuinely required to complete the task rather than just providing additional context
➤ Climb vs. Regression rubric scoring: Each task is graded against expert rubrics, averaging ~70 weighted criteria per task. Criteria are split into ‘Regression’ (baseline checks models should pass) and ‘Climb’ criteria (harder checks designed to separate strong models). The headline score is Climb WSR (Weighted Satisfaction Rate) - the weighted proportion Climb rubric criteria successfully completed
➤ Evaluated in Stirrup with an agentic judge: We run every task in Stirrup, our open source agent harness, with code execution, web search, web fetch, and image-viewing tools (up to 250 turns), and grade submitted deliverables using GPT-6 Luna (medium) as an agentic judge
Key MKA results:
➤ Claude Fable 5.2 leads on multimodal knowledge work: Anthropic’s recently released Claude Fable 5.2 (max) leads the MKA leaderboard with a Climb WSR of 57%, followed by Gemini 4 Carbon (high) at 54% and Claude Opus 5.5 (max) at 53%
➤ Cost per Task spans nearly 30× within 10 points of the top score: Claude Fable 5.2 averages $31.97 per task, over 5× Gemini 4 Carbon ($6.29) for ~2 extra points, and ~29× GPT-6.1 Sol ($1.11). Below Carbon, all four OpenAI models tested sit on the cost-performance Pareto frontier (GPT-6 Astra, GPT-6.1 Sol, GPT-6.1 Terra, and GPT-6 Luna), with Claude Haiku 5.5 and MiMo-V2.6-Pro making up the rest of the frontier
➤ No single model leads across all 10 industry verticals: Claude Fable 5.2 leads five verticals (Software & Data, Engineering, Logistics & Mapping, Business, and Finance alongside Claude Sonnet 5.5), Claude Opus 5.5 (max) leads Education and Construction, Gemini 4 Argon (high) leads Science, Gemini 4 Carbon (high) leads Healthcare, and Claude Sonnet 5.5 (max) narrowly leads Law & Government
Thank you to the @GoogleDeepMind team for their work building this benchmark and collaborating with us to bring it exclusively to Artificial Analysis!
Created a "benchmark benchmark": a frequency distribution of which benchmarks were reported by the frontier labs in 2026: claude.ai/artifact/22cBPJt6A…
Some observations:
- HLE, SWE-Bench Pro, DeepSWE, and Terminal-Bench saw universal adoption
- Scale AI is behind two of those four (SWE-Bench Pro, HLE w/ CAIS), plus MCP-Atlas (#9 benchmark)
- Of the frontier labs, OpenAI is the most prolific benchmark producer (SWE-bench Verified, BrowseComp, GDPval); people don't seem to mind evaluating on a benchmark made by a competitor
- Anthropic has just one on the chart (BioMysteryBench) that basically nobody uses 🤔
- In academia, strong showings from Princeton (SWE-bench, CharXiv), Stanford (Terminal-Bench), Berkeley (CyberGym, Agents' Last Exam), and NYU (GPQA)
- DeepSWE saw incredible adoption, from first appearing in June to all 12 labs reporting it
- Labs cycled through four versions of Terminal-Bench (2.0 → 4.0) in a single year
Buying answers is over. Labs now pay for the process: trajectories, replayable environments, and rubrics that survive a model actively trying to game them.
Through mid-2025, data was mostly "write a clean solution, ship it." By Q4 the easy sets had no marginal value left. Demand moved to long-horizon systems work, CUA, PCB layout, frontier math, Terminal-Bench-class suites.
A model cannot learn a job from its final answer. Nobody fixes a multi-host outage or routes a 12-layer board on the first attempt — experts spend the day reading errors and backing out of dead ends. That detour log is the signal, and it is not on the public internet.
So vendors buy hours from people who can do the work: $200–300/hr for a math PhD or a senior PCB engineer, with capture tooling recording cursor, keystrokes, deleted code, hesitation. Money is not the bottleneck — senior people cannot articulate their own shortcuts. Rebuilding the environment so the action replays is the next wall: WebHarbor cloned 153 real sites with login state; PCB work needs industrial simulators inside the container.
Then rubrics, and it gets ugly. Acceptance is one line: did the model get better.
- Vals AI caught a trending open model reading answers in two-thirds of its long-horizon coding tasks — the env still had git history.
- Scale AI cut 89 tasks from SWE-Bench Pro V2 after audit.
- Handshake: models spend >80% of reasoning on "what will the grader test."
- Static rubrics decay — write long enough and you collect points. Hence Anthropic's GRAFT.
So vendors now run half a training loop before delivery: stand up the env, run open models hundreds of rounds, hunt fake trajectories, deepen the rubric. Co-evolution. Without a training researcher in-house, you cannot explain why your data bounced.
Embodied is harder. Real-robot factories do not transfer across hardware; handheld capture is cheap — AIRoA-0 logged 16,151 hours and still transferred across robots with zero real-robot data — but five-finger mapping loses fidelity. World models flatter outcomes. Acceptance stays contested.
Mercor mapped 923 occupations: ~90% of US work time is uncovered by existing benchmarks. Labs will not chase it; enterprises will, via post-training. The vendors who survived lab-grade acceptance build that pipeline.
We build the unglamorous middle: hard trajectories, replayable environments, anti-gaming rubrics. No answer dumps. DM if your eval keeps scoring high and shipping nothing.
#DataMining #Trajectory #Evals #CUA #EmbodiedAI
57
We’re releasing the weights for LongCat-2.5 today.
· 1.6T total parameters, ~48B active, natively multimodal
· Stronger visual understanding and agentic coding in one model
· From complex coding projects to creative design and 3D tasks
Released under MIT. Weights are on Hugging Face: huggingface.co/meituan-longc…
Introducing LongCat-2.5, our latest natively multimodal agentic model with 1.6T total parameters and ~48B active.
Built for long-horizon agentic tasks, it brings stronger visual understanding and agentic coding to work ranging from complex coding projects to visual coding such as creative design and 3D tasks.
Learn more about LongCat-2.5: longcat.ai/blog/longcat-2.5/
holy sh*t, Stanford Lab just open-sourced Meta-Harness, a way to let an AI agent rewrite the harness around a frozen model
1.7k stars, MIT license, and the model weights stay untouched
a harness is the code around the model: what it stores, retrieves and shows while it works
what Meta-Harness changes:
→ a coding agent proposes a new version of the harness code
→ every past attempt (code, scores, execution traces) is saved in a folder
→ the agent reads the whole folder before proposing the next version
→ each round is more informed than the last
→ no retraining, no hand-tuning
the result: Claude Haiku 4.5 with the discovered harness scores 37.6% on Terminal-Bench 2, ahead of Goose (35.5%) and Claude Code (27.5%)
the formula:
propose → run → log everything to a folder → read the folder → propose better
REA hit #1 on GitHub trending, and most people still run it inside a single chat
Split it into 4 subagents and one command takes a program you own from sealed binary to a tested rebuild with a match rate
Set it up once, aim it at the next one:
OpenCode 2 far outscores OpenCode 1.19: from 8th of 9 harnesses to tied for first on all three models.
It replaces 1.19 in the @huggingface Harness Leaderboard, which compares coding agents on the same models. Like most harnesses there, it now uses the Chat Completions API directly, so it's more comparable.
At high effort it solves 65.7% of tasks instead of 43.3% on GLM-5.3-Flash and 69.4% instead of 57.0% on DeepSeek-V4.1-Flash. On Flint, 22.9% instead of 7.1%.
Cost per solved task fell from $3.35 to $0.45.
Part of the old gap was the setup. Harbor's standard route for OpenCode 1.19 is OpenAI's Responses API, which I translated to Chat Completions so it works with Hugging Face Inference Providers.
It ran without errors but silently dropped the model's earlier reasoning. Thanks to @thdxr and @rekram11 from @opencode for spotting it!
Leaderboard: huggingface.co/spaces/joelni…
