A high-signal data ecosystem leveraging data from industry & academia to deliver post-training to frontier labs, neolabs, & enterprises building advanced models
San Francisco, CA
Joined May 2026
- Tweets441
- Following294
- Followers107
- Likes250
Pinned Tweet
As AI makes expertise increasingly abundant, *trust* will become one of the most scarce and valuable assets.
AA-Paperclip from @ArtificialAnlys treats a document as correct only when all five decisions and ten extracted facts are right. That changes what a strong average means: scattered mistakes can leave many documents unusable. We like checking whether the complete output holds together, alongside measuring individual answers. artificialanalysis.ai/method…
A shared set of benchmarks gives labs a common language for progress. @_jasonwei's "benchmark benchmark" makes that ecosystem visible.
We'd love to see this tracked over time: which tests remain useful as models improve, and which new ones earn a place in launch reports. Building a benchmark that other labs keep using is a real contribution.
Created a "benchmark benchmark": a frequency distribution of which benchmarks were reported by the frontier labs in 2026: claude.ai/artifact/22cBPJt6A…
Some observations:
- HLE, SWE-Bench Pro, DeepSWE, and Terminal-Bench saw universal adoption
- Scale AI is behind two of those four (SWE-Bench Pro, HLE w/ CAIS), plus MCP-Atlas (#9 benchmark)
- Of the frontier labs, OpenAI is the most prolific benchmark producer (SWE-bench Verified, BrowseComp, GDPval); people don't seem to mind evaluating on a benchmark made by a competitor
- Anthropic has just one on the chart (BioMysteryBench) that basically nobody uses 🤔
- In academia, strong showings from Princeton (SWE-bench, CharXiv), Stanford (Terminal-Bench), Berkeley (CyberGym, Agents' Last Exam), and NYU (GPQA)
- DeepSWE saw incredible adoption, from first appearing in June to all 12 labs reporting it
- Labs cycled through four versions of Terminal-Bench (2.0 → 4.0) in a single year
ReasonCore AI @ NeurIPS 2026 retweeted
Created a "benchmark benchmark": a frequency distribution of which benchmarks were reported by the frontier labs in 2026: claude.ai/artifact/22cBPJt6A…
Some observations:
- HLE, SWE-Bench Pro, DeepSWE, and Terminal-Bench saw universal adoption
- Scale AI is behind two of those four (SWE-Bench Pro, HLE w/ CAIS), plus MCP-Atlas (#9 benchmark)
- Of the frontier labs, OpenAI is the most prolific benchmark producer (SWE-bench Verified, BrowseComp, GDPval); people don't seem to mind evaluating on a benchmark made by a competitor
- Anthropic has just one on the chart (BioMysteryBench) that basically nobody uses 🤔
- In academia, strong showings from Princeton (SWE-bench, CharXiv), Stanford (Terminal-Bench), Berkeley (CyberGym, Agents' Last Exam), and NYU (GPQA)
- DeepSWE saw incredible adoption, from first appearing in June to all 12 labs reporting it
- Labs cycled through four versions of Terminal-Bench (2.0 → 4.0) in a single year
AISLE, with @pschwllr and @SchwallerGroup among its contributors, tests models on real laboratory experiments. Its resin-transfer task separates getting the right volume once from keeping transfers within tolerance across repeated operations. Those measures tell different stories. ReasonCore supports this benchmark because we want scientific evaluations to distinguish the best result in a run from sustained control. aisle.science/
RSIJudge, from researchers at @PKU1898 and @NUSingapore, tests what improves when a model reads robot-task grading code and revises its reusable skills. Its extra score disappeared when the same simulated runs faced criteria rewritten to match the instructions. We see a useful test here: do the revised skills perform the task better, or fit one verifier better?
arxiv.org/html/2611.08318
1
25
StateBench, from @jowenpetty, @lambdaviking and @Ashish_S_AI at @allen_ai, separates remembering a value from tracking how it changes. Paired problems vary the update rules while controlling the variables and assignment positions. We like this diagnostic approach: finding a fact in a long context and maintaining the correct state through that context are different capabilities. allenai.org/blog/statebench
VisSearch-Bench, from Tsinghua, @AntGroup and Renmin researchers, tests visual search beyond identifying the first image. Tasks require models to keep objects, images and facts connected across a search, sometimes checking every candidate in a set. We like evaluating that completeness: finding several correct matches is different from establishing that nothing was missed. github.com/antins-labs/VisSe…
TURTLE, from @hjalexnam, @natashajaques and Stanford/Google DeepMind/UW collaborators, evolves tool-use tasks toward problems a model can solve, but not reliably. Proposed solutions are executed before tasks are tested against the model. We like targeting that boundary rather than making workflows longer for its own sake. arxiv.org/html/2610.27307
ReasonCore AI @ NeurIPS 2026 retweeted
Long horizon tasks are getting longer and longer!!!
Any prediction for 2030 and beyond: what will be a long horizon task then?
Terminal-Bench-Science launched in August with a top score of 30%. Today Claude Fable 5.2 reports 74.6%.
Three things worth noting:
1/ Resolution. It's 70 tasks, and the system card gives a task-clustered standard error of 4.3 points. Fable 5.2's 0.7-point gap to Mythos 5.2 is noise, and its 6.5-point gap to GPT-6 Astra is about 1.5 SE.
2/ Supply. 920 proposals became 70 merged tasks, a 7.6% yield after expert and technical review. Hard, verifiable science tasks are being used up faster than they're built.
3/ Signal. Near 75%, more tasks get solved on nearly every trial, and they stop separating models or giving RL a gradient.
A hard problem with an unreliable answer key is bad training data.
Frontier models need challenging tasks and solutions that hold up under scrutiny. We build both: expert-authored scientific and coding reasoning tasks with verified solutions.
ReasonCore has 5,000 CritPt-style tasks, 5,000 SciCode-style tasks 1,000 Terminal-Bench-Science-style tasks, and Terminal-Bench 4-style tasks available.
Building your next post-training run? DM us for samples.
Hard benchmarks make headlines. For training, “hard” is the wrong target.
A science task every model fails teaches RL nothing. So does one every model passes. The signal lives in the middle: a model solves it 1 to 6 times out of 8.
Hitting that band is the expensive part. One terminal-bench-science task can take a frontier agent up to 3 hours. Certifying it takes 8 runs, and all 8 again every time a new model ships or a single file changes.
That’s up to 24 agent-hours before a task is worth handing to a lab.
SeVerus, from @jff and Porto/Lisbon/Rutgers researchers, tests Rust verification as supporting lemmas are progressively removed. A model can complete a proof with help yet struggle to build that help itself. We like the controlled comparison: keep the program and its requirements fixed, vary the proof support, and check both verification and shortcuts that bypass the task. arxiv.org/html/2611.07194
BlockLock, from @JaviConD3 and UESTC/UPM researchers, keeps a request's execution geometry fixed while other requests in its batch change. The team reports repeatable outputs within tested deployments, with validated backend settings. We like this approach for evaluation: checking the serving setup belongs alongside checking the model. A fixed seed alone doesn't control the numerical path. arxiv.org/html/2611.06977
1
1
68
Terminal-Bench 4.1 improves both sides of verification: rejecting reward hacks and accepting valid solutions the original tests missed. @ryan_marten's FreeCAD example makes the distinction concrete: a correct 3D model shouldn't fail because it was built differently. We like @terminalbench and @harborframework's focus on measuring the result without prescribing the route. ReasonCore specializes in Terminal-Bench 4 style tasks.
Terminal-Bench 4.1 is here with updated verifiers for all tasks.
In this release, we focused on:
- resisting reward hacking
- reducing false negatives
- improving reproducibility
Re-grading the 4.0 leaderboard trials with the 4.1 verifiers, resulted in ~1% of rewards flipping.
Leaderboard scores remain within previously reported confidence intervals.
3
121
The useful distinction is that better verification should catch invalid solutions without rejecting valid alternatives. The FreeCAD task previously expected sketch extrusion and rejected correct models built with 3D primitives; 4.1 accepts both. The reported leaderboard scores stayed within their previous confidence intervals, so this is grading refinement, not evidence the earlier rankings collapsed. docs.harborframework.com/new…
1
45
AppCIF-Bench from @TencentHunyuan releases simulation engines that regenerate reference checklists for complex instruction-following tasks. A separate model judge checks responses against those checklists. We like making both stages inspectable: establishing the expected answer and judging a response are different jobs. github.com/Tencent-Hunyuan/A…
1
20
ReasonCore AI @ NeurIPS 2026 retweeted
Adversarial Speculative Reasoning Improves Long Horizon Agents
arxiv.org/abs/2611.05345
AI protein design from UCLA. Go Bruins!
@jasonzxzhang's @UCLA lab pairs AI-generated protein designs with experimental testing, including work aimed at difficult-to-target disordered proteins. We see real promise in this approach: using models to propose new molecules, then testing what they actually do in biology. An exciting path toward better research tools and, potentially, new therapies. newsroom.ucla.edu/stories/bi…
TB-MathModel, from @shawnwu630 and HKUST/IDEA researchers, checks reported answers against task constraints separately from their closeness to human reference solutions. Its checks can reject impossible values while flagging missing evidence for review. We see this separation as important for modeling evaluations: agreement with a reference is not evidence that a proposed solution is feasible. ReasonCore has 5,000 CritPt and 1,000 Terminal-Bench-Science tasks available. arxiv.org/html/2611.08237