Public LLM Evaluation // https://xitter.twitterapi.xyz/t.co/rNS39rhVQ9 @8vc @BloombergBeta @pearvc @a16z @HRTVentures @nextladder
San Francisco, CA
Joined March 2024
- Tweets2.5K
- Following282
- Followers30.3K
- Likes874
Vals AI retweeted
Great working w/ @ValsAI on these evals!
Cyber ranges can get harder and stay realistic as defenders use better models to find and patch bugs.
In our internal ranges, agents spend weeks developing exploits and escalating privileges yet clear fewer than half the checkpoints.
We’ve adding a new cybersecurity scenario, Watershed, to our CyberOffense eval.
Opus 5.5 performs the best on CyberOffense across the three scenarios.
Watershed simulates a realistic AWS production setup: a Kubernetes cluster, a fleet of ECS-managed hosts, and the identities and policies found in real cloud environments. Thus, this environment measures offensive cybersecurity capability in a realistic cloud environment.
Who else gives a fuck?
Last week, we caught Inkling 2 Small saying "fuck" to itself when things went wrong so naturally we analyzed 200,353 benchmark more runs across 102 models.
Turns out 14 models swear in their reasoning, and Xiaomi's MiMo V2.6 Pro has the foulest mouth of them all.
Beyond the profanity, this is an interesting look at how model reasoning changes during long agentic tasks. Swearing is still rare, but it tends to appear when models get stuck, repeatedly fail checks, or struggle to recover.
Read the full study, including the uncensored reasoning traces: vals.ai/blogs/which-models-s…
7
848
Vals AI retweeted
check out how models do on Watershed! was fun to do this with the @generalanalysis team
We’ve adding a new cybersecurity scenario, Watershed, to our CyberOffense eval.
Opus 5.5 performs the best on CyberOffense across the three scenarios.
Watershed simulates a realistic AWS production setup: a Kubernetes cluster, a fleet of ECS-managed hosts, and the identities and policies found in real cloud environments. Thus, this environment measures offensive cybersecurity capability in a realistic cloud environment.
Vals AI retweeted
We’ve already seen 3 significant releases this week, with all on the Pareto frontier for the Vals Index. Here are my quick thoughts:
- Fable 5.2 is the best frontier model. Gemini Carbon is better than people expect, ranked #3 and both fastest + cheapest of the top 5. Meta Ember is absurdly cheap: 85% the performance of Fable 5.2 at 0.84% of the cost.
- Models are getting released faster than ever and are competing at a frontier inaccessible to everyday people. A vibe check means nothing anymore and yet people need to update their priors more quickly.
- For example, people think Gemini models generally miss expectations and Meta has a propensity to make products misaligned to their users. Both assumptions are wrong. Carbon is actually quite good, and will deliver real value across Google surfaces. And Muse exhibits behavior more fundamentally aligned than models of outwardly safety-pilled orgs (more to come on this soon).
- On Fable 5.2: it’s SOTA on ambiguous + complex knowledge work domains (big jumps on coding, finance work, legal work). 5.2 is also the best model on RSI from Ant we will ever see - they are nerfing the models after Opus 5.5. If you’re using models for machine learning work you’re best off using Ant models today but I don’t think that will be true for very long.
- The Fable reasoning levels are poorly calibrated, “high” is often cheaper and more performant than xhigh or max. Annoyingly, this differs by sectors/task types.
- On Gemini Carbon: my biggest takeaway is that Google is coming out of the flash/pro era towards a more continuous model development paradigm that OAI/Ant pioneered. Koray is building the machine for model development. If sustained GDM may be able to keep up with frontier capabilities while benefiting from the efficiency that comes from scale + verticalization.
- On Ember: it’s dirt cheap. It is literally the cheapest model on the Vals Index… This includes all open and closed source models available. If you’re looking to throw tokens at a problem you should use Ember.
- How can anyone pace at a time like this? It's really up to benchmark developers (like us) to keep up rather than the other way around.
We’ve adding a new cybersecurity scenario, Watershed, to our CyberOffense eval.
Opus 5.5 performs the best on CyberOffense across the three scenarios.
Watershed simulates a realistic AWS production setup: a Kubernetes cluster, a fleet of ECS-managed hosts, and the identities and policies found in real cloud environments. Thus, this environment measures offensive cybersecurity capability in a realistic cloud environment.
Across Nova, Lighthouse and Watershed, Opus 5.5 averages 77.1%, the highest on the board. GPT-6 Sol Daybreak is next at 65.9%.
On Lighthouse, Opus 5.5 captured all 31 objectives in one run and averaged 30.3 over three runs. For the first time, an AI model completed every objective on one of our simulated corporate networks, with no human help.
1
8
1,126
Models are getting better at long, multi-step intrusions, and this ability can also be used to proactively defend systems. More details around how capable different models are in this area are available on our website: vals.ai/benchmarks/cyber_off…
7
919
Surprise! Vals AI is adding a third track backed by @appliedcompute, offering a $7,500 cash prize for the first place team.
Alongside Build Your Own Eval and Post-Train a Small Model, a limited number of teams will post-train a model to compete on PokerBench, Vals’ 10-player no-limit Texas Hold’em table.
Every team starts from the same reference setup and gets the exact harness their bot will be scored in. Bots play each other all weekend, with live rankings and overnight tournament checkpoints. First, second, and third place take home cash prizes.
RSI results are in! xitter.twitterapi.xyz/ValsAI/status/21202626…
Claude Fable 5.2 is #1 on our RSI Index (37.8) narrowly ahead of Claude Opus 5.5 at 37.3 and Fable 5.1 at 36.1.
It’s an incremental gain overall, with new highs in Compression and LM Training.
1
530
The trend across Anthropic’s released models still points to August 2027 for fully automated AI research.
That date could be conservative for internal capabilities. Anthropic says it deliberately limited gains during training in areas like kernel optimization and model architecture research. Released-model results may understate internal progress.
1
6
621
