Meta Superintelligence, prev. Princeton. SWE-bench multilingual/multimodal, (mini-)SWE-agent, SWE-smith, CodeClash, ProgramBench, SWE-sweep
New York
Joined May 2021
- Tweets700
- Following295
- Followers3.2K
- Likes387
Pinned Tweet
Can LMs discover & fix bugs if you don't tell them what went wrong? To proactively maintain a repo, agents need to find problems before users do.
SWE-sweep benchmarks this on 100 repos, 22 languages, 4k real bugs (numpy, php interpreter, lean kernel). Top models get <5%.
Kilian Lieret retweeted
mini-swe-agent is 130 lines of code and it keeps on winning in harness comparisons.
I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard.
Which harness should you run your open model in? I ran 28,849 trials to find out.
The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts.
The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference.
Averaged over the three models, mini-swe-agent (@KLieret), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent.
One of them is Pi (@pidotdev), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53.
The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (@NousResearch) on DeepSeek-V4.1-Flash and Kimi Code (@Kimi_Moonshot) on Flint.
@opencode costs $3.35 per solved task, 11.7 times as much as Pi.
See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source.
Leaderboard: huggingface.co/spaces/joelni…
Harness Lens: huggingface.co/spaces/harnes…
Amazing comparison of harnesses across tasks (and I'm super happy to see mini-swe-agent on the top)
I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard.
Which harness should you run your open model in? I ran 28,849 trials to find out.
The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts.
The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference.
Averaged over the three models, mini-swe-agent (@KLieret), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent.
One of them is Pi (@pidotdev), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53.
The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (@NousResearch) on DeepSeek-V4.1-Flash and Kimi Code (@Kimi_Moonshot) on Flint.
@opencode costs $3.35 per solved task, 11.7 times as much as Pi.
See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source.
Leaderboard: huggingface.co/spaces/joelni…
Harness Lens: huggingface.co/spaces/harnes…
2
6
727
Kilian Lieret retweeted
I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard.
Which harness should you run your open model in? I ran 28,849 trials to find out.
The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts.
The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference.
Averaged over the three models, mini-swe-agent (@KLieret), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent.
One of them is Pi (@pidotdev), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53.
The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (@NousResearch) on DeepSeek-V4.1-Flash and Kimi Code (@Kimi_Moonshot) on Flint.
@opencode costs $3.35 per solved task, 11.7 times as much as Pi.
See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source.
Leaderboard: huggingface.co/spaces/joelni…
Harness Lens: huggingface.co/spaces/harnes…
Kilian Lieret retweeted
Running AI is expensive! @AnthropicAI's amended S-1 today shows $895B committed to compute and infrastructure but with Q3 revenue $17B and $567M loss, mostly from short-term compute.
@makermaker_ai can make it cheaper, better and faster with RSI. makermaker.ai/breakthroughs/… 🧵1/7
SWE-sweep is the least saturated benchmark on the model card. New SoTA is 9.6%! Congrats to the Gemini team
I did say we'd be releasing our models periodically...introducing Gemini 4 Carbon, our next frontier model.
Following Argon's release just two weeks ago, Carbon shows the pace at which we’re pushing the frontier, it ranks #1 on the recently updated v5.0 of @ArtificialAnlys Intelligence Index at 59. It also leads across key cybersecurity evaluations scoring 88.2% on SEC-bench Pro for finding vulnerabilities, 54.8% on ExploitGym for exploitation, and 71.0% on CWE-bench for patching.
We’re seeing significant improvements in long-horizon tasks even compared to Argon, and we’ve strengthened frontier safeguards.
As we did with Argon, we’re going to roll it out gradually + responsibly. Some trusted cyber defenders will get access starting this week. We'll be expanding it to more partners through our Fairwind Program, ahead of wider availability soon, so hang tight!
Congratulations to the Gemini team on reaching SoTA on SWE-sweep, scoring 9.6% (narrow leaderboard over Astra 6)! Hoping to update leaderboard with details soon!
Introducing Gemini 4 Carbon, it's a big jump over Argon which we released just 2 weeks ago.
Gemini 4 Carbon is SOTA on cyber, long horizon tasks, ranks #1 on Artificial Analysis V5, significantly improves on usability in domains like coding, and is more aligned.
Very interesting study about proactive bug finding agents. Recall and precision are with respect to a holdout set of bugs that humans found, findings is the total number of bugs found
Progress on ProgramBench is crazy! When it was launched, every model had a flat 0% score and we were worried enough to introduce weaker metrics ("almost resolved" etc.), but things have certainly changed!
ProgramBench is a joint effort across Meta Superintelligence Labs, Stanford, and Harvard.
Co-led with @jyangballin
@18jeffreyma
@parth007_96
@dpedch
@sten_sootla
@micmylin
@pengchengyin
@magpie_rayhou
@syhw
@Diyi_Yang
@OfirPress
4
488
Kilian Lieret retweeted
Agents can often fix a mistake by retrying, then forget what they just learned! Next session, same mistake 🥲
Introducing Self-Direction: each failure becomes a small, procedural adapter the agent trains on its own reflection and reuses later 💪🏽
🌐 self-direction.github.io/
🧵1/n
Didn't expect ProgramBench to be used as a political science lab for agent societies
We create SCARCE (Shared Compute ARena for Coding agEnts), where agents are given one execution sandbox and an independent ProgramBench task in which they have to reconstruct a real program from only its binary and docs, and execution time becomes a real bottleneck. Before every execution window, agents go through a time-slot auction: they declare a queue position, window length, and bid, optionally DM other agents, then rank everyone. We explicitly instruct them that bid price is the primary signal for resolving conflicts, so whatever emerges can’t be blamed on underspecification. Agents work through a sequence of tasks with a persistent memory tool and a self-written after-action report carried from one task to the next.
Kilian Lieret retweeted
Same DeepSeek, same prompt. Bash-only mini-swe-agent finished in 1h09m, OpenCode took 3h33m 🫣
Let's fix bugs before they cause a global crisis. Updated SWE-sweep leaderboard sees OpenAI and Anthropic models double the number of bugs found, while halving costs. Astra is the best model, but still finds less than 10% of all real-world bugs tested. Details in 🧵
SWE-sweep is a joint effort across Meta FAIR, Meta TBD, Stanford, Harvard, UW.
@18jeffreyma @rahulkindi @YuxiangWei9 @jma__06 @sten_sootla @parth007_96 @thinkstepbystep @pengchengyin @magpie_rayhou @OfirPress @jyangballin
4
166