
Research Engineer @Meta MSL. Working on LLMs for coding - CodeLlama, Llama3, Muse.
Joined December 2017
- Tweets5
- Following140
- Followers39
- Likes18
Sten Sootla retweeted
SWE-sweep is the least saturated benchmark on the model card. New SoTA is 9.6%! Congrats to the Gemini team
I did say we'd be releasing our models periodically...introducing Gemini 4 Carbon, our next frontier model.
Following Argon's release just two weeks ago, Carbon shows the pace at which we’re pushing the frontier, it ranks #1 on the recently updated v5.0 of @ArtificialAnlys Intelligence Index at 59. It also leads across key cybersecurity evaluations scoring 88.2% on SEC-bench Pro for finding vulnerabilities, 54.8% on ExploitGym for exploitation, and 71.0% on CWE-bench for patching.
We’re seeing significant improvements in long-horizon tasks even compared to Argon, and we’ve strengthened frontier safeguards.
As we did with Argon, we’re going to roll it out gradually + responsibly. Some trusted cyber defenders will get access starting this week. We'll be expanding it to more partners through our Fairwind Program, ahead of wider availability soon, so hang tight!
Sten Sootla retweeted
Progress on ProgramBench is crazy! When it was launched, every model had a flat 0% score and we were worried enough to introduce weaker metrics ("almost resolved" etc.), but things have certainly changed!
Sten Sootla retweeted
Today we published a technical report on Muse Glimmer, our 30B open-weight model built for agentic tasks on consumer hardware.
It runs at practical speeds on a Mac or PC with a single consumer GPU at 4-bit precision, and leads comparably-sized open models on tool use and multi-step planning.
Read about the model architecture, distillation recipe, quantization and more: go.meta.me/4Bcvxol
Check out SWE-sweep, our new coding benchmark!
SWE-bench gives the model an issue to fix. SWE-sweep goes a step further, towards full autonomy: the issue description is gone - the model is dropped into a codebase and simply tasked to proactively find and fix as many latent bugs as possible.
Sten Sootla retweeted
SWE-sweep is a joint effort across Meta FAIR, Meta TBD, Stanford, Harvard, UW
@18jeffreyma @rahulkindi @YuxiangWei9 @jma__06 @sten_sootla @parth007_96 @thinkstepbystep @pengchengyin @magpie_rayhou @OfirPress @jyangballin
Website: swesweep.com/







