Sten Sootla (@sten_sootla) profile picture

Research Engineer @Meta MSL. Working on LLMs for coding - CodeLlama, Llama3, Muse.

Joined December 2017
Sten Sootla retweeted
SWE-sweep is the least saturated benchmark on the model card. New SoTA is 9.6%! Congrats to the Gemini team
I did say we'd be releasing our models periodically...introducing Gemini 4 Carbon, our next frontier model. Following Argon's release just two weeks ago, Carbon shows the pace at which we’re pushing the frontier, it ranks #1 on the recently updated v5.0 of @ArtificialAnlys Intelligence Index at 59. It also leads across key cybersecurity evaluations scoring 88.2% on SEC-bench Pro for finding vulnerabilities, 54.8% on ExploitGym for exploitation, and 71.0% on CWE-bench for patching. We’re seeing significant improvements in long-horizon tasks even compared to Argon, and we’ve strengthened frontier safeguards. As we did with Argon, we’re going to roll it out gradually + responsibly. Some trusted cyber defenders will get access starting this week. We'll be expanding it to more partners through our Fairwind Program, ahead of wider availability soon, so hang tight!
4
1
17
3,003
Sten Sootla retweeted
Progress on ProgramBench is crazy! When it was launched, every model had a flat 0% score and we were worried enough to introduce weaker metrics ("almost resolved" etc.), but things have certainly changed!
4
9
2
63
6,327
Sten Sootla retweeted
Today we published a technical report on Muse Glimmer, our 30B open-weight model built for agentic tasks on consumer hardware. It runs at practical speeds on a Mac or PC with a single consumer GPU at 4-bit precision, and leads comparably-sized open models on tool use and multi-step planning. Read about the model architecture, distillation recipe, quantization and more: go.meta.me/4Bcvxol
184
187
145
1,417
237,661
Check out SWE-sweep, our new coding benchmark! SWE-bench gives the model an issue to fix. SWE-sweep goes a step further, towards full autonomy: the issue description is gone - the model is dropped into a codebase and simply tasked to proactively find and fix as many latent bugs as possible.
Can LMs discover & fix bugs if you don't tell them what went wrong? To proactively maintain a repo, agents need to find problems before users do. SWE-sweep benchmarks this on 100 repos, 22 languages, 4k real bugs (numpy, php interpreter, lean kernel). Top models get <5%.
3
2
25
1,341