Astra 6 significantly outperforms all other models and gets closer than any other model to fully solving all bugs for one repository. This plot shows the distribution of % bugs resolved for every repository.
2
3
214
It's quite interesting that Astra 6 is also cheaper than Sol 5.6. This is a combination of taking less turns and also consuming less input tokens (however note that it's different model generations, Sol 6 would have probably been cheaper)
1
2
140
SWE-sweep is a joint effort across Meta FAIR, Meta TBD, Stanford, Harvard, UW.
@18jeffreyma @rahulkindi @YuxiangWei9 @jma__06 @sten_sootla @parth007_96 @thinkstepbystep @pengchengyin @magpie_rayhou @OfirPress @jyangballin
4
166