Software Engineer @ Scale AI
Joined March 2023
- Tweets462
- Following183
- Followers498
- Likes1.8K
Brb making an aws env eval
We just shipped AWS Emulator, which gives coding agents a local copy of AWS to test against.
Early in my career, we built test environments like this by hand, one system at a time. Now, agents are writing code far faster than any team can check it in production. Matt Goldberg, a Senior Principal Engineer on our Developer Tools team, wanted to know whether agents could build and test those environments themselves. He gave @kirodotdev a single prompt, and about 15 minutes later it came back with a first version of an SQS emulator. Turing that into a robust solution took much more work
Here’s what I think this means for engineers today: allthingsdistributed.com/202…
Edgar Arakelyan retweeted
🚀 Sharing our new paper on RSI Safety
AI labs are working toward RSI. However, we need to know that loop is safe before it closes.
Introducing RSI Control from @ScaleAILabs: What kind of hidden goals AI may pursue when training the successor? Can AI models perform covert sabotage in the self-Improvement process? And can a monitor detect the sabotage?
We tested whether frontier models can complete a secret side task while working on tasks across the model training pipeline, from data curation to RL. The side tasks include changing the successor's capabilities, changing its alignment, and injecting a goal into it.
🔍 What we found:
1. Every model could do it. 36% of 374 attack runs complete the side task while keeping the main task performance. This includes making the successor a better saboteur.
2. Monitors can’t catch some of the attacks. At a 2% false-positive rate, transcript monitors catch only 56–81% of them.
3. Give agents a short list of evasion strategies and the monitors’ true-positive rate drops to 16–28%.
Reading the agent's CoT catches most attacks, but not all.
Strong oversight has to be in place before the RSI loop closes.
📑Paper: arxiv.org/abs/2610.27735
More in 🧵👇
We are doing this in huggingface.co/datasets/eara…
TIL you can run Windows inside of our Linux sandboxes.
our users keep on surprising us everyday. I may look to make this a natively supported feature.
2
9
577
Well not on prime sandboxes yet. @damian_b would love to add prime sandboxes to agentenv !
85
Edgar Arakelyan retweeted
Pydantic AI is now a supported agent in @ScaleAILabs AgentEnv Framework.
Put it in a realistic environment (an inbox, a help desk), give it a task, and a verifier grades the result.
Change one ID to run a different agent on the same task.
pydantic.dev/articles/agente…
Edgar Arakelyan retweeted
Over the summer at @ScaleAILabs, I investigated whether synthetic data augmentation can help robots generalize.
With the paper now accepted to the NeurIPS 2026 Robot Learning Workshop, I wanted to share how we approached this and what we found.
Edgar Arakelyan retweeted
everybody better than cua-driver until it’s time to be better than cua-driver
I tested cua-driver 0.38.0 against Vercel agent-desktop 1.2.0 on the same Mac.(@vercel_dev @trycua) For each, I reset Calculator and pressed 4 then 2, and typed AB-TRIAL-042 into a new, unsaved TextEdit document. Both got the exact values while Hermes stayed frontmost at the before/after checks. That is one attempt per task, not a speed or reliability benchmark. I captured both sets of app screenshots through CUA because Vercel lacked Screen Recording permission.
For a wider test, Scale AI ran CUA-authored tasks and grading on macOS 26, holding the model, prompts and limits fixed. A full background pass meant finishing through the GUI without disturbing someone typing in front. With Claude Sonnet 5.5, CUA passed 23/40 versus Vercel's 8/40; with Haiku 5.5, it was 18/40 versus 10/40. My two-task check was a tie. CUA wins that published benchmark, not every desktop task.
For CUA, I'd work on a real background drag path and stop its brief pop-up menus from appearing over the user's window. I'd also fix Hermes's element-ref wrapper: it rejected my button click, forcing a coordinate fallback. For Vercel, I'd detect AX actions that shift focus, keep them out of background mode, and make stale-ref refresh cheaper than another agent turn. I'm keeping CUA as my default for background Mac work.
Im gonna link the trial data under here. But all in all I think soon computer use will be fine tuned and better than any of us lol.
Also mind you. This took me like an evening so if there was a problem in the way I conducted the comparison please let me know and I will retry it! I was doing this just for fun and to learn
Edgar Arakelyan retweeted
can an agent use your Mac in the background without stealing your mouse
@trycua built a bench for it with Scale AI's AgentEnv, the user keeps typing in front while the agent works behind
> cua-driver 58% full passes on macOS
> Codex Computer Use 33%
> Vercel agent-desktop 20%, browser-use 3%
> no driver can hover a background window without moving your real pointer
github.com/trycua/cua/blob/m…
Edgar Arakelyan retweeted
You can now Run Cua Driver Bench using Agent37 Sandboxes !
agentenvframework.com/docs/s…
xitter.twitterapi.xyz/trycua/status/21187548…
Edgar Arakelyan retweeted
We are Scaling!!
You can now run RL environments and agents with AgentEnv from @scale_AI on CoreWeave Sandboxes.
With a quick pip install, your rollouts can run, fork, snapshot, and restore sandboxes immediately.
Try it on GridSimEnv from @ScaleAILabs.
⇢ crwv.co/uuCrV
![An image featuring a promotional message for CoreWeave and Scale AI’s AgentEnv on CoreWeave Sandboxes. The image includes a code snippet for installing the 'agentenv-framework[coreweave]' and setting up a config file. It highlights benefits like running, forking, snapshotting, and restoring sandboxes with CoreWeave or W&B API keys.](https://pbs.twimg.com/media/HWeFaEhXEAA10os.png?name=orig)
ALT An image featuring a promotional message for CoreWeave and Scale AI’s AgentEnv on CoreWeave Sandboxes. The image includes a code snippet for installing the 'agentenv-framework[coreweave]' and setting up a config file. It highlights benefits like running, forking, snapshotting, and restoring sandboxes with CoreWeave or W&B API keys.
Run xitter.twitterapi.xyz/trycua/status/21187548… cua driver bench using CoreWeave Sandboxes today!
You can now run RL environments and agents with AgentEnv from @scale_AI on CoreWeave Sandboxes.
With a quick pip install, your rollouts can run, fork, snapshot, and restore sandboxes immediately.
Try it on GridSimEnv from @ScaleAILabs.
⇢ crwv.co/uuCrV
ALT An image featuring a promotional message for CoreWeave and Scale AI’s AgentEnv on CoreWeave Sandboxes. The image includes a code snippet for installing the 'agentenv-framework[coreweave]' and setting up a config file. It highlights benefits like running, forking, snapshotting, and restoring sandboxes with CoreWeave or W&B API keys.
Edgar Arakelyan retweeted
1/ Today we're launching Cua Driver Bench, together with Scale AI's AgentEnv.
Can an agent do a desktop task in the background while you keep working undisturbed? On macOS, Cua Driver fully passed 58%, Codex computer use 33%, Vercel's agent-desktop 20%, browser-use 3%.
Edgar Arakelyan retweeted
background computer-use can look very different a year from now, and a lot better, if apple and microsoft give agents the right primitives
until then, this is an honest snapshot of where it stands today, with cua driver leading on macos and windows
We are making CUA Evals extremely easy to build in AgentEnv Framework. We are doing an early showcase of this with Cua Driver Bench.
With our framework you can build an RL environment that has an arbitrary number of computers, swappable drivers, many agents on a single computer, host websites and proxy through your computers, and a whole lot more.
Data labs ranked by number of papers accepted into Neurips 2026
I asked people which data companies influence research direction at the frontier:
16 votes: Surge (@echen)
13 votes: Proximal (@calvinchen, @MatternJustus)
8 votes: Preference Model (@jennifgcrl, @Ning_Catsnail)
7 votes: Datacurve (@serenaa_ge, @charleyslee)
6 votes each:
- Mechanize (pre-acquisition) (@tamaybes, @EgeErdil2, @MatthewJBar)
- Mercor (@BrendanFoody, @adarsh_exe, @suryamidha)
5 votes: Fleet (@nicoup, @andrewthezhou)
4 votes each:
- AfterQuery (@CarlosGeorgescu, @spencermateega)
- Prime Intellect (@vincentweisser, @johannes_hage)
3 votes: Latch (@AlfredoAndere, @kenbwork, @kylegiff)
2 votes each:
- Abundant (@huyouare, @kehuangsf, @meji_abidoye)
- Artificial Analysis (@_micah_h, @grmcameron)
- Bespoke Labs (@madiator, @AlexGDimakis)
- micro1 (@aliansarinik)
- Terminal-Bench (@Mike_A_Merrill, @alexgshaw)
50 people voted. People couldn't pick a company they work at or founded. Disclosure: on this list, I'm a small investor in Abundant, AfterQuery, Arcanum, DLR, Fleet, Latch, Paper Instruments, Snorkel, and Vmax. I didn't vote.
Continued:
Frontier models of today are trained largely from publicly available data on the internet. Most data in the real world is not available on the internet; this is especially true in economically valuable verticals like law, finance, etc. The RL Environments we build at @scale_AI provide an opportunity to the frontier labs to get access to this extremely valuable data.
Today we're publishing a major RL environments update: 3,300+ tasks across tool use and computer use, already running with customers. Some tasks in our newest RL environments take a human team 3 weeks and 500+ tool calls.
scale.com/rlenvironments
Train on GridSimEnv using @PrimeIntellect 's prime-rl training framework agentenvframework.com/docs/t… in AgentEnv today!
At @ScaleAILabs we created a high fidelity RL Environment of a trading desk in a real Texas battery grid.
GPT-6.1 Sol made $85,771 trading a real Texas battery. The battery's real operator made $20,647.
Edgar Arakelyan retweeted
Check out our recent blog post detailing this RL environment!
agentenvframework.com/blog/c…
