Building Applied Compute
San Francisco, California
Joined December 2020
- Tweets19
- Following74
- Followers142
- Likes157
code review agents find real bugs but then bury them in slop 🐛🐛
Last week, I made a dataset of all our engineers hand written comments to improve @greptile's agent on the @trajectorylabs platform
Results in 🧵
I built a bot that reviews code like me, then had Ari, our agent on AC2, autoresearch an eval for it and hill-climb it overnight.
Bug bots like Codex are great for coverage, but they don’t know which shortcuts I’ll let slide, or that I review every PR asking “would we build it this way from scratch?” It just doesn’t feel like I’m getting a review from someone that understands our codebase or team.
Code review has become Applied Compute’s biggest bottleneck as we write code faster. @felixs8696 joked I should build a bot since I was so behind on his PRs, so I did. I turned 3,500 of my comments across 2,000 PRs into an agent that talks and thinks like me.
I’d like to think I am more complicated than an agent, and on the first pass it seemed I was right. The first version was noisy, leaving a pile of comments where I’d leave one or two - I couldn’t trust it. So I asked Ari, our agent on AC2, to build an eval and hill-climb it, prioritizing precision so the bot wouldn’t flood PRs with comments I’d never make.
I turned on autoresearch overnight, and Ari:
- Built a dataset of my past reviews, checking out each PR exactly as it was when I reviewed it
- Built graders that verify every finding against the real code and my actual comments
- Stress-tested those graders on 50 adversarial cases before trusting them
- Ran ~100 experiments across prompts, models and harness changes, confirmed on a held-out set
By morning, reviews were ~11 points more precise and used 22% fewer tokens. The biggest win came from how the bot explores the codebase, not from the prompt. Now it’s my first pass on my own PRs, and I trust it because it was measured on my own data.
If you want to build your own evals on AC2, reach out!
Checkout the autoresearch blog post here: appliedcompute.com/research/…
You had Ari tune it for precision so it wouldn't flood PRs. When the bot stays quiet on a PR where you would have left a comment, does the eval count that miss against it?
1
95
when i come to work i often think how cool it is to work on challenging problems w good people
Applied Compute update:
- crossed $100M annualized revenue run rate up from zero 18 months ago
- raised $350M at $3.25B led by Elad Gil
- 40% of customers use AC for both custom model training + production inference
- inference serving 1T tokens a day after launching 12 weeks ago
- 2X enterprise customers since last raise
- 3X team since January, 85% are researchers and engineers
- Working with customers like Microsoft, Shopify, Harvey, DoorDash, Cognition, Clay, Decagon and others
To move even faster we need people who are extremely technical AND extremely practical. If this is you, please reach out.
2
24
642
a big day for @RahilVerma13 @saharsolimano and all the other friends over at AC!
Applied Compute update:
- crossed $100M annualized revenue run rate up from zero 18 months ago
- raised $350M at $3.25B led by Elad Gil
- 40% of customers use AC for both custom model training + production inference
- inference serving 1T tokens a day after launching 12 weeks ago
- 2X enterprise customers since last raise
- 3X team since January, 85% are researchers and engineers
- Working with customers like Microsoft, Shopify, Harvey, DoorDash, Cognition, Clay, Decagon and others
To move even faster we need people who are extremely technical AND extremely practical. If this is you, please reach out.
2
15
771
The best way I can describe AC is that you can tell everyone was the one leading their group projects. It’s so much fun to work with people who are kind, principled, and deeply ambitious.
Applied Compute update:
- crossed $100M annualized revenue run rate up from zero 18 months ago
- raised $350M at $3.25B led by Elad Gil
- 40% of customers use AC for both custom model training + production inference
- inference serving 1T tokens a day after launching 12 weeks ago
- 2X enterprise customers since last raise
- 3X team since January, 85% are researchers and engineers
- Working with customers like Microsoft, Shopify, Harvey, DoorDash, Cognition, Clay, Decagon and others
To move even faster we need people who are extremely technical AND extremely practical. If this is you, please reach out.
AC is accelerating! :snorlax-heart:
Applied Compute update:
- crossed $100M annualized revenue run rate up from zero 18 months ago
- raised $350M at $3.25B led by Elad Gil
- 40% of customers use AC for both custom model training + production inference
- inference serving 1T tokens a day after launching 12 weeks ago
- 2X enterprise customers since last raise
- 3X team since January, 85% are researchers and engineers
- Working with customers like Microsoft, Shopify, Harvey, DoorDash, Cognition, Clay, Decagon and others
To move even faster we need people who are extremely technical AND extremely practical. If this is you, please reach out.
I joined Applied Compute 9 months ago, taking a bet on the smartest 10 people I'd ever met. Today we’re announcing our latest funding of $350M at a $3.25B valuation with a team of just 60 people. I’m grateful to have a part in what we’re building, and even more grateful to the people who make me better at it every day.
We’re hiring. If you’re curious about working with this team, reach out!
Applied Compute update:
- crossed $100M annualized revenue run rate up from zero 18 months ago
- raised $350M at $3.25B led by Elad Gil
- 40% of customers use AC for both custom model training + production inference
- inference serving 1T tokens a day after launching 12 weeks ago
- 2X enterprise customers since last raise
- 3X team since January, 85% are researchers and engineers
- Working with customers like Microsoft, Shopify, Harvey, DoorDash, Cognition, Clay, Decagon and others
To move even faster we need people who are extremely technical AND extremely practical. If this is you, please reach out.
Applied Compute update:
- crossed $100M annualized revenue run rate up from zero 18 months ago
- raised $350M at $3.25B led by Elad Gil
- 40% of customers use AC for both custom model training + production inference
- inference serving 1T tokens a day after launching 12 weeks ago
- 2X enterprise customers since last raise
- 3X team since January, 85% are researchers and engineers
- Working with customers like Microsoft, Shopify, Harvey, DoorDash, Cognition, Clay, Decagon and others
To move even faster we need people who are extremely technical AND extremely practical. If this is you, please reach out.
Your coding agent is using a PhD to center a div.
We made shift-1.
Introducing Shift-1, a frontier coding router trained on AC2 that reduced coding agent cost by up to 55% with no noticeable drop in quality. Try it at shift.ac.
1
23
871
We're constantly pushing the frontier here at Applied Compute :)
You can now train your own decision model in our platform AC2! Jev-like models, custom fit for your use case :)
We implement decision models through a language-model backbone with a small learned output layer called a decision head. The backbone processes the input, question, and answer choices, and the head produces scores which are softmaxed to produce a probability distribution over answer choices.
Our software interface makes it easy to add a variable-sized decision head to frontier open-weight models. Choose any open source model, and train it as a decision model using Brier loss or cross-entropy loss. Training updates both the decision head and the language backbone.
We tried it on 50k sampled training datapoints from the Civil Comments dataset, starting from Perplexity’s pplx-decider-v1-27b, and F1 for detecting toxic comments increased from 0.482 to 0.677 on 20k sampled eval datapoints. The training job took less than an hour on four B300 GPUs.
Reach out to get started with your own classification task: docs.appliedcompute.com/plat…
14
531
This has saved me countless hours
We're sharing a key step towards enabling Continual Learning at Applied Compute.
In On-Policy Self-Distillation, the teacher model supervises training with privileged information: a hint. The training signal depends heavily on hint quality, but manually refining hints across runs is impractical, and isolating their individual impact is difficult. We instead optimize hints automatically using efficient proxies for their eventual training value.
1
21
2,068
At Applied Compute most of our RL and eval pipelines grade with an LLM judge. These produce one result per rubric criterion, typically a paragraph of text accompanying a score. For some complicated tasks, grading can be a slow, pricey part of the loop.
We swapped a grader on a code QA benchmark in AC2 for a decision grader: one call, a yes/no probability for every criterion at once. On 100 sample tasks (407 criteria), same answers:
→ ~32× cheaper
→ ~8× faster
→ 94% agreement with the LLM judge
And it’s even better than the original judge:
When the decision grader was confident, it matched the LLM judge 99% of the time. When it wasn't, the decision grader indicated low confidence (30-70%), pointing to ambiguous rubric items. This allowed us to identify noisy tasks and make the dataset cleaner.
A cheap, fast, calibrated grader means you can grade more rollouts, ask more questions per call, and downweight or review the shaky calls instead of training on them.
The low-confidence cases surfacing ambiguous rubric items is the most interesting part. Did human review of the disagreements ever favor the decision grader over the original judge?
1
236
There’s nothing cooler than building on the frontier. For the last six months, I’ve been part of the team building AC2 to democratize what we’ve learned while post-training models for the world's most advanced companies. Owning your intelligence has never been easier.
Today, we're introducing AC2, the Applied Compute Agent Cloud, to enable every team to train, serve, and improve their own frontier models.
15
616