Rahil Verma

I built a bot that reviews code like me, then had Ari, our agent on AC2, autoresearch an eval for it and hill-climb it overnight. Bug bots like Codex are great for coverage, but they don’t know which shortcuts I’ll let slide, or that I review every PR asking “would we build it this way from scratch?” It just doesn’t feel like I’m getting a review from someone that understands our codebase or team. Code review has become Applied Compute’s biggest bottleneck as we write code faster. @felixs8696 joked I should build a bot since I was so behind on his PRs, so I did. I turned 3,500 of my comments across 2,000 PRs into an agent that talks and thinks like me. I’d like to think I am more complicated than an agent, and on the first pass it seemed I was right. The first version was noisy, leaving a pile of comments where I’d leave one or two - I couldn’t trust it. So I asked Ari, our agent on AC2, to build an eval and hill-climb it, prioritizing precision so the bot wouldn’t flood PRs with comments I’d never make. I turned on autoresearch overnight, and Ari: - Built a dataset of my past reviews, checking out each PR exactly as it was when I reviewed it - Built graders that verify every finding against the real code and my actual comments - Stress-tested those graders on 50 adversarial cases before trusting them - Ran ~100 experiments across prompts, models and harness changes, confirmed on a held-out set By morning, reviews were ~11 points more precise and used 22% fewer tokens. The biggest win came from how the bot explores the codebase, not from the prompt. Now it’s my first pass on my own PRs, and I trust it because it was measured on my own data. If you want to build your own evals on AC2, reach out! Checkout the autoresearch blog post here: appliedcompute.com/research/…

At Applied Compute most of our RL and eval pipelines grade with an LLM judge. These produce one result per rubric criterion, typically a paragraph of text accompanying a score. For some complicated tasks, grading can be a slow, pricey part of the loop. We swapped a grader on a code QA benchmark in AC2 for a decision grader: one call, a yes/no probability for every criterion at once. On 100 sample tasks (407 criteria), same answers: → ~32× cheaper
→ ~8× faster
→ 94% agreement with the LLM judge And it’s even better than the original judge: When the decision grader was confident, it matched the LLM judge 99% of the time. When it wasn't, the decision grader indicated low confidence (30-70%), pointing to ambiguous rubric items. This allowed us to identify noisy tasks and make the dataset cleaner. A cheap, fast, calibrated grader means you can grade more rollouts, ask more questions per call, and downweight or review the shaky calls instead of training on them.