San Francisco, CA
Joined July 2010
- Tweets11.8K
- Following165
- Followers56.9K
- Likes36.1K
Pinned Tweet
How to effectively debug with agents when you no longer read or understand the code
In this post, I propose a debugging framework for agents that doesn’t rely on human knowledge of code to get to the right “root cause”
The TL;DR are the screenshots.
Link to post in the reply.
CI as it currently exists (build all the code to run the full test suite) seems like a waste of compute, which can be better assigned to agents to analyze and test select parts of code but way more rigorously.
My pre-merge tests don’t do full builds; just targeted micro-tests.
Revisiting my tweets from 6m ago is funny, because I don’t run into any of these problems now at all.
What was missing in my setup 6 months ago was the verification process I now use.
70% of my daily token spend goes into verifying patches, and only ~20-30% into creating them.
This is 100% true.
Your reliability in general should be *higher* with agent written code, as measured by error rates, SLO violations, client regressions etc.
If you’re having more or equal incidents now than when you were manually writing code, that’s likely a “skill issue”.
How to design good verification loops?
Well, that’s most of the job now, isn’t it?
Last week, I finally landed a refactor I started in Jan. It’s good I waited b/c the Jan version was buggy af.
This is the verification process I used for refactors (have different ones for perf)
Getting agents to produce “high-quality, optimized outputs in a language or domain you’re personally unfamiliar with” is an emergent skill.
It doesn’t happen magically, but requires:
1. A change in mindset
2. Setting up the best domain references and rigorous verification loops
I still can't believe people are rewriting software into Rust for performance reasons without learning any Rust.
I'm optimizing a Go program right now and using Agents to do the work, but I have to steer them a lot or they get stuck in local minima, but I can only steer them because I have 10 years of doing performance work in Go.
If I was optimizing a Rust program I'd have to just keep saying: "make it faster" over and over again until it got stuck somewhere dumb.
Returning from international travel, followed by jet lag, physical exhaustion, remnants of a throat infection I picked up when traveling, dealing with a case of misplaced luggage across 3 carriers, catching up on work on a quasi-zombie state …
Yeah, I’m glad its the weekend ❤️
21
3,815
Didn’t Google just blog about how they solve this today? cloud.google.com/blog/topics…
Secrets are kept entirely out of the agent sandbox. A security proxy injects credentials when needed.
The “secure proxy” route to secure agent infra and secrets seems to be a popular pattern.
The reason I still review every line of code despite encoding all the software engineering best practices in my agentic workflow, is because sometimes it’s the *exceptions* to these generalized rules that work best.
The agent isn’t discerning enough to make these calls (as yet)
At Paris CDG, Terminal 2E, security check:
Gate K: no need to take your electronics out
Gate M: take all your electronics out and put it in a tray
Me: I wasn’t asked to do it in Gate K
Security: Gate K is Gate K, Gate M is Gate M
Never seen this in any other airport. 😕
20
68
8,659
Cindy Sridharan retweeted
I came back to Google to work on AI infra. Instead we got pulled in to solving the problem of exploding use of agents across the company.
Announcing Agent Compute Engine: bringing planet scale agentic infra to Google services and now to Google Cloud.
cloud.google.com/blog/topics…
I found this unpublished blog post from 6y ago (look at the hand drawn diagram lol).
I’d totally forgotten about this, but the key ideas here (embracing non-determinism as a first class citizen via “error kernel” centric design etc) work super well as guidelines for my agents.
5 months and many experiments later, this was totally a case of “skill issue” on my part 😅
Like with code, agents don’t magically write great tests
If your unit tests are slop (as mine were), that’s because you don’t have proper testing guidelines for your agents (as I didn’t)
We’ll still need end to end tests, and frankly, we’re going to need all the forms of testing I’ve been talking about for a decade.
But “unit tests are now slop” doesn’t have to be the foregone conclusion.
Unit tests can be very useful even if a human will never again read them.
Unfinished coding side projects are now way easier to complete than unfinished blog posts.
I’m perfectly OK with AI writing my code, even the docs.
But I flinch at the idea of AI writing my blog posts, which feels like an egregious violation of a fundamental ethnical contract.
I started this series on testing nearly a decade ago, but never published the final part copyconstruct.medium.com/tes…
There’ve been so many great advances in the storage space in the ~7y I feel this is worth completing.
Have a draft from years ago I hope I can wrap up in early 2027.
I’m literally that weirdo who wrote (all by hand) a 15K word blog post on:
- why one cant treat models as oracles
- how models gaslight you all the time
- what it takes to build confidence in the model’s output
copyconstruct.medium.com/how…
So yeah sure, I believe in magical woo-woo
To create my claude.md, I had it read:
- my favorite programming books + blogs
- the best oss projects
- the code of the engineers I respect the most
Claude (when given proper instructions, constraints, and context) now writes better code than me, almost always.
the coding agent haters are getting dunning-kruger'd so hard right now
do you really think that your engineering ability is better than the models at this point?
it is very very likely that fable is better than you in every branch of software
it has been trained on all the code of the internet, it knows almost every trick in the book, all the design patterns of every domain, every single low-level detail, every bit-twiddling hack, it's read every single blog post from every cracked engineer that ever existed
do you really think you can beat it? do you really think you can be more creative than it?
unless you are 99th percentile expert in some niche domain of software, it simply is not possible. sorry.
Many moons ago, I wrote about “post-commit” reviews.
In the agentic era, a workflow I find extremely useful is … *pre-commit* agentic reviews.
For large changes, the review agents take 10x the time it takes the coding agent to create a patch, but is totally worth it, IMO.
Blogged: Post commit reviews
Where I question if pre-commit mandatory code reviews are truly needed and what a post-commit review world would look like.
Caveat: post-commit review doesn’t necessarily mean post-deploy review.
bit.ly/2Oid8jG
It’s disappointing to see people say all code is going to be slop and we should be ok with it, because humans weren’t writing the best code either.
Quality and reliability should be higher (as quantitatively measured), not lower.
Why’re we lowering the bar for ourselves? 😔
Cindy Sridharan retweeted
Resharing this and urging everyone to read it for several reasons.
1. Everything Cindy writes is fantastic and serious engineering discourse
2. The examples here are concrete and clear and most importantly production services with real world consequences