The SQL assistant on my site had a bug. When you typed "is null", it generated IS NOT NULL. Inverted. The exact opposite of what you asked for, in a tool whose entire job is producing SQL you paste into a production database.
That tool had been live for months. Nobody reported it. I found it because I pointed Claude Code at my own site with one instruction: assume everything is broken, and prove it.
Here's the uncomfortable context: Claude Code wrote that tool in the first place. I shipped 86 free browser-based tools this year, almost all of them AI-written, reviewed by me, deployed to production. Then I made the same AI audit its own work in an adversarial loop. Across two campaigns it found and fixed roughly 140 verified defects. The July sweep alone fixed about 60 bugs touching 44 of the 86 tools.
Half my catalog. Let me show you the receipts, because the specific bugs matter more than the count.
The greatest hits
These are real defects from the audit log, each one shipped to production by an AI, caught later by the same AI in adversarial mode:
| Tool | The bug | Why it stings |
|---|---|---|
| AI SQL Assistant | "is null" generated IS NOT NULL | Inverted logic in output users paste into databases |
| chmod Calculator | 4-digit modes silently dropped setuid/setgid/sticky bits | A permissions tool that lies about permissions |
| Password Generator | Modulo bias in character selection | The UI literally said "cryptographically secure" |
| Cron tools | 6-field expressions mis-parsed seconds vs. year; day-of-week 7 rejected | Wrong schedules from a scheduling tool |
| Regex Tester | Catastrophic backtracking froze the whole tab | A regex tool that a regex could kill |
| QR / Screenshot Generator | Copy-to-clipboard wrote an empty string right after the image, clobbering it | The one button everyone clicks did nothing |
| CSV tools | Quoted fields with embedded newlines broke round-tripping | The single hardest thing about CSV, missed |
| Color tools | 3-digit hex shorthand produced rgba(NaN, NaN, NaN) | #fff is not an edge case |
| Weather Widget | Used the browser's clock for remote cities; classified EU air quality with US bands | Confidently wrong in two hemispheres at once |
| Docker Run to Compose | Unescaped quotes produced invalid YAML | Output that fails the moment you use it |
And my favorite meta-bug: the blog's table-of-contents generator scraped headings from inside code fences, creating ghost navigation entries with dead anchors. The AI-written blog engine was hallucinating structure from the AI-written blog posts.
What this actually proves (it's not "AI code is bad")
The lazy reading is "see, AI-generated code is garbage." That's not what the data says. Most of these 86 tools worked correctly for their primary path on day one. What the audit caught was almost entirely edge-case correctness: shorthand hex, embedded newlines, 6-field cron, non-UTC clocks, the second click instead of the first.
That distribution should look familiar to anyone who has done code review on human teams, because it's the same distribution. The difference is volume and confidence. An AI will produce a plausible, working-for-the-demo tool in minutes, and plausible-working is precisely the failure mode that survives a human skim review. I skimmed. I shipped. The demo path worked.
This is the same pattern I found auditing other people's setups in the 12-team AI coding audit: the expensive failures are never "the AI wrote nonsense." They're "the AI wrote something 95% right and the missing 5% was silent."
The method: adversarial self-review as a loop
One-shot "review this code" prompts found almost nothing. What worked was structuring it like an actual audit engagement:
- A standing instruction with a quota mindset. Not "check for bugs" but "assume defects exist; each iteration, run lint, typecheck, build, content scans, and link scans, then triage findings HIGH/MED/LOW and fix until none remain." Framing matters. A reviewer told to confirm quality confirms quality. A reviewer told to find what's broken finds what's broken.
- Fresh context per iteration. Each pass started clean instead of accumulating a growing transcript of self-congratulation. Stale context is how agents talk themselves into "looks fine." I covered why in the context engineering post.
- Verification before credit. A bug only counted when the fix built clean and behavior was confirmed, in-browser for anything user-facing. The final sweep verified all 86 tool routes returned 200 and re-tested the fixed tools by hand.
- Knowing when to stop. Around iteration 58 the loop went clean and the log's own recommendation was to cancel it, because further iterations would burn harness cost without finding new defects. Autonomous loops need an exit condition or they become the runaway-cost problem I wrote about in May.
Total cost: a weekend of intermittent supervision and a few dollars of tokens. Compare that to what 60 user-facing correctness bugs cost in credibility on a site whose pitch is "tools that just work."
The part that generalizes to your infrastructure
My blast radius was small. Worst case, someone got a wrong cron schedule from a free browser tool (I'm sorry) and hopefully caught it in review. Now re-run this story with the AI writing Terraform, IAM policies, and Kubernetes manifests, which is exactly what teams are doing at scale this year.
The same edge-case failure distribution applies. The 95%-right security group. The IAM policy that's correct except for one wildcard. The lifecycle rule that's inverted, like my IS NOT NULL. Except infrastructure doesn't fail like a color picker; it fails like a $47K AWS bill at 2 AM.
So the takeaway is not "don't let AI write code." I'll keep letting it write code; the 86 tools exist because that's economically unbeatable. The takeaway is:
- AI-written code needs AI-scale review. Human skim review cannot keep up with AI-speed shipping. The only reviewer with matching throughput is another agent pass, run adversarially.
- Make the auditor assume guilt. "Verify this works" and "prove this is broken" are different prompts with wildly different yields.
- Count only verified fixes. An agent that reports "fixed 60 bugs" without build-and-behavior proof has fixed approximately zero bugs.
- Give loops an exit condition. Clean iterations are a signal to stop paying, not proof of perfection.
- Weight the effort by blast radius. I did this for browser toys. If AI writes your IAM policies, this isn't a weekend hobby, it's a control. Pair it with deterministic guardrails for the mistakes that must never ship.
The SQL assistant now handles "is null" correctly. I know because the thing that broke it proved it fixed it, and this time I checked.