All the tests were green — what actually worked when reviewing AI-written code
- CodeRabbit
- AI
- ClaudeCode
- code review
- codex
In solo development, I leave most of the implementation to Claude Code and Codex.
The code grows quickly when you do, but the problematic cases are when all tests, lint, and build pass, yet it is still broken.
In this article, I’ll write about two cases I actually encountered and the review process that took shape from them.
Rather than general advice, everything here was taken from my development records over the past year.
Case 1: There was a path where a development secret value could be used in production
I had written the pepper (secret value) used to hash responses like this.
At first glance, this looked like a fail-safe design that would fail if SECRET_PEPPER was unset in production. The tests, lint, and build all passed.
However, in anadversarial review requested to "read this implementation with the intent of breaking it," it was flagged as a blocker.
The problem was that config.ts, which reads the settings,silently treated an unknown APP_ENV or an unset value as 'local'.
(It was intentional leniency to avoid a circular dependency with logger, and was not a problem on its own.)
In other words, merely forgetting to set APP_ENV in production or misspelling it would cause the following:
- env would become 'local'
- all hashes would be generated using the known pepper hard-coded in the source
- because answers such as personal names are short, they could be recovered from the hashes through brute force
This created a path to that outcome. The anomaly produced only a one-line warning log, and nobody reads production logs.
The fix was toAND two signals from different sources.
Since next build / next start set NODE_ENV=production themselves, it takes effect independently of forgetting to set the app-specific APP_ENV.
Lesson: Check whether the “permissive default” and the “safety-first decision” are linked through a single variable.
During code reviews, we made it a rule to always ask, “What happens if this condition becomes true by mistake?”
Case 2: A note deleted with × comes back — the boundary becomes apparent when shown to another AI
In a browser-based learning game, a user pointed out that “deleted notes were coming back.”
The cause was that the process of reconciling the data on the device and server took a union.
The rules for append-only data, such as “cleared chapters,” had also been applied to lists whose entries could be deleted. Even after the post-deletion list was sent, it came back merged with the entries still remaining on the server.
Two more bugs were lurking in the same path.
- Every time you match a response, the words you wrote yourself disappear
- The response-matching process calls save, which then schedules the next submission. As a result, even when doing nothing, it was communicating roughly every 2.5 seconds.
We added an update timestamp to the list and changed the approach to “keep the newer one in its entirety.”
Here, we had an AI (Codex) other than the one that wrote the fix review the diff, and it surfaced three points that neither the AI that implemented the fix nor I had considered.
- Handling cases where the same sentence appears on both sides at the moment of login
- Content edited during the conversation gets rolled back in the reply
- The time is based on the terminal's clock, so a terminal whose clock is ahead keeps winning (explicitly state the upper limit, and switch to the server-side version number if this becomes a problem)
Lesson: Rather than asking the AI that wrote it to "review it,"it is better to give only the diff to a different AI;this surfaces more edge cases. The same model misses the same things.
Established workflow
1. Separate the AI that writes the code from the AI that reviews it
We use Claude Code for implementation and Codex for reviewing the diff.
Since it is set to --sandbox read-only, the reviewer will not modify the code.
2. Get an independent opinion before showing the plan
For high-risk changes (authentication, data deletion, billing, locking, etc.),
before showing my plan,
I have another AI list the "anticipated bugs, invariants that must be preserved, and necessary tests."
3. Limit review requests to "one diff, with scope"
When the request is vague, the reviewing AI may explore the repository, use up time and quota, and end with zero findings.
I now write this in focus:
Including "If there are no issues, explicitly state that there are none" also makes it less likely that findings will be forced or contrived.
4. Don't use a single "green" run as evidence
An E2E test using a random seed would fail 1–3 times out of 6 runs with the same code. Moreover, because the deployment script piped the test results through `| tail`,the exit code became 0, making failures appear successful.
When an AI reports that "the tests passed,"I check what was run, how many times, and with which exit codeit passed.
Summary
| Examples | Test | Method used to find it |
|---|---|---|
| A development secret is used in production | All green | Adversarial review ("What if it becomes true by mistake?") |
| E2E fails 1–3 times out of 6 | Reported "green" | Multiple runs + exit code verification |
| The deleted memo comes back | — | User feedback → Differential review by another AI |
The most important thing when reviewing AI-written code isnot to treat green tests as evidence of correctness.
Tests only verify what the author (the AI) anticipated. To look beyond the assumptions, we introduce another perspective (another AI, the specification, or adversarial questions).
Go to list of users who liked
Go to list of comments
Register as a new user and use Qiita more conveniently
- You get articles that match your needs
- You can efficiently read back useful information
- You can use dark theme TAG578@@What you can do with signing up



