We built a dedicated bug fixing agent called Sherlock. It receives bugs from Slack and processes them as structured tasks in Sharkly.ai.

In the past 15 days, Sherlock completed 13% of the company's tasks. Out of the 100 tasks assigned recently, 90 were completed successfully.

What was important was not just improving the model, but controlling the agent's execution process.

1. A 10-stage state machine instead of a role prompt

Sherlock follows a fixed 10-stage process, not relying solely on the prompt "You are a great engineer." It cannot skip stages and cannot modify product code during the diagnosis stage.

2. Two explicit approval tokens

The first workflow had seven approval points, but engineers became a bottleneck. Therefore, we reduced it to two.

[START DEVELOPMENT] — approve code changes

[SUBMIT FOR TESTING] — approve push/merge

"Looks good" or "continue" are not treated as approvals.

3. Mandatory artifacts at each stage

The required files are diagnosis.md, idea.md, plan.md, and review-report.md. We do not consider "diagnosis complete" as a completion message.

4. Recovery from state.json instead of chat history

All states are saved in state.json. If a session is interrupted, the next execution will load the state from the disk. When reconstructing progress from the chat history, there was a problem where the agent would incorrectly generate progress.

5. Maximum of six self-reviews

After implementation, Sherlock checks the diff as if it were a separate reviewer. If there are serious issues, it fixes them and re-reviews, but stops after a maximum of six times and escalates to a human.

6. Before-and-after screenshots of UI changes

For UI changes, screenshots are not passed only by file path but are displayed directly within the conversation.

Real-world example

The bug "the list cannot be scrolled" was processed through the following flow: Slack → Structured Task → 14 minutes of autonomous diagnosis → 3 human judgments → Approval → 37 minutes of implementation → PR, preview URL, and screenshot.

The human judgments totaled 2 times.

Note that the result of 90/100 is limited to the tasks assigned to Sherlock. It is not the success rate for all bugs. Also, the data on re-open rate has not been released yet.

Detailed article:
https://dev.to/yukioikeda/how-to-build-an-ai-agent-that-fixes-bugs-reliably-practice-in-sharkly-o63