Debugging with LLMs: A 9-Minute Bug Fix Case Study and Cost Analysis

1. Background: A Persistent Engineering Problem

In a proprietary content distribution tool, a long-standing defect affected system stability. The issue manifested when a large number of pending items appeared simultaneously in the task queue: the scheduling module would occasionally consume the same record multiple times, leading to duplicate downstream writes. This bug was not consistently reproducible but occurred sporadically under high concurrency and uneven task durations, making it costly to locate.

Previously, the team attempted to mitigate the issue by increasing lock granularity, adjusting the state machine sequence, and introducing delayed retries. However, these measures only reduced the probability of recurrence without solving the root cause. After each deployment, the monitoring system still triggered intermittent duplicate consumption alerts. Since this module is part of the core pipeline and involves task state transitions, idempotency controls, and exception rollbacks, modifying it carried significant risk. Consequently, it remained in a “run-with-bugs” state.

This time, we decided to change our approach: instead of manually deducing the root cause line by line, we provided the complete engineering context to an AI coding tool, requiring it to execute a closed-loop process from localization to verification.

2. Task Setup and Input Preparation

To enable the tool to complete the task efficiently, input quality proved more critical than the model itself. Rather than simply pasting an error log, we provided three categories of context, following standard engineering debugging practices:

1. Module Documentation: Described the task scheduler’s responsibilities, state field definitions, consumption entry points, and retry paths.
2. Key Code Snippets: Three core logic blocks involving state updates, queue polling, and idempotency checks, along with their corresponding test cases.
3. Reproduction Conditions: Defined the sporadic duplicate consumption under high concurrency, supplemented with recent alert timestamps and key pipeline metrics.

We also established explicit constraints: no new external dependencies could be introduced; changes had to be compatible with existing data formats; regression tests were mandatory; and the final output had to include a modification summary and risk assessment.

This structured input effectively transitioned the tool into the mindset of a “developer familiar with the codebase,” rather than just answering an isolated question. The subsequent process proved that this step dictated the upper limit of the fix’s quality.

3. Execution Process: From Localization to Verification

3.1 Root Cause Localization

Instead of immediately outputting a “list of possible causes,” the tool traced the call chain layer by layer. It first identified a non-atomic window between task polling and state updates: after a task is read and before its status is marked as “processing,” an exception or concurrent context switch could cause the same record to be read by another worker thread.

More critically, it pinpointed a flaw in the idempotency control’s placement. Although the current implementation performed an idempotency check before the final write, the validation relied on an auto-incrementing primary key and state combination, rather than the task’s unique business key. When the same business key was split into multiple internal tasks, this validation failed.

3.2 Fix Implementation

Based on these findings, the tool proposed a combined modification strategy:

• Merge task polling and state marking into a single atomic update operation, utilizing a “claim first, process later” approach to prevent concurrent duplicate reads.
• Add the unique business key dimension to the idempotency check and establish a unique constraint or equivalent index.
• Perform a lightweight cleanup of historical存量 data to merge duplicated internal tasks.
• Add a high-frequency concurrency regression test simulating multiple threads simultaneously polling the same batch of tasks.

The proposed solution did not introduce complex distributed locks or transaction frameworks. Instead, it covered the risk window with minimal changes within the existing architecture. This aligned with the team’s expectations: fixing legacy issues should focus on precisely intercepting the defect path rather than rewriting the entire module.

3.3 Regression Verification and Results

Upon completing the code modifications, the tool synchronously updated the test files. The tests covered scenarios such as continuous polling, duplicate polling after an exception interruption, and cross-business-key conflicts. All tests passed locally.

Subsequently, a concurrent verification script was constructed, using multiple threads to contend for the same batch of tasks, logging consumption and write counts. Before the fix, the duplicate consumption ratio in extreme scenarios was approximately 0.8% to 1.2%; after the fix, consecutive executions showed zero duplicates.

The entire process, from task submission to verification completion, took approximately 9 minutes. During this time, no human manually wrote or modified any code; human involvement was limited to context preparation, result review, and deployment decisions.

4. Quantitative Observations on Cost and Efficiency

4.1 Single Task Cost

This fix involved inputting approximately 340 lines of context and multiple tool invocations. Based on current public general-purpose LLM API pricing models, the total cost for this single task was estimated to be between 0.11 RMB and 0.18 RMB. This cost covers multiple inference rounds, code generation, and test output, but excludes the human time spent organizing the context.

4.2 Long-Term Budget Projection

Observing this cost within the context of a long-term project provides better reference value. Assuming a developer produces approximately 270,000 lines of effective code annually, with about 60% eligible for AI-assisted generation or refactoring, the annual inference cost alone would range between 15,000 and 19,000 RMB.

This estimate is based on the following premises: code requires multiple interaction rounds to reach a usable state; some complex modules still require manual rewriting; and the average effective retention rate of generated content is about 65%. Even calculated at the upper limit, the cost remains significantly lower than equivalent human labor. Calculated by the hourly rate of a mid-to-senior level developer in a major tech hub, the labor cost for the same output volume is typically several times higher.

4.3 Implicit Benefits

Beyond direct expenses, the shift in debugging and maintenance costs cannot be ignored. If handled manually, this type of legacy defect typically requires a localization cycle ranging from hours to days, potentially disrupting other development schedules. With AI assistance, most mechanical reasoning and test writing are front-loaded, allowing human resources to focus purely on solution review and boundary confirmation.

However, the cost advantage is not unconditional. It relies on clear task decomposition, complete context, and explicit constraints. If the input is ambiguous, the model may repeatedly generate invalid proposals, inflating both invocation and review costs. Therefore, what truly needs improvement for long-term use is engineering expression and testing infrastructure capabilities.

5. Subsequent Practice: A Frontend Landing Page Implementation

To evaluate the tool’s performance on non-debugging tasks, we subsequently assigned a frontend implementation task: generating a status display landing page for a content distribution flow.

The input contained only field descriptions, target layouts, and interaction requirements. The tool output a runnable page skeleton, including a status list, filtering logic, and empty state prompts, along with responsive design considerations. The initially generated structure was largely functional, with primary manual adjustments focused on visual spacing and status color mapping.

This task demonstrated that when objectives are clear and logical boundaries are well-defined, the tool maintains high usability for frontend implementation tasks. However, visual details and business tone still require human calibration. It is best utilized as a “running starting point” rather than a “final draft requiring no review.”

6. Conclusion: How to Properly Utilize AI Coding Tools

The most significant takeaway from this practice is not the “9-minute bug fix” result, but the validation of a replicable workflow:

• Transform the problem into complete context rather than merely providing error snippets.
• Define explicit constraints, especially regarding dependency limits, compatibility requirements, and validation standards.
• Treat the tool’s output as a “pending engineering proposal” rather than production-ready code.
• Quantify the fix’s effectiveness using regression tests instead of relying on the subjective assumption that “it looks fine.”

Within appropriate task boundaries, AI coding tools can effectively reduce the marginal costs of fixing legacy defects, refactoring code, and template-based development. For long-term maintenance projects, the annual budget is controllable, and the benefits are clear. However, true efficiency gains still stem from the synergy of engineering capabilities: clear module boundaries, a verifiable testing system, and a sound human review mechanism. The tool amplifies these capabilities; it does not replace them.