AI Coding Assistants in Real Engineering: From Stubborn Bugs to a Nine-Minute Closed Loop

For independent developers and small teams, choosing an AI coding tool comes down to two ledgers: token costs and the real-world delivery rate on engineering troubleshooting. The first determines whether you can sustain usage long term; the second determines whether you can ship on time. Over the past three months, I worked solo alongside multiple AI assistants to produce roughly 270,000 lines of code and close to 3,000 commits, spending about 17,000 RMB per year on various AI tool subscriptions. This article is not about any specific product. Instead, it breaks down the methodology: using a long-dormant bug as a stress test, walking through the full four-step cycle of “reproduce, locate, fix, verify,” then reviewing baseline data and billing structure, and finally presenting a reusable cost model and context management strategy.

1. Why Use a “Stubborn Bug” as a Capability Benchmark

Benchmarks tell part of the story, but in real engineering three things matter more: whether a model can maintain attention across very long contexts, whether it can correctly execute commands in constrained terminal environments, and whether it can avoid introducing regressions during extended editing sessions. A bug that has been shelved for months and survived multiple model attempts happens to cover all three:

• It often spans multiple modules, requiring a large set of related files to be loaded at once, testing context window size and retrieval precision;
• Its reproduction steps typically depend on a specific runtime environment and command-line operations, testing terminal task capability;
• Its fix point is hidden, making it easy to “fix one spot, break three others,” testing the stability of continuous editing.

So I designed this test as an “engineering troubleshooting exercise” rather than a feature demo: provide only the goal and constraints, no hints on where to look, and observe whether the tool can converge autonomously.

2. The Nine-Minute Closed Loop: A Standardized Troubleshooting Workflow

The entire process—from pressing Enter to completing reproduction, modification, and verification—took about nine minutes, with no line of code written by hand. Breaking this down yields four reusable stages.

1. Reproduce: Make the Bug “Observable” First

The first step in troubleshooting is not reading code—it is making the bug appear reliably. A common approach is to write a minimal reproduction script for the trigger condition and include the runtime logs, exception stack, and input samples in the context.

# Minimal reproduction: fixed input, fixed environment, fixed assertion
$ run-repro --case bug_1042 --verbose
[FAIL] expected=ok actual=timeout at stage=dispatch

Key lesson: keep the reproduction script as small as possible, but the logs as complete as possible. A small script controls context size; complete logs ensure the model sees the actual failure path instead of guessing.

2. Locate: Replace “Reading Everything” with “Hypothesis–Verification”

A very long context does not mean letting the model read from start to finish. A more efficient approach is to have it generate several candidate hypotheses first, then verify each one through retrieval and minimal experiments:

1. List three stages that could cause the timeout (dispatch, serialization, retry);
2. Construct an observable metric for each stage (latency distribution, queue depth, retry count);
3. Use a single run to eliminate two, narrowing attention to the real problem.

The value here is turning “reading code” into “running experiments,” which saves tokens and prevents the model from being distracted by irrelevant files.

3. Fix: Small Commits, Rollback-Friendly

Once the specific function is located, apply the principle of minimal change: modify only one logic point at a time, and immediately run targeted test cases after each change. If the tool supports multi-file editing, require it to output a change description before writing code, making human review easier.

Change description (example)
- File: dispatcher/queue.py
- Change: retry decision changed from "count exceeded" to "exponential backoff + idempotency check"
- Risk: affects retry path, requires regression on idempotency test cases

4. Verify: Regression Tests and Smoke Checks Together

Post-fix verification should not only run the triggering test case but also the regression suite for adjacent modules and an end-to-end smoke test. Of the nine minutes, actual “fixing” time was often under two minutes—the rest went to reproduction and verification. This is the essential difference between engineered troubleshooting and “asking AI to write a function.”

3. How to Read Benchmark Data: Three Dimensions Beat One Composite Score

Composite leaderboards are useful for initial screening, but engineering selection should look at sub-scores that match task types. Three categories of metrics are worth watching:

• Long-duration continuous coding: measures a model’s ability to maintain consistency across multiple editing rounds, suited for “large-scale refactoring and cross-file fixes”;
• Autonomous task execution: measures a model’s ability to plan steps, invoke tools, and validate results on its own, suited for “unattended batch processing and inspection”;
• Constrained terminal tasks: measures a model’s operational accuracy in real command-line environments, suited for “deployment, troubleshooting, and script orchestration.”

Two caveats when reading this data: First, score gaps must be interpreted alongside task distribution—a few percentage points of lead on one sub-score may yield far greater real-world benefit on long-tail bugs than the composite gap suggests. Second, benchmarks are static while engineering environments are dynamic. Always run a small-sample replay using historical bugs from your own codebase to get your team’s “private benchmark.”

4. Cost Model: Converting Subscription Fees into “Deliverable Troubleshooting Sessions”

The key to a 17,000 RMB annual budget is not the number itself but how many effective deliveries it buys. A recommended accounting approach:

Item	Basis	Purpose
Input tokens	Billed per million tokens, distinguishing cache hits from misses	Primary cost of reading codebases and logs
Output tokens	Billed per million tokens, typically priced higher than input	Generating patches, explanations, test cases
Cache hit rate	Cached portion priced significantly lower	Determines whether long sessions are cost-effective
Effective delivery rate	Ratio of sessions that complete “fix + verify” in one pass	Converts to true unit cost

A practical estimation formula:

Cost per troubleshooting session ≈ (uncached input × input price + cached input × cache price + output × output price) / effective delivery rate

The “effective delivery rate” is the most easily overlooked factor yet the one that most affects conclusions. If only six out of ten sessions actually close the loop, the tokens consumed by the other four should also be amortized into the cost. Under this framework, long contexts do not necessarily get more expensive: by placing stable project descriptions, coding standards, and directory structures into the cached prefix, the cached portion can be an order of magnitude cheaper, and the unit cost of long sessions actually decreases.

5. Context Management: Making Long Windows Actually Work

The point of a million-token window is not “stuff everything in at once” but “layer the loading.” A three-layer structure is recommended:

1. Persistent layer: project conventions, build commands, directory index, common tool descriptions. Stable content, suitable for caching.
2. Task layer: files, logs, and reproduction scripts directly related to the current bug. Replaced as tasks change.
3. Temporary layer: experiment outputs, retrieval results, failed attempts. Discarded after use to avoid polluting subsequent reasoning.

Two accompanying disciplines: First, any log entering the context must be trimmed first (remove timestamps, deduplicate stacks)—information density over volume. Second, after a certain number of turns, proactively “restart” the session, carrying only the persistent layer and task conclusions, to prevent early irrelevant content from diluting attention.

6. Solidifying the Method into Team Practice

A single nine-minute fix can be a coincidence; codifying the process is where the real return lies. Four things to implement:

• Bug replay set: Select 20–30 representative bugs from historical issues to form a private evaluation set. Run it whenever a new tool or model is introduced;
• Prompt templates: Write the four steps—“reproduce, locate, fix, verify”—into a fixed template to reduce startup cost for each conversation;
• Change review gate: AI-generated patches must include a change description and impact scope. Human reviewers only examine “why change” and “will it break”—not syntax;
• Cost dashboard: Track token consumption, cache hit rate, and effective delivery rate weekly, turning budget decisions from “does it feel expensive” into “is the unit delivery cost decreasing.”

Conclusion

The value of AI coding assistants lies not in demos but in delivery. Use stubborn bugs as your benchmark, a four-step process to control pace, sub-scores for tool selection, an “effective delivery rate” to correct the cost ledger, and layered context to use long windows properly—this methodology does not depend on any specific product and holds when you switch tools. What truly needs to be accumulated is process and data: once your team has its own bug replay set and cost dashboard, tool selection is no longer a matter of intuition but a reproducible engineering decision.