From Long-Term Maintenance to Minute-Level Fixes: A Technical Ledger for LLM-Assisted Programming

I. Background: The Real Pressure of Maintaining a Commercial Delivery System Solo

Over the past few years, I have been maintaining a proprietary content distribution tool. The project has accumulated nearly 3,000 commits and consists of approximately 270,000 lines of code. While not massive, it handles real commercial delivery: content synchronization, distribution queues, template rendering, exception retries, and log tracking are tightly coupled. Any logic change in one module can impact overall system stability.

Operating with a very small team, I essentially run it like a “company of one.” Therefore, I must be extremely restrained with tool investments. My annual spending on various AI tool subscriptions is around 17,000 RMB. This budget is neither exceptionally high nor low. What matters is not the total amount spent, but whether each expenditure effectively replaces more expensive human time costs.

In daily development, I evaluate models based on two metrics. The first is token billing cost, which determines whether high-frequency calls are sustainable. The second is the actual engineering troubleshooting delivery rate, which dictates whether the model is genuinely usable in complex contexts, long dependency chains, and legacy code. Models with high benchmark scores but low delivery rates ultimately only increase trial-and-error costs.

II. A Legacy Defect Exposing Evaluation Blind Spots

The project harbored a long-standing, unresolved defect. The symptom: during specific content distribution tasks, queue consumption would intermittently timeout at a fixed stage, and retries would cause partial task duplication. This issue was not consistently reproducible; its probability depended on data scale and the timing of scheduled task triggers.

I had previously tried several common approaches: adding logs, implementing distributed locks, resetting consumption offsets, and adjusting timeout thresholds. Each attempt alleviated the surface symptoms but failed to eradicate the root cause. When using different models for analysis, some pointed out suspicious fields in the logs, others suggested adding retry backoff strategies, and some leaned towards refactoring the consumption logic. None, however, accurately located the root cause in a single pass.

This defect was stubborn because it spanned three modules: inconsistent time field formats on the upstream task production side; lack of strict idempotency checks on the mid-stream consumption side; and ambiguous handling of “partial success” states during exception recovery on the downstream write side. Individually, none of these issues were fatal, but combined, they created an intermittent pipeline failure.

III. A 9-Minute Fix: The Complete Process

Later, I handed this task entirely to a new-generation LLM. From task submission to reproduction, modification, and verification, the entire process took about 9 minutes. I did not write a single line of code. My only contribution was providing the model with the paths of relevant modules in the repository, key log snippets, reproduction steps, and a list of previously failed fix attempts.

The model first analyzed three suspicious points in the consumption chain and concluded that the problem likely did not lie in the consumer itself. Instead, the upstream task production side was writing time fields without a unified timezone standard, causing boundary deviations during downstream window calculations. It then pinpointed an upstream write function where certain code branches used local timestamps while others used UTC timestamps. In cross-timezone environments, this caused millisecond-level offsets, triggering false positives in the downstream timeout window.

The proposed modification involved only two changes: unifying the time field format and adding idempotency key validation on the consumer side. During the verification phase, the model constructed boundary time inputs simulating the original trigger conditions and confirmed that tasks were no longer duplicated post-fix. The entire fix took roughly 9 minutes: 4 minutes for reproduction and analysis, 3 minutes to generate the modification plan, and 2 minutes for verification.

IV. From Benchmarks to Delivery Rate: Evaluating True LLM Usability

This experience prompted me to re-examine my evaluation framework. Relying solely on composite benchmark scores rarely reflects true engineering value. For solo developers, three metrics demand more attention.

First, long-duration continuous coding capability. Real projects are not trivial algorithm challenges; they are engineering tasks spanning hours and multiple files. A model must maintain context, remember constraints, and adhere to existing code styles. Some models excel at isolated tasks but gradually lose their initial objective and drift during prolonged, multi-step operations.

Second, execution capability in terminal-constrained environments. Practical development often requires operating within resource-limited environments where arbitrary dependency installation is forbidden, existing environments cannot be disrupted, and current directory structures must be respected. Whether a model understands these constraints and provides boundary-respecting suggestions directly impacts the feasibility of the fix.

Third, the delivery rate of real engineering troubleshooting. This is the most critical metric. Delivery rate represents the proportion of model outputs that can be directly deployed, rather than those that “look reasonable but require rework.” Influencing factors include the ability to interpret logs, adapt to historical code styles, reproduce intermittent issues, and demonstrate sensitivity to boundary conditions.

V. Establishing a Sustainable, Neutral Evaluation Process

Because different projects and phases require different model capabilities, I gradually developed a stable internal evaluation process. Here are several generalized steps, not bound to any specific product.

Step 1: Collect a fixed sample set. Select 10 to 15 real defects from historical issue trackers, covering common types: concurrency conflicts, data consistency issues, timezone/time handling, asynchronous callback exceptions, and cross-module coupling. Each sample should include a problem description, key logs, a minimal reproduction path, and an expected fix approach. Samples should not be filtered for “easy resolution”; include a proportion of cases that originally took significant time to resolve.

Step 2: Standardize the task format. Provide identical contexts and requirements: locate the root cause first, propose a minimal modification plan second, and explain the verification method last. The evaluation does not require the model to successfully write the complete code; the focus is on localization capability and plan executability.

Step 3: Record three outcomes. Whether the root cause is accurate, whether the modification is deployable, and whether the verification is convincing. Only if all three pass is it counted as a valid delivery. This prevents being misled by outputs that have the right direction but lack practical execution.

Step 4: Calculate unit delivery cost. Convert the token consumption for each valid delivery into a monetary amount and combine it with the average duration to determine the unit delivery time cost. I prefer using “average time and money spent per effective fix” as the final evaluation dimension, rather than simply looking at how cheap a single API call is.

VI. How to Calculate the Ledger

Whether an annual budget of 17,000 RMB is reasonable depends on how much configuration cost it replaces. Configuration costs consist of two types: the time spent locating root causes, and the rework caused by trialing erroneous solutions. Troubleshooting a stubborn defect manually might take hours to a couple of days; bringing in external support costs even more.

Taking this 9-minute fix as an example, compared to standard manual troubleshooting, it saved at least half a day. If time costs are converted into developer hours, a single fix already covers a significant proportion of the monthly budget. Coupled with daily high-frequency tasks like code completion, unit test generation, log analysis, and deployment script writing, the overall return on investment is acceptable.

However, it is crucial to be wary of scenarios that merely “look convenient.” Many simple tasks do not warrant an expensive LLM call chain, especially those quickly resolved with existing templates, code snippets, or compiler assistance. Establishing a tiered execution strategy is vital: use low-cost solutions for simple problems and reserve high-quality models for complex troubleshooting.

VII. How to Ask a Good Question in Large Codebases

The quick resolution of this fix was largely due to the structured nature of the task description. Here is a reusable description template:

[Problem Module]
Briefly describe the code path, involved files, and functional positioning.
[Symptom Description]
Time of occurrence, trigger conditions, intermittent or consistent, impact scope.
[Key Logs]
Retain fields useful for localization in the logs; remove irrelevant noise.
[Attempted Solutions]
List previously tried fix methods and their results to avoid repeated suggestions.
[Constraints]
State limitations such as no large-scale refactoring allowed, version compatibility requirements, and runtime environment specifics.
[Expected Output]
Locate the root cause first, then provide a minimal modification plan, and finally offer a verification method.

This format is effective not only for LLMs but also for aligning issues within a team. Often, the quality of the problem description influences the final delivery more than the model’s capability.

VIII. From Fixing a Single Defect to Building Long-Term R&D Assets

A single successful fix does not warrant over-reliance. The true value lies in precipitating each fix process into queryable, reusable R&D assets. For example: document root cause analysis processes, extract fix patterns into code review checkpoints, and add common failure causes to test case suites.

As the project continues to evolve, the accumulation of such assets will gradually reduce the recurrence rate of similar issues. The value of LLM-assisted programming lies not just in “rapid fixes” but also in helping teams establish a more systematic knowledge retention mechanism.

IX. Conclusion

To summarize, for solo developers maintaining medium-to-large codebases long-term, evaluating LLMs should not be blindly dictated by composite benchmark scores. More attention should be paid to stability during prolonged engineering tasks, executability in constrained environments, and the delivery rate for real defects. Simultaneously, developers must establish their own internal evaluation samples and cost ledgers to avoid being dragged into rework loops by low-quality outputs.

Fixing a legacy defect in 9 minutes certainly involves an element of luck, but it is underpinned by description quality, sample management, and tiered invocation strategies. Transforming an “AI model bill” into an “engineering delivery ledger” is the only way to truly determine whether a technology warrants continuous investment.