Large-Scale AI Coding in Practice: What 30 Billion Tokens and 110,000 Lines of Code Revealed

1. Background and Evaluation Goals

Over a one-month development cycle, we used two types of coding LLMs to assist development on a mid-sized existing codebase. The total token consumption across the cycle reached approximately 3 billion, with around 110,000 lines of code generated and modified. During this period, the two models were integrated into daily development through a premium coding subscription and a fixed weekly quota respectively, covering the full pipeline from solution design and implementation to defect localization and end-to-end test closure.

The core objective was not to compare benchmark scores, but to establish a reusable engineering evaluation framework that answers the following questions:

• How consistent are LLM-generated solutions with the initial design in real-world projects?
• Can coding accuracy and runtime stability meet the baseline required for continuous integration?
• During defect diagnosis, do models stop at surface-level fixes or reach design-level root causes?
• Can the end-to-end test loop converge autonomously without human supervision?
• Is token consumption sustainable for long, iterative trial-and-error workflows?

To avoid bias from model names, the two systems are referred to as Model A and Model B throughout this article. The focus is on deriving a generalizable evaluation methodology and observations rather than recommending or comparing specific products.

2. Test Conditions and Resource Model

2.1 Engineering Environment

The target codebase is a mid-complexity backend system with the following characteristics:

• Primary languages: Java and TypeScript
• Modules: 14 core modules
• Existing codebase size: approximately 180,000 lines
• Build tooling: common build systems and dependency management tools
• Test infrastructure: unit tests, integration tests, and end-to-end smoke tests

Development tasks included adding new business modules, refactoring existing interfaces, adjusting cross-module dependencies, and diagnosing runtime defects. Tasks were assigned alternately so that both models encountered requirements of similar difficulty, reducing sample bias.

2.2 Resource Investment

Item	Model A	Model B
Subscription	Premium coding subscription	Fixed weekly quota
Total consumption	Approximately 3 billion tokens	Weekly quota fully exhausted for four consecutive weeks
Involvement	Design, coding, diagnosis, test repair	Design, coding, diagnosis, test repair

It is worth noting that token consumption and lines of code do not scale linearly. Models produce substantial non-code output during iterative trial and error, context replay, and design discussions. Therefore, the 110,000 lines of code associated with 3 billion tokens should not be interpreted simply as generation efficiency. The figure also includes extensive solution reasoning, defect localization dialogues, and test log analysis.

3. Evaluation Dimensions

To convert subjective experience into quantifiable observations, we defined five primary dimensions with corresponding evaluation criteria. This framework can also serve as a reference for other engineering teams introducing LLM-based coding capabilities.

Dimension	Sub-dimension	Evaluation Criteria
A	Solution design and code structure	Consistency between directories, files, path naming, and the initial design
B	Coding accuracy	Compilation/startup success rate and runtime crash frequency
C	Bug localization and repair	Whether a single conversation yields the root cause and minimal change; ability to detect deep design flaws
D	Autonomous end-to-end test closure	Whether the log analysis, code change, and retest loop can complete without human supervision
E	Token/quota sustainability	Whether the model supports prolonged iterative trial and error

3.1 Dimension Definitions

Solution design and code structure focuses on whether the model can produce a clear, executable design that remains consistent with the final code before implementation begins. In practice, many problems do not originate in the code itself, but in design-stage misjudgments about existing directories, naming conventions, and module boundaries.

Coding accuracy is a relatively surface-level indicator, but it directly affects development efficiency. If model-generated code frequently fails to compile or start, all downstream activities are disrupted.

Bug localization and repair measures the depth of attribution and the repair strategy after a defect appears. Surface-level fixes can stop bleeding quickly, but cannot resolve systemic design flaws.

Autonomous end-to-end test closure is the most discriminating dimension in this evaluation. It requires the model to read test logs, determine the cause of failure, independently modify code, and trigger retesting until the loop converges or the model explicitly reports that it cannot proceed.

Token/quota sustainability examines from a resource-economics perspective whether the model is suitable for long-term, high-volume engineering practice. A model that repairs precisely but consumes enormous resources may be unsustainable under real budget constraints.

4. Results by Dimension

4.1 Solution Design and Coding Phase

Common Strengths

Both models performed at a similar level in the following areas:

• Understanding the existing codebase: both quickly established a basic understanding of module structure, core call chains, and design patterns. The probability of proposing reasonable new file paths in the first conversation was high, indicating a structured grasp of directory organization.
• Markdown solution output: both produced well-formatted design documents listing new files, modified files, and related dependencies. The readability and completeness of these documents met the basic requirements for team review.
• Naming consistency: directory structures and file paths matched the initial design expectations more than 80% of the time. This suggests that both models are reasonably stable for adding or adjusting standard business modules.

Consistency Issues

Both models exhibited a certain level of redundant design or over-abstraction during the initial solution phase. Specific manifestations included:

• Introducing unnecessary interface-implementation separation for simple CRUD modules.
• Presetting multiple strategy patterns or factory structures when there was no clear extensibility requirement.
• Generating generic utility classes unrelated to the business, adding noise to the codebase.

These issues were related to an open-ended specification plugin we had enabled at the time. The plugin required models to output more complete design documents before coding, which unexpectedly induced models to generate a large volume of ceremonial abstraction layers with limited engineering value. After removing the plugin, the proportion of redundant code dropped significantly. This suggests that the constraints imposed by the toolchain can significantly influence a model’s design tendencies, and model capability should be evaluated with minimal toolchain interference.

Differences

At the low-level implementation detail level, both models produced a certain rate of errors that required multiple rounds of localized fixes to reach a compilable state. These errors typically involved:

• Incorrect cross-module dependency references.
• Mismatched constructor parameter order or types.
• Async calls that failed to handle return values or exceptions properly.
• Inconsistent data model field mappings.

From the perspective of the final repair rate to a compilable state, the difference between the two models was not significant. They can be classified in the same capability tier. The real divergence emerged in the subsequent defect diagnosis and test closure stages.

4.2 Defect Diagnosis and Repair

This was the dimension with the most pronounced differences. We recorded and compared the two models across four indicators.

Indicator	Model A	Model B
Diagnostic depth	Tended to identify design-level root causes in one pass	Mostly stayed at surface-level code errors
Interaction rounds	Averaged around 1.x rounds to convergence	Often required 3–5 rounds of guidance
Solution completeness	Offered multiple options for confirmation	Tended to modify code immediately, requiring mid-course interception
Probability of patchwork fixes	Low	High, often triggering new problems in a chain reaction

Differences in Diagnostic Depth

Model A exhibited a diagnostic approach closer to that of a senior engineer. When faced with a cross-module data consistency issue, Model A tended to reason from the data flow design and state management strategy, pointing out that the problem was not a localized logic error, but a lack of clear data ownership conventions between modules. This kind of diagnosis directly led us to revisit module boundaries rather than simply modifying a conditional check.

Model B stayed more at the surface-code level. For the same type of problem, Model B would locate the variable assignment order in a method or a missing null check in a class, and provide the corresponding code fix. These repairs were effective locally, but often failed to prevent the same problem from recurring in other modules.

Differences in Interaction Rounds and Strategy

In terms of interaction rounds, Model A averaged around 1.x rounds to deliver an acceptable solution. Model B typically required three to five rounds of guidance before moving from surface-level fixes to deeper design adjustments.

Another notable difference lay in how each model approached the solution process. Model A tended to present multiple options first, briefly explaining the pros, cons, and impact scope of each, and waited for developer confirmation before making changes. This align-first, execute-later approach is safer in collaborative environments or high-risk modules.

Model B leaned toward modifying code directly, sometimes completing changes before the developer had fully assessed the impact. This approach appears efficient, but in complex projects it can lead to patchwork fixes and cascading failures. During testing, we repeatedly had to stop Model B mid-course and re-evaluate the change scope. This process effectively offset the time advantage of its rapid response.

The Cost of Patchwork Fixes

When Model B applied patchwork fixes, it often triggered new problems in a chain reaction. For example, to fix an issue where an API returned a null value, Model B might adjust null handling on the caller side, serialization configuration, and frontend rendering logic all at once. This expanded modification might eventually resolve the current defect, but it also introduced new edge-case issues, causing the number of problems during the testing phase to increase rather than decrease.

Model A showed a significantly lower probability of patchwork fixes. Even when its initial judgment was off, it usually provided a clear impact scope after making changes, facilitating subsequent verification and rollback.

Summary: In defect diagnosis and repair, Model A showed clear advantages in deep root-cause attribution, controlled repair strategies, and communication style.

4.3 Autonomous End-to-End Test Closure

End-to-end test closure is the dimension that best reflects a model’s engineering viability. The core criterion is: given a failed test log, can the model autonomously complete the full cycle of log analysis, localization, code change, and retest triggering, and ultimately converge the test case?

Model A Performance

Model A exhibited near-semi-autonomous engineering behavior in this dimension. After multiple integration test failures, Model A was able to:

1. Read failure logs and identify key stack traces.
2. Locate the specific module and suspicious code locations.
3. Propose a minimally invasive fix.
4. Trigger a retest and wait for results.
5. Continue to the next round of analysis if the failure persisted.
6. After two or more consecutive non-convergent rounds, proactively pause and report blockers and hypotheses to the developer.

This closed-loop capability significantly reduced the need for human intervention. This was especially valuable during overnight batch integration test runs, where Model A could keep progressing through rounds of failure and repair until the problem converged or it explicitly reported a blocking point.

Model B Performance

Model B showed two typical problems in test closure:

First, its analysis of failure logs was not sufficiently focused. When faced with a long stack trace, Model B sometimes prioritized the exception message at the end of the log while ignoring the upstream root cause that actually triggered the failure. This led to misdirected fixes and new failures.

Second, it lacked autonomous convergence awareness. If a test still failed after one modification, Model B generally did not proactively enter a deeper re-analysis process. Instead, it tended to modify more related code. This fix-by-editing strategy occasionally resolved the issue by chance, but overall closure efficiency was low and it introduced uncontrolled changes.

Observed Cases

We recorded several representative cases:

• Case 1: Cross-module RPC call timeout. Model A identified through logs that the timeout parameter configuration conflicted with the retry strategy in the call chain, proposed a configuration adjustment, and achieved closure in one round. Model B modified the caller-side timeout value, which passed locally, but did not notice that the server-side retry wait window was mismatched, causing the failure to recur under higher concurrency.
• Case 2: Memory overflow in a scheduled task. Model A identified from stack traces and GC logs that the cache retained a large number of temporary objects, and proposed a design-level fix to adjust the caching strategy. Model B added an explicit reclamation call at the end of task execution, which temporarily alleviated the issue but did not fundamentally address the memory growth trend.

Summary: Model A demonstrated stronger autonomy and convergence awareness in end-to-end test closure, while Model B showed insufficient stability in analyzing long-chain failures.

4.4 Token/Quota Sustainability

Over four weeks of intensive use, Model A’s premium coding subscription consumed approximately 3 billion tokens, while Model B’s weekly quota was regularly exhausted. Given the differences in interaction rounds and misrepair rates, the consumption structures differed significantly.

Differences in Consumption Structure

Although Model A’s total token consumption was high, it was primarily spent on solution alignment, design discussions, and one-time deep root-cause analysis. The repair actions themselves tended to be restrained, so the average token consumption per defect repair was relatively manageable.

Model B’s consumption was disproportionately concentrated in the defect repair and test closure stages. The reasons include:

• Multiple interaction rounds causing repeated context replay.
• Frequent incorrect modifications and rollbacks consuming large numbers of tokens.
• Shallow post-failure analysis often requiring additional human-supplied information, increasing dialogue rounds.

Sustainability of Iterative Trial and Error

From the perspective of supporting prolonged iterative trial and error, Model A’s token consumption, while large, produced reusable design conclusions for downstream stages. Its conversation history generated significant reuse value in subsequent similar problems, reducing marginal consumption.

Model B’s quota was also fully used, but its consumption pattern leaned toward trial-and-error patching. This type of consumption has low reuse value because many conversations ultimately led to invalid modifications or rollbacks and could not be consolidated into inheritable engineering knowledge.

5. Engineering Selection and Usage Recommendations

Based on the results above, we do not attempt to draw an absolute conclusion about which model is better. Instead, we offer engineering selection and usage strategies based on different priorities.

5.1 Matching Models to Task Types

Scenarios suited to Model A

• Cross-module defect localization and design-level troubleshooting.
• Test repair workflows requiring end-to-end autonomous closure.
• High-risk code changes where aligning on the approach before execution is important.
• Long-term projects where preserving and reusing design conclusions matters.

Scenarios suited to Model B

• Code generation and routine implementation within a single module.
• Well-structured CRUD requirements with clear objectives.
• Short-cycle tasks with distributed budgets.
• Scenarios requiring rapid code drafts for subsequent human review and refactoring.

5.2 Toolchain and Process Optimization

One important lesson from this evaluation came from the impact of the toolchain. After removing the open-ended specification plugin, redundant design decreased significantly for both models. This is a reminder that model capability should be evaluated with a simple toolchain and process, avoiding extra constraints that introduce biases unrelated to the model itself.

For teams requiring end-to-end automated closure, we recommend the following mechanisms:

• Structured collection of build and test logs.
• Auxiliary markers for rapid root-cause localization of failures.
• Change-scope constraints for autonomous repairs.
• Automatic pause and human handover after two consecutive non-convergent rounds.

These mechanisms do not depend on any specific model. Instead, they amplify model strengths through process design while mitigating the risk of uncontrolled modifications.

6. Overall Assessment

This month-long evaluation gave us a more concrete understanding of the capability boundaries of LLMs in real-world engineering.

In the solution design and routine coding phases, the commonalities between the two models outweighed the differences. Both reached a usable level for development assistance. The differences mainly appeared in diagnostic depth, restraint in interaction strategy, and autonomy in end-to-end test closure.

The more valuable takeaway was the evaluation framework itself. Five dimensions—solution consistency, coding accuracy, defect attribution, test closure, and token sustainability—form a reusable observation structure. Any team can use this framework, combined with its own engineering context and budget constraints, to systematically validate coding LLMs without relying on scattered personal experience or benchmark scores alone.

Ultimately, we did not treat the models as fully autonomous code generators that could replace developers. Instead, we treated them as engineering collaborators with specific behavioral patterns. Understanding a model’s tendencies between deep attribution and surface-level patching, and its balance between autonomous closure and uncontrolled modification, is the key to determining whether it can deliver long-term value in real-world engineering.