How Large Language Models and Coding Agents Work Together: A Practical Evaluation of Code Generation Performance

Introduction

In the current wave of AI-assisted development, large language model capabilities continue to advance rapidly. At the same time, intelligent coding tools—often referred to as harnesses or coding agents—have become increasingly important as the bridge between models and local development environments. However, larger model parameters or improved theoretical capabilities do not always translate directly into higher productivity in real-world development tasks.

To investigate how different model–tool combinations perform in complex, realistic scenarios, we designed a stress test centered on pure code-generated video output.

1. Test Background and Design

The core task was to use a prompt to generate a 3D simulator teardown video entirely through code. The prompt introduced a specific “skill” concept: a code module capable of converting any machine object into an interactive 3D simulator.

To objectively evaluate the capabilities of each combination, every participant was given only one execution attempt with no human intervention.

The evaluation covered three dimensions:

1. Subjective output quality: video smoothness, fidelity to the prompt intent, and completeness of the code framework.
2. Objective resource consumption: total execution time and total token processing volume, with particular attention to cache hit rate and output token size.
3. Tool scheduling strategy: behavioral differences among coding tools in task parsing, skill module invocation, and multi-turn interaction with the model.

The evaluated solutions included five common collaboration patterns representing different technical approaches to combining intelligent coding tools with large language models.

2. Objective Performance Data: Time Versus Token Consumption

In complex code generation tasks, a tool’s scheduling strategy directly determines time cost and compute consumption. The measured results across the five solutions showed striking differences:

• Fast execution type: One combination completed the task in only 8 minutes. Its total token processing volume was approximately 3.5 million, with cache accounting for 97% and final output reaching 28,000 tokens. This indicates a highly parallel scheduling strategy that maximizes context cache reuse to reduce latency.
• High-quality reasoning type: Another combination in standard mode took 46 minutes, with total token processing surging to 46.44 million (99% cache) and output reaching 281,000 tokens. This heavy resource investment suggests deep multi-turn reflection and iterative code refinement.
• Balanced type: The remaining three solutions took approximately 17 minutes, 21 minutes, and 36 minutes respectively, with token consumption between 12 million and 14 million and output tokens ranging from 50,000 to 130,000.

Key insight:
In terms of token consumption and cache ratio, all mature tools achieved cache hit rates above 96%, demonstrating that modern coding tools generally have strong context management capabilities and can effectively avoid redundant prompt processing. However, the fastest and slowest combinations differed by nearly 6x in time and 13x in token consumption. This gap is not primarily caused by insufficient model compute but rather by different thresholds for “reasoning depth” set at the tool layer.

3. Output Quality and Technical Characteristics

Objective metrics only reflect engineering efficiency. The actual quality of generated code and video is what ultimately determines whether a technical approach is viable in practice.

3.1 The Double-Edged Sword of Deep Reasoning Mode

The combination that took 46 minutes exhibited clear signs of “overthinking.” In standard mode, the tool tended to repeatedly analyze and self-review every code module. While this mechanism offers advantages when troubleshooting complex logic defects, it generated substantial redundant output (281,000 output tokens) in a goal-oriented code generation task.

Additionally, this solution showed semantic drift when handling multilingual prompts. Despite the input being provided in Chinese, the generated video content did not accurately match the specific requirements of the Chinese-language context, revealing weaknesses in cross-language instruction following.

3.2 The Trade-Off Between Speed and Instruction Adherence

The 8-minute combination demonstrated a fundamentally different technical profile. This model–tool pairing excelled at direct execution: after receiving instructions, it quickly abandoned the complex skill framework referenced in the prompt and generated the final video through a more basic code path.

While this approach achieved extremely fast completion and likely incurred minimal API costs, it sacrificed invocation of the advanced skill module. The resulting code showed notable deficiencies in scalability and engineering elegance.

3.3 The Best Balance of Quality and Engineering Capability

In this evaluation, one combination that completed the task in 17 minutes with 130,000 output tokens demonstrated clear championship-level performance. It produced high-quality video output, accurately addressed the abstract problem described in the prompt, and properly handled both the physics engine logic required by the 3D teardown simulator and the skill framework invocation.

This indicates that the underlying model possesses strong code intent understanding, and the accompanying coding tool achieved an excellent balance in the “planning–generation–verification” workflow.

4. Scheduling Strategies and Best Practices for Intelligent Coding Tools

From this comparative evaluation, several key technical considerations emerge for AI-assisted development systems:

4.1 Coordinating Context Caching and Log Monitoring

Modern coding tools routinely generate millions of tokens in contextual interactions. Without efficient context caching mechanisms, API latency and costs would quickly become unacceptable.

In real-world engineering deployments, organizations should combine robust logging services with monitoring and alerting to audit token consumption paths. By analyzing cache hit logs, developers can optimize prompt structures and front-load frequently reused context, leveraging general-purpose caching acceleration capabilities offered by cloud providers to significantly reduce inference costs.

4.2 Matching Model Capability with Tool Mode

A model’s parameter scale determines its knowledge floor, but the coding tool’s operating mode determines its practical ceiling. For example, whether a tool provides both “minimal mode” and “standard mode” directly affects how problems are solved.

When facing rule-based, repetitive code generation tasks, developers should use a low-reasoning-threshold fast mode. For architecture design or complex algorithm refactoring, a standard mode with deep reflection capabilities is more appropriate.

Developers should not blindly pursue larger parameter counts in a single model, but should instead focus on how well a model integrates with the broader tooling ecosystem.

4.3 Engineering Standards for Skill-Based Invocation

The evaluation introduced the concept of “skills” modules, which represents a practical engineering extension of function calling technology. A well-designed coding tool should support dynamic loading of external skill packages—for example, installing a specific code simulator through command-line tools—and strictly follow the skill-defined framework during execution, rather than abandoning the framework for shortcuts like the fast-execution solution did.

This requires developers designing agent-based tools to establish rigorous validation mechanisms for skill invocation.

5. Conclusion

The capability boundaries of large language models continue to expand, but discussing model parameter superiority in isolation has lost much of its practical significance. In real code generation and engineering practice, three factors collectively form the foundation of AI development productivity:

• The scheduling strategy of the coding tool
• Context cache management capability
• Integration with external skill modules

When selecting technical solutions, developers should not be swayed solely by theoretical benchmark scores. Instead, they should evaluate their specific business requirements—whether they are more sensitive to execution time or output quality—and find the optimal balance between deep reasoning and fast execution.