Long-Context Large Models in Practice: Engineering Code Delivery and Long-Horizon AI Agents
As large models shift from answering questions to completing tasks, long-context capability has become a decisive factor in real deployments. Using the public technical disclosures of a next-generation model as a sample, this article examines how million-token context windows translate into engineering value across four dimensions: the technical foundations of context windows and why effective utilization degrades; code capabilities ranging from single-function generation to engineering-grade delivery; the architecture of long-horizon agents that turn one-off Q&A into resumable workflows; and layered strategies for factual accuracy, safety governance, and cost control. It closes with practical guidance on selecting models by task success rate, prioritizing context governance, treating verification as a system capability, designing costs upfront, and building layered applications on Baidu AI Cloud.
Introduction: Long Context Is a Starting Point, Not a Finish Line
In October 2026, a Chinese large-model vendor released a new flagship model. It uses a roughly 3-trillion-parameter scale, supports a 1M-token context window, and reaches or approaches the level of top international closed-source models across several code, agent, and knowledge benchmarks. It subsequently ranked first among Chinese models in an intelligence-index leaderboard published by a third-party evaluation organization.
The focus of industry competition has shifted from “how large is the parameter count” to “how much can it take in at once, and what can it actually accomplish after taking it in.” But “supporting a million tokens” only describes the upper limit of capability. In real business scenarios, developers care about four questions:
- Can ultra-long context be utilized stably, rather than existing only as a stated window size?
- Can the model stay correct across multi-step code tasks and long-horizon agent tasks?
- Are factual errors and safety risks controllable?
- How should the cost differences introduced by high-throughput inference and caching be accounted for?
Using the public technical disclosures of this class of model as an analysis sample, this article systematically explores the deployment path of long-context large models across four levels: technical principles, evaluation methodology, engineering practice, and cost governance.
1. Technical Principles: Where the Context Window Comes From and Why It Degrades
1.1 The Attention Mechanism and Complexity Constraints
Transformer models use a self-attention mechanism to model the relationships between tokens, where every token must compute relevance against every other token. The computation of standard full attention grows quadratically with sequence length, and the attention cache in memory grows linearly with length. Expanding the context window therefore relies primarily on three techniques:
- Sparse or block attention: reduces the number of token pairs that actually participate in computation;
- Position-encoding extrapolation and interpolation: lets the model retain positional discrimination when operating beyond its training length;
- Parallel training and memory optimization: tensor parallelism, sequence parallelism, and attention recomputation together support training on ultra-long sequences.
Window size thus becomes a system-design parameter, but what determines effectiveness is “long-context utilization.”
1.2 Why Long-Context Utilization Degrades
Even when a model claims to support an ultra-long window, performance degradation is common in practice, mainly for these reasons:
- Middle-section information loss: research consistently finds that models make better use of information at the beginning and end, while key information in the middle is easily overlooked;
- Context corruption: as the number of task turns grows, irrelevant details accumulate and interfere with the model’s judgment of the current goal;
- Attention dilution: a large amount of highly similar, repetitive content diffuses attention weights, drowning out critical instructions.
This explains a common phenomenon: multiplying the window tenfold does not necessarily improve task success rate proportionally.
1.3 Three Lessons for Engineering Practice
- Structure the context: place the most critical information at the beginning and end, and keep only distilled intermediate conclusions in the middle region;
- Actively compress the context: in long-horizon tasks, periodically summarize, archive, and prune rather than stuffing the entire history in verbatim;
- Validate with length-tiered evaluation: do not only check whether the model can hold long text; test key-information retrieval and multi-hop reasoning across different length tiers.
2. Code Capability: From Code Completion to Engineering-Grade Delivery
2.1 Breaking Down the Evaluation Dimensions
Code capability spans at least five levels:
- Single-function generation: assessing syntactic correctness and mastery of common algorithmic patterns;
- Repository-level understanding: maintaining logical consistency across cross-file, cross-module symbol references and dependencies;
- Terminal toolchain operation: compiling, running, and testing in a real command-line environment and iterating based on errors;
- Long-horizon software engineering tasks: the complete loop from requirement decomposition and implementation to self-verification;
- Specialized-domain script generation: game-engine scripts, parametric 3D-modeling scripts, and so on.
The last three levels are precisely where long-context models hold an advantage. A single long-horizon task can span dozens of files and involve multiple rounds of “execute — read the error — fix” loops, placing extremely high demands on context retention.
2.2 Terminal-Restricted Tasks: A Litmus Test in Realistic Engineering Environments
In a public evaluation of realistic terminal-restricted tasks, a new-generation long-context model achieved a 65.7% pass rate, exceeding a top closed-source model’s 58.2% and trailing the then-highest-scoring model (66.4%) by only 0.7 percentage points.
The constraint of these tasks is that the model can rely only on terminal output for feedback; it cannot “imagine” compilation results or runtime behavior and must localize problems step by step from real errors. Its analysis should grasp three points:
- It measures system-level capability: the model, the runtime environment, and tool invocation are all indispensable;
- Feedback quality determines convergence speed: the clearer the logs and the more complete the error messages, the faster the model self-corrects;
- The gap between models is narrowing: the top tier clusters within a few percentage points, so selection should not rely on the aggregate score alone but also examine the distribution across specific tasks.
2.3 Graphics and 3D Scripts: Highly Structured Scenarios
With tools such as game-engine scripts and parametric 3D-modeling scripts, a new-generation model scored 92.7 on a computer-aided-design task benchmark, close to the 96.2 and 95.9 of top closed-source models.
These scenarios impose a dual demand on long context:
- Reading multiple authoritative documents: the model must simultaneously grasp the scripting APIs of the engine or modeling software, coordinate-system rules, and unit conventions;
- Maintaining parameter consistency: after changing one dimensional parameter, associated features, assembly relationships, and constraints must be updated in sync.
In practice, the pattern of “parameter table + constraint rules + generated script” works well. Making design intent explicit as structured data, letting the model generate parametric scripts, and then having the software automatically rebuild and validate unifies “generating correctly” with “modifying in a controllable way.”
2.4 Long-Horizon Code Tasks: Internal Benchmarks and Engineering Practice
A new-generation model scored 61.2% on an internal long-horizon code evaluation (Kimi Code Bench 3.0 Long-horizon), an improvement of about 5.25 percentage points over the previous generation’s 55.95%. Because this benchmark is vendor-built, its interpretation warrants care:
- Limited comparability: it serves as an internal reference for generational comparison within the same model family and should not be directly compared against public leaderboards on other platforms;
- What the gain indicates: it shows that improvements in long-context and agent capability are primarily converted into better stability on long-cycle tasks.
For engineering practice, a three-stage flow of “retrieval to narrow the scope + model generation + automatic verification” is recommended:
- Retrieval stage: use symbol indexes and call graphs to locate relevant files, sending only the necessary context to the model;
- Generation stage: have the model output code that compiles and runs directly, rather than pseudocode or fragments;
- Verification stage: automatically run unit tests, static checks, and type checks, feeding failure messages back as context for the next round.
2.5 Staged Attribution of Long-Task Failures
Failures in long-horizon tasks are often not a single-point error but errors accumulating along the task chain. Attributing failures by stage makes optimization more targeted:
| Failure stage | Typical symptom | Optimization direction |
|---|---|---|
| Requirement understanding | Misunderstanding, doing the wrong thing | Task-spec templates, requirement-clarification steps |
| Context organization | Missing or stale key files | Retrieval quality, summary freshness, version consistency |
| Tool invocation | Wrong arguments, retry timeouts | Clearer tool descriptions, idempotent design, timeout control |
| Self-verification | Passing tests but flawed logic | More boundary cases, human spot checks, regression tests |
| State management | Unable to resume after interruption | Checkpoint persistence, failure-recovery mechanisms |
3. Long-Horizon Agents: Turning One-Off Q&A into Resumable Workflows
3.1 Capability Differences Concentrate in Long-Horizon Tasks
The core value of long-context models stands out most in long-horizon agent scenarios, with finance being a typical representative. A new-generation model can deliver workflows such as financial modeling, investment analysis, and industry research reports end to end, scoring 64.4 on an external office-document evaluation (OfficeQA Pro v2), which the vendor calls its current best result.
The characteristics of these tasks make long-horizon capability necessary:
- Multi-step dependencies: data cleaning, metric calculation, assumption setting, sensitivity analysis, and conclusion writing are tightly interlinked, so earlier errors amplify layer by layer;
- Large and heterogeneous documents: annual reports, announcements, market data, and research reports come in varying formats and must be cross-compared within an ultra-long context;
- Results must be reviewable: every key figure should be traceable to its original source.
3.2 Architectural Essentials of a Long-Horizon Agent
A robust long-horizon agent typically comprises five modules:
- Planner: decomposes the goal into verifiable subtasks and maintains task state;
- Context manager: handles layered storage, summary compression, retrieval recall, and budget control;
- Tool layer: executes model inference, file read/write, code execution, and data queries;
- Verifier: performs rule-based validation and consistency checks on intermediate outputs;
- Execution and scheduling layer: handles concurrency control, timeout retries, rate limiting, and cost control.
Illustrative code follows (it conveys the architectural idea only):
MAX_CONTEXT = 1_000_000OUTPUT_RESERVE = 20_000 # Reserve space for outputCOMPACT_RATIO = 0.7 # Trigger compaction when context usage exceeds 70%def run_long_horizon_task(task):session = ContextSession(messages=[system_prompt(), task],token_budget=MAX_CONTEXT - OUTPUT_RESERVE,)plan = Planner().decompose(task) # Produce a list of verifiable subtaskswhile plan.has_pending():# When the context budget nears its limit, compact older turns,# keeping key decisions and tool resultsif session.used_ratio() > COMPACT_RATIO:session.compact(keep=("conclusions", "key_data", "pending_items"))action = model.decide(session.render()) # Decide: call a tool or submit resultsresult = tools.execute(action)if not verify(result):result = tools.retry(action, feedback=verify(result).reason)session.append(action, result)plan.update(result)checkpoint.save(session, plan) # Persist a checkpoint to support resuming after interruptionreturn plan.report()
3.3 A Budget Model for Layered Context
It is advisable to divide the context into a fixed layer, a working layer, and a history layer, with an explicit budget and update strategy for each:
| Layer | Content | Update strategy | Budget reference |
|---|---|---|---|
| Fixed layer | System prompt, task spec, output-format constraints | Always present and unchanged; prioritize cache hits | 5–10K |
| Working layer | Files, retrieved snippets, and tool results needed for the current subtask | Updated dynamically per step | 50–200K |
| History layer | Summaries and decision records of completed steps | Periodically compressed and archived | 20–100K |
| Output reserve | Code or report body generated in the current round | Does not consume the input budget | ≥20K |
Specific values should be tuned against the measured task success rate. The principles are: keep the fixed layer as stable as possible to maximize cache hits; keep only the information necessary for the current step in the working layer; and let the history layer exist as conclusions rather than raw text.
4. Factual Accuracy: Layered Governance of the Hallucination Problem
4.1 Reading the Metrics
On an internal high-difficulty hallucination stress test (Kimi Hallu Bench Hard), a new-generation model reduced its single-fact statement error rate from the previous generation’s 23.4% to 8.3%. The proportion of responses containing at least one factual error dropped to about 42.4% of the previous generation (a reduction of roughly 57.6%). The vendor also emphasized that the test deliberately focuses on error-prone cases and that the error rate does not represent overall performance in everyday use.
This points to two common ways of misreading metrics:
- A drop in proportion does not mean errors have disappeared: an 8.3% error rate is still high-risk in finance, healthcare, and legal scenarios;
- A specialized benchmark cannot replace the real distribution: evaluation sets should be built from business data and updated regularly.
4.2 Three Lines of Defense
Line one: Retrieval-Augmented Generation (RAG)
RAG turns “recall-style answering” into “evidence-based answering” and is one of the most effective means of controlling factual errors. Keep in mind during deployment:
- Document chunking: split at semantic boundaries while preserving metadata such as heading hierarchy;
- Hybrid retrieval: combine vector retrieval with keyword retrieval to improve recall of long-tail facts;
- Reranking and citation: rerank retrieved results and require the model to cite its sources item by item.
Line two: Output constraints and cross-validation
Apply structured output (such as JSON Schema) and rule-based validation to key fields, and have numeric conclusions recomputed by a program and compared against the model’s output. In long-context tasks, add global consistency checks as well — for example, whether different sections of a research report describe the same metric consistently.
Line three: Explicit expression of uncertainty
State clearly in the system prompt that when evidence is insufficient, the model should answer “no basis found” and list the items to be verified, rather than giving a speculative conclusion.
An illustrative prompt fragment:
Role: A rigorous research and analysis assistantRules:1. Every factual statement must cite a source number corresponding to a retrieved snippet;2. When evidence is insufficient, output "Insufficient evidence" and list the material that needs to be supplied;3. All numbers must be checked against the original documents one by one; do not estimate and present directly;Output format: {"conclusion": "...", "evidence": ["[3]", "[5]"], "to_verify": ["..."]}
5. Safety Governance: The Capability Cost of Removing Offensive/Defensive Data
A new-generation model identified and removed network attack-and-defense-related data during both the mid-training and reinforcement-learning stages to reduce misuse risk. The cost is quantified as well: on a self-built attack-and-defense evaluation set (xdev-bench-hard), the unfiltered version solved 90.5% of problems at pass@3, while the final released version reached 81.0%.
This set of data reveals a general trade-off pattern:
- What pass@k means: the probability that at least one of k samples of the same problem passes, reflecting the upper bound of the model’s capability. A 9.5-percentage-point drop in pass@3 indicates that removing high-risk data did weaken some high-risk capabilities;
- An upper bound is not a lower bound: some capability loss can be offset through retries and multi-sampling, but not all losses can be recovered through engineering means;
- Tailor the approach by scenario: when enterprises need offensive/defensive capabilities for security research, they should evaluate, approve, and use them in an isolated environment, rather than expecting a general-purpose model to unconditionally meet such needs.
When enterprises build or fine-tune their own models, they should likewise establish mechanisms for data-source review, training-data deduplication, and harmful-content filtering, and incorporate safety evaluation into the release process.
6. Cost and Performance: The Billing Challenge of Long Context
6.1 Why Long Context Costs More
Long-context costs come mainly from three sources:
- Input-token inflation: system prompts, conversation history, tool outputs, and retrieved snippets stack up, so a single request can easily reach tens of thousands to hundreds of thousands of tokens;
- Memory usage and computation: the attention cache grows with length and inference throughput drops;
- Higher output costs: the per-token output price is usually significantly higher than the input price, making long report generation especially costly.
6.2 The Billing Logic of Caching and the High-Throughput Tier
Using the publicly listed pricing of a new-generation model as an example (CNY per million tokens):
| Model | Input | Output | Cache hit | Cache write (5 min) | Cache write (1 h) |
|---|---|---|---|---|---|
| Standard | 13.00 | 68.00 | 1.30 | 13.00 | 26.00 |
| High-throughput | 26.00 | 136.00 | 1.30 | 26.00 | 52.00 |
The design follows three rules:
- The cache-hit price is the lowest: only one-tenth of the input price;
- Cache writes are priced by validity period: a write is charged at the input price, and a cache with a one-hour validity costs double, making it suitable for slowly changing, long-lived contexts;
- The high-throughput tier is roughly twice the standard tier: a 2x output price means long-output tasks carry a more pronounced premium.
6.3 Cost Estimation Method
Cost per request ≈uncached input tokens × input unit price+ cache-hit tokens × cache-hit unit price+ cache-write tokens × cache-write unit price+ output tokens × output unit price
Assume a single request has 100K input tokens, of which 90K hit the cache and 10K are newly written (5-minute validity), with 5,000 output tokens:
- Standard:
- Uncached input: 0;
- Cache hit: 0.09 × 1.30 = 0.117;
- Cache write: 0.01 × 13.00 = 0.13;
- Output: 0.005 × 68.00 = 0.34;
- Total ≈ 0.59 CNY.
- High-throughput:
- 0.09 × 1.30 + 0.01 × 26.00 + 0.005 × 136.00 = 0.117 + 0.26 + 0.68 ≈ 1.06 CNY.
- Comparison: without caching, the standard tier costs 0.10 × 13.00 + 0.34 = 1.64 CNY. With cache hits, the total cost drops by about 64%.
Whether the cache can be hit depends on whether the shared prefix across requests is exactly identical. Frequently changing the system prompt, or writing timestamps or random IDs into the prompt, all cause the cache to miss.
6.4 Five Optimization Strategies
- Stable prefix: place the system prompt, tool definitions, and fixed knowledge at the very front and keep them byte-for-byte identical to maximize cache hits;
- Layered use of long context: use one-hour caching for fixed long documents and organize dynamic content separately to avoid mutual contamination;
- Output-budget control: cap the maximum output tokens and split long reports into multiple verifiable sections;
- Tiered routing: route simple subtasks to lightweight models and reserve the flagship model for complex steps only;
- Continuous observability: build a token-consumption dashboard, attribute usage by business, scenario, and user dimensions, and promptly locate anomalous calls.
7. Architecture Recommendations: Building Long-Context Applications on Baidu AI Cloud
A long-context application is quintessential systems engineering; a single model cannot independently support production-grade needs. On Baidu AI Cloud, you can build it in layers by responsibility:
- Model service layer: use the Qianfan model platform to access models of different sizes under a unified interface, supporting routing by business scenario and multi-model failover;
- Data storage layer: use object storage to hold raw files of long documents and knowledge bases, and a cache service to hold session state and checkpoints;
- Retrieval service layer: combine vector databases with full-text search to enable hybrid retrieval and reranking;
- Runtime and scheduling layer: deploy the agent runtime on container services, and use function compute to handle tool calls and asynchronous tasks;
- Security and governance layer: use log services, monitoring and alerting, and content-safety capabilities to observe the full chain of calls, token consumption, and anomalous outputs.
The benefit of a layered architecture is that when model capability is upgraded, the application layer only needs to adjust its invocation configuration rather than rework the business logic.
8. Deployment Recommendations
- Select by task success rate, not window size: build an evaluation set from real business tasks and test separately at different length tiers such as 128K, 256K, 512K, and 1M.
- Prioritize solving context-governance problems: before upgrading the model, first check whether retrieval quality, context compression, and budget allocation are reasonable — many “the model isn’t strong enough” problems are actually “the context isn’t provided well.”
- Make verification a system capability: a model cannot guarantee correctness on the first try, so build a quality loop from automated testing, rule-based validation, and human spot checks.
- Design costs upfront: complete the token-budget and caching-strategy design in the solution-design phase to avoid reactive optimization after launch.
- Clarify safety boundaries: define which capabilities are permitted and which require an isolated environment, and bind evaluation results to usage policies.
Conclusion
The technical progress represented by new-generation long-context models marks a shift for large models from “answering questions” to “completing tasks.” A 1M-token window and a roughly 3-trillion-parameter scale are the foundation, but true competitiveness comes from the systematic command of context, tools, verification, and cost. For developers, only by understanding the technical boundaries, evaluation criteria, and billing logic of long context — and by building a layered architecture on complete cloud infrastructure — can model capability be reliably converted into business value.
