Choosing a Million-Token Context LLM: Evaluating Coding and Agent Capabilities
Long-context open models with roughly 1-million-token windows are now a baseline, so the real differentiator is cost-effectiveness rather than raw capability. This article explains how to choose among capability-first and efficiency-first open models for coding and agent workloads. It covers the core concepts (total vs. activated MoE parameters, why a context window is not the same as effective context, and the "lost in the middle" effect), proposes an end-to-end evaluation framework built on success rate, cost per successful task, and stability, and details a three-layer context strategy, a practical cost model, and deployment through a unified gateway. It closes with a two-week PoC checklist and three common misconceptions, showing how to turn model capability into measurable, maintainable productivity.
1. Background: Long Context Is Now the Baseline — the Real Differentiator Is Cost-Effectiveness
The open-source model landscape in 2026 is clearly heating up. A wave of “near-frontier” open models has entered developers’ view: context windows commonly reach around 1 million tokens, coding and tool-calling capabilities are markedly stronger, and they can be integrated either through a hosted cloud API or by self-hosting the weights.
Parameter counts, context lengths, benchmark rankings, and list prices all differ, and each model emphasizes different selling points: some highlight general capability, long-horizon agent tasks, and multimodal input; others emphasize coding efficiency, long engineering tasks, and inference cost. Back in engineering reality, developers’ real question is not “who is absolutely stronger” but:
- Is my task code generation, bug fixing, cross-file refactoring, or turning a screenshot into a working implementation?
- How much context and how long an output does a single task need? What are the latency and concurrency requirements?
- Does the team prefer paying per call, or maintaining its own inference infrastructure?
Around these three points, evaluation can be carried out across five dimensions: capability profile and task fit, effective use of context, cost structure and cost per task, deployment and data compliance, and ecosystem maturity and iteration pace.
2. Core Concepts: Parameters, Context, and Effective Context
1. Total vs. activated parameters
Current flagship open models generally use a sparse mixture-of-experts (MoE) architecture: total parameters determine the model’s theoretical capacity, but each inference activates only a small fraction of them, and that fraction determines actual compute consumption. When a model card says “activated parameters are less than one-twentieth of total parameters,” this is what it refers to. This leads to three engineering consequences:
- Small activated parameters mean lower compute consumption and typically lower per-inference latency;
- Expert weights still need to stay resident in GPU memory, so memory requirements are driven by total parameters, not activated parameters;
- Load imbalance causes expert-routing skew, so distributed deployment needs capacity factors, load balancing, and similar mechanisms.
So “large total parameters but small activated parameters” does not necessarily mean high cost for low return. The key is to weigh the self-hosting threshold introduced by total parameters against the actual throughput enabled by activated parameters.
2. Context window is not the same as effective context
Window size is only theoretical capacity. When context enters inference, the KV cache grows linearly with sequence length and concurrency, and can be estimated with the approximation:
KV cache memory (GB) ≈ 2 × layers × KV heads × head_dim × sequence length × bytes per value × concurrency / 1024³
Long sequences have limited impact on a single request; what really amplifies cost is high concurrency stacked on top of long inputs. With, for example, 80,000-token inputs and 32-way concurrency, the KV cache can easily take up tens of GB of GPU memory, which can only be absorbed with a larger memory budget or more aggressive quantization.
3. Effective context and “lost in the middle”
It is widely observed that when key information sits in the middle of a long context, the model’s recall degrades, and instructions placed too early in a multi-turn conversation can get diluted. You can test recall at different depths and lengths using cross-paragraph key-information retrieval (a “needle in a haystack” approach).
A more engineering-relevant approach is to set up a “task degradation” experiment: for the same bug-fixing task, compare providing only the relevant file snippets against providing the entire repository directory — how do first-pass fix rate, output length, latency, and cost each change? This reflects real usability better than a window number does.
3. Evaluation Framework: From Benchmarks to End-to-End Tasks
Public leaderboards are good for a first screening, but selection should rely on end-to-end evaluation that stays close to real tasks, centered on three types of metrics:
- Success rate: whether the task is actually completed. Bug fixing is measured by unit-test pass rate, code generation by executability and test results, and refactoring by behavioral consistency.
- Cost: the average token consumption and expense per successful task, not the unit price itself.
- Stability: format-compliance rate, tool-call validity rate, and consistency of results across multiple runs.
The recommended approach is to extract 50–200 samples from real tickets to form a fixed test set, and to judge results with a single unified verifier. Example skeleton code:
# End-to-end evaluation skeleton: same test set, same verification logic, run against different modelsdef evaluate(candidate, tasks):report = {"passed": 0, "total_tokens": 0, "elapsed_ms": 0}for task in tasks:result = candidate.run(prompt=task.prompt, context=task.context)report["total_tokens"] += result.input_tokens + result.output_tokensreport["elapsed_ms"] += result.elapsed_msreport["passed"] += int(verify(task, result)) # run unit tests / static checks / rule validationreport["pass_rate"] = report["passed"] / len(tasks)report["cost_per_pass"] = price(report["total_tokens"]) / max(report["passed"], 1)return report
Note two points: first, run each task at least 3 times and take the average to avoid sampling randomness; second, cost_per_pass is closer to real spending than “price per million tokens,” because failed tasks consume tokens without producing usable results.
4. Capability Profiles and Boundaries of the Two Model Types
From public information, long-context open models fall roughly into two categories, each with obvious weaknesses.
| Dimension | Capability-First | Efficiency-First |
|---|---|---|
| Typical traits | High general intelligence, strong long-horizon agents, some support image/video input, higher unit price | Strong coding specialization, long engineering tasks approaching frontier closed-source levels, low unit price, high throughput |
| Suited scenarios | Complex system design, multi-step cross-file changes, UI changes from screenshots, multimodal requirement understanding | High-frequency completion, bug fixing, test generation, large-repository analysis and refactoring, large-scale batch tasks |
| Main constraints | High unit cost, high self-hosting memory threshold, heavy ops burden for small and medium teams | Weak multimodal ability, slightly lower ceiling on open-ended reasoning and cross-domain knowledge tasks |
| Evaluation focus | Long-horizon task completion, multimodal input quality, instruction-following stability | Code benchmark performance, first-token latency, cost per task |
Capability-first: look at the ceiling — and whether you can actually use it
General capability is the foundation of complex engineering tasks. Multi-step changes require weighing the blast radius, cross-service bugs require understanding business semantics, and design reviews must balance extensibility. The stronger the general intelligence, the higher the quality of the first-pass solution and the less rework it needs. Multimodal input can also bring UI screenshots, design mockups, and architecture diagrams directly into context, reducing manual restatement.
The cost is price. Long-output tasks (such as generating large amounts of test code) amplify the unit-price gap, so these models are better placed at the “hard-problem” spots in the pipeline.
Efficiency-first: look at the floor — and at stability
Most code tasks have clear goals and executable checks, and specialized models are enough for the common work: fixing bugs after reading a call chain, adding unit tests that match a repository’s existing style, and batch-migrating outdated APIs. High output determinism and speed, combined with high concurrency, make them suitable as the default model for daily development.
The weaknesses must also be faced: on open-ended design problems and cross-domain knowledge questions, reasoning depth is slightly lower; and image-input handling is limited. Some models iterate quickly and are sensitive to prompt formatting, so in engineering you need fixed prompt templates and pinned versions.
One-line conclusion: choose capability-first for the ceiling and multimodality, and efficiency-first for speed and cost; for most teams the answer is a combination of both.
5. Context Workflows: What Determines the Experience Isn’t Window Size
With the same 1 million tokens, different usage can differ several-fold in both effectiveness and cost. In practice you can follow a “three-layer context” strategy:
- Retrieval layer: recall relevant file snippets based on keyword and symbol indexes, kept to about 10%–20% of the window;
- Structure layer: inject the repository directory tree, module dependencies, and interface-definition summaries to help the model build a global picture;
- Instruction layer: placed in the system prompt, clearly stating output format, constraints, and acceptance criteria to avoid dilution in long context.
Long-horizon agent tasks can be split into a “plan–execute–verify” loop, feeding the model only the context relevant to the current subtask each round and compressing history into summaries:
def run_agent_task(task, repo_index, planner, executor, max_rounds=8):state = {"plan": planner(task, context=repo_index.summary()), "history": []}for _ in range(max_rounds):ctx = build_context(state, repo_index, budget_tokens=60_000)action = executor(task, ctx)observation = run_action(action) # run command / edit file / run testsif verify(task, observation):return statestate["history"].append(summarize(action, observation))return state # round limit reached, hand off to a human
Three more engineering tips:
- Prefer compression over truncation: summarize irrelevant history, but keep key constraints verbatim;
- Use prefix caching: place stable system prompts and repository structure at the very front of the context to reduce repeated-input billing and first-token latency;
- For long-output tasks, use streaming responses to improve the interaction experience and lower timeout risk.
6. Cost Model: Don’t Just Look at Unit Price — Compute “Cost per Successful Task”
Price is a deciding factor for many teams, but token unit price is only one term in the cost formula:
Cost per task = (input tokens × input price + output tokens × output price) × retry amplification factor
Three easily overlooked factors:
- Structural amplification: under an agent framework, one user request often contains multiple model calls, so actual consumption may be several to tens of times a single call;
- Retry amplification: raising the success rate from 70% to 85% means the same task set needs fewer retries, so “cost per successful task” actually drops — a higher unit price can still end up cheaper;
- Output-price weight: generative tasks produce far more output tokens than input, so the output price has a larger effect on total cost.
You can roughly estimate monthly cost as “daily task volume × average tokens per task × unit price × 30,” then reserve headroom for peak concurrency.
Self-hosting is another path. Open-weight models can be deployed with mainstream inference frameworks, but you must first calculate the resource bill: memory capacity determines whether all weights can be loaded, throughput and first-token latency determine concurrency capacity, and then you add machine depreciation, ops labor, and elastic-scaling costs. Saving only on tokens while ignoring hidden costs often turns out not to pay off. The key decision point is the break-even between daily call volume × unit price and total monthly self-hosting cost.
7. Deployment and Engineering: Turning Model Capability into a Stable Service
Whether you go via API or self-hosting, it is recommended to integrate through a unified gateway layer that hides the differences from callers:
- Routing strategy: simple requests go to a small or efficiency-first model, while complex requests escalate to a capability-first model;
- Caching and rate limiting: cache results for high-frequency repeated requests, and configure rate limits per tenant;
- Observability: log call chains, token consumption, success rates, and latency distributions, and connect them to logging and monitoring alerts for fast localization when anomalies occur;
- Security and compliance: sensitive code and data require assessing data-egress and retention policies; choose private deployment when necessary.
For example, Baidu AI Cloud’s Qianfan platform provides model services and invocation management, its container platform can host self-hosted inference services, and products such as its logging service and monitoring alerts support runtime observability. With general cloud capabilities like these, teams can build a hybrid “API + self-hosting” model access layer at relatively low cost. Refer to the relevant official documentation for the latest product scope and regional availability.
8. Selection Checklist: A Two-Week, Verifiable PoC
| Evaluation item | Key question | Suggested weight |
|---|---|---|
| Task fit | Pass rate and rework rate on your own test set | 30% |
| Cost | Cost per successful task, share of monthly budget | 25% |
| Stability and ecosystem | Format-compliance rate, version iteration, documentation and community activity | 20% |
| Deployment and compliance | API / self-hosting, data retention, license restrictions | 15% |
| Multimodal and extensibility | Screenshot/video understanding, tool calling, future task evolution | 10% |
Suggested PoC schedule: in the first week, build a 50–200-task real-world test set and complete baseline evaluation of both models; in the second week, integrate the gateway for gray-release dual-model routing, tally cost per successful task, first-token latency, and human acceptance rate, then finalize based on the data. Avoid two types of bias during evaluation: first, looking only at leaderboards and not at your own tasks; second, comparing only unit price and not total cost of ownership.
9. Common Misconceptions and Conclusion
Three common misconceptions
- Longer window is better: irrelevant content crowds out effective context and pushes up cost; retrieval and summarization matter more than filling the window up;
- Bigger parameters mean stronger: actual performance is jointly determined by activated parameters, data quality, and post-training alignment;
- One selection lasts forever: models iterate month by month, so keep a replaceable access layer and re-evaluate quarterly.
Conclusion
Selecting a long-context model is essentially an engineering tradeoff: capability-first provides the ceiling and multimodality, while efficiency-first provides cost and throughput. Rather than arguing about “who is stronger,” it is better to reframe the question as “under my task distribution and budget, which combination has the lowest cost per successful task.” Using your own test set as the yardstick and cost per successful task as the anchor, combined with layered context and a unified gateway, is what truly turns model capability into measurable, maintainable productivity.
