How to Evaluate Long-Context LLMs: Architecture and Long-Horizon Agents
As open-source large models push context windows toward the million-token level and shift their focus from single-turn Q&A to long-horizon agents, developers need a way to judge them beyond parameter counts and benchmark scores. This article proposes a six-dimension evaluation framework—architecture, long-context training, inference acceleration, serving systems, long-horizon agents, and engineering deployment—with concrete, actionable checklists for each dimension. You can use it to read any model technical report with a clear checklist in hand, or to run a small-scale selection validation within a few hours. The core focus is on whether long-context capability is truly trained rather than extrapolated, whether acceleration translates into real serving gains, and whether agents can reliably complete end-to-end tasks.
Introduction: Beyond Parameters and Scores, You Need an Evaluation Framework
Every time a leading LLM vendor releases a new generation of models, developer communities fill up with “how should we evaluate it” discussions. Some stay at the level of parameter counts, leaderboard scores, and context length; others focus on real-world usability and cost. This article doesn’t target any specific model. Instead, it tries to answer a more general question: When open-source large models push context to the million-token level and shift their primary use case from single-turn Q&A to long-horizon agents, what dimensions and methods should we use to evaluate them?
Given limited space, this article goes straight to a conclusive evaluation framework: a checklist built around six dimensions—architecture, training, inference acceleration, serving systems, long-horizon agents, and engineering deployment—with an actionable check item for each. You can carry this checklist while reading any model technical report, or use it to complete a small-scale selection validation within a few hours.
This article only discusses widely adopted industry concepts and general engineering experience. For any specific model, defer to official release information and technical reports, and base every conclusion on measured data from your own workload.
1. Evaluation Background: From “Long Context” to “Long Horizon”
Over the past two years, there’s a clear main thread in the evolution of large model capabilities: context windows growing from 128K toward the million-token range. Behind this thread is a shift in task shape:
- Code and knowledge bases: Before million-level context appeared, analyzing a medium-sized codebase required a lot of retrieval, chunking, and summarization—and any error in that pipeline would distort the conclusions. Now you can feed an entire repository into the model and, from a global perspective, perform cross-file refactoring, impact analysis, and regression test generation.
- Long-document and multi-document reasoning: Contracts, annual reports, or technical manuals hundreds of pages long can be input as a whole, and the model can directly perform cross-chapter checks, clause-conflict detection, and structured extraction.
- Long-horizon agents: Working continuously across long chains of tens to hundreds of tool calls, such as cross-service troubleshooting, batch data migration, and end-to-end feature development.
But window length itself is just a “container capacity” metric. The real dividing line is: whether content can be effectively used, whether inference cost stays controllable, and whether the agent can advance steadily across a long chain. This is precisely the difference between “long context” and “long horizon.” Long context measures the absorption capacity of a single input; long horizon measures sustained execution across multiple rounds and steps. Both require dedicated architecture and training design, and both need matching evaluation methods.
2. Evaluation Framework Overview
Before diving into details, here’s the overall framework. To evaluate an open-source large model that focuses on ultra-long context and long-horizon agents, I recommend six dimensions. They progress from shallow to deep, so work through them layer by layer:
| Dimension | Core question | Key observations | Typical evaluation method |
|---|---|---|---|
| 1. Architecture design | Can it support ultra-long context under controllable compute? | Sparse experts, sparse attention, index reuse | Read the technical report; check public configs and ablation studies |
| 2. Long-context training | Was long context “trained” or “extrapolated”? | Training stage and sequence length, position encoding extension method | Multi-depth retrieval tests, long-document reasoning, long-text quality checks |
| 3. Inference acceleration | Can it stably speed up both long-output and long-input scenarios? | Speculative decoding acceptance rate, compatibility with inference frameworks | Compare decode throughput and first-token latency; stress-test long inputs |
| 4. Serving systems | Can it serve stably and economically at million-level context? | Memory management, scheduling, cache reuse, observability | Concurrency stress tests, long-run stability tests, capacity planning |
| 5. Long-horizon agents | Can it advance correctly across tasks of dozens of steps or more? | Long-trajectory training loop, context compaction, reward anti-hacking | Multi-turn end-to-end task tests, long-trajectory stability analysis |
| 6. Engineering deployment | Does it fit real business and teams? | License terms, deployment barrier, ecosystem tooling, ops system | Small-traffic canary, cost estimation, compliance review |
It’s worth stressing that the six dimensions are not independent. Sparsification in the architecture directly affects the choice of training strategy; inference acceleration depends on inference framework support; long-horizon capability, in turn, relies heavily on serving system stability. So only by evaluating all six dimensions as a whole can you avoid the misjudgment of “excelling at one point but imbalanced overall.”
3. Dimension 1: Architecture Design—How Sparsification Supports Million-Level Context
3.1 Sparse MoE: Decoupling Total Parameters from Activated Parameters
Most mainstream open-source large models today adopt a sparse Mixture-of-Experts (MoE) architecture. Each layer contains a large number of expert networks, and a router selects a few experts to participate in computation for each token. Its essence is decoupling “model capacity” from “per-token compute”: total parameters determine how much knowledge the model can hold, while activated parameters determine the actual compute cost of each forward pass.
For evaluators, it’s important to understand what each number means:
- Total parameters: Represents model capacity and training cost, and also sets the floor for GPU memory usage. All expert weights must stay resident in memory; otherwise the runtime will frequently shuttle weights between memory and GPU memory, and latency will spike.
- Activated parameters: Determines per-token compute, directly related to decode throughput and single-GPU compute requirements.
- The ratio between them: A key indicator of sparsification. When the ratio of total to activated parameters reaches ten-plus times or higher, it’s an aggressive sparse configuration. A higher ratio means more knowledge carried per unit of compute, but also higher demands on routing quality, expert load balancing, and training stability.
When evaluating, focus on asking three things:
- How many activated parameters? This determines how much compute you need at minimum to deploy.
- How many total parameters? This determines your GPU memory budget and parallel sharding scheme.
- Is expert routing load-balanced? Tokens at different positions in a long context activate different experts. If routing is heavily skewed, a few experts become hotspots, and actual throughput will fall well below the theoretical value.
3.2 Dynamic Sparse Attention: Tackling the Compute Bottleneck of Long Context
Standard attention compute grows quadratically with sequence length. When a sequence expands from a few thousand tokens to a million, length grows roughly a hundredfold, but attention compute grows roughly ten-thousandfold. Without optimization, ultra-long context heads straight into “compute explosion.”
The idea behind dynamic sparse attention is: at runtime, dynamically select the subset of keys worth attending to based on the query content, performing exact computation only on key segments, while the remaining positions interact approximately through lightweight summaries or index information. It shares the same sparse philosophy as MoE, replacing “full interaction” with “on-demand interaction,” thereby greatly reducing per-token attention overhead.
When evaluating such designs, note three points:
- Whether it’s consistent with training: Is the sparse pattern introduced during training or added at inference time? The former lets the model learn to adapt to sparse attention during training and is generally more reliable; the latter requires additional adaptation validation.
- Whether it supports index reuse: Recomputing attention indices at every layer brings considerable overhead. Whether indices can be reused across layers is a key engineering detail that determines the actual speed of long context.
- Whether it sacrifices long-range dependencies: Sparsification may cause some cross-segment associations to be ignored, which must be validated through long-document reasoning tasks.
3.3 Evaluation Checklist (Architecture)
- How many total and activated parameters respectively? Is the ratio reasonable?
- Does it use dynamic sparse attention? Are the training and inference sparse patterns consistent?
- Does it have long-context-specific optimizations like cross-layer index reuse and KV sharing?
- Does the technical report provide ablation studies targeting the long-context architecture?
4. Dimension 2: Long-Context Training—Was the Capability “Trained” or “Extrapolated”?
4.1 Introducing at Training Time vs. Extrapolating at Inference Time
This is the core criterion for distinguishing “real long context” from “paper long context.” Inference-time extrapolation relies on tricks like position encoding interpolation and attention scaling to “force-feed” the model long sequences it never saw during training. Such approaches may perform acceptably at moderate lengths, but because the training data and inference distribution are inconsistent, they easily develop mid-segment forgetting, repetition degradation, and declining instruction following at ultra-long distances.
A more mature approach is systematically introducing long sequences during training: starting from the mid-training stage, mixing data like long documents, long code, and long conversations into the training corpus in proportion, and using an efficient attention implementation that supports ultra-long sequences to complete forward and backward passes. For example, a common approach is to introduce long-context capability at a mid-training stage (such as a 128K sequence-length stage). Although this significantly increases training cost, the model’s positional relationships and long-range dependency modeling are genuinely learned during training, making long-context performance more reliable.
You can judge from three levels during evaluation:
- Documentation level: Do the official technical docs clearly disclose the stage and data composition of long-sequence training?
- Code level: Are there corresponding long-sequence training configs or implementations in the repository?
- Paper level: Are there companion academic papers you can trace the mechanism back to?
It must be objectively pointed out that leading vendors’ new models are often disclosed primarily through technical blogs, and complete technical reports may not yet be published. Therefore, when evaluating, record disclosed information separately from assumptions still to be verified, and avoid treating speculation as conclusion.
4.2 Test Methods for Long-Context Effectiveness
- Multi-depth retrieval tests: At different lengths like 100K, 300K, and 1M tokens, randomly insert a “needle” (a key piece of information) at different depths of the document (beginning, middle, end) and test the model’s recall accuracy. Focus on whether performance on the middle of long texts is significantly worse than at the beginning and end.
- Multi-hop and aggregation reasoning: Retrieval tasks can only verify “can find it.” What truly tests long-context capability is associative reasoning across multiple pieces of information, such as aggregating interrelated conditions from dozens of documents to reach a conclusion.
- Long-text generation quality: Long inputs are often accompanied by long outputs. Check whether the model exhibits degradation like repeated paragraphs, internal contradictions, and broken formatting.
- Length extrapolation consistency: Compare the same model’s performance at 32K and 1M context. If performance drops sharply as length increases, it indicates long-context capability comes more from inference-time extrapolation than training-time adaptation.
5. Dimension 3: Inference Acceleration—Speculative Decoding and Inference Framework Compatibility
5.1 Multi-Token Prediction: Letting Each Decode Step “Go Further”
Autoregressive decoding generates only one token at a time, and token-by-token decoding is limited by GPU memory bandwidth, so GPU compute is often largely idle. Speculative decoding improves decoding efficiency through “guess first, then verify”: a small model or extra prediction module first produces several candidate tokens in a row, and the main model then verifies them all in parallel at once; accepted candidates are taken directly, and the first rejected position is corrected with the main model’s output. The average number of accepted tokens is called the acceptance length, which is the core metric for speculative decoding efficiency.
Multi-Token Prediction (MTP) is a training-built-in form of speculative decoding: while predicting the next token, the model also predicts several subsequent tokens through extra prediction modules. Because the prediction modules are jointly trained with the backbone, their guess distribution is highly consistent with the backbone, and the acceptance rate is typically higher than approaches using an attached small model. With a 7-step prediction configuration, the acceptance length can reach around 5 in publicly reported industry practice, and overall decode throughput gains can exceed 20%. As a comparison, common models without speculative decoding often have an acceptance length around 3.
But acceleration effects depend heavily on task characteristics:
- Code generation: Highly patterned and predictable, usually with a longer acceptance length.
- Open-ended Q&A and creative writing: High uncertainty, relatively shorter acceptance length.
- Long-context inputs: The prediction module also has to handle long context, and its overhead can’t be ignored.
So when evaluating, don’t just look at “maximum acceptance length”; measure end-to-end throughput on datasets close to your real workload.
5.2 Inference Framework Compatibility: Can Acceleration Actually Land?
Architectural innovation only translates into real serving capability when supported by mainstream inference frameworks. When evaluating a new model, pay attention to:
- Inference framework support: Whether it’s been adopted by mainstream inference frameworks, and whether they support the model’s specific optimizations like sparse attention and KV sharing.
- Quantization and parallelism support: Whether it supports low-bit quantization, tensor parallelism, and expert parallelism, and whether it runs stably on target hardware.
- Compatibility risk: When a new model is just released, dedicated optimizations may not yet be covered by frameworks, so allow for an adaptation period and keep a standard attention path as a fallback.
5.3 Acceleration Comparison Experiment
I recommend building a minimal comparison experiment on your own hardware, using the same set of requests to measure the following metrics:
- Decode throughput: output tokens per second (tokens/s);
- First-token latency (TTFT): strongly correlated with input length;
- End-to-end latency distribution: focus on the P95 and P99 long tail.
def benchmark_decoding(model_client, prompts, num_warmup=5, num_runs=20):"""Compare decode throughput and first-token latency. model_client is a self-wrapped inference call interface."""import time, statisticsfor _ in range(num_warmup): # Warm-up: avoid first-run compilation and cache loading affecting resultsmodel_client.generate(prompts[0], max_tokens=256)ttft_list, tps_list = [], []for _ in range(num_runs):for prompt in prompts:start = time.perf_counter()first_token_at, out_tokens = None, 0# Receive output in streaming mode, recording a timestamp for each tokenfor chunk in model_client.stream(prompt, max_tokens=1024):if first_token_at is None:first_token_at = time.perf_counter()out_tokens += 1end = time.perf_counter()ttft_list.append(first_token_at - start)decode_secs = end - first_token_atif decode_secs > 0:tps_list.append((out_tokens - 1) / decode_secs)return {"TTFT_P50(ms)": statistics.median(ttft_list) * 1000,"Decode_tokens_per_s": statistics.mean(tps_list),"Decode_tokens_per_s_P5": sorted(tps_list)[len(tps_list) // 20],}
6. Dimension 4: Serving Systems—How Inference Serving Supports Million-Level Context
6.1 The Memory Challenge of Ultra-Long Context
Whether a model can “support 1M context” and whether it can “serve many concurrent requests stably under 1M context” are problems at two different levels. The KV cache grows linearly with context length and is the largest memory overhead source in long-context serving. A rough estimate:
KV cache memory ≈ 2 (K and V) × batch size × sequence length × num layers × num KV heads × head dim × bytes per element
Using 80 layers, 1024 K/V elements per token per layer, and 2-byte precision for the KV cache, a single request’s KV cache at 1M context approaches 320GB. Add hundreds of GB to TB-level model weights, and long-context inference demands far more memory than conventional scenarios. Even if you cut the overhead several-fold through techniques like inter-layer KV sharing, low-bit quantization, and KV cache offloading, memory remains the constraint that most needs priority accounting in capacity planning.
6.2 Engineering Systems for Inference Serving
Mature ultra-long-context serving usually includes the following mechanisms. They aren’t unique to any one model but are general engineering requirements for serving systems:
- Prefill/Decode separation: The first-token computation for long inputs (Prefill) is compute-intensive, while decoding (Decode) is memory-bandwidth-intensive. Deploying them in different resource pools avoids long prefills blocking ongoing decode requests.
- Chunked prefill: Split an ultra-long prefill into multiple small chunks and interleave-schedule them with decode requests, avoiding a long-input request monopolizing compute and slowing down all online requests.
- KV cache reuse: System prompts, long documents, and code repositories are often repeatedly input as prefixes; a prefix cache hit can directly skip repeated computation.
- Memory paging and compression: Borrowing from the operating system’s virtual memory idea, manage the KV cache in pages, and support compression and cross-layer sharing to improve memory utilization.
- Long-run stability: A single ultra-long request may take tens of seconds or even minutes, so the service needs timeout control, request queuing, and graceful degradation to avoid request pileups triggering an avalanche.
6.3 Serving Capability Test Checklist
- Concurrency stress test: At 1M input length, gradually increase concurrency and observe the throughput inflection point and latency long tail.
- Mixed workload: Mix long-input requests with short ones to verify whether short-request latency is dragged down by long requests.
- Long-run stability test: Run continuously for more than 24 hours and observe whether issues like memory fragmentation accumulation and memory leaks appear.
- Capacity accounting: Estimate the maximum concurrency of a single instance from “model weights + peak KV cache + runtime overhead,” then use that to calculate the per-token serving cost.
7. Dimension 5: Long-Horizon Agents—From “Can Call Tools” to “Can Finish a Task”
7.1 The Real Difficulty of Long-Horizon Tasks
Single-turn tool calling tests the model’s understanding of tools; long-horizon tasks test the model’s ability to maintain goals, manage context, and recover from failures across chains of dozens of steps or more. The main challenges include:
- Context bloat: Each round of tool-call results enters the context, and a long chain quickly hits the window limit.
- Sparse intermediate feedback: There’s a clear positive signal only when the task finally completes; intermediate steps are hard to judge as good or bad.
- Goal drift: As steps increase, the model tends to gradually drift from the original goal or keep circling on local optima.
- Reward hacking: If the training signal is poorly designed, the model may learn to “appear to complete” rather than “actually complete.”
7.2 Three Key Links in Long-Horizon Agent Reinforcement Learning
- Context compaction: Periodically compress historical interactions into structured summaries within a long trajectory, keeping key decisions, completed steps, and pending items to free up context space. This needs to be implemented at inference time and also taught during training so the agent learns “when to compact and what to compact.”
- Critic model and PPO: Introduce a value-estimation model (Critic) to score task progress, combined with reinforcement learning algorithms like Proximal Policy Optimization (PPO), to provide dense feedback for intermediate steps and alleviate the sparse-reward problem.
- Anti-hacking mechanisms: Through rule validation, trajectory review, and automated unit-test verification, prevent the model from “fooling” the reward signal via methods like faking test results, skipping key validations, or repeating invalid calls.
7.3 Long-Horizon Capability Evaluation Methods
- End-to-end task success rate: Select real multi-step tasks (like “fix a specific bug and pass all tests”) and measure the full pass rate, not single-step accuracy.
- Average steps and token consumption: At the same success rate, the lower the steps and token consumption, the higher the agent’s planning efficiency.
- Long-trajectory stability: Observe whether goal forgetting, repeated operations, and tool-call format errors appear in the later part of task execution.
- Failure attribution analysis: Categorize failures into “planning errors,” “execution errors,” “context loss,” “external tool anomalies,” and so on, to determine whether the bottleneck is in the model or the environment.
{"task_id": "bugfix-0421","total_steps": 38,"compaction_triggered_at": [12, 27],"outcome": "success","failure_attribution": null,"notes": "After compaction at step 27, the original requirements and the list of passed test cases were retained"}
8. Dimension 6: Engineering Deployment—From Evaluation to Production
8.1 Open-Source License and Commercial Terms
The license is often a “single-veto item” in enterprise selection. Models under permissive licenses (like common permissive open-source licenses) usually allow commercial use, private deployment, and secondary fine-tuning. Before deployment, still check each item carefully:
- Whether commercial use is allowed, and whether derivative works must also be open-sourced;
- Attribution and usage restrictions on the model name and output content;
- Whether there are additional terms like an acceptable use policy.
8.2 Deployment Form and Cloud Resources
Ultra-long-context models demand high deployment resources, so you can plan using general cloud capabilities:
- Container platform: Hosts inference services, enabling elastic scaling and rolling upgrades;
- Object storage: Stores model weights, training data, and evaluation datasets, supporting large-scale distribution;
- Logging and monitoring/alerting: Collects metrics like TTFT, decode throughput, and memory watermarks, and configures threshold alerts;
- Message queue: Makes long-horizon tasks asynchronous with queuing, smoothing traffic peaks.
With the general cloud product capabilities from Baidu AI Cloud—containers, storage, logging, monitoring, and so on—you can quickly build an inference service platform with elasticity and observability. The deployment architecture should be designed by your team based on actual business needs.
8.3 Typical Application Scenarios
| Scenario | Context demand | Recommended strategy |
|---|---|---|
| Code repository understanding and refactoring | Million-level | Full-repo input + hybrid retrieval augmentation, with long-output quality checks |
| Long-document review and cross-checking | Hundreds of thousands to millions | Chunked preprocessing + global summarization, focusing on cross-chapter consistency |
| Multi-turn conversation and knowledge-base Q&A | Tens of thousands to hundreds of thousands | Regular context suffices; focus on cache reuse and cost control |
| Long-horizon agent tasks | Dynamically growing | Context compaction + task checkpoints; focus on long-run stability |
8.4 Canary Rollout and Continuous Evaluation
I recommend putting new models into a standard “evaluate—canary—full rollout” process:
- Offline evaluation: Complete the six-dimension evaluation on your own datasets;
- Shadow traffic: Copy real online requests to the new model, record only without returning, and compare quality and latency;
- Small-traffic canary: Roll out gradually by business line, continuously tracking success rate and cost;
- Full rollout and retrospective: Produce an evaluation report and accumulate it into the team’s internal model-selection knowledge base.
9. Conclusion and Evaluation Checklist
Back to the original question: how do you evaluate an open-source large model that focuses on million-level context and long-horizon agents? The core conclusion is that you shouldn’t stop at parameters and scores; instead, build a six-dimension evaluation system of “architecture—training—inference acceleration—serving systems—long-horizon agents—engineering deployment.” Three judgments are especially critical:
- Whether long context is “trained”: Distinguishing training-time extension from inference-time extrapolation is the first watershed in judging the authenticity of long context.
- Whether acceleration can “land”: The gains from speculative decoding depend heavily on the task and inference framework support, and must be validated with real business data.
- Whether long-horizon tasks can “finish”: End-to-end success rate, long-run stability, and anti-hacking mechanisms say more about an agent’s real level than single-step accuracy.
| Dimension | Core check item | Passing standard (reference) |
|---|---|---|
| Architecture design | Sparse MoE and sparse attention config | Fully disclosed officially, with long-context-specific optimizations |
| Long-context training | Training stage, long-document task performance | No significant mid-segment forgetting in long texts |
| Inference acceleration | Throughput, TTFT, framework support | Decode throughput gain ≥20% on business data |
| Serving systems | Concurrency, long-run stability, capacity | Controllable long-tail latency under mixed workloads |
| Long-horizon agents | End-to-end success rate, steps | Long tasks complete stably, with no reward-hacking behavior |
| Engineering deployment | License, deployment, ops | License-compliant, with complete monitoring and canary systems |
Finally, a reminder: what this article offers is an evaluation methodology, not an assessment of any specific model. A model’s real performance keeps changing with version iterations, hardware environments, and business scenarios, and any selection conclusion should come from reproducible testing and continuous data tracking. Only this way can you move from “leading parameters” to “usable capability, stable results, and controllable cost.”
