LLM Long Context in Practice: Effective Windows and Pitfall Guide

1. Nominal Context vs. Effective Context

Two years ago, a 256K token context window was considered top-tier in the industry. Today, mainstream Large Language Models (LLMs) frequently advertise windows reaching the 1M token mark, making 256K seem relatively short. However, assuming that a larger window directly translates to a better user experience leads to the first major pitfall.

A 1M token limit represents the maximum capacity of what the model can “hold,” not a guarantee of how well it can “use” that information. In practical applications, most models perform stably within the 200K token range. Once this threshold is exceeded, there is a noticeable decline in instruction-following accuracy and the completeness of detail retrieval. Teams with mature context engineering practices can indeed extend this stable range, but that requires additional engineering effort rather than relying solely on the model’s out-of-the-box capabilities.

The underlying reasons for this degradation are no secret:

• Attention Dilution: The more tokens the model needs to process simultaneously, the lower the attention weight allocated to individual pieces of critical information.
• Lost in the Middle: Information located in the middle of the context window has a significantly lower recall rate compared to information at the beginning and end.
• Task Pollution: Historical failed attempts and redundant discussions continuously interfere with subsequent reasoning.

While many model promotional materials still highlight “long context” as a core selling point, the metric users should genuinely care about is the effective context—the maximum window length the model can reliably utilize while maintaining output quality.

2. Model “Degradation”: The Biggest Pitfall of Long Context

The “degradation” frequently discussed in developer communities refers to the overall performance decline models exhibit as context length grows: responses become verbose, early constraints are forgotten, redundant work occurs, and instruction execution deviates. Empirically, the 100K to 200K token range serves as a reliable comfort zone where most tasks can be completed without noticeable degradation.

Two key reminders:

First, do not use static “needle in a haystack” test results to predict real-world task performance. Needle retrieval tests only verify whether the model “can find” specific information; they cannot verify whether the model “can consistently perform tasks correctly.” Multi-turn, dynamic, stateful real-world tasks place far more pressure on the context than one-off document Q&A.

Second, degradation is gradual, not a sudden cliff drop. It often begins with occasionally forgetting a single constraint and progressively evolves into a systemic deviation from the target. By the time the deviation becomes obvious, the task may have been off-track for a long time. Therefore, monitoring output quality and designing validation mechanisms proactively is far more critical than fixing issues retroactively.

3. Two Approaches: Single Context vs. Multi-Agent Orchestration

When facing tasks that exceed the effective context window, the industry has adopted two typical approaches.

Approach 1: Sticking with a Single Context.
The advantage is that global information is centralized, allowing the model to see the complete history. The trade-off is a rapid increase in token costs and an accumulating risk of degradation as the context grows, making longer tasks increasingly uncontrollable.

Approach 2: Sub-agent Orchestration.
Instead of forcing everything into a single context, tasks are delegated to sub-agents. The main control node only maintains goals, plans, and scheduling information, distributing specific sub-tasks to multiple sub-agents. Each sub-agent receives a clean, short, and focused context, returning only the conclusions upon completion. Public practices from various teams validate this trend: rather than stretching the context window, it is more effective to break tasks down so each execution unit operates within its optimal performance range.

Dimension	Single Context + Compaction	Multi-Agent Orchestration
Global Consistency	High	Depends on result return mechanism
Token Cost	Increases with context length	Controllable due to short sub-tasks
Degradation Risk	Accumulates with length	Low, due to short individual contexts
Orchestration Complexity	Low	High (scheduling, aggregation, conflict resolution)
Applicable Scenarios	Tightly coupled tasks requiring global consistency	Tasks with clear boundaries that are divisible and parallelizable

In real-world long-horizon engineering tasks, a hybrid approach is often used: the main controller handles planning and division, while sub-nodes handle execution and verification.

4. Context Engineering: The Key to Stable Long-Horizon Tasks

If the model’s window size determines the upper limit, context engineering determines the actual lower limit you can achieve. The core comes down to two things: externalizing the state and building a recovery mechanism.

4.1 Externalizing Process State: Writing to Files

Do not rely on conversation history to carry the entire memory of a task. Write intermediate artifacts, decision logs, and task checklists to external files, keeping only the minimal context required for the current work in the conversation. A common practice is to maintain a task state file:

{
  "goal": "Write a reconciliation service for the order system",
  "constraints": ["Compatible with existing database schema", "Reconciliation delay under 5 minutes"],
  "progress": [
    {"step": "Read database schema", "status": "done", "output": "docs/schema.md"},
    {"step": "Design reconciliation algorithm", "status": "in_progress", "output": "docs/design.md"}
  ],
  "next_action": "Complete algorithm design review and update design document"
}

As long as this file exists, the task can resume from the breakpoint even if the entire conversation history is lost. This is the value of writing process files: it converts volatile context into a persistent asset.

4.2 Accept the Fact: Compaction is Inevitable

For true long-horizon tasks, context compaction is not an anomaly; it is an inevitability. Developers have shared cases of long-horizon tasks processing approximately 147 million tokens cumulatively—a volume that no context window can hold. Such a task will inevitably undergo dozens or hundreds of compactions throughout its lifecycle. In other words: a task that does not trigger compaction cannot be considered a long-horizon task.

Therefore, the question is never “how to avoid compaction,” but rather “how to correctly continue after compaction.”

4.3 The Harness Must Recover Losslessly After Compaction

Compaction is a high-risk point for long-horizon tasks. Compaction is typically summary-based; if task goals, early constraints, or key decisions happen to be lost in the summary, the task will silently deviate. A robust harness (task execution framework) should achieve the following:

1. Before Compaction: Ensure critical states are persisted (goals, constraints, progress, next actions).
2. After Compaction: Immediately rebuild the task context from the state file, rather than relying on the summary’s “memory.”
3. Periodically: Persist execution results to disk and output validation reports when necessary.

A conceptual pseudocode example:

def run_long_task(model, state_file):
    state = load(state_file) or init_state()
    while not state.is_done():
        if model.context_usage() > COMPACT_THRESHOLD:
            flush_state(state)            # Persist to disk first
            model.compact()               # Then compact
            state = reload(state_file)    # Recover from file after compaction
        result = model.step(state.brief())
        state.update(result)
        flush_state(state)                # Persist to disk every step

This discipline of “persist first, compact, then recover” is the dividing line for whether a long-horizon task can run stably.

5. Session Limits and Task Continuity

Many managed platforms impose duration or usage limits on individual sessions (e.g., on the scale of hours). This leads to two conclusions:

1. Long-horizon tasks will inevitably span multiple session cycles; the harness cannot assume “session immortality.”
2. Whether a task can resume after a session interruption depends entirely on how well state externalization is implemented.

Rather than treating platform duration limits as obstacles, treat them as mandatory drills: any compaction or disconnection should be equivalent to “starting a new session, reading the file, and continuing the work.” Enterprise users running such tasks on cloud platforms (such as Baidu AI Cloud) should also integrate observability capabilities, like logging services and monitoring alerts, into their harness design, ensuring that critical events like compactions and breakpoints are trackable and traceable.

6. Practice Checklist: Turning Pitfalls into Processes

Finally, consolidate the above experiences into an actionable checklist:

1. Budget by Effective Window: Plan tasks using 100K~200K tokens as the single working range; do not budget based on the nominal window limit.
2. State as Files: Goals, constraints, progress, and next actions—the four key elements—must be persistable and recoverable at all times.
3. Proactive Compaction: Monitor context usage. Actively trigger compaction at the threshold and execute the persistence workflow; do not wait for the platform to passively truncate the session.
4. Output as Validation: Run assertions or tests after critical steps to detect early deviations caused by “degradation.”
5. Divide and Conquer: When sub-task boundaries are clear, prioritize multi-agent orchestration to keep every context within its optimal performance range.
6. Treat Compaction as the Norm: One of the acceptance criteria for the harness should be that “the task can resume losslessly at any moment of compaction or disconnection.”

Conclusion

The promotional numbers for long context windows are refreshed annually, but engineering realities remain constant: the effective window is much smaller than the nominal window, compaction is a constant in long-horizon tasks, and “state externalization + lossless recovery” is the true source of stability. When selecting models, look at the effective range; when designing, assume compaction will inevitably occur; when executing, let files carry the memory. By internalizing these three principles into your workflow, you can proactively bypass most of the pitfalls associated with long-context LLM tasks.