Engineering Guide for Long-Horizon LLM Agent Tasks
Long-horizon LLM agents face engineering challenges such as context bloat, task deviation, and unpredictable costs when handling multi-step complex tasks. This guide provides engineering practices for long-horizon agent tasks, covering multi-agent architecture division, context compression strategies, model tiering and cost control, and task-level evaluation systems to help developers stabilize model capabilities in real-world scenarios.
1. Background: New Challenges Brought by Long-Horizon Agents
As large language models (LLMs) evolve from single-turn Q&A to long-horizon agents, application forms have changed significantly. In the past, one request solved one problem. Now, a single task might involve dozens of steps: reading a codebase, breaking down requirements, calling tools, running tests, fixing errors, and summarizing results. Task durations stretch from a few seconds to tens of minutes, and the intermediate context expands accordingly.
This shift introduces three typical challenges:
- Context window overflow: The history fills the context window, pushing out early critical constraints, causing the model to deviate from the original goal.
- Premature termination or looping: During long chains, the model may end prematurely by claiming the task is complete without actually finishing it, or repeatedly trial-and-error on a single step, consuming excessive steps.
- Unpredictable costs: Call costs grow with the number of steps, varying greatly across tasks, making budgets hard to estimate.
For developers, understanding these issues is more practical than chasing the parameter size of a single model. Model capability is a prerequisite, but engineering structure determines whether that capability can be stably applied in real tasks. The following sections elaborate on architecture, context management, cost control, and evaluation.
2. Task Division: Orchestrator and Worker Agents
A multi-agent architecture is a common approach to handling complex tasks. A stable pattern involves setting up an orchestrator agent responsible for planning and acceptance, while multiple worker agents execute specific subtasks in parallel. This structure separates “figuring out what to do” from “actually doing it,” with each layer handling only what it excels at.
When dividing tasks, three principles can be followed:
- The orchestrator only holds the global goal, the subtask list, and acceptance criteria. It does not directly read large chunks of raw materials to avoid being overwhelmed by details in its own context.
- Worker agents receive only the minimum context required to complete the current subtask. Upon completion, they return results in a structured format, including status, artifact locations, and unresolved issues.
- Dependencies between subtasks are uniformly maintained by the orchestrator. Workers do not pass messages directly to each other, reducing coupling and making fault localization easier.
Below is illustrative pseudocode demonstrating the basic flow of batched parallel execution based on dependencies:
def run_task(goal: str) -> Report:plan = orchestrator.plan(goal) # Split into subtasks with dependenciesresults: dict[str, Result] = {}for batch in topological_batches(plan.subtasks): # Tasks within the same batch do not depend on each otherfutures = {t.id: worker_pool.submit(execute_subtask,t,context=build_context(t, results), # Inject only the output summary of dependency taskstimeout=t.timeout,)for t in batch}for task_id, future in futures.items():results[task_id] = future.result()return orchestrator.review(goal, results) # Orchestrator performs final check against acceptance criteria
During implementation, note a few points. Concurrency should have an upper limit; otherwise, multiple workers running simultaneously can quickly exhaust quotas or trigger rate limits. Each subtask should have timeout and retry limits, returning a clear failure status upon timeout instead of waiting indefinitely. The acceptance phase is best completed by an independent check step, such as running a test suite or validating output format, rather than letting the executor judge completion itself.
3. Context Growth and Compression Strategies
Context growth is the most common issue in long-horizon tasks. As conversation turns increase, historical content must be resubmitted with each call, raising costs and latency, while the model’s attention to early information degrades. Three common countermeasures exist:
1. Periodic Summarization and Compression
When the context exceeds a threshold (e.g., 70% of the window limit), compress the middle portion of historical conversations into a structured summary. The summary should consistently contain four fields: goal, constraints, completed steps, and open issues. Raw tool outputs in the middle can be discarded directly.
2. External Storage and On-Demand Retrieval
Large volumes of file content, logs, and test outputs should not be placed into the context wholesale. A more reasonable approach is writing them to external storage, keeping only paths, sizes, and summaries in the context, and retrieving fragments by line number or keyword when needed.
3. Context Isolation by Subtask
Each worker agent uses an isolated context. The orchestrator receives only the results, not the execution process. This drastically slows the growth of the orchestrator’s context and reduces interference from irrelevant information.
Illustrative code is as follows:
PINNED_ROLES = {"system"}KEEP_RECENT = 6def maybe_compact(messages: list[Message], limit_tokens: int) -> list[Message]:if count_tokens(messages) < limit_tokens * 0.7:return messagespinned = [m for m in messages if m.role in PINNED_ROLES or m.pinned]recent = messages[-KEEP_RECENT:]middle = [m for m in messages if m not in pinned and m not in recent]summary = summarize(middle,schema=["goal", "constraints", "done", "open_issues"],)return pinned + [summary_message(summary)] + recent
Compression itself carries risks. Summaries may lose key constraints, causing subsequent steps to deviate from the goal. Therefore, two protections are needed: first, mark constraint conditions as a non-compressible pinned area; second, after compression, require the model to restate the goal and constraints in its own words, and programmatically compare key fields for consistency. If the restatement is inconsistent, regenerate the summary instead of continuing execution.
4. Model Tiering and Cost Control
Not all subtasks require the same tier of model. Solution design, cross-module risk assessment, and final acceptance are reasoning-intensive stages suitable for more capable models. Massive repetitive file reads/writes, format conversions, and simple retrievals can be handed off to cheaper, faster models.
Applicable routing rules include:
- Routing by task type: Planning and acceptance use a high-capability tier; execution subtasks use a standard tier.
- Routing by context length: Long-context subtasks should be prioritized for splitting rather than directly using the longest window, because longer windows increase the cost and latency of a single call.
- Routing by budget: Set token limits for individual tasks and daily totals. When approaching the threshold, automatically downgrade to a low-cost model, or pause and request manual confirmation.
Cost accounting should be based on “cost per unit task” rather than the price of a single call. A cheap model that requires repeated retries might have a higher overall cost than a more expensive model that succeeds on the first try. It is recommended to record the following data for each task: call count, input/output token count, retry count, duration, and whether it finally passed acceptance. Over time, real data will reveal whether the routing rules are reasonable.
Below is illustrative code for the routing logic:
def pick_model(subtask: Subtask, budget: Budget) -> str:if budget.remaining_ratio() < 0.1:return "reserve" # Keep only minimal execution capabilities when budget is tightif subtask.kind in {"plan", "review"}:return "high"if subtask.context_tokens > 64_000:return "standard" # Prefer window-adapted models for long contexts, and split tasks if possiblereturn "standard"
5. Evaluation: Beyond Single-Turn Answer Quality
When evaluating long-horizon agents, simply looking at whether a single response is fluent is far from enough. Metrics should be designed around the outcome of the complete task, including at least the following five:
- Task completion rate: The proportion of tasks completed within the limited steps and passing independent acceptance.
- Premature termination rate: The proportion of tasks claiming completion without actually finishing, a typical failure mode in long-horizon scenarios.
- Tool call accuracy: The proportion of calls with correct parameters and successful execution.
- Cost and duration per unit task: The average tokens and time consumed per accepted task.
- Manual intervention count: The average number of times a task requires manual correction or supplementary information.
It is recommended to establish a fixed regression task set covering typical scenarios such as code modification, document organization, and data retrieval. Whenever changing models, adjusting prompts, or modifying compression strategies, run the same set of tasks again and compare the above metrics. Making launch decisions based on data is more reliable than relying on subjective experience, and it can promptly detect regressions in certain task types caused by a specific change.
6. Implementation Checklist
- Define clear and verifiable completion criteria for each subtask, avoiding ending with a “good enough” condition.
- Execute tests or validations using an independent check step; do not let the executor self-evaluate the results.
- Set concurrency limits, timeouts, and retry counts to prevent resource exhaustion or infinite task hanging.
- Automatically compress context when it exceeds a threshold, keeping constraints in a non-compressible area.
- Select models by tier based on task type, and record the unit cost of each task.
- Establish a fixed regression task set for comparative testing before and after each change.
