From Conversations to Autonomous Agents: Long-Horizon Task and Cost Engineering Practices

1. From Chatting to Working: The Shift in Model Capabilities

Over the past two years, the dominant form of LLM applications has been “conversation”: a user asks a question, the model provides a single-turn answer, and the session state resets upon completion. This paradigm demands high single-turn quality—breadth of knowledge, depth of reasoning, and fluency—metrics that standard benchmarks evaluate as “one-shot accuracy.”

However, in real production environments, significant value isn’t generated by answering a single question, but by “continuously completing a task”: locating and fixing an engineering bug, producing an industry research report, or driving an end-to-end content creation workflow. These tasks share common characteristics: they are goal-oriented rather than question-oriented; they require multi-step planning and self-correction; they frequently invoke external tools; and their execution cycles are measured in minutes or even hours. The application paradigm supporting these tasks is the Agent.

Recently, the industry has shown a clear trend: providers are allocating more post-training compute power toward true agentic environments—programming, financial research, 3D tool manipulation, and long-horizon task execution—rather than merely刷新ing single-turn benchmark scores. Simultaneously, the delivery surface for model capabilities is broadening. It extends from web chat to desktop clients and programming tools, embedding capabilities natively into workflows. Some providers also release full model weights to support enterprise private deployments. These actions point to a shared consensus: the competitive frontier is shifting from “who answers more accurately” to “who can continuously complete tasks at an acceptable cost.”

Consequently, evaluation metrics are evolving. Beyond single-point benchmark scores, engineering teams increasingly focus on four key metrics: end-to-end task success rate, long-horizon consistency (whether the model still adheres to initial constraints after dozens of steps), tool invocation reliability (correctness of parameter formats and timing), and per-task cost. The last metric is exactly where many teams transition from being “amazed” to experiencing “bill shock.”

2. Why Quotas Burn Out in Ten Minutes: The Agent Token Consumption Model

To understand agent costs, we must first examine the token consumption structure.

In a conversational scenario, a single request typically takes a few hundred to a few thousand input tokens and outputs a few hundred tokens. Consumption is dictated by the user’s typing volume; a dense ten-minute chat might only consume tens of thousands of tokens. In an agent’s main loop, however, every step must re-initiate a request carrying the full context: system prompts (thousands of tokens), tool definitions (thousands to tens of thousands of tokens), historical trajectory (continuously growing), and tool return results (webpage text, file contents, command-line outputs, often exceeding tens of thousands of tokens).

A rough estimation: assuming a single-step context stabilizes at 50,000 tokens, a 20-step task will accumulate roughly one million input tokens alone. If a task lasts for hours and involves hundreds of steps, the input for a single task can reach several million tokens. The output side is equally significant; planning text, tool call parameters, and intermediate summaries at each step all count as output tokens, and output pricing is typically significantly higher than input pricing. With both sides stacking up, quotas designed for conversational-era usage naturally cannot hold.

Therefore, “quotas depleting quickly” is not inherently because models became more expensive, but because the usage paradigm changed. The determining factor for token consumption shifted from “user input volume” to “task complexity.” This means cost governance must become a first-class citizen in agent engineering, not an afterthought.

3. Prompt Caching: The First Lever for Agent Economics

Mainstream inference services generally offer prompt caching capabilities, and their billing structures typically share three characteristics:

1. The unit price for cached input tokens is far lower than non-cached input, often by an order of magnitude;
2. The unit price for output tokens is noticeably higher than non-cached input;
3. Some services offer priority scheduling tiers, trading higher prices for lower latency and higher stability, suitable for latency-sensitive scenarios.

This is a massive boon for agentic scenarios because agent requests naturally possess a “stable prefix”: system prompts remain unchanged throughout; tool definitions remain constant during the task; and the historical trajectory grows append-only, meaning the past portion is identical to the previous step. As long as the prefix remains stable, the majority of input at each step can hit the cache, drastically reducing costs. Conversely, if dynamic content is mixed into the prefix (real-time timestamps, random IDs, changing environment variables), the cache will completely invalidate from the point of insertion onward, causing costs to immediately rise.

Engineering optimization checklist:

• Prefix Stability: Place system prompts and tool definitions at the very beginning of the message sequence, maintaining byte-level consistency; push dynamic information to the end.
• Session Affinity: Reuse the same session and routing path for a given task to prevent cross-node cache invalidation.
• Tool Result Pruning: Compress and truncate large returns like webpage text and file contents before writing them to the trajectory.
• Trajectory Summarization: Replace long-term historical text with summaries to flatten the context growth curve.

# Stable prefix: fixed content first, dynamic content last, maximizing cache hits
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},          # Unchanged throughout
    {"role": "system", "content": render_tool_schema(TOOLS)},  # Unchanged throughout
    {"role": "user",   "content": task_brief},             # Task input
    *trajectory,                                           # Trajectory grows append-only
    {"role": "user",   "content": f"Current time: {now()}"},   # Dynamic info placed at the tail
]

4. Managed Agents: Turning Runtime Scaffolding into Infrastructure

The model itself only decides “what to do next.” To make an agent truly run around the clock, a complete runtime scaffolding (often called a Harness) is required: tool registration and authentication, execution sandboxes, state persistence, failure retries, progress reporting, logging, and monitoring. A recent industry trend is exposing this scaffolding as cloud-managed capabilities for enterprises and developers, known as “Managed Agents”: developers declare goals and tools, and the platform handles long-horizon scheduling and multi-agent collaboration.

The problems this infrastructure solves are universal:

1. State Persistence and Checkpoint Recovery: Long-horizon tasks inevitably face interruptions (budget limits, failures, manual pauses). Context and intermediate artifacts must be persisted to disk as checkpoints to support resumption rather than starting from scratch.
2. Event-Driven Wakeup: 24/7 availability does not mean constant polling. Constant polling continuously consumes tokens. The correct approach is triggering via messages or timed events, maintaining zero consumption when the task is idle.
3. Multi-Agent Collaboration: Three common patterns exist: Master-Slave (an orchestrator decomposes tasks, executors do the work), Pipeline (artifacts passed sequentially), and Blackboard (shared state, parallel competition). Collaborative communication itself consumes tokens and should be budgeted.
4. Security and Isolation: Agents perform operations with real-world side effects. Sandbox execution, least-privilege access, and human approval for critical actions are indispensable.

Regarding reference architecture, enterprises can build it in-house: using container platforms for execution sandboxes, message queues for event-driven workflows, object storage for checkpoints and artifacts, and logging/monitoring services for observability. These general cloud capabilities are available across mainstream cloud providers like Baidu AI Cloud. Alternatively, teams can directly use provider-hosted model inference and managed runtime services, focusing their efforts on task design. The trade-off between these two routes lies in balancing autonomy and operational overhead.

5. Engineering Practices: Balancing Capability and Cost

5.1 Budget Governance: Turn “Burning Out” into “Circuit Breaking”. Set a token budget limit for each task. When exceeded, trigger a checkpoint and pause, rather than letting it fail blindly:

def run_agent(task, budget=2_000_000):
    while not task.done():
        if task.used_tokens() > budget:
            task.save_checkpoint()
            notify(f"Budget exceeded; paused at step {task.step}")
            return task.pause()
        step = model.decide(task.context())
        task.apply(step)   # Execute and update trajectory and usage
    return task.result()

Combined with tiered alerts (50% reminder, 80% restriction, 100% circuit breaker), this transforms cost anomalies from “after-the-fact bills” to “in-the-moment interventions.”

5.2 Model Tiering. Once tasks are decomposed, a large portion of steps are low-risk and pattern-based (format conversion, initial screening, summarization) and can be handed off to lightweight models. Key decisions (planning, complex reasoning, external outputs) should invoke flagship models. Tiered scheduling can significantly reduce per-task costs with almost no loss in end-to-end success rate.

5.3 Context Engineering. Compress tool results before entering the trajectory; use rolling summaries for distant history; expose only the subset of tools required for the current sub-task to reduce definition overhead; externalize large reference materials to Retrieval-Augmented Generation (RAG) and fetch on demand.

5.4 Reliability Design. Tool calls must be idempotent or compensable to ensure safe retries; use exponential backoff for rate limiting and network errors; save checkpoints at every step so any failure can be traced to a specific link, avoiding a total task restart.

5.5 Metric System. Track at least four metrics: task success rate, per-task token cost, human intervention rate, and cache hit rate. Cache hit rate is a direct signal of context engineering quality; per-task cost determines whether the agent can scale broadly.

6. Conclusion

The trajectory toward agentic models is clear: capability competition is shifting towards “continuous work,” delivery formats are evolving toward multi-device coverage and infrastructure, and cost structures are jointly dictated by cache billing and usage paradigms. For development teams, the real homework is not waiting for a stronger model, but mastering cost engineering: stabilizing prefixes, implementing budget circuit breakers, tiering models, and enabling checkpoint resumption. Solid execution in these areas is what transitions an agent from a “stunning demo” to a “production-ready system.”