Long-Running AI Agents: Cost Optimization and Stability Engineering Practices
Teams often begin by evaluating model response quality, but production agents quickly encounter repeated data reads, lost state, failed tool calls, uncontrolled retries, and rising costs. This article presents general engineering practices for building stable, observable, low-cost agent workflows. It explains why long-running tasks differ from ordinary chat, how to reduce wasted work through incremental processing and deterministic code, how to use state machines and idempotency for reliability, and how to govern context and monitor both cost and quality. It also provides an implementation roadmap and practical evaluation criteria. The focus is on making agents reason only where necessary, rather than on a specific model or platform.
In many automation projects, teams initially focus on answer quality: whether the model can understand natural language, write an accurate summary, or generate an executable plan. But once the model enters production and takes on scheduled collection, data filtering, API calls, message delivery, and similar tasks, problems usually surface quickly.
The model may perform well for the first few rounds and then repeatedly read the same batch of data after running for a while. It may forget completed steps and mistake “create a record” for “delete a record.” It may also retry repeatedly after a tool call fails, causing request volume and costs to keep growing. Clearly, chat capability and long-chain execution capability are not the same thing.
This article does not discuss a specific model or platform. Instead, it starts from general engineering practices and explains how to build stable, observable, low-cost agent workflows.
1. Why Long Tasks Are Harder Than Ordinary Chat
Ordinary chat usually has three characteristics: relatively few interaction turns, a relatively concentrated context, and a result that is mainly determined by one generation. A long-running agent, by contrast, must keep working, often involves multiple external systems at the same time, and is affected by networks, permissions, data changes, and tool failures during execution.
1.1 Tool Calls Easily Form Long Chains
A seemingly simple task such as “collect news and send a summary” may actually include the following steps:
- Read the last execution time;
- Request the data source;
- Retrieve data page by page;
- Remove content that has already been processed;
- Filter by keywords;
- Sort candidate content;
- Generate a summary;
- Check the summary format;
- Send it to the messaging channel;
- Save the execution state for this round.
If any one of these steps fails, it can affect the downstream flow. If the model does not have a clear state management mechanism, it may repeat earlier steps or even restart from the beginning after a failure, causing a large number of invalid calls.
1.2 Long Context Creates Hidden Costs
Many implementations include the full system prompt, the entire conversation history, and previous tool results in every request. This is simple in the short term, but when a task runs for dozens or even hundreds of rounds, the context keeps expanding.
Context expansion creates three types of problems:
- The number of input tokens increases, raising the cost of each request;
- Important state is buried in a large amount of historical information, making misjudgment more likely;
- Requests take longer, increasing the probability of timeouts and retries.
Therefore, conversation history cannot be treated directly as a database. Conversations are suitable for expressing process, while structured state is suitable for supporting long-term operation.
1.3 Complex Instructions Increase Execution Drift
Long tasks usually contain multiple constraints, such as “only process new content from the past hour,” “select no more than three items,” “keep the summary under one hundred words,” “do not send duplicates,” and “retry only twice after failure.” If all these conditions are packed into one natural-language paragraph, the model may understand the general idea but miss one of the restrictions.
A more reliable approach is to split the task into explicit stages and define the input, output, and completion condition for each stage. For example:
Stage A: Fetch dataInput: Last successful processing timeOutput: Raw data listCompletion condition: Return a structured list with unique identifiersStage B: Deduplicate and filterInput: Raw data list, historical unique identifiersOutput: Data list to be processedCompletion condition: Does not contain already processed recordsStage C: Content generationInput: Data list to be processedOutput: Summary in a fixed formatCompletion condition: Count does not exceed the limit, length meets requirementsStage D: Send and persistInput: Summary contentOutput: Send result, latest stateCompletion condition: Write the processing record after a successful send
This design reduces the model’s room for free-form behavior and also makes it easier to locate errors later.
2. The Core of Cost Reduction: Eliminate Wasted Work, Not Just Lower Unit Prices
Many teams first think about switching models or compressing prompts, but total cost is usually driven more by invalid calls, duplicate calls, and excessive calls.
A simplified formula can be used to estimate the cost of long-running tasks:
Total cost ≈ number of runs × input cost per run+ number of tool calls × tool cost+ number of failed retries × execution cost per attempt+ concurrency and storage costs
If a task runs once per hour, but there is usually only a small amount of new data, and every run still performs a full fetch, full analysis, and full generation, most resources are spent on repeated work. Optimization should focus on the following areas.
2.1 Move from Full Processing to Incremental Processing
The system should store a stable checkpoint, such as:
- Last successful execution time;
- Unique identifiers of processed content;
- Current pagination position;
- Most recent send result;
- Workflow version number.
On the next run, the system should read only data after the checkpoint rather than scanning all content again. For data that cannot be read incrementally by time, deduplication can also be performed through unique identifier sets, hash values, or cursors.
Note that state must be updated after the key step succeeds. It should not write the latest time immediately after fetching data; otherwise, if summary generation fails later, that batch of data may be incorrectly skipped.
2.2 Use Deterministic Logic for Simple Work
Tasks such as deduplication, sorting, field validation, time comparison, and count limits should not all be handed to the model. They are better handled by program code.
For example:
items = fetch_data(after=last_checkpoint)new_items = filter_by_id(items, processed_ids)matched_items = filter_by_keywords(new_items, keywords)selected_items = sort_and_limit(matched_items, limit=3)
The model is better suited to semantic understanding, content summarization, and complex judgment, not to mechanical logic that can be completed with a few lines of code. This reduces the number of calls and improves repeatability.
2.3 Adopt a Tiered Model Strategy
Different steps do not need models of the same complexity. A tiered strategy can be established:
- Rule layer: handles time, format, deduplication, and threshold judgment;
- Lightweight model layer: performs classification, keyword matching, and simple rewriting;
- High-capability model layer: handles complex reasoning, conflict judgment, and high-quality content generation;
- Human confirmation layer: handles high-risk or irreversible operations.
For example, news headline filtering can first use keywords and lightweight classification, and only when there are too few candidates or semantic ambiguity exists should the task be passed to a higher-capability model. This reduces high-cost calls and avoids sending unnecessary data to the model.
3. Stable Operation Requires a State Machine
Compared with “letting the model remember which step it has reached,” a state machine is a more reliable engineering solution. A workflow can be defined as:
Pending → Data Fetch → Filtering Complete → Summary Generation → Ready to Send→ Send Successful → Archived↘ Retry on Failure → Human Intervention
Each state should include the following information:
- Current step;
- Input data version;
- Execution time;
- Retry count;
- Error reason;
- Next action;
- Whether repeated execution is allowed.
Idempotency Is Critical
If the system suffers a network interruption after sending a message but before saving state, the next run may send the same content again. Therefore, send operations need an idempotency mechanism. For example, generate a unique business identifier for each task and check whether that identifier has already been successfully processed before sending.
Similarly, operations such as writing to a database, creating tasks, and generating files should be designed so that “repeated execution does not produce additional side effects.”
Retries Must Be Bounded
Retry strategies should distinguish between error types:
- Network timeout: retry a limited number of times;
- Temporary service unavailability: use gradual delay;
- Parameter error: do not repeat the request; record it and alert directly;
- Permission error: requires human handling;
- Content validation failure: return to the generation stage instead of redoing the entire workflow.
A common retry method is exponential backoff, in which the interval between retries increases gradually, with a maximum number of attempts and a maximum wait time.
4. Reduce Token Waste Through Context Governance
Context governance is not simply “deleting history.” It means retaining the information that is genuinely useful for the current task.
4.1 Use Summaries Instead of Full History
For completed steps, keep only a structured summary:
{"task_id": "task identifier","last_success_time": "last successful time","processed_count": 18,"pending_count": 2,"last_error": null,"workflow_version": "version number"}
Compared with carrying dozens of rounds of natural-language conversation, this approach is shorter, more stable, and easier for programs to read.
4.2 Control the Size of Tool Results
Tools should not return irrelevant fields, nor should they put complete original text into the context. They can return only:
- Unique identifier;
- Title;
- Time;
- Summary;
- Source type;
- Necessary fields related to the task.
If full text really needs to be viewed, it can be read on demand rather than loaded in the first round.
4.3 Set Context Budgets
Each step can set input limits, such as the maximum number of records to process, the maximum number of characters to retain per record, and the maximum number of days to retain historical state. When limits are exceeded, apply truncation, batching, or degraded processing.
5. Monitor Cost and Quality to Avoid Silent Runaway
A long-running agent cannot monitor only “whether the task succeeded.” It should also monitor resource and quality metrics at the same time.
At a minimum, record:
- Duration of each run;
- Number of model calls;
- Number of input and output tokens;
- Number of tool calls;
- Number of retries;
- Number of empty runs;
- Number of duplicate processing operations;
- Send success rate;
- Summary length and format compliance rate;
- Average cost per task.
“Empty runs” deserve particular attention. If the system checks once per hour but there is no new data most of the time, consider extending the check interval, using event triggers, or first determining through a low-cost method whether any change exists.
6. How to Evaluate Whether Optimization Works
Do not judge cost reduction only by intuition. Establish comparison metrics before and after optimization, and use the same batch of tasks for comparison.
Focus on observing:
- Whether the average number of calls per task decreases;
- Whether duplicate reads and duplicate sends decrease;
- Whether the average recovery time after failure is shortened;
- Whether summary quality remains stable;
- Whether the error rate increases after long-running operation;
- Whether the processing cost per thousand records decreases.
If model calls are reduced but missing data, incorrect sends, or additional manual rework increase, then the optimization is not truly effective. The ideal state is to achieve the same business goal with fewer calls while maintaining traceability, recoverability, and auditability.
7. Recommended Implementation Path
For new projects, implement in the following order:
First, split tasks into clear stages and define the input, output, and failure handling for each step.
Second, separate state from conversation history and store checkpoints in a database, cache, or persistent file.
Third, move deterministic work such as deduplication, sorting, format validation, and count limits out of the model.
Fourth, introduce incremental reading and idempotency design to avoid duplicate processing and duplicate sends.
Fifth, tier models according to task complexity: use lightweight solutions for simple tasks and call high-capability models only for complex tasks.
Sixth, add monitoring for call volume, tokens, errors, retries, and business results.
Seventh, use real operational data for continuous tuning rather than relying only on a few demonstration samples.
Overall, the core of agent cost optimization is not to make the model “work harder,” but to make it work only where reasoning is truly needed. Use incremental processing to reduce input, state machines to ensure continuity, deterministic code to replace mechanical judgment, bounded retries to prevent runaway behavior, and a complete monitoring and audit system. Only then can a long-running agent gradually evolve from a one-off demonstration tool into a stable, maintainable, and scalable production system.
