Taming Token Costs in Agent Applications: Attribution, Optimization, and Model Selection

If you maintain a persistent agent application—one with a fixed persona, long-term memory, scheduled heartbeat tasks, and tool-calling capabilities—you have almost certainly faced the problem of token consumption far exceeding expectations. A common first reaction is to switch to a cheaper, more cost-effective model. But practice repeatedly shows that the model’s unit price is rarely the root cause of runaway costs. The real cost drivers typically hide in three places: bloated session context, abused heartbeat tasks, and uncontrolled memory growth. This article follows a sequence of “attribute first, then govern, then select the model” to provide a practical cost optimization playbook.

1. Attribute Costs First, Don’t Switch Models First

Every response from a persistent agent is not a “bare API call.” Even a seemingly simple conversation typically bundles the following into each request:

1. System prompt and persona definition: Injected on every request, and often lengthy.
2. Tool definitions: The more tools registered, the higher the fixed system overhead.
3. Conversation history: The entire dialogue history is replayed with each request.
4. Memory content: Long-term memory, workspace files, and similar items are concatenated into the context.
5. Scheduled task wake-ups: Heartbeat requests generate traffic even when no user is speaking.

It is recommended to add a token accounting layer at the application level, estimating input/output size for each request and tagging it by session and task type:

def estimate_tokens(text: str) -> int:
    """Rough estimate: ~1 Chinese character ≈ 0.6–1 token, ~4 English characters ≈ 1 token"""
    cjk = sum(1 for ch in text if '\u4e00' <= ch <= '\u9fff')
    other = len(text) - cjk
    return int(cjk * 0.8 + other / 4)

After running this for a few days, most teams find the problem concentrated in two areas: long sessions that get more expensive the longer they run, and heartbeat tasks that keep accruing charges even when no one is interacting. Once the attribution is clear, governance can be targeted effectively.

2. Session Management: Split Tasks and Control Context Bloat

The inference cost of conversational models is tightly coupled with context length, and conversation history grows linearly with each turn. Critically, most chat APIs resend the full history with every request: the 50th turn carries all previous 49 turns, meaning the same content is repeatedly billed. The longer the session, the higher the marginal cost of each additional turn.

Practical recommendations:

1. Split sessions by task boundaries. When a task is completed or an issue is resolved, proactively start a new session rather than mixing work and casual chat in a single thread. Long-lived mixed sessions force large volumes of irrelevant history to “ride along” in every subsequent request.
2. Apply in-session summarization. When history exceeds a threshold (e.g., 20 turns or several thousand tokens), have the model compress the earlier content into a summary that replaces the original text, preserving conclusions while discarding process details.
3. Keep large files and long outputs out of history. Large logs or long documents returned by tools should be used and discarded immediately, leaving only the conclusions in context.

3. Heartbeat Tasks: The Biggest Hidden Cost Source

Many persistent agents include a heartbeat mechanism: a periodic automatic wake-up to check messages or run inspections. What’s easily overlooked is that every heartbeat is a full session—persona, memory, and tool definitions are all injected, even if the heartbeat does nothing. Optimization approaches, ordered from lowest to highest impact:

1. Reduce frequency and use conditional triggers. Change “every 5 minutes” to “every 30 minutes,” or shift to event-driven waking: only wake up when there are unhandled messages or file changes.
2. Minimize heartbeat payloads. Heartbeat scenarios do not need the full persona or entire memory store. Maintain a separate, minimal system prompt for this purpose.
3. Script recurring tasks. This is the highest-impact change: any task that runs on a fixed schedule with fixed steps should be converted into a script written by the model. The script then executes directly, and only the result summary is passed back to the model—ideally, only when an anomaly occurs.

# Example heartbeat governance configuration
heartbeat:
  interval_minutes: 30          # Reduce trigger frequency
  trigger: event_or_schedule    # Only wake up when unhandled events exist
  prompt_profile: minimal       # Use a minimal system prompt; omit full persona and memory
  report: on_anomaly_only       # Normal results do not enter the conversation; anomalies are reported

The difference before and after scripting is substantial: the model participates only in “writing the script” and “reviewing anomalies.” All repetitive work in between is handled by ordinary code, which costs nearly nothing to execute.

4. Memory Governance: Don’t Let Context Carry Historical Baggage

Long-term memory is the value proposition of persistent agents, but memory that only grows and never shrinks makes every request more expensive and also dilutes key information, degrading response quality. A three-layer management approach is recommended:

• Working memory: Temporary state for the current task; cleared when the task ends.
• Long-term memory: Stable facts such as user preferences and project background; kept to a limited number of entries.
• Archive: Low-frequency but potentially useful history, stored in workspace files and retrieved on demand, rather than resident in context.

Two supporting actions are important: first, periodically ask the agent to organize its own memory (merge duplicates, remove stale entries, compress verbose descriptions); second, directly edit memory files by hand. In many frameworks, memory is essentially a set of text or structured files—editing these directly is more precise and controllable than conversational management.

5. Model Selection: From “Which Is Cheaper” to “Which Model for Which Task”

Only after the governance steps above should you turn to model selection. The evaluation dimension should expand from “unit price” to “total cost of ownership”:

1. Unit price × call volume. Input and output prices differ, and the longer the context, the higher the share of input tokens. After session governance reduces actual consumption, the appeal of chasing the lowest-priced model diminishes.
2. Capability fit. Focus on four aspects: long-context support (determines how long sessions can run), multimodal capability (whether images or files need to be read), tool-calling and structured-output reliability (determines whether the agent can run dependably), and instruction-following ability.
3. Compatibility testing. Different models vary significantly in how well they adapt to agent frameworks. Some models generate field names or output formats that deviate from the framework’s expected schema during tool calls, causing parsing failures or even process exceptions. Before selecting a model, run regression tests using your actual workflow—do not rely solely on public benchmarks or third-party recommendations.
4. Quotas and pricing plans. Various providers—including cloud vendors’ LLM services—typically offer pay-as-you-go, resource packages, and developer subscription plans, each with different quota and rate-limit policies. Choose based on your call-volume curve, and be cautious of plans with limited quotas that high-frequency agent workloads may exhaust quickly.

A more effective strategy is tiered routing: not every request needs to go to the same model.

def route(task) -> str:
    if task.type in ("summarization", "format conversion", "simple Q&A"):
        return "SMALL_MODEL"     # High-frequency light tasks go to a low-cost model
    if task.needs_multimodal or task.context_tokens > 50_000:
        return "LARGE_MODEL"     # Long-context or multimodal scenarios use a flagship model
    return "DEFAULT_MODEL"

For example, scenarios requiring native multimodal support and long context can prioritize a capable model from a cloud provider’s LLM service. Pure code and text tasks, by contrast, can use a model without multimodal capabilities but with stable tool-calling behavior. A multi-model combination is often both cheaper and more reliable than a single-model strategy.

6. Engineering Cost-Reduction Checklist

Here is a directly executable checklist:

• Record input/output tokens for every call; tag by session and task type.
• Auto-summarize or open a new session when a long session exceeds the threshold.
• Reduce heartbeat frequency, use conditional triggers, and minimize prompt size.
• Script all periodic tasks; the model handles only exceptions and decisions.
• Layer memory; organize and archive regularly.
• Truncate and extract fields from large tool outputs before they enter context.
• If the provider supports context caching, reuse repeated prefixes to reduce input costs.
• Set daily/monthly token budgets with alerts to catch anomalous growth immediately.

7. Conclusion

Token cost optimization is fundamentally an exercise in context engineering: first, ensure each request carries only the necessary minimal information; then, move fixed workflows off the model and into scripts; finally, use tiered routing to match different tasks to appropriate models. Following this sequence, most persistent agent applications will see significant consumption reductions—and only then will comparing model prices and capabilities across providers become a clear and meaningful decision.