Why Frequent Status Updates from AI Coding Agents Quietly Burn Tokens

Recently, the behavior of AI coding agents that repeatedly re-read the entire conversation history when generating status messages has become a popular topic in technical communities. The problem is not conceptually difficult, but it exposes a long-underestimated cost sink in long-running application agents: in order to produce a few words of progress at the bottom of a terminal, a background sub-agent may feed the entire conversation history, from task start to the current point, back into the model.

For a complex engineering task that may run for tens of minutes and accumulate tens of thousands or even over a hundred thousand tokens of context, this “read everything, write one line” pattern can turn the terminal into a token-burning machine. This article starts from how token billing works, explains why the problem occurs, and provides practical optimization approaches.

1. Where the Cost Comes From: The Billing Structure of LLM Calls

To understand why “re-reading the context” is so expensive, it is important to clarify how LLM calls are billed.

Most mainstream models follow a basic principle: input tokens and output tokens are priced separately, and in most agent scenarios, input tokens far exceed output tokens.

A typical conversational request includes the following as input:

• The system prompt, which usually contains tool definitions, behavior rules, and other fixed overhead that may already amount to several thousand tokens.
• The full trajectory of the conversation history, including user instructions, every step of the model’s reasoning, and the inputs and outputs of tool calls.
• The incremental content that needs to be processed in the current request.

For a task such as updating a status line, the actual “output” may be only a dozen or so tokens, such as “analyzing the code structure” or “running test step 3.” This creates an extremely unbalanced ratio: to produce a dozen useful tokens, the model may need to reprocess thousands or even tens of thousands of input tokens. The cost is almost entirely determined by the input side.

2. Root Cause: “Full Replay” in Stateless Calls

Large language models are stateless. They do not inherently “remember” what happened in the previous turn. For every call, the required context must be reassembled into the prompt. This is the technical root of the problem.

When a background agent periodically refreshes its status, a naive implementation may rebuild a prompt containing the entire conversation history every time. As the task progresses, the context only grows, and the input cost of each individual call keeps rising.

More importantly, there is a frequency multiplier effect:

Input cost of a single status refresh ≈ system prompt + full conversation history, which grows linearly with task duration
Status refreshes per hour = 3600 / refresh interval in seconds

For example, if the status is refreshed every 30 seconds, there will be 120 full-context re-reads in one hour. Assuming the context remains stable at 100,000 tokens, status refreshing alone would consume about 12 million input tokens per hour, while the useful information gained from those tokens is almost negligible.

This is not a model capability issue. It is an architectural flaw in context management.

3. Optimization Approach 1: Use Lightweight Notes Instead of Full Re-reads

The most direct optimization is to break the strong coupling between “status updates” and “full history.”

If the status line only needs to summarize what is currently happening and which step is underway, the agent can maintain a structured lightweight state note, such as an agent scratchpad or progress notes. Instead of using the entire conversation as input for status generation, the agent can use only this note.

The core idea is to let the agent continuously maintain a concise working-state summary during execution, rather than repeatedly re-reading all raw records.

# Bad example: replay the full history on every refresh
def refresh_status_naive(full_history):
    prompt = SYSTEM_PROMPT + full_history   # Input cost grows with the history length
    return call_model(prompt, "Describe the current progress in one sentence.")
# Optimized: read only maintained lightweight notes
def refresh_status_with_notes(progress_notes):
    # progress_notes should contain only the goal, completed steps, current step, and key conclusions
    # It is usually a few hundred tokens and is decoupled from total task duration
    return call_model(SYSTEM_PROMPT, 
                      "Update the status in one sentence based on the following progress notes: " + progress_notes)

The key difference is that the size of the note is decoupled from the absolute duration of the task. Whether the task runs for five minutes or five hours, as long as the note is well maintained, the input cost of a single status update can remain within a small, stable range.

This is also the source of the cited claim that the cost of a single status update can be reduced by about 99%. What is saved is not the output, but the input that was being repeatedly replayed.

4. Optimization Approach 2: Context Compression and Incremental Management

Lightweight notes address the high-frequency, low-value scenario of status refreshing. However, agent systems also face a more general problem: as the task progresses, the context in the main reasoning path continues to expand.

Several common industry techniques can be combined to address this.

1. Summarization-based context compression

When the conversation history exceeds a certain threshold, older parts can be compressed into a structured summary that retains key decisions, code changes, and conclusions, while discarding redundant intermediate steps. Subsequent calls then carry a high-information-density summary plus recent raw context, rather than the full conversation log.

2. Retrieval-augmented context

Instead of injecting all history into every call, the system can retrieve only a small number of the most relevant historical fragments based on the current task. This requires the system to build a searchable index over the context.

3. Externalizing state

Task state, completed steps, and todo lists can be stored outside the model, such as in memory, cache, or a storage service. The model then reads only the necessary slice for each call, rather than carrying the entire state in the prompt.

5. Optimization Approach 3: Use Prompt Caching for Stable Prefixes

It is also worth noting that modern inference services often support prompt caching or prefix caching. Content that remains unchanged across multiple calls, such as system prompts and tool definitions, can be cached and reused, avoiding repeated billing and repeated computation of the same fixed prefix.

From an engineering perspective, prompts should be organized into a “stable prefix + dynamic suffix” structure. Place unchanged rules, tool schemas, and other stable content at the front to improve cache hit rates, and place changing history and incremental content after them. This complements the idea of sending only incremental context instead of full context, and together they help reduce the cost of repeated input.

6. A Practical Checklist for Token Cost Governance

The above ideas can be turned into a practical checklist for teams building long-running agents:

• Separate high-value calls from low-value calls. High-frequency, low-value calls such as status refreshing and progress broadcasting should never use the full context; they should use lightweight notes or state slices instead.
• Design different context budgets for different calls. Main reasoning requires richer context, while auxiliary tasks should be kept as concise as possible. A single prompt template should not be used for everything.
• Build automatic maintenance of state summaries. Let the agent update the summary whenever a milestone is completed, rather than re-reading all raw history when needed.
• Set context thresholds and trigger compression. When historical tokens exceed a configured value, automatically perform summary compression to prevent unbounded growth.
• Use observability metrics to monitor cost. Feed metrics such as input tokens, output tokens, and cache hit rate into monitoring and alerting systems to detect abnormally amplified calls early.

Conclusion

This discussion reveals a fact that is easy to overlook: in agent systems, what really burns money is often not the model’s “thinking,” but the “memory” being repeatedly moved around. When the architecture builds every small output on top of a full re-read, cost can grow out of control as the task duration increases.

The solution is not mysterious. Separate state from raw history, and replace blind full replay with lightweight notes, context compression, and cache reuse. This allows compute to be directed precisely to the places that genuinely need reasoning.

For teams building long-running, multi-step agent applications, this context cost governance strategy is worth considering from the first day of architectural design.