Context Cost Control in Practice for AI Coding Assistants

1. The problem is not “asking too much,” but “carrying too much”

In many discussions about the cost of AI coding assistants, the first reactions are usually “too many questions,” “the model output is too long,” or “tool calls are too frequent.” However, when looking at the actual usage structure, the key variable that really determines cost is often not the length of a single output, but how much context must be carried with each request.

If an AI coding assistant uses a typical conversational calling pattern, it usually does not send only the current question to the model. Instead, it also sends accumulated content from the earlier conversation, including but not limited to:

• system prompts;
• tool definitions;
• historical user inputs;
• historical model responses;
• read file snippets;
• code retrieval results;
• tool call return values;
• conversation summaries or intermediate state.

This means the later the conversation goes, the longer the context becomes. Even if the current question is very small, the model may still need to re-read the entire history. In other words, the major cost may not come from “the number of questions,” but from “how much conversation must be resent each time a question is asked.”

In a 7-day local usage record containing 4,423 requests, this structure is clear: the average request carried about 233,000 tokens of context, while the actual model output was only 469 tokens. The two differ by hundreds of times. If converted using public API billing, the 7-day cost would be on the order of hundreds of US dollars. About 60% was spent on “cache reads,” meaning the earlier content was read again in every conversation turn; about 30% was spent on cache writes; and the content truly generated by the model accounted for just over 10%.

This set of data highlights an important engineering fact: cost optimization for AI coding assistants cannot focus only on “output length.” It must first address “context length” and “context reuse efficiency.”

2. Why costs concentrate on cache reads

2.1 Stateless APIs force full resending

Most large-model service APIs are stateless. The server does not permanently remember the full state of a conversation. The client must provide the context again with every request. To maintain continuity, an AI coding assistant can only package and send historical conversations, tool definitions, and code snippets.

This creates a typical linear expansion: the first request carries a small amount of context, the second carries slightly more, the tenth may already contain a large amount of code and tool results, and the hundredth may become a huge context package. The number of requests itself may not be large, but the “fixed cost” of each request continues to rise.

A simplified cost formula can help understand this:

Total cost ≈ Σ(cache read tokens × cache read unit price
          + cache write tokens × cache write unit price
          + new input tokens × normal input unit price
          + output tokens × output unit price)

Cache reads usually have a lower unit price, but the read volume is huge. Cache writes usually have a higher unit price, but they occur less often. Output tokens may be valuable, but they usually make up a small share in coding assistants. Therefore, the cost structure often looks like this: the largest volume is cache reads, the most expensive unit is cache writes, and the lowest share is output.

2.2 A high cache hit rate does not mean low cost

Model services with prompt caching usually cache repeated prefix content. If the prefix of the next request is exactly the same as the previous one, it can hit the cache and be billed at a lower cache-read price.

In the 7-day test, the cache hit rate reached 98.6%. From the hit rate alone, this ratio seems high and suggests that the caching mechanism is working well. However, the cost remained high. The reason was not a low hit rate, but the absolute volume of cache reads being too large.

This can be understood as:

Cache read cost = total cache-hit tokens × cache read unit price

If every request rereads hundreds of thousands of tokens, then even if the unit price is only a small fraction of the normal input price, the total cost after multiplying by the number of requests can still be significant. In other words, caching can lower the “unit price,” but it cannot by itself lower the “total volume.” What really determines the upper limit of cost is still the context length itself.

2.3 Cache writes are a hidden cost amplifier

Cache writes usually occur when context is newly added, when cache expires, when the prefix changes, or when the conversation structure changes. Their unit price is often higher than cache reads, and also higher than normal input. Although the share of write tokens may not be high, if writes happen frequently, the cost can be amplified quickly.

In the 7-day data, cache reads accounted for 98.4% of all tokens and about 59.9% of cost; cache writes accounted for only 1.4% of tokens, but about 27.8% of cost; output accounted for 0.2% of tokens, but about 12.3% of cost.

This structure shows that cache writes are an easily ignored cost black hole. Cache writes may increase significantly in scenarios such as:

• cache expiration after a long-running conversation;
• frequent modification of the system prompt;
• changes in tool definitions;
• regeneration of context summaries;
• changes in the order of file content;
• insertion of timestamps, random IDs, or dynamic state;
• frequent rewriting of tool return results.

If the cache is frequently invalidated, the model service may need to rebuild the cache, and cost can rise quickly. Therefore, controlling context length is not only about reducing read volume, but also about avoiding cache invalidation and repeated writes.

3. Why common token-saving methods have limited benefits

Many developers try intuitive token-saving methods, such as truncating tool output, building code indexes, compressing logs, reducing model output, or limiting the number of files read. These methods are not without value, but if the context structure itself is not governed, the benefits are often limited, and in some cases may even become negative optimizations.

3.1 Truncating tool output only reduces the increment

Truncating tool output can indeed reduce the newly added tokens in a single request. For example, compressing a retrieval result of 50,000 tokens down to 2,000 tokens looks like a large saving. However, if the historical conversation already contains a large amount of context, the benefit of this single truncation may be only a small change relative to the whole request.

Suppose the total context of a request is 200,000 tokens, of which tool output accounts for 10,000 tokens. Even if tool output is compressed from 10,000 to 1,000 tokens, it reduces total context by less than 5%. If the historical conversation itself is not compressed, overall cost will still be mainly determined by the historical context.

Therefore, tool output governance is necessary, but not sufficient. It is more suitable as a “local optimization” than as a “global optimization.”

3.2 Code indexing is not inherently cost-saving

Building indexes for code, performing symbol retrieval, or using vector retrieval can reduce situations where an entire file is sent to the model. This can indeed lower context volume. However, code indexing itself also has costs:

• index construction requires compute resources;
• index updates require continuous maintenance;
• retrieval results still occupy context;
• if too many snippets are recalled, cost may rise instead;
• when retrieval quality is unstable, the model may need multiple retries.

Therefore, the value of code indexing is not that “having an index automatically saves tokens,” but whether it can express more effective code evidence with less context. The ideal state is: the model does not need to see the entire repository, only the small number of snippets most relevant to the current task.

3.3 Frequent summarization can also add consumption

Another common approach is to let the model summarize historical conversations and then use the summary to replace the original text. This direction is correct, but if the summarization frequency, summary format, and trigger conditions are poorly designed, it can also create additional cost:

• summarization itself requires an additional request;
• the summary result will still enter subsequent context;
• if the summary is too short, key constraints may be lost;
• if the summary is too long, it approaches the original text, and the benefit declines;
• frequent summary rewriting breaks cache continuity.

Therefore, summary compression must be designed as a system engineering problem, not as a simple “summarize history” button.

4. The truly effective direction: control context length

Based on the measured data, a core conclusion can be drawn: the priority for cost optimization of AI coding assistants should be:

Control context length > improve cache continuity > reduce ineffective writes > optimize tool output > optimize model output

Among these, controlling context length is the most fundamental lever. It directly determines how much content each request carries, and indirectly affects cache hit rate and write frequency.

4.1 Context compression: from “full memory” to “structured memory”

For long-session coding tasks, it is not recommended to let the model permanently retain all historical original text. A more reasonable approach is structured context compression.

A practical compression model can include the following parts:

1. Task goal: the current problem to be solved;
2. Key constraints: interfaces, versions, style, and security requirements that cannot be modified;
3. Completed actions: which files have been modified and which commands have been executed;
4. Key decisions: why a particular solution was chosen;
5. Open issues: which risks still need further verification;
6. Relevant file list: only paths and summaries, not full text;
7. Recent raw turns: retain recent details to maintain continuity;
8. Tool call result summaries: only conclusions, paths, and key fields.

In this way, the model no longer needs to reread all original conversations every time. Instead, it can continue working based on structured state. Context volume can be significantly reduced, while task continuity is not obviously damaged.

A simplified context preparation logic is as follows:

def prepare_context(messages, max_context_tokens=120000):
    if estimate_tokens(messages) <= max_context_tokens:
        return messages
    keep_recent = 6
    old_messages = messages[:-keep_recent]
    recent_messages = messages[-keep_recent:]
    summary = summarize_task_state(old_messages)
    compact_context = {
        "task": summary.task,
        "constraints": summary.constraints,
        "completed": summary.completed_steps,
        "decisions": summary.decisions,
        "open_issues": summary.open_issues,
        "files": summary.file_summaries,
        "recent": recent_messages,
    }
    return render_compact_context(compact_context)

The key point here is not “the more compressed, the better,” but retaining the minimum sufficient information needed for the model to continue completing the task.

4.2 Cache continuity: avoid repeatedly rewriting the entire context

Cache hits depend on prefix stability. If the system prompt, tool definitions, and historical message order are frequently changed, the cache is likely to be invalidated. To improve cache continuity, the following principles can be used:

• put stable content first: system prompts, tool definitions, and long-term constraints first;
• put dynamic content later: current questions, temporary files, and tool results later;
• avoid inserting timestamps, random IDs, or temporary paths into the cached prefix;
• do not frequently modify tool schemas;
• do not change the order of historical messages in every request;
• make summary generation occur at fixed stages, rather than being triggered randomly every turn.

A common anti-pattern is:

System prompt + current time + random session ID + tool definitions + historical conversation

If “current time” and “random session ID” are placed at the front, the prefix of every request will change, cache hit rate will drop, and cache write cost will rise.

A more recommended structure is:

Stable system prompt
Stable tool definitions
Stable long-term constraints
History conversation summary
Recent raw turns
Current question
Temporary context

This allows as much content as possible to fall into a stable prefix, improving cache reuse.

4.3 Session segmentation: do not let one session extend indefinitely

Long sessions may seem convenient, but they are not advisable from a cost perspective. The longer the session, the larger the context, and the higher the base for cache writes and reads. A more reasonable approach is to split a large task into multiple stages, with each stage having an independent context.

For example, a task to “fix a complex defect” can be split into:

Stage 1: Locate the issue
Stage 2: Read related code
Stage 3: Propose a fix
Stage 4: Implement changes
Stage 5: Run tests
Stage 6: Summarize changes

At the end of each stage, generate a structured handoff summary, and the next stage only reads the necessary content. This can both keep the task moving forward and avoid unlimited context expansion.

For coding assistants, session segmentation is not a concession in user experience, but a necessary design for cost control.

4.4 Tool output governance: summarize by default, expand on demand

Tool calls are an important capability of AI coding assistants, but tool return values are often large, such as search results, logs, file contents, test results, and API responses. If all of them are sent to the model in full, context will expand rapidly.

A tiered output strategy is recommended:

Tier 1: Summary
Return only task status, key fields, exception information, file paths, and line ranges.
Tier 2: Partial expansion
When the model determines that details are needed, read part of the content by specified path.
Tier 3: Full view
Read the complete file only when truly necessary, and set a clear token limit.

For example, when retrieving code, the result can be returned as:

File: src/service/order.ts
Matched symbol: createOrder
Lines: 120-180
Summary: The main flow function for creating orders
Key dependencies: validateOrder, paymentClient, repository
Suggested read range: 120-180

Instead of directly returning the entire file content. This way, the model can still obtain the necessary information, but the context volume will be much smaller.

4.5 Code indexing: return the “minimal evidence set”

Capabilities such as code indexing, symbol retrieval, and call graph analysis can reduce the model’s dependence on full-repository context. However, index results must obey a token budget rather than unlimited recall.

A reasonable retrieval output should satisfy the following:

• only return the snippets most relevant to the current task;
• attach path, symbol name, and line number to each snippet;
• avoid returning a large amount of duplicate code;
• avoid returning complete test logs;
• for long files, only return key functions or key diffs;
• deduplicate and rank search results;
• set limits for low-confidence results.

Retrieval results should be understood as “evidence,” not as a “full export of the knowledge base.” What the model needs is enough evidence to make a judgment, not a copy of the entire repository.

4.6 Monitoring and alerting: turn token cost into an observable metric

Cost optimization is difficult to sustain without monitoring. For team-level or enterprise-level use, it is recommended to include the following metrics in the observability system:

1. Context tokens per request;
2. Cache read tokens;
3. Cache write tokens;
4. Normal input tokens;
5. Output tokens;
6. Cache hit rate;
7. Number of cache expirations;
8. Cumulative cost per session;
9. Tool-call return token distribution;
10. Context length P50 / P95 / Max.

If the following signals appear, they usually indicate problems in context management:

• per-request context continues to grow;
• the cost share of cache writes rises significantly;
• cache hit rate declines;
• average tool output tokens are too large;
• average turns per session are too high;
• summary trigger frequency is abnormal;
• the same files are repeatedly read for the same task.

In enterprise environments, general capabilities related to Baidu AI Cloud, such as object storage, log services, and monitoring and alerting, can be used to build cost observability and anomaly alerting mechanisms. The key point is not to stack tools, but to make the cost structure of every request visible, explainable, and optimizable.

5. A practical optimization framework

If the above methods are integrated into an engineering framework, they can be divided into five layers.

Layer 1: Context budget

Set a context budget for each request. For example:

Maximum context tokens: 120000
System prompt and tool definitions: 8000
History summary: 20000
Recent raw text: 60000
Tool results: 30000
Output budget: 0

When the budget is exceeded, first compress old history, then truncate tool output, and finally reduce the amount of recent raw text retained. The budget mechanism can prevent context from growing without limit.

Layer 2: Session state compression

Abstract session state into a structured object:

{
  "task": "Fix the order payment failure issue",
  "constraints": [
    "Do not modify database table structures",
    "Keep existing API compatibility"
  ],
  "completed": [
    "Located the payment callback module",
    "Confirmed the exception was caused by missing null validation"
  ],
  "decisions": [
    "Use defensive null checks",
    "Do not introduce new dependencies"
  ],
  "open_issues": [
    "Need to add unit tests"
  ],
  "files": [
    "src/pay/callback.ts",
    "src/order/service.ts"
  ]
}

Subsequent requests only send this state object and the necessary raw text, rather than sending the complete chat history.

Layer 3: Cache-friendly construction

When constructing requests, follow the stable-prefix principle:

[System prompt]
[Tool definitions]
[Long-term rules]
[Session summary]
[Recent history]
[Current input]
[Temporary context]

At the same time, avoid:

[Dynamic timestamps]
[Random session IDs]
[Temporary debug information]
[Frequently changing tool descriptions]

These unstable contents should be placed after the cached prefix or handled through a separate mechanism.

Layer 4: Tool result governance

Tool return values must undergo standardized processing:

Search results: keep only top-k, with path, line number, and summary.
Log results: keep only exception segments, first and last fragments, and key fields.
File results: return only relevant functions or diff blocks.
Test results: return only failed test cases and key errors.
Command results: truncate standard output, keep error codes and exit status.

The more structured the tool results are, the easier it is for the model to understand them, and the lower the token cost will be.

Layer 5: Continuous optimization loop

Optimization is not a one-time effort, but requires continuous observation:

Collect cost metrics
→ Analyze high-cost sessions
→ Locate context expansion points
→ Adjust compression strategies
→ Observe cache hit rate and task success rate
→ Iterate again

In particular, it is important not to sacrifice task quality in order to reduce tokens. For example, if a summary excessively removes key constraints, the model may repeat mistakes; if tool output is excessively truncated, the model may be unable to locate the problem. Therefore, the optimization goal should be “reduce cost per task while maintaining task success rate.”

6. Common mistakes and precautions

6.1 Only looking at output cost, not input cost

Many teams limit model output length, for example by requiring “answers must not exceed 300 words.” This has some help for output cost, but in coding assistants, output usually accounts for only a small part of total cost. The real major cost is the context carried by each request. Therefore, limiting output only, without governing context, has limited optimization effect.

6.2 Assuming a high cache hit rate means there is no room for optimization

A high cache hit rate only indicates good prefix reuse. It does not mean the total context volume is reasonable. If every turn carries an extremely long context, then even if the cache is hit, the read volume is still large. The optimization focus is still on reducing unnecessary context.

6.3 Over-compression causing task failure

Context compression needs to retain key constraints. For example:

• which files are prohibited from being modified;
• which interfaces must remain compatible;
• which testing requirements must be followed;
• confirmed error causes;
• already excluded solutions;
• commands currently being executed.

If this information is dropped by the summary, the model may repeat mistakes, which instead increases the cost of multiple debugging rounds.

6.4 Treating code indexing as a silver bullet

Code indexing can only reduce “read scope.” It cannot automatically guarantee the shortest context. If index results are too many, too broad, or too messy, they will still push up token consumption. The index system needs to be combined with token budgets, relevance ranking, and minimal-evidence return mechanisms.

6.5 Ignoring cache write cost

Although cache writes account for only a small part of tokens, their unit price is usually high. If sessions frequently expire, prefixes frequently change, or tool definitions are unstable, write cost may rise quickly. When optimizing, both read volume and write frequency must be considered.

7. Summary: context governance is the main line of cost optimization

From this set of measured data, a clear conclusion can be drawn: the cost structure of AI coding assistants is not dominated by “model output,” but by “the context carried by each request.” Cache read volume is huge, cache write unit price is high, and output accounts for a low share. Therefore, the optimization focus must shift from “reducing answer length” to “reducing context length, improving context reuse, and lowering the frequency of cache invalidation.”

Truly effective optimization paths include:

1. Compress historical conversations and use structured session state;
2. Control per-request context length and set clear budgets;
3. Keep cache prefixes stable and improve cache continuity;
4. Reduce cache expiration and repeated writes;
5. Apply tiered summarization to tool output;
6. Use code indexes to return the minimal evidence set, rather than full files;
7. Split long tasks into short-context stages;
8. Establish monitoring for token cost and cache hit rate;
9. Reduce tokens while maintaining task success rate;
10. Make cost optimization a continuous loop, not a one-time script.

For developers and enterprise users, the value of AI coding assistants lies not only in generating code, but also in whether they can complete complex tasks continuously and stably at a predictable cost. As long as context governance is done solidly, token consumption can be turned from a black box that “looks large but cannot be explained” into an engineering metric that is measurable, controllable, and optimizable.