Evaluating LLM Token Consumption and Optimizing Usage in Multi-Turn Conversations

1. A Common Problem: Why the Quota Runs Out Before the Conversation Is Finished

In subscription-based LLM services, users pay for a fixed billing cycle and receive a corresponding usage quota. Many developers and heavy users have encountered the same situation: during a multi-turn conversation, the quota is consumed much faster than expected, and the task is interrupted halfway by rate limiting or a usage-limit notice. The only options are to wait for the next cycle or upgrade the plan.

This is not simply a matter of “the quota being too small.” There is a technical logic behind it that can be analyzed quantitatively. Three key facts matter:

1. LLM services generally use tokens as the basic unit for metering, billing, and rate limiting.
2. Multi-turn conversations have a context-accumulation property: each request usually carries all previous history.
3. This accumulation causes total consumption to grow approximately quadratically with the number of turns, while most users intuitively expect linear growth.

Once these three points are clear, it becomes easier to understand why “the conversation did not seem long, but the tokens disappeared.”

2. Understanding Tokens: Metering Principles and Request Composition

What Is a Token?

An LLM does not directly process characters or words. A tokenizer first splits text into tokens, which are the smallest units the model can process. Based on common industry experience, English text is roughly one token per four characters, while Chinese text is roughly one to 1.5 tokens per character. Different tokenizers produce different results. This also explains why, for the same “one thousand characters,” Chinese content usually has a higher token cost than English content.

What Consumes Tokens in a Single Request?

• Input side: system prompt (role setting and rule constraints) + conversation history + current user input + injected external materials (such as retrieved document snippets)
• Output side: the model-generated response
• Total consumption: input tokens + output tokens

Two Concepts That Are Often Confused: Context Window and Usage Quota

• Context window: the maximum number of tokens a single request can contain. It determines “how long a single conversation can be.”
• Usage quota: the total number of tokens allowed within a unit of time or a billing cycle. It determines “how much conversation can be handled in total.”

Many people assume that a large context window means they will not exceed the limit. In practice, the two are independent: no matter how large the window is, service can still be rate-limited once the quota is exhausted.

3. The Context Amplification Effect in Multi-Turn Conversations: An O(n²) Problem

Let the system prompt be S, the average new input per turn be I, the average output per turn be O, and the total number of turns be n. The input for turn i is approximately:

input_i ≈ S + (i-1) × (I + O) + I

This is because the model needs to “re-read” the entire history. The total consumption across n turns is approximately:

total ≈ n × S + n(n+1)/2 × (I + O)

In other words, consumption grows quadratically with the number of turns n. A short code example can verify this:

def session_total_tokens(n_turns: int, sys_tokens: int,
                         avg_in: int, avg_out: int) -> int:
    """Estimate total token consumption for an n-turn multi-turn conversation (input + output)"""
    total = 0
    for i in range(1, n_turns + 1):
        # Turn i must carry: system prompt + full history of previous i-1 turns
        carried = sys_tokens + (i - 1) * (avg_in + avg_out)
        total += carried + avg_in + avg_out
    return total
# Example: 500-token system prompt, 200 new input and 300 output per turn, 20 turns
print(session_total_tokens(20, 500, 200, 300))  # about 115000

A quick intuitive estimate might be: 20 turns × about 500 tokens per turn ≈ 10,000 tokens. The actual result is about 115,000 tokens, an order of magnitude higher. If capabilities such as long-document upload or long-term memory are added, the baseline increases further. This is the fundamental reason why conversations can hit the limit halfway through.

4. How to Estimate and Monitor Token Consumption

1. Quick estimation: For planning, use rules of thumb—Chinese text at roughly one to 1.5 tokens per character, English text at roughly one token per four characters—then multiply by the expected number of turns and adjust using the quadratic formula above.
2. Exact measurement: Major model services generally include input and output token usage fields in their responses. Server-side statistics should be treated as authoritative; client-side estimates are only for pre-planning.
3. Build usage logs and dashboards: Record the time, business scenario, input/output tokens, and cache-hit status of each call. Then use general logging and monitoring/alerting capabilities to aggregate metrics such as average daily consumption, per-session consumption distribution, and the top N high-consumption scenarios.
4. Assess plan fit: First measure the average daily consumption over one to two weeks, then compare it with the cycle quota of the plan being used. It is advisable to reserve 30%–50% headroom for peaks, so that a plan does not look sufficient on average but exceed the limit during peak usage.

5. Eight Engineering Practices for Reducing Token Consumption

1. Streamline the system prompt: Merge duplicate rules and remove redundant examples. The system prompt is a fixed cost paid on every turn, so reducing it often produces the highest return.
2. Context trimming and sliding windows: Keep only the most recent N turns of original text plus key state, such as user preferences and confirmed conclusions. Move older content out of the context.
3. History summarization and compression: When a conversation exceeds a threshold, have the model compress older dialogue into a structured summary, replacing thousands of tokens of original text with a few hundred tokens.
4. Constrain output length and format: Request bullet-point or structured output, and set a reasonable maximum generation length to avoid excessively long model responses.
5. Use retrieval augmentation instead of feeding full documents: Store knowledge externally in a vector database or document service, retrieve only the most relevant snippets on demand, and inject them into the context instead of inserting an entire document into every turn.
6. Caching and reuse: Enable caching for identical prefixes, such as fixed system prompts, and for frequently asked questions. Cache hits can significantly reduce repeated consumption.
7. Model tiering and routing: Route lightweight tasks such as summarization, classification, and format conversion to smaller models. Use flagship models only for complex reasoning, balancing cost and quality.
8. Session lifecycle management: Split sessions by task and avoid extending a single session indefinitely. Archive sessions promptly when tasks are complete so that historical baggage does not keep growing.

6. Building Usage Management and Cost Control

1. Budgets and threshold alerts: Set a periodic token budget for each project or business line. Trigger alerts when consumption reaches thresholds such as 70% and 90%, rather than reacting passively after rate limiting occurs.
2. Tiered quotas: Protect core production traffic with baseline quotas and rate-limit experimental and debugging traffic. This prevents abnormal loop calls in debugging code from exhausting the overall quota.
3. Anomaly detection: Monitor outliers in per-request token usage and promptly detect engineering defects such as “forgetting to truncate history” or “retrying indefinitely after failure.”
4. Use platform capabilities: When integrating model services on cloud platforms such as Baidu AI Cloud, you can often combine the platform’s general capabilities for usage statistics, logging, and monitoring/alerting to turn the mechanisms above into automated dashboards and alert pipelines.

7. Conclusion: Turn “Quota Anxiety” into a Computable Engineering Problem

Exceeding token limits is not mysterious. It is a quantifiable, predictable, and optimizable engineering problem. A three-step approach is recommended:

1. Quantify first: Use estimation formulas and usage logs to understand actual consumption.
2. Diagnose next: Identify the sources of amplification—whether it is an overly long system prompt, an endlessly extended session, or full-document feeding.
3. Optimize last: Combine trimming, summarization, retrieval augmentation, caching, and model tiering/routing to bring consumption back to a reasonable range.

For individual users, compressing prompts, starting new sessions regularly, and controlling output length can often make the same quota last several times longer. For enterprise teams, token management should be incorporated into cost governance, with data-driven model selection and quota planning. When you can accurately answer “how many tokens does my application consume per thousand conversations,” the usage limit is no longer a threat hanging overhead—it is just another engineering parameter.