Managing LLM Token Consumption: Reasoning Budgets and Chain-of-Thought Control

    This article explains why large language models (LLMs) generate increasingly long outputs, how to manage token consumption through reasoning budgets and chain-of-thought (CoT) control, and why output truncation is not equivalent to a wrong answer. It covers the difference between max_tokens and context windows, practical strategies for tiered token allocation, and a Python example for implementing dynamic budget assignment across task complexity levels. Readers will learn how to evaluate truncated outputs properly and report benchmark scores in a way that separates model capability from budget limitations.

    As large language models become more capable, the internal reasoning processes they rely on to solve complex problems have grown significantly longer. These reasoning processes consume substantial numbers of tokens, which directly impacts API call costs and response latency, and can also skew evaluation results. Recent industry discussions about a popular open-source model’s new version have centered on its performance in both reasoning and non-reasoning modes, with one key point of debate being how output truncation—triggered when token consumption exceeds the budget—affects scoring. This highlights that reasoning budget management deserves as much attention as accuracy improvement.

    1. Why “Non-Reasoning” Mode Can Also Produce Longer Outputs

    Traditional non-reasoning models typically generate answers directly without displaying explicit step-by-step reasoning. However, some newer versions of non-reasoning mode behave similarly to “adaptive reasoning models with a lower thinking budget.” Simple questions still receive short, quick answers, but when faced with multi-step logic, complex code, or long-text computation, the model spontaneously expands into intermediate reasoning, noticeably increasing the output length.

    The magnitude of this change depends on the model version, task type, and input length—there is no single fixed number that captures it across all scenarios. Teams should therefore measure the output token distribution on their own task sets, paying close attention to the median and the 95th percentile (P95), rather than relying only on the average. Averages hide the long tail, and it is exactly the long-tail requests that are most prone to hitting truncation limits.

    2. max_tokens vs. Context Window: Two Distinct Limits

    max_tokens limits how many output tokens a single request can generate at most. The context window, on the other hand, constrains the total length of input plus output. Both limits apply simultaneously: even if max_tokens is set very high, a long input consumes space in the context window, leaving less room for the output.

    When setting max_tokens, three guidelines are recommended:

    • First, estimate the input length and verify that input plus output stays within the context window.
    • Second, assign budgets by task complexity levels rather than using one value for every request.
    • Third, enforce a platform-level hard cap to prevent any individual request from consuming unbounded resources.

    3. Hitting the Limit Is Not the Same as Answering Incorrectly

    When the output reaches max_tokens, generation is forcibly stopped. Most API interfaces return a finish reason in the response—for example, finish_reason set to length indicates the generation stopped due to the length limit, not because the model naturally completed its answer. Such results lack a final answer and should be tagged as “incomplete (truncated),” not automatically recorded as “wrong.”

    When scoring evaluation results, it is necessary to track three categories separately:

    • Completed and correct;
    • Completed but incorrect;
    • Incomplete (truncated).

    Different evaluation protocols handle truncation differently: some assign zero points, some exclude the sample entirely, and some allow re-running the request. Regardless of the approach, the report should explicitly state the method used and also provide both the completion rate and the accuracy rate on completed samples. This allows readers to determine whether a low score stems from insufficient capability or from budget constraints. If only a single aggregate score is reported and truncated items are scored as zero, the overall score can be significantly dragged down by the budget setting alone, leading to misinterpretation of the conclusions.

    4. The Long Chain-of-Thought Problem in Reasoning Mode

    Reasoning models break down problems through explicit chains of thought. The longer the chain, the longer the output. The relationship is not strictly exponential; it more closely resembles an upward trend as the number of reasoning steps increases, sometimes showing stepwise growth. For a small number of extremely difficult problems, the model may repeatedly verify certain steps, leading to “overthinking” that pushes the output beyond the preset limit.

    For example, even with a limit set to 80K tokens, a particularly hard problem may still produce a chain of thought exceeding that value. Note that 80K here is just one illustrative example—the actual threshold should be determined from your team’s measured distribution on real tasks.

    The impact of truncation on the median score depends on how truncated results are scored. We recommend reporting both the median score when truncated items are included and when they are excluded. The gap between these two numbers itself reflects the degree of budget insufficiency.

    5. Dynamic Token Budget Allocation

    For evaluation or production API calls, a one-size-fits-all cap is rarely ideal. A practical approach is to use a lightweight pre-classification step—such as intent recognition or a complexity classifier—to assign smaller budgets to simple requests and larger budgets to complex tasks. Below is a simplified example:

    1. from typing import Optional
    2. BUDGET_BY_LEVEL = {
    3. "simple": 4000,
    4. "complex": 32000,
    5. "extreme": 64000,
    6. }
    7. DEFAULT_BUDGET = 8000 # conservative budget used when classification fails or confidence is low
    8. SAFETY_MARGIN = 0.9 # reserve headroom for input-length estimation errors
    9. PLATFORM_HARD_CAP = 64000 # platform-level hard cap, applied regardless of classification result
    10. def get_max_tokens(
    11. task_complexity: Optional[str],
    12. input_tokens: int,
    13. context_window: int,
    14. ) -> int:
    15. """Return the max_tokens value that should be used for this request."""
    16. if task_complexity and task_complexity in BUDGET_BY_LEVEL:
    17. budget = BUDGET_BY_LEVEL[task_complexity]
    18. else:
    19. budget = DEFAULT_BUDGET
    20. # Apply the safety margin to account for input-length estimation error
    21. budget = int(budget * SAFETY_MARGIN)
    22. # Ensure the output budget does not exceed available space in the context window
    23. available_for_output = context_window - input_tokens
    24. budget = min(budget, available_for_output)
    25. # Enforce the platform-level hard cap
    26. budget = min(budget, PLATFORM_HARD_CAP)
    27. # Ensure the budget is at least 1 token
    28. return max(budget, 1)

    In production, the classification layer can be a lightweight model or heuristic rules. The budget levels, safety margin, and hard cap should be calibrated empirically against your own workload distribution and cost constraints. This granular approach lowers waste on easy queries while preserving sufficient headroom for the hard ones, reducing the chance of premature truncation where it matters most.