Choosing LLMs by Token Usage: Measurement Scope, Tracking, and Selection

    This article explains why token usage is a more accurate measure of LLM workload than request counts. It covers how measurement scope, such as input versus output tokens and time window, changes usage share results. It also describes how to build a reliable usage tracking pipeline and why usage rankings alone should not drive model selection.

    1. Why Look at LLM Usage Through Tokens

    LLM services are typically billed by tokens. A token is the basic unit a model uses to process text. It can correspond to a word, a character, or a subword segment, and the exact splitting rules are set by each model’s tokenizer. The same Chinese text can produce noticeably different token counts across models. For this reason, measuring usage in tokens reflects the actual computational load better than counting requests. A single request may consume only a few dozen tokens, or it may carry tens of thousands of tokens of long context. Counting requests hides this difference.

    Industry reports often cite two types of metrics. One is developer scale, which reflects the breadth of adoption. The other is the share of token calls, which reflects the depth of actual use. The two answer different questions. Developer counts measure registered or connected accounts, which does not show whether each account keeps calling the service. Token share measures actual consumption, but it depends heavily on how the statistic is defined. Readers need to understand the definition first, and then judge the conclusion.

    2. How Measurement Scope Shapes Conclusions

    The same dataset can lead to very different conclusions depending on how it is measured. When interpreting share-based data, first confirm the following factors.

    • Scope: Does it cover all models, or a specific category, such as open-weight models or closed-source models?
    • Time window: Is it daily, weekly, or a cumulative value up to a given date?
    • Metered units: Does it count input tokens only, output tokens only, or both? Output tokens usually cost more, so counting only input tokens underestimates the computational cost of the service.
    • Data source: Is it published by the service provider, collected by a third-party monitor, or sampled at the gateway?

    The example code below shows how including or excluding output tokens affects the shares.

    1. # Example: aggregate token usage by model family and calculate shares
    2. usage_records = [
    3. {"model_family": "family_a", "input_tokens": 1200, "output_tokens": 300},
    4. {"model_family": "family_b", "input_tokens": 800, "output_tokens": 500},
    5. {"model_family": "family_c", "input_tokens": 400, "output_tokens": 100},
    6. ]
    7. def token_share(records, include_output=True):
    8. totals = {}
    9. for r in records:
    10. total = r["input_tokens"] + (r["output_tokens"] if include_output else 0)
    11. totals[r["model_family"]] = totals.get(r["model_family"], 0) + total
    12. grand = sum(totals.values())
    13. return {k: round(v / grand, 4) for k, v in totals.items()}
    14. print(token_share(usage_records)) # input and output combined
    15. print(token_share(usage_records, include_output=False)) # input tokens only

    With this sample data, family_a’s share is about 45% when output tokens are included and about 50% when they are excluded. The data itself has not changed. Only the scope of measurement has. Before comparing share figures from different sources, check that they use the same definition.

    3. Building a Reliable Usage Tracking Pipeline

    Usage data should be collected where the call happens. A common approach is to record each request at a unified gateway or in the call wrapper of the application. Each record should include the following fields: request timestamp, unique request ID, model version identifier, input token count, output token count, latency, and status code.

    After collection, data is usually written to a log service or data warehouse and aggregated by hour or day. Two points are easy to get wrong.

    The first is normalizing model identifiers. If one model has several aliases or version spellings, it will be split into multiple statistical entries and distort the shares. Build a mapping table that consolidates aliases into standard version identifiers.

    The second is deduplicating retried requests. When a call fails and is retried, the service may count usage for each attempt. Deduplicate by unique request ID during aggregation, and record the usage from retries separately so it can be reconciled against the bill.

    4. Don’t Rely Only on Usage Rankings for Model Selection

    Usage rankings reflect market usage. They do not directly indicate the best choice for a specific business scenario. Evaluate models against your own business needs, using the following dimensions.

    • Task quality: Run evaluations on your own datasets, and do not rely only on published results.