LLM Token Cost Measurement and Tiered Model Routing Architecture

Introduction: The Cost Dilemma in LLM Application Deployment

In current LLM application development, developers and enterprises face a serious engineering challenge: the stronger the model, the higher the token consumption cost. Calling top-tier complex reasoning models for every business step can ensure output quality, but the resulting API bills are often prohibitive. Conversely, relying entirely on lightweight models fails to handle complex logical reasoning tasks.

To find the optimal balance between performance and cost, we need to accurately evaluate the actual token consumption rates of various large models and build a scientific “tiered model routing architecture” on that basis. This article uses a set of real business call records to back-calculate the credit consumption rates of several typical model tiers through data fitting, and explains in detail how to use these data to guide production architecture design.

1. Data Fitting and Model Cost Back-Calculation

1.1 Test Background and Methodology

To obtain accurate cost data, we extracted real business LLM call records and performed multivariate linear regression, using total credit consumption as the dependent variable and input tokens, output tokens, and cache-hit tokens as independent variables.

The fitting results were extremely strong: the coefficient of determination R² = 1.0, with residuals approaching zero. This means the back-calculated credit consumption rate formulas are almost perfectly consistent with the actual billing rules. Note that the following data are personal measurement and back-calculation results, not official price lists; billing rules from different cloud providers may change over time.

1.2 Credit Consumption Rates of Typical Model Tiers

Based on the fitting results, we divided common industry models into four typical tiers by capability and cost (specific brands and model versions are omitted for neutrality, and tiers are used instead):

• Complex reasoning model (analogous to a top-tier general-purpose LLM)
  • Input: 80,000 / 1M tokens
  • Output: 400,000 / 1M tokens
• Medium reasoning model (analogous to a standard-tier LLM)
  • Input: 32,000 / 1M tokens
  • Output: 112,000 / 1M tokens
• Light execution model (analogous to a fast-tier LLM)
  • Input: 4,000 / 1M tokens
  • Output: 8,000 / 1M tokens
  • Cache hit: 80 / 1M tokens (only 2% of the normal input cost)
• Micro assistant model (analogous to a compact-tier LLM)
  • Unified input/output: 2,000 / 1M tokens

1.3 Quantitative Comparison of Cost Differences

The data above show that cost differences between tiers are orders of magnitude apart. Most intuitively: the output cost of the complex reasoning model is 200 times that of the micro assistant model. Meanwhile, once the light execution model hits the context cache, its input cost drops sharply to 2% of the normal level—a massive cost advantage for scenarios that frequently pass long system prompts.

2. Capacity Estimation and Call Quota Management

Before designing the architecture, we need to estimate the actual model capacity under given quotas based on the consumption rates above. Assume the business side allocates a quota of 60,000 credits per 5 hours (short-term burst) and 600,000 credits per week (long-term steady state).

In typical interaction scenarios, the ratio of input tokens to output tokens is about 8:2 (80% context input, 20% content generation). Ignoring cache hits for now, we can estimate the theoretical capacity of each model tier within 5 hours:

• Complex reasoning model: about 417,000 tokens / 5 hours
• Medium reasoning model: about 1.25 million tokens / 5 hours
• Light execution model: about 12.5 million tokens / 5 hours
• Micro assistant model: about 30 million tokens / 5 hours

These figures clearly show that under the same quota, low-cost lightweight models can handle dozens or even hundreds of times more text throughput than high-end models. Blindly using high-end models for all tasks is therefore a major waste of compute and budget.

3. Tiered Routing Architecture Design and Practice

Based on the cost differences and capacity data above, we propose a “tiered routing” LLM invocation architecture whose core idea is: expensive models think, cheap models do. Specifically, invocation logic can be divided into four tiers by task type:

3.1 Complex Reasoning Model: Top-Level Planning and Final Review

Because this tier has extremely high output cost but strong logical reasoning ability, it is suitable for global task decomposition and quality control. In the business pipeline, it only receives the user’s initial request, decomposes it into multiple subtasks, and at the final step receives the execution results of all subtasks for unified logical review and polishing. By restricting it to planning work with “input + short output,” its expensive output cost can be effectively controlled.

3.2 Medium Reasoning Model: Complex Logical Reasoning

For tasks requiring some contextual understanding and multi-constraint reasoning (such as code generation or complex structured data extraction), the medium reasoning model can be used. Its cost is intermediate, and it can handle relatively complex execution logic while maintaining a certain quality level.

3.3 Light Execution Model: Primary Execution Workload

This tier carries the largest workload in the entire architecture. For standard tasks such as routine text expansion, translation, and format conversion, the light execution model is the best choice thanks to its extremely high throughput and very low cost. In addition, the business side should fully leverage its “extremely low cache-hit cost” by passing fixed system prompts and long reference documents as cached input, further reducing input costs.

3.4 Micro Assistant Model: Batch Simple Tasks

For highly patterned simple tasks such as classification, intent recognition, sensitive-word filtering, and tag extraction, call the micro assistant model directly. Such tasks have very short outputs and usually short inputs; the extremely low unit price of the micro model makes it well suited to digesting massive data through concurrent batch processing.

4. Architecture Implementation Example and Routing Strategy

In actual engineering deployment, developers can introduce a gateway layer for unified request routing and distribution. Below is an illustrative pseudocode for routing decision logic:

class ModelRouter:
    def route_request(self, task_type, input_tokens, require_reasoning):
        # Rule 1: Top-level planning and final review go to the advanced model
        if task_type in ["task_planning", "final_review"]:
            return "complex_reasoning_model"
        # Rule 2: Tasks requiring deep logical reasoning
        if require_reasoning and task_type in ["code_generation", "complex_extraction"]:
            return "medium_reasoning_model"
        # Rule 3: Long-text execution tasks with fixed prompts trigger light-model caching
        if input_tokens > 4096 and task_type in ["translation", "expansion"]:
            # Set cache_prompt to leverage the 2% cached input cost advantage
            return "light_execution_model_with_cache"
        # Rule 4: Default simple tasks go to the micro model
        if task_type in ["classification", "intent_recognition", "tag_extraction"]:
            return "micro_assistant_model"
        # Fallback strategy
        return "light_execution_model"

Through this tiered routing mechanism, the system can automatically direct requests to the most cost-effective model. In a cloud-native environment based on container platforms and microservice architecture, developers can use a standard API gateway together with message queues to achieve asynchronous decoupling and smooth scaling of this routing strategy. Combined with conventional logging and monitoring alerts, the actual token consumption of each model tier can be tracked in real time so that routing weights can be adjusted dynamically.

5. Summary and Considerations

Through rigorous data fitting and cost back-calculation, the huge cost gap between different LLMs becomes clear. Adopting a “tiered invocation, each tier doing its part” architecture is a key path for enterprises to control IT budgets while maintaining LLM application quality.

When implementing this architecture, developers should note the following:

1. Cache strategy dependency: Fully leveraging context caching is an important means of cost reduction, but you must ensure that the API gateway or underlying LLM service supports prompt caching.
2. Dynamic routing adjustment: Model capabilities are constantly evolving, and routing rules should not remain static. Establish an A/B testing mechanism to regularly evaluate whether lower-cost models can take over tasks originally handled by higher-cost models.
3. Watch for billing rule changes: The consumption rates in this article are based on measured data fitted at a specific point in time. LLM providers may adjust billing policies at any time, so the architecture should keep billing parameters configurable to adapt quickly to market changes.