LLM Token Cost Benchmarking and Multi-Level Agent Task Scheduling Strategies

LLM Token Cost Benchmarking and Multi-Level Agent Task Scheduling Strategies

As AI applications increasingly shift towards agentic architectures, the cost of invoking Large Language Models (LLMs) remains a focal point for developers and enterprises. Recently, certain platforms have introduced subscription billing plans based on total token volume. To evaluate the actual cost-efficiency of such plans, this article records API invocation data under real-world business scenarios. By applying data fitting methods, we reverse-engineered the actual credit consumption rates of several mainstream LLMs on a given platform. Based on these findings, we propose a multi-level task scheduling strategy that balances both performance and cost.

I. LLM Token Consumption Rate Fitting and Comparison

To obtain accurate real-world costs, we recorded extensive API call logs under identical business logic. We then performed linear regression fitting on the input tokens, output tokens, and consumed credits. The results showed an extremely high goodness of fit, with residuals approaching zero. After data abstraction, the actual resource consumption characteristics of several typical industry LLMs are as follows:

1. Frontier Comprehensive LLM (Benchmarked against top-tier closed-source models)
This category of model possesses the most comprehensive capabilities. In our tests, its input cost was approximately 80,000 tokens per million credit units, while its output cost reached up to 400,000 tokens per million units. It is an industry consensus that generation costs are significantly higher than comprehension costs. However, the output cost of this model is 200 times that of ultra-low-frequency lightweight models, reflecting extremely high computational resource consumption.

2. Deep Reasoning LLM
This type of model excels in logical reasoning and code generation. Its input cost is about 32,000 tokens per million units, and its output cost is roughly 112,000 tokens per million units. Compared to the comprehensive model, its overall cost is reduced by over 60%, making it suitable for handling medium-complexity logical chain inference tasks.

3. High-Speed Lightweight LLM (Flash / Flash Lite class)
These models are specifically designed for high-concurrency, low-latency scenarios. For standard high-speed models, the input cost is approximately 4,000 tokens per million units, and the output cost is about 8,000 tokens per million units. More impressively, when context caching is enabled and hit, the single input cost drops to merely 2% of the normal rate, drastically reducing the repeated computation overhead in long-prompt scenarios. As for the ultra-lightweight version, input and output are billed uniformly at only 2,000 tokens per million units.

II. Quota Accounting and Throughput Deduction

Assume a platform offers a base quota of 60,000 credits every 5 hours, with a weekly limit of 600,000 credits. In actual business scenarios, the input-to-output ratio for LLMs is typically 80% to 20% (long prompt inputs, short result outputs). Excluding cache hits, the maximum token processing limits for each model type within a 5-hour window are as follows:

• Frontier Comprehensive LLM: Calculated using the weights (80% * 80,000) + (20% * 400,000), one credit processes approximately 7 tokens. The total throughput over 5 hours is about 417,000 tokens.
• Deep Reasoning LLM: One credit processes roughly 21 tokens. The 5-hour total throughput is approximately 1.25 million tokens.
• High-Speed Lightweight LLM: One credit processes about 167 tokens. The 5-hour total throughput is roughly 12.5 million tokens.
• Ultra-Lightweight LLM: One credit processes approximately 500 tokens. The 5-hour total throughput is about 30 million tokens.

Data comparison indicates that the throughput capacity difference between comprehensive models and lightweight models within a given time frame can exceed 70-fold.

III. Multi-Level Agent Task Scheduling Architecture Design

Given the stark cost and throughput disparities described above, routing all processes through a top-tier model in a complex Agent application will instantly exhaust the quota. To address this, we designed a multi-level scheduling architecture. The core philosophy is: “Expensive models think, cheap models execute.”

1. Planning and Review Layer (Frontier Comprehensive LLM)
The comprehensive model has the broadest knowledge base and strongest instruction-following capabilities. In this architecture, it is only invoked at the start and end of a task. At the start, it parses user intent, decomposes complex tasks into multiple sub-steps, and generates an execution plan. At the end, it performs semantic review, polishing, and format validation on the final aggregated results, ensuring the highest ceiling of output quality.

2. Logical Reasoning Layer (Deep Reasoning LLM)
Sub-tasks requiring multi-step logical inference, mathematical calculations, or complex code refactoring are delegated to the deep reasoning model. It offers significant cost advantages while retaining robust chain-of-thought capabilities, acting as the “midfield coordinator” executing instructions from the planning layer.

3. Primary Execution Layer (High-Speed Lightweight LLM)
High-frequency tasks such as routine tool call parsing, API parameter extraction, and standard code generation are handled by the high-speed lightweight model. These tasks require above-average intelligence but demand extremely high response speed and concurrency. Integrating high-speed models into this layer significantly boosts the system’s overall operational efficiency while controlling baseline costs.

4. Batch Processing Layer (Ultra-Lightweight LLM & Caching Mechanism)
Highly mechanical, low-difficulty tasks like text classification, sensitive word filtering, log structuring, and text summarization slicing are routed directly to the ultra-lightweight model. Furthermore, for fixed texts like system prompts, the context caching mechanism must be forcibly utilized. This reduces the input cost for this portion to 2% of the original, thoroughly eliminating the redundant overhead associated with long contexts.

IV. Engineering Best Practices

Implementing this scheduling architecture requires the support of underlying cloud-native infrastructure. In engineering practice, developers should leverage general container platforms and message queue services to build an elastic, scalable invocation gateway.

The schematic pseudo-code is as follows:

def route_task(task_type, prompt):
    if task_type == "complex_planning":
        # Invoke the comprehensive model for top-level planning
        return call_llm(model="premium_model", prompt=prompt)
    elif task_type == "logical_reasoning":
        # Invoke the reasoning model for complex logic
        return call_llm(model="reasoning_model", prompt=prompt)
    elif task_type == "tool_execution":
        # Invoke the high-speed model for routine execution
        return call_llm(model="flash_model", prompt=prompt, cache_enabled=True)
    else:
        # Batch simple tasks, invoke the ultra-lightweight model
        return call_llm(model="lite_model", prompt=prompt)

At the logging service and monitoring alert level, the system must aggregate and track the token consumption of each model type in real time. If an abnormal surge in the invocation frequency of expensive models is detected, a rate-limiting or graceful degradation strategy is triggered. This automatically offloads non-critical tasks to lightweight models, ensuring the system quota is not exceeded.

Through this fine-grained “model as a resource pool” scheduling management, enterprises and developers can build robust AI application systems within limited budgets. These systems are capable of handling high-difficulty reasoning while simultaneously managing high-concurrency requests.

Note: The model consumption rates mentioned in this article are derived from personal empirical data fitting and reverse-engineering. They do not represent official pricing tables, and actual performance may vary with platform rule adjustments.