Multi-Model Intelligent Routing for Table Generation: Cutting LLM Costs in Practice

    This article shows how multi-model intelligent routing can significantly cut the cost of large language model (LLM) applications in table generation scenarios without sacrificing output quality. Using a controlled sample of 1,000 table generation tasks and clearly defined unit prices, acceptance criteria, and caching rules, it demonstrates a reduction in per-thousand-task cost from 18.6 CNY to 2.8 CNY, roughly 85%. It covers root-cause analysis of cost differences, a four-layer architecture (unified access, task identification, routing decisions, and observability), key techniques such as task tiering, cascading calls, context compression, output constraints, cache reuse, batch processing, and budget controls, as well as quality assurance, phased rollout steps, and a FAQ. It is intended for engineers and teams building observable, governable, and cost-efficient LLM routing systems.

    1. Background: From “Choosing the Strongest Model” to “Routing Each Task to the Right Model”

    As large language models move into business scenarios such as table generation, field extraction, format conversion, and summarization, the focus of enterprises has shifted from “can we call a model?” to “can we deliver reliably at a lower cost?” In table generation scenarios, capabilities like multi-model routing, LLM cost reduction, and model scheduling often come up together. High-capability models are usually better suited to complex reasoning, long-text comprehension, code generation, and high-compliance tasks. However, if every request defaults to a high-capability model, it leads to obvious token waste. Conversely, if simple tasks blindly use low-capability models, it can cause missing fields, formatting errors, and repeated retries, ultimately raising the total cost instead.

    The key to multi-model applications, therefore, is not simply comparing “which model is stronger,” but building a scheduling mechanism that is task-aware, cost-aware, and quality-aware: routing simple tasks through low-cost paths, complex tasks through high-capability paths, letting repeated tasks hit the cache as much as possible, and keeping human review in place for high-risk tasks.

    2. Test Sample and Cost Methodology: How the 85% Reduction Is Calculated

    To avoid overstating the conclusion, this example constrains the reduction to a clear sample, unit prices, acceptance criteria, and caching strategy. The test task is table generation, with 1,000 items in total, covering 250 header generation cases, 300 field completion cases, 250 format conversion cases, and 200 batch extraction cases. Three categories of models take part: high-capability Model A, balanced Model B, and lightweight Model C.

    This calculation uses illustrative unit prices, in CNY per 1,000 tokens, to explain the cost methodology; actual figures should follow the model provider’s bill:

    • Model A: input price 0.0040, output price 0.0080
    • Model B: input price 0.0035, output price 0.0072
    • Model C: input price 0.0020, output price 0.0040

    Retry cost is amortized as an extra call: about 0.0020 CNY per call for Model A, about 0.0040 CNY per call for Model B, and about 0.0030 CNY per call for Model C. Cache hit cost is 0.0003 CNY per hit, covering cache reads, retrieval, permission checks, and basic structural validation.

    The cost methodology is as follows:

    Per-task cost = input tokens × input price / 1000 + output tokens × output price / 1000 + retry call cost
    Per-thousand-task cost = average per-task cost × 1000
    Auto-routing per-thousand cost = actual model call cost + cache hit cost + retry cost

    It is worth noting that a cache hit produces no model token consumption, but it does incur cache reads, retrieval, permission checks, and lightweight validation. This cost must be included in the total auto-routing cost and the overall cost comparison.

    The baseline strategy is: route all tasks to the high-capability Model A, with caching, context compression, task tiering, and auto-routing disabled. The acceptance criteria are: complete fields, consistent data types, controlled null ratio, output readable by a standard table parser, and a sampled manual score of no less than 4.5. This acceptance standard is a sampling-based standard; it means maintaining field completeness and format pass rates under sampling acceptance, and does not make an absolute commitment for all unsampled tasks.

    Strategy Avg input tokens Avg output tokens Total tasks Actual model calls Cache hits Retries Quality pass rate Per-thousand-task cost
    High-capability model direct 2800 900 1000 1000 0 120 98.8% 18.6 CNY
    Lightweight model direct 2800 760 1000 1000 0 260 92.1% 9.4 CNY
    Human-assigned fixed model 2400 820 1000 1000 0 140 97.6% 15.2 CNY
    Auto-routing 1800 560 1000 400 600 90 98.9% 2.8 CNY

    Notes: For the auto-routing row, the “1000 total tasks” includes cache hits; there are 400 actual model calls and 600 cache hits. Average input and output tokens count only actual model calls, excluding the cache-hit portion. The auto-routing per-thousand cost includes model call cost, cache hit cost, and retry cost. The human-assigned fixed model uses the balanced Model B, including a small amount of manual spot-checking and retry cost, without caching or auto-routing enabled.

    Calculation details:

    • High-capability model direct: 1,000 model calls; per-call model cost ≈ 2.8 × 0.0040 + 0.9 × 0.0080 = 0.0184 CNY; 120 retries ≈ 0.24 CNY; total ≈ 18.64 CNY, about 18.6 CNY.
    • Lightweight model direct: 1,000 model calls; per-call model cost ≈ 2.8 × 0.0020 + 0.76 × 0.0040 = 0.00864 CNY; per-thousand model cost ≈ 8.64 CNY; 260 retries ≈ 0.78 CNY; total ≈ 9.42 CNY, about 9.4 CNY.
    • Auto-routing: 400 model calls; average input 1800 tokens, average output 560 tokens. Auto-routing is dominated by Model C, with a small amount of Model B and Model A for medium- or high-risk tasks; after blended conversion, the per-thousand model cost is about 2.34 CNY; 600 cache hits, cache cost ≈ 0.18 CNY; 90 retries ≈ 0.27 CNY; total ≈ 2.79 CNY, about 2.8 CNY.

    In the auto-routing strategy, average input tokens drop from the 2800 baseline to 1800, a context compression rate of about 35.7%; average output tokens drop from 900 to 560, a reduction in output redundancy of about 37.8%. In this task distribution, table generation tasks with clear formats and stable fields make up a high proportion, bringing the cache hit rate to 60%, an important source of the cost reduction.

    Under this methodology, auto-routing reduces the per-thousand-task cost from 18.6 CNY to 2.8 CNY. The cost reduction is:

    Cost reduction = (baseline cost − optimized cost) / baseline cost × 100% = (18.6 − 2.8) / 18.6 × 100% ≈ 84.9%, about 85%.

    It should be emphasized that this result depends on the task distribution, cache hit rate, degree of context compressibility, strength of output constraints, changes in model unit prices, and the actual call-chain cost, and does not necessarily hold in every business scenario.

    3. Root Cause Analysis: Why the Same Task Costs So Differently

    For generating the same well-formed table, the cost difference between strategies usually comes from five areas.

    Root cause Common symptom Optimization direction
    Mismatch between model capability and task complexity Simple format conversions also use a high-capability model, raising input/output tokens; complex extraction uses a lightweight model, causing missing fields Establish task tiering and complexity scoring
    Different context length control System prompts, conversation history, business documents, and examples are all concatenated into the request, inflating input tokens Context compression, summarization, retrieval, field trimming
    Different levels of output redundancy The model returns explanatory text, repeated descriptions, Markdown wrappers, or unnecessary notes Clear output format and field constraints
    Immature retry and validation mechanisms No format or field validation after output; a single failure triggers multiple retries Upfront validation, limited retries, escalation on failure
    Lack of unified scheduling and cost observability Models are called in a scattered way, with no unified entry point, budget control, or monitoring dashboard Unified access, routing decisions, cost observability

    Among these, context and output redundancy are very common but easily overlooked cost sources in table generation scenarios. The task itself may look simple, but if you stuff the full history, long documents, and many examples into every request, input tokens expand rapidly. On the output side, even when the task only requires CSV or JSON, if the model generates extra explanatory text, it also increases parsing and output costs.

    4. Overall Architecture: Unified Access, Task Identification, Routing Decisions, and Observability Feedback

    It is recommended to split multi-model scheduling into four layers.

    4.1 Unified Access Layer

    The unified access layer abstracts away the differences between models and provides a standardized call entry point. Its core capabilities include a unified request format, unified context templates, unified output validation, unified logging, unified quota accounting, and a unified failure/retry policy. Business systems only need to describe the task type, input data, output requirements, and constraints, without worrying about differences in the underlying models.

    The unified access layer should also record each call’s task ID, business line, tenant, user role, permission scope, model version, schema version, cache status, retry status, and final cost. Only then is there a data foundation for subsequent routing optimization and cost analysis.

    4.2 Task Identification Layer

    The task identification layer decides which model strategy a task is suited to. It can score across dimensions such as task type, risk level, number of fields, rule clarity, historical success rate, and output format. To ease implementation, the complexity score can be designed as a normalized weighted score:

    Complexity score S = 0.35 × F + 0.25 × R + 0.20 × H + 0.20 × K

    Where:

    • F: field-count complexity, F = min(field count / 30, 1)
    • R: rule-clarity gap, R = 1 − rule clarity; the vaguer the rules, the higher R
    • H: normalized historical failure rate, H = min(7-day failure rate / 0.20, 1)
    • K: normalized risk level, 0 for low risk, 0.5 for medium risk, 1 for high risk

    Recommended thresholds:

    • S ≤ 0.35: simple task, prefer lightweight Model C
    • 0.35 < S < 0.65: medium task, use balanced Model B
    • S ≥ 0.65: complex or high-risk task, use high-capability Model A, and keep human review

    Examples:

    Task feature Fields Rule clarity Historical failure rate Risk level Complexity score Recommended strategy
    Fixed-template header generation 10 0.90 0.02 Low ~0.16 Lightweight Model C
    Multi-field completion 18 0.70 0.05 Medium ~0.44 Balanced Model B
    Contract key-field extraction 25 0.45 0.12 High ~0.75 High-capability Model A + human review

    If the score is low and the rules are clear, prefer the lightweight model; if the score is high, or the scenario involves high-risk areas such as finance, contracts, permissions, or production configuration, escalate to the high-capability model and introduce human review.

    4.3 Routing Decision Layer

    The routing decision layer should handle the cache and high-risk tasks first. A cache hit can significantly reduce repeated calls, but the cache key must include the input content, task type, tenant, permission scope, model version, schema version, and key constraints. Even when a high-risk task hits the cache, it must not bypass human review.

    Illustrative logic:

    1. def select_strategy(task, cache, policy):
    2. if task.risk_level == "high" or policy.need_human_review(task):
    3. return "strong_with_human"
    4. cache_key = build_cache_key(
    5. task.input,
    6. task.type,
    7. task.tenant,
    8. task.permission_scope,
    9. task.model_version,
    10. task.schema_version
    11. )
    12. if cache.exists(cache_key) and cache.permission_match(task, cache_key):
    13. if cache.schema_valid(cache_key, task.schema_version):
    14. return "cache"
    15. if task.type == "format_transform" and task.complexity_score <= 0.35:
    16. return "light"
    17. if task.complexity_score >= 0.65:
    18. return "strong"
    19. if task.complexity_score >= 0.35:
    20. return "balanced"
    21. return "auto"

    Even after a cache hit, schema validation and business validation are still required. If the cached result fails to parse, has missing fields, has a permission mismatch, or has an inconsistent schema version, it should immediately fall back to the model call path and record an audit log and alert.

    4.4 Observability Feedback Layer

    The observability layer should record at least the following metrics:

    • Input tokens
    • Output tokens
    • Total tasks
    • Actual model call count
    • Cache hit count
    • Retry count
    • Task success rate
    • Format validation pass rate
    • Field completeness rate
    • Average response time
    • Per-task cost
    • Per-thousand-task cost
    • Model hit rate
    • Cache hit rate
    • Permission check failure rate
    • Number of fallbacks and fallback reasons

    Teams can use this data to continuously optimize routing rules rather than relying on experience-based judgment.

    5. Key Optimization Techniques

    5.1 Task Tiering

    Divide tasks into three levels: simple, medium, and complex. Simple tasks use the lightweight model, medium tasks use the balanced model, and complex tasks escalate to the high-capability model. This ensures quality for complex tasks while avoiding overconsumption on simple ones.

    In table generation scenarios, tasks can be tiered further by type: fixed header generation and rule-clear format conversion suit the lightweight model; multi-field completion and field-relation extraction suit the balanced model; complex business definitions, contract extraction, and financial reconciliation suit the high-capability model with review retained.

    5.2 Cascading Calls

    You can adopt an “easy first, hard later” strategy: try the lightweight model first, and if the output passes validation, return it directly; if it fails, retry the lightweight model once; if it still fails, switch to the balanced or high-capability model. Cascading calls reduce the indiscriminate use of high-cost models.

    Two alternatives:

    • Pure rule-based routing: suitable for scenarios with few task types, clear rules, weak semantic-caching needs, and simple business chains. For example, fixed header generation and fixed format conversion can map to models directly by task type.
    • Budget-first routing: suitable for scenarios with strong cost constraints, non-real-time tasks, and batch asynchronous tasks. When a budget threshold is triggered, the system prioritizes the cache, lightweight models, and batch processing; for tasks that are not time-sensitive, it can defer execution to lower-price windows.

    5.3 Context Compression

    Before a call, you can remove duplicate history messages, trim the system prompt, keep only relevant fields, summarize or retrieve long documents first, and compress examples to the smallest usable set. Note that compression must not delete required fields and should retain an audit snapshot; if output validation fails after compression, fall back to the uncompressed version or escalate the model.

    In table generation scenarios, prioritize compressing conversation history, long business documents, repeated examples, and irrelevant field descriptions. For required fields, type constraints, enum ranges, date formats, and permission boundaries, keep them in the compressed context.

    5.4 Output Constraints

    For table generation, field extraction, and structured conversion tasks, specify the target format clearly, such as CSV, JSON, or XLSX, and constrain the model to return only the target content. Field-level validation can include:

    • Whether headers are complete
    • Whether required items are empty
    • Whether data types are consistent
    • Whether enum values are legal
    • Whether field lengths exceed limits
    • Whether date and amount formats meet specifications
    • Whether the output can be read by a standard table parser

    If output parsing fails, adopt a degradation or escalation strategy: first let the lightweight model try to repair the structure, then try the balanced model, and finally escalate to the high-capability model; if it fails repeatedly, move it to the human handling queue.

    5.5 Cache Reuse

    Caching can significantly reduce repeated calls, but you must design TTL, invalidation policies, permission isolation, and masking. Template-type tables can set a longer TTL, such as 7 days; dynamic data such as orders and reconciliations should set a shorter TTL, such as 24 hours, and support event-driven invalidation.

    The cache key should include the input content, task type, tenant, permission scope, model version, and schema version. When sensitive fields such as phone numbers, ID card numbers, or contract numbers are involved, mask them before writing to the cache. Permission isolation must distinguish tenants, user roles, and business scenarios to avoid unauthorized hits.

    Before writing to the cache, apply minimal masking, keeping only the fields needed for routing and reuse. Audit logs should record the caller, tenant, permission scope, hit result, validation result, cache TTL, invalidation reason, and access time. Audit log retention is recommended to be no less than 180 days, or per enterprise compliance requirements. If cache permission checks fail, masking fails, or schema validation fails, the cache result must not be returned; fall back to a model call and trigger an alert.

    Semantic matching is only recommended for low-risk, structured, field-stable tasks, with a high similarity threshold, such as 0.96; once a hit fails validation, fall back to a model call immediately.

    5.6 Batch Processing and Asynchronous Execution

    For batch tasks, you can merge multiple small requests to reduce scheduling overhead; for non-real-time tasks, you can place them in an asynchronous queue and choose the execution time based on cost and resources. This improves throughput efficiency and lowers the per-task unit cost.

    In batch table generation scenarios, you can combine the header, field completion, and format conversion tasks of the same batch into one batch task, do cache hit checks uniformly, do output validation uniformly, and retry anomalous samples separately.

    5.7 Budget Control and Alerting

    Enterprises should set call budgets and quota limits for different business lines. When an application, task, or time period consumes abnormally, the system should alert in time. For example:

    • A single-task cost exceeding 2× the historical average
    • Daily call cost growing more than 30% compared with the previous period
    • Format validation pass rate below 95%
    • Retry rate above 10%
    • Abnormal rise in cache permission check failures
    • Per-thousand-task cost exceeding 110% of the budget baseline

    After an alert is triggered, check the routing rules, cache hit rate, context compression ratio, and model unit price changes in combination with the fallback strategy.

    6. Quality Assurance: Cost Reduction Must Not Sacrifice Reliability

    The premise of cost optimization is controllable quality. Before returning any model output to the business system, it should undergo format validation and business validation, including field completeness, type correctness, null anomalies, enum rules, and downstream parsability.

    If the lightweight model repeatedly fails to produce a qualified result, it should automatically escalate to a stronger model rather than retrying indefinitely. For high-risk scenarios such as finance, contracts, compliance, permissions, and production configuration, even if the model’s accuracy is high, human review or secondary confirmation should be retained.

    During multi-model scheduling, enterprises should also mask sensitive data and control log access permissions to prevent sensitive information from entering model call logs or the cache system. For cache hits, model fallbacks, human reviews, and failed retries, traceable records should be retained for subsequent audit and optimization.

    7. Implementation Steps

    Step 1: Inventory task types. Survey scenarios such as table generation, Q&A, summarization, extraction, generation, review, and code assistance, and gather call volumes, failure rates, and cost distribution. For table generation scenarios, focus on the proportion of subtasks such as header generation, field completion, format conversion, and batch extraction.

    Step 2: Build a unified call layer. Consolidate model calls scattered across business modules into a standard entry point, making it easier to measure, govern, and control costs.

    Step 3: Design routing rules. In the early stage, it is recommended to use “rules first, models second,” setting strategies based on task complexity, risk level, output format, historical success rate, and budget. Routing rules should ensure stability first, then gradually introduce caching, context compression, and cascading calls.

    Step 4: Run canary validation. A canary ratio of 5% to 20% is recommended, with a canary period of no less than 7 days, or a sample size of no less than 1,000 items. Suggested quantitative thresholds for canary acceptance:

    • Cache hit rate: no less than 40% during canary, no less than 50% when stable
    • Format validation pass rate: no less than 98%
    • Field completeness rate: no less than 99%
    • Quality sampling score: no less than 4.5
    • Retry rate: no more than 10%
    • Per-task cost anomaly threshold: trigger rollback when it exceeds 2× the historical average
    • Per-thousand cost anomaly threshold: trigger rollback when it exceeds 110% of the baseline
    • Daily cost period-over-period growth: trigger an alert and inspection when it exceeds 30%
    • Cache permission check failure: immediately fall back to a model call and record an audit log

    If, during canary, format parsing failure rates rise, retry rates spike abnormally, cache permissions mismatch, or costs exceed thresholds, pause the rollout and fall back to the baseline strategy or the previous stable version.

    Step 5: Continuous optimization. Adjust routing rules, compress context, expand cache hits, and tune alert thresholds based on observability data, and periodically review high-risk scenarios. Whether the cost reduction is sustainable depends on the accuracy of task tiering, whether context can be trimmed, the reliability of cache quality, the completeness of output validation, and the timeliness of observability feedback.

    8. Conclusion and FAQ

    The phenomenon that “multiple models can all complete basic tasks, yet the bills differ significantly” shows that LLM applications have entered a phase of refined operations. The value of multi-model intelligent routing lies not in simply choosing a particular model, but in building an observable, governable, and optimizable scheduling system. For table generation scenarios, cost optimization should focus not only on model unit prices but also on task distribution, cache hits, context length, output redundancy, retry strategies, and permission isolation.

    FAQ

    Q: What is multi-model intelligent routing?
    A: It is a scheduling mechanism that automatically selects an appropriate model or cached result based on task complexity, cost budget, output requirements, permissions, and risk level.

    Q: How do you reduce costs in table generation scenarios?
    A: Start with task tiering, context compression, output constraints, cache reuse, limited retries, and cost observability. Simple fixed-header tasks should preferentially use the lightweight model, repeated tasks should preferentially hit the cache, and complex tasks should escalate to the high-capability model.

    Q: How do you choose a model?
    A: Simple rule-based tasks prefer the lightweight model, medium tasks use the balanced model, and complex or high-risk tasks use the high-capability model, with validation and human review retained.

    Q: How do you calculate call costs?
    A: Per-task cost = input token cost + output token cost + retry cost; the auto-routing per-thousand cost should also include cache hit cost. Optimization rate = (baseline cost − optimized cost) / baseline cost.

    Q: What are common mistakes in multi-model routing?
    A: Common problems include routing all tasks to high-capability models, caches lacking permission isolation, outputs lacking format validation, retries without a limit, overly long context, and no cost observability or budget alerting.

    Q: Can you return the result directly after a cache hit?
    A: It is not recommended to return it directly. After a cache hit, permission checks, schema validation, and business validation are still required. If permissions mismatch, fields are missing, or parsing fails, fall back to the model call path immediately.