Mid-Tier LLM Selection: Evaluating Cost-Effectiveness for Automation Scenarios

1. Background: Tiered Model Supply and the “Middle Gap”

The current LLM market is generally characterized by a tiered supply structure. Flagship models pursue peak capability, handling the most complex reasoning and creative tasks at the highest pricing. Lightweight models serve simple, high-volume requests with extremely low prices and low latency. Between these two, there is often a noticeable gap in both price and capability—the input and output unit prices between the two tiers can sometimes differ by an order of magnitude.

However, the actual workload distribution on the enterprise side rarely matches this tiered structure. The vast majority of automation requirements are “high-frequency, batch, and process-oriented” tasks: document and ticket extraction, form filling, data transfer between systems, desktop UI operations, and batch content review. These tasks do not require cutting-edge reasoning capabilities but are extremely sensitive to per-call cost, throughput speed, and stability. The larger the gap, the greater the need for a “mid-tier主力” role to bridge it: capabilities close to flagship models, significantly lower costs, and faster speeds.

Understanding this is the starting point for model selection: first determine whether your task “requires peak capability” or “requires scale and efficiency,” and then decide which tier to select from. The root cause of many automation project failures is choosing the wrong tier from the start—either running full volumes on a flagship model causing budget overruns, or running complex tasks on a lightweight model leading to uncontrollable failure rates.

2. Typical Positioning of Mid-Tier Models: Efficient “Hands and Eyes” vs. the “Smartest Brain”

Looking at common industry product positioning, the capability profile of mid-tier models is typically as follows: approaching flagship models in technical writing and communication; matching or closely following flagship models in visual understanding and computer/desktop operations, but with much faster execution speeds; and slightly weaker in the highest-difficulty reasoning tasks.

This positioning can be summarized by a formula:

Mid-tier model ≈ Flagship-level “eyes and hands” (visual understanding + tool/desktop operations) + multiple times the execution speed + significantly lower per-token pricing

This leads to an important engineering conclusion: do not use mid-tier models to tackle world-class puzzles or the most complex creative generation—leave those tasks to flagship models. The value of mid-tier models lies in “sitting in front of the computer and doing the dirty, tiring, and batch work” for you. The first step in selection is honestly categorizing your task list: what is “problem-solving” and what is “labor.”

3. Four Core Evaluation Dimensions

3.1 Visual Understanding and Desktop/Tool Operation Capabilities

The main work of office automation is “looking at the screen + operating the UI + filling out forms.” Therefore, multimodal perception and operation capabilities directly determine the boundary of automatable tasks. When evaluating, focus on three types of benchmarks that closely resemble real workplace scenarios:

• Long-horizon agentic tasks: For example, converting design drafts into structured deliverables, reviewing media assets one by one, and outputting structured logs with timestamps. These tasks have long chains and low fault tolerance, best reflecting real workplace performance.
• Real desktop environment operations: Completing installation, configuration, and cross-software collaboration actions in a simulated operating system. This tests the capability of the “hands.”
• Multimodal visual reasoning: Chart understanding, screenshot information extraction, and layout analysis. This tests the capability of the “eyes.”

When interpreting these results, there are two key points. First, always compare scores under the same reasoning tier; differences between tiers can exceed differences between model generations. Second, focus on per-task cost rather than just scores—a model that scores 5% higher but costs twice as much may not be cost-effective in batch scenarios.

3.2 Execution Speed and Throughput

Speed should be analyzed across three levels:

1. First-token latency: Affects the user experience in interactive scenarios but has less impact on batch processing.
2. Steady-state output speed (tokens/s): Determines the throughput upper limit of a batch processing pipeline.
3. End-to-end task completion time: The total time including multi-step tool calls, retries, and waiting. This is the true capacity metric for an automation pipeline.

A rule of thumb: a multiple-fold increase in end-to-end speed means the pipeline capacity scales proportionally under the same budget and concurrency resources. For scenarios with SLA constraints (e.g., promised ticket response times), speed may even take priority over benchmark scores. Additionally, note that officially advertised speeds should be retested under your actual workloads—context length, concurrency, and network links will significantly affect measured values. Discrepancies between measured and advertised speeds are normal.

3.3 Cost: Per-Task Cost ≠ Unit Price

This is the most misunderstood dimension. The correct calculation is:

Per-task cost = Input tokens × Input unit price + Output tokens × Output unit price + Failure retry cost + Manual validation cost

This is determined by two variables: unit price and token efficiency. Some models have lower unit prices but are verbose and have high failure rates, resulting in a higher per-task cost when calculated. Conversely, a model with high token efficiency (completing the same task with fewer tokens) can process more tasks under a fixed budget. For example, assume a mid-tier model’s input/output unit prices are half of a flagship’s, but the output tokens required to complete the same task are only two-thirds of the flagship’s. The per-task cost would be roughly one-third of the flagship’s—“price reduction” and “efficiency improvement” have a multiplicative relationship.

From a budget perspective, an N-fold decrease in per-task cost means an N-fold increase in the volume of tasks that can be automated. For automation projects, this is the most critical ROI lever, more important than any single benchmark score.

3.4 Stability and Hallucination Rate

The unique characteristic of automation scenarios is that they are unattended: errors are not intercepted by humans but flow directly downstream and are amplified. Here, hallucinations manifest as fabricating fields, clicking wrong buttons, or inventing conclusions. When evaluating, pay special attention to the error rate under lower reasoning tiers—because production environments, for cost and latency reasons, mostly run on low to medium tiers. If a model is only reliable when pushed to the highest tier, its cost model is invalid for batch scenarios.

Common engineering methods to reduce hallucinations include structured output constraints (JSON Schema, function calling), requiring models to provide evidence (citing original text or screenshot regions), secondary confirmation for critical actions, and confidence-based routing + sampling manual review. The introduction of these methods inherently increases token consumption and should be included in the per-task cost calculation.

4. How to Correctly Interpret Benchmarks and Official Claims

1. Beware of the “Aggregate Score Illusion”: Aggregate scores mask task-level differences. If your business primarily involves document extraction, you should weight document-related tasks rather than looking at overall leaderboard rankings.
2. Tier as Configuration: Different reasoning tiers of the same model represent different “cost-effectiveness points.” The correct approach is to plot a “score-cost” frontier curve and select a point based on the business’s accuracy requirements, rather than defaulting to the highest tier.
3. Beware of Benchmark Contamination: General evaluation sets may already be covered by training data, leading to inflated scores. The most credible scores always come from your own private evaluation sets.
4. Official Data is an Ideal Upper Bound: Real business contexts are longer, formats are dirtier, and exceptions are more frequent. Measured performance is usually lower than official claims, so leave a margin when budgeting.

5. Landing Practice: Building Custom Evaluation Sets and Canary Routing

5.1 Five-Step Implementation Method

1. Task Profiling: Outline input/output modalities, context length, latency budget, accuracy requirements, and per-task output structure.
2. Golden Set: Sample 100–500 real cases from production traffic, annotating expected outputs and judgment rules.
3. Evaluation Script: Rule-based scoring (field hit rate, format compliance) + LLM-as-judge (semantic quality) + manual sampling as a fallback.
4. Cost and Speed Measurement: Measure output speed, first-token latency, and per-task cost under target concurrency and real contexts.
5. Canary and Degradation: Run shadow traffic in parallel, route based on confidence, automatically escalate hard cases to flagship models, and fall back to humans on failure.

5.2 Evaluation Script Example (Pseudo-code)

# Speed and cost calculation (pseudo-code, replace API names with actual platform interfaces)
import time
PRICE_IN, PRICE_OUT = 1.0, 5.0   # Example values: price per million tokens, replace with actual billing
def measure(run, prompt: str, golden: dict) -> dict:
    t0 = time.perf_counter()
    resp = run(prompt)                    # Wrap a single model call
    dt = time.perf_counter() - t0
    u = resp["usage"]
    cost = (u["input_tokens"] / 1e6 * PRICE_IN
          + u["output_tokens"] / 1e6 * PRICE_OUT)
    return {
        "tps": u["output_tokens"] / dt,   # Steady-state output speed
        "cost": cost,                     # Per-task cost
        "pass": check(resp["text"], golden),  # Rule-based scoring
    }

5.3 Routing and Degradation Example (Pseudo-code)

def route(task) -> str:
    if task.risk == "high" or task.needs_deep_reasoning:
        return "flagship"      # Hard cases, high risk -> Flagship
    return "mid_tier"          # High-frequency batch -> Mid-tier
def handle(task):
    out = call(route(task), task)
    if not validate(out):
        out = call("flagship", task)   # One escalation retry
    return out if validate(out) else escalate_to_human(task)

If you access models via a cloud platform (for example, using model services provided by platforms like Baidu AI Cloud, along with general capabilities like logging services and monitoring alerts), you can integrate call logs, cost statistics, and exception alerts directly into your existing operations system, significantly reducing the cost of building evaluation and canary pipelines.

6. Common Pitfalls and Selection Checklist

Four high-frequency pitfalls: Score obsession (leaderboard rank ≠ business fit); unit price obsession (ignoring token efficiency and retry costs); tier mismatch (running full volume on flagship or hard cases on lightweight models); one-time evaluation (lacking continuous regression after launch).

Copyable checklist:

• □ Task profiling and golden set are ready
• □ Horizontal comparison of at least 2–3 candidate models across 2 reasoning tiers
• □ Recorded metrics: task score, tokens/s, first-token latency, per-task cost, failure rate
• □ Plotted “score-cost” frontier curve, selected point based on SLA
• □ Launched routing, degradation, alerting, and periodic regression evaluation

7. Conclusion

The value of mid-tier models lies not in being “stronger,” but in being “just right, and faster and cheaper.” The essence of selection is using a measurable evaluation system to translate vague business requirements into four numbers: task score, per-task cost, throughput speed, and failure rate, and then making rational trade-offs on the “score-cost” frontier. By treating evaluation, routing, and degradation as serious engineering problems, automation projects can truly achieve large-scale implementation.