LLM Selection in Practice: Moving Beyond Benchmarks to Planner-Executor Workflows

Whenever a new generation of large language models is released, the community tends to split into two opposing camps. One side points to dramatically improved benchmark scores and claims a leap in capability. The other side, after running into problems in real business scenarios, dismisses the model as overhyped. Both views seem contradictory, but the real source of the disagreement is often a difference in evaluation criteria. This article aims to clarify the issue and provide a practical model selection methodology.

1. The Gap Between Benchmarks and Real-World Experience: Why Evaluations Go Wrong

A rational assessment of any new model is usually this: it is neither as impressive as the benchmark scores suggest, nor as poor as scattered negative feedback implies. Three technical factors typically explain the gap.

1. Benchmarks measure peak capability, while daily work reflects average performance. Public benchmark datasets contain difficult questions with clear boundaries, which makes targeted optimization easier. Real business problems, by contrast, come with ambiguous constraints, long contexts, and multiple competing objectives. Performance naturally drops in such conditions.

2. Small-sample evaluation has high variance. Drawing conclusions from a handful of prompts or conversations is fundamentally small-sample inference. A good answer today may simply mean the prompt happened to fall within the model’s training distribution. A poor answer tomorrow may be nothing more than sampling randomness. Anecdotal experience can either overestimate or underestimate a model’s true ability.

3. Benchmark data contamination. If training data overlaps heavily with public evaluation sets, scores can be systematically inflated, masking the model’s actual generalization ability.

The conclusion is straightforward: the only reliable way to determine whether a model fits your use case is to build your own evaluation system, rather than chasing leaderboards.

2. Building Your Own Evaluation Set: A Three-Layer Design

Layer 1: Capability domain decomposition. Break down your business problems into capability domains: solution planning, code generation, code understanding and review, complex issue localization, long-context handling, structured output, and so on. For each domain, collect 20 to 50 real business problems to form a fixed evaluation set with version management.

Layer 2: Repeated sampling. Sample each question 3 to 5 times and record two metrics:

• Best answer quality: measures the model’s capability ceiling, representing the best it can achieve.
• Average answer quality and variance: measures engineering usability, representing what it can reliably deliver day to day.

Layer 3: Structured logging. Use a unified template to record results for longitudinal comparison across models and versions. Example:

{
  "capability": "architecture_planning",
  "question_id": "AP-017",
  "runs": 5,
  "best_score": 9,
  "avg_score": 7.2,
  "std": 1.1,
  "latency_first_token_ms": 4200,
  "tokens_per_run_avg": 18500,
  "notes": "Covers constraints well, but occasionally misses non-functional requirements"
}

This kind of record-keeping turns subjective impressions into data and replaces debates with controlled comparisons.

3. The Value of Parameter Scale: Where Reasoning Breadth Comes From

The industry has recently seen extremely large models with hundreds of billions or even trillions of parameters. The core value of these models is not producing a spectacular answer to a single question, but rather breadth of reasoning:

• More comprehensive consideration of dimensions: On open-ended tasks such as architecture design and technology selection, they are less likely to miss key constraints and boundary conditions.
• More stable context retention: In long documents and multi-file code scenarios, they are better at referencing and integrating distant information.
• Fewer knowledge blind spots: A lower rate of common-sense errors on cross-domain problems.

Based on observations from internal test samples, these models do show impressive ceilings on planning and analysis tasks. But scale is a double-edged sword: more parameters mean higher inference overhead, slower responses, and greater token consumption per call. This makes extremely large models naturally suited to low-frequency, high-value tasks rather than high-frequency, routine ones.

4. Unavoidable Engineering Constraints: Speed and Token Budget

1. Speed determines where a model fits in the pipeline. Interactive scenarios such as IDE completion, conversational debugging, and online Q&A are highly sensitive to first-token latency and generation speed. Offline batch processing such as document summarization, bulk refactoring, and data cleaning can tolerate slower responses. A slow model is not unusable; the key is placing it in the right position within your pipeline.

2. Token consumption is a hidden cost. Reasoning-oriented models often generate long chain-of-thought processes before producing a visible answer, and those hidden tokens count toward your quota. A single complex problem can easily consume tens of thousands of tokens. If such a model is used for simple tasks like format conversion or document polishing, the token budget will deplete alarmingly fast.

3. Learn to do the math. A simple cost model can help with monthly estimates:

def estimate_monthly_cost(calls_per_day, tokens_per_call, price_per_1k):
    monthly_tokens = calls_per_day * tokens_per_call * 30
    return monthly_tokens / 1000 * price_per_1k
# Planning tasks: low frequency, high per-call consumption
planner_cost = estimate_monthly_cost(calls_per_day=50,
                                     tokens_per_call=20000,
                                     price_per_1k=P1)
# Execution tasks: high frequency, low per-call consumption
executor_cost = estimate_monthly_cost(calls_per_day=2000,
                                      tokens_per_call=3000,
                                      price_per_1k=P2)

If the platform supports budget controls for output length or reasoning depth, enable them proactively to prevent the model from overthinking simple problems.

5. Planner Models vs. Executor Models: Matching the Right Model to the Right Job

Considering capability ceiling, speed, and cost together, a pragmatic division of labor has emerged in engineering practice:

• Planner models: Responsible for architecture design, task decomposition, technical solution review, and complex defect localization. These tasks are low-frequency and high-value per call, require breadth and depth of reasoning, and can tolerate slower responses and higher per-call costs.
• Executor models: Responsible for mechanical tasks such as code generation, unit test writing, format conversion, comments, and documentation. These tasks are high-volume with fixed patterns, requiring speed and cost efficiency rather than a high capability ceiling. Many models currently offer a strong cost-performance advantage on execution tasks, which is precisely the market condition that makes this division of labor viable.

Example task routing logic:

ROUTE_TABLE = {
    "architecture_design": "planner",
    "task_decomposition":  "planner",
    "complex_debugging":   "planner",
    "code_generation":     "executor",
    "unit_test_writing":   "executor",
    "doc_polish":          "executor",
}
def route(task_type, complexity="normal"):
    role = ROUTE_TABLE.get(task_type, "executor")
    # Escalate to the planner model when complexity increases
    if complexity == "high":
        role = "planner"
    return role

Two points deserve emphasis:

1. The boundary is not absolute. In small and medium-sized projects, planning scenarios are often not complex enough to require a massive model. An executor model may produce perfectly adequate solutions, and in such cases you should delegate aggressively to reduce costs. Where the boundary lies should be determined by your own evaluation set, not by copying someone else’s conclusions.
2. Do not treat the planner model as a general-purpose model. Routing every request indiscriminately to the most expensive, slowest model is the most common cost disaster. The essence of a routing layer is trading task understanding for inference cost.

For teams that want to formalize multi-model access, canary rollouts, and usage and cost monitoring as infrastructure, cloud-based model inference and invocation services can provide a unified access layer for routing and observability, avoiding the need to rebuild these capabilities in every application.

6. Model Selection Checklist

Before adopting or switching to any new model, run through the following checklist:

1. Capability: Record peak and average performance separately; pay attention to variance rather than single-point results.
2. Speed: Measure first-token latency and generation speed separately, distinguishing between interactive and batch scenarios.
3. Cost: Per-task token consumption, quota depletion rate, and monthly cost estimates.
4. Stability: Consistency across repeated samples, and whether performance degrades in long sessions.
5. Engineering readiness: Structured output, function calling, reasoning budget controls, and context length must meet integration requirements.
6. Replaceability: Whether the interface abstraction is sufficient to switch to a better model at low cost when one emerges.

7. Conclusion

When evaluating a new model, benchmark scores are a starting point, not the destination. What truly determines whether a model enters your production pipeline is its average performance on your evaluation set, its response speed within your latency budget, and its per-unit value under your cost model. The planner-executor split provides a pragmatic path to adoption: use extremely large models for low-frequency, high-value tasks where reasoning breadth matters, and use lightweight, efficient models for high-frequency, routine tasks where throughput and cost are paramount. The essence of model selection is not finding the single best model, but building the most appropriate model combination.