LLM Service Selection: Build a Quantifiable Evaluation System

1. Redefine the problem: from “which model is stronger” to “which setup fits my workload”

Personal impressions of a model often come from ad hoc chat-style trials: one model feels better at writing, another feels more reliable for code, and another feels more human in conversation. Such impressions are weak evidence for engineering selection for three reasons:

1. Sampling bias: Manual prompts tend to be short, isolated, and free of context dependencies. Production traffic often includes long documents, multi-turn state, and structured output constraints.
2. Single-metric bias: Subjective impressions focus on answer quality and ignore first-token latency, throughput, failure rate, and per-task cost, all of which become visible only at scale.
3. Lack of reproducibility: Different prompt tweaks, decoding parameters such as temperature or maximum output length, and reasoning modes make conclusions impossible to reproduce or regression-test.

The engineering question is therefore not “which model is best,” but:

Under my task distribution, latency budget, compliance boundaries, and unit-cost constraints, which combination of models and service configurations has the lowest total cost?

This is a multi-objective optimization problem. Model capability is only one variable.

The real object of evaluation is an LLM service, which has at least five relatively independent dimensions:

Dimension	Key considerations
Model capability	Instruction following, long-context understanding, structured output stability, tool calling, multimodal understanding
Service interface	Streaming, batch or async endpoints, concurrency quotas and rate-limit policies, timeout and retry semantics
Engineering controllability	Parameter exposure, prefix caching, observability of logs and usage
Economics	Separate input and output pricing, cache-hit billing, break-even analysis between subscription and pay-as-you-go models
Governance and compliance	Whether data is used for training, log retention and redaction, data residency, private deployment

Separating interface, billing, governance, and capability layers is the prerequisite for all later work.

2. Building the evaluation set: the first-order factor in evaluation credibility

Evaluation-set quality matters more than the sophistication of the evaluation method. A practical construction process is:

• Stratified sampling by scenario: Sample according to real business traffic distribution, such as Q&A, summarization, extraction, classification, code generation, and tool-call orchestration. Allocate sample size by traffic share to avoid letting demo-style problems dominate the benchmark.
• Difficulty stratification: Split each scenario into simple, routine, and edge cases. Edge cases often provide the strongest discrimination because most models pass routine cases.
• Prefer decidability: Define automatically checkable assertions for each sample, such as exact match, regex, JSON Schema validation, unit-test pass, or numeric tolerance. Keep subjective items to a minimum.
• Contamination control: Use private business data or recently added data to build the core evaluation set. Public benchmarks can serve only as auxiliary references. Also maintain a regression set that is not publicly disclosed to avoid overfitting from repeated prompt tuning.
• Annotation consistency: Subjective scoring requires a rubric and dual annotation. Report inter-annotator agreement, such as Cohen’s Kappa. If agreement is below an acceptable threshold, fix the rubric before comparing models.

A minimal but complete storage structure for evaluation samples can be:

{
  "id": "kb-qa-000137",
  "category": "long_context_qa",
  "difficulty": "hard",
  "input": {
    "system": "...",
    "user": "...",
    "documents": ["doc_ref_1"]
  },
  "constraints": {
    "max_output_tokens": 400,
    "temperature": 0.0
  },
  "assertions": [
    {
      "type": "json_schema",
      "schema_ref": "answer_v2"
    },
    {
      "type": "contains_all",
      "keywords": ["refund", "7 days"]
    },
    {
      "type": "not_contains",
      "keywords": ["guaranteed return"]
    }
  ],
  "reference": "Optional reference answer or rubric ID"
}

assertions are the key to the whole system. They turn “good or bad” into “satisfies constraints or not,” making evaluation automatable, regression-testable, and runnable in CI.

3. Metrics: four categories must be evaluated together

3.1 Effectiveness

Beyond task-level accuracy or pass rate, three engineering-relevant metrics deserve separate tracking:

• Format and schema compliance rate: In structured-output scenarios, this is often the first failure source and directly affects downstream parsing success.
• Instruction-following rate: Joint satisfaction of length constraints, language constraints, prohibited-content constraints, and similar requirements.
• Harmful-output rate and refusal rate: Excessive refusal should also count as failure. Otherwise, a model can improve safety metrics by refusing to answer anything.

3.2 Efficiency

Do not use average latency as the primary decision metric. Use percentiles:

• TTFT (time to first token): Determines the perceived responsiveness of streaming interactions and should have its own SLO.
• TPOT (time per output token): Determines total duration for long-output tasks.
• End-to-end latency P50/P95/P99: P99 reflects rate limiting, queueing, and retry amplification.
• Effective throughput: Requests completed per second under agreed concurrency limits, not theoretical peak throughput.

3.3 Economics

Looking only at unit price leads to wrong conclusions. Use two composite metrics:

• Cost per 1,000 requests: Include input, output, cache-hit billing, retry amplification, and sunk cost from failed requests.
• Cost per successful task: Total cost divided by the number of tasks that pass evaluation. A cheaper per-token price can still result in higher cost per success if retry rate is high or first-attempt success rate is low.

Subscription and pay-as-you-go pricing also need modeling. The break-even point can be estimated only after defining variables such as monthly request volume, average input and output lengths, and cache-hit rate.

def cost_per_success(reqs, in_tok, out_tok, price_in, price_out,
                     success_rate, retry_factor=1.0, cache_hit=0.0):
    billable_in = in_tok * (1 - cache_hit * 0.75)  # Discounted billing for cache hits
    per_req = (billable_in * price_in + out_tok * price_out) * retry_factor
    successful = reqs * success_rate
    return per_req / successful if successful else float("inf")

3.4 Stability and engineering readiness

Track error-code distribution, including rate limiting, timeouts, upstream 5xx responses, and content blocking. Also measure rate-limit trigger rate, retry success rate, regional availability, and whether usage and call logs are observable. These metrics often appear only during load testing, yet they contribute heavily to production incidents.

4. Bias control for automated evaluation

Using a model to judge models, commonly called LLM-as-a-judge, is efficient, but it has well-documented systematic biases that must be explicitly suppressed:

• Position bias: In pairwise comparison, the judge may favor the first or second answer. Mitigation: evaluate with A/B order swapped and keep only consistent results.
• Length bias: The judge may prefer longer or more visually formatted answers. Mitigation: require in the rubric that “concise and correct is preferred over verbose.”
• Self-preference bias: The judging model may favor outputs from its own model family. Mitigation: use a judge from a different model family than the candidates and retain human spot checks.
• Score drift: The same rubric may produce different scores at different times. Mitigation: pin the judge model version and parameters, and periodically run consistency checks on a fixed anchor sample set.

def pairwise_judge(judge, prompt, ans_a, ans_b, swaps=2):
    verdicts = []
    for i in range(swaps):
        first, second = (ans_a, ans_b) if i % 2 == 0 else (ans_b, ans_a)
        v = judge(prompt, first, second)          # -> "A" / "B" / "TIE"
        v = v if i % 2 == 0 else {"A": "B", "B": "A", "TIE": "TIE"}[v]
        verdicts.append(v)
    wins = verdicts.count("A")
    return "A" if wins > swaps / 2 else ("B" if wins < swaps / 2 else "TIE")

One principle remains essential: automated evaluation is used for regression and shortlisting, but final decisions require human blind-review calibration. For each automated run, sample a subset for human review and track the correlation between human and automated conclusions as a health metric for the evaluation system itself.

5. Architecture: replace “pick one model” with a unified adapter layer

Mature systems do not choose one model. They treat models as replaceable compute resources and abstract them on the application side:

class LLMBackend(Protocol):
    name: str
    def complete(self, req: Request) -> Response: ...
    def stream(self, req: Request) -> Iterator[Chunk]: ...
def route(req: Request, registry: dict[str, LLMBackend], budget: Budget) -> LLMBackend:
    tier = classify(req)                     # Task tiering: simple / routine / hard
    for name in pick_candidates(tier, budget):   # Ordered by capability, latency, and cost
        be = registry[name]
        if be.healthy() and budget.can_afford(name, req):
            return be
    return registry[fallback_name()]

Four capabilities determine system resilience:

1. Routing strategy: Dispatch by task tier, such as sending simple classification or extraction tasks to lower-cost models and complex reasoning to higher-performance models. Adjust weights dynamically based on real-time latency and failure rates.
2. Fallback and failover: Automatically switch to a backup backend when the primary backend times out or hits rate limits. For streaming scenarios, define degradation semantics such as “partial result plus explicit notice.”
3. Caching: Exact caching is suitable for frequent repeated requests. Semantic caching can provide substantial benefits when some business error is acceptable, but it must have similarity thresholds and invalidation policies. Otherwise, it may return wrong answers.
4. Retry and budget: Apply exponential backoff retries only to idempotent failures that are clearly infrastructure errors. Set token and cost budgets per tenant. When budgets are exceeded, degrade instead of failing outright.

If an organization already uses platforms such as Baidu AI Cloud, it can combine the platform’s large model services with general capabilities such as object storage, message queues, logging, monitoring and alerting, and container platforms to build a complete evaluation and operation pipeline: evaluation datasets and call logs stored in object storage, evaluation jobs scheduled asynchronously through message queues, and metrics sent to monitoring and alerting. This is a general engineering practice and does not rely on any vendor-specific requirement.

6. Turn evaluation into a pipeline: from one-time comparison to continuous regression

Model capabilities and billing change continuously. One-time comparisons usually expire within months. The correct approach is to turn evaluation into a pipeline:

# eval-gate.yml (example)
stages:
  - lint_prompts_and_schemas
  - run_offline_eval:        {dataset: regression_v7, samples: 1200}
  - run_latency_probe:       {concurrency: [1, 8, 32]}
  - gate:
      pass_rate_min: 0.93
      schema_valid_min: 0.995
      p95_latency_max_ms: 2500
      cost_per_success_max: 0.014
  - shadow_traffic:          {ratio: 0.05, duration: 24h}
  - report_and_archive

The key elements are gates and shadow traffic. Candidate configurations first run offline evaluation. After passing gates, they enter real environments through 5% shadow traffic or gradual rollout to compare online effectiveness and cost. Only after approval should traffic be increased. Any change to prompts, model versions, parameters, or routing strategy should go through the same pipeline to avoid breaking multiple systems with one small change.

7. Common pitfalls and implementation roadmap

Four common mistakes:

• Replacing internal evaluation with public leaderboards: Public benchmarks differ significantly from business traffic distributions and may be contaminated by training data.
• Testing capability only, without cost and stability: At scale, the combined effect of failure rate and unit price can exceed capability differences.
• Hard-coding prompts into business code: This makes cross-model comparison difficult and prevents fast backend switching.
• Ignoring data governance terms: Requirements for training-data use, log retention, redaction, and data residency should be confirmed before technical validation. Otherwise, later replacement can be very expensive.

Recommended phased roadmap:

1. Weeks 1–2: Definition and data collection
   Define 3–5 core scenarios, sample from real traffic, and build an evaluation set of 500–1,500 samples with assertions.
2. Weeks 3–4: Offline comparison
   Standardize parameters and prompts. Produce a comparison matrix covering capability, latency, and cost. Use human blind review to calibrate automated evaluation.
3. Weeks 5–6: Adapter layer and gradual rollout
   Implement a unified backend abstraction, routing, fallback, caching, and budget controls. Connect monitoring, alerting, and usage dashboards.
4. Week 7 onward: Ongoing operation
   Integrate the evaluation pipeline into CI. Re-run comparisons quarterly. Make renewal and quota decisions based on cost per successful task rather than unit price alone.

8. Conclusion

The overall performance of an LLM service is not a static, rankable property. It is the joint result of capability × latency × cost × stability × compliance under a specific workload.

Turning subjective impressions into assertable, regression-testable, gate-ready evaluation assets, and turning “choose one model” into “a replaceable multi-backend routing architecture,” is the controllable path for keeping up with a fast-changing model ecosystem. Once the evaluation set, metric definitions, and switching capability are in place, adding or replacing any backend becomes a routine pipeline change rather than an architecture-level rewrite.