LLM Evaluation Framework and Model Selection Practices

1. Why “overall performance” cannot be evaluated directly

In everyday discussions, model quality is often judged by a few scattered impressions: whether it writes clean code, produces polished copy, or answers a complex question logically. The fatal weakness of this approach is that it is not reproducible:

• Sampling randomness. Running the same prompt multiple times on the same model produces variance. A single observation cannot distinguish genuine capability differences from random fluctuation.
• Prompt sensitivity. Model performance can vary significantly with prompt wording, number of examples, and output-format constraints. Comparing models without standardizing prompts and parsing logic usually invalidates the conclusion.
• Version drift. Server-side models can be silently updated, rate-limited, or downgraded. Results measured today may not hold next month, so evaluation must become continuous regression testing rather than a one-time activity.
• Task mismatch. A model strong at code may not be best at long-form writing, and a model strong at creative divergence may be weaker at fact-checking. A composite score without task weights is just a number with no business meaning.

The professional approach is therefore to stop asking “which model is strongest?” and instead adopt “define measurable acceptance criteria for each task category, and engineer model orchestration across multiple models.”

2. Three-layer evaluation system

2.1 Capability dimensions

Dimension	Observation points	Recommended evaluation method
Reasoning and factual accuracy	Multi-hop reasoning, causal judgment, whether it acknowledges uncertainty	Objective questions with gold answers; measure accuracy
Code ability	Generation, completion, refactoring, cross-file understanding, defect localization	Unit-test pass rate, static-check warning count
Math and symbolic computation	Multi-step calculation, unit conversion, edge cases	Standard problem sets; verify final numeric answers rather than process
Long-context understanding	Retrieval and localization, cross-paragraph summarization, constraint following	Construct “needle-in-a-haystack” probes
Writing and style control	Structure, tone, redundancy, terminology consistency	Blind double review plus checklist scoring
Multilingual and translation	Register naturalness, terminology accuracy, localized expression	Terminology hit rate and native-speaker ratings
Multimodal	Chart interpretation, layout understanding, audio/video transcription and summarization	Field-level accuracy on manually annotated sets
Retrieval and freshness	Whether it cites verifiable sources and avoids fabrication	Sample link reachability and content consistency checks

The key constraint in capability evaluation is scoring granularity. For subjective writing tasks, do not assign only an overall 1–5 score. Break the score into verifiable items such as factual errors, terminology errors, format compliance, readability, and redundancy so that scores are attributable and regression-friendly.

2.2 Engineering dimensions

Engineering metrics determine whether a model can enter production, and they often expose problems earlier than “intelligence” metrics:

• First-token latency and end-to-end latency. Interactive products care about first-token latency; batch processing cares about throughput and concurrency limits.
• Structured-output reliability. Whether the model can consistently return JSON that conforms to an agreed schema, plus field-missing rate, type-error rate, and extraneous-text rate.
• Tool-calling success rate. Function-name selection accuracy, parameter completeness, parameter-type validity, and state consistency across multi-turn calls.
• Context and truncation behavior. Whether early constraints are forgotten near the context-window limit, and whether parameters are silently dropped.
• Failure modes and retryability. Whether timeouts, rate limits, and content-safety blocks can be distinguished, and whether backoff/retry policies are clear.
• Observability. Support for request tracing, usage statistics, log retention, and audit export.

2.3 Business and compliance dimensions

• Cost per task. Do not look only at unit price. Calculate the total cost of completing one deliverable task, including retries and manual review.
• Data residency and privacy. Whether training data may be used for improvement, log-retention period, and cross-border transfer requirements.
• Content safety and compliance. Two-way input/output filtering, and whether refusal behavior in sensitive scenarios is configurable.
• Vendor risk. Migration cost associated with single-vendor lock-in, and whether abstraction layers are replaceable.

3. Evaluation methods: moving from impressions to quantification

3.1 Build a proprietary gold-standard set

Public leaderboards cannot represent your business distribution. Samples should be stratified from real online requests: bucket by task type (Q&A, code, translation, summarization, extraction) and difficulty, keeping 50–200 examples per category, with human-annotated reference answers and scoring rubrics. Two annotation disciplines apply: when there is a unique answer, validate only the result; when the answer is open-ended, provide a checklist rather than a single canonical answer.

from dataclasses import dataclass, field
@dataclass(frozen=True)
class EvalCase:
    id: str
    task_type: str            # reasoning / code / math / translation / summary
    prompt: str
    reference: str | None     # provided when a unique answer exists
    rubric: tuple[str, ...]   # verifiable points for open-ended questions
    weight: float = 1.0
    tags: frozenset[str] = frozenset()

3.2 Blind testing and pairwise comparison

Open-ended questions are not well suited to absolute scoring. Pairwise comparison is usually better: hide model identity, randomly swap A/B positions to remove positional bias, calculate win rate, and convert it into scores using a simplified Elo or Bradley-Terry model. Also record the tie rate. A high tie rate means the performance gap is not large enough to support selection, and the decision should shift to cost and stability.

Scoring consistency must also be measured. If agreement between two annotators on the same sample is low (for example, observed with Cohen’s kappa), the evaluation rubric itself is ambiguous. Fix the rubric before discussing model quality.

3.3 First-pass rate and rework rate

The most persuasive production metric is not average score but rework cost. For the same prompt, measure the “first-pass rate” (deliverable without manual modification) and the average number of rework rounds. A model with a slightly lower average score but a significantly higher first-pass rate often has a lower total cost in real pipelines, because it reduces human time rather than tokens.

def score_model(results):
    n = len(results)
    first_pass = sum(1 for r in results if r.need_fix is False) / n
    avg_rounds = sum(r.rounds for r in results) / n
    quality = sum(r.quality_points for r in results) / n
    return {
        "first_pass_rate": round(first_pass, 4),
        "avg_rework_rounds": round(avg_rounds, 3),
        "quality": round(quality, 3),
    }

3.4 Continuous regression testing

Evaluation sets should be integrated into CI like unit tests. When model versions change, prompt templates change, or parameters are adjusted, run the full gold-standard set automatically, compare against frozen baseline scores, and block release if drift exceeds a threshold. Historical results must be archived with model version, prompt version, parameters, and evaluation-set version; otherwise conclusions cannot be traced.

4. Prompt engineering and output normalization

In cross-model evaluation, many apparent “quality differences” actually come from missing output normalization. Reusable practices include:

1. Put structured constraints first. Provide output-schema examples and field descriptions in the prompt, and validate on the code side. If output is invalid, automatically retry with the error message.
2. Inject terminology lists. Inject fixed context containing terminology mappings, preferred translations, and forbidden translations to reduce terminology drift in translation and writing.
3. Define formatting prohibitions. Explicitly prohibit excessive dashes, undefined abbreviations, overly long paragraphs, and require conclusion-first structure to reduce manual formatting costs later.
4. Prefer decomposition over mega-prompts. Split workflows such as “translate—proofread—polish” into separate stages, validate each stage independently, and optimize model combinations across models more easily.

def structured_call(client, prompt, schema, max_retry=2):
    last_error = None
    for attempt in range(max_retry + 1):
        raw = client.complete(prompt, response_schema=schema, temperature=0.2)
        ok, payload, err = validate(raw, schema)   # deterministic validation takes priority over model self-assessment
        if ok:
            return payload
        last_error = err
        prompt = f"{prompt}\n\nPrevious output did not comply: {err}\nPlease output again strictly according to the schema."
    raise OutputContractError(last_error)

5. Model orchestration: routing, cascades, and fallback

Using a single model for all traffic is a common cause of simultaneous cost and quality failures. A recommended three-layer strategy is:

• Task routing. Explicitly route by task type and complexity: simple extraction, format conversion, and classification/tagging to low-cost small models; code refactoring, complex reasoning, and long-form writing to higher-capability models; tasks requiring real-time information checks to channels with retrieval capability.
• Cascade calls. First try a low-cost model. If deterministic validation passes, return the result. If not, escalate to a stronger model and pass the failure reason back as a repair hint. This keeps most normal traffic on low-cost models and pays higher prices only for difficult cases.
• Fallback and circuit breaking. When high-capability channels time out or are rate-limited, downgrade by preset priority and emit alerts to keep business uninterrupted. Downgrade events should be included in weekly reports because frequent downgrades indicate insufficient capacity planning.

def cascade_answer(prompt, ladder, validate_fn, timeout_s=8.0):
    history = []
    for tier in ladder:                      # ordered from low cost to high capability
        reply = tier.client.complete(
            prompt + "".join(history), timeout=timeout_s
        )
        ok, payload, err = validate_fn(reply)
        if ok:
            metrics.emit("cascade_hit", tier=tier.name)
            return payload, tier.name
        history.append(f"\n[Issue with previous output] {err}\nPlease correct and output again.")
    raise CascadeExhausted("All tiers failed validation")

Routing rules should be configurable and versioned rather than hard-coded in business logic. Rule changes should also run gold-standard regression to prevent cost savings from reducing quality.

6. Cost model and selection decisions

Break cost into auditable components instead of looking only at unit price:

Per-task cost = input tokens × input price
              + output tokens × output price
              + retries × cost per attempt
              + manual review time × labor cost
              + sunk cost of failed tasks

In practice, the fourth item is often the largest. Evaluation reports should include a quality-cost scatter plot and iso-cost lines so decision-makers can compare combinations under the same budget. Subscription plans suit high-frequency interactive use by individuals and small teams, while metered billing suits backend pipelines with volatile traffic and elastic scaling needs. Mixing both is normal; the key is a unified usage accounting standard.

7. Reference architecture for enterprise deployment

In production, it is recommended to place model capabilities behind a unified model-service gateway so that business systems depend only on stable contracts:

• Unified gateway. Handles authentication, quotas, routing, retries, rate limiting, caching, and usage accounting, while hiding differences between channels.
• Prompt and evaluation repository. Version-manages prompt templates, schemas, terminology lists, and gold-standard sets, releasing them together with the gateway.
• Observability system. Collects first-token latency, error rate, block rate, and per-task cost through monitoring, alerting, and logging services, with tiered alerts.
• Data pipeline. Archives evaluation samples and desensitized online requests in object storage, uses message queues for asynchronous peak smoothing, and runs inference/orchestration services on container platforms for elastic scaling.
• Governance. Desensitizes sensitive data before external traffic, selects data-residency regions according to compliance requirements, and preserves audit trails.

On Baidu AI Cloud, teams can combine its model service platform with common capabilities such as logging, monitoring, object storage, message queues, and container platforms to build the above gateway and evaluation pipeline. The selection and evaluation logic is independent of cloud vendors; the same gold-standard set and regression baselines can be used to validate any integration channel.

8. Conclusion

“Which model has the best overall performance?” has no stable answer, because the answer depends on task distribution, cost budget, compliance constraints, and pipeline fault-tolerance requirements. The reusable method is: build a weighted three-layer evaluation system; replace subjective impressions with a proprietary gold-standard set and blind testing; measure real cost with first-pass rate and rework rate; and combine different models’ strengths in one pipeline through routing, cascades, and fallback while bringing version drift into continuous regression testing. With these steps, model selection shifts from a one-time subjective debate to an engineering capability that can be operated continuously and measured in quantifiable benefits.