Three-Scale LLM Evaluation: From Benchmarks to Production Acceptance

1. Start with the right question: three scorecards, not one

Every flagship model release is accompanied by dense capability comparisons: who leads in coding tasks, whose reasoning score is higher, whose context window is longer. This discussion has real value, but it answers only a narrow question: how strong is the model’s problem-solving ability?

In an industrial delivery context, the decision chain actually contains three completely different scorecards:

• Capability scorecard: model answer quality, instruction following, tool-call accuracy, long-context stability;
• Process scorecard: whether an experiment or change can be traced, reproduced, and audited; whether the team can run without the vendor;
• Business scorecard: whether a PoC can pass recorded acceptance, whether the system can be operated independently after handover, and whether the next budget cycle can be renewed.

The common mistake is not overestimating the model. It is using the score from the first scorecard to predict the outcomes of the other two. A full score on the capability scorecard only means “can do”; passing the process scorecard means “safe to use”; passing the business scorecard means “can keep buying”. There is no automatic transfer among these three. Mixing them into one scorecard is the most common evaluation bias in current agent and AI for Science projects.

2. Tier 1: Capability - make model IQ measurement trustworthy

The difficulty in capability evaluation is not “run it once”. It is “run it credibly”. At minimum, engineering needs to solve four things.

1. Private evaluation sets and contamination control. Once a public evaluation set enters the pretraining corpus, its score loses discriminative power. A practical approach is to split the evaluation set into two parts: a public set for routine regression testing, and a private holdout set that is run only once before a release. The private set should never enter training data and should never leave the internal environment.

2. Multiple sampling and confidence intervals. A single output from an LLM is a random variable. A single-point score has almost no decision value. On a fixed evaluation set, sample each item N times, report the mean, variance, and confidence interval, and compare against the baseline using paired comparisons:

Metric report template
  Evaluation set version: eval-set-2026Q1-private (312 items)
  Sampling strategy: temperature=0.0 / N=1        -> first-pass rate 0.61
  Sampling strategy: temperature=0.2 / N=5        -> pass@1 0.68, pass@5 0.84
  Baseline comparison: previous version 0.64 -> new version 0.68 (95% CI: [-0.01, +0.09])
  Conclusion: difference is not statistically significant; insufficient to justify version switch

The last line is the key point: many apparent improvements are not statistically significant. If they are used as the basis for model selection, all later acceptance work will be overturned by reality.

3. Layered metrics, not one total score. Task-level metrics, such as single-step accuracy, and workflow-level metrics, such as end-to-end completion rate, number of manual interventions, and rework rate, often move in opposite directions. A model may be smarter at a single step, but more verbose in tool selection, which makes the end-to-end workflow slower.

4. Failure-mode classification. In addition to the total score, track the proportion of failure types such as hallucinated citations, unauthorized operations, parameter-format errors, and timeout-triggered retry storms. This is often more predictive of production incidents than the total score itself.

3. Tier 2: Process - reproducibility and auditability are the real gate

The core contradiction in the process tier is this: LLM outputs are inherently non-reproducible, but business decisions must be reproducible. The solution is not to pursue byte-for-byte identical outputs. It is to make the five elements - input, configuration, version, artifact, and approval - replayable.

1. Run identity and parameter snapshot. Every invocation should produce an immutable execution manifest. The manifest should include at least the model version identifier, prompt version, sampling parameters, tool set and permission scope, retrieval-library snapshot ID, random seed, and input digest.

{
  "run_id": "exp-20260826-0431",
  "idempotency_key": "design-campaign-A-v7",
  "model": {"endpoint": "inference-svc", "version_tag": "v-2026-08"},
  "prompt": {"template_id": "chem-design", "version": "7", "sha256": "9f2c..."},
  "sampling": {"temperature": 0.2, "top_p": 0.9, "max_output_tokens": 4096},
  "tools": ["query_property_db", "submit_wet_lab_order"],
  "permissions": {"query_property_db": "read", "submit_wet_lab_order": "require_approval"},
  "knowledge_snapshot": "kb-2026-08-20",
  "input_sha256": "1a7d...",
  "output_uri": "object-storage://lab-evidence/0431/",
  "operator": "user:zhang",
  "created_at": "2026-08-26T09:31:02Z"
}

2. Idempotent execution and resume/rerun. Actions with business side effects, such as submitting an experiment order or changing a production configuration, must go through approval and an idempotency key. Repeated submission should not produce secondary side effects:

function execute_with_audit(task, manifest):
    run = registry.begin(manifest)            # Write manifest and obtain run_id
    if registry.has_result(run.idempotency_key):
        return registry.replay(run.idempotency_key)   # Idempotent replay; do not submit twice
    try:
        result = agent.run(task, manifest)     # Main execution path
    except TransientError as e:
        retry.with_backoff(max_attempts=3, budget_ms=15000)
    except PolicyViolation:
        audit.mark_blocked(run.run_id)         # Block unauthorized action and preserve evidence
        raise
    evidence.put(run.run_id, result)           # Archive input, output, and logs to object storage
    registry.commit(run.run_id, digest(result))
    return result

3. The audit chain must answer three questions. Who approved this decision, which model version and knowledge-base version were used at the time, and where are the output artifacts? If these three questions cannot be answered, the compliance review will not pass. A common implementation is to write artifacts to object storage with versioning and write-after protection, send invocation and approval events to a logging service, and connect key metrics to monitoring and alerting. On Baidu AI Cloud, the common combination is object storage, log service, monitoring and alerting, and a container platform. These are generic capabilities and do not require vendor-proprietary interfaces.

4. Runbooks and vendor decoupling. The handover package must include an operations manual, an exception-handling tree, an abstraction layer for model integration so that the model or cloud can be replaced, the evaluation set and regression scripts, and the on-call and escalation path. The acceptance criterion is this: after the vendor team leaves, the customer’s engineers can independently complete one full process drill.

4. Tier 3: Business - turn acceptance into recorded facts

The typical procurement questions almost all fall into the second and third tiers:

1. Can the parameters from the previous execution be found in the system?
2. If another person takes over, can they reproduce the result using the same process?
3. After you leave, can our people run it themselves?

The four phases and exit criteria for a PoC should be explicitly written into the proposal:

Phase 1 Alignment (1-2 weeks)
  Exit criteria: written confirmation of scenario boundaries, data scope, success metrics, and acceptance method
Phase 2 Demonstrable (2-4 weeks)
  Exit criteria: end-to-end run on real data; outputs can be used directly by the business side
Phase 3 Recorded acceptance (2-4 weeks)
  Exit criteria: metrics met + audit chain complete + exception-path drill passed
            Acceptance artifacts are screen recordings and execution logs, not slides
Phase 4 Handover (1-2 weeks)
  Exit criteria: customer team independently executes at least 3 real tasks, including one failure-handling drill

At the renewal stage, what really matters is operational data, not the peak capability. The relevant indicators include the task success-rate trend, the decline curve of manual intervention rate, unit task cost, and incident and recovery time. These metrics should be designed into the instrumentation in the first week, not patched together temporarily before delivery.

5. Cost and stability governance

Capability upgrades often come with a higher unit price. Governance is as important as capability evaluation:

• Tiered routing: low-risk tasks go to smaller models or rule engines; only high-risk tasks are escalated to an LLM;
• Cache reuse: use content-addressed caching for retrieval results and structured intermediate artifacts; cache hit rate directly determines unit cost;
• Budgeted retries: more retries are not always better. Set a total latency budget and a maximum call limit to avoid retry storms;
• Rate limiting and degradation: concurrency limits, queueing policies, and timeout degradation paths should be defined in advance and drilled;
• Cost observability: break down call volume and cost by scenario, tenant, and model version so that “where the cost is high” can be located.

6. Team and roles: what is missing is not another algorithm headline

Process-driven delivery really needs three types of people:

• Delivery/solution engineer: can carry a 4-12 week PoC and connect the four stages of alignment, demonstration, recorded acceptance, and handover, rather than simply stacking workload by person-days;
• Scenario translator: can talk with domain scientists, business owners, and procurement at the same time, translating “what the model can do” into “which process steps need to change”;
• Transitional technical lead: fills the gap between demo narrative and procurement narrative, and is responsible for architecture convergence, risk backstopping, and acceptance-artifact formation.

Correspondingly, interview assessment should shift from “which models have you tuned” to: how to design an evaluation set, how to build an audit chain, how to handle idempotency and unauthorized actions, how to define PoC exit criteria, and how to conduct a complete failure retrospective.

7. Common misjudgment checklist

• Using public evaluation-set scores to replace business-scenario validation;
• Drawing conclusions from a single sample without variance or confidence intervals;
• Counting only successful cases without classifying failure modes;
• Having a demo but no audit chain, so parameters and versions cannot be replayed;
• Lacking approval and idempotency controls for side-effect actions;
• Missing runbooks, so handover degrades into “just need a person on site”;
• Having a PoC without written exit criteria, so acceptance becomes subjective judgment;
• Adding instrumentation only before delivery, leaving no operational data for renewal discussions.

8. Summary

Flagship model capability jumps are worth watching, but they only answer “how much can it do”. Whether a system can be put into production depends on two other things: process-level reproducibility and auditability, and business-level acceptance and handover. If evaluation is split into three tiers, with clear metrics and exit criteria assigned to each tier, a team can turn the heat of a launch event into a production line that can be continuously operated and renewed.