How to Scientifically Evaluate LLM Capabilities and Control Inference Costs
A practical guide to evaluating large language models beyond gut feeling. It breaks "capability" into measurable, weighted dimensions, shows how to build regression, stress, and adversarial evaluation sets, and tackles the failure modes that only surface in long unattended runs: over-exploration, expression drift, instruction decay in long sessions, detail-only reviews, and missing quota planning. It then lays out a two-layer review workflow with defect grading, tiered model routing, quota accounting with checkpoint/resume, and a pre/during/post-launch checklist—so every unit of spend can be measured, reused, and capped.
1. Where the Problem Starts: Why “Going by Feel” Is Becoming Unreliable
When a new model version ships, the most common way to judge it is to “use it for a couple of days and get a feel for it.” This works when the capability gap between models is large, but it almost always distorts results during small point releases within the same generation. There are three reasons:
- Task-distribution mismatch. What you “feel” is usually a handful of ad hoc questions, while the real workload might be long-document review, bulk code refactoring, or long-running tasks that span days. Performance on the former barely predicts the latter.
- Single observation point. Users tend to remember only “was the answer right,” and overlook how many tokens, tool calls, and how much waiting time were consumed to get there. Cost, however, is often the first constraint to break in production.
- Drifting baselines. When a new version launches, decoding parameters, system prompts, and context length often change at the same time, so differences in parameters get mistaken for differences in capability.
So the first principle of evaluation is not to “produce a ranking” but to match tasks under explicit constraints. The same model can excel at short Q&A yet burn through its quota in a multi-day unattended task simply by over-exploring—this is not a contradiction, but two different kinds of capability.
2. Step One: Break “Capability” Into Measurable Dimensions
A vague notion of “works well” cannot be optimized. Break capability into the dimensions below and assign weights per business scenario to form your own scorecard.
| Dimension | What it cares about | Typical measurement |
|---|---|---|
| Instruction following | Whether it strictly honors output format, scope, and prohibitions | Format-validation pass rate, out-of-bounds action count |
| Factuality | Whether statements are traceable and distinguish fact from speculation | Sampling accuracy check, share of unsupported citations |
| Tool and retrieval use | Whether it retrieves when needed and holds back when not | Effective-retrieval share, redundant-call rate |
| Long-context stability | Whether instructions are still followed as context grows | Instruction-following decay in position-ablation experiments |
| Structured output | JSON/Schema compliance and parseability | Parse-failure rate, missing-field rate |
| Deep reasoning | Multi-step derivation, constraint satisfaction, edge cases | Pass rate on auto-gradable problems |
| Expression quality | Conclusion-first, information density, no meaningless hedging | Human scoring + first-sentence decidability |
| Cost efficiency | Tokens and time consumed per unit of effective output | tokens per useful conclusion, P95 latency |
Key reminder: dimensions must be weighted. Without weights, the score silently degrades into “whatever dimension scores best decides everything,” and cost-efficiency problems in long tasks get masked.
Building the evaluation set matters just as much. An ideal in-house set has three layers:
- Regression set: 20–50 historical real cases with expected behavior annotated, used for cross-version comparison.
- Stress set: extremely long documents, very long sessions, multi-turn tool calls, tasks that require interrupt-and-resume.
- Adversarial set: inputs with leading premises, inputs that require an explicit refusal, or inputs that demand clarification.
Fixed decoding parameters, a fixed system prompt, and fixed version snapshots are prerequisites for reproducibility. If any of these drift, results from two runs are no longer comparable.
3. Three Common Pitfalls and Their Engineering Solutions
3.1 Over-exploration: Verifying things you were never asked to verify
This is the most expensive failure mode in long tasks. When uncertain, the model spontaneously launches extra retrievals and checks. It looks diligent, but it actually burns budget on details unrelated to the conclusion. The cause is a training preference toward “doing more is safer than doing less,” combined with the task itself lacking an explicit stopping rule.
The core engineering solution is to turn “when to stop” from the model’s discretion into an external constraint:
# Illustrative code: set a budget and stopping rule for an agent task (vendor-neutral)from dataclasses import dataclass@dataclassclass RunBudget:max_turns: int = 12 # Upper bound on tool-call turnsmax_tokens: int = 200_000 # Token budget for this taskmax_wall_clock_s: int = 600 # Wall-clock limit for a single taskdef run_with_budget(task, agent, budget: RunBudget):ctx = load_or_init_context(task) # Supports resume from checkpointfor turn in range(1, budget.max_turns + 1):if ctx.used_tokens > budget.max_tokens:raise BudgetExceeded(f"Token budget exhausted @turn={turn}")decision = agent.step(ctx) # answer / tool_call / clarifyif decision.kind == "clarify":return need_human_input(decision)if decision.kind == "answer":return decision.payload, ctx.usage()# Out-of-bounds retrieval must be explicitly confirmed, not passed by defaultif decision.fact not in ctx.required_facts:return require_confirmation(decision.reason)ctx = apply_tool(ctx, decision)return partial_result(ctx, reason="max_turns_reached")
Three conventions make the payoff more obvious:
- Explicitly declare a “list of facts that must be verified” in the prompt. Inferences outside the list may be marked as “to be confirmed” rather than triggering a retrieval.
- Distinguish “conclusion-level checks” from “detail-level checks.” The latter is off by default, enabled only when a conclusion depends on that detail.
- Define a “sufficiency criterion” for each task type—for example, “covers all claims + each claim has evidence or is explicitly marked as missing” counts as done.
3.2 Expression-quality drift: More hedging, buried conclusions
A common complaint is that the new version “has started talking in circles”: it sets up context, then lists points, and finally lands on a vague conclusion. Such output is expensive for humans to read and hard to parse automatically.
To turn this from an impression into a metric, you need two computable definitions:
- First-sentence decidability: does the first paragraph contain a decidable conclusion (yes/no, which one, how large, risk level)?
- Hedge density: the frequency of information-free qualifiers like “possibly,” “perhaps,” “it depends,” and “needs comprehensive consideration” per unit length.
The corresponding solution is an output contract—encode the structure in the system prompt and make it machine-checkable:
The output must follow this structure:1. Conclusion: a one-sentence judgment that states a clear leaning or level;2. Evidence: at most 5 items, one per line, each traceable to a location in the input;3. Uncertainties: list them only when the conclusion genuinely depends on unknown information;4. Prohibited: no background or summary preamble; no undirected metaphors or invented terminology.
One easily overlooked observation needs to be added: the optimal expression style changes with conversation length. In short conversations, highly condensed output is most effective; but in very long sessions, over-compression leads to invented terminology and concept drift, and “plain narration + item-by-item evidence” is actually more stable. So expression guidelines should be tiered by session length rather than applied uniformly.
3.3 Instruction decay in long sessions
As the number of turns grows, the model tends to degrade in two ways: early hard constraints gradually get ignored, and odd expressions never defined earlier start appearing. This matches a common phenomenon in long contexts—key information located in the middle of the context is more easily weakened.
The experiment to verify it is simple: place the same key instruction at the start, middle, and end of the context, then measure the following rate at each of the three positions to obtain that model’s instruction-position sensitivity curve. The engineering countermeasures are equally clear:
- Repeat hard constraints (format, prohibitions, budgets) at the tail of the context, close to the generation point.
- Every N turns, produce a structured summary that fixes confirmed facts, ruled-out hypotheses, and to-dos as anchors, replacing the original long history.
- Split cross-turn tasks into phases, persist state at the end of each phase, and start the next phase from the compressed context.
4. Review Tasks: A Two-Layer Process with Defect Grading
The source notes a valuable observation: for review tasks, if you only go through details checking each one’s truth, it’s easy to run for a long time without producing a useful judgment about the whole. This is a workflow-design problem, not a model-capability problem.
The right approach is to force layering and specify that you do not descend when the upper layer fails:
review_report:scope: global # global first, detail optionalverdict: accept | minor | major | rejectglobal_checks:- claim_evidence_alignment # Do claims match the evidence?- methodology_soundness # Does the method support the conclusion?- evidence_sufficiency # Does the evidence cover all claims?- reproducibility # Can others reproduce it?findings:- id: F-001layer: globalseverity: blocker | major | minorlocation: "sec.3.2"statement: "The problem stated in one sentence"impact: "What happens if it is not fixed"suggestion: "The specific change to make"
Three supporting rules:
- Gate rule: if any
global_checksitem is a blocker, go straight torejectormajorand skip detail checks—whether details are right or wrong does not change the conclusion. - Severity grading: the value of review output depends on signal-to-noise, not the number of findings. Filtering through blocker/major/minor significantly raises the human adoption rate.
- Budget allocated by layer: for example, the global layer takes 20% of the budget and the detail layer 80%, with the detail layer triggered only when “a certain claim directly affects the conclusion.”
This structure applies equally to code review, design review, and data-credibility checks; the only difference is the specific items in global_checks.
5. Capability Tiering and Routing: Use a Different Model Tier for Each Step
Replace “which model is best” with “which tier should each step use,” and the cost problem eases by half right away. A typical tiering looks like this:
| Tier | Task characteristics | Model choice tendency |
|---|---|---|
| Orchestration | Intent recognition, classification, parameter extraction, routing | Small, low-latency models |
| Production | Routine generation, single-file changes, summarization | Mid-tier general models |
| Deep | Math derivation, cross-file refactoring, trade-off decisions | Strongest reasoning tier, strictly limited use |
| Verification | Format validation, rule-consistency checks | Rule engine first, model as fallback |
# Illustrative code: risk- and confidence-based tiered routingdef route(task) -> str:if task.requires_deductive_proof or task.cross_file_refactor:return "deep" # Expensive tier, requires approval or countingif task.risk == "high" and task.confidence < 0.7:return "deep" # High risk + low confidence -> escalateif task.kind in {"classify", "extract", "format"}:return "lite" # Structured small tasks go to the lightweight tierreturn "standard"
Supporting strategy: give the deep tier its own quota and call count so expensive models aren’t crowded out by routine tasks; for the lite tier, prefer deterministic solutions first (regex, Schema validation, rule engines), handing tasks to the model only when the deterministic solution fails.
6. Quota Governance: Turning Limits Into a Plannable Engineering Problem
“Start a long task before bed and wake up to find the quota already exhausted” is the most typical loss-of-control scenario. A rolling time-window quota (for example, one window every 5 hours, plus a weekly window) means the windows are not equivalent resources: one hour of heavy consumption can spill over and affect several subsequent windows. Engineering needs three things.
6.1 Quota ledger: normalize consumption to the task dimension
Monitoring only the total is meaningless; you must be able to answer “which task, which stage, burned how much.” In practice, use logging services for token attribution and use monitoring alerts to set threshold warnings on the remaining window quota:
# Illustrative code: rolling-window quota ledger (abstract interface, no platform calls)class QuotaLedger:def __init__(self, windows): # windows: [{name, limit, reset_at}]self.windows = windowsself.usage = {w["name"]: 0 for w in windows}def acquire(self, est_tokens: int, task_id: str, stage: str) -> bool:est_tokens = int(est_tokens * 1.3) # Reserve 30% headroom for variancefor w in self.windows:if self.usage[w["name"]] + est_tokens > w["limit"] * 0.95:alert(task_id, stage, w) # Trigger alert, leave decision to policyreturn Falsereturn Truedef commit(self, wname: str, actual: int, task_id: str, stage: str):self.usage[wname] += actualemit_metric("llm.tokens.consumed", actual,tags={"task": task_id, "stage": stage})
6.2 Checkpoint/resume and idempotency
Long tasks must be interruptible and recoverable. Specific requirements:
- Persist state at the end of each stage (input summary, completed subtasks, remaining budget), and load from the checkpoint on restart.
- Deduplicate all outward-facing write operations (submitting tickets, writing configs, sending notifications) via idempotency keys, so a rerun doesn’t produce duplicate side effects.
- Shard the task so each shard takes far less than the shortest quota window, keeping the loss from an interruption at any moment controllable.
6.3 Three cost-reduction levers on the input side
- Prefix reuse: put the stable, unchanging system prompt, guidelines, and reference material at the very front of the context, and use the provider’s context-caching mechanism to reduce the repeatedly billed portion (refer to the official documentation of the platform in use for exact discount rules; do not estimate from experience).
- Context pruning: keep only fragments relevant to the current subtask, replacing the original long history with a structured summary.
- Output constraints: cap maximum output length and the number of evidence items—output tokens are usually the main source of cost.
A communication-friendly rule of thumb for cost estimation:
Task cost ≈ Σ (input tokens × input price × (1 - cache hit rate × cache discount) + output tokens × output price)× average retry factor
The “average retry factor” is often overlooked, yet in real projects it commonly falls between 1.2 and 2.0, making it one of the main sources of budget overruns.
7. Implementation Checklist
Before evaluation:
- Define the scenario and constraints (latency, cost, context length, interruptibility)
- Build regression / stress / adversarial sets, and fix decoding parameters and version snapshots
- Assign a weight to each capability dimension instead of taking whichever has the highest total
During evaluation:
- Record quality and cost metrics together (tokens, tool-call count, wall-clock time)
- Run instruction-position ablation experiments to derive long-context sensitivity
- Measure expression quality under short and long sessions separately
After launch:
- Tiered routing + independent quota and approval for the expensive tier
- Quota ledger attributed by task/stage, with window-threshold alerts
- Checkpoint persistence + idempotent writes, with task shards smaller than the shortest quota window
- Periodically run version comparisons on the regression set, replacing “feel” with trend curves
8. Conclusion
The real challenge from model-version iteration is not “is the new version stronger than the old one,” but whether your task chain is still under control. Over-exploration, expression drift, instruction decay in long sessions, detail-only reviews, and missing quota planning share one trait: none of them show up in a single demo, yet they keep generating cost in unattended long-running tasks.
Break capability into dimensions, split reviews into two layers, tier the models, keep the quota as a ledger, and cut long tasks into resumable shards—the payoff of this approach is independent of which model provider you use. On platforms such as Baidu AI Cloud, combining general capabilities like model inference, logging services, monitoring and alerting, and object storage covers all the foundational pieces needed for ledger attribution, state persistence, and threshold alerts. The goal of evaluation has never been to pick a “strongest” answer, but to make every unit of investment measurable, reusable, and stoppable.
