LLM Valuation Returns to Data: How Model Selection Moves to Quantifiable Evaluation
As leading model providers' financing and IPO activity push valuations back toward verifiable financials, engineering teams face two concrete questions: how good is a given model on our own workloads, and what latency and cost does each effective task actually incur? This article lays out an engineering method for moving from narrative metrics to quantifiable evaluation. It covers a three-tier metric system (narrative, measurement, business) and the quantitative relationships between them, how to build reproducible layered evaluation sets, automated scoring (L0–L2) with human calibration, unified token/latency/throughput cost definitions, cost-reduction strategies such as tiered routing, semantic caching, context trimming, and distillation with quantified benefit calculations, and continuous evaluation and governance. It also lists common pitfalls and a phased rollout roadmap, showing how model selection and cost governance can be made data-driven, reproducible, and durable as models and prices keep changing.
1. Introduction: After the Narrative Fades, Evaluation Has to Rely on Data
The financing and IPO cadence of leading model providers has shifted noticeably recently. One major overseas model company has kicked off its IPO roadshow, with an implied market cap that has climbed past the USD 1 trillion mark at the top of the indicative price range. Its disclosed financials show that roughly 80% of revenue comes from usage-based API calls, while nearly all contractual obligations and commitments relate to compute infrastructure. Another leading overseas vendor’s latest funding round fell short of early expectations; although its adjusted operating margin improved quarter over quarter, the absolute value remains deeply negative.
Meanwhile, after a dense summer of launches, some regional model vendors have slowed their release cadence. The market reads this as model-company valuations returning from “narrative support” to verifiable financial numbers.
A shift in valuation logic quickly propagates to technical decision-making. Once the industry starts asking “who is actually being used?” and “how much compute cost does each unit of revenue require?”, engineering teams should be able to answer two concrete questions of their own:
- How good is this model on our business, really?
- How much latency and cost does each effective task cost us?
Starting from this question, this article systematically lays out an engineering method that moves from narrative metrics to quantifiable evaluation, covering evaluation-set construction, automated scoring, cost accounting, cost-reduction strategies, and ongoing governance.
2. What the Industry Signals Mean for Engineering Teams
Three signals from the capital-market shift in valuations relate directly to engineering practice.
Signal 1: Call volume has become a core asset. With revenue concentrated in usage-based billing, real call volume is an important support for valuation. On the engineering side, what we care about is effective call volume—outputs actually adopted by the business—rather than raw request volume that includes retries, dry runs, and invalid generations.
Signal 2: Transparent pricing makes cost accountable. Mainstream model services bill by token and distinguish input, output, and cache hits. The high share of usage-based revenue shows this billing model has become the industry norm. Because call volume and cost are almost linearly related, every request can be precisely converted into a monetary figure.
Signal 3: The supply model shapes the cost structure. Leading vendors’ contractual commitments are heavily concentrated in compute infrastructure, consistent with the long-horizon compute-reservation model common across the industry. For teams building their own inference or purchasing compute, utilization directly determines unit cost. Idle compute during off-peak hours is essentially wasted reserved cost.
An easily overlooked detail: definitions (the “unit of measure”). The financial disclosure that a leading vendor shows a positive adjusted operating profit while still posting a loss under accounting standards is a reminder: whether the metric is average latency or P95 latency, call count or successful-task count, if the definitions are not aligned, the conclusions are not comparable. The first task of any engineering measurement is to define the units clearly.
Core point: The shift in valuation from narrative to financials is essentially a shift in the evaluation standard from “telling a story” to “verifiable results.” Model selection should follow the same logic: let task success rate, latency, and unit cost on your own business data do the talking.
3. Methodology: Three-Tier Metrics and Mapping from Narrative to Business
Traditional “narrative-style” evaluation usually stops at three levels:
- Leaderboard rankings: composite scores from general-capability tests;
- Parameter scale: model parameter count, maximum context window;
- Marketing language: unverifiable claims like “near human-level.”
These are useful as first-pass screening signals, but they cannot answer “how will it perform inside my system?” Below we build a three-tier evaluation framework.
3.1 Three-Tier Metric Definitions
| Tier | Focus | Representative metrics | Common pitfall |
|---|---|---|---|
| Narrative (signal) | External information, initial screening | Leaderboard composite score, context length, multimodal support | Unrelated to your own business distribution, not reproducible |
| Measurement (core) | Task-level quality, performance, and resources | Task success rate, field-level accuracy, P50/P95 latency, token consumption | Only taking the mean, unfixed sampling parameters, inconsistent definitions across models |
| Business (goal) | Value output and operating results | Adoption rate, human-fallback rate, cost per successful task, cost-savings rate | Using unit price instead of total cost, ignoring failed retries and human cost |
The three tiers support each other from bottom to top: the measurement tier answers “is it good and fast?”, and the business tier answers “is it worth it?” Information from the narrative tier enters only as a signal for validation in the measurement tier and does not participate in decisions directly.
3.2 Quantitative Relationships Between Metrics
To genuinely connect the three tiers, we can establish the following relationships:
Average cost per task ≈ avg input tokens × input unit price + avg output tokens × output unit priceCost per successful task = average cost per task / task success rateEffective throughput ≈ successful tasks per unit time × average output tokens
Expanded, this shows that cost and quality are not independent:
- Raising the success rate directly lowers the cost per unit of effective output;
- Failed requests still consume input tokens and may even trigger retries;
- Long context drives up both latency and cost.
So model selection and optimization are essentially about finding the Pareto-optimal point in a space defined by “quality–cost–latency,” not about competing for rank on a single leaderboard.
4. Step 1: Build a Reproducible, Layered Evaluation Set
The evaluation set is the cornerstone of the whole system. Using a customer-service scenario as an example, evaluation should cover the following dimensions:
- Business domain: pre-sales inquiries, order after-sales, refund disputes, and so on;
- Input length: short questions, long conversation histories, attached long documents;
- Output form: free text, structured fields, multi-turn dialogue;
- Language: Chinese, English, and the minor languages the business touches.
When sampling, stratify by online traffic share so the evaluation set is distributed like real requests. Five points matter in the design:
- Sufficient sample size. With too few samples, a difference of a few percentage points may just be random noise. Estimate the required sample size from the “minimum effect size you want to distinguish” and the “acceptable confidence interval” before organizing annotations.
- Real sources. Rely mainly on desensitized sampling of online logs, supplemented by manual construction. This covers long-tail cases while avoiding a test that only probes “topics the model is good at but the business rarely hits.”
- Difficulty and diversity. During annotation, record input length, whether it is multi-turn, whether ambiguity exists, and so on, so bottlenecks can later be broken down by dimension.
- Contamination prevention and compliance. Deduplicate evaluation items against online requests; avoid heavy overlap with public test sets to prevent inflated scores; user data must be desensitized and handled in line with the relevant compliance requirements.
- Versioning and updates. Record the evaluation-set version and scoring-rule version for every model or prompt change; periodically add hard cases from new traffic to prevent overfitting.
The evaluation set can be split by purpose into three types:
| Type | Suggested size | Update cadence | Main use |
|---|---|---|---|
| Smoke set | A few dozen items | As business changes | Fast regression on every change |
| Regression set | A few hundred items | Supplemented quarterly | Comparison when models or prompts are upgraded |
| Challenge set | A few dozen items | Continuously accumulated | Finding capability limits, guiding routing and fallback strategies |
Here is an illustrative code sample:
import jsonimport randomfrom dataclasses import dataclass, fieldfrom typing import Any, Callable@dataclassclass Task:"""A single evaluation task; messages is a list of desensitized conversation messages"""tid: strmessages: list[dict[str, Any]]expect_schema: dict[str, Any] | None = None # Expected output structure (JSON Schema)reference: Any = None # Expected output (for structured tasks)rubric: list[dict[str, Any]] = field(default_factory=list) # Scoring dimensions and weightstags: dict[str, str] = field(default_factory=dict) # domain/language/input length, etc.weight: float = 1.0 # corresponding online traffic sharedef sample_layered(tasks: list[Task], ratio: float = 0.2, seed: int = 42) -> list[Task]:"""Stratified sampling by tag combination, keeping each distribution dimension at the same ratio"""groups: dict[tuple, list[Task]] = {}for t in tasks:key = tuple(sorted(t.tags.items()))groups.setdefault(key, []).append(t)picked: list[Task] = []rng = random.Random(seed)for group in groups.values():k = max(1, round(len(group) * ratio))picked.extend(rng.sample(group, k))return pickeddef bootstrap_ci(values: list[float], n_boot: int = 1000, alpha: float = 0.05) -> tuple[float, float]:"""Bootstrap resampling of task-level scores to get the mean CI; especially useful for small samples"""rng = random.Random(42)means = []for _ in range(n_boot):sample = [values[rng.randrange(len(values))] for _ in values]means.append(sum(sample) / len(sample))means.sort()lo = means[int(alpha / 2 * n_boot)]hi = means[int((1 - alpha / 2) * n_boot) - 1]return lo, hidef compare(task_scores: dict[str, list[float]]) -> dict[str, dict[str, float]]:"""Paired comparison of two models; if the difference CI spans 0, no significant difference is claimed"""a, b = list(task_scores)assert len(task_scores[a]) == len(task_scores[b]), "task sets differ; cannot do paired comparison"diffs = [x - y for x, y in zip(task_scores[a], task_scores[b])]lo, hi = bootstrap_ci(diffs)mean_a = sum(task_scores[a]) / len(diffs)mean_b = sum(task_scores[b]) / len(diffs)return {a: {"mean": mean_a},b: {"mean": mean_b},"diff": {"lo": lo, "hi": hi, "significant": lo > 0 or hi < 0},}
With a reproducible evaluation set in place, the next step is to automate “scoring” while controlling the scoring error itself.
5. Step 2: Automated Scoring and Human Calibration
The scoring method should be chosen by task type and layered by reliability, from highest to lowest:
- L0 Deterministic checks: schema validation, regex and enum matching, numeric-tolerance comparison, running code in an isolated sandbox and passing unit tests;
- L1 Reference comparison: field-by-field comparison for structured tasks, semantic similarity for text tasks;
- L2 Model judge: for open-ended tasks, a judge model scores against a detailed rubric.
Specific recommendations:
- Hard checks first. Anything that can be decided by L0 should never be handed to a judge model—this saves cost and avoids scoring randomness.
- Break rubrics down to dimensions. Give each dimension explicit score anchors (e.g., 0/1/2 for “not met / partially met / fully met”) rather than a vague “good/bad.”
- Calibrate the judge’s results. Periodically sample for human re-review and measure the agreement rate between judge and human scores; when agreement is too low, revise the rubric first rather than trusting the judge outright.
- Score long outputs in segments. For long-form generation, score segment by segment and then aggregate by task weight, so early content does not bias the overall judgment.
- Aggregate execution results. Run a single model 3 to 5 times, take the median and track the variance, to separate real capability differences from sampling noise.
If you use a model judge, keep the following in mind:
- Double-blind and position randomization: do not tell the judge which model produced which candidate output, and swap the candidate order for a second pass, to offset brand bias and position preference;
- Fix the judge version and parameters: set temperature to 0, and prompt the judge to rely only on the rubric and to give a reason item by item;
- Disclose limitations: model-judge scores have systematic biases (e.g., a preference for longer, more neatly formatted answers), so conclusions should note the judge model and rubric version.
Here is an illustrative code sample:
def validate_schema(obj: Any, schema: dict[str, Any]) -> bool:"""Lightweight check of required fields and field types for an object schema (illustrative)"""if schema.get("type") != "object" or not isinstance(obj, dict):return Falsetype_map = {"string": str, "number": (int, float), "boolean": bool, "object": dict}for key in schema.get("required", []):if key not in obj:return Falsefor key, spec in schema.get("properties", {}).items():expected = type_map.get(spec.get("type"))if key in obj and expected is not None and not isinstance(obj[key], expected):return Falsereturn Truedef grade_with_asserts(output: Any, task: Task) -> float:"""L0 deterministic scoring: returns only 0.0 or 1.0 to guarantee reproducibility"""if task.expect_schema and not validate_schema(output, task.expect_schema):return 0.0if task.reference is not None and output != task.reference:return 0.0return 1.0def score_task(task: Task,output: Any,judge: Callable[[Task, Any], float] | None = None,) -> float:"""Per-task scoring: L0 first, L1 reference comparison, L2 judge fallback; layer weights sum to 1"""if grade_with_asserts(output, task) == 0.0:return 0.0score = 0.6 # base score when all L0 checks passif task.reference is not None:score += 0.2 * semantic_similarity(output, task.reference)if task.rubric and judge is not None:score += 0.2 * judge(task, output) # judge must be double-blind with swapped orderreturn scoredef weighted_score(task_scores: dict[str, list[float]], weights: dict[str, float]) -> float:"""Aggregate into an overall score weighted by online traffic"""total = sum(w for w in weights.values())return sum(weights[tid] * sum(s) / len(s) for tid, s in task_scores.items()) / total
Once evaluation can reliably output a “task success rate,” you can wire it into the cost model.
6. Step 3: Unify Cost and Performance Definitions
6.1 Token definitions
Different models use different tokenizers, so the same text can produce noticeably different token counts across models. Unit prices must therefore be compared under the same token-counting convention; it is recommended to express everything as “price per million tokens.”
Distinguish three kinds of tokens:
- Input tokens;
- Output tokens, usually at a higher unit price;
- Cache-hit tokens, usually discounted.
If the business involves multimodal inputs such as images or audio, pay attention to their separate billing rules.
6.2 Latency definitions
Define and collect separately:
- Time to first token (TTFT): the key metric for interactive products;
- Average time per output token: determines the overall experience of long-form generation;
- End-to-end duration: observed at the P50, P95, and P99 percentiles.
The mean is easily dragged down by a few very fast requests; tail percentiles better reflect the real experience.
6.3 Throughput definitions
Do not look only at the theoretical peak. Distinguish “request throughput” (requests per second) from “token throughput” (tokens generated per second), and measure them empirically under different concurrency levels and context lengths to inform capacity planning.
6.4 Illustrative calculation
All numbers in the following example are hypothetical:
- 1,000 input tokens at CNY 0.002 per 1K tokens;
- 200 output tokens at CNY 0.01 per 1K tokens.
Then the cost per call is 1000 × 0.002 ÷ 1000 + 200 × 0.01 ÷ 1000 = CNY 0.004. If the task success rate is 97%, the cost per successful task is about 0.004 ÷ 0.97 ≈ CNY 0.0041.
7. Step 4: Cost-Reduction Strategies and Quantified Benefits
7.1 Strategy matrix
| Strategy | Mechanism | Implementation notes | Main risk |
|---|---|---|---|
| Tiered routing | Route simple requests to small models, complex ones to large models | Use a weak classifier or rules for difficulty detection first, and use small-model confidence and extraction-schema validation results as fallback signals | Routing errors directly lose quality; route accuracy must be evaluated separately |
| Semantic caching | Reuse historical results for similar questions | Set the vector-similarity threshold according to business tolerance, and add cache expiry | Similarity misjudgment may return the wrong answer |
| Context trimming | Cut retrieved snippets, compress dialogue history, remove redundant examples | Aim for the “minimum context needed to answer” | Over-trimming drops critical information |
| Output-length control | Cap maximum output tokens and constrain the output format | Structured output plus schema validation to reduce parse-failure retries | Truncation causes incomplete content |
| Batching and off-peak scheduling | Batch inference to raise hardware utilization, schedule offline tasks to troughs | Pool online and offline tasks separately to avoid interference | Larger batches can raise single-request latency |
| Model compression and distillation | Use smaller or quantized models for high-frequency simple tasks | Precision loss must be verified against the regression set | Capability degradation on hard cases |
| On-demand long context | Trim context for routine requests; enable long-context inference only for complex tasks | Route by input length and task type | Fluctuations in long-context cache hit rate cause cost variance |
7.2 Routing decision example
Here is an illustrative code sample:
@dataclassclass CallResult:output: Anyprompt_tokens: int # measured uniformly by the ingress layercompletion_tokens: intlatency_ms: floatmodel: strdef cost_of(result: CallResult, prices: dict[str, dict[str, float]]) -> float:"""Prices stored as price-per-million-tokens, version-bound at call time to avoid historical distortion after repricing"""p = prices[result.model]return (result.prompt_tokens * p["input"]+ result.completion_tokens * p["output"]) / 1_000_000def route(task: Task,small: Callable[[Task], CallResult],large: Callable[[Task], CallResult],prices: dict[str, dict[str, float]],threshold: float = 0.8,) -> CallResult:"""Illustrative: escalate to the large model if the small model's confidence is low,output fails validation, or it times out"""try:r = small(task)passed = grade_with_asserts(r.output, task) == 1.0if passed and r.confidence >= threshold:return rexcept Exception:pass # timeouts, rate limits, etc. also trigger escalationreturn large(task)
7.3 Benefit calculation example
Suppose 60% of requests in a scenario are simple tasks:
- All on the large model: CNY 0.0100 per call, 99.0% success rate;
- After tiered routing: simple tasks handled by the small model (CNY 0.0010, of which 3% escalate to the large model, with 95% routing accuracy), while complex tasks are still handled by the large model.
The combined cost per successful task after two-tier routing is approximately:
0.6 × (0.0010 + 0.03×0.0100) / (0.95×0.99 + 0.05×0.98)+ 0.4 × 0.0100 / 0.99≈ CNY 0.00511
Compared with the original ~CNY 0.0101, cost drops by about half.
One caveat deserves special attention: 95% routing accuracy means 5% of simple requests are wrongly escalated and 5% of complex requests are wrongly downgraded. Once a downgraded complex request fails and is retried, the cost can actually exceed calling the large model directly. So before a routing strategy goes live, you must build a separate routing evaluation set and continuously monitor the escalation and retry rates in production.
7.4 Recommended rollout order
Proceed from lowest to highest risk:
- Output length and format constraints: almost no quality impact, implement first;
- Context trimming: first validate information retention on the evaluation set;
- Semantic caching: start canarying from low-risk scenarios;
- Tiered routing: pair it with a routing evaluation set and a downgrade/retry mechanism;
- Model compression and distillation: highest precision risk, must go through full regression and canary.
8. Step 5: Continuous Evaluation and Governance
8.1 Unified metering at the ingress layer
All model calls should pass through a unified ingress layer (i.e., a model gateway) that centrally handles authentication, routing, retries, rate limiting, and metering. Each call should collect at least the following fields:
- Request ID, business line, scenario;
- Model name and version, prompt-template version;
- Input token count, output token count, cache-hit token count;
- Time to first token, total duration;
- Termination reason (normal finish, length limit reached, content-safety block, timeout, etc.);
- Retry count and per-call cost.
Cost is computed in real time from token counts and the price table bound at call time, then persisted, to avoid historical-data distortion from later price changes. Whether you use a self-hosted inference cluster or a cloud service, meter everything under this same set of definitions.
8.2 Metrics system and alerting
| Metric category | Key metrics | Purpose |
|---|---|---|
| Cost | Cost per successful task, cost per 1K requests, cache savings | Cost attribution and budget control |
| Quality | Task success rate, field accuracy, human-fallback rate, user adoption rate | Quality monitoring and routing adjustments |
| Performance | TTFT, end-to-end P95/P99, effective throughput | Capacity planning and experience optimization |
| Stability | Error rate (broken down by error type), timeout rate, rate-limit triggers | Fault localization and contingency execution |
Alerting recommendations:
- Set cost thresholds by day-over-day and week-over-week comparison, combined with baseline-detection methods such as year-over-year, period-over-period, moving average, and same-period averages, to catch abnormal spikes;
- Base success-rate and P95-latency alerts on historical same-period baselines, not arbitrary hard-coded numbers;
- Every alert maps to a clear remediation plan, such as scaling out, rollback, or switching to a backup model.
8.3 Offline regression and canary release
Wire regression evaluation into the CI pipeline so it triggers automatically whenever the model or prompt changes. Here is an illustrative code sample:
from dataclasses import dataclass@dataclassclass QualityGate:"""Release quality gate; thresholds should come from historical baselines, shown here for illustration"""min_success_rate: float = 0.95 # must not fall below 98% of current online success ratemax_success_drop: float = 0.02 # max allowed period-over-period drop in success ratemax_p95_latency_ms: float = 3000.0ci_min: float = 0.05 # lower bound of regression-score CI must exceed the baselinedef evaluate_gate(base: dict[str, float],candidate: dict[str, float],candidate_ci: tuple[float, float],) -> tuple[bool, list[str]]:"""base and candidate each hold three metrics: success_rate, p95_latency_ms, cost_per_success"""reasons: list[str] = []if candidate["success_rate"] < base["success_rate"] * 0.98:reasons.append("success rate dropped >2% relative to baseline")if candidate["success_rate"] < QualityGate.min_success_rate:reasons.append("success rate below absolute threshold")if candidate["p95_latency_ms"] > QualityGate.max_p95_latency_ms:reasons.append("P95 latency exceeds threshold")if candidate_ci[0] <= base["score"]:reasons.append("lower bound of regression-score CI did not exceed baseline")return not reasons, reasons
Online releases follow the flow “offline gate → shadow traffic → small-traffic canary → full rollout,” with each stage measured against baseline metrics and with predefined rollback conditions.
8.4 Capacity planning and model portfolio
Capacity planning should use P95, or even P99, peak traffic as the baseline and combine it with business-growth forecasts to set a sensible ratio of reserved to elastic compute. If long-horizon compute reservations and commitments do not match business growth, insufficient utilization directly drives up unit cost. In practice, maintain a portfolio of “primary large model + lightweight small model + backup model”:
- The primary large model guarantees the quality ceiling;
- The lightweight small model handles high-frequency simple tasks to lower average cost;
- The backup model ensures business continuity when the primary model suffers a regional outage or rate limiting.
All three should be regression-tested periodically, and the portfolio adjusted dynamically as the business distribution changes.
9. Common Pitfalls and an Avoidance Checklist
- Leaderboard as evaluation: a high leaderboard score does not mean it is good on your business. You must re-test on your own evaluation set.
- Random sampling: different settings like temperature make results irreproducible and prevent model-to-model comparison.
- Average latency only: the mean hides the long tail; P95/P99 are what decide the experience.
- Confusing call volume with success volume: failed retries multiply token consumption, so cost should be amortized per successful task.
- Optimization without a control: optimizations like caching, routing, and compression should be validated with treatment and control groups running at the same time, so business fluctuations are not mistaken for optimization gains.
- Ignoring evaluation-set aging: after a prompt or model upgrade, old test items may already have been “overfitted to,” so keep adding new ones.
- Ignoring data compliance: when the evaluation set contains user data, it must be desensitized and meet compliance requirements; when evaluating through external services, also consider cross-border data transfer.
10. Phased Rollout Roadmap
| Phase | Reference duration | Core tasks | Deliverables |
|---|---|---|---|
| Phase 1: Solidify the foundation | 2–4 weeks | Unified ingress and instrumentation, define metric conventions, build a basic dashboard | Per-request cost and performance data, metric-definition doc |
| Phase 2: Build the evaluation loop | 4–6 weeks | Build the layered evaluation set, implement L0–L2 scoring, calibrate the judge by hand | Baseline evaluation report, one-click regression pipeline |
| Phase 3: Optimize and govern | Ongoing | Canary rollout of routing, caching, context trimming, etc.; capacity planning and alerting | Before/after comparison data, capacity plan, change-management standard |
Set quantifiable acceptance criteria for each phase:
- Phase 1: cost data covers all model calls;
- Phase 2: core-scenario evaluation coverage is at least 90%, and judge–human agreement reaches the agreed threshold;
- Phase 3: cost per unit of effective output keeps falling, while online success rate and P95 latency are no worse than baseline.
11. Conclusion
Valuations returning from narrative to financials are essentially a swap of “invisible capability” for “verifiable results.” For engineering teams, the same shift means:
- Model selection no longer depends on leaderboards and marketing language;
- Instead, it is built on reproducible evaluation, unified cost and latency definitions, and a continuously running regression and canary system.
From building a layered evaluation set, to using “cost per successful task” to unify the measurement of quality and spend, to tiered routing, caching, context trimming, and capacity planning, every step can land on concrete data and code.
Models will keep iterating and prices will keep adjusting, but this engineering method of “data-driven model selection and cost governance” will remain effective over the long term—and will ultimately compound into a core competitive advantage for the team.
