Evaluating New LLMs: A 3D Framework for Capability, Cost, and Availability

1. Why “Just Looking at Benchmarks” Misleads

Whenever a new model is released, the immediate reaction is often to quote benchmark scores: ranking on a comprehensive leaderboard or percentage improvements on a specific task. This narrative is simple and direct but offers limited value for engineering decisions, for three main reasons.

First, benchmarks carry data contamination risks. Most mainstream evaluation datasets are publicly downloadable. If they are mixed into post-training data, the scores reflect memorization rather than generalization. A common industry practice is to retain private, undisclosed test sets or periodically change the questions. However, as a user, you cannot verify the boundaries of a vendor’s training data.

Second, score differences may lack statistical significance. A one or two-percentage-point difference between two models on the same benchmark might simply be due to variations in sampling parameters, evaluation prompt templates, or answer parsing methods. Running the same model again with a different template could cause similar fluctuations. Scores that do not report variance and confidence intervals across multiple runs should be heavily discounted.

Third, benchmark capabilities often misalign with business needs. Comprehensive leaderboards measure “average capabilities,” while business requests typically follow a long-tail distribution. Your system might concentrate 80% of its traffic on document Q&A. A model with a higher overall ranking but mediocre performance in your core scenario is not necessarily better than a lower-ranked model that happens to be highly stable in that specific scenario.

Therefore, a more rational approach is to treat benchmark scores as an “initial screening signal”—they tell you which models are worth advancing to the next round of empirical testing, rather than serving as the final conclusion.

2. The First Pillar: Capability — Translating Benchmark Scores into Business Language

Before formal testing, clarify which capability dimensions your business relies on, then design tests targeting those areas.

1. Long Context Capability. “Supporting a million-token context” and “accurately retrieving information from the middle of a million-token context” are two different things. Key metrics include recall rate in “needle in a haystack” retrieval, cross-document information integration, and the quality degradation curve as context grows. The testing method is straightforward: construct multiple long documents, embed key facts at various depths, and observe if the model can reliably recall them.

2. Multimodal Understanding. Native vision capabilities are highly valuable in scenarios like receipt recognition, chart understanding, and UI auditing. During evaluation, distinguish between “can see images” and “sees accurately.” The former merely means API support; the latter requires validation using labeled image test sets that cover the real image distribution of your business (e.g., the proportion of screenshots, scanned documents, handwritten notes, and charts).

3. Code and Agent Capabilities. If the model is used for coding assistants or automated Agents, the multi-step task completion rate in a real environment is more meaningful than single-turn code generation. You can build a sandbox and let the model complete a full pipeline: “pull code — locate issue — modify — run tests,” tracking the end-to-end success rate.

4. Instruction Following and Format Stability. For engineering systems, stable output formatting (like strictly valid JSON) is often more critical than a model being “smarter.” The usability gap in a pipeline between a model that outputs valid structured results 99% of the time and one at 95% is massive.

A minimalist custom evaluation framework looks like this:

import json
EVAL_SET = json.load(open("golden_set.jsonl"))  # Labeled data sampled from business logs
def run_eval(model_client, cases, n_runs=3):
    scores = []
    for _ in range(n_runs):            # Run multiple times to observe variance
        correct = 0
        for case in cases:
            resp = model_client.chat(
                messages=[{"role": "user", "content": case["prompt"]}],
                temperature=0.2,
            )
            correct += judge(case["expect"], resp)  # Rule-based judging + manual spot-check
        scores.append(correct / len(cases))
    return {"mean": sum(scores) / len(scores),
            "min": min(scores)}        # Focus on worst-case performance, not best

Three key takeaways: The evaluation set must come from real business distributions; run the same use case multiple times to get the mean and observe fluctuations; and use a combination of “rules + manual spot-checks” for judging to avoid the systemic bias of using the same model as both the player and the referee.

3. The Second Pillar: Cost — Understanding the Structure Behind Token Pricing

Once capabilities meet the standard, cost becomes the deciding factor for scalability. When reading a price sheet, four structural issues are easily overlooked.

1. Input vs. Output Price Disparity. The industry standard is that output token pricing is multiples of input token pricing, because generation is a token-by-token autoregressive decoding process, which is far more compute-intensive than the parallel processing of inputs. When estimating costs, don’t just look at the input price—for generative tasks like writing reports or code, output tokens often make up the bulk of the consumption.

2. Cache Hit vs. Cache Miss Price Disparity. Mainstream model services generally offer prompt caching: when the request prefix (system prompts, long documents, few-shot examples) matches a historical request, these tokens are billed at a substantial discount. This offers an order-of-magnitude optimization for three scenarios: Agents with fixed long system prompts, multi-turn conversations, and batch Q&A based on the same set of documents. Engineering-wise, you should proactively place stable content at the beginning of the prompt and variable content at the end to consistently hit the cache.

3. Hidden Costs. The longer the context, the more input tokens per request; multi-turn Agent calls repeatedly carry historical messages, meaning actual consumption can be multiples of a surface-level estimate. Costing should be calculated per “complete session” rather than per “single call.”

4. Total Cost of Ownership (TCO) Perspective. Beyond token fees, you must also factor in evaluation costs, migration and refactoring costs, and the fixed overhead of maintaining a backup model to hedge against availability risks.

A ready-to-use cost estimation function:

def estimate_cost(in_tokens, out_tokens, price, cache_hit_ratio=0.0):
    """price unit is CNY/million tokens, below is a fictional example price"""
    hit  = in_tokens * cache_hit_ratio
    miss = in_tokens - hit
    return (hit  * price["cache_hit"]  / 1e6
          + miss * price["cache_miss"] / 1e6
          + out_tokens * price["output"] / 1e6)
# Example: An Agent carries 50k historical tokens per turn (90% cache hit), generates 2k tokens
# Total cost ≈ 0.045 + 0.05 + 0.1 ≈ 0.2 CNY/turn

At the same capability tier, the list prices of different services can vary by multiples. Furthermore, how well you utilize caching strategies can widen the gap by additional multiples on top of the list price. The potential for cost optimization often exceeds the capability gap between models—this is exactly why you should “look at the price sheet first.”

4. The Third Pillar: Availability — The Most Overlooked Metric at Launch

A common phenomenon: a model launches with impressive scores and aggressive pricing, only for the provider or community to acknowledge service instability days later. This isn’t specific to one provider; it’s an industry norm. New models experience peak traffic immediately upon launch, and even the most thorough capacity planning can be overwhelmed. For engineering teams, availability must be evaluated independently:

1. Focus on Core Metrics, Not Slogans. Error rates (especially rate limits and server-side errors), P99 Time to First Token (TTFT), rate limits (requests and tokens per minute), and formal Service Level Objective (SLO) commitments. This information is typically found on the platform’s status page and documentation. Checking it a week after launch provides a much more realistic picture than checking on launch day.

2. Clients Must Have Built-in Fault Tolerance. Regardless of the service chosen, exponential backoff retries, timeout control, and graceful degradation are standard practices:

import time, random
def call_with_retry(fn, max_retries=5, base_delay=0.5):
    for attempt in range(max_retries):
        try:
            return fn()
        except RateLimitError:
            delay = base_delay * (2 ** attempt) + random.uniform(0, 0.3)
            time.sleep(delay)            # Exponential backoff + jitter to avoid retry storms
        except TimeoutError:
            if attempt == max_retries - 1:
                return fallback_model()  # Degrade to a backup model
    raise RuntimeError("retries exhausted")

3. Multi-Model Routing is the Last Line of Defense for Stability. Abstract model calls into a unified gateway layer, configured with a “primary model + backup model” setup, automatically switching upon health check failures. If a team doesn’t want to build this infrastructure themselves, they can directly use managed capabilities on major cloud platforms—such as Baidu AI Cloud and other mainstream clouds that offer common model access, monitoring, alerting, and logging services. This essentially pushes “availability engineering” down to the platform layer, letting teams focus on business logic.

4. Write Availability into the Final Selection Conclusion. It is recommended to maintain a tracking table for candidate models: record the daily failure rate and P99 latency for two consecutive weeks, then combine this with pricing and capability scores to make the final decision. A model that leads in capability by 5% but has a failure rate an order of magnitude higher may not be the optimal solution in a production environment.

5. A Practical, Step-by-Step Evaluation Process

Connecting the three pillars yields a reusable workflow:

1. Initial Screening (1 day): Review public leaderboards and list prices. Eliminate candidates with clearly inadequate capabilities or costs multiples over budget, retaining 2–3 for empirical testing.
2. Capability Testing (3–5 days): Sample 100–300 labeled use cases from business logs, covering core scenarios like long context, multimodal, code/Agent, and format adherence. Run multiple times to get the mean and worst-case values.
3. Cost Estimation (1 day): Simulate a week of traffic based on the real session structure (including cache hit ratios) to calculate the monthly cost range, rather than just looking at unit prices.
4. Availability Trial (2 weeks): Route about 10% of traffic in a canary release, recording failure rates, P99 latency, and rate limits, while validating retry and degradation logic.
5. Final Decision (1 day): Aggregate the 3D data into a comparison table, weight it according to business priorities, and form a reproducible, rollback-ready model selection record.

6. Conclusion

The frequency of new model releases will only increase. Chasing the latest model is not wrong; what is wrong is substituting a benchmark screenshot for an engineering decision. Benchmark scores get you to the candidate list, the price sheet answers whether scaling is viable, and availability data determines if it can stay in production. Turning these three pillars into your team’s standard operating procedure ensures that every model update is no longer an emotional spectacle, but a reproducible, rollback-ready engineering upgrade.