How to Evaluate LLM Capabilities Scientifically: Benchmarking, Quota Governance, and Cost Optimization

    Large language models (LLMs) often feel like they "suddenly get worse" after running smoothly at first, and developers tend to blame the model itself. In reality, many factors beyond the model affect final output, so this perceived degradation is usually configuration drift rather than a loss of capability. This article shows how to replace gut-feel judgments with data-driven decisions: building a multi-dimensional, reproducible evaluation pipeline, treating quotas and concurrency as a resource to be governed, and using multi-model cascading routes driven by verifiable signals. It also covers context and prompt engineering for stability, observability for cost control, and a practical checklist for putting all of this into production. The goal is to keep quality, cost, and stability under control at the same time.

    1. Background: Why Do LLMs “Suddenly Get Worse”?

    In real-world engineering, many developers describe the same experience: a model feels great at first, then gradually seems to “get dumber,” so they conclude that “model quality has dropped.” Treating this subjective impression as a conclusion and using it to drive model-selection decisions is dangerous. On a real serving path, far more than the model itself influences the final output:

    1. Server-side rolling upgrades and canary switches: The same model identifier may route to different weight versions or quantization configurations at different points in time and under different traffic ratios.
    2. Different reasoning effort (thinking budget): Whether deep reasoning is enabled, and how the reasoning-chain length limit is set, can produce large differences on the same question.
    3. Requests being downgraded or forwarded: During peak traffic, the system may route requests to smaller, cheaper model branches to protect availability.
    4. Changes in context and tool-call configuration: Different system prompts, tool definitions, and retrieval results can cause output quality to fluctuate significantly.
    5. Tighter quota and rate-limiting policies: Changes to quotas, concurrency, and priority tiers directly affect response quality and how much work can be completed.

    In other words, “getting dumber” is very likely configuration drift rather than capability regression. The sound engineering response is to upgrade from “judging by feel” to “judging by data”: first establish reproducible evaluation and monitoring, then talk about selection and optimization.

    2. Capability Evaluation: Building a Multi-Dimensional Quantitative System

    A single-dimension score cannot support production selection. It is recommended to cover at least the following six categories of metrics:

    Dimension Key Metrics Recommended Evaluation Method
    Reasoning & Math Multi-step derivation accuracy, process consistency Automated grading of objective questions
    Coding Unit-test pass rate, defect detection rate, patch compilability Sandbox execution of test cases
    Long Context Long-document retrieval hit rate, cross-file dependency understanding Retrieval-QA benchmarks
    Instruction Following Structured-output compliance rate, format violation rate Schema validators
    Stability Consistency across repeated samples, variance Repeated sampling on fixed inputs
    Availability & Cost First-token latency, throughput, rate-limit trigger rate, per-task token cost Load testing + billing attribution

    “Stability” is the most easily overlooked dimension. If the same question, sampled multiple times, swings widely between “correct” and “incorrect,” the model is unreliable under the current prompt and temperature settings. In that case, rather than trying to raise the accuracy of a single call, it is better to add a safety net through self-consistency voting or validation-and-retry.

    3. A Reproducible Evaluation Pipeline

    Evaluation must be reproducible, or no conclusion can be retained over time. The core elements are: a fixed set of cases, fixed scoring rules, fixed decoding parameters, and complete call records.

    1. # Normalized description of evaluation cases (illustrative)
    2. CASES = [
    3. {
    4. "id": "reasoning-001",
    5. "task": "multi-step reasoning",
    6. "prompt": "...",
    7. "checker": "exact_match", # exact_match / unit_test / json_schema / judge
    8. "expected": "...",
    9. "params": {"temperature": 0.0, "max_tokens": 4096},
    10. },
    11. ]

    When running the evaluation, it is recommended to record the following for each case: model identifier and version, prompt version number, decoding parameters, output, token consumption, first-token latency, whether a downgrade or retry was triggered, and the final score. This way, if quality fluctuates, you can attribute it layer by layer along “prompt version → model version → configuration item,” instead of guessing and swapping models.

    For scoring, prioritize objectively judgeable signals (exact match, unit tests, schema validation). When model-based scoring is genuinely needed, you must use blind pairwise comparison plus human spot-checks, and calibrate for consistency among different raters.

    4. Quota and Concurrency Governance: Design “Quota” as a Resource

    In production, what usually slows down the experience is not the speed of a single inference call, but how quotas and concurrency are allocated. A common trap is putting interactive tasks, batch tasks, and sub-agent tasks all into the same quota, so that overnight batch jobs consume the daytime interactive quota, causing an interactive-experience cliff the next day.

    It is recommended to govern at three levels:

    1. Quota isolation: Split into independent call credentials / billing units by business domain or task type, so that interactive paths and offline batch processing do not encroach on each other.
    2. Budget admission control: Estimate token consumption before calling; requests that exceed budget should queue rather than fire blindly, avoiding traffic amplification from being rejected outright and then retried.
    1. # Pseudocode: token-budget admission based on a rolling window + priority queue
    2. class TokenBudget:
    3. def __init__(self, window_seconds: int, token_limit: int):
    4. self.window_seconds = window_seconds
    5. self.token_limit = token_limit
    6. self.events = [] # (timestamp, tokens)
    7. def used(self, now: float) -> int:
    8. cutoff = now - self.window_seconds
    9. self.events = [(t, n) for t, n in self.events if t >= cutoff]
    10. return sum(n for _, n in self.events)
    11. def try_acquire(self, need: int, now: float) -> bool:
    12. if self.used(now) + need <= self.token_limit:
    13. self.events.append((now, need))
    14. return True
    15. return False
    16. def wait_seconds(self, need: int, now: float) -> float:
    17. cutoff = now - self.window_seconds
    18. for t, n in sorted(self.events):
    19. if t >= cutoff:
    20. need -= n
    21. if need <= 0:
    22. return max(0.0, t + self.window_seconds - now)
    23. return self.window_seconds
    24. def submit(task, priority: int = 1):
    25. need = estimate_tokens(task) * (1 + task.retry_budget)
    26. if budget.try_acquire(need, now()):
    27. return call_model(task)
    28. scheduler.enqueue(task, priority=priority, eta=budget.wait_seconds(need, now()))
    1. Restrained failure handling: For 429 / rate-limit errors, use exponential backoff with jitter and set a maximum retry count; for authentication or permission issues such as 401/403, do not retry, and alert directly. A retry storm is the most common reason a quota gets exhausted quickly.

    5. Multi-Model Cascading Routes: Let Expensive Models Do Only What They Should

    A practical engineering pattern is “a cheap model as the baseline, a strong model as the final reviewer”:

    • Routing layer classifies tasks first: format extraction, log summarization, unit-test generation → lightweight model; defect localization, code review, solution design → strong reasoning model.
    • Escalation mechanism: When structured validation fails, confidence is low, or unit tests fail, automatically escalate the task to a stronger model instead of having a weak model repeatedly self-correct.
    • Sub-agent offloading: Delegate verifiable, fine-grained tasks like “find bugs, read code, patch small features” to the cheap model, while the main model handles planning, decomposition, and final merging.
    1. ROUTES = [
    2. {"match": "extract", "model": "light", "params": {"temperature": 0.0}},
    3. {"match": "code_review", "model": "mid", "validator": "unit_test"},
    4. {"match": "hard_reasoning", "model": "strong", "allow_escalate": True},
    5. ]
    6. def dispatch(task):
    7. for r in ROUTES:
    8. if r["match"] != task.kind:
    9. continue
    10. resp = call(r["model"], task, **r.get("params", {}))
    11. validator = r.get("validator")
    12. if validator and not validate(validator, resp, task):
    13. if r.get("allow_escalate"):
    14. return call("strong", task) # escalate only on validation failure to control cost
    15. raise ValidationError(task.id)
    16. return resp
    17. raise LookupError(task.kind)

    The key principle is: escalation must be driven by verifiable signals, not blind retries based on a vague “this looks wrong,” otherwise cost and latency will spiral out of control.

    6. Context and Prompt Engineering: The Other Half of Stability

    Many “unstable model output” problems originate on the input side:

    • Prioritize structured constraints: Mechanisms like JSON Schema and function calling constrain output format more reliably than repeatedly pleading in the prompt.
    • Inject context on demand: Control context length through relevance retrieval and summary compression, avoiding long-text stacking that dilutes key information.
    • Version your prompts: Manage system prompts in a code repository, route each change through a review process, and record the version number, so that “prompt drift” is not mistaken for a decline in model capability.
    • Fix decoding parameters: Align parameters such as temperature and maximum output length between evaluation and production as much as possible; when sampling variance is too large, introduce multiple sampling plus self-consistency voting.

    7. Cost and Observability: Close the Loop

    It is recommended to build a minimal viable observability loop:

    • Call-chain tracing: For each call, record the request ID, model identifier, prompt version, input/output tokens, latency, retry count, and whether a downgrade occurred.
    • Quota monitoring and alerting: Aggregate token consumption by project / API / model, and set tiered alerts and automatic degradation policies.
    • Cache optimization: Combine exact caching with semantic caching, and treat cache hit rate as one of the core cost metrics.
    • Off-peak scheduling and batching: Schedule offline tasks to use batch capabilities during idle periods, avoiding competition with the interactive path for quota.

    When implementing on the cloud, you can build the above chain on top of general-purpose capabilities such as managed model services, logging services, monitoring and alerting, and object storage. Cloud platforms such as Baidu AI Cloud typically provide multi-tier models, quota management, and call observability as foundational capabilities. It is recommended to access them uniformly through a self-built gateway, so that when evaluation conclusions change or canary switching is needed, you can replace models and adjust traffic ratios at relatively low cost.

    8. Production Checklist

    1. Is there a fixed regression evaluation set that automatically runs after any model or configuration change?
    2. Do you distinguish between two problem types: “model capability decline” and “configuration/context drift”?
    3. Are interactive paths and offline batch processing quota-isolated?
    4. Do rate-limit errors use backoff retries rather than blind retries?
    5. Is there a validator-driven escalation mechanism from small models to stronger ones?
    6. Are prompts under version control and traceable?
    7. Is token cost aggregated by dimension and alerted on?
    8. Are models accessed through a unified gateway to keep the engineering cost of switching models manageable?

    9. Conclusion

    Judging whether an LLM is “good to use” should not stop at subjective impressions. Making evaluation a reproducible pipeline, treating quotas as a resource that needs governance, and making multi-model collaboration a routing mechanism driven by validation signals are what allow quality, cost, and stability to be controlled at the same time. Model capabilities will keep evolving, but the evaluation system, quota governance, and routing architecture are long-term capabilities that can be accumulated as team assets.