How to Evaluate LLM Services: Capability Testing and Quota Rules Explained

    This guide explains how to evaluate large language model services along two independent dimensions: model capability and service terms. It covers reproducible capability testing with layered question sets and multiple sampling passes, how to interpret multi-period quota rules including rate multipliers, reset schedules and utilization, and how to verify pricing transparency. A practical checklist is included for procurement decisions.

    1. Why Evaluate Capability and Service Terms Separately

    When choosing an LLM service, buyers typically care about two things: whether the model can complete the task, and whether the pricing, speed, and quota rules are reasonable. Many people mix the two judgments together, and impressions end up outweighing evidence. A service with strong capability but confusing quotas will frustrate users in daily operation; a service with clear quotas but weak reasoning will fail to retain professional users.

    Therefore, the evaluation framework should be split into two dimensions. The capability dimension focuses on the correctness and stability of model output. The service dimension focuses on quotas, rate multipliers, reset rules, pricing, and over-limit handling. Collect evidence for each dimension separately, then combine them into a final judgment.

    This article is organized as follows: Section 2 describes a reproducible testing method for mathematical reasoning capability; Section 3 explains multi-period quota multipliers, reset time zones, and utilization; Section 4 discusses pricing and rule transparency; Section 5 provides a ready-to-use checklist.

    2. Capability Evaluation: How to Run Reproducible Tests

    Key takeaway: fix the test conditions first, then use a layered question set, sample each question multiple times, score with two metrics, and only then compare services.

    2.1 Common Pitfalls

    • Treating one or two successful cases as representative. Successes may come from lucky sampling or from questions closely resembling training data.
    • Taking a complete-looking answer as proof of correctness. Long derivations with skipped steps or incorrect citations hide errors better the longer they are.
    • Comparing a single run. The same model can perform noticeably differently across sampling runs; a single result says nothing about stability.

    2.2 Conditions to Fix Before Testing

    Test results are only comparable when conditions are fixed. Record the following for every test run:

    • Model name and version string, plus the test date.
    • Sampling parameters, including temperature, top_p, and max_tokens.
    • The system prompt, and whether tool calls are allowed.
    • The invocation path — direct API call or through a gateway. Gateways may rewrite parameters and must be documented separately.

    If the provider does not publish a fixed version number, note “version unknown” in the report and limit conclusions to the test date.

    2.3 Building the Question Set

    The question set is the core of reproducible testing. Follow these principles:

    1. Each question must have a standard answer that a program can verify automatically, such as an integer, fraction, or a mathematical expression.
    2. Layer the questions across categories such as algebra, number theory, combinatorial counting, and probability. Keep category sizes roughly equal so no single category dominates the total score.
    3. Hold out a subset used only for final evaluation — never for tuning or prompt optimization.
    4. Rotate some questions periodically and check whether questions have appeared in public materials, to reduce data contamination.
    5. Aim for at least 100 questions total, with a minimum per category, e.g., 15 per category. These are engineering rules of thumb, not statistically proven thresholds. The smaller the set, the noisier the results — keep conclusions conservative.

    2.4 Sampling Count and Scoring Metrics

    Sample each question independently n times (e.g., n=5) and report two metrics:

    • Estimated pass@1: the number of correct answers across all questions and all samples, divided by (number of questions × n). It estimates the average single-attempt pass rate and reflects average performance. When reporting, state n and the sampling parameters to avoid confusion with the pass@k convention in other benchmarks.
    • Consistent pass rate: the fraction of questions answered correctly in all n attempts. For production environments, this metric is more informative.

    Multiple samples of the same question are not independent — question difficulty causes results to cluster. Therefore, confidence intervals should be computed with bootstrap resampling at the question level. Applying the single-trial binomial formula directly will produce intervals that are too narrow.

    Score only whether the final answer matches the standard answer. The derivation does not count toward the score, but you should manually review a sample and tally errors by type: calculation errors, skipped steps, unit errors, and failure to follow the required output format. These data do not affect the score, but they reveal where the model is unstable.

    If a result deviates from previous runs beyond normal variance, first check the version, parameters, and gateway configuration before concluding that the model itself has changed.

    2.5 Result Recording Template

    Field Example Notes
    Test date YYYY-MM-DD (actual date) For tracing version changes
    Model identifier Full name returned by the provider Do not use abbreviations
    Sampling parameters temperature=0.7, top_p=1.0, max_tokens=4096 Fill in every field
    Question set version v2.3, 120 questions total Record the set ID
    Category distribution Algebra 30 / Number theory 30 / Combinatorics 30 / Probability 30 Verify balance
    Samples per question (n) 5 Must match the scoring metric
    Estimated pass@1 78.3%, CI 74.1%–82.0% One decimal place
    Consistent pass rate 61.7% Same as above
    Review error tally 1 in 20 questions reviewed; 3 skipped steps, 1 unit error State review ratio and error types

    3. Multi-Period Quotas: Multipliers, Resets, and Utilization

    Many services set quotas across multiple periods, for example a 5-hour rolling window plus a weekly cap. Such rules can conflict with each other. Verify in this order: read the official rules first, then compute multipliers, then derive reset times from the time zone, and finally compute utilization. Skipping any step can lead to wrong conclusions.

    3.1 Computing Multipliers

    Multipliers define how much quota different models or features consume per unit of work. Suppose a service meters usage in “base units” with these rules:

    • Model A: 1 unit per 1,000 input tokens, 4 units per 1,000 output tokens.
    • Model B: 2 units per 1,000 input tokens, 12 units per 1,000 output tokens.

    Model B’s input multiplier is 2× Model A’s, and its output multiplier is 3×. Since the effective ratio depends on the input/output mix, use a blended multiplier:

    Blended multiplier = input share × input multiplier + output share × output multiplier

    Assume a task with 70% input and 30% output. Model A’s blended multiplier is 0.7 × 1 + 0.3 × 4 = 1.9; Model B’s is 0.7 × 2 + 0.3 × 12 = 5.0. The blended ratio for B is about 2.6, between the pure-input ratio of 2 and the pure-output ratio of 3. These figures only illustrate the calculation; always defer to the provider’s official rules.

    When verifying, confirm: whether multipliers are given separately for input, output, cache hits, and thinking tokens; and whether there is a minimum charge or a fixed per-request fee.

    3.2 Reset Windows and Time Zones

    Reset rules determine when quota is restored. Check three things:

    1. Window start: Does the window start from the first request, or is it aligned to natural time (e.g., midnight)? In the first case, window boundaries differ per user; in the second, they are identical for all users.
    2. Time zone: Which time zone governs the reset? If resets follow UTC, a user in UTC+8 will hit the reset around 8 a.m. local time.
    3. Window interaction: When multiple windows coexist, a request must pass the checks of all windows simultaneously. If any window is exhausted, the request is rejected even when other windows still have headroom.

    The third point must be verified empirically:

    • Record the remaining quota of both windows, then send a small request. Confirm that both windows decrease by the same amount.
    • Then send a request exceeding the short window’s remaining quota. Confirm the request is rejected and the long window’s remaining quota is unchanged.
    • If results differ, record it as a blocking or metering anomaly and flag it separately in the report.

    3.3 Utilization Formula

    Utilization measures how fast quota is being consumed:

    Utilization = quota consumed in the window ÷ total quota for the window × 100%

    Example: a weekly window has a total of 500 units; 380 units are consumed by Friday. With Monday as the window start, 5 of 7 days have elapsed — a time progress of about 71%. Utilization is 76%, ahead of time progress. At the current pace of roughly 76 units per day, the remaining two days would need about 152 units — more than the 120 units left — so the quota will most likely run out before the weekend.

    Therefore, when judging whether limits will be hit, compare utilization against time progress simultaneously. If utilization consistently runs ahead of time progress, consumption is outpacing the window’s rhythm.

    Over-limit handling: providers typically return a specific error code or fall back to a smaller model. During testing, record the error code, retry hints, whether billing occurs, and when quota is restored. If the provider does not publish this information, note it in the report.

    4. Pricing and Rule Transparency

    Pricing is transparent when a buyer can trace every charge using only the published rules and a single bill. Opaque pricing drives budgets out of control even when the unit price looks low.

    4.1 Verification Steps

    1. Confirm the billing unit: Check whether prices are per 1K or per 1M tokens. Unit conversion mistakes are the most common source of discrepancies.
    2. Itemize charges: Input, output, cache hits, cache writes, and thinking tokens are usually priced differently — list each separately.
    3. Confirm minimum and fixed charges: Some services impose a per-request floor, making short requests effectively more expensive than the listed rate.
    4. Confirm taxes, exchange rates, and invoicing: The gap between tax-inclusive and tax-exclusive prices can be significant. For foreign-currency settlement, record the exchange rate date.
    5. Confirm the scope of free credits and discounts: Do free credits count toward quota windows? Do discounts apply only to certain models? Spell it out.
    6. Sample reconciliation: Randomly pick 10 requests, read the usage fields from the responses, hand-calculate the expected charge using the official formula, and compare against the bill. If the deviation exceeds your budget tolerance (e.g., 1%), record the request IDs and investigate.

    5. Checklist

    Check each item before use. For anything unconfirmed, cite the source or mark it as “unverified.”

    • Full model version string and test date recorded.
    • temperature, top_p, max_tokens, and the system prompt recorded.
    • Invocation path confirmed; gateway parameter rewriting checked.
    • Question set has standard answers verifiable by program.
    • Question set layered by category, with roughly equal counts meeting the minimum per category.
    • Held-out set not used for tuning.
    • Each question sampled n times; both estimated pass@1 and consistent pass rate reported.
    • Confidence intervals computed with question-level bootstrap.
    • Derivations reviewed and errors tallied by type.
    • All quota windows listed, with start and duration confirmed.
    • Reset time zone confirmed and converted to the user’s local time.
    • Multi-window blocking and metering logic verified empirically.
    • Utilization computed for each window and compared against time progress.
    • Error codes, fallback behavior, and recovery time recorded for over-limit events.
    • Billing unit, per-item prices, minimum charges, and taxes confirmed.
    • At least 10 sampled reconciliations completed with deviations recorded.
    • Data cutoff date noted, and all “unverified” items listed.