How to Design LLM Subscription Quotas and Evaluate Model Capabilities

    This article explores two critical aspects of Large Language Model (LLM) subscription services: quota design and model capability evaluation. It provides a structured approach to calculating the ratio between sliding window and total period quotas, emphasizing transparent reset rules and user-facing consumption visibility. Furthermore, it introduces an evaluation framework using unpublished mathematical propositions to avoid data contamination, outlining a rigorous scoring methodology and a tiered capability assessment method, offering actionable insights for developers and LLM service providers.

    1. Quota Design: Calculate First, Optimize Experience Later

    The quota rules of a subscription service are often treated as a mere technical detail, but they directly determine how much actual work users can accomplish. Model capability is constrained by algorithms and computing power, making it a technical issue. How subscription tiers are divided and when quotas reset, however, are product design issues. Prices can be high, but the rules must be clear; users should not have to repeatedly guess their limits while using the service.

    1. How to Determine the Ratio Between Window and Total Period Quotas?

    Most subscription services adopt a two-layer rate limiting mechanism. One layer is a sliding window quota, such as usage limits per 5 hours. The other is a total period quota, such as a weekly or monthly cap. The ratio between these two layers determines where the bottleneck occurs.

    Let X be the 5-hour window quota, and Y be the total period quota within a billing cycle. A week consists of 168 hours. Calculating based on a 5-hour window, there are approximately 33.6 windows, rounded up to 34. If a user maxes out their quota in every window throughout the week, the total consumption would be 34X. Therefore, the total period quota should at least be:

    Y >= 34 * X

    If considering only a 5-day workweek (120 hours), the number of windows is 24, making the formula:

    Y >= 24 * X

    The assumption here is that the user maxes out every window without idle time, and the quota refreshes in fixed amounts. In actual usage, users rarely operate at continuous maximum capacity, so this formula provides a conservative upper bound. If the weekly total quota is lower than these values, users might be blocked by the total cap mid-week, diminishing the significance of the window quota.

    Conversely, if Y is significantly greater than 34X, the window quota becomes the actual limit. The size of the task a user can run each time is determined by the window, while the weekly total acts merely as a safety net. Whether this ratio is reasonable depends on the usage rhythm the service provider expects: continuous short-term usage or concentrated bursts.

    To verify if the ratio is reasonable, you can track the window usage distribution of real users over a week. If most high-frequency users deplete their weekly total quota before Wednesday, the ratio is likely too low. Using the window count formula for validation is more reliable than relying on intuition.

    2. Are Reset Rules Consistent for Continuous Subscribers?

    Another common issue arises with monthly or annual subscribers. When a monthly total expires, it resets, but the weekly limit might not. This carries over the end-of-month weekly limit state into the new cycle.

    This design presents two problems. First, cross-period coupling lacks a clear business rationale. Usage in a new cycle should ideally be independent of the previous one; inheriting old states makes prediction difficult for users. Second, inconsistent rules are exposed in multi-account or cross-period scenarios, leading users to question the fairness of the rules. This indicates either a high explanatory cost for the rules or inconsistent implementation across different quota layers. Whether this is an intentional risk control measure should be determined by the risk strategy and billing system implementation, not just inferred from user perception.

    For subscriptions billed by independent cycles, a recommended principle is: the issuance, reset, and clearance boundaries of quotas should align with the billing cycle boundaries, with each layer calculated independently. If annual subscribers have their total quota rolled out on a monthly, weekly, or 5-hour basis, the value of the discount is eroded during the rolling process, making the subscription feel less cost-effective.

    There are exceptions to this principle. Scenarios such as bonus quotas, risk-control freezes, and refund recalculations may require separate handling. Service providers should clearly state these exceptions on the rules page and announce them in advance when rules change, ensuring users know what might affect their used quotas.

    3. How Can Users View Their Quota Status?

    No matter how reasonable the quota rules are, if users cannot see their consumption, trust is lost. Service providers should display the following fields in the console:

    • Quota used and remaining for the current window;
    • Quota used and remaining for the current billing week;
    • Time until the next window refresh;
    • Time of billing week end and reset;
    • Effective dates for rule changes and links to announcements.

    Alert thresholds can also be set. For instance, notify users when weekly total consumption exceeds 70%, and prompt them to reduce frequency or switch models when it exceeds 90%. The implementation cost of these fields and thresholds is low, but they significantly reduce the user’s cognitive load.

    With this information, users can schedule their work effectively: placing high-intensity tasks right after a window refresh and splitting long tasks across multiple windows.

    2. Model Capability Evaluation: Using Uncontaminated Problems as Probes

    Beyond quotas, evaluating model capability is equally important. Mathematical reasoning is a highly differentiating task, so it serves as a good example to illustrate an actionable evaluation methodology.

    1. Why Use Unpublished Propositions?

    The key to evaluating mathematical capability is avoiding data contamination. If both the problem and the answer appear in public training corpora, the model might complete the task through memorization, testing retrieval rather than reasoning ability. One approach is to select conclusions from public literature that only state the proposition without providing the proof, asking the model to complete the proof and output structured results. For example:

    1. Please prove the following proposition and write it as an English lecture in LaTeX format.
    2. Requirements: Clear definitions, step-by-step derivation, and annotation of dependencies for key lemmas.

    Since there is no ready-made proof to memorize for such problems, the rigor of the output, symbol consistency, and completeness of the derivation chain better reflect the model’s reasoning level. Note that “unpublished proof” only reduces the risk of contamination but does not eliminate it entirely. If the proposition has been proven in subsequent papers or blogs, the model might still be affected. Therefore, subsequent literature for the proposition should be searched before use.

    2. What Criteria Should Be Used for Scoring?

    The outcome of a single problem is subject to chance and should not lead to direct conclusions. It is recommended to establish a scoring methodology covering at least three dimensions:

    • Correctness: Whether the logic of the proof holds and if there are errors in key steps;
    • Rigor: Whether definitions are clear and boundary conditions are omitted;
    • Symbol Consistency: Whether symbols are consistent throughout and if the same symbol denotes different objects.

    Each dimension can be scored from 0 to 2, with at least two reviewers with a mathematical background scoring independently. In case of significant disagreement, a third reviewer should audit. Multiple sampling is best for the same problem, such as running each problem 5 times independently and counting pass rates, rather than looking at a single output.

    The number of samples and problems should be included in the conclusion. For example, “Tested on 20 propositions, run 5 times each for 100 total outputs, with a pass rate of X%” is much more informative than “the model is very strong.”

    3. Tiered Evaluation Framework

    Based on the above methodology, a model’s mathematical capability can be roughly divided into three tiers. The criteria for each tier are as follows, though specific thresholds need calibration based on problem difficulty and review results.

    1. Top Tier: Capable of handling open mathematical problems and providing derivations with research value. The criteria: on unsolved or recently advanced problems, it can provide intermediate conclusions recognized by experts. Few models currently reach this tier, so specific evaluation results should dictate this classification.
    2. Strong Reasoning Tier: Capable of reliably completing theorem proofs and construction problems at the undergraduate to graduate level. The criteria: on standardized proof problems, the pass rate across multiple samples remains consistently high, and the error rate in key steps of long derivation chains is low.
    3. Mainstream Tier: Capable of completing routine derivations, proof reproductions, and lecture organization. The criteria: simple proofs pass, but in derivations exceeding ten steps, manual verification is frequently required.

    The significance of tiering lies in providing observable evidence. If grading is based solely on subjective impressions, conclusions are easily skewed by personal bias.

    4. What Should Everyday Developers Focus On?

    For everyday developers, benchmark scores do not always reflect the actual user experience. More practical dimensions include:

    • The trade-off between chain-of-thought length and cost: The longer the chain of thought, the higher the consumption per run. You need to evaluate the quota consumed by each type of problem.
    • Output format stability: Whether it can output LaTeX, JSON, or specified structures as requested.
    • Symbol convention consistency in long contexts: Whether definitions remain consistent throughout long conversations.

    These metrics can be tested using your own real tasks. For example, select ten representative tasks, record the quota consumption ratio for each, whether the output is directly usable, and how many modifications are needed. This yields a cost-performance estimate much closer to actual work scenarios.

    3. How to Make a Judgment

    The value of a subscription service depends on two parts: model capability, and the rationality of quotas and service rules. Model capability largely depends on technical investment, while quota rules depend on whether the product design is transparent and consistent.

    For service providers, quota ratios can be validated using window counts, reset boundaries should align with billing cycles, and the console should display real consumption status. The cost of these improvements is typically low, but they directly reduce user confusion.