Evaluating Mathematical Reasoning in LLMs and Managing Quota Limits in Long-Running Tasks

Introduction: The Dual Challenge of Capability Assessment and Resource Consumption

When selecting and using large language models (LLMs), developers frequently face two interrelated issues. First, how can one objectively determine a model’s true capability in advanced mathematical reasoning? Second, how can long-running autonomous tasks avoid being interrupted by quota limits? The first issue affects model selection decisions; the second affects engineering usability. This article abstracts both problems into general technical propositions and provides a practical methodology.

It is important to note that mathematical reasoning ability is not the same as general conversational ability. A model that performs well on writing or code generation may not reliably handle open-ended mathematical problems. Therefore, evaluation must be conducted across multiple dimensions and task types rather than relying on a single overall impression.

A Framework for Evaluating Mathematical Reasoning Capability

Task Stratification: From Standard Problems to Open Questions

The first principle of evaluation is task stratification. We recommend dividing mathematical tasks into three layers:

1. Standard problem set layer: Competition or textbook problems with definitive answers, used to measure baseline accuracy.
2. Semi-open layer: Problems with a defined answer format but multiple valid solution paths, used to measure reasoning stability.
3. Open problem layer: Problems where the conclusion itself requires exploration or may not yet have a complete solution, used to measure a model’s exploration and self-verification ability.

Most models perform similarly on layer 1. The real differences appear in layers 2 and 3. The ability to handle open mathematical problems, as mentioned in the original context, belongs to layer 3 and is the key differentiator between model tiers. In engineering practice, we recommend preparing independent evaluation sets for each layer and recording both pass rates and average reasoning steps per layer.

Evaluation Dimensions: Beyond Accuracy

Accuracy alone is insufficient. We recommend collecting the following metrics as well:

• Chain-of-thought length and complexity: Excessively long chains indicate redundant reasoning, higher cost, and a greater chance of errors midway. Chains that are too short may skip steps, producing inflated accuracy that is not reliable.
• Reasoning conciseness: At the same accuracy level, models with more concise chains generally generalize better and require less manual review effort.
• Human readability of output style: In scenarios requiring human review of reasoning processes, such as teaching or auditing, readability directly affects deployment efficiency.
• Stability: The variance of multiple sampling runs on the same problem. High variance suggests the model has not yet formed a stable strategy for that type of problem.

A Sound Approach to Tier Ranking

The original context proposed a tier-ranking approach: using open-problem capability as the highest-tier criterion, then using chain-of-thought conciseness to distinguish the next tier, and finally placing models with longer chains but similar capability in the third tier. This approach can be abstracted into the following evaluation flow:

def rank_models(models, open_set, semi_set, std_set):
    scores = {}
    for m in models:
        s_open = eval_pass_rate(m, open_set)      # pass rate on open problems
        s_semi = eval_pass_rate(m, semi_set)      # pass rate on semi-open problems
        s_std  = eval_pass_rate(m, std_set)       # pass rate on standard problems
        chain  = avg_chain_length(m, semi_set)    # average chain length
        scores[m] = (s_open, s_semi, s_std, chain)
    # First stratify by open-problem performance, then sort by chain length/conciseness within each tier
    return sorted(scores.items(),
                  key=lambda x: (-x[1][0], x[1][3]))

The key point is: stratify first, then sort. Comparing all models on a single scalar score masks the qualitative leap represented by open-problem capability.

Avoiding Evaluation Pitfalls

• Sample bias: Testing only familiar problem types can overestimate a model’s ability. Coverage should span algebra, geometry, combinatorics, number theory, and other areas.
• Prompt sensitivity: The same model may perform quite differently under different prompt styles. Evaluation should either fix a prompt template or average results across multiple templates.
• Version drift: Minor model version updates can significantly change mathematical performance. Evaluation results should include the version and date, with periodic regression testing.

Engineering Management of Quota Consumption

The scenario described in the original context—starting a long task at night, hitting a 5-hour window limit, then resuming after recovery and hitting a 7-day window limit—is a typical conflict between long-running autonomous tasks and quota windows. The root cause is a mismatch between the token consumption rate of the task and the quota refresh cycle. The following governance approach addresses this issue.

Task Grading and Budget Estimation

Before submitting a task, perform a budget estimate:

def estimate_budget(task, model):
    # Estimate based on historical token consumption for similar tasks
    base = task.complexity * model.avg_tokens_per_step
    chain_factor = model.expected_chain_ratio  # higher for models with longer chains
    return base * chain_factor * task.max_retries

Classify tasks as light, medium, or heavy based on the estimated budget. Schedule heavy tasks at the beginning of a fresh quota window, and avoid starting them near the end of a window.

Resumable Execution and State Persistence

Long tasks must support interruption and recovery. The core approach is to break the reasoning process into persistable step units:

class ResumableTask:
    def run(self):
        state = self.load_state() or self.init_state()
        while not state.done:
            if self.quota_exhausted():
                self.save_state(state)   # persist and exit, wait for next window
                return
            step = self.next_step(state)
            state = self.apply(state, step)
        self.save_state(state, final=True)

The state should record: completed steps, intermediate conclusions, estimated tokens consumed, and retry counts. This way, execution can resume from the checkpoint after the quota is restored, rather than starting over.

Rate Control and Backoff Strategies

• Token bucket rate limiting: Derive an allowed tokens-per-minute rate from the window quota and proactively throttle requests instead of hitting the limit passively.
• Exponential backoff: On rate-limit errors, retry with exponential backoff and record the time of the rate-limit event for analyzing window boundaries.
• Task interleaving: Run lightweight tasks during rate-limited gaps to improve overall quota utilization.

Observability: Identifying the Root Cause of Rate Limiting

Collect three types of metrics: token consumption distribution per task, cumulative consumption curve within a window, and timestamps of rate-limit events. By comparing the consumption curve with window boundaries, you can determine whether the limit was hit due to a single overly heavy task or excessive concurrency. The former calls for splitting the task; the latter calls for reducing concurrency.

How to Present Model Selection Conclusions

When communicating model selection results, avoid absolute statements. Use conditional phrasing instead:

• “For open-ended mathematical problem scenarios, models in tier A currently lead.”
• “At the same accuracy level, models in tier B produce more concise reasoning chains, reducing manual review cost.”
• “Models in tier C have the longest chains but comparable capability, making them suitable for scenarios that require complete process traceability.”
• “For mid-level mathematical tasks, models in tier D fall between two tiers and can serve as a cost-effective option.”

This style preserves the informational value of the original assessment while avoiding the presentation of subjective evaluation as objective fact, which aligns with the rigor expected in technical writing.

Summary

This article decomposes “mathematical reasoning capability evaluation” and “quota management for long-running tasks” into two engineering problems. The former is addressed through task stratification, multi-dimensional metrics, and a stratify-then-sort evaluation process. The latter is addressed through budget estimation, resumable execution, proactive rate limiting, and observability. Combining these two approaches allows teams to both select the right model and run it reliably in real-world projects. For teams that run inference tasks at scale in the cloud, a complete governance loop can be built using general-purpose cloud capabilities—such as object storage for saving task states, log services for collecting rate-limit events, and monitoring and alerting for tracking quota consumption curves.