Evaluating LLM Capabilities in Practice: Math Reasoning, Code, and Quota Management

Whenever a new generation of large models is released, the technical community inevitably produces a wave of “how good is it” discussions: some rank capabilities, some complain about usage limits, and others share workflow experiences. These discussions often contain genuine insight, but most remain at the level of subjective impression—a few test questions and an evening of use are enough for people to declare which model is stronger or weaker. This article attempts to distill the most informative facets of these discussions—capability evaluation, chain-of-thought quality, and quota and cost management—into a reproducible, comparable methodology that helps developers maintain rational judgment when facing a flood of new models.

1. Why Subjective Testing Distorts Results

Subjective experience is not without value—it reflects the overall feel of real workflows—but it suffers from at least three systematic biases:

1. Interface differences. The same base model can perform noticeably differently across entry points (chat interface, workbench, API): default system prompts differ, context truncation strategies differ, and whether deep reasoning is enabled by default differs. Using experience from one interface to dismiss capabilities on another produces conclusions that cannot stand. Fix the entry point and parameters before evaluation; otherwise cross-interface comparisons are meaningless.

2. Sampling randomness. LLM output is fundamentally probabilistic sampling—running the same problem multiple times can yield vastly different results. Declaring that “a model can’t handle a certain type of problem” after only one or two runs is statistically insufficient. The rigorous approach is to sample the same problem at least three to five times and look at pass rates rather than single-run success or failure.

3. Problem representativeness. Randomly chosen problems often fall within a model’s comfort zone or happen to hit questions seen during training. You think you are testing reasoning, but you are actually testing memorization.

Only after addressing these three points can subjective experience be upgraded into method.

2. Math Reasoning: Open-Ended Problems Are the Real Test

When evaluating math ability, public benchmarks have a fatal weakness: data contamination. Models may have “memorized” answers and solutions during pretraining, and a high benchmark score does not necessarily indicate genuine reasoning ability.

A more reliable test is open-ended math problems, characterized by:

• No ready-made standard answer, or an answer that requires construction and derivation rather than retrieval;
• Multi-step symbolic manipulation, case analysis, or proof by contradiction;
• Intermediate steps that can be cross-checked with symbolic computation tools.

For evaluation, build your own bank of 20–30 problems covering algebraic construction, combinatorial counting, inequality proofs, and algorithm correctness arguments, and periodically change variables and problem framing. Score on three dimensions: final correctness (verifiable by computational tools), intermediate step accuracy (sampled manual review), and failure mode analysis (conceptual error, calculation carelessness, or giving up). With this problem bank, running the same set when new versions or models are released yields vertically comparable capability curves.

3. Chain-of-Thought Quality: Length Does Not Equal Depth

Reasoning styles vary widely across models: some produce extremely long chains with elaborate branching; others are concise and direct, their prose even resembling how a human engineer thinks. When evaluating chain-of-thought, it is easy to fall into the illusion that “longer is stronger.” In reality:

The cost of long chains. High token consumption and long wait times; more critically, errors propagate along the chain—one small mistake early on may lead the model to repeatedly rationalize in subsequent steps, going further and further astray, ultimately producing a solution that is “rigorously wrong all the way to the end.”

The value and risk of short chains. Concise chains indicate the model grasps the problem structure directly, but steps may be skipped, and key skips require manual review.

It is advisable to score chain-of-thought on three separate dimensions: step correctness (no factual or reasoning errors), conciseness (no redundant detours), and self-correction ability (whether the model can find and fix its own errors). High scores on all three indicate genuinely high-quality reasoning; readability that “writes like a human” is an additional bonus in teaching and collaborative scenarios.

4. Coding Ability: Verify at Three Levels

The demands of daily development on models can be split into three levels, each with its own test tasks:

1. Comprehension layer—code review. Give the model a piece of real code that “runs but smells bad” and see whether it can identify design problems, missing boundary conditions, concurrency risks, and exception-handling gaps. This tests reading and judgment.

2. Localization layer—defect investigation. Give the model a module with a clear bug and its error symptoms, and see how accurately it locates the root cause and whether its investigation path is economical (it does not insert logs aimlessly everywhere).

3. Production layer—incremental features. Add a small feature to an existing system and examine respect for the existing architecture: does it follow project conventions, does it arbitrarily change public interfaces, does it add the necessary tests.

A special reminder: do not use toy problems like “write a quicksort” for evaluation—mainstream models have long been able to write these correctly, and the discriminative power is near zero. Incremental tasks in real codebases are where differences emerge.

5. Quota Management: Understanding the Logic of Multi-Window Stacked Limits

Subscription-based model services generally adopt a multi-window stacked limit design, typically “an hourly rolling window plus a weekly window.” Many users watch only one dimension and unexpectedly hit a wall on the other—the short window still has plenty of quota left, but the weekly quota is exhausted first. A few practical principles:

1. Understand the quota structure first. Probe with small tasks: roughly how many rounds of deep reasoning does the hourly quota support? What is the approximate ratio between weekly and short-window quotas? This directly determines your usage rhythm and prevents squandering quota in the first two or three days.

2. Reserve high-value quota for high-value tasks. Limits are essentially a budget: reserve deep reasoning and difficult bugs for the high-intensity tier, and delegate trivial tasks to lightweight models (see the next section).

3. Add your own layer of usage tracking. No matter how detailed the server-side reports are, keeping a local call log is always prudent:

# Usage tracking illustration: token usage statistics within a sliding window
import time
from collections import deque
class QuotaTracker:
    def __init__(self, window_seconds: int, budget: int):
        self.window = window_seconds
        self.budget = budget
        self.records = deque()          # [(timestamp, tokens), ...]
    def log(self, tokens: int):
        self.records.append((time.time(), tokens))
        self._evict()
    def _evict(self):
        now = time.time()
        while self.records and now - self.records[0][0] > self.window:
            self.records.popleft()
    def usage_ratio(self) -> float:
        self._evict()
        return sum(t for _, t in self.records) / self.budget
# Instantiate two windows, corresponding to the short window and the weekly window
tracker_short  = QuotaTracker(5 * 3600,      budget_5h)
tracker_weekly = QuotaTracker(7 * 24 * 3600, budget_7d)

The right response after hitting a limit is not to wait idly for recovery but to downgrade the task to a lightweight model and keep making progress. In team collaboration scenarios, you can also use a unified cloud-based model service entry point, together with general invocation and monitoring capabilities (for example, related general service categories on Baidu AI Cloud), to centralize quota records on the server side and avoid each member “flying blind” individually.

6. Multi-Agent Division of Labor: Let Lightweight Models Handle Secondary Tasks

Another lever for cost control is tiered task orchestration: the main model acts as commander, and lightweight models act as execution workers.

• Main model (high-intensity tier): responsible for task decomposition, solution design, difficult reasoning, and final review;
• Lightweight model (low unit price, low latency tier): handles deterministic chores such as initial code review screening, simple bug localization, and small utility function implementation.

You can maintain a “division-of-labor policy” file in the project as a shared behavioral convention for all agents:

# Division of labor policy (illustration)
1. Complex reasoning, architecture decisions, and difficult defects: handled personally by the main model (high-intensity tier);
2. Routine code review, simple bug localization, and small function implementation: delegated to lightweight models first;
3. Every delegated task must include: relevant file paths, acceptance criteria, and expected output format;
4. Output from lightweight models must be reviewed by the main model before adoption; it must not be merged directly without review.

This approach yields three benefits: saving money and quota—the main model only handles problems that truly require it; speed—chores and main tasks proceed in parallel; cross-validation—the “delegate + review” process naturally forms a dual-model mutual check, and the error detection rate is actually higher.

7. Turn Experience into Your Own Evaluation Baseline

Finally, solidify the above methods into an evaluation checklist and run it whenever a new model or version is released:

Capability dimension	Test set	Core metrics
Math reasoning	Self-built open-ended problem bank (20–30 problems)	Pass rate, step accuracy
Chain-of-thought	Sampled problems	Conciseness, self-correction rate
Code comprehension	Real project review tasks	Issue detection rate
Code localization	Modules with injected defects	Root cause hit rate
Code production	Incremental feature tasks	Runs, has tests, follows architecture
Cost efficiency	Fixed workload	Token consumption, end-to-end latency

Record score changes vertically for each version and compare strengths and weaknesses across models horizontally. After three months, you will have an evaluation table that fits your real workload better than any public leaderboard—and when you next see a discussion about “how good is this new model,” you will have your own scale: which model to use in which scenario, how to spend quota, and how to split tasks will all be clear.