How to Evaluate an LLM’s Math and Coding Ability Without Fooling Yourself

1. The Value and Pitfalls of Experiential Evaluation

A common evaluation style in technical communities looks like this: a developer runs a model through several difficult tasks and then ranks it based on intuition—“Model A is stronger than Model B,” or “Model C sits between Model D and Model E.” These first-hand impressions are not worthless. They come from real workloads, so they often reflect day-to-day usage better than leaderboard scores.

But if you accept such impressions without constraints, at least three systematic biases creep in:

1. Task distribution bias. Evaluators tend to use problems from fields they already know well. The sample is small and unrepresentative. This is especially acute in math: competition problems, standard textbook exercises, and genuinely open mathematical questions test different levels of ability. Mixing them together and ranking models on the result has limited meaning.
2. Scoring capability bias. Closed problems have standard answers and can be machine-graded. Open mathematical problems often cannot even be quickly verified as “correct or not,” so conclusions depend heavily on the evaluator’s own expertise. The error is hard to control.
3. Recency bias. Model versions iterate quickly. A conclusion from two or three months ago may already be stale, yet community discussions often keep citing old results.

A better approach is not to dismiss experiential evaluation, but to engineer it: fix the tasks, fix the process, record metrics, and let subjective experience settle into reproducible data.

2. Building a Personal Evaluation Framework: Layering, Controlling Variables, and Tracking Metrics

2.1 Design Layered Evaluation Tasks

Mathematical reasoning should be evaluated in at least three layers:

• Closed verification layer: Problems with a single standard answer, such as competition problems and textbook exercises. These can be auto-graded and are suitable for large-scale pass-rate statistics.
• Semi-open layer: Problems that may have multiple answers but can be cross-validated, such as propositions with several valid proof paths. The evaluator judges whether the reasoning process is sound.
• Open exploration layer: Problems with no accepted solution. These are used only to observe a model’s exploration strategy, conjecture quality, and self-correction ability. They are not scored directly.

Coding ability also needs layering: bug localization, code review, small incremental feature development, and cross-module refactoring differ in difficulty and test different skills. The typical daily workloads developers describe—finding bugs, reviewing code, and adding small features to a system—cover mid-level engineering tasks and serve as a useful reference for layer design.

2.2 Control the Variables

• Use fixed prompt templates to avoid noise from wording differences.
• Fix sampling parameters such as temperature.
• Sample each problem multiple times and record the pass rate rather than judging from a single run.
• Record the test date and model version number so results can be reproduced later.

2.3 Track Multiple Metrics

Beyond accuracy, record chain-of-thought length, tokens consumed before reaching the correct answer, the number of detours and self-corrections, and whether human hints were needed. Together these form a capability profile rather than a single ranking.

3. Chain-of-Thought Quality: An Underrated Evaluation Signal

Chain-of-thought styles vary widely across models. Some are concise and direct; others are verbose and repetitive. Many experienced users have observed that “some models have the most concise chains of thought and express themselves more like humans, while others have the longest and most complex chains.” That is precisely the quality dimension at work.

From an engineering perspective, chain-of-thought carries at least three signals worth recording:

1. Cost signal. Chain-of-thought tokens directly affect cost and latency. Solving the same problem with three thousand tokens versus thirty thousand tokens makes an enormous economic difference in batch workloads.
2. Convergence signal. A long chain of thought does not necessarily mean weak capability. Some models trade longer exploration for higher accuracy. But “length increases while accuracy does not” is a clear negative signal.
3. Readability signal. Whether the chain of thought is easy for humans to audit directly affects in-the-loop verification efficiency.

A practical composite metric is “expected cost per correct answer”: accuracy divided by average token consumption, or average latency. Use it to compare cost-effectiveness across models on the same task layer.

4. Use Interval Positioning Instead of Exact Rankings

A professional phrasing worth adopting: say a model “falls between two other versions” rather than forcing a first, second, and third place. This better reflects the reality of model capability—it is a multidimensional vector, and any single ranking inevitably loses information.

Consider using this conclusion format in personal evaluation reports:

• Mathematical reasoning: Based on N problems with M samples each, consistently stronger than Tier X models, mixed results against Tier Y models, and weaker than Tier Z models.
• Code review: Comparable to Tier X models, but with a more concise chain of thought and lower average cost per problem.
• Conclusion validity: Note the test date and version number, and state that the conclusion is only valid within that time window.

This “interval plus validity period” style is more robust than absolute rankings and less likely to mislead others.

5. Understanding Quota Mechanics in AI Coding Tools

Quota exhaustion is another frequent pain point. Many AI coding tools use a rolling quota model that combines a short window and a long window. For example, a short window may reset every five hours, while a long window resets weekly. Your actual available capacity is determined by whichever constraint hits its limit first.

Two typical failure scenarios emerge under this mechanism:

1. Running long unattended tasks. You start a long task before going to sleep. The model exhausts the short-window quota, waits for the window to reset, and continues. The short window repeatedly recovers and gets consumed again, and only when the long window is quickly drained do you realize that a week’s quota has been burned through in a few days.
2. Using the highest tier for everything. Every session runs on the top-tier model. Each task consumes a large amount of quota, and the long window bottoms out quickly.

Corresponding strategies:

• Estimate consumption before starting a task, using historical data such as “this type of task consumes roughly X percent of the short-window quota per hour.”
• Split long tasks into interruptible subtasks with checkpoints, and confirm at each checkpoint whether to continue.
• Use monitoring to make consumption visible instead of discovering it only when you hit the limit.

6. Multi-Agent Task Routing: Save the High Tier for What Matters

The core method for controlling consumption is task routing. A common pattern is the “main agent plus sub-agents” architecture:

• Main agent uses a high-tier model and handles task planning, complex reasoning, architectural decisions, and final integration.
• Sub-agents use lightweight models for mechanically verifiable work: bug localization, code review, and small incremental feature implementation.

Many AI coding tools allow you to declare sub-agent model tiers and responsibilities in an agent policy file. A typical configuration looks like this, with fields shown as pseudocode. Consult your tool’s documentation for exact syntax:

session:
  model_tier: high            # Main session: high tier
subagents:
  - name: reviewer
    model_tier: light         # Sub-agent: lightweight tier
    tasks: [bug_hunt, code_review, small_feature]
policy: |
  Route mechanically verifiable work such as bug localization,
  code review, and small incremental features to lightweight
  sub-agents first. Use the main model only for architectural
  decisions and complex reasoning.

Practical experience shows that for workloads like “finding bugs, reviewing code, and adding small features,” explicitly dispatching most of the work to lightweight sub-agents can complete noticeably more sessions in the same time window and reduce quota consumption by a meaningful margin. The key is to write the routing rules as explicit instructions in the policy file rather than relying on the model to make the right call on its own.

This “high-tier planning plus lightweight execution” pattern is applicable across different platforms and model services, not tied to any single vendor.

7. Quota Observability: Make Consumption Auditable

Finally, close the management loop by recording the time, tier, duration, task type, and quota consumption percentage of each session. A minimal log entry looks like this:

import time, json
def log_session(tier, minutes, task_type, quota_used_pct):
    with open("quota_log.jsonl", "a") as f:
        f.write(json.dumps({
            "ts": time.strftime("%Y-%m-%d %H:%M"),
            "tier": tier,
            "minutes": minutes,
            "task": task_type,
            "quota_used_pct": quota_used_pct,
        }) + "\n")

After one to two weeks of data, you can estimate how much of the short-window and long-window quota an average high-tier session consumes per hour. Before accepting a new task, you can then answer two key questions: how much quota is left today, and whether this task deserves the high tier. When more competing models are released and your options expand, this ledger also becomes the basis for switching plans and re-comparing costs.

8. Conclusion

Returning to the original question: “How do you evaluate a model?” A mature answer is not a ranking. It is a methodology. Use layered evaluation sets to produce a capability range. Use chain-of-thought quality and expected cost per correct answer to describe cost-effectiveness. Replace absolute rankings with time-stamped interval conclusions.

On the usage side, understand how rolling-window quotas work. Use multi-agent routing to reserve the high tier for tasks that genuinely require strong reasoning. Use session logs to make every unit of quota auditable.

Models will keep iterating. The methodology is the asset you can reuse over the long term.