How to Systematically Evaluate LLM Capabilities in Math and Coding
A practical framework for evaluating large language models: using open-ended math problems to test deep reasoning, real-world engineering tasks to assess coding ability, layered agent collaboration, and quota/limit awareness to measure long-task reliability—plus a concrete evaluation methodology and optimization tips for developers.
How to Systematically Evaluate LLM Capabilities in Math and Coding
1. Don’t Judge a Model by Whether It Can “Chat”
When discussing whether a large language model is strong, it’s easy to fall into two traps: judging it only by how fluently it handles everyday Q&A, or by whether it gets one specific question right. In reality, model capability spans at least the following dimensions:
- Knowledge comprehension: whether the model accurately understands problem context, constraints, and domain terminology.
- Logical reasoning: whether it maintains consistency across multi-step derivations.
- Open-ended problem solving: whether it can independently propose and verify approaches for problems with no fixed template or known answer.
- Code engineering: whether it can read existing projects, locate defects, design changes, and deliver working implementations.
- Tool and agent collaboration: whether it can decompose tasks sensibly, call tools, inspect results, and correct itself based on feedback.
- Service stability and cost: response speed, context length, quota policies, continuity in long tasks, and failure recovery.
Evaluating a general-purpose LLM therefore shouldn’t be reduced to “better than model X” or “worse than model Y.” A more sound approach is to build a task matrix and observe the model’s overall performance under different workloads.
2. Open-Ended Math Problems Are a Key Differentiator
Standard math problems usually follow recognizable patterns. A model can often get the right answer simply by identifying the problem type and applying a familiar solution template. Open-ended math problems are different. They typically have these characteristics:
- The problem statement is incomplete, requiring the model to clarify assumptions;
- Multiple solution paths exist, or no established template applies;
- Intermediate conclusions need repeated verification;
- The final answer must not only be correct, but also explain why it holds;
- The problem may require constructing counterexamples, finding boundary conditions, or proving a proposition.
In these tasks, the key capability is not how long the model’s “chain of thought” looks, but whether it can close the loop:
Understand the problem → form hypotheses → construct a method → derive results → check for gaps → revise conclusions.
When evaluating, focus on these metrics:
1. Correctness of conclusions
This is the most basic metric, but don’t just check the final number. Confirm that the conclusion satisfies every condition of the problem and that no hidden edge cases are missed.
2. Completeness of derivation
Some models jump straight to the result while omitting key steps. For research, engineering, and teaching purposes, such answers have limited reusability. A high-quality answer should clearly state key transformations, the conditions under which theorems apply, and the boundaries of the reasoning.
3. Self-verification
Whether a model proactively checks extreme cases, simple examples, and counterexamples is a major factor separating ordinary answers from high-quality reasoning. Especially in complex proofs, the ability to discover that its own assumptions don’t hold is often more valuable than producing a polished answer in one shot.
4. Handling uncertainty
A genuinely reliable model won’t force a definitive answer when information is insufficient. Instead, it will point out the missing conditions and explain what different conclusions follow under different assumptions.
3. Coding Ability Should Be Measured on Real Engineering Tasks
Algorithm challenges alone are not enough to evaluate coding ability. Common tasks in enterprise development include:
- Reading unfamiliar code and tracing call chains;
- Locating intermittent bugs;
- Adding a small feature without breaking existing logic;
- Writing unit tests and boundary tests;
- Optimizing slow queries, memory usage, or concurrency handling;
- Reviewing security risks and exception handling;
- Modifying configuration, logging, and monitoring logic.
To evaluate a model, use a small but real codebase with several tasks that have clear acceptance criteria. For each task, record:
- Whether the first proposed solution was executable;
- Whether unrelated files were modified;
- Whether existing interfaces were broken;
- Whether tests were added;
- Whether the model could continue fixing issues based on error logs;
- How much human review time was ultimately needed.
What to Focus on in Code Review
When a model performs code review, don’t just check whether it catches syntax errors. Pay attention to:
- Boundary conditions: are null values, timeouts, duplicate requests, and extreme inputs handled?
- Resource cleanup: are files, connections, locks, and threads properly released on exception paths?
- Concurrency safety: does shared state have race conditions?
- Error propagation: are exceptions swallowed? Is error information sufficient for debugging?
- Security: input validation, access control, sensitive data leakage, and injection risks?
- Maintainability: does the change fit the existing architecture, or does it introduce excessive abstraction?
A model may perform well at generating new code but struggle with understanding legacy code, following project conventions, and controlling the scope of changes. Practical engineering evaluation should therefore prioritize “modifying an existing system correctly” over “writing a demo from scratch.”
4. Agent Effectiveness Depends on Task Division
Many developers assign all work to a single main model. The result is not only higher quota consumption, but also wasted reasoning capacity on trivial tasks. A better approach is layered collaboration:
- Main model: requirement understanding, architecture decisions, complex reasoning, and final acceptance;
- Helper model: code search, format conversion, simple refactoring, log summarization, and preliminary testing;
- Tool system: running tests, static checks, builds, log queries, and generating diffs/patches;
- Human developer: confirming business goals, risk boundaries, and final merge.
The core principle is not “more models is better,” but matching tasks of different difficulty to different capability levels and costs.
For example, a code-fixing workflow might look like this:
- The main model reads the requirements and clarifies acceptance criteria;
- The helper model scans relevant directories and lists potentially affected files;
- The main model analyzes call relationships and drafts a change plan;
- The helper model generates test cases or runs basic checks;
- The main model refines the implementation based on test results;
- The tool system runs the full test suite;
- The main model summarizes changes, risks, and rollback procedures.
Note that agent systems need clear boundaries. Helper models should not modify core modules at will, and should never execute high-risk operations without confirmation. Configuration files should explicitly define task scope, output formats, forbidden actions, and failure-handling procedures.
5. Quota Limits Directly Affect Long-Task Experience
No matter how capable a model is, if long tasks are frequently interrupted by time or quota limits, real-world productivity suffers. Common restrictions include:
- Per-session time limits;
- Short-cycle usage quotas;
- Periodic total quotas;
- Separate metering for advanced reasoning modes;
- Limits on the number of parallel sessions;
- Caps on tool call counts.
These limits can create an illusion: users feel the model has “suddenly gotten weaker,” when in fact the current session may simply have been downgraded, paused, or switched to a different service tier.
Before committing to long-term use, clarify three questions:
1. How are quotas calculated?
Some systems count requests, some count input/output volume, and others factor in reasoning complexity, tool call counts, and concurrency. You cannot infer long-term availability from a single short conversation.
2. Do different modes share quotas?
Heavy reasoning, regular chat, code execution, and agent tasks may share one pool of resources or be metered separately. If multiple sessions run simultaneously, actual consumption typically rises noticeably.
3. Can context be recovered after interruption?
If a task cannot be accurately resumed after interruption, don’t run long tasks for hours at a stretch. A safer approach is staged checkpointing: ensure each phase produces independently verifiable artifacts, such as design documents, patch files, test reports, and issue lists.
6. Building a More Reliable Evaluation Framework
Four categories of test sets are recommended:
Category 1: Baseline capability tests
Standard math, code generation, text comprehension, and structured output—used to confirm the model is basically usable.
Category 2: Hard reasoning tests
Open-ended math problems, complex constraint analysis, counterexample construction, and multi-stage proofs—used to assess stability on unfamiliar problems.
Category 3: Engineering practice tests
Using real or anonymized projects, test defect localization, incremental changes, test supplementation, performance analysis, and security review.
Category 4: Long-task and recovery tests
Have the model work continuously on extended tasks and observe whether it forgets constraints, repeats work, drifts from goals, and whether it can resume after interruption.
Final scoring should go beyond “right or wrong.” Also record:
- Completion time;
- Number of interaction rounds;
- Number of human revisions;
- Scale of code changes;
- Test pass rate;
- Error recovery ability;
- Resource consumption;
- Explainability of the output.
7. Practical Optimization Tips
Developers can adopt the following strategies to reduce failure rates:
- Break large tasks into phases, with clear inputs, outputs, and acceptance criteria for each phase.
- Have the model analyze first before allowing changes, to avoid large-scale code writing from the start.
- Keep diff records for every change, making rollback and human review easier.
- Let the model generate tests proactively, but never treat the tests themselves as the sole proof of correctness.
- Use high-capability modes for complex reasoning; delegate mechanical tasks to lightweight modes.
- Set directory, command, and permission boundaries for agents to prevent accidental operations.
- Periodically save summaries during long tasks, recording what’s done, what remains, and current assumptions.
- Don’t rely solely on model leaderboards—validate with your own real tasks.
Conclusion
What truly matters when evaluating an LLM is neither a marketing leaderboard nor one impressive answer, but whether it can deliver results in real tasks consistently, stably, and verifiably. Open-ended math problems test deep reasoning; code engineering tasks test comprehension and execution; agent collaboration tests task planning; and quota and rate-limiting mechanisms determine whether these capabilities can translate into long-term productivity.
When choosing a model, evaluate both the “capability ceiling” and the “usability floor”: the former determines how hard a problem it can solve, the latter determines whether it remains dependable in everyday work. Only by placing reasoning quality, engineering reliability, task recovery, and resource cost within the same evaluation framework can you reach objective, reproducible, and genuinely useful conclusions.
