Evaluating LLM Workflows for Coding and Mathematical Reasoning

    This article presents a workflow-level evaluation framework for using large models in coding and mathematical reasoning. It covers task and capability tiers, short- and long-window quota constraints, role separation across primary models, high-speed tiers, subagents, and independent reviewers, as well as practical proof-review and risk-governance practices. Readers gain a structured way to assess sustainable output, quota efficiency, review requirements, and auditable engineering workflows rather than relying on one-off model impressions.

    1. Problem Definition: From One-Off Experience to Sustainable Productivity

    A common mistake when discussing whether a large model can serve as a primary tool is to treat a single conversation result, one code snippet, or one mathematical proof as the whole conclusion. A more reasonable object of evaluation is not an isolated answer but the model’s sustainable output in a real workflow. This includes at least five dimensions: task completion, result correctness, end-to-end latency, quota consumption, and auditability. A model may produce high-quality answers in single turns but have high latency, frequent short-window rate limits, and insufficient long-window quota, so its effective output may actually decline. Another model may be slightly weaker in single-turn capability but faster, parallelizable, and easier to review, making it more economical for small and medium tasks.

    Evaluation should therefore be organized around task tiers. Complex system changes, cross-file refactoring, and open mathematical proofs are high-difficulty tasks that require a high-capability tier, longer context, and stronger planning ability. Defect localization, small feature completion, log analysis, and test generation are medium tasks that can be handled by a high-speed or lightweight tier. Code review, proof review, and risk checks should be performed by an independent review role to avoid a model certifying its own correctness. The original experience of “the primary model is capable enough but slow, another model is brought in for final review, and some tasks are switched to the high-speed tier” is essentially task tiering and role separation.

    2. Capability Tiers: Do Not Replace Task Evaluation with a Single Leaderboard

    Large-model capability should not be reduced to a binary distinction between strong and weak. For developers, at least four capability categories should be distinguished: complex coding and architecture understanding, defect repair and small-step iteration, mathematical proof and rigorous derivation, and documentation writing and readable expression. Complex coding tasks value long context, tool calling, plan decomposition, and cross-file consistency. Small-step iteration values first-token latency, throughput, and the number of attempts per unit time. Mathematical proofs value definition consistency, dependency completeness, counterexample awareness, and formalization potential. Documentation writing values structure, terminology accuracy, and reviewability.

    Community comparisons often show a tendency to place models with different positioning on the same leaderboard. A safer approach is to build a task matrix: for each task category, record success rate, rework rate, average latency, quota consumption, and the number of issues found in review. Mathematical capability requires particular caution: open mathematical problem solving remains scarce, a long chain of thought does not equal correctness, and a concise chain of thought does not equal weak capability. Generating lecture notes for propositions whose proofs are not provided in public literature is a high-value test scenario because it exposes whether the model truly organizes a proof rather than restating available material. Such output must still be reviewed by a human or an independent model, and the generated text must not be treated directly as a theorem proof.

    Task Type Recommended Tier Key Metrics Review Requirement
    Complex coding and refactoring High-capability tier Cross-file consistency, rework rate, latency Independent review required
    Defect localization and small features High-speed tier / lightweight tier Fixes per unit time, incorrect-change rate Sampling review
    Mathematical proofs and lecture notes High-capability tier + review tier Correctness, dependency completeness, readability Strong review and counterexample checks
    Documentation and explanation generation High-speed tier Structural completeness, terminology accuracy Human sign-off

    3. Quotas and Throughput: Short- and Long-Window Limits Are Hard Constraints

    In practice, the biggest constraint beyond capability is often quota. Short-window rate limits affect continuous work, while long-window quota affects weekly throughput. If long-window quota can only be converted into a few short-window sessions, users will clearly feel that effective working time is insufficient, especially during nightly batch processing, continuous debugging, or multi-session parallelism. Evaluation should treat quota as a production resource, not an accessory parameter.

    A quota-budget mechanism is recommended: estimate call count, context length, and review cost before starting a task; reserve long-window headroom for high-priority tasks; use the high-speed or lightweight tier for low-priority tasks; and automatically downgrade, defer, or split tasks when remaining quota falls below a threshold. The pseudo-configuration below shows the general idea:

    1. policy:
    2. task_class: complex_code | math_proof | small_patch | review
    3. tier:
    4. complex_code: high_quality
    5. math_proof: high_quality + independent_review
    6. small_patch: high_speed
    7. review: independent_reviewer
    8. quota_guard:
    9. before_run: estimate_tokens_and_calls()
    10. if short_window_remaining < threshold: defer_or_downgrade()
    11. if long_window_remaining < reserve_for_review: stop()
    12. after_run: record_usage_and_rework()

    Quota governance also requires observability: record each session’s consumption, task type, whether rework occurred, and review finding rate; set budget alerts; and distinguish exploratory consumption from delivery consumption. Otherwise, it is easy to encounter the situation where “it feels like little was used, but quota drops quickly.” For teams, quota consumption should be included in project scheduling rather than switching tools only after hitting a limit.

    4. Workflow Design: Primary Model, High-Speed Tier, Subagents, and Independent Review

    A more robust workflow is role separation: the primary model handles planning and high-difficulty tasks, the high-speed tier handles rapid iteration, the lightweight tier or subagents handle defect localization, code review, and small feature completion, and an independent review role handles final verification. Agent instruction files should clearly define the division of labor: which tasks must be completed by the high-capability tier, which can be delegated to the lightweight tier, and which must retain human sign-off. This improves throughput and reduces long occupation by a single model.

    Note that subagents are not “free parallelism.” Each agent consumes quota, and multi-agent collaboration increases context synchronization and result-merging costs. It is advisable to delegate only clearly bounded, verifiable tasks, such as finding defects, adding tests, checking interface consistency, and generating small patches. Tasks involving architecture decisions, security boundaries, and core steps of mathematical proofs should still be led by the high-capability tier with review headroom reserved.

    5. Mathematical Proof Practice: Generating Lecture Notes Is Not Completing a Proof

    A reasonable process for handling propositions in mathematical literature with a large model is: provide the proposition, context, and output requirements; require the model to explicitly cite dependencies, derive step by step, and mark unproven assumptions; output English lecture notes and compilable typesetting source; and finally perform independent review. Review should at least cover whether definitions are consistent, whether lemma dependencies are complete, whether circular reasoning exists, whether boundary conditions are missing, whether counterexamples can be constructed, and whether key steps can be formalized. For propositions whose proofs are not provided in public literature, the model should be required to state which steps are standard inferences and which are self-supplemented, so reviewers can locate risks.

    When evaluating mathematical output, do not look only at length and typesetting quality; look at verifiability. Readability, dependency completeness, and reproducibility are more important than a very long chain of thought. If the team has the conditions, key lemmas can be converted into formal checks or cross-validated by an independent review model; final sign-off by a domain expert is still required. Generating lecture notes is a means of improving efficiency, not a reason to replace proof responsibility.

    6. Evaluation Methods and Risk Governance

    A fixed task set is recommended: complex refactoring, defect repair, mathematical proof, documentation generation, and review findings. For each task, record success rate, rework rate, latency, quota consumption, and the number of human review issues. Conduct small-scale controlled experiments, but do not draw conclusions from a single result; pay attention to long-tail failures and quota-exhaustion scenarios. Keep logs during the evaluation period so it is easier to review whether the problem was insufficient capability, insufficient quota, or unreasonable workflow design.

    Risk governance includes avoiding single-point dependency and keeping a replaceable model abstraction layer; auditing generated code and proofs; setting quota alerts and degradation strategies; enforcing access control for sensitive code and data; and retaining provenance and review records for generated artifacts. Waiting for new model releases is not a long-term strategy. Establishing task tiering, budget mechanisms, and independent review processes is the sustainable approach. The conclusion is that a primary model being “capable enough but slow” is a common state; high-speed tiers and subagents can improve throughput; review cannot be omitted; quota is a hard constraint; and evaluation should return to an observable, auditable, and reusable engineering system.