Evaluating LLMs for Complex Math and Programming Tasks: A Practical Framework

    As LLM applications move from simple chat to long-running engineering execution, evaluation methods must evolve too. This article presents a practical framework for assessing large models on complex math proofs and automated programming tasks. It covers multi-layer metrics for math proofs—correctness, rigor, verifiability, and expression quality—along with engineering evaluation of automated coding agents built on main/sub agent roles, clear task boundaries, and acceptance-first workflows. It also explains quota governance and scheduling for long-running tasks, including checkpoint resumption and degradation strategies, plus quality monitoring and observability design. The article closes with team-level adoption advice and common pitfalls, pointing toward evaluation that is stable, reproducible, and governable under real constraints.

    1. Background: From Single Q&A to Long-Running Complex Tasks

    As LLM applications evolve from “chat and Q&A” toward “engineering execution,” evaluation methods must evolve with them. In the past, people often judged model capability with a single simple question, a one-off code completion, or a text summary. But when a task becomes “prove a complex mathematical proposition and produce English lecture notes” or “let an automated coding agent run for hours to make code changes, discover defects, and iterate on small features,” the evaluation target is no longer a single output point. Instead, it is the model’s stability, rigor, resource management, and engineering controllability across a long task chain.

    These tasks typically have three characteristics:

    1. Complex but decomposable goals
      A math proof is not a single answer but a complete chain of definitions, lemmas, reasoning, boundary checks, counterexample elimination, and expression organization. Programming tasks are similar: understanding requirements, locating code, modifying implementations, running verification, and performing regression checks.

    2. Long processes with scattered failure points
      The model may fail at any stage: concept understanding, formula derivation, symbol consistency, code context matching, or test case generation. If you only look at the final output, you can easily misjudge capability.

    3. Significant resource constraints
      Long-running tasks involve session windows, cycle quotas, concurrency, context length, and call costs. Without a scheduling strategy, a task may be interrupted near its limit, leading to irreproducible results or reduced engineering value.

    Therefore, evaluating LLM capability should evolve from “generating one answer” to “continuously producing verifiable results under constraints.”


    2. An Evaluation Framework for Complex Math Proof Tasks

    Take “prove a mathematical proposition and output English lecture notes” as an example. It is not enough to observe whether the model writes a plausible-looking proof. A more rigorous approach is to establish multi-layer evaluation metrics.

    2.1 Correctness: conclusions and reasoning must be traceable

    Correctness is the first standard. For math tasks, check:

    • Whether the proposition’s premises are accurately understood;
    • Whether the target conclusion is clearly stated;
    • Whether unproven content is treated as known;
    • Whether there is circular reasoning;
    • Whether boundary conditions are omitted;
    • Whether it fails in special cases;
    • Whether a necessary condition is written as a sufficient condition, or vice versa.

    During evaluation, you can ask the model to output a “proof skeleton”:

    1. Proposition:
    2. Assumptions:
    3. Goal:
    4. Proof Outline:
    5. Step 1:
    6. Step 2:
    7. ...
    8. Conclusion:

    This structure helps make implicit reasoning explicit, making manual or automated review easier.

    2.2 Rigor: definitions, symbols, and logical connections must be clear

    A common problem in math proofs is not “wrong direction” but “jumps that are too large.” For example:

    • Definitions not stated in advance;
    • Inconsistent symbol usage;
    • Jumping directly from the general case to the conclusion;
    • Lack of explanation for the use of certain lemmas;
    • Careless handling of key steps such as quantifier swapping, limit operations, and set inclusion relations.

    Therefore, lecture-note output should include sections for “definitions,” “symbol table,” “key lemmas,” “main proof,” and “discussion and boundaries.” This elevates model output from a “narrative article” to a “structured technical document.”

    2.3 Verifiability: convert natural proofs into checkable forms where possible

    If conditions allow, ask the model to provide verifiable intermediate results, such as:

    • Describing key algorithms or construction processes with pseudocode;
    • Expressing core invariants with formulas;
    • Using boundary conditions to check whether the proof covers degenerate cases;
    • Using counterexample methods to eliminate wrong propositions;
    • Using formal fragments or rule-based steps to reduce ambiguity.

    Even without relying on a specific formal system, you can improve verifiability through a “checklist”:

    1. checklist:
    2. - Are all assumptions listed?
    3. - Are all non-constant symbols defined?
    4. - Is the validity of the main construction explained?
    5. - Are extreme cases checked?
    6. - Is counterexample elimination provided?
    7. - Are references or known results cited?
    8. - Are there any unproven jumps?

    2.4 Expression quality: English lecture notes need to be academic and teachable

    When the goal is not just “the proof holds” but “written as English lecture notes,” text organization is equally important. High-quality lecture notes should have:

    • Clear heading hierarchy;
    • Abstract or proposition overview;
    • Background motivation;
    • Theorem statement;
    • Proof idea;
    • Step-by-step proof;
    • Key examples or counterexamples;
    • Discussion of common mistakes;
    • Follow-up questions or generalization directions.

    Evaluate from three angles:

    Dimension Focus
    Structure Whether there is an introduction, definitions, theorem, proof, and discussion
    Language Whether terminology is accurate, sentences are natural, and logical connections are clear
    Teachability Whether it helps readers understand the motivation behind the proof, rather than just piling up conclusions

    3. Engineering Evaluation of Automated Programming Agents

    Beyond math proofs, long-running automated programming tasks are another important scenario for testing the engineering capability of LLMs. Such tasks are often completed by “intelligent coding agents,” and their core is not a single completion but continuously completing code changes, defect localization, feature enhancement, and quality review across multiple rounds of execution.

    3.1 Division of labor between main and sub agents

    In complex programming tasks, a single agent is prone to context overload, attention dispersion, and task omission. A more reasonable approach is to split roles:

    • Main agent: responsible for requirement understanding, task decomposition, progress control, and result acceptance;
    • Sub agents: responsible for local tasks, such as finding defects, reviewing code, generating tests, supplementing docs, and implementing small features;
    • Review agent: responsible for checking whether changes comply with standards and whether they introduce regression risks;
    • Test agent: responsible for running unit tests, integration tests, or static checks.

    For example, an automated agent system can be configured as:

    1. roles:
    2. planner:
    3. responsibility: Decompose requirements and assign tasks
    4. outputs:
    5. - task_plan
    6. - acceptance_criteria
    7. implementer:
    8. responsibility: Complete code changes and feature implementation
    9. outputs:
    10. - code_diff
    11. - change_summary
    12. reviewer:
    13. responsibility: Check correctness, maintainability, and style consistency
    14. outputs:
    15. - review_comments
    16. - risk_list
    17. tester:
    18. responsibility: Run tests and summarize failure causes
    19. outputs:
    20. - test_report
    21. - failure_analysis

    The value of this division: the main agent does not have to handle all detail execution, sub agents can focus on local problems, thereby reducing context complexity and improving task success rate.

    3.2 Task boundaries must be clear

    The most common problem with automated agents is “drifting further off course as they work.” To avoid this, each subtask should have clear boundaries:

    1. {
    2. "task_id": "fix-null-check",
    3. "goal": "Fix the exception caused by empty user input",
    4. "scope": ["src/handler.js", "tests/handler.test.js"],
    5. "acceptance": [
    6. "Empty input returns a friendly message",
    7. "Non-empty input behavior unchanged",
    8. "Related tests all pass"
    9. ],
    10. "constraints": [
    11. "Do not modify public interface signatures",
    12. "Do not introduce new third-party dependencies",
    13. "Must run tests before committing"
    14. ]
    15. }

    If task boundaries are unclear, the model may modify unrelated files, over-refactor, change interface semantics, or even introduce new regression problems.

    3.3 Acceptance criteria should be defined up front

    In programming agent tasks, acceptance criteria should come before implementation. Good acceptance criteria include:

    • Functional criteria: how inputs and outputs change;
    • Quality criteria: whether tests and static checks pass;
    • Security criteria: whether null values, out-of-bounds access, permissions, and exceptions are handled;
    • Compatibility criteria: whether old interfaces or old user paths are affected;
    • Rollback criteria: how to restore the original state when a change fails.

    A recommended “acceptance-first” workflow:

    1. Main agent generates the task description;
    2. Review agent adds acceptance conditions;
    3. Execution agent starts making changes;
    4. Test agent runs verification;
    5. Review agent checks the change summary;
    6. Main agent decides whether to merge or roll back.

    4. Quota Governance and Scheduling Strategies for Long-Running Tasks

    A core pain point noted in practical experience is the conflict between long-task execution and quota limits. Abstracted, this is essentially a “long-running task scheduling problem under limited resources.”

    4.1 Model short-term session windows and cycle quotas separately

    Assume a general budget model:

    1. window_budget: available quota within the current time window
    2. cycle_budget: total quota within the current cycle
    3. task_cost: estimated consumption of a single task
    4. retry_cost: additional consumption from failed retries

    Before execution, make a budget judgment:

    1. def can_execute(task_cost, window_budget, cycle_budget):
    2. if task_cost > window_budget:
    3. return False
    4. if task_cost > cycle_budget:
    5. return False
    6. return True

    If a task is large, split it:

    1. def plan_tasks(tasks, window_budget, cycle_budget):
    2. planned = []
    3. used_window = 0
    4. used_cycle = 0
    5. for task in sorted(tasks, key=lambda t: t.priority, reverse=True):
    6. if used_window + task.estimated_cost <= window_budget:
    7. planned.append(task)
    8. used_window += task.estimated_cost
    9. used_cycle += task.estimated_cost
    10. return planned

    The point here is not the specific algorithm but making scheduling “budget-aware.”

    4.2 Priority scheduling: do high-value, verifiable tasks first

    Long-running tasks cannot spread effort evenly. Recommended priority order:

    1. Blocking tasks: those affecting the main flow, key features, or release quality;
    2. High-risk tasks: those involving complex logic, data changes, or access control;
    3. High-verification-value tasks: those that improve test coverage or reduce regression risk;
    4. Low-complexity batch tasks: such as documentation supplements, comment cleanup, or simple renaming;
    5. Low-priority tasks: non-critical optimizations or experimental changes.

    If cycle quota is tight, prioritize completing the full loop of high-value tasks rather than spreading resources across low-value tasks.

    4.3 Checkpoint resumption and state persistence

    Long-running tasks must support checkpoint resumption. Each execution should record:

    • Current task goal;
    • Completed steps;
    • List of modified files;
    • Test results;
    • Failure reasons;
    • Items to retry;
    • Budget consumed.

    You can maintain an execution state file:

    1. session:
    2. task: "refactor-auth-module"
    3. status: "partial"
    4. completed_steps:
    5. - "analyze_dependencies"
    6. - "modify_interface"
    7. pending_steps:
    8. - "update_tests"
    9. - "run_regression"
    10. budget_used:
    11. window: 35
    12. cycle: 24

    This way, even if a quota limit is reached or execution is interrupted, you can resume from the checkpoint instead of starting over.

    4.4 Degradation strategies: avoid task failure due to resource exhaustion

    When the remaining budget is insufficient, activate a degradation path:

    • Degrade from “full implementation” to “issue report”;
    • Degrade from “auto-fix” to “generate fix suggestions”;
    • Degrade from “full testing” to “critical-path testing”;
    • Degrade from “multi-file refactoring” to “single-point patch”;
    • Degrade from “auto-commit” to “commit after manual confirmation.”

    Degradation is not failure; it is a necessary mechanism for engineering systems to maintain availability.


    5. Quality Monitoring and Observability Design

    Evaluating complex tasks cannot rely on subjective impressions but should establish observable metrics. At minimum, it should include the following categories.

    5.1 Correctness metrics

    • Whether proofs pass manual review;
    • Whether code changes pass tests;
    • Whether static checks show no new warnings;
    • Whether regression defects occur;
    • Whether irreversible errors are produced.

    5.2 Process metrics

    • Task start time;
    • Time per stage;
    • Context usage length;
    • Number of calls;
    • Number of retries;
    • Distribution of failure locations.

    5.3 Resource metrics

    • Current window quota usage rate;
    • Cycle quota usage rate;
    • Average consumption per task;
    • Average consumption of failed tasks;
    • Coverage rate of high-value tasks.

    5.4 Quality metrics

    • Manual correction ratio;
    • First-pass rate;
    • Review rejection rate;
    • Test pass rate;
    • Proof rigor score;
    • Lecture-note readability score.

    Recommended alerting mechanisms:

    1. When cycle quota usage exceeds 80%, reduce the frequency of low-priority tasks.
    2. When verification fails twice in a row, pause similar tasks and trigger manual inspection.
    3. When the test failure rate rises rapidly, automatically roll back recent changes.
    4. When a sub agent's output quality stays low, limit its task scope.

    6. Practical Advice for Team Adoption

    If a team wants to use LLMs for math reasoning, coding agents, and automated development, start with the following aspects.

    6.1 Build a tiered evaluation set

    Do not evaluate models with only scattered questions. Build a layered task set:

    • Basic layer: definitions, theorems, common code patterns;
    • Advanced layer: multi-step proofs, complex defect localization, cross-module changes;
    • Engineering layer: long-running tasks, quota limits, agent collaboration, regression verification;
    • High-difficulty layer: open-ended problem exploration, non-standard proofs, complex system design.

    Only tiered evaluation can avoid capability misjudgment caused by “one accidental success.”

    6.2 Design the verification loop before the task

    Whether math proofs or code changes, the verification method should be defined before starting. For example:

    • Math tasks require outputting the proof structure and checklist;
    • Programming tasks require generating test cases;
    • Agent tasks require providing a change summary and rollback plan;
    • Automated tasks require recording failure reasons and budget consumption.

    6.3 Human-machine collaboration over full automation

    In complex tasks, fully automatic execution is often uncontrollable. A more robust model is:

    • Machines handle generation, checking, and retries;
    • Humans set goals, confirm boundaries, and review high-risk changes;
    • The system handles logging, alerting, and rollback.

    This collaboration balances efficiency and risk control.

    6.4 Build reusable prompts and task templates

    Models and agents may differ in capability, so prompts should be templated. For example, a math proof task can use:

    1. Please complete the following task:
    2. 1. Restate the proposition and list all assumptions;
    3. 2. State the proof goal;
    4. 3. Output the proof outline;
    5. 4. Expand the proof step by step;
    6. 5. Annotate key definitions and lemmas;
    7. 6. Check boundary cases;
    8. 7. If there are gaps, point them out and fix them;
    9. 8. Finally, generate the English lecture-note structure.

    A coding agent task can use:

    1. Please complete the change based on the following requirements:
    2. 1. List the affected files;
    3. 2. Provide a change plan;
    4. 3. Maintain compatibility with existing interfaces;
    5. 4. Add or update tests;
    6. 5. Output a change summary;
    7. 6. List risk points and the rollback plan;
    8. 7. Run tests and report results.

    Templating not only improves stability but also makes it easy to compare different models horizontally.

    6.5 Control “unauthorized changes” by automated agents

    Agent tasks should limit the scope of changes to prevent the model from performing large-scale unrelated refactoring just to “complete the task.” Recommended settings:

    • File whitelist;
    • Interface change approval;
    • Dependency introduction limits;
    • Diff review before commit;
    • Manual confirmation for high-risk files;
    • Automatic rollback threshold.

    This can significantly reduce the risk of automated development.


    7. Common Pitfalls and Improvement Directions

    Pitfall 1: Only looking at final results, not process quality

    Even if a proof or code change “looks correct,” it may contain hidden bugs, logical jumps, or unreasonable assumptions. The process must be reviewed.

    Pitfall 2: Treating long tasks as short Q&A

    Short Q&A can rely on a single generation, while long tasks require state management, budget control, checkpoint recovery, and quality monitoring. Their engineering complexity is completely different.

    Pitfall 3: Only pursuing speed without quota governance

    If a task lacks budget awareness, it is easy to be interrupted when cycle quota is exhausted, leaving high-value tasks unfinished.

    Pitfall 4: A single agent handling all complexity

    Complex tasks should be split among different roles. The main agent handles planning and acceptance, sub agents handle local execution, and the review agent handles risk control.

    Pitfall 5: Ignoring observability

    Without logs, metrics, and alerts, you cannot determine whether a problem comes from model capability, prompt design, task decomposition, or insufficient resources.


    8. Conclusion

    Evaluating LLM capability in complex math proofs and automated programming tasks cannot stop at the single-Q&A or single-point generation level. Instead, it should establish a comprehensive evaluation system oriented toward long-running engineering tasks. This system should cover correctness, rigor, verifiability, stability, resource governance, agent collaboration, and quality monitoring.

    For complex math tasks, the focus is on structuring, checklisting, and making proofs reviewable; for automated programming tasks, the focus is on clarifying task boundaries, establishing main/sub agent division, designing acceptance criteria, and ensuring task controllability through quota scheduling and degradation strategies.

    A truly mature evaluation method does not simply judge whether a model “can do” something, but whether it “can complete tasks stably, reproducibly, and governably under real constraints.” Only by examining model capability within an engineering framework can we form a practical, replicable, and continuously optimizable path.