LLM Mathematical Proof Evaluation and Long-Running Agent Governance

    This article presents an engineering framework for evaluating reasoning models through verifiable mathematical proof tasks and for governing long-running agents under real capacity constraints. It covers proof scoring criteria, capability tiers, failure modes, budgeted routing, quota control, observability, and retrospectives. The goal is to separate retrieval from reasoning, manage capacity before quotas are exhausted, and make long-running tasks more stable and cost-controlled.

    LLM Mathematical Proof Evaluation and Long-Running Agent Governance

    Evaluating a reasoning model should not rely only on the impression from a single question-and-answer exchange, nor should “whether it can produce a fluent answer” be the sole criterion. A more reasonable approach is to divide evaluation into three dimensions: whether the task is verifiable, which capability tier the model is operating at, and whether resource consumption is controllable.

    A useful core scenario from the original discussion is as follows: given a proposition in a public mathematical literature corpus that has not yet been proved, require the model to complete the proof and write it as English lecture notes in a standard mathematical typesetting format. This task tests long-context understanding, mathematical derivation, structural organization, and output specification at the same time. It cannot simply be equated with retrieval or paraphrasing. Another core scenario is long-running agent execution: after the user sets a goal, the system runs continuously, and the results quickly hit quota limits in both short-cycle windows and weekly windows. Looking at these two types of problems together yields a more engineering-oriented evaluation and governance framework.

    1. Mathematical proof tasks cannot be judged by final text alone

    The difference between proof-oriented tasks and ordinary question answering is that correctness can be checked step by step. Evaluation should at least focus on the following points.

    First, whether the proposition originally lacked an existing proof. If a public proof already exists, the model may complete the task by memorization or retrieval, making it difficult to distinguish derivation ability from retrieval ability.

    Second, whether the proof structure is complete. A qualified output should include definition clarification, lemma statements, key steps, boundary conditions, and conclusions, rather than only a concluding paragraph.

    Third, whether the derivation can be reviewed. Key lemmas should be sampled to check whether they are proved, whether circular reasoning exists, and whether undeclared conditions are implicitly used.

    Fourth, whether the output specification is stable. When English lecture notes are required, check terminology consistency, symbol definitions, paragraph hierarchy, and typesetting markup.

    Fifth, whether long-context constraints are maintained. A common problem in long tasks is conflict between definitions earlier in the text and symbols later in the text. Evaluation should specifically check cross-paragraph consistency.

    Therefore, mathematical capability evaluation should use a scoring rubric rather than merely judging whether the output “looks right.” For example, six items can be used: proposition understanding, proof correctness, structural completeness, symbol consistency, typesetting compliance, and reviewability. Each item should receive a pass/fail result and an evidence location. For open mathematical problems, a further distinction should be made between “closed-problem derivation” and “open-problem exploration.” The former has a clear answer, while the latter requires testable conjectures, counterexamples, or proof routes. Being able to handle closed problems does not mean being able to handle open problems; the two should be evaluated separately. In practice, a model may perform stably on closed problems but provide only heuristic directions on open problems. This is not contradictory; it reflects different capability tiers.

    2. Capability tiers are more meaningful than simple rankings

    When discussing model strength, it is easy to fall into a single ranking. A more reliable approach is tiering: retrieval and paraphrase tier, closed-problem derivation tier, open-problem exploration tier, and long-chain engineering execution tier.

    The retrieval and paraphrase tier focuses on information recall and expression. The closed-problem derivation tier focuses on multi-step reasoning and self-consistency. The open-problem exploration tier focuses on proposing verifiable routes, constructing counterexamples, and self-correction. The long-chain engineering execution tier focuses on task decomposition, tool invocation, state maintenance, and budget control. A model may be strong at one tier and weak at another. The question raised in the original discussion—“whether it can handle open mathematical problems”—is precisely a manifestation of higher-tier capability, but such judgments require multiple problems, multiple rounds, blind review, and human review. They cannot be concluded from a single experience.

    From an engineering perspective, failure modes should also be recorded: whether the model misunderstands the proposition, misses boundary conditions, drifts in symbols, skips proof steps, or loses the goal in long tasks. Failure modes are more instructive than total scores. For mathematical lecture-note outputs, it is advisable to retain intermediate drafts and final drafts so that reviewers can compare whether the model corrected errors during revision. For high-risk tasks, human review and formal verification tools should be introduced. For low-risk tasks, a low-cost model can be used for initial screening, and then a high-reasoning-tier model can review key steps.

    3. Long-running agent quotas are fundamentally a capacity planning problem

    Another pain point in the original discussion is that after an agent runs long tasks continuously, short-cycle windows and weekly windows are quickly exhausted, and there is even an observation that “the weekly quota is only equivalent to several short-cycle windows.” Regardless of the specific platform rules, this is a typical capacity planning problem. The consumption of a long-running agent does not come only from a single generation. It also includes task decomposition, sub-agent calls, tool execution, code inspection, log return, and failure retries. Without a budget model, the system will unknowingly treat the weekly quota as an unlimited resource.

    Governance should cover four actions.

    First, estimate before the task. Estimate tokens, request count, tool-call count, and retry probability by task type, and provide two-level budgets for short-cycle and weekly windows.

    Second, enforce hard limits during execution. Set maximum budgets for each session, each subtask, and each sub-agent. When a threshold is exceeded, pause and request confirmation.

    Third, route by tier. Route low-risk tasks such as bug finding, code review, and small feature changes to low-cost models. Reserve mathematical proofs, architecture decisions, and critical reviews for high-reasoning-tier models.

    Fourth, use circuit breaking and degradation. When short-cycle quota consumption exceeds a threshold, automatically reduce concurrency, shrink context, disable unnecessary sub-agents, or split tasks into the next window.

    The following pseudocode expresses routing and budget policy:

    1. task_policy:
    2. math_proof:
    3. target_tier: high_reasoning
    4. require_review: true
    5. max_retry: 2
    6. bug_scan:
    7. target_tier: low_cost
    8. max_retry: 1
    9. code_review:
    10. target_tier: low_cost
    11. escalate_on_risk: high_reasoning
    12. budget:
    13. short_window_ratio: 0.6
    14. weekly_ratio: 0.5
    15. per_session_cap: 200000
    16. on_exceed: pause_and_confirm
    17. routing:
    18. - if: task_type in ["bug_scan", "small_feature"]
    19. use: low_cost
    20. - if: task_type == "math_proof"
    21. use: high_reasoning

    It should be emphasized that these example fields are only an engineering expression, not parameters for any specific product. The key is to make “budget” a first-class citizen instead of waiting for bills or rate limits to appear before taking remedial action.

    4. Observability and retrospectives determine long-term cost

    A long-running task system must be observable. At minimum, record the following for each session: model tier, input and output size, number of tool calls, number of sub-agents, number of failure retries, quota consumption ratio, and task completion status. Without these data, it is impossible to determine whether the task itself is too difficult or the routing policy is too coarse.

    For example, on platforms such as Baidu AI Cloud, container platforms can host task executors, while logging and monitoring services collect quota consumption, failure rates, and average completion time. This is a general cloud capability composition and does not involve any customized service.

    During retrospectives, the following questions should be answered: Which tasks could have been completed by a low-cost model? Which retries could have been avoided? Which context was repeatedly injected? Which sub-agent calls produced no output? Only through continuous correction based on these questions can “it runs” become “it runs stably and economically.”

    Conclusion

    Reasoning models should be evaluated on both verifiable mathematical capability and long-running agent governance capability. Mathematical proof tasks should use scoring rubrics and human review to distinguish retrieval, derivation, and open-ended exploration. Long-running task systems should use tiered routing, hard budget limits, circuit breaking and degradation, and observability to prevent quota loss of control. A single performance can only serve as a clue; it cannot replace systematic evaluation.