Evaluating Agents in Formal Proof and Scientific Computing: Metrics, Verification, and Practical Workflows

Why Model Capability Needs a New Evaluation Lens

Large-model evaluation has long relied on question-answering accuracy and general benchmark scores. These metrics reflect language understanding, but they struggle to answer the questions engineering teams care about most: Can the model complete a multistep task independently? How much time and compute does it require? Can a third party verify the result?

Once a model is placed inside an agent framework, it plans steps, calls tools, reads and writes files, and iteratively corrects its work. The unit of evaluation therefore shifts from a single answer to a task workflow. Evaluation needs to cover result quality, process cost, and the degree of human intervention at the same time.

Four Core Evaluation Dimensions

1. Task completion: Score against predefined intermediate goals instead of judging only whether the final result looks complete.
2. Resource consumption: Record input and output tokens, number of calls, wall-clock time, and compute resource usage.
3. Verifiability: Determine whether the output can be checked by deterministic tools such as compilation, unit tests, formal verification, or simulation replay.
4. Human intervention rate: Count the number of times human prompting, correction, or goal resetting is required.

The following is an example run-record structure:

from dataclasses import dataclass, field
@dataclass
class RunRecord:
    task_id: str
    milestones_total: int
    milestones_passed: int
    input_tokens: int
    output_tokens: int
    wall_clock_hours: float
    human_interventions: int
    verifier_passed: bool
    notes: list[str] = field(default_factory=list)
    def completion_rate(self) -> float:
        if self.milestones_total == 0:
            return 0.0
        return self.milestones_passed / self.milestones_total
    def tokens_per_milestone(self) -> float:
        passed = max(self.milestones_passed, 1)
        return (self.input_tokens + self.output_tokens) / passed

This structure makes results from different models or configurations comparable in a single table. For a valid comparison, the task set, time budget, tool permissions, and prompt templates must be fixed; otherwise, the numbers are not comparable.

Beyond single-point numbers, progress over time should also be recorded. If the completion rate barely increases after a certain point, the agent has stalled, and additional time is unlikely to produce much return.

It is also important to distinguish “unit price” from “total cost.” A lower per-token price does not necessarily mean a lower total cost for completing the same task, because the number of calls and context length required can also change. The correct metric is “cost per completed milestone.”

Formal Proof: Replacing Manual Review with Machine Checking

Formal proof uses a dedicated proof language to describe each step of a derivation, and a proof checker verifies the steps one by one. If the checker passes, the proof holds within the corresponding formal system; if the checker reports an error, the problem is localized to a specific step. This turns “looks correct” into “machine-confirmable.”

This kind of environment is well suited to closed-loop agent iteration: generate code, compile, read errors, modify, and compile again. Task decomposition is critical. Common practices include:

• Having a human or an existing tool produce a theorem list first, as measurable milestones;
• Assigning the list to multiple parallel work units, each maintaining an independent code module;
• Using a unified interface declaration to avoid naming conflicts between modules;
• Merging periodically and compiling the whole project, while recording the pass status of each milestone.

There are two boundaries to note. First, a passing checker only proves that the formal statement holds; whether the formal statement accurately corresponds to the original mathematical problem still requires human review. Second, if a problem has been public for a long time, training data may contain related solutions. Evaluators should disclose data sources and, where possible, use new problems that have not been made public.

Protein Capsid Structure Prediction: Dividing a Large Problem into Symmetric Subproblems

Viral capsids are assembled from dozens of protein subunits according to symmetry rules. Single-chain structure prediction methods are good at handling individual proteins. Directly predicting the entire assembly significantly increases both computational scale and inter-chain interactions.

One practical approach is divide and conquer:

1. Use a known symmetry group (for example, a common rotational symmetry) to determine the smallest asymmetric unit;
2. Predict adjacent chain segments within the unit separately, retaining overlapping regions for alignment;
3. Use symmetry transformations to copy and assemble the segments into the full structure;
4. Evaluate quality using consistency in overlapping regions and overall geometric metrics.

Structural quality is commonly measured by root-mean-square deviation (RMSD), in angstroms (1 Å is approximately 0.1 nm). If results report the proportion of cases “within 3 Å error,” they should also specify the number of samples, structure type, and source of the reference structure; otherwise, readers cannot judge reliability.

The following code illustrates the core steps of symmetric assembly and omits alignment and deduplication logic:

import numpy as np
def apply_rotation(coords: np.ndarray, rotation: np.ndarray) -> np.ndarray:
    """coords: (N, 3) atom coordinates; rotation: (3, 3) rotation matrix"""
    return coords @ rotation.T
def assemble_symmetric(unit_coords: np.ndarray, rotations: list[np.ndarray]) -> np.ndarray:
    copies = [apply_rotation(unit_coords, R) for R in rotations]
    return np.vstack(copies)

A segmentation strategy can reduce memory usage for a single computation, but total computation may not decrease, because overlap between segments and assembly itself also incur overhead. Actual resource requirements depend on sequence length, model scale, and alignment strategy.

Robot Control: Evaluation Must Separate Seen and Unseen Tasks

Dual-arm robot control evaluation is often conducted in simulation, with tasks such as grasping, assembly, and tidying. A score alone is not very meaningful; the training distribution, whether the task appeared during training, and the gap between simulation and the real robot must also be reported.

A suggested evaluation design is shown below:

Evaluation group	Meaning	What to report
Seen tasks	Tasks that appeared during training	Success rate, number of trials
Unseen same-category	Same skill, different objects	Success rate, gap versus seen tasks
Unseen cross-category	New skill combinations	Success rate, typical failure modes

In addition, each group should include confidence intervals for success rates. With few trials, a difference of a few percentage points may be random fluctuation and should not be used for ranking. When conditions allow, some tasks should also be tested on the real robot, with the deviation between simulation and real-robot results recorded. Random seeds and environment versions should be fixed so others can reproduce the results.

Vulnerability Discovery: Recall and False-Positive Rate Matter Equally

Regression testing against a set of publicly disclosed vulnerabilities is a common practice. Three numbers need to be tracked: recall on known vulnerabilities, number of false positives, and number of valid vulnerabilities found only by the tool. Unique findings are valuable, but they must be manually confirmed and deduplicated; otherwise, duplicate and invalid reports inflate the results.

From an engineering perspective, the process can be organized as follows:

1. Run analysis tools in an isolated environment with read-only access only;
2. Send all candidate issues into a unified triage queue;
3. Have security engineers reproduce issues and determine severity;
4. Notify maintainers through a responsible disclosure process, agree on a remediation deadline, and avoid publishing technical details before fixes are available.

Deployment Checklist

• Decompose tasks into milestones that can be checked automatically
• Record tokens, duration, number of calls, and human interventions
• Disclose overlap between the evaluation set and training data
• Minimize agent permissions; require human confirmation for write operations
• Keep complete logs that can trace every operation
• Have key conclusions independently reviewed by tools or humans

Boundaries to Keep in Mind

A single experimental run is not enough to represent stable capability. The same task should be run repeatedly, with mean and variation reported. Cost comparisons should use “cost per completed milestone” as the metric, with fixed time budgets and tool permissions. In high-risk areas such as security vulnerabilities, biological information, and robot control, automated output should be treated only as supporting evidence; final judgments still need to be made by personnel with relevant expertise.