Evaluating Open-Weight Agent Models: From Execution Reliability to Controllability

1. Background: Why Agent Tasks Need More Granular Evaluation

As large models are deployed in automation, command execution, code generation, and conversational interaction, relying solely on static Q&A accuracy no longer reflects a model’s real capabilities. In agent tasks especially, a model must not only generate answers but also adjust execution paths based on environment feedback, invoke tools, handle failures, and keep working toward multi-step goals. As a result, the industry has gradually introduced evaluation methods that are closer to real usage environments, observing overall model performance through long-horizon task sessions and user feedback.

Against this backdrop, the significance of open-weight models lies in the fact that developers can deploy, debug, and fine-tune them independently to better fit specific business scenarios. But open weights alone do not equal sufficient capability. Credible evaluation metrics are required to judge whether a model is ready for production. This article analyzes an agent model capability evaluation framework across dimensions including execution reliability, user satisfaction, controllability, and confirmation success.

2. Breaking Down Evaluation Dimensions: Reading Capability Boundaries from Numbers

Evaluation data typically comes from real user sessions rather than offline benchmarks. The advantage of this approach is that tasks are more complex and feedback is more realistic, exposing weak points in a model’s long-term execution. Taking a set of public evaluation results as an example, an open-weight model achieved a net improvement rate of +5.1% on an agent leaderboard, compared to only +0.9% for the previous generation. The net improvement rate reflects the proportion of users who, after comparing experiences, considered the new model a substantive improvement over the old one. Though simple, this metric can comprehensively reflect overall progress across multiple task types.

Beyond the overall metric, granular dimensions also warrant attention. For example, in Shell command execution reliability, the new model reached +7.9%, while the previous generation was at -4.5%. This indicates that the older model had a clear weakness in command execution, potentially causing task interruptions or erroneous operations. For scenarios requiring automated operations or batch script execution, improvement in this metric is especially critical.

User satisfaction metrics also deserve attention. The new model achieved a +7.7% praise-to-complaint differential, far above the +1.3% of the previous generation. This shows that users care not only about whether a task is completed, but also about the interaction experience—whether the model accurately understands instructions, provides clear explanations, and proactively recovers after failures.

3. Controllability: A Core Capability of Agent Models

The controllability metric measures whether a model can adjust its execution direction in a timely manner based on user feedback. In real usage, users often provide corrections mid-task, such as “don’t delete temporary files” or “try a different way to process the data.” If the model cannot effectively absorb this feedback, it will repeat errors or deviate from the goal.

The new model achieved +6.5% in controllability, ranking first among open-weight models. This means the model can maintain goal consistency across multi-turn interactions while possessing the ability to revise its strategy based on feedback. For developers, this capability can reduce the frequency of manual intervention and improve the stability of automated workflows.

To improve controllability, models typically need to incorporate multi-turn interaction data and feedback alignment mechanisms during training. For example, training samples can include execution trajectories after user corrections, teaching the model to abandon its original plan and re-plan steps upon receiving new instructions. This capability cannot be measured through simple Q&A benchmarks—it can only be observed through long-horizon task sessions.

4. Confirmation Success and Task Closure

The confirmation success metric reflects whether a model, after completing a series of operations, can provide a clear and verifiable success result. The new model scored +3.9%, slightly higher than the previous generation. Although the numerical improvement is modest, task closure capability is critical in real scenarios. For example, in an automated deployment pipeline, the model must not only execute installation commands but also verify whether the service started correctly, whether ports are listening, and whether logs are error-free. Without a confirmation step, users may assume the task is complete when the system is actually still in an abnormal state.

Therefore, when evaluating agent models, developers should focus on testing whether the model has the habit of proactive verification and result confirmation. This can be done by constructing tasks that include failure branches and observing whether the model checks status and reports results at key nodes.

5. How to Apply These Metrics to Model Selection

For enterprise developers, choosing an agent model should not rely solely on leaderboard totals. Instead, they should consider their own business characteristics and focus on the following aspects:

1. High proportion of execution tasks: Prioritize command execution reliability and confirmation success to avoid environment damage from erroneous operations.
2. Many multi-turn interaction scenarios: Focus on controllability and user satisfaction to ensure the model can understand feedback and adjust behavior.
3. Strong need for open deployment: Choose models with open weights that support local deployment, facilitating subsequent fine-tuning and security compliance management.
4. High requirements for long-term stability: Reference net improvement rates and real session scale to avoid being misled by high scores from a small sample.

During technical implementation, models can be deployed on container platforms, with logging services recording execution trajectories, failure nodes, and user feedback for each task session to form an internal evaluation loop. Combined with monitoring and alerting capabilities, when a model repeatedly exhibits low-reliability operations, the system can promptly switch to a backup strategy or escalate to human handling.

6. Code Example: Designing an Internal Evaluation Script

Below is a general-purpose agent task evaluation script framework for testing model controllability and confirmation success in a self-hosted environment. This example illustrates flow logic only and does not depend on any specific platform API.

from typing import List, Dict
def run_agent_task(model, task_steps: List[str], feedback: Dict[int, str]):
    """
    Simulate a multi-step agent task and inject user feedback at specified steps.
    Return whether each step was executed as expected and whether the direction
    was adjusted based on feedback.
    """
    results = []
    current_plan = task_steps.copy()
    for step_idx, step in enumerate(current_plan):
        response = model.execute(step)
        if step_idx in feedback:
            # User provides a correction
            adjusted = model.adjust_plan(feedback[step_idx])
            results.append({
                "step": step_idx,
                "adjusted": adjusted,
                "matched_feedback": feedback[step_idx] in response
            })
        else:
            results.append({
                "step": step_idx,
                "adjusted": False,
                "matched_feedback": None
            })
    return results
def evaluate_confirmation(model, verification_prompt: str) -> bool:
    """Check whether the model proactively performs result confirmation."""
    output = model.execute(verification_prompt)
    # General check: does the output contain status verification or validation descriptions
    confirm_keywords = ["check", "verify", "confirm", "status", "log"]
    return any(kw in output for kw in confirm_keywords)

The core value of the above code lies in translating abstract evaluation metrics into executable test workflows. Developers can extend step definitions, feedback injection methods, and confirmation criteria based on their own business needs to establish an internal model capability baseline.

7. Summary and Outlook

Open-weight agent models are evolving from “can converse” to “can reliably execute complex tasks.” The evaluation system is also shifting from a single accuracy metric to multi-dimensional capability observation, including execution reliability, user satisfaction, controllability, and confirmation success. For developers, understanding the engineering implications behind these metrics is more valuable than simply chasing leaderboard rankings.

In the future, as agents are applied more deeply in automated operations, data analysis, and code generation, evaluation methods will become further refined. For example, constraint enforcement for security compliance, link tracing across tool invocations, and memory management for long-horizon tasks will all become important directions for measuring model practicality. Developers should continuously monitor real user session evaluation data and combine it with internal test environments to form their own selection criteria, so they can truly leverage the advantages of open-weight models.