Background: Agent Capabilities Are the New Battleground for LLMs

Large language model applications are evolving from single-turn, question-and-answer interactions into multi-step agent workflows that follow a perceive–plan–act–feedback loop. An agent must call tools, run commands, read and write files, operate in its environment, and stay aligned with its goal across dozens of interaction turns. This places fundamentally different demands on the model than pure conversation: it is not enough to “say the right thing” — the agent has to actually get things done.

At the same time, open-weight models (models whose parameters are publicly released so users can download and deploy them themselves) have become a major industry trend. For enterprises, open weights mean:

• Data security and compliance: inference can run in private environments, keeping sensitive data in-house;
• Predictable costs: compute can be planned around your own workload, avoiding long-term pay-per-call fees;
• Customizability: the model can be further trained or preference-aligned on business data;
• Portability: no lock-in to a single provider, preserving room for replacement and gradual migration.

Open weights, however, create a practical problem: with no official service to fall back on, enterprises must judge for themselves whether a model actually works on their tasks. This makes a solid evaluation framework more important than ever.

Why Static Benchmarks Fall Short for Agent Evaluation

Traditional static benchmarks (multiple-choice questions, fixed coding problems, and so on) show three limitations in agent scenarios:

1. Data contamination risk: public questions may leak into training corpora, inflating scores and masking true generalization ability;
2. Distorted task formats: real agent tasks tend to be long-running, multi-constraint, and dynamically changing, while static tests are mostly single-turn, closed, one-shot problems;
3. No interaction dimension: the core difficulty of agents is “continuously adjusting based on environment and user feedback,” which static benchmarks cannot measure.

The industry has therefore converged on a consensus: agent evaluation should be built on real, long-horizon task sessions. A typical approach:

• Collect large volumes of real user task sessions covering coding, office work, Q&A, and other scenarios;
• Run the candidate model and a baseline model against the same real traffic;
• Use statistical methods to aggregate the share of sessions that are “significantly better than baseline” and “significantly worse than baseline,” yielding a Net Improvement Rate:

Net Improvement Rate = share of sessions significantly better than baseline − share of sessions significantly worse than baseline

A net improvement rate above zero means the new model is better than the baseline overall on the real task distribution; the absolute value, together with sample size, determines confidence in the conclusion. This approach costs more, but it correlates with production performance far better than offline leaderboards.

Core Metrics for Agent Evaluation

Around real sessions, the industry typically builds its metric system along the following dimensions:

1. Execution Reliability

Definition: the proportion of tool calls (especially shell commands and code execution) produced by the model that succeed on the first attempt in the target environment without causing destructive side effects.

Agents often fail not because they “think wrong” but because they “act wrong”: misspelled commands, incompatible arguments, accidentally deleted files, operations on nonexistent paths, and so on. Execution reliability determines whether the agent’s engineering scaffolding is solid. Key measurement points include:

• Log the return code, stderr output, and duration of every tool call;
• Flag destructive commands (such as recursive deletes and forced overwrites) separately;
• Distinguish “first-attempt success” from “success after retries” — they reflect different levels of problems.

2. User Satisfaction (Praise-to-Complaint Ratio)

Definition: extract “praise” and “complaint” signals from explicit and implicit feedback within sessions, and compute their relative strength.

Explicit feedback includes likes and ratings; implicit feedback includes complaint-style statements like “still wrong” or “you misunderstood me,” as well as confirmation-style statements like “great, exactly what I wanted.” This metric reflects the end user’s subjective experience and complements execution reliability: a task where every command executed successfully but the direction drifted will still draw complaints.

3. Controllability

Definition: the model’s ability to adjust its execution direction based on mid-task user feedback and converge on the expected goal.

Agent tasks often change mid-flight: a user may only say at turn 5, “don’t use recursion, switch to iteration.” Controllability measures how many turns the model needs to converge after a correction is issued, and whether it repeatedly regresses to the old approach. A poorly controllable model dramatically increases human supervision costs — often more damaging in production than single-point accuracy.

4. Confirmation Success Rate

Definition: the proportion of tasks that end with the user explicitly confirming “the task is complete and the result is usable.”

This deliberately differs from “model self-assessed completion.” A common agent failure mode is “declaring victory too early” — claiming a bug is fixed before the code even runs. Anchoring on user confirmation suppresses the model’s self-flattering bias.

5. Scenario-Specific Evaluation

The same model can perform very differently across work, coding, and chat categories. Enterprises should focus on the subsets that match their own task profiles rather than overall rankings alone.

Building an In-House Agent Evaluation Pipeline

Step 1: Retain session traces in a structured format. Every session should record a complete trace: user input, model plan, tool calls and results, intermediate artifacts, final response, and user feedback. A sample structure:

{
  "session_id": "s-20240601-001",
  "turns": [
    {
      "role": "user",
      "text": "Aggregate the errors in the logs by type"
    },
    {
      "role": "agent",
      "plan": "Parse logs → extract error lines → aggregate counts",
      "tool_calls": [
        {
          "type": "shell",
          "command": "grep -i error app.log | sort | uniq -c",
          "exit_code": 0,
          "duration_ms": 120
        }
      ],
      "final_answer": "3 error types in total, distributed as follows..."
    }
  ],
  "feedback": { "type": "praise", "text": "Exactly what I wanted" }
}

Step 2: Build a metric computation layer. The following pseudocode shows how the core metrics are computed:

def evaluate(sessions):
    reliability, confirmed, steer_rounds = [], 0, []
    for s in sessions:
        cmds = [t for t in s.tool_calls if t.type == "shell"]
        if cmds:
            ok = [t for t in cmds if t.exit_code == 0 and not t.destructive]
            reliability.append(len(ok) / len(cmds))
        confirmed += 1 if s.user_confirmed_success else 0
        if s.corrections:  # user corrections exist
            steer_rounds.append(s.rounds_to_converge)
    return {
        "bash_reliability": mean(reliability),
        "confirmed_success": confirmed / len(sessions),
        "avg_steer_rounds": mean(steer_rounds),
    }

Step 3: Run offline replay and online comparison in parallel. Offline, replay historical sessions to quickly screen candidate models; online, run a small-traffic A/B test to validate the net improvement rate on the real distribution, then roll out gradually once targets are met.

Step 4: Safety guardrails. Evaluation and runtime environments must be sandboxed and isolated: restrict file system and network access, require a second confirmation for high-risk commands, and keep complete audit logs.

Managed Agent Services: The Last Mile from Weights to Usable Capability

Open weights solve “getting the model”; managed agent services solve “using it well.” A general-purpose managed agent service typically includes these capability layers:

• Model inference serving: weight hosting, elastic scaling, batch processing, and streaming output;
• Tool runtime: sandboxed code execution, command execution, and external call environments;
• Orchestration and memory: task planning, multi-turn context management, and long-term memory storage;
• Observability: session tracing, metric dashboards, and anomaly alerting, integrated with the evaluation loop;
• Security and governance: access control, data isolation, and content and behavior auditing.

Enterprises can assemble these layers from general-purpose cloud capabilities (such as container platforms, object storage, logging services, monitoring and alerting, and model serving platforms). Cloud platforms such as Baidu AI Cloud typically offer this kind of combination of general-purpose capabilities: enterprises can choose managed inference to reduce operational burden, or download open weights and deploy them in their own environments, then use a unified evaluation pipeline to keep quality consistent across both paths.

Three recommendations for putting this into practice:

1. Evaluate first: before any model enters production, run it through your in-house evaluation pipeline and let the net improvement rate and scenario-specific metrics speak;
2. Version-control the weights: bring model weights, system prompts, and tool lists under version management so any change can be rolled back and compared;
3. Close the feedback loop: stream live user feedback back into the evaluation dataset so that model iterations and evaluation benchmarks evolve together.

Conclusion

In the agent era, “is the model good?” increasingly depends on “is it stable in real tasks?” Execution reliability, user satisfaction, controllability, and confirmation success rate break the vague “feel” of model quality into engineering objects that can be measured, attributed, and improved. For developers, real competitiveness lies not in chasing single leaderboard spots but in building a complete loop — real session collection, multi-dimensional metric evaluation, staged rollout, and feedback reflow — and, through managed services or self-hosting, steadily converting the potential of open-weight models into business value.