From Open Weights to Agent Services: Evaluation Systems and Implementation Practices

1. Background: When “Open Weights” Become the Industry Norm

In recent years, the open-source trend in the large model community has accelerated significantly. After completing training, leading teams often publish model weight files to model hosting platforms, allowing anyone to download, locally deploy, and further develop them. It should be clarified that “open weights” is not entirely equivalent to strict “open source”—while the weights are downloadable, the training data and code are typically not included, and usage is bound by license terms (e.g., whether commercial use or derivative training is permitted).

For enterprises and technical teams, the value of releasing open weights primarily manifests in three areas:

• Private Deployment: Models can run on self-built infrastructure or dedicated compute environments, ensuring sensitive data remains on-premises to meet compliance requirements.
• Deep Customization: Based on open weights, teams can perform domain-specific fine-tuning (e.g., SFT, LoRA) to adapt general models to vertical business needs.
• Autonomous Evaluation: Instead of relying on vendor marketing, teams can conduct regression testing and horizontal comparisons using their own business data.

However, opening weights is just the starting point. The real challenges lie in two questions: how to objectively evaluate a model’s true capabilities in “agentic” scenarios, and how to transform a weight file into a stable, usable online service. These two points are the focus of this article.

2. Agent Evaluation: From “Benchmarking” to “Real-World Task Sessions”

2.1 Limitations of Static Benchmarks

Traditional evaluations heavily rely on static datasets (e.g., multiple-choice questions, code generation, mathematical reasoning). While these benchmarks are easy to reproduce, they have two inherent flaws:

1. Data Contamination: The test questions may have been mixed into the model’s pre-training corpus, leading to artificially inflated scores.
2. Distribution Shift: Real users propose complex tasks that are long-cycle, multi-turn, and involve extensive tool usage, which differ vastly from static questions.

Therefore, the industry has recently shifted toward dynamic evaluations based on real user sessions. This involves anonymously collecting tens of thousands of real agentic task sessions, having participating models execute tasks from the same starting line, and scoring them combined with explicit user feedback. Because the samples come from a real distribution and scale to tens of thousands, the conclusions offer significantly higher reference value for production environments. Evaluating an “agent” rather than a “chatbot” essentially means evaluating the model’s ability to act within a tool environment, not just its language proficiency.

2.2 Four Core Metrics Worth Noting

In agentic scenarios, the following four categories of metrics form a relatively complete capability profile:

(1) Shell Command Execution Reliability
Agents completing engineering tasks rely heavily on the terminal: installing dependencies, running tests, and checking logs all require commands. This metric measures the model’s “first-pass execution success rate” for issued commands, as well as its self-repair ability after a failure. Notably, this metric can be negative—if erroneous commands and destructive operations outnumber successful corrections, a negative net value indicates the model is not yet reliable in a real terminal environment. Moving from a “negative value” to a “significant positive value” often signifies a generational upgrade for the model at the tool-calling level, making it the single most important metric for engineering teams to prioritize.

(2) Praise vs. Complaint Ratio (User Positive/Negative Feedback Ratio)
This is the ratio of explicit positive feedback (likes, thanks) to negative feedback (complaints, corrections) given by users during a session. It does not measure single-point capabilities but serves as an aggregate signal of the end-to-end experience, exposing comprehensive issues such as “answering the wrong question,” “over-execution,” or “messy formatting.”

(3) Controllability
This measures the model’s ability to adjust its execution direction based on mid-task user feedback. Real tasks rarely succeed on the first try; users might say “wrong direction,” “try a different implementation,” or “don’t touch this file yet.” A model with poor controllability will stubbornly proceed along the original path, resulting in a terrible multi-turn experience. A highly controllable model, on the other hand, can converge to the user’s true intent in the fewest turns possible.

(4) Confirmation Success Rate
The proportion of sessions where the user explicitly confirms that the “task is completed and the result is correct.” This is the hardest metric closest to “delivery quality”—no matter how smooth the process is, it doesn’t count as a success unless the user confirms it.

2.3 How to Interpret the “Net Improvement Rate”

In comparative evaluations, the common “Net Improvement Rate” metric can be approximately understood as: Net Improvement Rate ≈ Win Session Ratio − Loss Session Ratio. When interpreting this, keep three key points in mind:

• The sign is more important than the absolute value: Moving from near 0 to a clearly positive value indicates the model has transitioned from “comparable to the baseline” to “better in most scenarios.”
• Look at the sub-metrics: A total net improvement coupled with a decline in command reliability might mean the model is “more pleasing in chat but sloppier in execution,” requiring caution for engineering scenario selection.
• Consider sample size and confidence intervals: A ranking based on tens of thousands of sessions is significantly more reliable than one based on a few hundred.

A minimal snippet of code can reproduce this statistical caliber in an in-house evaluation:

def net_improvement(wins: int, losses: int) -> float:
    """Net Improvement Rate: Win ratio - Loss ratio, ties are not in the denominator"""
    total = wins + losses
    return (wins - losses) / total if total else 0.0
def shell_reliability(sessions: list[dict]) -> float:
    """Command Execution Reliability: Ratio of sessions succeeding on first execution"""
    if not sessions:
        return 0.0
    ok = sum(1 for s in sessions if s.get("first_pass_exec"))
    return ok / len(sessions)

3. From Weight Files to Managed Agent Services

Evaluation solves the question of “is it worth using,” while implementation solves “how to put it into use.” Transforming an open weight file into a usable agent service for a team typically requires building four layers.

3.1 Inference Serving: Turning Weights into APIs

• Deployment Form: Mainstream inference frameworks generally support tensor parallelism, continuous batching, and general low-bit quantization for open weights to improve throughput under limited VRAM.
• Elastic Scaling: Container platforms can scale based on load, automatically scaling down during off-peak nighttime hours to reduce costs.
• Typical Trade-off: Private deployment offers high control but comes with steep operational costs. Alternatively, using a cloud provider’s managed model services and supporting capabilities (such as the model inference services, container platforms, logging services, and monitoring alerts provided by Baidu AI Cloud) allows for pay-as-you-go billing and eliminates self-managed GPU operations. The compromise path for most teams is to use managed services for rapid validation during the evaluation phase, and decide whether to privatize after scaling up.

3.2 Agent Orchestration Layer: Plan, Execute, Reflect

Weights are just the “brain”; an agent also needs a “workflow.” A minimal but complete agent loop consists of three steps:

# Framework-agnostic minimal agent loop illustration
history = [system_prompt, user_task]
while not task_done:
    action = llm(history)                # 1. Plan: Decide the next action
    if action.type == "shell":
        result = sandbox.run(action.cmd) # 2. Execute: Run command in an isolated sandbox
        history.append(observe(result))  # 3. Reflect: Feed execution result back to the model
    elif action.type == "ask_user":
        history.append(user_reply())     # Controllability: Actively ask when uncertain
    elif action.type == "finish":
        task_done = True

Although this loop seems simple, it covers the key components of the evaluation system: execution result feedback corresponds to reliability repair, active inquiry corresponds to controllability, and the end condition corresponds to the confirmation success rate.

3.3 Tools and Sandboxes: Security is the Baseline

Command-execution agents must prioritize security:

• Least Privilege: Run inside containers, restricting network egress and filesystem mount scopes.
• Dangerous Command Interception: Set manual confirmation thresholds for commands involving deletion, privilege escalation, or data exfiltration.
• End-to-End Audit: Log every command and model output to support session replay and post-incident accountability.

3.4 Feedback Loop: Making the Service Better Over Time

The core value-add of a managed agent service often lies not in “how strong the model is,” but in the closed-loop construction: session-level scoring and user feedback collection → regular regression evaluation → targeted fine-tuning or prompt iteration → canary release. This mechanism naturally connects with the evaluation metrics in Section 2: the praise-to-complaint ratio and confirmation success rate can serve directly as the North Star metrics for regression testing.

4. Implementation Advice for Enterprises and Developers

1. Select Models Based on Real Tasks, Not Just Leaderboards: Prioritize comparative data based on real sessions, and conduct small-scale retests using your own business samples.
2. Invest in Phases: Start with managed services before moving to private deployment; start with single-point scenarios (like code review or ops inspection) before full workflows.
3. Integrate Evaluation into CI: Incorporate agent task success rates and command reliability into continuous integration; run regressions for every change to the model or prompt.
4. Security First: Sandbox isolation, least-privilege access, and audit logs are all indispensable.
5. Pay Attention to Licenses: Commercial terms and derivative model constraints should be confirmed jointly by legal and engineering teams during the selection phase.

5. Conclusion

Open weights have significantly lowered the barrier to “accessing advanced models,” but what determines implementation success is the scientific rigor of the evaluation system and the completeness of the service engineering. By using real task sessions as the evaluation foundation and “plan-execute-reflect-feedback” as the service framework, teams can truly transform a weight file into a dependable, iterable, and auditable agent capability.