LLM Evaluation Methodology and Computer-Use Agent Practices

1. Background: Escaping the Model Selection Dilemma

Over the past two years, large language models have largely evolved along two parallel tracks: a general-purpose reasoning track targeting conversational, analytical, and writing tasks, and a specialized coding track deeply optimized for software engineering. Each track has its own focus, but together they create practical headaches for engineering teams:

• High selection cost: For every requirement, you first have to decide which model to call—the routing logic itself becomes a new source of complexity.
• Inconsistent experience: Different model tracks differ in capability boundaries, output styles, and error patterns, making them difficult to unify at the product layer.
• Capability silos: Coding models excel at tool calling and structured output, while reasoning models excel at long-horizon planning, and neither can easily borrow the other’s strengths.

The industry’s emerging answer is the unified model: merging general reasoning and coding capabilities into a single system that covers different scenarios through different delivery forms (chat products, APIs, software engineering platforms) rather than stitching multiple models together. For developers, interfaces become more convergent and the cognitive burden shrinks; for platform providers, capabilities can be transferred and reused within the same foundation—tool-use skills gained from code training can feed back into desktop control, and long-horizon reasoning can feed back into task planning.

2. Three Key Capabilities in the Unified Model Era

2.1 Interactive Reasoning: Turning the “Thought Process” into an Intervenable Dialogue

Traditional reasoning models tend to “think everything through before answering,” leaving users to accept the result or retry. A new generation of interactive reasoning allows users to insert feedback mid-thought and redirect the model—for example, adding constraints while it is working through a solution. The engineering value is clear:

• Lower rework rates: Directional errors are intercepted early, instead of being overturned after a long output has already been produced.
• More natural human-AI collaboration: Reasoning shifts from “black-box waiting” to “whiteboard co-creation.”

2.2 Long Context: From “Reading Documents” to “Reading Repositories”

Million-token context windows allow models to load medium-to-large code repositories, complete legal case files, or quarterly business data directly into a single session and perform cross-file correlation analysis. Combined with tiered speed modes (trading a small amount of quality for higher throughput and lower cost), teams can call the model in layers: full mode for complex planning, fast mode for bulk lightweight tasks.

2.3 Computer Use: From “Dialog Box” to “Operations Console”

This is the capability leap that has drawn the most attention over the past year: models directly operating real desktops and browsers to complete tasks. It is covered in detail below.

3. How to Evaluate a Large Model Scientifically

The more capable models become, the easier it is for benchmark scores to become distorted. A credible evaluation system must answer at least three questions: what to test, who scores, and how to quantify hallucinations.

3.1 What to Test: From Static Question Banks to Real Deliverables

Static benchmarks (multiple-choice questions, fixed coding problems) face two chronic problems: data contamination and capability saturation. Frontier models now approach perfect scores on most classic benchmarks, and discriminative power has dropped sharply. A more effective approach is building real work task sets:

• Cover pillar industries and typical occupations with high economic contribution—for example, sales presentations, financial modeling, legal analysis, staff scheduling, and manufacturing drawings.
• Use “deliverables” as the evaluation unit rather than single-point Q&A: what gets scored is a slide deck ready to use or an auditable financial model, not a single correct answer.
• Keep task sources close to real business workflows and rotate them regularly to prevent targeted overfitting.

Recent public third-party evaluations have widely adopted this approach: task sets composed of dozens of occupations and hundreds of real tasks, followed by structured scoring of the outputs. The conclusions of such evaluations tend to be far closer to the essential question—“is the model actually good at the job?”—than traditional benchmark scores.

3.2 Who Scores: Multi-Judge Cross-Scoring Plus Human Calibration

A single judge model or a single human evaluator carries systematic bias. A more robust approach is multi-judge cross-scoring:

# Multi-judge cross-scoring workflow (illustrative)
def evaluate(model, task_set, judges, human_panel):
    records = []
    for task in task_set:
        deliverable = model.run(task.prompt, tools=task.tools)
        machine_scores = [j.score(task, deliverable) for j in judges]  # Multiple judge models
        human_score = human_panel.sample_review(task, deliverable)     # Human spot-check calibration
        records.append({
            "task": task.id,
            "scores": machine_scores + [human_score],
            "preference_rate": pairwise_win_rate(deliverable, task.baseline),
        })
    return aggregate(records)

Three points matter here. Multi-judge cross-scoring: different judge models have different bias directions, and taking the consensus interval after cross-scoring significantly reduces noise such as “a judge preferring a particular writing style.” Human calibration: sample and re-check machine scores, and continuously calibrate the judge models, to avoid an overfitting pattern in which “the model panders to the judges.” Preference-rate comparison: have human evaluators blindly choose between old and new solutions on the same task; the resulting preference rate carries more business meaning than absolute scores.

3.3 Quantifying Hallucinations: Two Metrics Are Both Indispensable

Hallucination evaluation cannot rest on an overall “feels credible” impression. It should be split into two independent metrics:

• Per-claim error rate: Decompose a response into atomic factual claims and verify them one by one, measuring “how many errors occur per ten statements.”
• Whole-response error rate: A response counts as erroneous if it contains at least one mistake, measuring “whether a deliverable can be used directly.”

The two metrics reveal different problems: the former reflects the model’s factual foundation, while the latter directly determines deliverable usability. Public evaluations show that through retrieval augmentation, citation grounding, and training-side alignment, new-generation models have improved both metrics by double-digit percentages compared with the previous generation. In engineering practice, it is advisable to add a verification pipeline on the business side as well: key claims are required to carry citation sources, and claims without sources are downweighted or flagged.

3.4 Four Engineering Baselines for Evaluation

• Reproducibility: Fix sampling parameters and version the task sets and prompts so that any evaluation can be replayed.
• Contamination resistance: Keep task sets confidential and rotate them; treat public benchmarks as reference only, never as conclusions.
• Statistical significance: The sample size must support the conclusion; report confidence intervals rather than single-point scores.
• Cost accounting: Factor token consumption and latency into scoring to avoid “quality at any cost.”

4. Computer-Use Agents: Principles, Safety, and Evaluation

4.1 Capability Architecture: A Perception–Planning–Execution–Verification Loop

Computer use refers to models understanding screen content via screenshots and then generating mouse and keyboard commands to operate a real desktop, or controlling web pages through browser automation interfaces, forming a closed task loop:

# Main loop of a computer-use agent (illustrative)
goal = "Export this month's report from the business system"
policy = ConfirmPolicy(sensitive=["payment", "delete", "external_send"])
while not done:
    screen = desktop.screenshot()                  # Perception: pixel-level screenshot
    elements = accessibility_tree(desktop)         # Perception: control tree for precise localization
    action = planner.next(goal, screen, elements)  # Planning: decide the next step
    if action.is_sensitive and not policy.approve(action):
        break                                      # Safety: human confirmation for sensitive actions
    desktop.execute(action)                        # Execution: mouse/keyboard/browser commands
    done = verifier.check(desktop.screenshot(), goal)  # Verification: goal achieved?

Key technical points include: dual-channel perception—screenshots provide visual semantics while the accessibility tree provides precise localization, and the two complement each other; constrained action space—only atomic actions such as click, type, and scroll are exposed, with complex operations composed by the model, which makes actions easy to audit; step-by-step verification—a screenshot confirms the state after each step, avoiding cascading errors in which the agent “keeps clicking after a misclick.”

4.2 Safety Mechanisms: Making Confirmation Policy a First-Class Citizen

Computer use inherently involves permission risks, and mature implementations front-load safety policy: tiered confirmation (low-risk actions run automatically, while sensitive actions such as payments, deletions, and external sends require explicit human confirmation), sandbox isolation (run in a controlled environment with restricted network and filesystem access), and full-chain audit (every screenshot and action is logged, enabling replay and accountability).

4.3 How to Evaluate: Efficiency Beyond Success Rate

Mainstream evaluation uses end-to-end real tasks, with core metrics including first-attempt success rate (“one-shot” ability, independent of luck), success rate within N attempts (convergence ability when retries are allowed), and session duration and token consumption (which determine cost per task). Industry observations show that leading models can match or even slightly exceed human baselines on standardized desktop-control tests; in publicly reported real-world portal automation cases, agents achieved first-attempt success rates above 90% across tens of thousands of business websites, near-full coverage within three attempts, several-fold faster sessions, and token consumption cut by more than half. These figures suggest the capability has moved from “demo” into the “usable” range—but production rollout should still be tiered according to operational risk.

4.4 Deployment Path: Three Steps from Read-Only to Transactional

1. Read-only scenarios: information lookup, cross-system data reconciliation, screenshot archiving.
2. Form scenarios: data entry, report export, and process initiation, paired with human confirmation.
3. Transactional scenarios: gradually opened only under complete audit and rollback mechanisms.

5. Enterprise Selection and Deployment Recommendations

Based on the trends above, here are four actionable recommendations for engineering teams:

1. Build your own evaluation set: Extract 50–200 real tasks from historical business tickets, covering high-frequency scenarios, and use them as a “private benchmark” for model selection and regression testing.
2. Shadow-traffic gray release: Run new models on shadow traffic first, compare error rates, latency, and cost against the production model, and then decide on the traffic-shift ratio.
3. Favor a unified foundation: Prefer unified models with converged capabilities to reduce routing complexity, while keeping dedicated channels as a fallback for critical scenarios.
4. Leverage cloud toolchains: Cloud platforms such as Baidu AI Cloud typically provide general-purpose capabilities including model services, evaluation tools, monitoring and alerting, and logging services. You can reuse this infrastructure directly and focus your own effort on business task sets and safety policies.

6. Conclusion

The competition in model capabilities is shifting from “single-point benchmark scores” to “real-world deliverables”: unified models dissolve the selection dilemma, interactive reasoning reshapes human-AI collaboration, and computer use brings models from the dialog box into the operations console. For engineers, what matters more than chasing any specific model is building a credible evaluation system and a safe deployment path—so that every model upgrade can be measured, verified, and audited.