Robot Control Foundation Model Evaluation: Benchmarks and Cost-Performance

From “Can It Demo?” to “Can It Deploy?”: Core Questions in Robot Control Evaluation

When foundation models move from text generation, code assistance, and multimodal understanding toward robot control, developers and enterprise users usually focus on two practical questions rather than a single impressive demo: first, whether the model can reliably complete real or near-real manipulation tasks; second, whether the computation, latency, deployment, and operating costs required to complete those tasks are acceptable.

Robot control is not a simple image recognition or question answering problem. It is a closed-loop system. The system must continuously receive camera images, sensor states, task instructions, and feedback information, then output action decisions and continuously correct behavior as the environment changes. Therefore, evaluating a robot control model cannot rely only on whether a single demonstration succeeds. It must also consider stability in complex tasks, abnormal states, continuous operation, and long-running conditions.

From this perspective, robot control model evaluation should answer at least the following questions:

1. Can the model understand objects, goals, and constraints in the current scene?
2. Can the model decompose natural language or task instructions into executable steps?
3. Can the model generate smooth, safe, and effective action trajectories?
4. Can the model recover from abnormalities such as grasp failure, collision, timeout, or occlusion?
5. Do the model’s inference latency, compute consumption, and deployment costs meet business requirements?

Only by combining these dimensions can one determine whether a model has true engineering value.

Why Simulation Testing Has Become an Important Entry Point for Robot Control Evaluation

In robot control, simulation testing is important because real robot experiments are expensive, risky, and slow in sample acquisition. Physical robots may involve mechanical wear, sensor calibration, force-control safety, and site restrictions, making them difficult to rerun as frequently as software tests.

By contrast, simulation environments can construct task scenarios in batches and quickly generate different object positions, lighting conditions, interference factors, and manipulation goals. For example, the same dual-arm robot task can be tested repeatedly in simulation, recording trajectory, duration, success rate, and failure causes for each run. For foundation models, simulation can also unify visual input, language instructions, action output, and reward feedback within the same evaluation framework, making cross-comparison easier.

However, simulation testing has prerequisites. If different models run under different execution frameworks, action spaces, sampling frequencies, collision rules, or maximum inference intensity, score differences may come from differences in the test environment rather than from differences in model capability. Therefore, more rigorous evaluations usually emphasize a “unified test framework” and “unified execution conditions.” Only when models use the same observation interface, action constraints, timeout rules, and inference configuration can their scores be compared meaningfully.

This is also why many robot control evaluations emphasize average scores rather than showing a few demonstration videos. Average scores can cover more task types and reduce randomness, while demonstration videos are better for showing capability boundaries but are not sufficient to represent overall stability.

What Average Scores Can and Cannot Show

Average score is a common metric in robot control evaluation, but it must be understood correctly. It is often not simply “number of completed tasks divided by total number of tasks,” but may be a weighted result of completion across multiple subtasks.

For example, a task may be divided into stages such as grasping, moving, placing, aligning, twisting, and drawing. A system may complete most stages but fail at the last step. It may also encounter partial collisions, but because safety mechanisms take effect, the result may be recorded as partially completed. As a result, the final score may be calculated from fully successful, partially successful, failed, timed-out, and abnormally terminated outcomes.

Therefore, if an evaluation shows that the top model has a relatively high average score, this does not mean all tasks are fully successful. It more likely means that the model performs relatively well across the overall task set. In other words, a high score can show that the model has a relatively high average completion rate across multiple tasks, but it does not directly equal “every task succeeds stably.”

Another issue to consider is who conducts the evaluation. Even if the evaluation is organized by the model provider, uses a unified test framework, and has reasonable task design, one should still examine whether the provider publicly shares per-task results, failure causes, random seeds, repetition counts, and confidence intervals. Without such details, outside observers cannot easily determine whether the results are broadly applicable.

When interpreting robot control model scores, it is therefore advisable to focus on the following types of information:

1. Task set composition: Is it simple grasping, or complex long-horizon manipulation?
2. Score calculation method: Is it binary success, or weighted completion across stages?
3. Repetition: Is the same model tested multiple times and averaged?
4. Failure distribution: Are failures mainly in perception, planning, or action execution?
5. Public per-task results: Can model weaknesses be located on specific tasks?
6. Internal or independent testing: Can results be aligned with third-party leaderboards or independent re-tests?

Only when this information is complete can scores become meaningful selection criteria.

Breaking Down Robot Control Model Capabilities

From a technical perspective, the capabilities of robot control foundation models can be broken down into perception, planning, control, recovery, and generalization. Each layer affects final task performance differently.

1. Perception and Scene Understanding

A robot must first “see” the environment. It must identify object categories, positions, poses, and graspable regions. It must also determine which objects are obstacles, which objects need to be moved, and which areas cannot be touched. For dual-arm robots, it must also understand the operable ranges of the left and right arms and avoid collisions between them.

Insufficient perception can lead to incorrect grasp points, object misclassification, or failed occlusion handling. For example, when a target object is partially occluded by other objects, the model needs to infer the full shape rather than relying only on visible local textures. This requires high capabilities in visual encoding, multimodal fusion, and spatial reasoning.

2. Task Planning and Step Decomposition

Robot control is not a single-step reaction but a multi-step task. For example, “put the block in the box,” “open the container,” and “use a tool to draw a pattern” all require the model to understand the goal first and then decompose it into a sequence of actions.

High-quality task planning usually has several characteristics:

1. It can recognize task dependencies, such as grasp first, then move, then place.
2. It can adjust plans based on the current state rather than mechanically executing a fixed script.
3. It can handle constraints, such as obstacle avoidance, force-control limits, and action order.
4. It can perform local replanning after failure instead of restarting the whole task.

If a model is only good at single-step action generation but weak at long-horizon planning, its stability in complex tasks will clearly decline.

3. Motion Control and Dual-Arm Coordination

Ultimately, a robot must complete actions through joints, end effectors, grippers, or other tools. Motion control quality directly affects task success rate.

Key indicators include:

1. Whether the trajectory is smooth or contains sudden changes.
2. Whether the grasping force is appropriate and does not damage objects.
3. Whether the placement position is accurate and avoids collisions.
4. Whether the two arms act in coordination without interfering with each other.
5. Whether jitter or delay occurs under high-frequency control.

For dual-arm robots, control difficulty is usually higher than for single-arm robots. The two arms must not only complete actions independently but also handle spatial relationships, coordinated grasping, load distribution, and collision avoidance.

4. Failure Recovery and Exception Handling

Robot tasks in real environments rarely succeed on the first attempt. Grasps may slip, objects may be pushed to the edge, cameras may be temporarily occluded, and tools may not be aligned. Whether the model has failure recovery capability is an important criterion for distinguishing “demonstration-level capability” from “engineering-level capability.”

Good failure recovery mechanisms include:

1. Detecting grasp failure and adjusting the gripper position.
2. Detecting object displacement and re-localizing the object.
3. Pausing or backing off when collision risk is detected.
4. Switching to a backup strategy after a timeout.
5. Stopping safely and requesting human intervention when the task cannot be completed.

A model without recovery mechanisms may bring higher maintenance costs in production environments even if its average score is high.

5. Generalization and Robustness

Robot control models must deal with object position changes, lighting changes, background changes, material changes, and slight mechanical deviations. If a model performs well only in fixed scenarios and lacks sufficient generalization, its practical deployment value is limited.

Generalization can usually be evaluated through the following methods:

1. Changing object colors, shapes, and materials.
2. Changing initial placement positions.
3. Adding distractors.
4. Adjusting camera angles.
5. Changing the robot’s initial pose.
6. Introducing random noise and slight delay.

If a model can remain stable under disturbances, it is closer to a deployable state.

6. Inference Intensity, Latency, and Cost

Many robot control evaluations use the maximum inference intensity because higher inference intensity often improves performance on complex tasks. However, engineering deployment cannot look at scores alone; latency and cost must also be considered.

High inference intensity may increase response time. In real-time control scenarios, excessive latency can cause delayed actions, unstable trajectories, or safety risks. Therefore, the inference strategy should match the task type: lightweight models or low inference intensity may be used for simple tasks, while higher inference intensity can be enabled for complex tasks.

Cost-Performance Is Not a Single Price, but Total Cost of Ownership

In robot control foundation model selection, “cost-performance” should not be understood simply as invocation price. It should be understood as the cost per effective task completed. Even if a model is cheap per inference, low success rate, frequent retries, and large amounts of human takeover can make the overall cost higher.

A more reasonable cost-performance framework should include the following indicators:

1. Task success rate: The proportion of fully completed tasks.
2. Partial completion rate: The proportion of tasks that complete major stages but not fully.
3. Stability: Whether results remain consistent across multiple runs.
4. Safety violation rate: Whether collisions, boundary violations, overload, or other risks occur.
5. Average latency: Time from receiving an instruction to outputting an action.
6. P95 latency: Whether tail latency remains controllable.
7. Unit success cost: Total resources consumed to complete one effective task.
8. Operating cost: Investment in monitoring, logging, replay, tuning, and human takeover.

A simplified evaluation function can express this idea:

def evaluate_robot_model(success_rate, safety_score, stability_score,
                         p95_latency, unit_cost, weights=None):
    if weights is None:
        weights = {
            "success": 0.45,
            "safety": 0.20,
            "stability": 0.15,
            "latency": 0.10,
            "cost": 0.10,
        }
    latency_score = 1 / (1 + p95_latency)
    cost_score = 1 / (1 + unit_cost)
    score = (
        weights["success"] * success_rate
        + weights["safety"] * safety_score
        + weights["stability"] * stability_score
        + weights["latency"] * latency_score
        + weights["cost"] * cost_score
    )
    return score

This example is not a fixed formula but an evaluation approach. Different businesses assign different weights to safety, latency, and cost. For example, in laboratory research, more attention may be given to task exploration capability; in industrial settings, safety, repeatability, and maintainability must be prioritized.

Further, unit successful task cost can be introduced:

Unit successful task cost = Total resource cost / Number of effective successful tasks

Total resource cost includes not only model inference cost, but also data collection, labeling, training, deployment, monitoring, exception handling, and human takeover. Only when total cost is included can one determine whether a model is truly suitable for large-scale use.

High Simulation Scores Do Not Mean Physical-Robot Readiness

Sim-to-real transfer has long been a difficult problem in robot control. Models that perform well in simulation may experience performance degradation on physical robots due to many factors.

Common differences include:

1. Camera calibration errors causing visual localization bias.
2. Lighting changes affecting object recognition.
3. Robotic arm dynamics errors causing trajectory deviations.
4. Differences between simulation and reality in object material, friction coefficient, and weight.
5. Insufficient grasp-force feedback from the gripper.
6. Sensor noise affecting state estimation.
7. Network latency or control-cycle jitter affecting real-time performance.

Therefore, even if simulation tests show a relatively high overall score for a model, physical robot validation is still necessary. Physical robot testing usually focuses more on the following issues:

1. What is the first-attempt success rate?
2. How stable is the system after ten or one hundred continuous runs?
3. Can it stop safely when abnormalities occur?
4. Does it require frequent human takeover?
5. Are mechanical wear and energy consumption acceptable?
6. Can it transfer across different workstations and lighting conditions?

From an engineering perspective, simulation testing is suitable for large-scale cross-comparison and regression validation, while physical robot testing is suitable for pre-deployment acceptance. They cannot replace each other.

What Physical-Robot Demonstration Tasks Show

Robot control model demonstrations often include tasks such as solving puzzle blocks, opening bottle caps, and drawing patterns. These tasks may look simple, but each evaluates different capabilities.

1. Puzzle Block Tasks

Puzzle block tasks usually test spatial reasoning, step planning, and object manipulation. The robot needs to understand the current state of the blocks, the target state, and the intermediate steps. The difficulty is not a single grasp, but long-horizon planning and state tracking.

If a model lacks stable memory and state judgment, it may forget the goal midway or execute steps in the wrong order.

2. Bottle Cap Opening Tasks

Bottle cap opening tasks focus on continuous motion control, force feedback, and tool use. The robot needs to determine the cap position, adjust the gripper angle, apply appropriate rotation force, and re-align after failure. These tasks require high action precision and closed-loop control.

If a model only outputs coarse-grained actions without fine-grained control strategies, the success rate may clearly drop.

3. Drawing Pattern Tasks

Drawing pattern tasks evaluate tool use and trajectory generation. The robot must not only control the drawing tool to contact the paper but also maintain stable pressure and move along the target path. This task requires coordinate mapping, trajectory smoothness, pose control, and visual feedback.

These tasks reflect whether a model can generate continuous actions from abstract goals, rather than only completing discrete grasps.

Layered Architecture Recommendations for Engineering Deployment

If robot control foundation models are to be used in production environments, a layered architecture is recommended instead of concentrating all capabilities in a single model.

A generally applicable layered design is as follows:

1. High-Level Task Planning Layer

This layer is responsible for understanding task goals, decomposing steps, and generating plans. It can be deployed in the cloud or on edge servers and is mainly used for complex reasoning and strategy generation.

2. Mid-Level Skill Control Layer

This layer converts high-level plans into specific action sequences, such as grasping, moving, rotating, and placing. It needs to focus on action validity, collision detection, and execution constraints.

3. Low-Level Real-Time Control Layer

This layer is responsible for high-frequency control, joint commands, force-feedback processing, and safety protection. It usually runs on the robot controller or edge device and has the highest requirements for latency and stability.

4. Data and Observation Layer

This layer collects camera streams, sensor data, joint states, task logs, and exception information. The data can be used for replay, analysis, training, and continuous optimization.

5. Safety and Monitoring Layer

This layer is responsible for anomaly detection, emergency stop, limit control, alerting, and human takeover. It is essential in production environments.

In cloud-edge collaboration scenarios, object storage can be used to save task logs and replay data, message queues can be used for asynchronous event reporting, logging services can be used for failure-cause analysis, monitoring and alerting can track model latency, exception rates, and resource usage, and container platforms can support versioned deployment. This allows model capability, runtime state, and operating cost to be managed together.

Building a Sustainable Evaluation Loop

Robot control models cannot be finalized through a single evaluation. As scenes change, objects change, hardware wears, and task complexity increases, model capabilities need continuous validation.

A recommended evaluation loop is as follows:

1. Define the task set: Cover basic operations, complex long-horizon tasks, and exception recovery tasks.
2. Unify the execution environment: Fix action space, observation interface, timeout rules, and collision rules.
3. Run repeated tests: Record mean, variance, maximum, and minimum values.
4. Classify failures: Perception failure, planning failure, control failure, and safety failure.
5. Perform physical-robot retests: Verify whether simulation results can be transferred.
6. Build regression sets: Automatically retest before a new version is released.
7. Monitor online metrics: Success rate, latency, exception rate, and human takeover rate.
8. Review cost regularly: Inference cost, compute cost, operating cost, and business benefit.

The value of this loop is that it not only answers “which model has a higher score,” but also answers “which model is more suitable for the current business.”

Rational View of High Scores: Capability and Cost-Performance Must Return to the Scenario

High scores of robot control foundation models are worth attention, but scores are only the entry point, not the conclusion. What truly determines model value is its stable performance, safety boundaries, and total cost in a specific scenario.

If the business scenario is laboratory research, more attention may be given to exploration capability, complex task completion, and generalization. If the business scenario is industrial operation, repeatability, safety, maintainability, and unit task cost become more important. If the business scenario is service robots, interaction safety, exception recovery, and user experience also need to be considered.

Therefore, model selection should avoid relying only on a single leaderboard. A comprehensive judgment should consider the following aspects:

1. Whether it matches task complexity.
2. Whether it meets real-time requirements.
3. Whether it has failure recovery capability.
4. Whether it has sufficient safety protection.
5. Whether it supports continuous evaluation and version regression.
6. Whether it can control inference and operating costs.
7. Whether it can transition smoothly from simulation to physical robots.

In summary, competition among robot control foundation models is shifting from “impressive single demonstrations” to “reliable long-term operation,” and from “outstanding single capabilities” to “a comprehensive balance of tasks, cost, safety, and engineering readiness.” For developers and enterprise users, the most rational approach is not to blindly chase the highest score, but to build their own evaluation system and use unified metrics, real-world scenarios, and total cost models to determine whether a model is truly usable, reliable, and sustainable.