Evaluating Foundation Models for Dual-Arm Robots: Metrics, Sim-to-Real Transfer, and Model Selection

Which Conditions Must Be Standardized Before Evaluation?

All models must be evaluated under the same conditions. Otherwise, their scores cannot be compared fairly. The following settings should be stored as fixed configuration values and loaded for every run rather than changed manually:

• Task descriptions: Use exactly the same natural-language instruction templates.
• Observation interface: Provide camera images only, with identical camera positions, resolution, and frame rate.
• Action space: Select either joint angles or end-effector poses, then keep the control frequency and value range fixed.

• Inference budget: Evaluate using two separate protocols:

  • For capability-ceiling evaluation, allow each model to use its highest available configuration and record the exact parameters.
  • For cost and deployment selection, use the same deployment configuration for every model, including hardware, numerical precision, and batch size. Record P95 latency, maximum concurrency, cost per inference, and retry cost.

  Results from these two protocols must not be mixed in the same table.

• Random seeds: Use a fixed list of seeds for every task to support reproducibility.
• Environment versions: Record the versions of the simulator, object assets, and robot model files.
• Post-processing versions: Record the script versions for action smoothing, clipping, and retry logic.
• Episode count and timeout: Any episode that exceeds the time limit is counted as a failure.
• Task-level success rates: Record results for each task instead of retaining only an overall score.

Store these fields in a version-locked configuration file so that other teams can reproduce the experiment. For example:

eval_config:
  task_prompt_template: "prompts/v3_fixed.txt"
  observation: {cameras: ["head", "wrist_left", "wrist_right"], resolution: [640, 480], fps: 15}
  action_space: {type: "joint_angle", control_hz: 10, clip_range: [-1.0, 1.0]}
  inference_mode: "capability_ceiling"   # or "cost_selection"
  seeds: [11, 23, 47]
  env_version: "sim-2.4.1 / assets-1.8.0 / robot-urdf-0.9.3"
  postprocess_version: "post-1.2.0"
  max_episodes_per_task: 20
  episode_timeout_s: 120

Differences between internal evaluation results and public benchmark rankings usually come from one or more of these fields. If any field is inconsistent, report the two data sets separately.

What Does the Average Score Actually Mean?

The average score is an aggregate metric across multiple tasks and episodes. It does not mean that every task succeeds, nor does it show how the model performs on its weakest task.

Assume there are (T) tasks. Task (t) is run for (Nt) episodes, and the result of episode (n), (y{t,n}), is 1 for success and 0 for failure. Then:

• Task-level success rate:

[
pt = \frac{1}{N_t}\sum_n y{t,n}
]

• Task-weighted average score:

[
\bar{S} = \frac{1}{T}\sum_t p_t
]

If tasks use partial scores, replace (y) with a value between 0 and 1; the formulas remain unchanged. Task-weighted and episode-weighted averages can produce different results, so reports must specify which method was used.

Success rates can fluctuate substantially when the number of episodes is small. For example, if a task is run 20 times and succeeds 14 times, its success rate is 0.70. With (z = 1.96), corresponding to a 95% confidence level, the Wilson confidence interval is approximately 0.48 to 0.85.

When two models differ by only a few percentage points, first check whether their confidence intervals overlap. Overlapping intervals mean that the current sample does not distinguish the models; they do not prove that the models are equivalent. A stronger conclusion requires more episodes or a paired comparison in which both models use the same seeds.

How Should Simulation Task Distributions Be Interpreted?

The task distribution determines what an average score means. The same score of 70 can result from moderate performance across all tasks, or from complete success on some tasks and complete failure on others.

The following categories provide one possible structure. The sample size of 20 episodes is illustrative and should be adjusted according to task variance and available compute; it is not a universal standard.

Category	Representative task	Suggested episodes	Success criterion	Episode timeout	Primary metrics
Object rearrangement	Move multiple objects to target locations in a specified order	20	Position error for every object is below the threshold	2× the human completion time	Success rate, total steps
Tool use	Use a gripper or spoon to complete a pick-and-place task	20	Both the tool and target object reach the specified states	2× the human completion time	Success rate, contact count
Assembly	Insert two components within the required tolerance	20	Insertion depth and angle are within tolerance	2× the human completion time	Success rate, completion time
Fine manipulation	Twist a bottle cap, thread an object, or open and close a drawer	20	The target rotation or displacement is reached	2× the human completion time	Success rate, peak force

High scores often reflect better modeling of geometric constraints, contact mechanics, and multi-step planning. In addition to the overall score, report the lowest task success rate in each category. This helps reveal whether a model is strong only in one type of manipulation.

Separating Zero-Shot Transfer from In-Distribution Generalization

Transfer capability is one of the easiest properties to overestimate. For models that have not been fine-tuned for a specific robot morphology, performance on unseen tasks can serve as a reference for zero-shot transfer.

An unseen task should not contain the same task template, object combination, or initial state as the training data. A task that is close to the training distribution may use the same action type while changing the objects or layout. These two categories must be reported separately; otherwise, zero-shot performance will be overstated.

Before publishing transfer results, complete three checks:

1. Training-data contamination check. Search for the test task’s instruction text, object asset IDs, and initial scene configuration in the training data. Use hashes for exact text matching and perceptual hashes or embedding similarity for approximate image matching. Record the overlap ratio. If it exceeds the threshold defined by the team, remove the task from the zero-shot results.

2. Stratified reporting. Report zero-shot and few-shot success rates separately, using the same test tasks. Document the number and source of few-shot examples, including whether any examples came from the test tasks themselves.

3. Failure-task list. List tasks whose success rate is below the team-defined threshold, such as 0.3. For each task, record typical failure modes—empty grasps, collisions, dropped objects, or timeouts—and assess whether the failure is acceptable in the target application.

Can a High Simulation Score Be Used Directly on a Real Robot?

No. A successful simulation run shows that closed-loop control works in that environment. Real robots also face friction, elasticity, camera calibration errors, and execution delays that are difficult to model completely.

Use the following three-stage release process. If a stage fails, return to the previous stage for diagnosis.

Safety Prerequisites

Before operating a real robot, complete the following safety requirements:

• Perform a written risk assessment that identifies potential collision targets and worst-case consequences.
• Install an emergency-stop button that directly cuts power through hardware circuitry rather than relying on control software.
• Install safety fencing or light curtains to define the robot’s operating area.
• Train operators and conduct at least one manual takeover drill before formal operation.

Stage 1: Closed-Loop Simulation

Run the full task suite in the standardized framework.

Suggested release conditions include:

• Each task’s success rate is no lower than the team’s historical baseline.
• For the same task run with three different seeds, the success-rate range does not exceed 0.2. The team should define this threshold based on its own evaluation requirements.
• No out-of-bounds actions occur.
• No episode exceeds its timeout without being reported as a failure.

Stage 2: Low-Speed Trials in an Isolated Workcell

Select two or three representative tasks and run them at a limited speed in an isolated workcell. The speed limit must be determined by the safety assessment.

Suggested release conditions include:

• 50 consecutive runs without a collision.
• Zero manual takeovers.
• The mean end-effector error between real and simulated trajectories is below the defined threshold. The trajectory-alignment method must be documented.

Any collision or emergency-stop activation must be recorded as an incident, followed by a return to Stage 1 to investigate the simulation model and parameters.

Stage 3: Force Sensing and Redundant Visual Monitoring

Install a six-axis force sensor at the end effector and deploy an independent monitoring camera. The two systems must not share the same data source.

The following thresholds are examples only. Actual values must be determined according to the load tolerance of the end-effector components and the safety assessment:

• When peak contact force reaches 70% of the rated value, reduce speed.
• When it reaches 90%, stop motion.
• If the monitoring camera detects a person or an unregistered obstacle entering the work area, stop motion immediately.

Suggested release conditions for Stage 3 include:

• No force-limit events during 200 consecutive runs.
• When a simulated sensor disconnection occurs, the system enters a stopped state within the defined time limit and passes the corresponding test.
• An operator can switch to stop or teaching mode with a single action.

If Stage 3 fails, return to the previous stage, preserve all logs and video, fix the issue, and restart from Stage 2. If the same issue occurs twice, suspend all real-robot trials for that model and review the source of the simulation-to-reality mismatch.

How Should Models Be Selected Using Latency, Concurrency, and Failure Cost?

Success rate answers whether a model can complete a task. Deployment selection must also consider the cost of completing the task and whether the same hardware can serve the required number of robots concurrently.

Collect the following four metrics using a consistent methodology:

• P50 and P95 latency: Measure the end-to-end time from camera capture to action output, including preprocessing, inference, and post-processing. Collect at least 1,000 requests and report latency in milliseconds. P50 reflects typical performance, while P95 reflects tail latency, which has a greater impact on control loops.
• Maximum concurrency: The maximum number of simultaneous requests that one hardware configuration can serve while keeping P95 latency within the control window.
• Cost per inference: For local hardware, divide the hourly hardware cost by the number of inferences completed per hour. For a cloud API, use the billed unit price.
• Cost per failure: Estimate the average cost of manual reset labor, material loss, and equipment downtime caused by one failure, based on recent operational records.

These metrics can be used to estimate the expected cost per completed task:

[
E = c{\text{inf}} \times n{\text{inf}} + (1-p) \times c_{\text{fail}}
]

Where:

• (c_{\text{inf}}) is the cost of one inference.
• (n_{\text{inf}}) is the average number of inferences per task, including retries.
• (p) is the task success rate.
• (c_{\text{fail}}) is the cost of one failure.

Retries increase the number of inferences while potentially reducing the failure rate. The retry strategy must therefore be included in both (n_{\text{inf}}) and (p), rather than evaluated using only one of them.

The following fictional data illustrates the calculation. The cost of one failure is assumed to be 50 currency units.

Model	Success rate	P95 latency	Maximum concurrency	Cost per inference	Average inferences	Expected cost per task
A	0.80	420 ms	4	0.02	12	10.24
B	0.90	980 ms	2	0.05	30	6.50
C	0.60	150 ms	8	0.01	8	20.08

Use the following four-step selection process:

1. Apply hard constraints. P95 latency must not exceed the control window, which is 1,000 ms in this example. Maximum concurrency must meet the deployment requirement. If a production line needs to control three robots simultaneously, Model B, with concurrency of two, should be rejected immediately.

2. Check the safety floor. The lower bound of the 95% confidence interval for success rate must not fall below the team’s defined safety threshold. Relying only on the point estimate can lead to the wrong choice; the lower confidence bound is the relevant release criterion.

3. Compare expected task cost. Among the models that pass the first two checks, select the one with the lowest expected cost.

4. Resolve close results. If two models have similar expected costs, prefer the model with less severe failure consequences and failures that can be detected promptly by monitoring.

In the example, Model B is rejected because its concurrency is insufficient. Model A has an expected task cost of 10.24, lower than Model C’s 20.08. Model C has the lowest latency, but its success rate is only 0.60, resulting in more failures and higher costs for manual intervention and reset.

If Model A’s lower confidence bound falls below the safety threshold, it cannot be released. The team must first improve its success rate or choose another solution. Every selected model must still pass Stages 1 through 3 of the validation process. Lower cost alone is not a reason to bypass real-world verification.