Embodied AI: How the Harness Determines Robot Control Performance

    This article examines why the harness—the translation layer that connects a large language model to the physical world—often matters more than the model itself when it comes to embodied AI. It explains what a harness does, why letting a model drive robot joints directly almost always fails, and how layered abstractions, skill libraries, pretrained controllers, and tool-based interfaces close the gap. It also offers six practical design principles, guidance for evaluating embodied systems, and the infrastructure required to run them in production.

    1. From “Lab Toy” to “Engineering Problem”: A Shift Is Underway

    Since this past summer, “letting a large model directly operate a robot” has evolved from the exploration of a few researchers into a direction that many engineering teams are seriously evaluating. The early intuition was straightforward: if a model can already understand natural language and call software APIs, then feeding it a robot’s sensor readings and letting it output action commands should let you drive a robotic arm, a quadruped, or a humanoid “in one step.”

    Hands-on practice quickly produced a counterintuitive conclusion: whether a model can handle a robotic task depends not only on the model’s own intelligence, but even more on the interface through which it connects to the physical world. Some systematic evaluations at the industry frontier have found that the same generation of models can perform drastically differently under different connection methods—sometimes by a margin larger than the capability gap between two model generations. In other words, the problem likely does not lie with the model, but with the “shell” we build around it. The industry usually calls this shell the harness.

    2. What Is a Harness: The Translation Layer Between Model and Physical World

    If you think of the large model as the brain, then the harness is the nervous and muscular system that connects the brain to the body. It typically handles several things:

    • Perception translation: organizing raw sensor data (camera images, point clouds, joint encoder readings, IMU poses, and so on) into observations the model can “understand”;
    • Action translation: converting the model’s intent into control commands the robot can execute;
    • Memory and state: maintaining context such as task progress, action history, and environment maps;
    • Feedback loop: returning execution results, failure causes, and environment changes to the model so it can decide the next step;
    • Safety constraints: intercepting out-of-bounds actions and enforcing speed and amplitude limits.

    It’s worth emphasizing that the harness is not “auxiliary code”—it is a first-class citizen. It determines what the world looks like to the model, which actions the model can take, and whether the model gets a chance to correct itself after an error. The quality of the harness design directly sets the ceiling on whether the model’s capability can actually be realized.

    3. Why Letting the Model Control Joints Directly Almost Always Fails

    The most naive approach is to use each joint’s target angle or torque as the action space and have the model output low-level control signals frame by frame. This path almost never works, for several reasons:

    1. The action space is too high-dimensional and strongly coupled. Humanoids and quadrupeds often have anywhere from a dozen to dozens of degrees of freedom, and the joints influence one another. The model would need to do precise millisecond-level planning in a continuous, high-dimensional space—far beyond the discrete, semantic decisions that language models are good at.
    2. Temporal granularity mismatch. Language models reason on the order of “seconds,” while low-level control requires “millisecond” closed loops. A module that reasons once per second simply cannot directly produce stable control signals.
    3. No physical priors. The model is not explicitly given constraints like dynamics, contact, and friction, so its outputs are often actions that “look reasonable but are physically infeasible.”
    4. No chance to correct errors. Once low-level action errors accumulate, the model neither sees the feedback nor has time to fix it, and the task fails outright.

    Stacked together, these four points make “direct joint control” a classic failure mode. A pattern that recurs in evaluations is this: the model’s capability is sufficient, but it simply “cannot reach” the robot’s execution layer.

    4. Layered Abstraction: Replace Raw Actions with Skills and Pretrained Controllers

    The approach that actually works is to insert a layered abstraction between the model and the hardware, encapsulating the “low-level control that is hard to decide directly” into “high-level skills the model can call directly.”

    • Pretrained controllers: legged locomotion, robotic-arm grasping, obstacle avoidance and following, and similar tasks already have mature reinforcement learning or planning implementations. The model does not need to reinvent them—it just calls them as “commands,” for example “walk forward two meters” or “pick up the red block on the table.”
    • Skill library: turn common behaviors into reusable skills with semantic names, preconditions, and success criteria. What the model faces is no longer an “array of torques” but “move_forward,” “grasp,” and “place_on.”
    • Hierarchical planning: the high level has the model do task decomposition and decision-making, while the low level has dedicated controllers guarantee safety and stability. This effectively reuses the mature “decision–planning–control” layering idea from autonomous driving.

    The payoff is direct: the model’s action space converges from “high-dimensional and continuous, hard to act on” to “low-dimensional and discrete, semantically clear”—which is exactly the domain language models excel at.

    Below is an illustrative interface design showing how the harness should expose “skills” rather than “joints” to the model:

    1. # The harness exposes "semantic skills" to the model, not low-level joint commands
    2. class RobotHarness:
    3. def observe(self) -> Observation:
    4. """Translate raw sensor data into a structured observation"""
    5. return Observation(
    6. scene=self.vision.describe_scene(),
    7. robot_pose=self.localizer.current_pose(),
    8. inventory=self.gripper.state(),
    9. )
    10. def available_actions(self) -> list[Skill]:
    11. """Expose only the skills executable in the current context, with their preconditions"""
    12. return [
    13. Skill(name="move_to", args={"target": "Point(x,y)"}),
    14. Skill(name="grasp", args={"object": "DetectedObject"}),
    15. Skill(name="place_on", args={"surface": "Surface"}),
    16. ]
    17. def step(self, skill: Skill) -> Result:
    18. """Hand the high-level skill to a pretrained controller and collect feedback"""
    19. if not self.preconditions_met(skill):
    20. return Result(ok=False, reason="precondition_failed")
    21. outcome = self.controllers.run(skill) # low-level stability guaranteed by dedicated controllers
    22. return Result(ok=outcome.success, feedback=outcome.observation)

    The key design here is that the model sees “a list of executable actions + structured observations + execution feedback,” not raw joint signals. This conversion is often the dividing line between “it works” and “it doesn’t.”

    5. Give the Model Tools: Localization, Navigation, and Manipulation Primitives

    Beyond skills, the harness can also give the model a set of “tools,” letting it invoke robot capabilities the way a coding agent calls functions:

    • Localization tools: query the robot’s position and heading on the map, answering “Where am I?” and “Where is the target?”;
    • Navigation tools: given a target point, return a traversable path or segmented waypoints;
    • Manipulation primitives: open/close the gripper, detect and recognize objects, read force/position feedback, and so on.

    One trend worth noting: when these tools are properly exposed, the ease with which models drive equipment like drones and mobile bases is rapidly approaching the level of a coding agent calling software APIs. The only difference is that a robot tool’s execution result happens in the physical world, so the harness must additionally take on the responsibility of safety fallbacks and failure recovery.

    6. Six Design Principles for the Harness

    1. Semantic actions: preferentially expose “intent-level” skills and leave low-level control to dedicated modules.
    2. Structured observations: don’t make the model “guess” at raw data—actively translate it into structures with labels, coordinates, and confidence.
    3. Immediate feedback: return success/failure and key state after every execution to close the loop.
    4. Manageable context: control context length with sliding windows, summaries, and external memory to avoid “forgetting the original goal” on long tasks.
    5. Safety up front: set hard constraints on speed, force, and workspace so out-of-bounds actions are intercepted before they reach the hardware.
    6. Recoverable failures: design “retry, roll back, request clarification” as first-class citizens rather than after-the-fact patches.

    7. Evaluation Takeaways: Scores Depend on the “Connection Method”

    The evaluation phenomenon mentioned earlier is essentially a reminder of how to correctly measure an embodied system:

    • When reporting model scores, you must also report the harness version. Otherwise, “the same model differs by an order of magnitude in score” becomes impossible to explain.
    • When comparing different models, keep the harness unchanged as much as possible; otherwise you are comparing “shell differences” rather than “model differences.”
    • Fold harness ablation studies into the evaluation: remove the localization tools, remove the skill abstraction, remove the feedback loop one at a time, and observe how the score changes to pinpoint the bottleneck.

    A pragmatic conclusion: in embodied AI, the shortest path to raising a system’s ceiling is often not “switch to a bigger model,” but “rebuild the harness once.”

    8. Engineering Implementation and Infrastructure

    Actually running a system like this also requires reliable infrastructure support:

    • Model inference services: host online inference for the policy model and guarantee low latency;
    • Container platforms: isolate the execution of controllers, simulation environments, and tool services;
    • Message queues: decouple the perception, decision, and execution stages and buffer high-concurrency data streams;
    • Object storage: accumulate simulation scenes, trajectory data, and skill assets;
    • Logging services and monitoring/alerting: perform full-link tracing for every task to detect abnormal actions and failure modes in time.

    These general-purpose capabilities are standardized on mature cloud platforms. Baidu AI Cloud, for example, offers general categories such as model services, container platforms, message queues, object storage, and logging and monitoring/alerting that can directly support building the architecture above, letting teams focus their energy on the harness and the skills themselves rather than repeatedly reinventing the infrastructure wheel.

    Conclusion

    The difficulty of putting embodied AI into practice is shifting from “is the model smart enough” to “can the model be used well.” As the translation layer connecting the model to the physical world, the harness determines the quality of observations, the granularity of actions, the closure of feedback, and the safety baseline. When a team is willing to treat the harness as a first-class engineering problem—replacing raw joints with pretrained controllers, replacing torque arrays with semantic skills, and replacing direct manipulation with tooled capabilities—a model that originally scored poorly can often deliver performance far beyond expectations. This may well be the most cost-effective way to accelerate embodied agents at the current stage.