From Operating Software to Controlling Physical Devices: Engineering Large Model-Driven Physical Agents

In recent years, the role of general-purpose large models has continued to expand. They began as tools for conversation and text generation. Later, through tool calling and code generation, they started operating browsers, terminals, file systems, and software APIs, giving rise to an agent paradigm centered on “given a goal, decompose it autonomously, and execute iteratively.” The next shift is connecting the decision-making capability of large models from virtual environments to physical actuators—robotic arms, gimbal cameras, grippers, laboratory instruments, sensor arrays, and even an entire small workflow.

The threshold is not simply writing another device driver. In software environments, trial and error is nearly free: errors can be rerun, and files can be rolled back. The physical world is different. An incorrect grasp may damage a workpiece, an incorrect trajectory may damage an end effector, and a calibration error of a few millimeters can collapse the success rate of the entire task. Therefore, integrating a large model into physical devices is a systems engineering reconstruction, not a toolset expansion. The following sections break this down by architecture, implementation path, common pitfalls, and evaluation method.

1. Why the two environments require different architectures

The first step in understanding physical agents is to recognize that they inherit engineering constraints from robotics, with an additional layer of stronger semantic understanding. The differences are concentrated in four areas:

• Irreversibility: Software operations can be rolled back transactionally, but physical actions cannot be undone. The system must decide whether an action can be executed before executing it, rather than correcting after execution.
• Real-time layering: Visual servoing, force control, and collision detection operate at millisecond scale, while task planning and parameter generation operate at second scale. They cannot be placed in the same loop.
• More sources of uncertainty: Changes in lighting, occlusion, object deformation, friction coefficients, and end-effector looseness can cause the same high-level instruction to produce different physical results.
• Safety risk spillover: Risk is no longer limited to data and processes; it can directly affect people, equipment, and materials. Safety must be a first-class concern, not a patch applied later.

The core principle that follows is: let large models handle understanding and planning, and let the traditional control stack handle determinism and safety. Forcing semantic capabilities into a real-time control loop is one of the most common architectural mistakes in this type of system.

2. Five-layer reference architecture

A reusable physical agent system can be divided into five layers with clear responsibilities and boundaries.

Layer 1: Device access and capability abstraction. Each device’s capabilities and constraints should be explicitly described as a machine-readable “capability contract,” including degrees of freedom, workspace, payload, speed limits, supported action primitives, and error-code semantics. The key output of this layer is a unified library of action primitives, not raw device SDKs exposed directly to the model. Primitive granularity matters: primitives that are too fine, such as a single joint angle, force the model to take on an excessive motion-control burden, while primitives that are too coarse, such as “complete a painting,” lose controllability.

Layer 2: Perception and calibration. Camera intrinsics, extrinsics, hand-eye calibration, depth alignment, and workpiece pose estimation are completed in this layer. It outputs a world state with uncertainty, rather than “where the model thinks the object is.” In practice, uncertainty should be propagated explicitly: pose estimates should include covariance, segmentation masks should include confidence, and downstream components should use this information to decide whether execution is allowed.

Layer 3: Cognition and task planning. This is the main domain of large models: decomposing a natural-language goal into ordered steps, generating parameters for each step, diagnosing failures, and selecting recovery actions. It should generate code or actions constrained to a restricted DSL or parameterized API, rather than freely writing arbitrary programs.

Layer 4: Motion and safety control. Deterministic controllers handle this layer: trajectory interpolation, limiting, collision detection, torque monitoring, and emergency-stop chains. All commands from upper layers are rechecked here. If validation fails, execution is rejected and the reason is reported.

Layer 5: Data and evaluation. This layer records the instruction, state, pose, duration, and failure reason for each step, supporting trajectory replay, offline evaluation, and later policy training. It determines whether the system can continuously improve, rather than merely completing one run.

3. Key implementation: slow loop + fast loop + hard guardrails

The most worthwhile engineering pattern is an explicit dual-loop structure. The slow loop is driven by the large model, operates at a lower frequency, tolerates latency, and produces outputs that require validation. The fast loop is a local real-time controller, fully decoupled from the network and the model.

Device capability contracts can be described as follows:

from dataclasses import dataclass, field
from typing import Literal
@dataclass(frozen=True)
class AxisLimit:
    lower: float
    upper: float
    max_velocity: float      # units per second
    max_accel: float
@dataclass(frozen=True)
class DeviceCapability:
    name: str
    dof: int
    payload_kg: float
    workspace_radius_m: float
    primitives: tuple[str, ...]      # action primitives allowed for the model
    axes: dict[str, AxisLimit] = field(default_factory=dict)
arm = DeviceCapability(
    name="tabletop_arm",
    dof=6,
    payload_kg=0.5,
    workspace_radius_m=0.6,
    primitives=("move_to_pose", "grasp", "release", "trace_path", "set_pressure"),
    axes={"j1": AxisLimit(-170, 170, 1.2, 6.0),
          "j2": AxisLimit(-100, 100, 1.2, 6.0)},
)

The guardrail layer translates “intent provided by the model” into “commands allowed by the device.” Any failed check is rejected directly:

class SafetyGuard:
    def __init__(self, cap: DeviceCapability, max_force_n: float = 10.0):
        self.cap = cap
        self.max_force_n = max_force_n
    def check(self, primitive: str, params: dict, pose_cov: float) -> list[str]:
        errs = []
        if primitive not in self.cap.primitives:
            errs.append(f"unsupported primitive: {primitive}")
        if params.get("speed", 0) > self.cap.axes["j1"].max_velocity:
            errs.append("speed exceeds axis limit")
        if params.get("force_n", 0) > self.max_force_n:
            errs.append("force exceeds contact budget")
        # Contact primitives are rejected when pose uncertainty is too high
        if primitive in ("grasp", "release") and pose_cov > 0.02:
            errs.append("pose covariance too high for contact primitive")
        return errs

The slow-loop planner limits generated content to a restricted language, performs structural validation before execution, and runs a simulation pre-check:

def run_task(goal: str, planner, guard: SafetyGuard, sim, driver, max_steps: int = 12):
    state = sim.snapshot()
    for step in range(max_steps):
        plan = planner.next_step(goal, state)     # returns {primitive, params, precondition}
        errors = guard.check(plan["primitive"], plan["params"], state["pose_cov"])
        if errors:
            return {"status": "rejected", "reason": errors}
        if not sim.is_collision_free(plan):        # validate the trajectory in simulation before real execution
            return {"status": "blocked_by_sim", "step": step}
        result = driver.execute(plan)              # enters deterministic fast loop with limit enforcement and emergency stop
        state = sim.snapshot()
        if result.failed:
            recovery = planner.diagnose(goal, state, result.error_code)
            if recovery is None:
                return {"status": "failed", "step": step}
    return {"status": "goal_check_required", "state": state}

Three design points deserve emphasis. First, contact-type actions must be gated by low-uncertainty preconditions. Second, simulation pre-validation should be a mandatory gate before real-device execution. Third, failure paths should be explicitly handed to the planner for diagnosis, rather than having the controller retry blindly.

4. Three practical entry paths

Different teams have very different resources and capabilities, so entering physical agent systems should be done in layers.

Path 1: Build new workflows for existing equipment. Many laboratories and factories already have usable hardware but lack orchestration capability. After abstracting device capabilities into a primitive library, the large model’s task is to generate “device collaboration scripts”: which instrument samples first, which instrument measures later, how data is transferred between steps, and how to skip exceptions. This type of transformation has the lowest investment and the fastest results. The main difficulty is unified packaging of heterogeneous device protocols.

Path 2: Simulation first, convergence on real hardware. Trajectory generation, grasping policies, and algorithm validation can be completed in simulators, then transferred to real devices through domain randomization—randomizing friction, mass, lighting, and sensor noise—and fine-tuned with a small amount of real-device data. The core metric of this path is not absolute success rate inside simulation, but the degree of success-rate degradation when moving from simulation to real hardware.

Path 3: Data-driven behavior policies. State-action pairs can be collected through teleoperation and used to train end-to-end vision-to-action policies for tasks that are difficult to model precisely, such as grasping, insertion/removal, and wiping. The bottleneck is usually not the model architecture, but data quality and collection efficiency: it requires unified time synchronization, clear success labels, and a reproducible camera calibration process.

These three paths can evolve together: first use Path 1 to establish the data pipeline, use Path 2 to reduce cost, and finally introduce Path 3 for high-frequency, high-value workstations.

5. Common pitfall checklist

• Treating calibration as a one-time task. If a camera bracket is bumped or an end effector is replaced, calibration becomes invalid. Calibration should be a process with periodic self-checks, and extrinsics should be versioned after each self-check.
• Lack of intermediate states and rollback points. Long-running tasks must support resumption from breakpoints and manual takeover. Otherwise, any small failure means starting the entire run over.
• Treating natural-language goals as executable parameters. Model-generated coordinates, force values, and speeds must all pass guardrail validation. They cannot be sent just because they “look reasonable.”
• Underestimating observation differences between simulation and real hardware. Lighting changes and occlusion are the most common transfer killers. Sufficient visual domain randomization should be introduced during training.
• Running the safety chain through a software path. Emergency stop, limiting, and torque monitoring should be guaranteed by hardware or a local real-time layer. They cannot depend on the model or network being online.
• Only tracking the “run-through rate.” One successful demo cannot hide long-tail failures. A systematic evaluation system is required.

6. Cloud and edge division of labor

The data volume and compute requirements of physical agents correspond to several categories of general-purpose cloud capabilities. Trajectories and videos are suitable for object storage and should be managed in partitions by task and time. Device commands and telemetry are suitable for message queues to buffer peaks and decouple systems, preventing the control chain from being blocked by data reporting. Batch simulation pre-runs and policy training are suitable for elastic scheduling on container platforms, started on demand and released after use. Device logs, exception codes, and status metrics should be connected to a unified logging service and monitoring/alerting system, with an independent alerting channel for safety incidents. Device identities and access credentials should be managed centrally through secret management capabilities, with hardcoding prohibited.

It is important to emphasize that inference and control involved in any real-time control loop should be pushed down to edge nodes. The cloud should only handle planning, training, storage, and analysis. Placing a control loop behind a cross-network call is one of the hardest failure sources to troubleshoot in this type of system.

7. How to evaluate a physical agent

At least six metrics should be tracked: task-level success rate (end-to-end pass rate from goal to completion), per-step success rate (identifying at which layer failures occur), human intervention rate (number of human takeovers per 100 tasks), average completion time, failure recovery rate (proportion of disturbances after which the system can recover autonomously), and safety incident rate (number of collisions, limit violations, or excessive contact forces). Among these, human intervention rate and failure recovery rate best reflect real usability and are most easily ignored in demo-style evaluations.

Conclusion

The expansion from operating software to controlling devices has not reduced the engineering difficulty of large models. It has moved the difficulty from “understanding intent” to “reliable execution.” What truly creates differentiation is not who can demonstrate an impressive physical action once, but who can steadily improve the success rate in mundane daily conditions such as calibration drift, lighting changes, and workpiece shifts. Decoupling semantic capability from deterministic control, making safety guardrails unbypassable gates, and turning evaluation and data loops into infrastructure—these three things are what make the physical world a stage where large models can sustainably contribute value.