General-Purpose LLMs Enter Physical Devices: New Possibilities for Embodied Intelligence

From Digital Agents to Physical Agents

In recent years, the expansion of large language model (LLM) capabilities has largely taken place in digital space. Models initially could only converse; they then gained the ability to call search engines, read files, execute commands, and write code. The core shift in this phase was that models evolved from “answering questions” to “completing tasks.”

The next step in this capability expansion is extending the model’s range of action into the physical world. Model outputs are no longer just text or code, but also joint angles for robotic arms, the open/close timing of grippers, and the framing angles of cameras. Feedback from the physical world also flows back into the model, forming a closed loop of perception, decision-making, and execution. Such systems are commonly referred to as embodied intelligence or physical agents.

In 2026, an independent developer has already used a robotic arm to complete a line drawing. The system’s input is a photograph; the intermediate process involves image contour extraction, path planning, and motion control. The final output is a drawing of approximately 256 strokes on paper, with nearly 4,000 path points and a total pen-tip travel distance exceeding 11 meters. The entire process takes about 65 minutes. This example shows that, once the toolchain matures, a single developer plus a reusable program can build a practical robot application with tangible output.

The Technical Layers of a Physical Agent

To understand physical agents, the system can be broken down into five layers. The reliability of each layer affects the final outcome.

1. Perception layer. Cameras, force sensors, position encoders, and various instruments collect environmental information. The quality of perception data directly determines the accuracy of subsequent judgment. Cameras require calibration—that is, establishing the correspondence between pixel coordinates and real-world spatial coordinates. Calibration errors can cause a robotic arm to execute incorrect actions despite correct visual judgment.

2. Understanding and planning layer. In this layer, the LLM converts natural language goals into a sequence of steps. For example, “place the red block on the left tray” would be decomposed into steps such as identifying the target, computing the grasping pose, moving, closing the gripper, moving to the target position, and releasing. Models excel at this semantic-level decomposition, but their judgment on geometric precision and physical constraints is not reliable.

3. Control layer. The control layer is responsible for converting the planned result into smooth, constrained motion. Common techniques include inverse kinematics solving, trajectory interpolation, velocity and acceleration limiting, and closed-loop control algorithms such as PID or impedance control. This layer should be implemented by deterministic controllers, rather than having the model directly output raw commands.

4. Execution layer. Hardware components—robotic arms, motors, grippers, and mobile bases—perform the actual physical actions. This layer carries the highest risk, because once an action occurs, it produces physical consequences.

5. Feedback and learning layer. The system records the input, output, sensor data, and result of each action for debugging, and can also be used to train control policies. The value of a data feedback loop lies in the systematic collection and correction of failure cases.

The following is an illustrative execution loop that shows how the layers cooperate. The functions in the code are conceptual examples, not specific interfaces.

def run_task(goal, robot, camera, planner, safety):
    observation = camera.capture()
    plan = planner.plan(goal, observation, robot.read_state())
    for step in plan.steps:
        # Deterministic validation in the control layer, independent of model output
        if not safety.within_limits(step, robot.limits):
            safety.halt("Action exceeds safety boundary")
            return False
        robot.execute(step, timeout=2.0)
        # Feedback check: re-perceive after execution to confirm the result matches expectations
        feedback = camera.capture()
        if not planner.verify(feedback, step.expected_result):
            planner.replan(goal, feedback)
            break
    return True

This example reflects three design principles. First, plans generated by the model must pass an independent safety check. Second, every execution step requires a timeout control. Third, perception-based verification after execution is necessary—model judgment of the result cannot replace actual measurement.

Possibilities: Which Scenarios Are Worth Attention

Personal creation and small studios. Robots can take over repetitive tasks such as writing, drawing, engraving, and polishing, while creators retain aesthetic judgment and directional control. The value of the toolchain lies in lowering the cost of trial and error, allowing creators to quickly iterate on a reusable workflow.

Research and laboratory automation. Pipetting, weighing, parameter scanning, and sample recording are repetitive and error-prone. Automated systems can operate continuously according to preset protocols, storing each experiment’s parameters and results in a structured format for reproducibility and analysis.

Small-batch flexible manufacturing. For scenarios with small order volumes and frequent product changes, the programming cost of traditional production lines is high. If a robotic arm’s tasks can be quickly configured through natural language descriptions and demonstration examples, production line changeover efficiency improves. Quality inspection is also well-suited for a combination of visual judgment and manual review.

Education and training. Physical robots make programming, control theory, and sensor principles tangible. Students can directly observe how code changes affect the robotic arm’s movements—a feedback loop far more intuitive than pure simulation.

Data flywheel. The value of physical operation data lies in its direct coupling with real environments. Every successful and failed operation can serve as a training sample. Teams with a stable data collection workflow will build an advantage through long-term accumulation.

Constraints and Risks

The risks of physical agents differ from those of digital agents, primarily in the following areas:

Safety risks. Once a physical action is executed, errors cannot be undone as easily as deleting a file. Systems need mechanisms such as speed limits, force limits, hardware and software emergency stops, workspace isolation, and designated human safety zones. Safety mechanisms must be enforced by hardware or the underlying controller, not left to the model’s self-restraint.

Reliability risks. LLM outputs are non-deterministic—identical inputs can yield different plans. Engineering practice typically relies on deterministic controllers as a fallback, constrained action sets, and fallback to a safe state upon failure.

Calibration and drift. Camera position changes, mechanical wear, and lighting variations can all cause perception errors. Systems require periodic recalibration and should pause operation when anomalies are detected.

Data and compliance. Captured video and sensor data may contain personal information or trade secrets. Storage and access should follow the principle of least privilege, with clearly defined data retention periods.

Liability. When equipment causes loss or damage, how responsibility is assigned should be defined in advance through policies and contracts.

Paths for Getting Involved

Different roles have different entry points. The following paths offer a way forward:

For developers.

1. Start with simulation environments to validate motion planning and collision detection logic before connecting to real hardware.
2. Save complete logs for every run, including inputs, plans, sensor data, and execution results, for replays and analysis.
3. Keep deterministic control logic separate from model outputs, with explicit boundary conditions for every action.

For enterprise teams.

1. Prioritize low-risk, highly repetitive scenarios such as equipment checks, data entry, and sample sorting.
2. Define success metrics during pilot phases, such as task completion rate, frequency of human intervention, and time per operation.
3. Establish equipment inventory, access permissions, and emergency response procedures, integrating physical systems into existing operations and safety management frameworks.

For general readers.

1. Start by understanding the basic concepts of perception, planning, control, and execution—this is the foundation for evaluating any physical intelligence solution.
2. Follow open-source communities and public technical documentation to learn fundamentals such as calibration, kinematics, and sensor data processing.
3. Practice on small desktop devices or educational kits to experience how a feedback loop works in practice.

Common Practices for Cloud-Device Coordination

As the number of devices grows, cloud capabilities can handle data storage, remote monitoring, and task scheduling. Common approaches include: using object storage to save trajectory data and video clips; using log services to aggregate device operation records; using monitoring and alerting systems to set thresholds for temperature, current, task timeouts, and other metrics; and using message queues to pass task states between devices and scheduling services. These capabilities fall under general-purpose infrastructure, and selection should consider data volume, latency requirements, and security/compliance constraints.

Conclusion

The entry of general-purpose models into the physical world changes not only robot capabilities but also the barrier to entry. As the barrier lowers, the key question becomes how to design reliable perception, planning, control, and feedback chains—and how to accumulate data within safe boundaries. For anyone willing to get hands-on, the most valuable starting point is to understand the layered architecture of such systems and to begin with a small, well-bounded task.