Large Models from Software Control to Physical Device Control: New Possibilities

The Fundamental Shift from Operating Software to Controlling Reality

Over the past few years, the most common use of large models has been to operate software: generating code, calling APIs, operating browsers and terminals. These tasks have clear fault-tolerance logic: mistakes can be undone, rolled back, or retried, and the cost is mainly time and compute. But when the same model is connected to physical devices—such as robotic arms, valves, motors, and conveyors—the nature of the error changes. A wrong action can produce deviation in the physical world, and many physical states are irreversible: spilled material cannot be recovered, out-of-spec process parameters can contaminate an entire batch, and a misoperation can create personal safety risks.

This is not something that can be solved simply by increasing software fault tolerance. The physical world introduces three things that are relatively lightweight in the software world: real-time constraints, safety boundaries, and responsibility assignment. A model’s “inference” is no longer a purely digital behavior; it can produce real consequences through actuators.

Engineering Prerequisites: Three Questions to Answer Before Deployment

In highly regulated industries such as pharmaceuticals, three questions must be answered before asking whether a model can control a system:

1. Who approves each action? Is a model-provided action a suggestion or an instruction? Before closing the loop, who bears final responsibility for each actual action?
2. Who handles exceptions? When sensor readings deviate, actuators do not respond, or model outputs go out of bounds, does a human take over, or does another deterministic logic layer?
3. How is everything traced afterward? Can each action, each decision, and each data snapshot be fully reconstructed and audited?

There is no field for “the model thinks” in a compliance framework. This means that any model entering a control loop must be wrapped in an engineering shell that is approvable, circuit-breakable, and traceable, rather than directly connected to actuators.

A Three-Stage Path: From Read-Only to Closed Loop

A practical path is not one-step intelligent control, but a gradual three-stage evolution. Each stage creates value on its own, and the risks in the first two stages are much lower than in the third.

Stage 1: Read-Only

The system only reads data, performs analysis, and raises alerts; it does not touch any actuator. Typical capabilities include reading instrument values, reading batch records, and comparing historical baselines for anomaly warnings. Much downtime in the field is actually consumed by “not being able to tell what is wrong.” Therefore, simply being able to see anomalies can already bring considerable benefits.

# Illustration: read-only stage; collect data and issue alerts only; do not drive any actuator
def monitor_only(sensor_stream, baseline):
    for reading in sensor_stream:
        anomaly = detect_deviation(reading, baseline)   # Compare against baseline/historical distribution
        if anomaly.score > anomaly.threshold:
            alert_service.push({                        # Push alert only; do not write back to the device
                "severity": anomaly.severity,
                "signal": reading.tag,
                "deviation": anomaly.score,
            })
        # Key constraint: in this stage, no code path exists to drive valves, motors, or switches

The core value of this stage is to build a trustworthy data pipeline and sensing capability: only after data ingestion, cleaning, alignment, and alerting are stable can later control be considered.

Stage 2: Suggestion Mode (Human-in-the-Loop)

The model begins to provide parameter suggestions, but instructions are issued by humans. For example, the model suggests adjusting the temperature of a process section, and an engineer decides whether to execute it after review. The essence of this stage is “the model proposes, humans decide,” keeping the responsibility gate clearly on the human side.

# Illustration: action pipeline with a human approval gate and complete audit records
def execute_with_gate(candidate_action, context):
    risk = assess_risk(candidate_action, context)       # 1. Classify action risk
    if risk.level in ("high", "irreversible"):
        approval = request_human_approval(candidate_action, context)
        if not approval.granted:                         # 2. High-risk actions must be released by a human
            audit_log.append({"action": candidate_action,
                              "decision": "rejected", "by": approval.user})
            return "blocked"
    snapshot = capture_state(context)                    # 3. Snapshot before execution to ensure traceability
    result = actuator.execute(candidate_action)
    audit_log.append({                                   # 4. Who approved it, what was done, and what was the result
        "action": candidate_action, "risk": risk.level,
        "approver": approval.user if risk.level != "low" else "auto",
        "before": snapshot, "result": result,
    })
    return result

Stage 3: Closed-Loop Control

Only after the first two stages have accumulated sufficient behavioral data, established mature exception-handling mechanisms, and built an audit system can the model be allowed to execute within a limited, controlled closed loop. Even then, hard safety boundaries—such as threshold clamping and circuit-break shutdown—and a human takeover channel—should be retained, so that the model works within guardrails rather than replacing all deterministic logic.

New Possibility: Making Legacy Equipment “Understandable”

The most practical value of this shift is not buying an entirely new intelligent production line, but making old equipment understandable and usable without replacing hardware. Through external sensing (cameras, sensors), behavior data collection, and model analysis, an old device without an open control protocol can also be “read.” Its cost is an order of magnitude lower than replacing the entire production line.

A common pattern is that developers connect large models to open-source robotic arm kits, use cameras to identify targets and grippers to attempt grasping, and continuously collect action data to train control models. Some studios also use intelligent programming tools to write motion-control programs, handle camera calibration, and check motion paths. With only an individual or a small team, they can build an automation workflow that can be continuously improved. The exploration of pushing AI agents from “operating software” to “controlling devices” is repeating the path of the earlier automation-script wave—early demos are rough, but they accumulate long-term valuable capabilities.

But there is one key principle: treat it as an instrument, not as an operator. Letting the model understand device state, predict anomalies, and provide suggestions has great value and controllable risk; letting it directly replace human operational decisions requires extreme caution.

Key Engineering Capabilities That Support Device Intelligence

For the above three stages to be implemented, a set of general capabilities is needed as the foundation. These capabilities can be organized using industry-standard product categories and do not depend on any specific vendor:

• Data ingestion and storage: Device time-series data, batch records, images, and other data need reliable collection and persistence. Object storage and message queues can carry real-time and batch data flows.
• Perception and calibration: Camera calibration, multi-sensor time alignment, and action-data labeling are prerequisites for control-related tasks.
• Model training and inference: Visual recognition, anomaly detection, and control strategies require reproducible training and inference pipelines.
• Monitoring, alerting, and log auditing: Every action, alert, and decision record must be traceable. Logging services and monitoring/alerting are infrastructure for compliance auditing.
• Containerization and elastic scheduling: Control services need stable deployment and gradual release. Container platforms can provide isolation and rollback capabilities.

Governance and Compliance: An Indispensable Part

Return to the initial three questions—approval, handling, and tracing—because they determine whether the system can enter real production. At a minimum, engineering must achieve: action risk classification, mandatory human approval for high-risk actions, state snapshots before and after execution, and complete audit-log retention. This governance logic is independent of which model or device is used; it is the necessary engineering between a “model that can reason” and a “trusted production system.”

Conclusion

Large models moving from software operation to controlling physical devices open up real new possibilities: low-cost intelligence for legacy equipment, visual anomaly warning, and acceleration of R&D and automation workflows. But the premise remains: first build read-only data and sensing capabilities, then use human-in-the-loop to carry responsibility, and only then discuss closed-loop control within guardrails. Treating the model as an “instrument” that improves judgment, rather than an “operator” that replaces decisions, is the stable and sustainable way forward on this path.