General Multimodal Models and Embodied AI: Reassessing In-House R&D Budgets

Embodied AI is undergoing a recalibration of its R&D path. In the past, teams typically started with data collection, model pretraining, and task fine-tuning, gradually improving a robot’s ability to understand visual scenes, natural-language instructions, and operational workflows. Today, general multimodal models show strong visual understanding, reasoning, and planning capabilities in some tasks. Robot systems can complete tasks through model calls, tool orchestration, and specialized control modules without training every capability from scratch.

This does not mean that embodied model R&D has lost its value, but it does mean that “training a large model from zero” is no longer the default answer. The real question for companies is: which capabilities can be obtained quickly through general models, and which capabilities must be built through proprietary data, specialized models, and robotics systems engineering?

The Shift Comes From Capability Composition, Not Just Model Scale

A robot completing a real-world task usually goes through multiple stages: perceiving the environment, understanding instructions, decomposing steps, selecting actions, executing control, and revising the plan based on feedback. In the past, these stages often depended on different modules and specially trained data, and system capabilities were easily limited by task scope, scene variation, and data coverage.

General multimodal models bring visual and language understanding, common-sense reasoning, and step planning into a more general model. A robot system can pass scene information and user instructions to the model to generate a task plan, and then have an execution module with robot interface capabilities complete the actions. For applications with clear task structures and low error costs, this combination can shorten the cycle from prototype to validation.

The industry impact therefore does not come simply from “higher model scores,” but from a change in how capabilities are acquired: some upper-layer capabilities that previously required large amounts of specialized data and repeated training may now be obtained more quickly through model integration, prompt design, and tool calling. For R&D teams, this directly affects project schedules, staffing, and budget allocation.

General Models Can Fill Gaps but Cannot Replace the Full Robot System

It is necessary to distinguish “understanding and planning tasks” from “reliably controlling real devices.” A model can generate seemingly reasonable steps, but that does not mean a robot can execute them stably in a physical environment. The real world involves occlusion, friction, object deformation, sensor noise, and execution error. The same instruction may correspond to completely different operation paths in different sites or with different object layouts.

Robot systems still need to handle a series of engineering problems:

• State estimation and perception. The system must continuously determine object position, pose, operable regions, and its own state. Semantic understanding from a model cannot replace high-precision localization and real-time sensor processing.
• Motion planning and low-level control. From “place the object on the table” to specific joint trajectories, kinematics, collision detection, control policies, and safety constraints must still work together.
• Real-time performance and fault tolerance. Network latency, uncertain model outputs, or a failed single-step action can interrupt a task. The system needs timeout handling, state readback, retries, and safe-stop mechanisms.
• Safety and responsibility boundaries. For tasks involving people, valuable equipment, or hazardous environments, action constraints, human confirmation, and traceable execution records must be established. Model-generated content cannot be treated directly as trusted control commands.

A more feasible engineering approach is to let general models handle high-level semantic understanding and task planning, while specialized modules handle precise perception, action control, and safety validation, with closed-loop feedback continuously correcting execution. Model outputs should undergo structured parsing and constraint checks rather than being converted directly into unverified physical actions.

In-House Decisions Should Move From All-or-Nothing to Layered Development

When general models can cover some general capabilities, companies that still direct their entire budget toward foundation model pretraining may face long investment cycles, high data costs, and duplicated capability building. But completely abandoning in-house development also carries risks: reliance on external models can be affected by cost, service availability, data governance, and the pace of capability iteration, and may not suit scenarios with high requirements for real-time performance, privacy, or stability.

A more reasonable approach is to split investment by capability layer.

1. Prioritize Reuse for General Capabilities

Capabilities such as language understanding, basic visual question answering, and common task decomposition can first be validated through general models. The evaluation focus should not be demo performance, but whether the model can complete tasks stably in the target scenario, whether frequent human intervention is required, and whether the latency and cost of the call chain are acceptable.

2. Continue Accumulating Core Scenario Data

A company’s long-term barrier is often not only model parameters, but its understanding of specific devices, environments, and business processes. Teams should continuously accumulate sensor data, failure cases, human correction records, and task completion results from real tasks, and maintain proper data labeling, version management, and permission governance. This data can be used both to improve specialized models and to evaluate whether general models fit the business.

3. Focus Specialized Capabilities on Differentiated Segments

For precision operations, device-specific control, long-duration continuous tasks, or high-safety scenarios, specialized algorithms and in-house models may still have clear value. Investment should focus on areas that general models cannot cover reliably and that are directly tied to business success, rather than pursuing full-stack in-house development just to “own a foundation model.”

4. System Integration Becomes a Key Asset

Model switching, tool calling, state management, action validation, and failure recovery form the engineering foundation of robot applications. Even if the underlying model changes, a well-designed system can still reuse task interfaces, evaluation processes, and control components. Rather than betting on a single model, building a replaceable, observable, and rollback-capable system architecture is more conducive to controlling long-term risk.

Build Evaluation Mechanisms for Real Tasks

A single leaderboard or short demo is not enough to support R&D decisions. Companies need evaluation sets relevant to actual business needs, covering normal workflows, edge cases, and failure recovery, and validating them in stages in simulation environments and on real devices.

Evaluation should focus on at least the following metrics:

1. Task success rate: Whether the robot completes the goal, not whether the model gives a fluent explanation.
2. Execution stability: Whether results are consistent under different initial states and environmental changes.
3. Human intervention rate: Whether tasks frequently depend on operators supplementing instructions or manually correcting errors.
4. End-to-end latency and cost: Total time from receiving an instruction to task completion, along with model call and device runtime costs.
5. Safety performance: Whether out-of-bounds actions, collision risks, or violations of task constraints occur.
6. Anomaly recovery capability: When an object is unreachable, recognition is incorrect, or execution fails, whether the system can detect the problem and safely exit or replan.

Evaluation should also record model versions, prompt templates, environmental conditions, device configurations, and human interventions. Only when the testing process is reproducible can teams determine whether performance changes come from model updates, system changes, or scenario differences.

Budgets Shift From Training Investment to Capability Composition

In-house R&D budgets should not simply be cut sharply because general models have improved, nor should companies keep the same investment structure as before. More effective budget management means splitting projects into data and evaluation, model validation, robot control, system integration, safety governance, and operations, and adjusting dynamically according to stage goals.

In the early stage, priority should be given to validating whether general models can create real value in the target task, whether system integration is feasible, and whether failure risks are controllable. If validation passes, the company can then decide whether targeted fine-tuning, specialized policy training, or building proprietary model capabilities is needed. If the general solution is unstable on key tasks, the budget should shift toward data supplementation, perception and control improvements, or specific capability R&D, rather than blindly expanding pretraining scale.

Team assessment should also gradually shift from process metrics such as parameter count and training epochs to task completion quality, deployment stability, and maintenance cost. This helps avoid investment mismatches where “the model is larger, but the robot still cannot work reliably.”

In-House R&D Is Not Disappearing; Its Boundaries Are Changing

General multimodal models lower the barrier to acquiring some embodied capabilities and force companies to revisit past R&D inertia. They change the division of capabilities rather than signaling the end of specialized R&D: general models improve the efficiency of understanding and planning, specialized modules ensure perception, control, and safety, and companies build long-term competitiveness through real data and systems engineering.

Therefore, what needs to be reassessed is not “whether we should still do in-house R&D,” but “which capabilities must we control ourselves.” By placing reuse, specialized R&D, and engineering integration in the right positions, companies can control budget risk amid rapid technological change and continuously improve robot reliability in real-world scenarios.