General-Purpose Large Models for Chassis Control: Technical Challenges and the Evolution of Autonomous Driving

1. Introduction: From Cognitive Reasoning to Physical-World Control

As artificial general intelligence and multimodal foundation models continue to advance in common-sense reasoning, code generation, and natural-language understanding, researchers and industry teams are exploring how to extend these capabilities to embodied physical systems.

Several highly experimental approaches have recently emerged. By using the function-calling interfaces of pretrained multimodal models, researchers have sent steering, braking, and throttle commands directly to the drive-by-wire chassis of modified test vehicles. In closed, low-speed environments, these vehicles have performed tasks such as starting, navigating around cones, and reversing into parking spaces.

However, test results reveal a significant technical gap. A small number of models with strong generalization and reasoning capabilities have completed basic routes after multiple attempts, but only at extremely low average speeds—approximately 1–2 km/h. Most models have shown low completion rates when handling more complex dynamic paths.

These results indicate that using text or multimodal models to drive physical entities through discrete reasoning is fundamentally different from achieving production-grade autonomous driving. Where should autonomous-driving technology evolve from here? This article examines the question from three perspectives: system architecture, control-engineering constraints, and system-design paradigms.

2. System Architecture: How Tool-Driven Vehicle Control Works

In an experimental architecture that connects a general-purpose model directly to a vehicle chassis, the system typically consists of three major components: multimodal input processing, a cognitive reasoning core, and a tool-execution gateway.

[Surround-View Cameras / Sensor Data]
                │
                ▼
[Multimodal Feature Encoding]
                │
                ▼
[General-Purpose Model: Cognition and Reasoning]
        ──> [Natural-Language CoT / Discrete Decisions]
                │
                ▼ (Tool / Function Calling API)
[Drive-by-Wire Chassis Gateway (CAN/Ethernet)]
                │
                ▼
[Steering / Throttle / Brake Actuators]

In this control flow, the large model acts as the decision-maker for the entire perception–planning–control stack. It periodically receives camera frames and ego-motion data, generates control parameters from the multimodal context—such as a target steering angle or longitudinal acceleration—and then passes them through a standardized software interface to produce messages for the vehicle’s underlying bus system.

Although this pipeline is logically feasible, a large model is fundamentally a predictor optimized over a discrete probability distribution. Its discrete planning process is therefore in natural conflict with the continuous and nonlinear dynamics of a vehicle operating in the physical world.

3. Core Technical Bottlenecks of Direct Large-Model Chassis Control

3.1 Mismatch Between Inference Latency and Real-Time Control Cycles

Vehicle-dynamics control requires extremely fast system responses. In standard autonomous-driving architectures, low-level lateral and longitudinal control loops typically refresh every 10–20 ms, corresponding to 50–100 Hz.

By contrast, a multimodal model with tens or hundreds of billions of parameters may require hundreds of milliseconds to several seconds for a single forward pass and autoregressive generation.

This severe difference in time scale means that the system’s control commands are based on outdated snapshots of the surrounding environment and vehicle state. During continuous curves or obstacle avoidance, the vehicle may experience significant control overshoot and oscillation. This latency mismatch is a fundamental reason why test vehicles can maintain only very low average speeds.

3.2 The Gap Between Spatial Geometry and Vehicle-Dynamics Modeling

General-purpose multimodal models are effective at high-level semantic understanding—for example, identifying a traffic cone or a parking-space marking. However, they remain limited in precise three-dimensional metric reconstruction and real-time ego-dynamics modeling.

• Geometric distortion: Monocular images do not provide highly accurate metric depth information, making the model vulnerable to distance-estimation drift.
• Missing dynamics constraints: The model has difficulty estimating the tire-grip limit, sideslip angle, and road-surface friction coefficient in real time. As a result, it may produce control commands that exceed the vehicle’s physical limits.

3.3 The Conflict Between Probabilistic Hallucinations and Deterministic Safety Requirements

Automotive electrical and electronic systems are built around the ISO 26262 functional-safety framework, which requires extremely low failure probabilities under strict ASIL thresholds.

General-purpose models, however, produce probabilistic outputs and may exhibit decision hallucinations. At a critical moment, a model could generate an unexpected sharp steering command or abnormal braking action. Such behavior is unacceptable under production-vehicle safety standards.

4. A Potential Direction: System 1/System 2 Architecture

To address the latency of general-purpose models and the limited generalization of traditional rule-based systems, the industry is increasingly moving toward a layered, heterogeneous architecture. This design separates a fast system, analogous to human intuition, from a slow system responsible for complex reasoning and planning.

"""
Example of fast–slow system cooperation
System 2 handles high-level topological navigation and semantic decisions
at a low frequency, such as 1–2 Hz.
System 1 generates trajectories and performs high-frequency dynamic tracking
at a high frequency, such as 50–100 Hz.
"""
class FastSlowAutonomousDrivingSystem:
    def __init__(self, slow_planner_llm, fast_policy_network, chassis_interface):
        self.slow_planner = slow_planner_llm      # Cognitive reasoning model (System 2)
        self.fast_policy = fast_policy_network     # Specialized end-to-end/VLA model (System 1)
        self.chassis = chassis_interface          # Drive-by-wire chassis interface
        self.current_semantic_plan = None
    def system2_reasoning_loop(self, multi_modal_inputs):
        """Low-frequency cognitive loop for long-tail semantics and
        medium- to long-horizon spatiotemporal corridors."""
        while True:
            # Extract semantic information from complex interactions
            self.current_semantic_plan = self.slow_planner.generate_spatiotemporal_corridor(
                images=multi_modal_inputs.get_camera_frames(),
                nav_goal=multi_modal_inputs.get_target_waypoint()
            )
            time.sleep(0.5)  # 2 Hz planning cycle
    def system1_control_loop(self, sensor_stream):
        """High-frequency control loop for vehicle dynamics correction,
        continuous trajectory tracking, and safety protection."""
        while True:
            current_state = sensor_stream.get_realtime_state()
            # Combine System 2 constraints with high-frequency perception
            # to generate a deterministic action
            control_action = self.fast_policy.predict_continuous_action(
                state=current_state,
                corridor_constraint=self.current_semantic_plan
            )
            # Validate the action against a safety envelope
            safe_action = self.apply_safety_envelope(control_action, current_state)
            # Send control messages to the chassis actuators
            self.chassis.send_control_command(
                steering=safe_action.steering_angle,
                torque=safe_action.drive_torque,
                brake=safe_action.brake_pressure
            )
            time.sleep(0.02)  # 50 Hz execution cycle
    def apply_safety_envelope(self, action, state):
        # Deterministically clamp actions to satisfy vehicle-dynamics constraints
        return rule_based_clamping(action, state.kinematics)

Under this paradigm:

• The slow system, represented by a cognitive foundation model: It focuses on long-tail interaction scenarios, global topological route reasoning, and the interpretation of unstructured roads. Its output is a spatiotemporal corridor or a driving preference—not a low-level steering, throttle, or braking command.
• The fast system, combining an end-to-end network with conventional control: It uses a specialized vision-language-action (VLA) model or a trajectory-tracking algorithm based on deterministic physical models. It performs lateral and longitudinal control, as well as emergency obstacle avoidance, within millisecond-level time windows to maintain continuity and stability.

5. Infrastructure and Cloud Collaboration: Building the Data Flywheel

The computing and power constraints of an onboard platform make it impractical to deploy a large general-purpose model in its entirety on every vehicle. Future competition will therefore depend on a coordinated hardware–software architecture that combines lightweight, efficient vehicle-side execution with advanced cloud-based generalization.

1. Vehicle-side optimization and distillation of Mixture-of-Experts models: General-purpose cognitive capabilities at the hundred-billion-parameter scale can be distilled and compressed for specific tasks into models with tens of billions or even billions of parameters. This approach aims to balance spatial understanding with low-power, real-time inference.

2. Cloud-based world-model simulation and validation: Cloud-native high-performance computing and object storage can support a closed-loop data factory. This enables large-scale mining of extreme long-tail corner cases and safety regression testing for autonomous-driving algorithms in high-fidelity virtual environments, potentially covering trillions of simulated kilometers.

3. Hardware–software safety isolation and redundancy: Regardless of how the upper-layer model evolves, the underlying steering and braking systems must retain a hardware-level safety foundation that is independent of the software reasoning layer. If model decisions diverge or the system malfunctions, the vehicle must be able to trigger a deterministic fallback and degradation strategy.

6. Conclusion

Basic control tests in closed environments demonstrate the early potential of general-purpose models to control physical-world behavior. At the same time, they clearly expose major limitations in vehicle-dynamics latency, deterministic spatial measurement, and automotive safety requirements.

Autonomous-driving companies are not without a path forward, and general-purpose models are not simple replacements for existing driving systems. Instead, the industry is moving toward a new consensus: future autonomous-driving systems will combine general-purpose multimodal cognition with high-precision vehicle-dynamics control.

A layered fast–slow architecture can convert the cognitive intelligence of advanced models into constraints and planning boundaries, while a highly responsive, specialized policy performs the physical execution. This division of responsibilities is essential for bridging the engineering gap between laboratory demonstrations and production-grade autonomous driving.