Autonomous Driving in the Age of General-Purpose LLMs: Is There Only One Path to Transformation?

After general-purpose large models began extending their capabilities into physical tasks, the autonomous driving industry has faced an unavoidable question: if a general-purpose model can control a vehicle through tool interfaces to start, detour, reverse, and park, will the engineering barriers that traditional autonomous driving systems rely on be redefined?

One point must be clarified first: a low-speed demonstration in a controlled environment is not equivalent to mass-production autonomous driving capability. A route of a few hundred meters, simple traffic conditions, low average speed, and permission for repeated attempts are fundamentally different from urban roads, complex traffic flows, unpredictable pedestrians, all-weather operation, and the daily commutes of millions of users.

The symbolic value of such demonstrations is greater than their immediate practical value. They show that general-purpose models may be able to understand tasks, invoke tools, and form action sequences. They also indicate that models participating directly in driving decisions are no longer just a theoretical idea.

For autonomous driving companies, however, the real question is not whether they can win a single demonstration. It is whether they can turn large-model capabilities into a driving system that can be mass-produced, validated, iterated, and safely backed up. If this transition cannot be completed within a limited window, the risk is not simply falling behind in one feature. It is a broader generational gap in technical paradigms, data closed loops, and organizational efficiency.

1. Two Different Coordinate Systems: Demonstration Capability Is Not Production Capability

The driving-related capabilities currently demonstrated by general-purpose models largely exist in a coordinate system defined by controlled environments, low speeds, simple routes, and permission for trial and error. Production autonomous driving systems operate in a very different coordinate system: open roads, continuous operation, high reliability, low intervention rates, and strict safety constraints.

The gap between these systems is not simply a matter of model parameters. It is an engineering-system gap.

First, real-time requirements are different. In production autonomous driving, perception, decision-making, and control must form a closed loop within milliseconds to hundreds of milliseconds. Model inference must meet deterministic latency budgets. A general-purpose model operating in the cloud or through a remote toolchain may have more time to reason and can retry a task multiple times. A vehicle cannot wait for long periods, nor can it depend on stable network transmission for safety-critical actions.

Second, the safety boundary is different. A failed demonstration can be retried. A production failure can create a safety risk. Therefore, the output of a large model cannot directly become a vehicle control command. It must pass through independent safety constraints, redundant monitoring, and failure-degradation mechanisms.

Third, the data distribution is different. Controlled environments contain limited scenarios. Open roads include construction zones, unusual vehicles, temporary traffic arrangements, severe weather, and non-standard behavior by road users. General-purpose models may have advantages in common-sense reasoning, but they may not cover the fine-grained rules and extreme distributions found in real driving environments.

Fourth, the evaluation system is different. Demonstrations are judged by whether a task is completed. Production systems are evaluated across multiple dimensions, including average distance between interventions, hard-braking frequency, ride comfort, traffic efficiency, scenario pass rates, and failure traceability. Without a complete evaluation system, model capability cannot become a deliverable product.

For this reason, the transformation path for autonomous driving companies is not simply to introduce a general-purpose LLM. It is to build a hybrid system composed of a driving-specialized large model, a safety constraint layer, a data closed loop, and vehicle-side engineering.

General-purpose models can provide scene understanding, common-sense reasoning, behavior explanation, and multimodal fusion. Specialized models and safety rules are responsible for real-time performance, determinism, and compliance. These are not replacement relationships. They are layered and collaborative relationships.

2. A Paradigm Shift: From Modular Pipelines to Hybrid Large-Model Architectures

Traditional autonomous driving systems typically follow a modular pipeline: perception, prediction, planning, and control. Its advantages are clear interfaces, well-defined responsibility boundaries, and the ability to validate each module separately. Its limitations are also clear: errors can accumulate across modules, long-tail scenarios often require extensive hand-crafted rules, and the overall system may have limited ability to understand complex semantics and unknown environments.

The core change introduced by large models is the ability to understand scenes at a higher level, generate behavioral candidates, explain decision rationale, and reduce manual rules through data-driven learning. End-to-end models can shorten the decision chain. World models can strengthen simulation and prediction. Vision-language-action models can introduce common-sense knowledge into the driving stack.

However, purely end-to-end black-box approaches still face major challenges in explainability, determinism, and safety validation during mass production.

A more practical path is a hybrid architecture:

Multimodal inputs
   -> Driving-specialized large model: scene understanding, risk identification, behavior candidate generation
   -> Safety constraint layer: vehicle dynamics constraints, collision detection, traffic regulations, comfort thresholds
   -> Trajectory optimization and arbitration: multi-candidate evaluation, redundant switching, minimum-risk strategies
   -> Real-time controller: steering, braking, and propulsion execution

This can be abstracted in pseudocode:

candidates = driving_model(multimodal_input)
safe_candidates = safety_filter(candidates, hard_constraints)
best_trajectory = evaluate_and_optimize(safe_candidates, comfort, rules)
control_command = controller(best_trajectory)

The key principle is straightforward: the large model is responsible for understanding and candidate generation; the safety layer is responsible for constraints and fallback protection; and the controller is responsible for real-time execution.

As model capability improves, the quality of generated candidates can improve. But the safety layer must remain independent of the model. The same black-box system should not both generate and validate a driving action. Only by embedding model capabilities within a verifiable engineering framework can autonomous driving companies gain the generalization benefits of large models while meeting production safety requirements.

3. Data Closed Loops Become the Core Asset: From Collecting Data to Creating Data, Mining Edge Cases, and Validating Models

As autonomous driving becomes more dependent on large models, the importance of data closed loops does not decline. It increases. As model scale grows, data quality, scenario coverage, and evaluation efficiency determine iteration speed.

Autonomous driving companies need an end-to-end loop that covers vehicle-side collection, cloud-based cleaning, automated annotation, edge-case mining, simulation generation, training and evaluation, and staged deployment.

First, shadow-mode operation on production vehicles is a low-cost source of data. While vehicles are operated manually or with driver assistance, they can return critical data such as segments where the model is uncertain, where rules conflict, or what occurred before and after driver intervention.

Second, automated edge-case mining is more important than blindly increasing data volume. Mining strategies should focus on dimensions such as low-confidence predictions, sudden braking, close-range cut-ins, construction detours, and non-standard road users.

Third, simulation reconstruction and synthetic generation can help fill gaps in dangerous and low-frequency scenarios.

Fourth, regression evaluation must cover functionality, safety, comfort, and regulatory compliance. This prevents models from overfitting to a single metric.

A representative workflow looks like this:

Vehicle-side shadow operation
  -> Identify high-uncertainty segments
  -> De-identify and upload data slices
  -> Automated annotation and scene reconstruction
  -> Edge-case repository and simulation augmentation
  -> Model training and regression evaluation
  -> Staged deployment and online monitoring
  -> New edge cases flow back into the loop

Whether this flywheel can operate at high speed depends on the integration of the data platform, training platform, simulation platform, and vehicle-side engineering.

If data, training, and evaluation are distributed across disconnected teams, organizational interfaces will slow the model iteration cycle. Large-model transformation may appear to be an algorithm problem, but at its foundation, it is a problem of data infrastructure and engineering coordination.

General infrastructure categories—such as object storage, message queues, container platforms, logging services, and monitoring and alerting systems—need to be assembled around the training and evaluation loop rather than built as isolated components.

4. Vehicle-Side Inference and Cloud Training: The Triangle of Compute, Latency, and Cost

Deploying large models in vehicles cannot be evaluated only by model performance. It must also account for in-vehicle compute capacity, power consumption, thermal limits, memory bandwidth, and cost.

A production vehicle is not a data center. Vehicle-side inference requires model compression, distillation, quantization, sparsification, operator optimization, and coordination across heterogeneous computing resources.

The division of capabilities between the vehicle and the cloud should be designed according to latency, bandwidth, privacy, and cost.

A reasonable layered approach is as follows:

• Small vehicle-side models or specialized models handle real-time perception and safety-critical decisions.
• Cloud-based large models handle training, simulation, edge-case mining, and offline evaluation.
• Vehicle-cloud collaboration supports map updates, scenario feedback, and staged model rollout.

The vehicle cannot rely on unstable network connectivity to complete a safety-critical closed loop. Likewise, the cloud cannot replace real-time vehicle-side control.

Typical engineering constraints include strict latency budgets for perception and decision pipelines, deterministic output requirements for control commands, degradation strategies when model confidence is low, and prioritization of safety functions when vehicle resources are constrained.

These examples describe engineering methods, not performance commitments for any specific product. Each company should establish its own latency budget, resource budget, and safety-degradation matrix, then validate them through stress testing.

5. Safety Validation: The Stronger the Model, the More Important an Independent Safety Layer Becomes

When large models enter driving systems, safety validation becomes more difficult. Models are probabilistic and uncertain, and their outputs cannot be exhaustively enumerated one by one.

Safety responsibility therefore cannot be assigned entirely to the model. A multilayer protection system is required.

First, there must be a hard-constraint layer. Regardless of the trajectory proposed by the model, it must pass collision detection, vehicle dynamics constraints, traffic regulation constraints, and comfort-threshold filtering.

Second, there must be a redundant monitoring layer. Independent monitoring modules should continuously evaluate model-output confidence, sensor consistency, and overall system health.

Third, there must be a minimum-risk strategy. When confidence is low, sensors fail, or computing resources become abnormal, the system should degrade safely through actions such as speed reduction, pulling over, or requesting driver takeover.

Fourth, evaluation must be traceable. Every model update should go through scenario-library regression testing, closed-loop simulation, controlled-site testing, and supervised on-road validation. Decision logs must be retained for later analysis and traceability.

Safety validation must also include data compliance and privacy protection. Vehicle-collected data should be de-identified and uploaded only when necessary. Training data should be auditable. Model updates should be released gradually and support rollback.

Large-model transformation does not bypass functional safety processes. It brings models into a stricter safety engineering framework.

6. Organization and Business: Not a Single-Team Algorithm Upgrade, but an End-to-End Rebuild

The transition of autonomous driving companies toward a large-model paradigm cannot be achieved simply by having algorithm teams add more parameters. It requires coordinated work across data platforms, training platforms, simulation and evaluation, vehicle-side engineering, safety and compliance, operations, and monitoring.

Organizational objectives must shift from “delivering modules” to “operating models and data flywheels.” Responsibilities need to be explicit: who mines edge cases, who builds evaluation datasets, who owns safety fallback mechanisms, and who manages staged deployment and rollback.

From a business perspective, the market has not yet fully consolidated, which suggests that industry reshuffling is still at an early stage. The future may not leave only one company, but it is likely to leave one dominant technical direction: large-model capabilities as the core, data closed loops as the engine, safety engineering as the foundation, and real-time vehicle systems as the delivery vehicle.

Companies that fail to build this foundation may not be defeated by one isolated feature. Instead, they may fall further behind in iteration speed, scenario coverage, and cost efficiency.

The conclusion is clear: a general-purpose large model controlling a real vehicle in a closed task is an important signal, but it is not the final answer. The path forward for autonomous driving companies is not simply to become “large-model companies.” It is to engineer large-model capabilities into production-ready driving systems.

The window may be only two or three years: first establish a hybrid architecture, then build an effective data closed loop, then strengthen vehicle-side inference and safety validation, and finally develop organization-wide iteration capability.

The companies that can transform general capabilities into safe, real-time, and verifiable driving productivity will have the best chance of competing in the next phase of autonomous driving.