General-Purpose LLMs Enter Autonomous Driving: Paradigm Shift and Engineering Deployment Path

1. A Signal Worth Taking Seriously

Recently, an independent evaluation organization conducted a public test: several general-purpose large models were given direct control of a production passenger car through tool-calling interfaces, completing tasks such as starting, navigating around traffic cones, reversing, and parking in a closed course—approximately 134 meters in total. The best-performing model completed the course in about 5 minutes and 22 seconds on its second attempt, averaging roughly 1.5 km/h; other models achieved completion rates ranging from single digits to over forty percent. During the same period, on a robotics simulation benchmark, general-purpose models without dedicated robot training also achieved top-tier results.

To be fair, the engineering reference value of this test is limited: closed course, low speed, simple routes, multiple attempts allowed, no complex traffic flows, no random pedestrians, no continuous urban road driving. The evaluation organization itself acknowledged that the tasks were “all very simple.”

But its symbolic significance cannot be ignored: a general-purpose large model with no task-specific driving training, for the first time, closed the loop through a tool-calling interface to control a real car and complete a full perception–decision–execution sequence. This marks the transition of “general-purpose large models can control real vehicles to complete driving tasks” from speculation to observable fact.

For the autonomous driving industry, this is not a stunt but an alignment of coordinate systems: the industry previously debated “how much more can modular pipelines be optimized”; now it must answer “when and how will the end-to-end large model paradigm replace them.”

2. Two Coordinate Systems: The Gap Between Current Capabilities and Production Requirements

Placing this test back onto real production autonomous driving roads, the gap between the two coordinate systems becomes immediately clear.

Current capability boundaries of general-purpose large models:

• Operating environment: closed course, no social vehicles
• Speed range: 1–2 km/h magnitude
• Task complexity: fixed routes, few static obstacles
• Reliability: multiple attempts allowed, single-attempt success rate unstable
• Latency: single-step decision time far exceeds real-time control requirements

Current production autonomous driving system level:

• Operating environment: open urban roads, mixed traffic flows
• Speed range: average 20–30 km/h, covering highway scenarios
• Reliability metrics: leading solutions have achieved hundred-kilometer-level mean distance between interventions (MPI)
• Usage scale: used daily by millions of users for commuting
• Safety constraints: redundancy design under dual constraints of functional safety and safety of the intended functionality

The gap between the two is not a “version number” gap but an engineering constraint gap. For large models to enter production autonomous driving, four hard thresholds must be crossed:

1. Real-time performance: Vehicle control loops require millisecond to hundred-millisecond responses, while large model inference (especially long-chain reasoning) has latency an order of magnitude higher. This means model distillation, quantization, speculative decoding, and even splitting “slow thinking” from “fast reaction” into dual systems.
2. Determinism: Production systems require interpretable, reproducible, and accountable decision chains, while large model outputs are probabilistic. Constrained decoding, safety guardrails, and formal verification layers are needed to close the gap.
3. Long-tail coverage: General models’ driving common sense comes from internet corpora and lacks distributed representations of China-specific road conditions (complex intersections, mixed non-motorized traffic, construction lane occupations). This must be filled through proprietary data closed loops.
4. Compute cost: Vehicle-side compute is limited; cloud training costs are high. End-cloud collaborative inference分工 and tiered model deployment are mandatory questions.

3. The Essence of the Paradigm Shift: From “Rule Pipelines” to “Data-Driven Closed Loops”

The key to understanding this impact is not comparing scores of a particular model but understanding the structural differences between two R&D paradigms.

Traditional modular paradigm: perception → prediction → planning → control, with each module optimized independently, relying on manual rules and scenario libraries to cover long tails. Advantages: controllable, debuggable, clear responsibility boundaries; bottleneck: marginal cost of long-tail scenarios keeps rising, and rule entropy increase leads to exponential maintenance burden.

End-to-end large model paradigm: a unified network directly maps sensor inputs to driving behavior outputs, absorbing long tails through data scale and model capacity. Advantages: high ceiling, concise paradigm; challenges: data quality and closed-loop efficiency determine success, and interpretability and safety validation systems need rebuilding.

A simplified pseudocode can illustrate the R&D loop differences between the two paradigms:

# Paradigm A: modular pipeline, long tail covered by rule patches
def modular_pipeline(sensors):
    perception = detect_and_track(sensors)
    prediction = predict_agents(perception)
    plan = rule_based_planner(prediction, scenario_library)
    return control(plan)   # new long tail -> new rules/scenarios
# Paradigm B: data-driven closed loop, long tail covered by data reflow and retraining
def data_driven_loop(fleet_data, model):
    hard_cases = mine_long_tail(fleet_data)      # automatic hard case mining
    pseudo_labels = auto_label(hard_cases)       # auto labeling / simulation replay
    model = retrain(model, hard_cases, pseudo_labels)
    return validate_and_deploy(model)            # shadow mode validation then gradual rollout

The core competitiveness of Paradigm B lies not in the model architecture itself but in data flywheel speed: vehicle-side upload → hard case mining → auto labeling → simulation replay → retraining → shadow validation → gradual release. The automation level and iteration cycle of this chain are the watershed for transformation success.

4. Transformation Paths: Three Options and Phased Deployment

For autonomous driving companies that have not yet completed the paradigm transformation, there are roughly three routes, each with its own applicable conditions.

Route 1: Full-stack end-to-end, one step to the finish.
Directly build a unified large model, training end-to-end from sensors to control outputs. Suitable for teams with large data volumes, sufficient compute reserves, and short organizational decision chains. Risk: long validation system rebuild cycle and short-term production delivery pressure.

Route 2: Dual-system parallel, gradual replacement.
Retain the mature modular pipeline as a safety fallback, overlay a large model “co-pilot” for decision suggestions, compare outputs through shadow mode, and gradually expand model takeover ratio. This is the lowest-engineering-risk path and the realistic choice for most teams. The key is designing evaluation alignment mechanisms: offline metrics, simulation pass rates, and shadow mode consistency rates must converge in sync.

Route 3: Focus on data closed loops, build infrastructure first.
Rather than rushing to replace models, first build data upload, hard case mining, auto labeling, and simulation replay infrastructure so existing pipelines also benefit from data dividends, stockpiling ammunition for subsequent paradigm switching. Suitable for teams with reasonable data scale but weaker engineering capabilities.

Phased deployment recommendations:

• Phase 1 (0–6 months): Establish unified data formats and upload channels; build hard case mining and auto labeling pipelines; run the minimal end-to-end model closed loop in simulation environments.
• Phase 2 (6–18 months): Dual-system parallel, accumulating comparison data through shadow mode; establish safety guardrails and constrained decoding layers for large model outputs; complete latency and compute budget assessments for on-device inference.
• Phase 3 (18–30 months): Gradually expand model takeover ratio by scenario; complete functional safety and safety of the intended functionality arguments; form a weekly iteration rhythm of “data–training–validation–release.”

5. Unavoidable Engineering Details

Data closed loops: The essence of transformation is making data the primary means of production. This requires unified definition of vehicle-side trigger strategies (hard braking, takeover, close cut-in, etc.), establishing upload priority queues, and controlling bandwidth and storage costs. The cloud can use object storage to retain raw data, message queues to decouple upload and processing chains, container platforms for elastic scheduling of training and simulation tasks, and logging and monitoring services to ensure pipeline observability—these are general-purpose capabilities that can be evaluated on demand during selection.

Compute scheduling: Training tasks have obvious tidal characteristics, requiring elastic compute pools and priority scheduling strategies to prevent simulation replay and model training from competing for resources.

Safety redundancy: Large model outputs must pass through an independent safety verification layer, including physical feasibility checks, traffic rule constraints, and collision risk fallbacks. Redundancy is not distrust of large models but engineering discipline for production systems.

Organizational adjustment: Paradigm shifts often accompany team restructuring—the division boundaries between perception, planning, and control are broken, replaced by three vertical capability lines: data engineering, training platforms, and validation systems. Without organizational change, paradigm transformation is difficult.

6. Window Assessment and Action Recommendations

The market landscape offers another observation angle: the current high-level autonomous driving market share is divided among multiple players, with no absolute monopoly yet formed. This means that on one hand, reshuffling has just begun and transformers still have room; on the other hand, teams that complete the paradigm switch first will widen the gap with faster iteration speeds, and latecomers’ catch-up costs will rise over time.

A pragmatic window assessment: the next two to three years are the critical period for paradigm switching. The criterion is not “whether an end-to-end model has been released” but three quantifiable indicators—whether the data closed-loop iteration cycle has entered the weekly level, whether shadow mode consistency rates are continuously converging, and whether on-device inference latency and compute meet production constraints.

Action recommendations in priority order:

1. Build data infrastructure first—this is the common prerequisite for all routes;
2. Use dual-system parallelism to control risk, not pursuing one-step completion;
3. Place validation system rebuilding at equal importance to model R&D;
4. Start engineering assessments for on-device inference early to avoid situations where model capabilities are ready but deployment conditions are not met.

General-purpose large models controlling real cars through closed-course tests is only the prologue. The real competition is who can spin the data flywheel, build the safety validation system, and chew through the engineering constraints. Paradigm shifts are not marked by a single demonstration but by a reproducible, iterable, and producible engineering chain as the endpoint.