From Pre-training to End-to-End: Engineering Practices for LLM Technical Judgment

    This article examines how to make sound technical judgments about large language models before an industry consensus forms. It argues that shifts in underlying capability tend to precede public consensus by one to two years, and that any early call must rest on verifiable technical anchors rather than narrative. Three anchors are explored in depth: scaling laws as an extrapolatable power-law framework for turning investment into a computable budget, generalization as the dividing line between task-specific models and unified ones, and end-to-end data-driven pipelines versus stitched ones. It then covers serving a model across four layers, model routing for cost-quality balance, and a due diligence checklist for evaluating LLM technology during anti-consensus periods.

    1. The Window Before Consensus: Why the Technology Curve Lags Engineering Reality

    A phenomenon worth noting is that changes in underlying technical capability often precede the formation of industry consensus by one to two years. When capital interest was concentrated on concepts such as energy transition and virtual spaces, autoregressive language models were still running parameter-scaling experiments in a handful of labs, and Scaling Law had not yet become a publicly accepted methodological consensus. This lag is not accidental; it stems from three thresholds rising at the same time:

    • Compute threshold: Training a single observable scale jump requires a 10,000-GPU-class cluster, high-bandwidth interconnect, and a stable parallelism strategy—trial and error is out of reach for ordinary teams.
    • Data threshold: General capability depends on governing trillions of tokens. Deduplication, decontamination, and quality tiering are engineering efforts, not algorithmic ones.
    • Evaluation threshold: Capability growth cannot be captured by a single metric. Traditional leaderboards are easily contaminated by data, and trustworthy out-of-distribution (OOD) evaluation is lacking.

    For these reasons, any “half a step ahead” judgment must rest on verifiable technical anchors rather than narrative. The three anchors below still form a working judgment framework today.

    2. Anchor One: Scaling Laws—Constraining Investment with an Extrapolatable Power Law

    The core of scaling laws is an empirical power-law relationship: among parameter count, data volume, and training compute, loss falls as a power law that can be extrapolated as all three grow. Their engineering value lies not in prediction accuracy but in turning uncertain investment into a computable budget.

    Illustrative code (it only conveys the fitting idea, not the interface of any specific implementation):

    1. import numpy as np
    2. def fit_power_law(params, loss, loss_inf=1.6):
    3. """
    4. L(N) ~= (N / A) ** (-alpha) + L_inf
    5. Fit alpha from small-scale experiments, then extrapolate the expected loss at the target scale.
    6. """
    7. x = np.log(np.asarray(params, dtype=float))
    8. y = np.log(np.maximum(np.asarray(loss, dtype=float) - loss_inf, 1e-6))
    9. slope, intercept = np.polyfit(x, y, 1)
    10. return {"alpha": -slope, "A": float(np.exp(-intercept))}
    11. # Three small-scale observation points are enough to give an extrapolation range
    12. obs = fit_power_law(params=[1e8, 3e8, 1e9], loss=[3.1, 2.85, 2.62])
    13. print(obs["alpha"])

    Three pitfalls to watch in practice:

    1. The irreducible loss term: Setting L_inf directly to 0 systematically overestimates extrapolation gains and lets the compute budget spiral out of control.
    2. Compute-optimal allocation: For the same compute budget, there is an optimal ratio between parameters and training tokens; when the ratio is off, the marginal return of adding parameters decays quickly. This is the technical answer to the “stack parameters vs. stack data” debate.
    3. The reverse constraint of inference cost: The training-side power law looks appealing, but online inference cost grows roughly linearly with parameters. To judge whether a technical route is deployable, you must plot both the “training cost curve” and the “per-request cost curve” and find where the two intersect.

    3. Anchor Two: Generalization—The Key Dividing Line from Task-Specific to Universal

    The word most easily misread in early judgments is “generalization.” It is often treated as a marketing adjective, but in engineering it has a precise meaning: the same set of weights, with no architectural change and no targeted fine-tuning, can produce usable output on tasks it has never seen.

    The way to judge this is not a single-point demo but the transfer curve as the number of few-shot examples varies:

    1. TASKS = ["qa", "summarize", "translate", "code", "math", "structured_extract"]
    2. def generalization_matrix(model, tasks=TASKS, shots=(0, 1, 5, 10), n=200):
    3. """Measure cross-task transfer performance under the same model and decoding configuration."""
    4. matrix = {}
    5. for k in shots:
    6. row = {}
    7. for task in tasks:
    8. row[task] = model.evaluate(
    9. prompt=build_prompt(task, k), # k-shot example injection
    10. n_samples=n,
    11. metric=default_metric(task),
    12. )
    13. matrix[k] = row
    14. return matrix
    15. def is_truly_general(matrix, min_delta=0.02):
    16. """True generalization: most tasks are monotonically non-decreasing as shots increase;
    17. a single-point rise is treated as an overfit demo."""
    18. wins = 0
    19. for task in TASKS:
    20. curve = [matrix[k][task] for k in sorted(matrix)]
    21. if all(b >= a - 1e-9 for a, b in zip(curve, curve[1:])) and curve[-1] - curve[0] > min_delta:
    22. wins += 1
    23. return wins >= len(TASKS) * 0.6

    Three evaluation disciplines that go with it:

    • Contamination isolation: Run n-gram overlap detection between the evaluation set and the training corpus, and remove overlapping samples before testing; otherwise the scores are not trustworthy.
    • Out-of-distribution testing: Keep a set of tasks that clearly differ from the training distribution (e.g., long documents, low-resource languages, structured extraction). Only out-of-distribution performance can separate “memorization” from “generalization.”
    • Reproducibility of emergent capabilities: Some capabilities appear only near specific scales, so multi-point sampling is needed to confirm they are not random fluctuation or the accidental result of prompt tuning.

    4. Anchor Three: End-to-End Data-Driven—Why It Beats a Stitched Pipeline

    A traditional dialogue system is a stitched pipeline: speech recognition → semantic parsing → dialogue management → generation → post-processing. Each module is optimized in isolation, errors amplify stage by stage, and the interfaces between modules cap the system’s ceiling. The point of an end-to-end, data-driven approach is to absorb every stage with a single objective function, shifting the optimization target from “module accuracy” to “the final experience.”

    The direct benefits are consistent with a logic repeatedly validated in autonomous driving:

    • Errors no longer cascade and amplify; long-tail scenarios can be solved by adding data rather than piling up rules;
    • The iteration unit shifts from “changing code” to “changing data,” letting the same architecture keep absorbing new scenarios;
    • The evaluation standard is unified, so diagnosis shifts from “which module broke” to “which type of sample is not covered.”

    Data mixing therefore becomes a core engineering asset rather than an accessory:

    1. MIX = {
    2. "web_general": 0.45, # general language capability
    3. "code": 0.15, # structured and reasoning capability
    4. "math_reason": 0.07, # symbolic and multi-step reasoning
    5. "instruction": 0.20, # instruction following, determines usability
    6. "preference": 0.08, # preference alignment, determines subjective experience
    7. "safety": 0.05, # safety and refusal boundaries
    8. }
    9. PIPELINE = [
    10. ("normalize", "Unify encoding and templates, remove garbled and truncated samples"),
    11. ("dedup", "Document-level + paragraph-level near-deduplication, suppress repeated memorization"),
    12. ("decontam", "Overlap detection against the evaluation set and removal, ensure trustworthy scores"),
    13. ("filter", "Quality scoring tiering, down-weight low-score samples rather than discarding them outright"),
    14. ("mixture", "Sample by MIX and fix the random seed, ensure experiment reproducibility"),
    15. ("snapshot", "Freeze the data version, bind training tasks to data snapshots one-to-one"),
    16. ]

    The key point: snapshot and a fixed random seed are prerequisites for long-term iteration. Without data snapshots, any “it got better / it got worse” conclusion cannot be attributed, and experiments degrade into guesswork.

    5. Engineering Deployment: Turning Model Capability into an Operable System

    Once the technical anchors hold, the real difficulty moves to serving. A usable model service must cover at least four layers, and each should preferentially reuse general cloud product categories (object storage, container platforms, message queues, log services, monitoring and alerting) rather than building them from scratch:

    1. Access and orchestration layer: a unified request gateway, authentication, quotas, and rate limiting; expose different-scale models as switchable backends.
    2. Inference service layer: containerized deployment, elastic scaling, batching, and KV cache reuse; long-context scenarios need dedicated memory and sharding strategies.
    3. Data and asset layer: corpus and model versions stored in object storage, training and inference jobs run on a container platform, and experiment metadata kept in a separate database.
    4. Observability layer: a log service carries requests and sampled outputs, while monitoring and alerting cover latency percentiles, error rate, token consumption, and refusal rate.

    In the capability systems of general cloud platforms such as Baidu AI Cloud, these four categories are essentially offered as standard cloud services, so a team’s main work is designing sensible layering and thresholds rather than rebuilding infrastructure.

    Model routing is the core lever for balancing cost and quality:

    1. TIERS = [
    2. {"name": "small", "max_ctx": 8_000, "rel_cost": 1, "p95_ms": 400},
    3. {"name": "mid", "max_ctx": 32_000, "rel_cost": 6, "p95_ms": 1500},
    4. {"name": "large", "max_ctx": 128_000, "rel_cost": 20, "p95_ms": 4000},
    5. ]
    6. def route(req):
    7. if req.exact_cache_hit: # exact cache
    8. return ("cache", req.cache_key)
    9. if req.semantic_cache_hit: # semantic cache, requires a similarity threshold
    10. return ("cache_semantic", req.semantic_key)
    11. for t in TIERS:
    12. if req.est_tokens <= t["max_ctx"]:
    13. break
    14. else:
    15. return ("degrade", summarize_then_answer(req)) # summarize overly long input first
    16. if t["name"] == "small" and req.confidence_hint < 0.5:
    17. return ("mid", req) # escalate low-confidence requests to the model
    18. return (t["name"], req)

    The cost and quality guardrails that go with it:

    • Cache first: high-frequency repeated requests (especially structured extraction and fixed prompt templates) are absorbed through exact/semantic caching, which usually lowers per-request cost substantially.
    • Tiered degradation: set timeout and error-rate thresholds; once triggered, automatically fall back to a smaller model or a cached result, prioritizing availability.
    • Unit economics accounting: budget by “cost per thousand successful requests” rather than “price per token,” which reflects the real business expense.
    • Security boundary: on the input side, detect prompt injection and privilege escalation; on the output side, apply content-safety filtering; and desensitize sensitive fields before they reach the logs.

    6. A Technical Due Diligence Checklist for Anti-Consensus Periods

    Whether you are selecting technology or doing an early evaluation, you can collapse “judgment” into an executable checklist:

    Dimension Question to answer Red flag
    Reproducibility Can results of a similar order of magnitude be reproduced with a public recipe Demo only, no evaluation protocol
    Extrapolation validity How large is the gap between scaling-law extrapolation and actual observation Extrapolation error exceeds an order of magnitude
    Generalization evidence Does it still hold under out-of-distribution and zero-shot conditions Excels only on a single task and single distribution
    Data asset Is the mixing / dedup / decontamination loop closed No data versioning, results cannot be attributed
    Cost curve How does per-request cost change with scale Training is cheap but inference is unaffordable
    Engineering support Are there serving, monitoring, and degradation plans Stuck at scripts and notebooks
    Compliance and security Are data sources, licensing, and output risks controllable Corpus provenance cannot be traced

    7. Conclusion

    Looking back at that curve, what was truly scarce back then was not capital but the judgment to identify technical signals amid noise. This judgment breaks down into three concrete actions: use scaling laws to turn investment into a computable extrapolation, use out-of-distribution evaluation to turn “looks smart” into “verifiable generalization,” and use an end-to-end data loop to turn one-off results into a continuously iterating capability. Stacked together and backed by a scalable, observable, cost-controllable serving engineering, technical judgment no longer stays at the level of narrative. For today’s developers and enterprise users, the value of this method is not in replicating some past early decision, but in having reusable tools to test the next technology curve while it has not yet become consensus.