Engineering Best Practices for Large Model Training and Inference Serving
This guide covers the practical engineering work behind training and serving large models: estimating effective training time under hardware failures, tuning checkpoint intervals, interpreting MFU correctly, building evaluation suites that reflect real capability, understanding the trade-offs of inference optimizations such as quantization, paged KV cache, speculative decoding, and prefix caching, handling autoscaling cold starts, and following a rollout checklist with SLA metrics, regression monitoring, cost accounting, and troubleshooting procedures.
1. Training: Start by Estimating Effective Training Time
With limited compute, training runs often last weeks to months. What determines project success is usually not peak FLOPS but the fraction of total time spent on effective training. This fraction can be roughly expressed as:
Effective ratio = 1 − (downtime + checkpoint write overhead + rollback loss after failures) / total time
Here is a hypothetical example to illustrate the estimation method. Assume a single GPU has a mean time between failures (MTBF) of 50,000 hours (a placeholder value—replace it with your team’s historical data), the cluster has 512 GPUs, and failures are independent across GPUs. The cluster-level MTBF is about 50,000 / 512 ≈ 98 hours, meaning an interruption roughly every four days.
The checkpoint interval balances two costs. More frequent checkpoints reduce the amount of rolled-back work after each failure but increase storage write overhead. If you checkpoint every 2 hours and each write takes 6 minutes, write overhead is about 5%. Each failure rolls back half an interval on average (60 minutes), plus 10 minutes of recovery time, so each 98-hour cycle loses about 70 minutes—roughly 1.2%. Together, the effective ratio drops by about 6%. Checkpoint write time depends on storage bandwidth and model state size, so the same strategy can produce very different results on different clusters.
Regarding recovery time, “minute-level recovery” is a goal, not a default outcome. Actual recovery time consists of failure detection, node isolation, image pulling on the replacement node, loading weights and optimizer states from the checkpoint, and re-establishing communication groups. Record timestamps for every failure, compute P50 and P90 statistics, and only then decide whether hot standby nodes are needed.
MFU (Model FLOPS Utilization) is defined as the actual model computation achieved divided by the hardware’s theoretical peak. Teams do not agree on what counts as “model computation”: some include recomputation, others do not; some count communication wait time in the denominator, others do not. Before comparing MFU numbers, confirm that the definitions, sequence lengths, and model architectures are consistent. Cross-project MFU comparisons have limited meaning; before-and-after comparisons under the same configuration are far more reliable.
Data and training schedules also need early validation. Learning rate warmup, data mixing ratios, and curriculum design have significant impact. Run ablation experiments on small models first, and only scale to full size once the configuration is stable.
2. Model Capability: Define Improvement Direction with Evaluation Sets
A common problem in model iteration is relying on a single leaderboard score, which leads to poor performance in real business scenarios. A more robust approach is to design separate evaluations for each capability category and keep a private evaluation set that is never used for tuning.
Public benchmarks have value. For code, consider HumanEval and LiveCodeBench; for math, GSM8K and MATH. But public datasets may already appear in pretraining corpora, inflating scores. Public benchmarks are therefore only suitable for coarse screening; final decisions should rely on self-built data.
Long-chain reasoning and tool calling have characteristic failure modes that evaluations should specifically cover:
- Tool parameter type errors, such as passing a string to a field that requires an integer;
- Repeatedly calling the same tool without using the previous return value;
- Failing to verify an incorrect intermediate result before producing the final answer;
- Losing constraints mid-task, such as user-specified units or time zones.
Long-context capability should also be evaluated in separate dimensions. Low perplexity only shows the model models language well—it does not mean it can accurately retrieve a specific passage from a long document. Research has found that models typically use information in the middle of the context less effectively than at the beginning or end (Liu et al., 2023, “Lost in the Middle”). Position encoding extrapolation methods need separate validation at the target length, and retrieval-style tasks and multi-passage integration tasks should be tested independently.
Alignment training requires monitoring two error types simultaneously: failing to refuse what should be refused, and refusing what should not be refused. Tracking only one type makes it easy for the model to drift toward the other extreme. In multilingual scenarios, safety evaluations should use samples in the target language, not just translations of English samples.
3. Inference Serving: Preconditions and Costs of Each Optimization
There are many optimization techniques for serving, but each has preconditions. The table below summarizes common trade-offs; concrete numbers must be measured on your own model and traffic.
| Technique | Best for | Main benefit | Main risk | How to validate |
|---|---|---|---|---|
| 8-bit weight quantization | General chat, summarization | Lower memory footprint; larger models fit on one GPU | Slight accuracy loss | Compare scores on the same evaluation set before and after |
| 4-bit weight quantization | Cost-sensitive workloads tolerant of minor accuracy loss | Further memory reduction | More visible degradation on math and code tasks | Per-task evaluation; check consistency of long outputs |
| Paged KV cache | High concurrency with varied request lengths | Less memory fragmentation, higher concurrency | Preemption and recomputation under memory pressure cause latency jitter | Monitor KV cache occupancy and preemption counts |
| Speculative decoding | Predictable output such as code completion, templated text | Lower per-request generation latency | Can be slower than normal decoding at low acceptance rates | Measure acceptance rate and end-to-end latency |
| Prefix caching | Fixed long system prompts, multi-turn dialogue | Lower time-to-first-token and less repeated computation | Cache invalidates when templates change; potential information leakage across tenants | Cache hit rate; run cross-tenant isolation tests |
Whether quantization accuracy loss is acceptable depends on the task. A safe default is 8-bit quantization, with separate evaluation for math and code tasks. After quantization, also check long-form generation for repetition, truncation, or formatting errors—issues that short-sample evaluations rarely expose.
Speculative decoding gains depend on acceptance rate. The closer the draft model’s output distribution is to the target model’s, and the more predictable the task, the higher the acceptance rate. Open-ended creative tasks usually have low acceptance rates, where speculative decoding may slow down the service. Under high-concurrency batching, gains also shrink, so test throughput under traffic close to production load before rollout.
Prefix cache keys must include the complete prefix and the template version. Once a template changes, old cache entries must be invalidated, or the model will generate content in the wrong context. In multi-tenant scenarios, also verify that timing differences cannot leak one user’s prompt content to another.
Autoscaling must handle cold starts. Loading large model weights can take tens of seconds or longer, and scaling requests cannot relieve pressure until new instances are ready. Below is an example scaling policy with cooldown and a sliding window:
def autoscale(current, window_util, state, cfg):# window_util: GPU utilization sampled every 30s over the last 5 minutesavg_util = sum(window_util) / len(window_util)# No adjustments during cooldown to avoid oscillationif state.in_cooldown():return currentif avg_util > cfg.scale_out_util:target = min(current + cfg.step_out, cfg.max_replicas)elif avg_util < cfg.scale_in_util:target = max(current - cfg.step_in, cfg.min_replicas)else:return currentstate.start_cooldown(cfg.cooldown_seconds)return target
GPU utilization alone does not fully reflect queuing. While requests wait in the queue, GPUs may still be busy, but P99 latency has already breached the target. Collect queue length and P99 latency as well, and include them in scale-out decisions. The minimum replica count should handle baseline traffic and account for request spikes during cold starts.
4. Rollout Checklist
The following steps apply to most enterprise AI engineering teams and can be executed in order.
- Build evaluation sets. Extract representative samples from real logs, at least 200 per task category. Stratify by task type and reserve a portion that is never used for tuning.
- Define SLA metrics. Common metrics include time to first token (TTFT), average inter-token latency (TPOT), throughput (tokens/s), availability, and cost per thousand tokens. Every metric should specify its statistical window and percentile.
- Set up regression monitoring. Every model version or inference engine change must run the same regression suite. Block releases that fail thresholds instead of fixing issues after the fact.
- Calculate costs. Unit cost can be computed as “GPU cost per hour ÷ effective output tokens per hour.” The denominator should count only effective output, excluding idle instances and failed requests.
- Design migration and rollback paths. Roll out new versions to a small traffic percentage first, and run shadow-traffic comparisons against the old version. Rollback should have a clear time objective, with the previous version’s weights and images retained.
When a metric misses its target, troubleshoot in the following order:
- Latency increase: check queue length and KV cache occupancy first, then whether preemption occurred, and finally batching parameters;
- Throughput drop: first confirm whether instances are cold-starting or being evicted, then check whether quantization and speculative decoding toggles are in the expected state;
- Cost increase: first look at the idle instance ratio, then whether average output length has changed, and finally whether the cache hit rate has dropped;
- Quality regression: first compare model versions and template versions, then review per-task evaluation results to find the most degraded category.
Training, evaluation, and serving influence each other. Data mixing ratios in training affect tool-calling stability, quantization schemes affect long-text consistency, and scaling policies affect P99 latency. When optimizing one dimension, teams should monitor all three to avoid trading one metric for another.
