LLM Training and Inference Performance Optimization: A Practical Guide
This guide traces the technical evolution from statistical language models to today's large language models, then focuses on the key algorithms and engineering techniques that drive performance on both the training and inference sides. It covers parallelization strategies, memory optimization, mixed-precision and low-precision training, communication tuning, MoE and long-context training, and training stability, followed by inference-side topics such as continuous batching, KV cache management, speculative decoding, and quantization. It closes with an optimization methodology and a reusable practice checklist, helping readers decide where each technique applies and how to deliver optimization gains reliably to production.
1. Introduction: From Language Models to Large Models
A language model (LM) characterizes the distributional regularities of natural language through statistical and neural-network methods. It is the foundational capability behind a wide range of NLP applications, including search ranking, machine translation, text generation, and intelligent question answering. Over more than two decades, language models have gone through four stages: statistical language models, neural representation learning methods, the pretrain-then-fine-tune paradigm, and today’s era of “large models” defined by hundreds of billions to trillions of parameters.
From an industrial standpoint, this evolution is not merely an expansion in parameter count; it is also an upgrade of the entire engineering system. On the training side, it involves parallel partitioning, communication scheduling, and fault recovery across ten-thousand-GPU clusters. On the inference side, it requires balancing time-to-first-token, generation speed, and concurrent throughput under limited memory and bandwidth. This article first reviews the technical lineage from language models to conversational large models, then examines the key algorithms and engineering techniques for performance optimization on the training side and the inference side separately, and finally provides a reusable practice checklist.
2. Technical Lineage: From Language Models to Large Models
2.1 The Essence of a Language Model: Modeling Sequence Probability
Mathematically, a language model assigns a probability to a token sequence:
P(X) = P(x1, x2, ..., xn) = Π P(xt | x1, ..., x(t-1))
By the chain rule, the probability of the whole sequence decomposes into the product of conditional probabilities at each position given the preceding context. Therefore, “predicting the next token given the previous context” is the most fundamental and most general task form of a language model. Classic approaches include:
- N-gram models: estimate conditional probabilities from word-frequency statistics, but suffer from data sparsity and difficulty modeling long-range dependencies;
- Embedding-based sequence models: map discrete symbols to low-dimensional dense vectors first, then model context with structures such as recurrent networks;
- Self-attention architectures: replace recurrence with an attention mechanism, enabling direct interaction between any two positions and making the architecture naturally suited to parallel computation.
The quality of a language model is usually measured by perplexity. It approximates the average number of candidate tokens the model must distinguish between at each position: the lower the value, the more accurately the model captures the regularities of language.
2.2 Scaling Laws: Predictable Performance Gains with Scale
Around 2020, a series of large-scale experiments in the industry revealed a fairly consistent pattern: when model parameter count, training data volume (number of tokens), and compute are scaled up together, the model’s loss decreases smoothly in a power-law fashion. This is known as the Scaling Law, and its significance lies in two points:
- Resource allocation can be planned in advance: given a compute budget, one can fit optimal parameter-to-data ratios at a small scale and extrapolate them to large-scale training;
- Large-model training shifts from “experience-driven” to “budgetable and predictable,” and baseline experiments before training become a standard part of the workflow.
As scale continues to grow, some capabilities (such as multi-step reasoning and in-context learning) “emerge” abruptly once scale crosses a certain threshold, a phenomenon known as emergent ability. This also explains why large models are far stronger than small models on general tasks: small models can usually solve only a single task, whereas large models acquire fairly general capabilities under the same prediction objective.
2.3 Architecture Lineage: Three Main Lines and Hybrid Routes
Across the diverse ecosystem of open-source and closed-source models, industry surveys often map the lineage among mainstream models as an “evolutionary tree.” Its value lies in three points:
- Structural commonality: the vast majority of mainstream models are built on the self-attention Transformer architecture, with only a few early branches diverging from it;
- Three coexisting paradigms: covering different task forms such as understanding, generation, and conditional generation;
- Continuity of research: foundational model research and architectural innovation are the prerequisites for every subsequent leap in scale.
| Type | Structural characteristics | Attention pattern | Typical training objective | Capability positioning |
|---|---|---|---|---|
| Encoder-Only | Stacked bidirectional encoding layers | Bidirectional full attention | Masked language modeling | Text classification, information extraction, semantic retrieval |
| Decoder-Only | Stacked autoregressive layers | Self-attention with a causal mask | Next-token prediction | Conversational generation, content creation, code generation |
| Encoder-Decoder | Encoder and decoder connected in series | Bidirectional in the encoder, causal in the decoder, linked by cross-attention | Sequence denoising / span reconstruction | Translation, summarization, text editing |
In recent years, to mitigate the problem that attention computation grows quadratically with sequence length, some models introduce linear attention or state-space sequence modeling between attention layers. These hybrid architectures represent a new line of exploration beyond the “pure attention route,” and are no longer comparable to the hybrid structures of the early recurrent-network era.
From an industrial standpoint, the primary driver of architectural evolution is efficiency: sparse activation expands parameter capacity while controlling compute; compressive attention reduces the cache overhead of long contexts; and hybrid architectures attempt to balance language-modeling capability with linear complexity.
3. Comparing Two Pretraining Paradigms: Bidirectional Understanding vs. Autoregressive Generation
3.1 Masked Language Modeling: Learning Context Representations Through “Fill-in-the-Blank”
The representative approach for encoder-only models is Masked Language Modeling (MLM): randomly mask part of the tokens in the input sequence (in practice often combined with whole-word masking and sentence-level replacement strategies), and have the model reconstruct the masked content using the context on both sides.
- Bidirectional attention: each position can see both the preceding and following context at once, producing representations that are “globally conditioned”;
- Objective function: maximize the probability of correctly reconstructing the masked positions;
- Capability profile: strong at understanding and discriminative tasks, and can be fine-tuned by attaching a classification head on top.
3.2 Autoregressive Language Modeling: Learning Generation Through “Continuation”
Decoder-only models use Causal Language Modeling (CLM), predicting the next token strictly from left to right:
# Autoregressive forward pass (illustrative): the output at position t depends only on 0..t-1# During training all positions are computed in parallel (teacher forcing); during inference tokens are generated one by onedef causal_logits(tokens, weights):hidden = embedding(tokens) # [B, T, D]for layer in weights.blocks: # each block has an attention and a feed-forward sublayerattn = causal_attention(hidden, layer) # apply the lower-triangular maskhidden = residual_norm(hidden + attn)hidden = residual_norm(hidden + ffn(hidden))return output_head(hidden) # [B, T, V], where V is the vocabulary size
The key is a lower-triangular causal mask applied on top of the attention matrix:
# Lower-triangular mask (illustrative): ✓ = allowed to attend, · = masked out# rows = current query position q, columns = attended key/value position k# k \ q 0 1 2 3# 0 ✓ · · ·# 1 ✓ ✓ · ·# 2 ✓ ✓ ✓ ·# 3 ✓ ✓ ✓ ✓
During training, teacher forcing is used, so a single forward pass computes the prediction loss for all positions in parallel. During inference, tokens are generated autoregressively one at a time, and each new token must be computed together with the key-value pairs (the Key-Value Cache, or KV cache) of the preceding context. This difference is the fundamental starting point for inference optimization.
3.3 Comparing the Two Paradigms and Choosing Between Them
| Dimension | Masked Language Modeling | Autoregressive Language Modeling |
|---|---|---|
| Context view | Bidirectional | Unidirectional (left context only) |
| Training parallelism | High (masked positions are mutually independent) | High (all positions parallel during training) |
| Inference paradigm | A single forward pass produces output | Requires token-by-token autoregressive generation |
| Suitable tasks | Discrimination, extraction, vectorized retrieval | Open-ended generation, dialogue, code, reasoning |
| Fine-tuning cost | Low, with a simple task head | Low, mainly via instruction fine-tuning |
Almost all mainstream conversational large models today adopt the decoder-only architecture, because the autoregressive objective unifies virtually every task form: anything expressible as text can be turned into “predicting the next token.” Masked models, thanks to their smaller parameter counts, lower inference latency, and lower deployment cost, are still widely used in scenarios such as semantic retrieval, content moderation, and ranking. In practice, paradigm selection should follow the principle of “let the task dictate the paradigm” rather than blindly pursuing generative capability.
4. Conversational Large Models: From a Base Model to a Usable Assistant
4.1 Base Capability: Pretraining Sets the Ceiling
During pretraining, next-token prediction is performed on massive text (often including large amounts of code and multilingual corpora). This stage determines the breadth of the model’s knowledge and the ceiling of its language capability, typically consumes more than 90% of the compute across the entire training cycle, and is the key stage at which long-context capability takes shape.
4.2 Instruction Fine-Tuning: Making the Model “Understand Tasks”
A base model is essentially a “continuation machine,” and when faced with a question it may simply keep fabricating more questions. Supervised Fine-Tuning (SFT) uses “instruction–response” sample pairs for supervised training, computing the loss only on the response portion:
# Illustrative sample construction and loss masking for multi-turn dialoguedef build_sample(system_prompt, turns):segments, labels = [], []segments.append(apply_template(system_prompt)); labels.append(IGNORE) # not counted in the lossfor user_msg, assistant_msg in turns:segments.append(apply_template(user_msg)); labels.append(IGNORE) # user input is not counted in the losssegments.append(apply_template(assistant_msg)); labels.append(1) # only the model output participates in trainingreturn build_loss_mask(segments, labels)
Key points in constructing samples include:
- Set a reasonable system prompt to unify the model’s behavioral boundaries;
- Ensure consistent formatting of role markers and special tokens across multiple dialogue turns;
- Compute the loss only on the response portion, avoiding wasting the model’s learning capacity on imitating user input.
4.3 Alignment Training: Making the Model “Match Preferences”
A model with only instruction fine-tuning may still produce verbose, format-violating, or insufficiently safe responses. The alignment stage introduces human preference signals, with two mainstream routes:
- Reward modeling + reinforcement learning: first train a reward model to rank multiple candidate responses, then optimize the policy with reinforcement learning. The classic approach is a class of Proximal Policy Optimization (PPO) algorithms, whose flow can be summarized in four steps:
Reinforcement learning alignment (general flow)1. Sampling: the policy model generates multiple responses for a prompt2. Scoring: the reward model outputs a scalar reward; a KL penalty can be added to constrain the degree of deviation3. Estimation: the value network estimates state values, combined with rewards to compute the advantage function4. Update: update the policy weighted by advantage, using a clipping mechanism to limit the per-step update magnitude
- Direct Preference Optimization: skips the explicit reward model and directly constructs a loss function from “preferred/dispreferred” paired samples, making the flow simpler and training more stable.
With the development of reasoning models, “reinforcement learning with verifiable rewards” is gradually becoming an important means of strengthening mathematical, code, and logical reasoning abilities. It exploits the fact that answers can be automatically verified (for example, by running unit tests or comparing against ground-truth answers) to construct reward signals, and estimates advantages through within-group relative scoring, eliminating the need for a separate value network.
For reasoning tasks, better answers may appear in only a few samples within a group, so tuning the number of samples, the temperature parameter, and the sampling strategy directly determines the proportion of effective samples for reinforcement learning. At the same time, reasoning-model outputs often contain lengthy chains of thought, placing higher demands on the long-output capacity of the inference service.
5. Training-Side Performance Optimization
5.1 Performance Metrics and Bottleneck Identification
The first step in training optimization is to establish a quantifiable evaluation system. Common metrics include:
- Per-step time and per-GPU throughput: directly reflect hardware utilization;
- Model FLOPs Utilization (MFU): the ratio of effective floating-point operations in training to the hardware’s peak compute. Publicly reported MFU for large-model pretraining is mostly in the 30%–50% range, with the remaining compute consumed by memory-access stalls, communication, and pipeline bubbles;
- Memory usage and peak watermark: determine the trainable model size and batch size;
- Loss-curve health: used to detect numerical overflow and gradient explosion.
A mainstream troubleshooting approach can be aided by the Roofline model: split each computation into “amount of computation” and “amount of data to be moved,” whose ratio is the arithmetic intensity. When the arithmetic intensity is below the hardware’s ratio of “peak compute / peak bandwidth,” the workload is memory-bound and the memory-access path should be optimized first; otherwise it is compute-bound and computational efficiency should be optimized.
5.2 Parallelism Strategies: Splitting the Model and the Data
A single machine’s memory cannot hold all the state of a large model, so the model must be partitioned across multiple GPUs and the data across multiple data-parallel groups. The common dimensions are:
| Parallelism dimension | Partitioning target | Primary communication pattern | Suitable location | Main cost |
|---|---|---|---|---|
| Data Parallel (DP) | Input batches | AllReduce, or ReduceScatter + AllGather | Across nodes | Each GPU must hold the full model state |
| Tensor Parallel (TP) | Individual operators (split by row/column) | AllReduce / AllGather | Within-node high-speed interconnect | Frequent communication, highly bandwidth-sensitive |
| Pipeline Parallel (PP) | Layers split into multiple stages | Send/Recv (point-to-point) | Across nodes | Pipeline bubbles |
| Sequence/Context Parallel (SP/CP) | Sequence dimension | AllGather / ring-style P2P | Long-sequence scenarios | Requires redoing attention pre/post-processing |
| Expert Parallel (EP) | Experts in sparse architectures | All-to-All | Across nodes | Communication stalls under load imbalance |
The basic principles for combining them can be summarized in three points:
- Topology matching: keep tensor parallelism, the most communication-intensive, within the node’s high-speed interconnect domain;
- Layered partitioning: pipeline parallelism passes activations between nodes, with far less communication than gradient synchronization;
- Global coverage: data parallelism sits at the outermost layer and provides linear scaling of overall throughput.
In addition, optimizer-state sharding (each GPU stores and updates only the state corresponding to a subset of parameters) lets data parallelism spread the state memory across the entire parallel group with almost no increase in communication.
5.3 Memory Optimization: Turning “Won’t Fit” into “Fits”
Memory usage in large-model training can be broken down into four categories: model weights, gradients, optimizer states, and activations. With the Adam optimizer and BF16 mixed precision, each parameter requires roughly 16 bytes (weights 2 + gradients 2 + first moment 4 + second moment 4 + FP32 master weights 4), so a 70B model needs about 1.1TB of memory for its states alone. Common techniques include:
- Activation recomputation: retain only the input of each Transformer layer in the forward pass and recompute intermediate activations during the backward pass, trading a small amount of compute for memory. In practice, recomputation can be selective at layer granularity to balance throughput;
- Memory offloading: migrate temporarily unused optimizer states or parameters to host memory/local storage, swapping them in and out on demand;
- Sequence sharding: split long sequences across multiple devices to reduce peak per-GPU activation memory;
- Communication buffer reuse and defragmentation: fixed allocation, ring buffers, and on-demand release reduce memory fragmentation and implicit synchronization;
- Dynamic batch-size scheduling: first set the per-GPU micro-batch size according to the memory budget, then adjust the number of gradient-accumulation steps, rather than blindly increasing the batch size.
5.4 Communication and Parallel Efficiency
Communication optimization focuses on “reducing communication volume, shortening communication time, and hiding communication overhead”:
- Overlapping compute and communication: trigger gradient reduction per bucket during backpropagation so it overlaps with subsequent computation;
- Gradient accumulation: reduce synchronization frequency by merging the gradients of multiple micro-batches before reducing them;
- Sequence parallelism: on top of tensor parallelism, split operators that cannot be tensor-parallelized—such as LayerNorm and Dropout—along the sequence dimension, turning an extra AllReduce into two cheaper collective communications;
- MoE All-to-All optimization: All-to-All grouped by expert requires a sensible grouping order and buffer reuse to avoid communication and computation blocking each other;
- Pipeline scheduling: use interleaved 1F1B (one-forward-one-backward alternation) scheduling and reduce the number of pipeline stages to compress the bubble ratio. The bubble ratio is approximately
(p-1)/(m+p-1), where p is the number of stages and m is the number of micro-batches, so increasing the number of gradient-accumulation micro-batches is a simple and effective measure.
5.5 Mixed-Precision and Low-Precision Training
Mainstream large-model pretraining generally uses BF16 mixed precision:
- Compute precision: matrix multiplications use BF16, whose exponent range is the same as FP32 and is less prone to overflow;
- Master weight copy: weights are maintained in FP32, and updates are performed in FP32, preventing small gradients from being rounded away during repeated accumulation;
- Loss scaling: FP16 scenarios require dynamic loss scaling to prevent gradients from underflowing to zero.
For higher throughput, FP8 training is gradually being adopted in the industry; its key lies in a sound scaling-factor strategy (per-tensor or block-wise scaling) and precision fallback for numerically sensitive layers. The essence of low-precision training is not “getting it to run” but verifying that the convergence curve aligns with the full-precision baseline.
5.6 Operator and Computation-Graph Optimization
- Operator fusion: fuse element-wise operations such as RMSNorm, activation functions, and residual additions into adjacent compute kernels to reduce reads and writes of intermediate results;
- Attention kernels: use precise attention kernels with tiling and recomputation strategies that avoid materializing the full attention matrix, significantly reducing memory-bandwidth pressure;
- Compilation optimization: leverage automatic operator generation and graph-level compilation to reduce kernel-launch overhead and apply static shape specialization;
- Execution-graph capture: freeze the static computation graph into a replayable execution unit to lower scheduling overhead.
5.7 Key Points of Sparse Architecture (MoE) Training
Mixture of Experts (MoE) trades sparse activation for capacity:
- Routing design: Top-k routing with capacity limits, combined with a load-balancing (auxiliary) loss to avoid expert collapse;
- Capacity and dropping: too small a capacity factor drops tokens, while too large a factor wastes memory, so it needs to be tuned according to the training stage;
- Expert-parallel communication: uneven send/receive in All-to-All causes a bucket effect, requiring grouping order to be optimized alongside routing statistics;
- Training stability: abnormal magnitudes of routing logits can trigger loss spikes; the industry commonly uses approaches such as router z-loss to constrain them.
5.8 Long-Context Training
- Positional-encoding extrapolation: based on Rotary Position Embedding (RoPE), extend the processable context length by adjusting rotation frequencies (e.g., position interpolation, frequency scaling);
- Context parallelism: split the sequence across different devices, using ring-style attention communication to exchange key-values;
- Data mixing: long documents need a sufficient proportion, otherwise the model learns only “position” rather than “dependency”;
- Memory bottleneck: the attention cache grows linearly with sequence length, so it must be combined with paged caching and quantization strategies.
5.9 Training Stability and Fault Tolerance
- Numerical monitoring: watch gradient norms, loss spikes, and activation overflow (Inf/NaN), and pinpoint them to specific layers and specific ranks;
- Gradient clipping and learning-rate warmup: prevent early-training divergence;
- State persistence: use sharded saving and asynchronous writes to disk to shorten checkpoint time;
- Fault self-healing: hardware failures are inevitable in large-scale training, so it is advisable to combine health checks, automatic isolation, and elastic reconfiguration of training jobs.
6. Inference-Side Performance Optimization
6.1 The Two Inference Phases and Key Metrics
Large-model inference is divided into two phases:
- Prefill: process all input tokens at once; compute-intensive, and it determines time-to-first-token (TTFT);
- Decode: generate token by token; memory-intensive, and it determines per-token generation latency (TPOT).
Explained from a Roofline perspective: the decode phase reads all weights and the KV cache for every token generated, while the compute scales only with batch size, so single-request decoding is inherently memory-bound. Increasing the concurrent batch size is precisely the key to amortizing memory-access cost over more requests.
| Metric | Meaning | Primary optimization techniques |
|---|---|---|
| TTFT (time-to-first-token) | From request to return of the first token | Chunked prefill, prefix caching, parallel partitioning |
| TPOT (per-token latency) | Time to generate each subsequent token | Larger batch size, weight quantization, KV cache compression |
| Throughput (tokens/s) | Total output per unit time | Continuous batching, separating long and short requests |
| Cache hit rate | Prefix-reuse ratio | Prefix caching, request-routing affinity |
6.2 Batching and Scheduling: From “Queue and Wait for Alignment” to “Continuous Batching”
The naive approach is static batching: all requests in a batch must be generated to the same length before being returned together, forcing short requests to wait for long ones and leaving compute underutilized. Continuous Batching schedules at iteration granularity: after each decode round, finished requests exit and idle slots immediately admit new requests. Its scheduling pseudocode is:
def scheduler_loop(runtime):while not runtime.is_drained():# 1) Admit new requests: chunked prefill may be needed so long inputs don't block decodefor req in pending_queue:if runtime.can_admit(req):runtime.admit(req) # allocate KV cache blocks# 2) Execute one forward pass (mix of prefill and decode)logits = runtime.step()# 3) Process outputs: release the cache if the end token is hit or max length is reachedfor seq in runtime.active:token = sample(logits[seq])if seq.finished(token):runtime.release(seq) # release the cache for reuse by new requestselse:append_token(seq, token)
Further techniques include:
- Chunked prefill: split overly long inputs into multiple chunks and interleave them with decode rounds, preventing a long input from monopolizing compute and “starving” decode requests;
- Prefill/decode disaggregation: the two request types have different compute profiles, so deploy them on separate resource pools that scale independently;
- Priority and preemption: schedule latency-sensitive online requests and offline batch jobs at different levels, suspending low-priority sequences when necessary.
6.3 KV Cache Management: The Main Battleground for Inference Memory
- Paged management: use fixed-size blocks as the allocation unit, requesting and releasing on demand to significantly reduce fragmentation;
- Prefix sharing: identical prefixes such as system prompts, few-shot examples, and long documents can share the same cache across requests, noticeably lowering time-to-first-token;
- KV quantization and compression: reduce the cache from FP16 to INT8/FP8, or introduce low-rank compression at the attention-structure level, trading a small amount of precision for longer contexts and greater concurrency;
- Cache offloading: migrate the caches of inactive sequences to host memory, trading off capacity and latency.
6.4 Parallelism and Decode Acceleration
- Parallelism strategies: tensor parallelism reduces per-GPU memory pressure and per-token latency, while pipeline parallelism boosts the throughput of multiple replicas;
- Speculative decoding: use a small draft model or parallel multi-head prediction to propose candidate tokens, then have the large model verify them in a single pass, merging multiple memory accesses into one forward pass. The speedup depends on the draft hit rate and needs to be tuned to the business distribution;
- Parallel decoding / tree candidates: generate and verify multiple candidate paths at once, suitable for scenarios with a fairly concentrated output distribution;
- Operator level: fuse sampling, penalties, and top-k/top-p filtering to reduce kernel launches and intermediate tensors.
6.5 Quantized Deployment
- Weight quantization: configurations such as W8A8 and W4A16 balance memory and bandwidth, and must go live only after passing precision evaluations (perplexity, task metrics, manual spot checks);
- Calibration data: the calibration set should closely match the real business distribution, otherwise quantization error concentrates in long-tail scenarios;
- Activation quantization: apply per-channel scaling and outlier handling to the activation distribution to prevent a few outliers from dragging down overall precision;
- End-to-end validation: the quantized model must be re-evaluated for long-text generation quality, not just short-text metrics.
6.6 Serving and Observability
Inference performance optimization ultimately has to land in the serving architecture:
- The access layer handles authentication, rate limiting, and request queuing;
- The gateway routes to different instance pools by model version, context length, and priority;
- Latency-sensitive and throughput-oriented workloads are configured with separate batching parameters;
- Full-link instrumentation collects TTFT, TPOT, queue time, cache hit rate, and memory watermark;
- Combined with logging services and monitoring alerts, this forms a closed loop that automatically scales up or removes instances on anomalies.
On a cloud platform such as Baidu AI Cloud, these capabilities are typically delivered by running inference instances on a container platform, storing models and checkpoints in object storage, and relying on logging and monitoring services for observability, forming a general and standardized deployment pattern.
7. Methodology and the Metrics Closed Loop for Performance Optimization
The general optimization workflow can be summarized in four steps:
- Model: use Roofline to determine whether the workload is compute-bound or memory-bound, avoiding blind parameter tuning;
- Decompose: break end-to-end time down to phases (prefill/decode) and modules (attention, feed-forward, sampling, communication);
- Quantify: estimate the theoretical upper bound of each optimization item’s benefit, and prioritize items with high payoff and low risk;
- Validate: establish a fixed evaluation set and benchmark scripts, and compare any change against the baseline for both precision and performance.
Common judgment pitfalls include:
- Looking only at single-request latency while ignoring throughput collapse under concurrency;
- Looking only at peak throughput while ignoring P99 tail latency;
- Optimizing only operators while ignoring the structural gains from scheduling and caching strategies;
- Focusing only on training MFU while ignoring the hidden overhead of the data pipeline and checkpoint writes.
8. Practice Checklist
Training side
- Run small-scale baseline experiments to estimate the parameter-to-data ratio, then scale up to full training;
- Match the parallelism strategy to the cluster topology, and keep tensor parallelism within a node;
- Account for the memory budget item by item as “weights + gradients + optimizer states + activations + buffers”;
- Default to BF16 mixed precision, and validate convergence alignment before using lower precision;
- Use activation recomputation and sequence parallelism to address the memory bottleneck of long sequences first;
- For MoE models, focus on monitoring routing load balance and expert utilization;
- Save checkpoints in shards asynchronously, and combine with health checks for fast fault recovery.
Inference side
- Clearly distinguish online interaction from offline batch processing, and set separate SLOs for each;
- Enable continuous batching and paged KV caching first;
- Enable cache reuse for stable prefixes, and load-test the cache hit rate;
- Choose chunked prefill and prefill/decode disaggregation strategies based on the business length distribution;
- Subject both weight quantization and KV quantization to precision acceptance;
- Build a unified monitoring dashboard for TTFT, TPOT, throughput, memory watermark, and error rate;
- Run regular regression evaluations to prevent optimizations from introducing quality regressions.
Overall, performance optimization for large-model training and inference is not a pile of isolated tricks, but collaborative engineering across six layers: architecture, parallelism, precision, scheduling, caching, and observability. Understanding the evolutionary logic of language models—from statistical to neural, and from bidirectional to autoregressive—helps judge the boundaries within which different optimization techniques apply; and building a quantifiable, regression-testable, and observable engineering system is the fundamental guarantee for reliably delivering optimization gains to the business side.
