A Comprehensive Guide to AI Infrastructure Interview Questions for Large Language Models
Preparing for an AI Infrastructure (AI Infra) interview? This guide breaks down high-frequency questions and essential technical concepts, focusing on the evolution of attention mechanisms, memory management, parallelism strategies, and inference serving. Learn how to structure your answers with quantified metrics, understand the shifting bottlenecks from training to inference, and master system design for long-context LLMs to ace your next interview.
A Comprehensive Guide to AI Infrastructure Interview Questions for Large Language Models
Introduction: Why AI Infra Interviews Revolve Around Attention Mechanisms
Every seamless long-context conversation and smooth streaming output relies on an underlying “efficiency war” over attention mechanisms. A few years ago, flagship models were constrained to context windows of just over 2,000 tokens. Today, mainstream large language models (LLMs) push these windows to 128K or 256K tokens, with some moving toward the million-token mark. This quantitative leap is not simply the result of scaling up parameters, but the product of co-evolution across architecture, operators, memory management, parallel strategies, and serving systems.
For candidates, the implication is straightforward: AI Infra interviews will almost certainly start with the attention mechanism and extend upstream and downstream into memory, communication, operators, and serving systems. Attention is the only module in the model whose complexity grows super-linearly with sequence length, and the only one that exhibits completely different bottleneck profiles during the “training” and “inference” phases. Whoever can clearly articulate this main thread holds the leverage to answer most follow-up questions.
This article unfolds in the order of “Examination Map → Technical Thread → High-Frequency Questions → System Design → Communication Methodology,” helping you organize scattered knowledge points into a reusable logical chain capable of withstanding rigorous interview scrutiny.
1. The Examination Map for AI Infra Roles
AI Infra interviews typically do not test isolated knowledge points. Instead, they assess whether you can “train, run, and scale a model under a real compute budget.” The common sections can be categorized into six areas:
| Category | Core Assessment Points | Typical Follow-up Depth |
|---|---|---|
| Memory & Parallelism | Memory composition accounting, Data/Tensor/Pipeline/Sequence/Expert Parallelism | Derive viable parallel combinations given a specific number of GPUs and model size |
| Communication & Topology | Collective communication primitives, communication volume & bandwidth lower bounds, topology affinity | Why TP shouldn’t span across nodes, how to calculate PP bubbles |
| Compute Efficiency | MFU/HFU, arithmetic intensity, operator fusion, recomputation | Why a training run only achieves 40% utilization and how to profile it |
| Inference Serving | KV cache, continuous batching, paged cache, prefix caching, speculative decoding | How TTFT and TPOT trade off against each other |
| Precision & Compression | Quantization (weights/activations/KV), distillation, low-rank decomposition | How to profile and compensate for accuracy drops after quantization |
| Stability & Cost | Long-term stable training, fault detection & recovery, elasticity, per-token cost | Your solution for a 1,000-GPU training run crashing once every three weeks |
Interviews generally take four formats: conceptual deep-dives, coding on a whiteboard, system design, and incident retrospectives (handing you a production incident to analyze). The last category best differentiates candidates because it requires you to understand math, hardware, systems, and business simultaneously.
2. Technical Thread: Three Bottleneck Shifts in Attention Mechanisms
To answer “why is long context so difficult,” the most convincing approach is not to recite conclusions, but to trace the trajectory of bottleneck shifts.
2.1 The Pre-Transformer Era: Serial Execution Locked Out Compute Utilization
Before self-attention became mainstream, sequence modeling relied heavily on Recurrent Neural Networks (RNNs) and their gated variants. They had two unavoidable flaws:
First, time-step serial execution. The computation at step t had to wait for the output of step t-1. There was no parallelizable dimension along the sequence. This meant that even with thousands of compute cores, the critical path of a single sample could only light up a fraction of them, leaving accelerators idle and stretching long-corpus training into months.
Second, long-distance dependency decay. To transmit information across dozens or hundreds of steps, gradients underwent repeated multiplications under the chain rule, leading to almost inevitable decay or explosion. Maintaining referential and contextual relationships between the beginning and end of a sentence was highly unstable.
The true contribution of the foundational 2017 attention paper (Attention Is All You Need) can be summarized in one sentence: It replaced the temporal dependency of recurrent structures with a fully connected structure along the sequence dimension. This substitution yielded two chained results—the sequence dimension could be chunked and parallelized entirely, leading to an order-of-magnitude improvement in training efficiency on equivalent hardware; and the path length between any two tokens became 1, completely unlocking the ability to express long-distance dependencies. Subsequent BERT-style pre-trained models, the LLaMA series, and today’s mainstream LLMs are all built on this foundation.
2.2 The First Bottleneck: O(N²) on the Training Side and the Memory Wall
The original Multi-Head Attention (MHA) maintained independent Q/K/V projections for each head, capturing dependencies in different subspaces (syntax, reference, logic, positional patterns). The expressive power was high, and the cost was straightforward:
Assuming a sequence length of N, a head dimension of d, and a head count of h, the attention matrix size is h × N × N. Both computational load and intermediate activation memory grow quadratically with N. When N grows from 4K to 128K, the compute and memory overhead for the attention portion increases by about 1,000x. This is why early model context windows maxed out at 2048: it wasn’t a choice, but a limitation. During training, the activation memory couldn’t hold it, and even if it could, the fragmented small operators in attention would cause accelerators to stall.
Solutions for the training phase exist on three layers, which should be explained hierarchically during an interview:
- Operator Layer: IO-aware fused implementations. The core idea is “tiling + online normalization.” Q, K, and V are sliced along the sequence into blocks that fit into on-chip high-speed cache. Local attention is computed block by block, using online softmax to incrementally maintain the maximum value and denominator, avoiding writing the full N×N attention matrix back to global memory. The result is that intermediate attention memory drops from O(N²) to O(N). By reducing repeated reads and writes to global memory, actual speed increases severalfold. The key insight is: attention during training is memory-bandwidth and IO bound, not purely compute bound.
- Graph Optimization Layer: Operator fusion and selective recomputation. LayerNorm, residual connections, and activation functions are fused with adjacent operators into single kernels. During backpropagation, not all intermediate activations are saved; instead, they are recomputed layer-by-layer or block-by-block, trading 20%~30% extra compute for 30%~60% activation memory savings.
- Parallelism Layer: Sequence dimension splitting. When a single GPU cannot hold an ultra-long sequence, the sequence dimension is sliced into segments and distributed across different devices. Ring communication rotates K/V shards among devices, supporting million-level sequences without increasing peak memory. This is the final piece of the puzzle for “running long-context training.”
2.3 The Second Bottleneck: KV Cache on the Inference Side and the Memory Bandwidth Wall
Inference is divided into two stages. Many people confuse their bottlenecks; distinguishing them clearly in an interview is a bonus:
- Prefill: Processes the entire prompt at once. Q/K/V for all tokens can be computed in parallel. This is compute-bound, with the bottleneck on compute power. It determines Time To First Token (TTFT).
- Decode: Adds only one token per step, but its Query must attend to the K/V of all historical tokens. At this point, compute power is almost idle. Generating each token requires moving the entire weight set and the entire KV cache from memory to the compute unit. This is memory-access bound, with bottlenecks on memory bandwidth and capacity. It determines Time Per Output Token (TPOT).
This distinction is crucial because it explains the rationale behind almost all inference optimization techniques.
KV Cache is the most fundamental technique: storing the K/V of historical tokens for reuse, avoiding recomputation at each step, trading memory for compute. However, it shifts the problem from “not enough compute” to “not enough memory.” The size of the KV cache can be approximated as:
KV_bytes ≈ 2 (K and V) × num_layers L × num_kv_heads × head_dim × seq_len × batch_size × bytes_per_precision
For a model with dozens of layers, a hidden dimension in the thousands, and FP16 precision, a single 128K context request can consume several GB to over a dozen GB of memory. This means the actual concurrency limit of an inference service is often not tuned by batch size, but capped by the memory budget of the KV cache. In an interview, if you can write out this estimation formula on the spot and calculate “how many long-context requests this machine can serve concurrently,” you are basically safe.
Optimizations around this point form a clear technical path:
- Reducing KV head count: Starting from MHA, multiple Query heads share a set of K/V heads (Multi-Query Attention). In the extreme case, all Query heads share one set. The KV cache shrinks severalfold at the cost of expressive power. The industry generally adopts a compromise grouping scheme (Grouped-Query Attention, GQA), compressing the KV cache to 1/4~1/8 with almost no performance drop.
- Compressing KV representations: Applying low-rank joint compression to K/V, caching a latent vector much smaller than the original dimension, and projecting it back during use. Mathematically equivalent to structuredly compressing the KV cache, it can compress the cache to extremely low levels. Since the compression is done jointly during model training, accuracy loss is controllable.
- Paged cache management: Slicing the KV cache into fixed-size blocks, mapping logical block tables to physical memory, and allocating/reclaiming on demand. This avoids the fragmentation waste of reserving contiguous memory for the “maximum possible length.” The benefits here are often more direct than model-side modifications.
- Prefix reuse: System prompts, few-shot examples, and long documents are identical across multi-turn conversations. Computing prefill only once for these common prefixes and caching K/V for subsequent requests to hit can reduce the TTFT of repeated requests by an order of magnitude. This is a key engineering method for landing long-context products.
- Continuous batching: Instead of waiting for an entire batch of requests to finish before swapping in new ones, dynamically swapping requests in and out at the granularity of token iterations. GPU utilization shifts from “batch busy, batch empty” to sustained full load, increasing throughput severalfold.
- Prefill/Decode disaggregation: Since the two phases have different bottlenecks, they are placed in separate resource pools, deployed with compute-intensive and bandwidth-intensive machine ratios respectively, passing KV caches via message channels. This is the current mainstream direction for long-context serving architecture evolution and an excellent system design question answer.
2.4 The Third Bottleneck: The Battle for “Effective Attention” in Long Contexts
Even with all the engineering methods above, O(N²) complexity remains uneconomical beyond 128K—human reading efficiency is limited, and a model’s attention budget should be too. Thus, two routes emerged:
Sparsification route: Instead of letting every token attend to all others, retain a combination of local windows, cross-block summary tokens, and a few “important” tokens. Representative forms include various window attentions, block-sparse attentions, and recently trainable natively sparse schemes—learning “where to look” during training and computing only a few selected K/V blocks during inference, compressing complexity from quadratic to near-linear. The engineering benefits are very tangible: attention overhead for long sequences drops significantly, KV cache read volume decreases synchronously, and TPOT improves.
Positional encoding extrapolation route: Models trained only on short sequences must handle long sequences during inference, causing positional encoding to mismatch. The industry standard involves a family of methods like positional interpolation, frequency dimension scaling, and temperature adjustment, combined with a small amount of continued pre-training on long texts, allowing the model to maintain a reasonable attention distribution beyond its original training length. Interview point: positional encoding extrapolation is a training-side algorithmic issue, but it ultimately exposes itself in inference serving as “output quality collapse under long context.” Infra teams must understand it to do capacity planning correctly.
Stringing the three bottlenecks together gives you a very powerful interview narrative:
The efficiency evolution of attention mechanisms is essentially the process of bottlenecks migrating sequentially between “sequence parallelism → training memory & IO → inference memory bandwidth → effective compute volume.” Each optimization compresses one dimension to release another.
3. High-Frequency Questions and Answering Frameworks
Below are questions that repeatedly appear in real interviews, categorized by section, with answering skeletons.
Memory & Parallelism
- Where does memory go when training a model? Four parts: parameters, gradients, optimizer states (under mixed-precision training, Adam-style momentum and variance are memory hogs, often several times the size of parameters), and activations. Add fragmentation, communication buffers, and temporary tensors. Answer by giving categories first, then quantitative estimations, and finally optimization methods (ZeRO-style parameter/gradient/optimizer state sharding, tensor parallelism, activation recomputation, offloading).
- How to choose between Tensor Parallelism (TP) and Pipeline Parallelism (PP)? TP splits single-layer operators, communicating frequently but with small payloads, suitable for intra-node high-speed interconnect domains, usually not exceeding 8 GPUs per node. PP splits by layers, with low communication volume but introducing pipeline bubbles, requiring finer splitting and scheduling to compress bubble ratio to acceptable levels. DP only splits data, relying on gradient sync for convergence, usually combined with gradient accumulation and sharded optimizer states.
- Why doesn’t TP generally span across nodes? Because TP requires collective communication twice per transformer layer in attention and FFN, which is synchronous blocking and extremely sensitive to latency. Spanning nodes over general networks causes communication time to exceed compute time, reducing the speedup ratio. This is a classic “trading bandwidth for latency” trade-off question.
Communication & Collective Operations
- Where are AllReduce / AllGather / ReduceScatter / All2All used? Data parallel gradient sync uses AllReduce; sharded optimizer state parameter updates use a combination of AllGather + ReduceScatter; expert parallelism token routing uses All2All. You must be able to explain the data flow direction and communication volume of each primitive.
- How to estimate communication volume lower bounds? Taking AllReduce as an example, with data size M and P participating devices, the classic ring implementation transmits/receives about 2M(P-1)/P bytes per device, approaching 2M, roughly independent of P. This conclusion explains why “adding machines doesn’t necessarily make it faster.”
Inference Serving
- How to optimize TTFT and TPOT, and why do they sometimes conflict? Increasing batch size improves throughput but elongates single-request latency; chunked prefill splits long prompt prefills into smaller chunks, preventing one long request from blocking the entire decoding batch, improving TPOT jitter; increasing concurrency squeezes the KV cache budget, triggering preemption and recomputation. You must articulate the “throughput-latency-memory” triangle relationship.
- Should the KV cache be quantized? Usually yields clear benefits (dual reduction in memory and bandwidth), but you need to distinguish the sensitivity of outliers in KV to quantization. Per-head or per-channel scaling is more stable than global scaling; if necessary, retain high precision for a few critical channels.
Precision & Compression
- How to troubleshoot accuracy drops after quantization? First, see if it’s a weight or activation quantization issue (compare output distributions layer by layer). Then, check if it’s concentrated in a few sensitive layers (usually the first/last few layers and attention output projections). Finally, consider mixed-precision schemes, GPTQ-style layer-by-layer error compensation, or distillation fine-tuning for recovery.
Stability & Cost
- How to ensure a 3-week stable run on 1,000 GPUs? Before training, perform slow node detection (comparing collective communication times across different GPUs quickly identifies degraded NICs). During training, add periodic consistency checks. Save asynchronous checkpoints to remove saving overhead from the critical path. Establish fault-domain-aware rescheduling mechanisms to isolate and repair replaced nodes first. Monitor compute utilization, communication latency, temperature, and GPU dropout rates throughout the process.
Coding Exercises
Common questions: implement cached self-attention forward pass, implement GQA KV head replication and splitting, implement paged KV cache block allocation and reclamation, implement a continuous batching scheduler, derive per-GPU memory usage given a parallel splitting strategy. When writing code, first clarify inputs, outputs, and tensor shapes, then write the main body, and finally proactively mention edge cases and performance considerations.
4. Sample Code: GQA Cache Access During Decoding
The following pseudo-code demonstrates the most critical step on the inference side—how to reuse shared K/V when the number of Query heads does not equal the number of KV heads, without explicit data copying:
import torchimport torch.nn.functional as Fdef gqa_attention(x, kv_cache_k, kv_cache_v, num_heads, num_kv_heads, scale=None):"""x: [batch, num_heads, 1, head_dim] Query for the current stepkv_cache_k: [batch, num_kv_heads, seq_len, head_dim] Historical K cachekv_cache_v: [batch, num_kv_heads, seq_len, head_dim] Historical V cachereturn: [batch, num_heads, 1, head_dim]"""b, h, _, d = x.shapeif scale is None:scale = d ** -0.5# Key: use repeat_interleave to logically expand KV heads.# In production, this should be read via broadcasting index without# creating a num_heads/num_kv_heads times memory copy.group = h // num_kv_headsk = kv_cache_k.repeat_interleave(group, dim=1)v = kv_cache_v.repeat_interleave(group, dim=1)# [batch, h, 1, seq_len]scores = torch.matmul(x, k.transpose(-1, -2)) * scale# A production implementation of chunked online softmax would break this# into a tile loop, avoiding allocating a full intermediate scores tensor# for ultra-large seq_len.probs = F.softmax(scores, dim=-1)out = torch.matmul(probs, v)return out
Three points worth expanding on from this code are all high-frequency follow-ups: First, repeat_interleave is just a logical view expansion; production implementations should avoid actual memory copying. Second, when seq_len reaches the 100,000 level, the intermediate tensor scores of shape [b, h, 1, seq_len] must be computed in chunks to avoid OOM. Third, during the decoding phase, the operator shape for matmul degenerates into “vector × matrix,” resulting in extremely low arithmetic intensity. The compute units of the accelerator are essentially idle, which is code-level proof of the assertion that “inference relies on bandwidth, not compute.”
5. System Design Example
Prompt: Design an online inference service supporting 128K context with controllable P99 Time To First Token (TTFT).
A viable answering structure:
- Calculate capacity first. Use the KV cache formula to estimate memory usage per request. Divide by “single GPU memory − weight footprint − runtime overhead” to get the concurrent requests per GPU. Then reverse-engineer the required number of GPUs based on target QPS. This step determines whether the subsequent design is realistic.
- Disaggregate architecture into pools. The Prefill pool is provisioned with compute-intensive machines to receive long prompts and perform chunked prefilling. The Decode pool is provisioned based on memory bandwidth and capacity to sustain continuous decoding. High-speed networks and message queues pass KV cache blocks in between.
- Tiered caching. GPU memory holds hot KV caches, host memory holds warm caches, and remote object storage holds persistent prefix caches for long documents. Use block hashing for global prefix reuse, hitting common system prompts and documents across requests.
- Scheduling strategy. Use the memory budget as a hard constraint for admission control. Queue long and short requests separately to avoid head-of-line blocking. When memory is tight, preempt by priority; preserve the computed prefixes of preempted requests for recovery rather than full recomputation.
- Elasticity and observability. Autoscale based on KV cache hit rate, memory watermarks, and queue times rather than simple CPU usage. Incorporate TTFT, TPOT, cache hit rate, preemption rate, and per-token cost into monitoring and alerting.
- Degradation plan. When context limits are exceeded or memory is critically low, switch to a degraded path of sliding window truncation + summary compression to ensure availability over completeness.
In a real enterprise environment, this architecture usually relies on general cloud infrastructure capabilities to host: container platforms manage inference service orchestration and elastic scaling, object storage handles model weights and long document caches, and logging services with monitoring and alerting systems support end-to-end latency breakdown and fault localization. As a user of a platform like Baidu AI Cloud, the engineering focus remains on connecting “model-side optimizations” with “infrastructure-side elasticity, observability, and cost governance,” rather than reinventing the wheel.
6. Communication Methodology: Passing All Follow-up Questions
Scoring in an AI Infra interview often hinges not on “do you know it,” but on “is your judgment quantifiable and verifiable.” It is recommended to structure every answer in four steps:
- Conclusion first: Give the trade-off judgment in one sentence. For example, “In this scenario, I would prioritize GQA over sparse attention because the accuracy risk is lower than the engineering complexity benefits.”
- Provide quantification: Write out the complexity or memory estimation formula, plug in specific numbers, and state the GPU count, sequence length, and precision.
- Explain the trade-off: Clearly state what you gave up, what the cost is, and under what conditions the conclusion would change.
- Provide evidence: Tell a story about measured data you collected or a pit you fell into (e.g., communication ratio under a specific parallel combination, or an accuracy recovery process after quantization). Real numbers are far more persuasive than adjectives.
Also, prepare three “ballast stories”: a performance bottleneck localization process (how you narrowed down the scope layer by layer from monitoring metrics), a production incident retrospective (root cause, impact scope, improvements), and a cost vs. performance trade-off decision (how much you saved, what you sacrificed). These three types of stories cover the vast majority of behavioral and incident retrospective questions.
Conclusion
From the initial dividends of sequence parallelism, to IO-aware operators during training, to KV cache and bandwidth constraints during inference, and now to the evolution toward near-linear complexity effective attention, this thread is itself a compressed history of the industrialization of large models. The assessment for AI Infra roles seems divergent—memory, communication, operators, scheduling, stability, cost—but the underlying logic is highly unified: under a given hardware budget, converting every ounce of compute and bandwidth into model capability.
Understand this main thread thoroughly, and back up every judgment with quantitative estimations and measured data. Your interview transforms from “guessing test points” to “clearly explaining a system you truly understand.”
