LLM AI Infrastructure Interview Questions: A Systematic Guide
LLM AI infrastructure interviews test whether candidates can turn model computation into reliable, efficient systems on large hardware clusters. This guide organizes common questions across attention, memory, distributed training, inference serving, quantization, and platform design, while showing how to structure answers around principles, bottlenecks, solutions, trade-offs, and validation metrics. It emphasizes the constraints linking compute, memory, communication, scheduling, and reliability.
1. What AI Infrastructure Interviews Actually Assess
Large-model AI infrastructure is not just about whether a candidate can call a model API. Candidates need to understand how model computation works and how models are deployed on large hardware clusters. They also need to improve training and inference efficiency under limited compute, memory, network capacity, and cost.
Interview questions usually focus on four levels:
- Model computation principles: Does the candidate understand attention mechanisms, parameter counts, compute requirements, memory footprint, and cache structures?
- System parallelism: Does the candidate understand data parallelism, tensor parallelism, pipeline parallelism, and hybrid parallelism?
- Performance optimization: Can the candidate identify compute bottlenecks, communication bottlenecks, memory bottlenecks, and scheduling bottlenecks?
- Engineering reliability: Does the candidate consider failure recovery, task scheduling, monitoring and alerting, service rate limiting, and capacity planning?
A strong answer should not stop at definitions. It should form a complete loop: principle → bottleneck → solution → cost → validation metric.
2. Attention Mechanisms: The Most Common Foundation Question
2.1 What is multi-head attention?
Given an input sequence, the model first applies three different linear transformations to produce query vectors, key vectors, and value vectors, denoted as (Q), (K), and (V). Attention is usually computed as:
[
\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V
]
Here, (d_k) is the dimension of the key vectors. It scales the dot-product results to prevent the values from becoming too large and making the Softmax gradients too sharp.
The core idea of multi-head attention is to split the hidden vector into multiple subspaces. Each head computes attention independently, and the results are then concatenated and projected back to the original dimension. Different heads can learn different types of relationships, such as local syntactic dependencies, long-distance coreference, and structural information.
Common follow-up questions include:
- Why divide by (\sqrt{d_k})?
- Why use multiple heads instead of one larger attention layer?
- What is the space complexity of the attention matrix?
- What is the difference between attention computation during training and during inference?
2.2 Why is standard attention quadratic?
Assume the sequence length is (N) and the hidden dimension is (D). Computing (QK^T) requires producing an (N \times N) association matrix, so the time complexity is usually:
[
O(N^2D)
]
The memory footprint also includes intermediate results related to (N^2).
This means that when context length grows, compute and memory costs do not increase linearly. For example, if sequence length doubles, attention-related computation may be close to four times larger. Therefore, the difficulty of long-context capability is not only about extending the range of positional encoding. It also involves reducing the cost caused by the attention matrix.
When answering this type of question, it is important to note that the full attention matrix is not necessarily materialized in every implementation. Real systems may use block computation, operator fusion, or online reduction to reduce intermediate tensors. Even so, the underlying computation relationship is still affected by quadratic complexity.
3. Where Memory Is Consumed in Training and Inference
This is a very common category of AI infrastructure interview question.
3.1 Memory components during training
Training memory usually includes:
- Model parameters;
- Parameter gradients;
- Optimizer states;
- Activations saved during the forward pass;
- Temporary computation buffers;
- Extra buffers required for communication.
Activations are usually closely related to batch size, sequence length, and the number of network layers. During long-sequence training, activation storage can become the main source of memory use.
With mixed precision, parameters and activations can be stored at lower precision, but optimizer states or some master weights may still need higher precision to ensure training stability. Therefore, it is not correct to assume that “half-precision training cuts memory in half.”
3.2 Memory components during inference
Inference usually does not need to save activations and gradients for backpropagation, so the memory structure is simpler. It mainly includes:
- Model weights;
- Input and output for current requests;
- Attention cache;
- Batching and scheduling caches.
For autoregressive generation models, the attention cache grows with the number of generated tokens. If many requests are processed concurrently, the cache may become the main source of inference memory consumption.
A useful interview answer framework is:
Training memory focuses on parameters, gradients, optimizer states, and activations; inference memory focuses on parameters, request intermediate states, and attention caches that grow with sequence length.
4. What Is the Attention Cache, and Why Does It Improve Inference Speed?
Autoregressive generation produces only one new token at a time. If the key and value vectors of all historical tokens were recomputed at every step, a large amount of redundant computation would result.
The attention cache, often called the KV cache, stores the key and value vectors already computed for historical tokens during generation. At the next step, the model only needs to compute the query vector for the new token and match it against the cached history.
This avoids recomputing the historical part and significantly reduces the computation required per generation step. But the cache is not free:
- It consumes a large amount of memory;
- The longer the context, the larger the cache for a single request;
- The more concurrent requests, the more complex cache management becomes;
- When request lengths differ, memory fragmentation is more likely;
- Cache reads and writes may be limited by memory bandwidth.
In an interview, candidates can further discuss cache management strategies such as:
- Allocating in fixed blocks to reduce the need for large contiguous memory;
- Dynamically scheduling different requests;
- Compressing or trimming excessively long histories;
- Using lower-precision storage when accuracy allows;
- Reclaiming caches from idle requests in a timely manner;
- Reusing prefixes to reduce redundant computation.
Cache optimization should not focus only on memory usage. It should also observe first-token latency, per-token latency, throughput, and request queueing time together.
5. How to Answer Questions About MHA, MQA, and GQA
In multi-head attention, each query head usually has its own keys and values. This provides strong representational capability, but it also increases cache size.
To reduce cache overhead during inference, multiple query heads can share a set of keys and values:
- Multi-head attention (MHA): The number of query, key, and value heads is generally similar. It has strong representational capability and a relatively large cache.
- Multi-query attention (MQA): Multiple query heads share the same set of keys and values. The cache is significantly reduced, but there may be some loss in representational capability.
- Grouped-query attention (GQA): Query heads are divided into groups, with each group sharing keys and values. This provides a compromise between model quality and cache overhead.
The key to this question is not memorizing definitions, but explaining the design trade-offs:
- The number of query heads mainly affects query representational capability;
- The number of key-value heads directly affects cache size;
- Fewer key-value heads usually reduce memory and bandwidth pressure during inference;
- Too much sharing may affect model quality and must be validated experimentally.
6. Distributed Training Questions
6.1 Data parallelism
Data parallelism is the easiest approach to understand. Each compute device holds a full copy of the model, different devices process different data shards, and gradients are synchronized after computation.
Its advantages are relative simplicity and suitability when the model fits into a single device’s memory. The disadvantages are that every device must hold a full model copy, so model size is limited by single-device capacity; gradient synchronization can also create significant communication overhead.
6.2 Tensor parallelism
Tensor parallelism splits matrix operations within a layer across multiple devices. For example, a large linear layer can be split by columns or rows. Multiple devices compute local results separately, and collective communication is then used for concatenation or reduction.
This approach is suitable when the parameters of a single layer are large and cannot be processed on one device. However, communication is frequent, placing high demands on interconnect bandwidth and communication latency between devices.
6.3 Pipeline parallelism
Pipeline parallelism splits the model by layers, with different devices responsible for different layers. A batch is further split into multiple micro-batches so that different pipeline stages can work at the same time as much as possible.
The main issue is idle time during pipeline startup and drain, commonly known as pipeline bubbles. Increasing the number of micro-batches can improve device utilization, but it also increases scheduling complexity and activation memory pressure.
6.4 Hybrid parallelism
When model scale, data scale, and cluster scale are all large, multiple parallelism strategies usually need to be combined. For example, tensor parallelism can be used within a node, data parallelism across nodes, and pipeline parallelism can be added as another dimension.
When answering, candidates should explain how parallelism strategies are selected:
- Whether the model fits on a single device;
- The network topology between devices;
- The communication ratio;
- Per-step computation time;
- Global batch size;
- Training stability and scaling efficiency.
7. How to Analyze a Training Performance Drop
If training throughput does not increase as the number of devices grows, the problem should not be attributed directly to “insufficient compute.” Instead, it should be investigated layer by layer.
7.1 Compute side
Check compute-device utilization, the proportion of matrix operations, operator execution time, and memory bandwidth. If compute-device utilization is low, possible causes include operator granularity being too small, frequent synchronization, or input preparation not keeping up.
7.2 Communication side
Observe the time spent on gradient synchronization, tensor aggregation, and pipeline transfer. If communication takes up too high a proportion, consider adjusting the parallelism partition, increasing compute-communication overlap, reducing synchronization frequency, or optimizing data layout.
7.3 Input side
Data reading, decoding, random shuffling, and batch assembly can all become bottlenecks. Data prefetching and computation should run in parallel to prevent compute devices from waiting for data.
7.4 Scheduling side
Uneven task partitioning and large differences in pipeline-stage duration can also leave some devices idle. In this case, network layers need to be repartitioned or the micro-batch strategy adjusted.
7.5 Stability side
Frequent retries, device throttling, and task migration triggered by insufficient memory can also reduce throughput. Performance analysis must combine logs, metrics, and timelines rather than looking only at average utilization.
Common performance metrics include:
- Tokens processed per second;
- Per-step training time;
- Compute-device utilization;
- Communication time percentage;
- Memory usage;
- Input wait time;
- Scaling efficiency;
- Failure retry rate.
8. Inference Serving: Throughput, Latency, and Scheduling
Inference services usually include two types of phases:
- Input-processing phase: Processes user input and builds the initial cache. This phase is usually more compute-intensive.
- Token-by-token generation phase: Generates a small amount of content at each step and is usually more affected by memory bandwidth and cache access.
The optimal batching strategy is different for these two phases. The input-processing phase is suitable for aggregating larger requests, while the generation phase needs continuous scheduling of different requests to improve device utilization.
In an interview, candidates can discuss dynamic batching:
- Add newly arriving requests to a running batch;
- Decide admission based on remaining cache and maximum sequence length;
- Release resources promptly for requests that are about to finish;
- Set fairness policies for long requests so that short requests are not queued for too long;
- Dynamically adjust batch size based on first-token latency and generation latency.
Several metrics need to be distinguished:
- First-token latency: The time from request arrival until the first token is returned;
- Per-token latency: The average token interval during continuous generation;
- End-to-end latency: The total time from when the user sends a request until the complete result is returned;
- Throughput: The number of tokens or requests processed per unit of time.
High throughput does not necessarily mean low latency. Real systems need to make scheduling and capacity trade-offs between the two based on business objectives.
9. How to Answer Questions About Quantization, Low Precision, and Operator Optimization
The basic idea of quantization is to use lower bit-width representations for weights, activations, or caches, thereby reducing memory usage and memory-access pressure.
Common effects include:
- Reduced weight memory usage;
- Reduced cache capacity requirements;
- Lower memory bandwidth demand;
- Improved compute efficiency on some hardware;
- Possible accuracy loss or numerical stability issues.
In an interview, candidates should emphasize that quantization is not simply converting every value to low precision. The following need attention:
- Whether the distributions of weights and activations contain outliers;
- Whether per-layer, per-channel, or group-wise scaling is used;
- Whether quantization error is concentrated in sensitive layers;
- Whether inference results meet business quality requirements;
- Whether low-precision operators are actually executed efficiently by the hardware.
Operator fusion is another common optimization technique. Combining multiple consecutive operations can reduce intermediate tensor writes and reads and lower the number of memory accesses. However, fusion may increase implementation complexity and must remain compatible with different shapes, precisions, and hardware environments.
10. How to Answer “Design an LLM Inference Platform”
The answer can be organized into the following layers:
10.1 Request ingress layer
Responsible for authentication, rate limiting, parameter validation, timeout control, and request routing.
10.2 Scheduling layer
Queues and batches requests based on request length, priority, remaining resources, and cache occupancy.
10.3 Execution layer
Responsible for model loading, parallel execution, cache management, batch construction, and result generation.
10.4 Resource management layer
Manages compute devices, memory, nodes, task instances, and failure migration.
10.5 Observability and governance layer
Collects request latency, throughput, error rate, memory usage, device utilization, queue length, and cache hit conditions. It also supports alerting and capacity forecasting.
In a design question, candidates should proactively discuss abnormal scenarios:
- A single request is extremely long;
- A compute device fails;
- Model loading fails;
- Memory fragmentation becomes severe;
- A burst of traffic exceeds capacity;
- A certain type of request occupies resources for a long time;
- Network jitter occurs between nodes;
- Some requests generate at abnormal speed.
A complete solution must include degradation strategies, such as limiting maximum input length, reducing concurrency, enabling standby instances, returning partial results, or temporarily disabling low-priority requests.
11. A General Method for Interview Answers
For system-design questions without a standard answer, use the five-step method:
- Define the objective first: Is the goal throughput, low latency, low cost, or stability?
- Clarify constraints: What are the model size, context length, number of devices, network conditions, and business peak?
- Break down bottlenecks: Are there problems in compute, memory, communication, input, scheduling, and storage?
- Propose a solution: Design across parallelism, caching, batching, quantization, and fault tolerance.
- Explain validation: Which metrics, stress-test methods, and fault drills will be used to verify the result?
For example, when asked “How would you support longer context?”, do not only answer by extending the range of positional encoding. Also discuss attention complexity, cache capacity, memory bandwidth, long-sequence training data, positional representation, batching concurrency, and quality evaluation.
Ultimately, AI infrastructure interviews assess the ability to turn model algorithms into stable systems. Understanding attention computation is only the starting point. More important is understanding the mutual constraints among computation, storage, communication, scheduling, and reliability, and being able to locate problems with metrics and make engineering trade-offs under constraints.
