LLM AI Infrastructure Interview Guide: Key Topics and Answer Frameworks
This article provides a structured interview-prep guide for large language model (LLM) AI infrastructure roles. It explains what interviewers assess, then covers training topics such as parallelism strategy selection, memory estimation, long-sequence training, and MoE; inference topics such as KV cache management, continuous batching, scheduling, and decoding optimization; and platform observability and troubleshooting. It concludes with a reusable answer framework and project-review preparation advice.
Over the past few years, LLM context windows have expanded from around 2K tokens to 128K–256K tokens. This progress has not come from a single parameter or one-off optimization. It reflects coordinated advances across positional encoding, attention computation, parallel training, memory management, cache scheduling, and service governance. In AI infrastructure interviews, interviewers usually do not ask only “What is a concept?” They use follow-up questions to test whether candidates truly understand system bottlenecks, resource models, and engineering trade-offs. The following sections provide a reusable knowledge map and answer framework for common questions.
1. What Interviewers Really Assess
The core of an AI infrastructure role is not memorizing terminology. It is making system trade-offs. Common assessment dimensions include:
- Resource efficiency: Given cluster size, model size, sequence length, and throughput targets, how do you choose the parallelism strategy, batch size, recomputation strategy, and cache strategy?
- Correctness and stability: In distributed training, are gradients consistent, is numerical precision controllable, and can the job recover from failures? In inference serving, are outputs reproducible, and are timeouts and retries controlled?
- Scalability: When scaling from a single node with multiple GPUs to multiple nodes, or from one replica to multiple replicas, the bottleneck may shift from computation to communication, storage, scheduling, or networking.
- Cost and SLOs: Training focuses on effective throughput and cluster utilization. Inference focuses on time to first token (TTFT), time per output token (TPOT), P99 latency, throughput, and cost per request.
- Engineering closure: Can you define metrics, locate bottlenecks, design experiments, perform canary validation, and roll back?
When answering, use the following sequence: clarify goals, build a resource model, propose solutions, explain trade-offs, and describe validation and fallback. Even if your specific experience does not match perfectly, this structure demonstrates systematic thinking.
2. Training-Side Questions
2.1 How Do You Choose Between Data, Tensor, Pipeline, Expert, and Sequence Parallelism?
This is one of the most fundamental and common training-system questions. Data parallelism assigns different batches to different devices. It is suitable when the model fits on a single device or when parameter sharding is used. Tensor parallelism splits matrix computation within a layer; because it involves frequent communication, it should generally be placed inside a high-speed interconnect domain. Pipeline parallelism splits the model by layers and reduces per-device memory, but it introduces pipeline bubbles and requires microbatch scheduling. Expert parallelism targets MoE architectures, where the core concerns are token routing and All-to-All communication. Sequence parallelism is often used for long-sequence scenarios, splitting the sequence dimension to reduce activation memory.
Do not just list terms. Give selection criteria: model parameter count, per-device memory, sequence length, cluster topology, interconnect bandwidth, global batch size, and training throughput target. A common principle is to first ensure per-device memory feasibility, then minimize communication across high-speed interconnect domains. Tensor parallelism should not span low-bandwidth domains. Pipeline parallelism should use microbatch count to control bubbles. For MoE, pay attention to expert load balance and communication amplification.
A configuration-checking pseudocode example:
def validate_parallel(world_size, tp, pp, dp):assert world_size == tp * pp * dp, "The product of parallelism degrees must equal total devices"assert tp <= world_size, "Tensor parallelism cannot exceed total devices"# Further checks: whether tp stays within a high-bandwidth domain and whether pp has enough microbatchesreturn {"tp": tp, "pp": pp, "dp": dp}
Interviewers may follow up by asking how to estimate pipeline bubbles. A useful answer is that the bubble ratio depends on the number of pipeline stages and the microbatch count. Increasing microbatches reduces bubbles but increases activation memory and scheduling overhead, so you need to balance throughput against memory.
2.2 How Do You Estimate Memory Usage?
Training memory typically consists of parameters, gradients, optimizer states, activations, communication buffers, and temporary workspaces. Parameters and gradients depend on model scale. Optimizer states depend on the optimizer type. Activation memory is strongly related to batch size, sequence length, layer count, and whether recomputation is enabled. In long-sequence training, activation memory often becomes the bottleneck before parameter memory does.
When answering, first clarify precision, optimizer, whether sharding is used, and whether recomputation is enabled. Then provide a formula-based estimate. For example, inference-side KV cache can be estimated as follows:
def estimate_kv_cache(tokens, layers, kv_heads, head_dim, bytes_per_elem, batch=1):# Each token requires both K and V caches at every layerper_token = 2 * layers * kv_heads * head_dim * bytes_per_elemreturn batch * tokens * per_token
Further explain that KV cache can be reduced by using fewer layers, fewer KV heads, quantized cache, paged management, prefix reuse, and eviction policies. Each method affects accuracy, latency, or implementation complexity.
2.3 What Are the Core Challenges of Long-Sequence Training?
Long-sequence training is not simply increasing the sequence length. First, attention computation and memory grow significantly with sequence length. Second, activation memory and temporary buffer pressure increase. Third, communication and data layout become more complex. Fourth, the extrapolation capability of positional encoding needs to be validated. Fifth, data packing and variable-length masks can easily introduce correctness issues.
Common approaches include sequence parallelism, chunked computation, IO-aware efficient attention kernels, activation recomputation, gradient checkpointing, variable-length sequence packing, and sparse or hierarchical attention. In interviews, emphasize that optimization should not focus only on per-step speed. It must also consider end-to-end effective throughput, numerical stability, and model quality.
2.4 Why Is MoE Training Difficult?
MoE improves parameter capacity through sparse activation, but it introduces high engineering complexity. Common topics include how to partition experts, how to estimate All-to-All communication volume, how to set the capacity factor, how to mitigate load imbalance, how to stabilize routing jitter, and how to avoid expert stragglers. In your answer, point out that the MoE bottleneck is often not matrix computation but token routing, communication, and load balancing. If the capacity factor is too small, tokens are dropped. If it is too large, memory and compute are wasted. You need to combine auxiliary losses, routing statistics, and dynamic adjustment strategies.
3. Inference-Side Questions
3.1 How Should KV Cache Be Managed?
KV cache is a core resource in inference serving. Interviewers often ask: How do you estimate maximum concurrency? How do you implement paged or chunked management? How do you support prefix reuse? How do you perform eviction? How do you avoid fragmentation? In your answer, split the request lifecycle into prefill and decode. Prefill is compute-intensive, while decode is affected by memory bandwidth. Managing KV cache in blocks can reduce fragmentation, but it requires block tables and reference counting. Prefix caching can improve hit rates for multi-turn conversations and shared system prompts, but it requires careful consistency and eviction policies.
3.2 How Should Continuous Batching and Scheduling Be Designed?
Continuous batching allows new requests to join a batch while older requests are still decoding, improving throughput. However, scheduling must balance fairness, SLOs, mixed long and short requests, and preemption. Common strategies include selecting requests by priority and remaining latency; reserving resources for long requests; preempting low-priority requests when KV blocks are insufficient; and limiting tokens per batch to avoid tail-latency degradation.
Pseudocode example:
while request_queue:req = select_by_slo(request_queue)if kv_blocks_available(req):admit(req)else:victim = pick_low_priority_running_request()preempt(victim)
Typical follow-up questions include: How do you avoid starvation? How do you evaluate P99? How do you handle extremely long requests? Return to the metrics: throughput, TTFT, TPOT, P99, and resource utilization.
3.3 What Directions Exist for Decoding Optimization?
Common directions include quantization, operator fusion, memory layout optimization, speculative decoding, parallel decoding, prefix caching, and batching. In interviews, explain the applicable boundary of each optimization. Quantization reduces memory usage and bandwidth but may sacrifice accuracy. Speculative decoding improves decoding speed but adds verification overhead. Batching improves throughput but may increase tail latency. Cache reuse depends on request similarity. Do not just list technology names. Provide benefit metrics and fallback plans.
4. Platform Engineering and Observability
AI infrastructure interviews also assess platform engineering capability. Common questions include: How do you locate a failed training job? How do you implement checkpoints and elastic recovery? How does an inference service autoscale? How do you implement multi-tenant quotas and priorities? How do you monitor GPU utilization, communication congestion, OOM events, request queuing, and cache hit rates?
It helps to establish layered metrics. At the resource layer, monitor GPU utilization, memory, network throughput, and storage throughput. At the job layer, monitor effective throughput, queue time, checkpoint duration, and failure rate. At the service layer, monitor TTFT, TPOT, P99, error rate, and cost per request. During troubleshooting, first review the timeline, then resource curves, and finally logs and traces. Avoid immediately attributing the issue to the model or framework.
General-purpose cloud platforms can support these capabilities through container platforms, object storage, logging services, and monitoring and alerting. However, interviews focus more on whether candidates can define metrics, establish causal chains, and propose verifiable optimization plans.
5. Answer Framework and Preparation Advice
An interview answer can follow four steps. First, clarify model size, sequence length, cluster topology, and SLOs. Second, provide a resource model or formula. Third, propose two or three solutions and compare them in terms of communication, memory, latency, and cost. Fourth, explain validation methods, monitoring metrics, and rollback strategies. When reviewing past projects, prepare three cases: one throughput optimization, one stability incident, and one cost-governance initiative. For each case, quantify the before-and-after metrics.
In summary, common LLM AI infrastructure interview questions may appear scattered, but they revolve around six keywords: compute, storage, communication, scheduling, stability, and cost. Frame every question as a system trade-off under resource constraints, and support your conclusions with metrics and an experimental feedback loop. This is how you demonstrate genuine engineering depth in an interview.
