Long Context in LLM Inference: Deep Dive into High-Frequency AI Infra Interview Questions

    A practical guide to the most frequently asked AI Infra interview questions around long-context LLMs, covering attention complexity and IO-aware computation, KV Cache and memory management, position encoding, distributed parallelism, speculative decoding, and quantization—with preparation tips for giving structured, system-level answers.

    Why Long Context Has Become a Core AI Infra Interview Topic

    Early mainstream models supported only a few thousand tokens of context, while today’s models routinely handle hundreds of thousands. This leap was not simply a matter of scaling parameters—it was a multi-year architectural and systems engineering revolution driven by dozens of key papers. Understanding each layer of this revolution is exactly what AI Infra interviewers use to probe a candidate’s depth.

    Interviewers favor this thread for three reasons. First, long context is where the tensions among GPU memory, compute, and communication are most concentrated, making it a comprehensive test of systems thinking. Second, the optimization techniques form clear layers—from operator level to cluster level—that lend themselves to progressive follow-up questions. Third, it is a real cost bottleneck for today’s inference services, so answer quality directly reflects the candidate’s ability to solve practical problems.

    Fundamentals: Attention Mechanism Questions You Will Always Get

    Starting with Complexity

    Almost every interview begins with the time and space complexity of attention. Standard self-attention has a computational complexity of O(n²·d), and memory usage also grows quadratically with sequence length. This means that when context expands from 4K to 128K, the theoretical attention compute grows roughly 1000x. Interviewers expect candidates to not just recite the formula but to explain why quadratic complexity becomes unacceptable at long sequence lengths, and which dimensions industry has used to attack the problem.

    IO-Aware, Tiled Computation

    The core insight behind modern efficient attention implementations is that attention is memory-bandwidth bound rather than compute bound. By loading Q, K, and V in blocks into fast cache and maintaining online softmax statistics (the running max and normalization denominator), implementations avoid materializing the full n×n intermediate attention matrix, dramatically reducing high-bandwidth memory reads and writes. Being able to explain the “online softmax rescaling” principle clearly usually earns extra credit. A simplified illustration:

    1. # Illustrative online softmax: iterate over Key/Value blocks
    2. m = -inf; l = 0; acc = 0
    3. for k_block, v_block in chunks(K, V):
    4. s = Q @ k_block.T / sqrt(d)
    5. m_new = max(m, s.max())
    6. l = l * exp(m - m_new) + exp(s - m_new).sum()
    7. acc = acc * exp(m - m_new) + exp(s - m_new) @ v_block
    8. m = m_new
    9. output = acc / l

    Attention Variants and Sparsification

    Sparse attention, sliding-window attention, and linear attention are standard advanced topics. A common question: why can sliding-window attention be combined with global attention? The key is a layered design—local windows capture short-range dependencies while a small number of global tokens carry long-range information—controlling compute while preserving critical long-distance semantic connections.

    KV Cache and Memory Management: The Main Battlefield of Inference Optimization

    Why KV Cache Is Always Asked

    Autoregressive decoding generates tokens incrementally: each step only needs the Q of the new token, but the K and V of all previous tokens must be cached and reused. KV Cache memory usage can be estimated as:

    1. Memory = 2 (K/V) × layers × sequence length × KV heads × head dim × bytes per element × batch size

    Frequent follow-ups include: why can GQA (Grouped-Query Attention) and MQA compress the KV Cache? The answer is that multiple query heads share the same KV heads, shrinking the KV Cache to a fraction—or even a tenth—of its original size, at the cost of requiring careful training or migration strategies to recover accuracy.

    Memory Fragmentation and Paged Management

    Traditional inference frameworks pre-allocate contiguous memory based on the maximum sequence length, causing severe internal and external fragmentation; actual utilization can fall below 30%. The industry-standard solution borrows virtual memory concepts from operating systems: split the KV Cache into fixed-size blocks and use a block table to map logical positions to physical blocks, enabling on-demand allocation and memory sharing across requests (e.g., shared prefixes). Candidates who proactively draw the analogy to OS page tables often open up bonus topics like copy-on-write and prefix-aware scheduling.

    Batch Scheduling Strategies

    Continuous batching replacing static batching was a key step in improving inference throughput: requests dynamically join or leave the batch after each decode iteration, eliminating head-of-line blocking where short requests wait for long ones. Interviews often ask how it fundamentally differs from traditional batching, and how it combines with two-stage Prefill/Decode disaggregated scheduling (so-called “PD disaggregation”), which places stages with different compute characteristics on separate resources.

    Architectural Techniques Supporting Long Context

    Extrapolation and Interpolation of Positional Encoding

    One hidden factor limiting context length is positional encoding. Rotary Position Embedding (RoPE) degrades sharply when extrapolating beyond the training length. Mainstream solutions include:

    • Linear interpolation: scale all position coordinates uniformly, trading resolution for length coverage;
    • NTK-style scaling: adjust frequency components non-uniformly—low-frequency components handle extrapolation while high-frequency components preserve local precision;
    • Progressive length extension during training: train on short sequences first, then continue training on longer data.

    A classic interview question: why does direct extrapolation collapse while interpolation works? A good answer discusses the wavelengths of RoPE’s different frequency components and their coverage in the training distribution.

    Long-Sequence Engineering for Training and Inference

    Long-context training faces activation memory explosion, requiring activation recomputation, sequence parallelism, and context parallelism to shard long sequences across multiple GPUs, with full attention communication only at shard boundaries. Interviewers often ask how context parallelism differs from tensor parallelism: the former splits along the sequence dimension, the latter along weights/heads—their communication patterns and applicable stages are entirely different.

    Distributed Parallelism: Must-Know Answers on the Training Side

    Combinations of Data Parallelism (DP), Tensor Parallelism (TP), Pipeline Parallelism (PP), and Expert Parallelism (EP) are standard for large model training. High-frequency questions include:

    1. Why does TP split at attention heads and the MLP intermediate layer? Communication volume is minimal and can overlap with forward computation;
    2. How does 1F1B scheduling in PP reduce pipeline bubbles? By interleaving micro-batches so forward and backward passes overlap in the pipeline;
    3. What distinguishes the three ZeRO stages? Progressive sharding of optimizer states, then gradients, then parameters—trading communication for memory;
    4. Where is the EP communication bottleneck in MoE models? All-to-All token dispatch and aggregation, which must be analyzed together with capacity factors and load-balancing losses.

    When answering these questions, a good framework is “memory composition analysis → communication volume calculation → choice of sharding dimension,” demonstrating systematic reasoning.

    Decoding Strategies and Quantization

    Speculative Decoding

    Speculative decoding uses a small model to quickly draft multiple candidate tokens, which the large model verifies in a single parallel pass—turning serial decoding into “multi-step drafting + single-step verification” and accelerating generation without changing the output distribution. A common interview question: why must the verification stage use strict rejection sampling to guarantee consistency with the large model’s distribution? This tests mathematical understanding of Markov chains and accept-reject mechanisms.

    The Quantization Landscape

    Key interview topics include: the difference between weight-only and weight-activation quantization, symmetric vs. asymmetric quantization, granularity trade-offs in grouped quantization, why KV Cache quantization matters especially for long contexts, why outliers become prominent at low bit widths, and how scale-migration approaches like SmoothQuant mitigate them. Being able to sketch the “FP16 to INT8/INT4 accuracy-memory-throughput trade-off curve” and explain the inflection points is a hallmark of a high-scoring answer.

    Interview Preparation Tips

    1. Build a layered knowledge tree: organize knowledge by “operator layer → scheduling layer → architecture layer → cluster layer,” and prepare 2–3 questions per layer whose formulas you can derive by hand;
    2. Master memory estimation problems: these appear in nearly every interview—practice quick estimates of parameter memory, activation memory, and KV Cache memory;
    3. Use illustrative code: like the online softmax example above, it instantly boosts the credibility and professionalism of your answers;
    4. Adopt a cost perspective: tie technical choices to inference cost (e.g., serving cost per million tokens) to demonstrate business awareness;
    5. Get hands-on: run experiments with or reproduce mainstream open-source inference engines to connect paper principles with real engineering and tuning experience.

    At its core, the AI Infra interview tests whether a candidate can move fluidly among mathematical principles, system constraints, and engineering trade-offs. Anchoring your preparation on the long-context thread and connecting the four pillars—attention optimization, KV Cache management, parallelism strategies, and quantization—into a coherent system will let you deliver logically complete, high-quality answers to any follow-up question.