LLM Techniques for Agentic Coding: Principles, Costs, and a Deployment Checklist

    This article breaks down the large-model techniques that matter for agentic coding workflows. It covers the three main bottlenecks (attention compute, KV Cache memory, and training signal), then explains sparse attention, multi-token prediction, and mixture-of-experts (MoE) architectures, including the formulas behind their costs. It also shows how repeated tool calls amplify long-context cost and how prefix caching reduces it, and closes with a pre-launch evaluation and deployment checklist plus common mistakes to avoid. All numbers are worked examples derived from formulas, not measured benchmarks.

    1. Background and Bottlenecks

    Early code tools only completed a single line or a single function, leaving problem decomposition, structure design, and incremental verification to the developer. Today’s agents read the codebase on their own, locate defects, modify multiple files, run tests, and keep iterating based on errors. A single task may trigger dozens of model calls, and each call carries an ever-growing history.

    This article is aimed at engineers who connect large models to these agent workflows. The scope is general mechanisms, not any specific model. The numbers below are examples computed from formulas, not measured results.

    Three bottlenecks are common in practice.

    The first is attention compute. The scoring part of standard self-attention scales with the square of the sequence length N and can be written as: compute ∝ N² × d, where d is the dimension of a single attention head. When the sequence grows from 4K to 128K, the scoring part grows by roughly (128K/4K)² = 32² = 1024×. The compute of the linear layers only grows linearly with N, so in long contexts this quadratic term becomes the dominant factor.

    The second is GPU memory usage. During inference, every layer caches the keys and values of the history, i.e., the KV Cache. Its memory formula is:

    KV Cache memory = 2 × num_layers × num_KV_heads × head_dim × seq_len × batch_size × bytes_per_element

    Example: 32 layers, 8 KV heads, head dimension 128, sequence length 128K, batch 1, BF16 precision (2 bytes per element) gives roughly 2 × 32 × 8 × 128 × 131072 × 1 × 2 bytes ≈ 17.2 GB, or about 16 GiB. This does not include the model weights. When computing, distinguish two phases. Prefill processes the entire input at once and is compute-heavy. Decode generates tokens one by one, reading the entire KV Cache at every step, and is mainly limited by memory bandwidth.

    The third is the training signal. Agents must learn multi-step decision making, and next-token prediction on static text alone struggles to teach this capability. Training requires verifiable environmental feedback, such as whether tests pass, command exit codes, and whether a task is completed. The scale, quality, and diversity of the data together determine the capability ceiling.

    2. Sparse Attention: Read Only the Most Relevant History

    The idea behind sparse attention is to let each query compute relevance against only a subset of historical positions. The three common approaches compare as follows.

    Method Long-range dependency Compute cost Suitable scenarios
    Sliding window Weak; information outside the window is invisible Low; simple to implement Local syntax, edits to adjacent lines
    Fixed pattern Depends on preset rules; may miss key positions Low Well-structured sequences
    Dynamic selection Strong; picks historical positions by content Medium; needs an extra indexing module Cross-file references, locating issues in long logs

    The core formula for dynamic selection is:

    output = sum over each position i in set S of softmax(q·k_i / √d) × v_i

    where S is the k positions chosen by the indexing module, q is the query vector, k_i and v_i are the key and value of position i, and d is the head dimension.

    Illustrative code follows.

    1. import math
    2. def topk_sparse_attention(query, keys, values, k):
    3. # Illustrative code: omits the /sqrt(d) scaling, causal mask, batch, and multi-head dimensions
    4. # 1. Compute the dot-product score between the query and each historical position
    5. scores = [sum(q * x for q, x in zip(query, key)) for key in keys]
    6. # 2. Select the top-k highest-scoring positions
    7. top_idx = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:k]
    8. # 3. Apply softmax only over the selected positions
    9. selected = [scores[i] for i in top_idx]
    10. m = max(selected)
    11. exps = [math.exp(s - m) for s in selected]
    12. total = sum(exps)
    13. weights = [e / total for e in exps]
    14. # 4. Weighted sum over the corresponding value vectors
    15. dim = len(values[0])
    16. out = [0.0] * dim
    17. for w, i in zip(weights, top_idx):
    18. for d in range(dim):
    19. out[d] += w * values[i][d]
    20. return out

    The code above only illustrates the selection flow and must not be used directly for model inference or in production. Two points deserve attention. First, scoring still has to scan all N positions. Real systems keep the indexing module very small so that the scan cost stays far below full attention. Second, exact attention applies only to the k positions, so when k is much smaller than N, the compute for this part drops noticeably.

    The cost of sparse schemes is recall risk. If the indexing module misses a key position, the output deviates from the full-attention result. Before launch, recall should be evaluated separately:

    recall = number of selected true key positions ÷ total number of true key positions

    The test set can include needle-in-a-haystack retrieval (insert one piece of information in the middle of a long text and then ask about it), cross-file function references, and locating errors in long logs. Testing only the beginning or the end of the text overstates the effect, because information in the middle is often harder for the model to exploit. In practice, the indexing module is usually trained jointly with the language modeling objective; otherwise the index scores and the true attention weights may not align.

    3. Multi-Token Prediction: Denser Training Signal, Not Necessarily Faster Inference

    The conclusion first: multi-token prediction gives each position more supervision signal during training, but a denser training signal does not guarantee faster inference. These two things must be verified separately.

    Standard training predicts only the next token at each position. Multi-token prediction adds m−1 prediction heads alongside the backbone, where head j predicts the (j+1)-th future token. The training loss is:

    L = L_1 + λ × (L_2 + … + L_m)

    where L_j is the cross-entropy loss of head j and λ is the weight of the auxiliary loss. If λ is too large, it interferes with the main head, so it needs to be tuned through experiments on your own training data.

    Illustrative code follows.

    1. def mtp_loss(head_logits, head_targets, lam=0.3):
    2. # Illustrative code: cross_entropy is an assumed cross-entropy function
    3. # head_logits[0] predicts the next token; head_logits[j] predicts the (j+1)-th future token
    4. losses = [cross_entropy(logits, target)
    5. for logits, target in zip(head_logits, head_targets)]
    6. return losses[0] + lam * sum(losses[1:])

    During inference, the extra prediction heads can propose draft tokens, and the main model verifies them in parallel with a single forward pass. This is a form of speculative decoding. The code below demonstrates only the greedy decoding acceptance rule; sampled decoding needs separate rejection-sampling rules.

    1. def speculative_step(model, mtp_heads, context, m):
    2. # Illustrative code: demonstrates only the greedy decoding acceptance rule
    3. draft = mtp_heads.propose(context, m) # draft tokens, length m
    4. verified = model.greedy_predictions(context + draft) # length m+1
    5. accepted = []
    6. for i, d in enumerate(draft):
    7. if d != verified[i]:
    8. break
    9. accepted.append(d)
    10. # The main model always produces one more token: it is the correction when a draft token is rejected, or an extra token when all drafts are accepted
    11. accepted.append(verified[len(accepted)])
    12. return accepted

    The upper bound on the benefit can be estimated. Assume each draft token is accepted with probability α and that acceptances are independent. With m draft tokens, the average number of tokens produced per verification is E = 1 + α + α² + … + α^m. For example, with α = 0.8 and m = 2, E = 1 + 0.8 + 0.64 = 2.44. But the actual speedup must subtract the extra compute from the prediction heads, the additional compute during verification, and scheduling overhead, so end-to-end speedup must be measured.

    The acceptance rate α must be measured on your own business data and cannot borrow someone else’s numbers. Acceptance rates differ sharply across code, test logs, and natural language text. When the sampling temperature rises, the acceptance rate usually drops. When the acceptance rate falls below the profitability threshold, the system should automatically turn off multi-token prediction and fall back to standard decoding.

    4. Mixture-of-Experts Structure and Long-Context Cost

    The mixture-of-experts (MoE) structure splits the feed-forward layer into M experts. The router first scores every token and selects only the top-k highest-scoring experts to participate in the computation:

    g = softmax(W_r × x)
    y = sum over the selected k experts e of g_e × E_e(x)

    where x is the hidden vector of the token, W_r is the router’s parameters, and E_e is the e-th expert. The total parameter count is determined by the number of experts, while the parameters actually activated per token include only the selected k experts plus the shared layers. For example, with 64 experts and 6 selected per token, the activation ratio of expert parameters is about 6/64, or 9.4% (excluding shared layers). This decouples model capacity from per-call compute.

    MoE has two common problems. The first is load imbalance. Some experts are selected by a large number of tokens while the rest sit nearly idle. During training, an auxiliary loss is often added:

    L_aux = β × M × sum over e from 1 to M of f_e × P_e

    where f_e is the fraction of tokens assigned to expert e, P_e is the router’s average probability for expert e, and β is the weight. The second is communication overhead. When experts are distributed across multiple devices, every layer needs an all-to-all communication. If communication bandwidth is insufficient or the batch is very small, expert utilization drops and the benefit shrinks accordingly. So MoE is better suited to deployment environments with enough devices and reasonably large batches.

    The cost of long context comes mainly from the amplification of call counts. Suppose a task makes T calls in total and the input length of the t-th call is L_t. The cumulative input of the task is:

    cumulative input = sum over t from 1 to T of L_t

    Example: a base prompt of 8000 tokens, 2000 new tokens per round, and 30 rounds in total. Without caching, the cumulative input is 30 × 8000 + 2000 × (0 + 1 + … + 29) = 240000 + 870000 = 1110000 tokens. If the provider supports prefix caching, only the new part per round needs to be computed, i.e., 8000 + 2000 × 29 = 66000 tokens, with the rest read from the cache. Prefix caching applies only to exactly identical prefixes, and billing rules follow the provider.

    Common ways to cut cost include: compressing past rounds, keeping only conclusions and essential errors; truncating overly long tool outputs while keeping the beginning and the end; placing the system prompt and tool definitions at the very front and keeping them unchanged, without inserting timestamps into the prefix; and limiting the maximum number of rounds per task.

    5. Evaluation and Deployment Checklist

    The table below lists the items to check before launch. Pass conditions are set by each team according to its own task set; no universal thresholds are given here.

    Check item Measurement method Pass condition
    Context length Test with the longest input from real tasks Key-information recall at the target length reaches the team’s set threshold
    KV Cache memory Compute the peak by formula and measure with the longest request Peak (weights + KV Cache + activations) stays below GPU memory capacity with headroom
    Call cost Track calls per task, input/output tokens, and cache hit rate Per-task cost no higher than the baseline, and traceable per task
    Test pass rate Fixed task set, averaged over multiple runs Not lower than the pre-optimization baseline
    Sparse recall Compute by the recall formula Not lower than the set ratio of the full-attention baseline
    Multi-token prediction Track acceptance rate, tokens produced per step, and end-to-end latency End-to-end latency decreases and the test pass rate does not drop
    MoE load Track token share per expert and communication time No clearly overloaded or idle experts
    Fallback switch Run tests with sparse attention and multi-token prediction turned off Results return to baseline, and switch toggling has test coverage
    Security Inspect the code execution environment and key locations Generated code runs in an isolated environment; keys are not written into prompts or logs

    Common mistakes are as follows.

    1. Looking only at the model’s total parameter count and ignoring the number of parameters actually activated per token.
    2. Testing recall with a single long text and ignoring performance on middle positions.
    3. Treating a drop in training loss as inference speedup.
    4. Not monitoring the cache hit rate and inserting timestamps into the prefix, which invalidates the entire cache.
    5. Concatenating tool outputs into the context without limit.
    6. Running the evaluation only once and ignoring result variance from sampling.
    7. Using the example code directly in production.

    In conclusion, these techniques are best suited to tasks involving large repositories, multi-file modifications, and repeated test runs. For single-round short code completion, the context and the number of calls are both small, so the benefit of these optimizations is usually limited. Readers should first use the baseline data in the table above to identify where the bottleneck lies, then decide which technique to introduce.