Asymmetrizing Activation Cost: How Inference Architecture Evolves Under Long-Input, Short-Output Workloads
Agent-based workflows have inverted the balance between reading context (prefill) and writing output (decode), with input-to-output ratios often reaching 10:1 or 20:1. This article explains why asymmetric activation, a causal encoder-decoder layer split, two-level sparse indexing, and lookup-based memory each address a mismatch between one-time and high-frequency costs in conventional symmetric MoE. It also covers the evaluation pitfalls that make throughput claims vary by 3x, the engineering tradeoffs and assumptions that need independent validation, and a practical checklist for deciding whether these techniques fit your workload.
1. The Workload Has Changed: Agents Split “Reading” from “Writing”
To understand any evolution in inference architecture, start with how the workload is shaped.
In traditional chat scenarios, inputs run from tens to a few thousand tokens and outputs from a few hundred to a few thousand tokens, so prefill and decode are roughly the same order of magnitude. In agent flows built around tool calls, that distribution inverts structurally: a single request first packs in the system prompt, tool definitions, retrieved repository snippets, multiple rounds of tool return values, and the interaction history, easily reaching tens or even hundreds of thousands of input tokens. Meanwhile, what actually gets “written” is often just a short stretch of tens to hundreds of tokens: call arguments or a small piece of code.
The read-to-write ratio is frequently 10:1 or even 20:1, and this ratio dictates where optimization should focus. The prefill phase must compress massive context into a usable representation in one pass; it is a “one-time cost” that can tolerate higher parameter activation because it is highly parallel and can saturate compute. The decode phase reruns the network to produce every single token; it is a “high-frequency cost,” and any extra activated parameters get multiplied by the output length.
The Two Phases Don’t Share the Same Bottleneck
| Dimension | Prefill (read) | Decode (write) |
|---|---|---|
| Parallelism | High; sequence dimension is splittable | Very low; strictly autoregressive and serial |
| Main bottleneck | Compute, bandwidth, context-compression capacity | Memory bandwidth, KV reads, per-step latency |
| User experience metric | TTFT (time to first token) | TPOT / generation speed |
| Cost profile | One-time; amortizable via caching | Accumulates linearly with output length |
Here’s the problem: traditional MoE uses symmetric activation, so prefill and decode go through the same routing and computation graph with identical activated parameters. The two optimization goals are therefore forced into alignment. Lowering activated parameters to cut decode latency weakens prefill’s context-modeling capacity; scaling up activation to improve context-compression quality makes every output token more expensive.
2. Asymmetric Activation: A Separate Budget for Each Phase
The core idea of asymmetric activation is simple: decouple input-side activated parameters from output-side activated parameters, giving prefill and decode their own independent compute budgets.
A schematic sketch (pseudocode expressing the mechanism only, not any specific implementation):
# Schematic: prefill and decode use different expert/parameter budgetsdef forward(tokens, phase):if phase == "prefill":k = k_prefill # Larger budget: for context compression and representation buildingelse:k = k_decode # Smaller budget: to guarantee per-token latencyrouter = top_k_gating(hidden, k=k)return moe_layers(hidden, router)
From an engineering standpoint, the payoff is “structural” rather than the result of parameter tuning:
- Pay for context compression once, reuse it many times. A wider prefill path encodes long context into a more compact representation, and that cost is paid only once at the start of the request.
- Output-side latency decoupled from input length. Decode runs only the smaller half of the budget, so per-step compute is independent of input length, assuming KV access is optimized (see the next section).
- Cache-friendly. When the prefix is stable, prefill results can be reused across requests, and the asymmetric design places “the expensive half” exactly where it can be cached.
The cost is equally clear: two activation budgets mean two sets of hyperparameters and two capacity-planning tracks, and both training and inference scheduling must branch by phase.
3. Causal Encoder-Decoder Split: Who Handles “Memory,” Who Handles “Generation”
For asymmetric activation to actually land, it needs a matching layer structure. One natural approach is to cut the network vertically into two segments:
- Lower layers as the Encoder: run only in the prefill phase, compressing long context layer by layer into a compact representation, and not participating in per-token generation.
- Upper layers as the Decoder: this half does most of the work in the decode phase.
- Global KV is projected once from the encoder’s final-layer hidden states, so decoder layers no longer each rebuild their own global KV from their own hidden states.
The third point is the key one and deserves its own explanation. In a conventional decoder-only Transformer, every layer maintains and reads/writes its own KV cache, and the total KV size under long context is roughly layers × sequence_length × heads × head_dim. This directly determines memory footprint and the bandwidth pressure during decode: the more layers and the longer the context, the more KV reads dominate the cost.
If instead you store a single global KV, projected from the encoder’s final layer, and have decoder layers access it via projection or modulation:
# Schematic: encoder produces context representation; decoder layers reuse the same global KVctx_hidden = encoder_stack(prompt_embeds) # Runs only in the prefill phaseglobal_kv = project_kv(ctx_hidden) # One projection, shared across all layersfor layer in decoder_layers:local_kv = layer.self_attn.local_cache() # Only tokens within the window; cheapattn = layer.self_attn(query, local_kv, global_kv)
The chain of benefits: context representations can be more compact, because decoder layers no longer need to rebuild global information at each layer, so the global context only needs to keep one thoroughly compressed summary. In one sentence: it turns “each layer’s global memory” from repeated storage into a single shared projection.
The engineering caveats are equally clear: shared KV tightens the coupling between “layers” and “context memory,” so scaling along the depth dimension needs to be revalidated; and the encoder’s compression is lossy, making tasks like multi-hop reasoning, which repeatedly backtrack over original details, the highest-risk cases.
4. Two-Level Sparse Indexing: Pulling Attention Out of O(n²)
With shared KV in place, the remaining question is “how much to read.” The general form of two-level sparse indexing is:
- Level one (coarse): score the context by block/segment, select a few candidate blocks, and shrink the candidate set.
- Level two (fine): do token-level scoring within the candidate blocks and select the true top-k tokens to participate in attention.
- Layer on a local sliding window to ensure nearby dependencies aren’t missed.
# Schematic: two-level sparse selectiondef sparse_attention(q, k, v, block_size=64, top_blocks=8, top_tokens=256):scores_b = block_score(q, k, block_size) # Level one: block-levelcand = select_top_blocks(scores_b, top_blocks) # Coarse filterscores_t = token_score(q, k[cand]) # Level two: token-level within blockidx = select_top_tokens(scores_t, top_tokens) # Fine filteridx = merge_with_local_window(idx, q) # Guarantee a local windowreturn softmax(q @ k[idx].T) @ v[idx]
The value of the two-level structure is that it trades a controllable extra scoring cost for a drop in attention complexity, and level one can run on lower-precision or coarser representations, making the actual cost far lower than the computation saved. What needs watching is retrieval recall: if the block-level coarse filter misses a key block, no amount of level-two refinement can recover it. Before shipping, validate recall with two evaluation sets, “long-context needle-in-a-haystack” and “multi-hop QA,” rather than looking at throughput alone.
5. Lookup-Based Memory: Separating “Memory” from “Computation”
One clear architectural trend over the past year: not everything that needs parameters is worth carrying via matrix multiplication.
A large number of low-frequency but clearly valuable patterns—fixed collocations, entity names, proper nouns, code identifiers, API signatures, internal project conventions—are inherently deterministic lookup relationships. Using multi-layer Transformers to “compute” them amounts to using general-purpose computation to simulate a hash table.
A lookup-based memory module works like this: hash a local n-gram, look up the corresponding embedding vector, then inject it into the backbone via residual or modulation.
# Schematic: n-gram lookup memorydef ngram_memory(h, tokens, table, n=3):idx = hash_ngrams(tokens, n) # Hash local spansmem = table[idx] # Lookup, almost zero FLOPsmem = mem * gate(h) # Gated injection, avoid unconditional additionreturn h + proj(mem)
The tradeoffs of this mechanism are very clear:
- Benefit: injects the ability to “remember specific patterns” at extremely low compute cost, letting backbone parameters focus on reasoning and compositional generalization; the same lookup memory can scale across tasks and domains.
- Cost 1, memory footprint. Large embedding tables are themselves big and usually resident in host memory, fetched on demand, which introduces lookup latency and scheduling complexity.
- Cost 2, hash collisions. Different spans mapping to the same slot cause memory pollution, requiring collision-rate monitoring and multi-hash/multi-table design.
- Cost 3, fuzzy boundaries. What belongs in the lookup table versus the Transformer has no theoretically optimal answer; it can only be tuned to the task distribution, and the payoff drops on OOD inputs.
An important calibration: lookup memory improves “local pattern reproduction and naming consistency,” not knowledge updates. Expecting it to replace retrieval or a knowledge base is aiming in the wrong direction.
6. Evaluation: Why Throughput Numbers Vary by 3x
The same architecture can yield wildly different conclusions under different measurement methods. When you see a claim like “several times faster generation,” first confirm the following:
- Whether prefill time is included. Counting only decode-phase tok/s systematically overstates end-to-end experience.
- Input/output lengths. tok/s at 1k input and at 100k input are not directly comparable.
- Cache hit rate and TTFT. In agent scenarios, stable prefixes are re-read heavily; the end-to-end gap between a 99% and a 90% hit rate may be larger than the gap between models themselves.
- Concurrency and batch size. Single-request latency and throughput are two separate curves and must be reported separately.
- Quantization, parallelism strategy, hardware model. These variables often matter more than architectural differences.
I recommend collecting at least the following metrics, reported by phase:
# Schematic: per-phase timing and end-to-end accountingdef bench(model, prompt, n_out):t0 = time.perf_counter()prefill_out = model.prefill(prompt)t1 = time.perf_counter()for _ in range(n_out):model.decode_step()t2 = time.perf_counter()ttft = t1 - t0tpot = (t2 - t1) / n_outreturn {"ttft_s": ttft,"tpot_ms": tpot * 1000,"decode_tok_s": 1.0 / tpot,"prefill_tok_s": len(prompt) / ttft,"e2e_tok_s": (len(prompt) + n_out) / (t2 - t0),"cache_hit": model.metrics.cache_hit_ratio,"kv_bytes_peak": model.metrics.kv_peak_bytes,}
For agent-oriented, real-world business evaluation, also treat total task time, total token consumption (including failed retries), tool-call format compliance rate, and cache hit rate as first-class metrics, not just tokens per second.
7. Engineering Costs and Assumptions That Need Independent Validation
Look at all these mechanisms together and they are mutually coupled; the benefits don’t simply add up. The following must be measured on your own scenario:
- Training stability. Introducing asymmetric activation, the encoder-decoder split, sparse indexing, and lookup injection all at once significantly raises the hyperparameter space and initialization sensitivity, so budget for recovery-training costs up front.
- Inference-stack coupling. The shared KV layout, two-level sparse selection, and lookup memory access all require custom kernels. General-purpose inference frameworks have limited support for custom attention structures, so the model and inference engine become tightly bound and the upgrade window narrows.
- Lookup-memory gains are statistical. It improves local pattern consistency; it cannot replace retrieval augmentation or fix factual errors.
- Sparse-index recall risk. Long-context retrieval, multi-hop reasoning, and cross-document synthesis are three high-risk scenarios that must each be evaluated with a dedicated test set.
- Depth and width scaling. Sharing global KV changes the relationship between layers and context, so the original scaling laws need to be revalidated and cannot be directly extrapolated.
- Tail latency. Both lookup memory access and the variable-length selection in sparse indexing can introduce long tails, and P99 latency often hurts real experience more than P50.
8. Deployment Advice and a Selection Checklist
First, judge whether the workload matches. If your requests are generally “long input + short output” (code completion, repository Q&A, tool-call orchestration, document extraction), the prefill-to-decode ratio far exceeds 1, and asymmetric activation plus shared KV pay off most; if your scenario is long-form generation with token-by-token output, symmetric activation is the more reasonable choice.
Put caching on your side. Pin stable prefixes (system prompts, tool definitions, knowledge-base summaries) to the very front of the request so the cacheable region is maximized, and push dynamic content as far back as possible. The return on this single change usually beats switching models.
Instrument by phase, and don’t look only at averages. In monitoring, collect TTFT, TPOT, prefill throughput, peak KV usage, cache hit rate, and tail percentiles separately at minimum. These metrics can be captured through general monitoring, alerting, and logging services on cloud platforms to build a baseline and form comparable historical curves.
Ask three questions when selecting:
- How much does long-context recall drop after sparsification on your own test set?
- Does the memory footprint of lookup memory eat up the space saved by KV compression?
- Once the inference framework is upgraded, how costly is migrating this custom attention?
Back to the original workload insight: the direction of architecture optimization is always determined by the ratio of “reading” to “writing.” The shared logic behind asymmetric activation, shared KV, two-level sparse indexing, and lookup memory is to price one-time costs and high-frequency costs separately, so each part’s budget goes to the phase that truly needs it. Understand this logic, and even facing entirely different implementation details, you can quickly judge whether something is worth the complexity for your scenario.
