Agent Inference Optimization: Low-Bit Decoding and Hybrid Attention
This article explains how to control inference cost for AI agents by framing cost around completing a task rather than raw per-token price. It covers optimization across four layers: architecture (sparse activation and MoE, hybrid attention for long context, low-rank experts, and scaling ladders), post-training (self-distillation, token-based length constraints, and dynamic thinking budgets), evaluation (moving from public leaderboards to internal task-level test sets), and inference (continuous batching with paged KV, low-bit decoding with W4A16 and KV quantization, operator fusion, MoE expert parallelism, and speculative decoding). It closes with a pre-launch checklist and stresses reporting both performance and cost under clear, consistent test conditions.
1. Background: Why Agent Inference Costs Are Getting Harder to Control
In agent applications, users rarely send a single simple question. Instead, they submit an entire chain of tasks. For example, “generate a quarterly report,” “restructure a table,” or “turn meeting notes into a to-do list.” The model must read source material, call tools, produce intermediate results, check formatting, spot errors, and repair them iteratively. The final output may be a document, a spreadsheet, a slide deck, code, or structured data.
This leads to three direct changes:
- Longer context: System prompts, tool definitions, conversation history, and long-document summaries keep accumulating, so input tokens keep growing.
- Longer outputs: To “say everything,” the model generates more reasoning, explanation, and formatting content, which pushes up output tokens and cost.
- Higher concurrency pressure: Enterprise services must handle long requests, short requests, traffic spikes, and tool-call round trips at the same time, raising the bar for throughput and tail latency.
As a result, what enterprises care about is no longer “how high the leaderboard score is,” but three questions:
- Can the task be completed quickly and well?
- What is the total cost of completing one task?
- Can the service scale reliably at an acceptable cost?
1.1 From per-token price to per-task cost
The traditional view looks at the price per million tokens, but what users actually pay for is “completing a task.” Per-task cost can be written as:
Cost(task) = Σ_i [ P_in_i × N_in_i + P_out_i × N_out_i ] - Save_cache
Here, P_in_i and P_out_i are the input/output unit prices of the i-th call, N_in_i and N_out_i are the corresponding token counts, and Save_cache is the cost saved by reuse mechanisms such as prefix caching.
This formula shows that cost depends not only on unit price but also on token consumption. Cutting only the unit price without controlling length, or only shortening outputs without optimizing input reuse, yields only local gains. A complete cost picture should also account for hidden overhead such as retries, failure rollbacks, cache storage, and manual correction.
1.2 Three optimization directions
| Direction | Common techniques | Main benefit |
|---|---|---|
| Lower per-token cost | Sparse activation, low-bit quantization, operator fusion, cache discounts | Reduces compute and memory-access overhead |
| Reduce token consumption | Prefix caching, context compression, length constraints, tool-call batching | Reduces total input and output volume |
| Shorten task latency | Decoding optimization, continuous batching, prefill/decode disaggregation, speculative decoding | Improves first-token speed and generation throughput |
The goal is not simply “make the model smaller,” but to reduce per-task cost and shorten task duration at the same time while preserving task quality.
2. Architecture Layer: Reducing Parameter Computation and KV Cache Pressure
2.1 Sparse activation: decoupling total and active parameters
In a dense model, every token passes through all parameters, so the larger the model, the higher the compute per forward pass. Mixture-of-experts (MoE) models use a routing mechanism that lets each token activate only a small number of experts, reducing actual compute while preserving knowledge capacity.
For example, a model can contain hundreds of experts, but each token is routed to only a few of them, supplemented by shared experts that handle general capabilities. This splits model scale into two dimensions:
- Total parameters: determine knowledge capacity and memory footprint.
- Active parameters: determine the actual compute per token.
The value of this decoupling is that a model can expand its knowledge capacity without significantly increasing per-call compute, making inference serving easier to keep under cost control.
2.2 Hybrid attention: compressing long context beyond GQA and MQA
In long-context scenarios, the KV cache is the main burden on memory and bandwidth. During autoregressive decoding, every generated token requires reading the keys/values of all historical tokens. For a model with 48 layers, 64 attention heads, a head dimension of 128, and an FP16 cache, the per-token KV usage per layer is:
2 × 64 × 128 × 2 = 32768 bytes ≈ 32 KiB
The 128K-context estimates for different attention schemes are as follows:
| Attention scheme | KV heads / compression per token | 48-layer 128K context estimate | Relative compression |
|---|---|---|---|
| MHA | 64 KV heads | ~192 GiB | — |
| GQA (8 groups) | 8 KV heads | ~24 GiB | ~8× |
| MQA | 1 KV head | ~3 GiB | ~64× |
| Low-rank latent scheme | Latent compression | Illustrative ~16 GiB, depends on dimensions | Needs measurement |
| Hybrid scheme | 1 global layer every 5 layers, sliding window on the rest, window 4096 | ~44.8 GiB | ~4.3× |
The hybrid scheme uses a consistent accounting: among 48 layers, one global KV layer is kept every five layers, and the remaining layers keep KV only within the window. Under this convention, there are about 10 global layers and about 38 sliding-window layers.
The estimate proceeds as follows:
- Global layers: 10 layers × 128K tokens × 32 KiB/token ≈ 40 GiB
- Sliding-window layers: 38 layers × 4096 tokens × 32 KiB/token ≈ 4.75 GiB
- Total: ~44.75 GiB
The core idea of hybrid attention is division of labor across layers: most layers focus only on a local window, preserving local detail while compressing the cache; a few layers keep global attention to handle long-range dependencies across sections. Unlike GQA and MQA, which mainly compress the number of KV heads, hybrid attention further distinguishes global and local caches by layer, making it better suited to long-context serving.
Note that compression is not lossless. Before going live, you should validate perplexity, long-context retrieval ability, and performance on real business tasks to confirm there is no clear regression.
2.3 Low-rank experts: reducing expert-weight memory-access pressure
In MoE, an expert is usually a feed-forward network. If the input dimension is d and the intermediate dimension is m, the two weight matrices contain 2 × m × d parameters. A low-rank expert rewrites the weight matrix as the product of two low-rank matrices:
W1 = A · B, A ∈ ℝ^{m×r}, B ∈ ℝ^{r×d}, r << min(m, d)Parameter count is roughly 2 × r × (d + m)
For example, with d = m = 2048, the original parameter count is about 8.4 million; with r = 256, the parameter count is about 1.05 million, roughly 8× compression.
The benefit of low-rank experts is mainly in the decoding stage. Decoding is often limited by memory bandwidth, so reducing the weights that need to be moved can directly improve throughput. The tradeoff is that representational capacity may drop, so you can keep full rank on a few critical layers, or increase the number of experts, and compensate during subsequent training.
2.4 Scaling ladder: validating architecture at low cost
If architecture changes are tried directly at large scale, the cost of trial and error is high. A more reliable approach is to build a scaling ladder:
- Choose proxy scales such as 1B, 3B, 7B, and 14B.
- At each scale, train different architecture variants — for example dense vs. MoE, different attention ratios, and whether low-rank experts are enabled.
- Fit the relationship between loss and compute/data, and extrapolate to the target scale.
- Choose a design based on combined training cost and inference cost.
The decision criterion should not look only at training loss, but also at the target capability curves — for example, the inflection points at which capabilities such as coding, tool calling, and long-context understanding change with scale.
3. Post-training: Making Models Smarter and Output More Controllable
3.1 Self-distillation: raising the intelligence density of small models
Complex reasoning, instruction following, and multi-step task planning often require supervision from a stronger model. The idea of self-distillation is to form a “teacher-student” structure within the model: first perform strong post-training on a large base, then use its outputs as supervision signals to train the target model.
Common forms include:
- Response-level distillation: train the student model with questions, answers, chains of thought, or tool-call traces generated by the teacher.
- Distribution-level distillation: align the student model’s output distribution with the teacher’s soft-label distribution.
The essence of self-distillation is to amortize reasoning-stage capability forward: the teacher only contributes during training, while the inference stage carries only the cost of the student model.
Three things need attention during implementation:
- Data filtering: remove factual errors and low-quality samples from the teacher’s outputs.
- Capability mix: capabilities such as general dialogue, code, reasoning, and tool calling need to be mixed in reasonable proportions.
- Forgetting monitoring: continuously regression-test the student model’s pre-existing capabilities from before distillation.
3.2 Length constraints: controlling output by token count, not character count
Verbose outputs push up both token cost and latency. During the reinforcement learning stage, you can introduce length-aware reward shaping, but length statistics must be based on the model tokenizer, not character count. Example:
def task_reward(completion, verifier, tokenizer, target_tokens: int,alpha: float = 0.1, max_penalty: float = 0.3):"""target_tokens is the target number of output tokens, not characters.n_tokens is computed with the model tokenizer to stay consistent with the inference-time token count.max_penalty caps the length penalty so the model does not over-shorten its answers."""success = verifier(completion) # 1.0 / 0.0n_tokens = len(tokenizer.encode(completion))over_ratio = max(0, n_tokens - target_tokens) / target_tokenslength_penalty = min(alpha * over_ratio, max_penalty)conciseness_bonus = 0.05 if n_tokens <= target_tokens else 0.0return success - length_penalty + conciseness_bonus
Design points:
- Task completion first: the main reward must dominate, so the model does not cut corners just to shorten output.
- Soft-constraint penalty: deduct points proportionally when over budget, and set a penalty cap.
- Anti-gaming: prevent the model from boosting the length metric by truncating answers or omitting necessary steps.
3.3 Thinking budget: allocating reasoning resources by task difficulty
Simple tasks can be answered directly, while complex tasks can enable deeper reasoning. A more practical approach is to set dynamic policies in the system:
- Format conversion, information extraction, simple rewriting: use fast mode.
- Multi-step analysis, code generation, complex tool calls: enable deep thinking and set a maximum reasoning-token limit.
This avoids spending too much reasoning resource on simple tasks while ensuring complex tasks have enough room to finish.
4. Evaluation Layer: From Leaderboard-driven to Task-level Evaluation
Public leaderboards have value for academic research, but they have limits when used for product selection: task distributions may deviate from the business scenario, question banks may be optimized against once public, most leaderboards favor single-turn Q&A, and they look only at accuracy — not at latency, token consumption, or cost.
A more reasonable approach is to build an internal task evaluation set. For example, an office scenario can cover:
| Category | Typical task | Acceptance focus |
|---|---|---|
| Document generation | Generate a report from source material | Complete structure, factual accuracy |
| Document editing | Adjust format or tone per instructions | Instruction following, accurate scope of edits |
| Table repair | Fix misaligned tables | Correct formatting, no data loss |
| Human-model co-editing | User and model edit alternately | Context consistency, correct incremental edits |
| Audio/video processing | Meeting minutes, subtitle generation | Accurate transcription, clear summaries |
| Visual understanding | Chart Q&A, screenshot understanding | Correct spatial relations and visual-element recognition |
Evaluation should stick to four principles:
- Artifact-driven: judge the final deliverable, not just the model’s text.
- End-to-end: include tool calls, retries, and corrections.
- Automatable: prefer rule checks, code execution, and format validation.
- Dynamically updated: periodically sample from real requests to keep the set realistic.
Metrics should not look only at accuracy, but should include at least artifact quality, output length, end-to-end time, per-task cost, cache hit rate, and failure/retry rate.
5. Inference Layer: Low-bit Decoding and Decoding Acceleration
5.1 Why decoding speed matters more to experience than theoretical compute
Generative inference has two phases:
| Phase | Characteristics | Main bottleneck | User impact |
|---|---|---|---|
| Prefill | Processes input tokens in parallel | Compute-intensive | Affects first-token latency |
| Decode | Generates autoregressively token by token | Memory-bandwidth-intensive | Affects generation speed and total duration |
Agent tasks produce long outputs, so decoding time often accounts for the bulk of the total. Tool calls add multiple rounds of waiting, so decoding throughput has a large impact on experience.
5.2 Continuous batching and paged KV
Static batching must wait for all requests in the batch to finish, so GPU utilization is low when long and short requests are mixed. Continuous batching lets completed requests exit and new requests join immediately, improving GPU utilization.
Paged KV cache splits the KV cache into fixed-size pages allocated on demand, avoiding fragmentation from reserving contiguous memory for the maximum length. Combining the two improves single-card concurrency.
5.3 How to do low-bit decoding: W4A16 and KV quantization
Weight quantization compresses FP16 weights to 4-bit, reducing memory footprint and bandwidth pressure. A common scheme is W4A16: weights are stored in 4-bit and dequantized to FP16 to participate in matrix multiplication during computation.
W4A16 considerations:
- Use group-wise or channel-wise quantization to reduce local error.
- Keep high precision on critical layers to avoid a clear quality drop.
- Handle activation outliers, using mixed precision where necessary.
- Use high-quality calibration data, or adopt quantization-aware training.
- Verify that the dequantization operator and fused kernel actually deliver throughput gains.
KV quantization applies to long context. Compressing the KV cache to 8-bit or 4-bit can free memory and support longer context or higher concurrency. But KV quantization is sensitive to precision, so it should be combined with quantization strategies aware of rotary position embeddings, channel-wise scaling, or grouping, and validated through long-context task regression.
5.4 Operator fusion: reducing kernel launches and memory-access round trips
In the decode phase, generating each token requires running many small operators — for example RMSNorm, QKV projection, RoPE, attention, residual connections, FFN/MoE routing, and sampling. Every kernel has launch overhead, and repeatedly reading and writing intermediate results to memory wastes bandwidth.
The goal of operator fusion is to merge many small operators into a few large kernels so that intermediate results stay in registers or shared memory as much as possible, reducing HBM round trips and improving parallelism and pipeline efficiency. For small-batch, high-frequency, bandwidth-sensitive decode workloads, the fusion gains are usually clear.
5.5 MoE inference: expert parallelism and routing optimization
The main challenges of MoE inference include load balancing, All-to-All communication, and routing overhead. Common optimizations:
- Expert parallelism, to reduce the number of experts per device.
- Grouped routing, to reduce cross-node communication.
- Dynamic batching, to merge tokens hitting the same expert from different requests into one computation.
- Expert caching, to keep high-frequency experts resident in memory.
- Coordinated computation of shared experts and routed experts, to improve general capability.
5.6 Speculative decoding: improving decode throughput with a draft model
Speculative decoding uses a lightweight draft model to generate multiple candidate tokens, which the target model then verifies in parallel. Accepted tokens are kept, everything after the first mismatch is discarded, and generation continues.
The speedup depends on the acceptance rate and the draft cost. If the draft model shares the target model’s architecture, or uses a self-speculation scheme that skips some layers, you can keep the draft cost under control while maintaining a high acceptance rate. Because the final output is still verified by the target model, this does not introduce additional distribution bias in principle.
5.7 Performance claims must include the test conditions
Engineering articles often state things like “exceeds 150 TPS,” but without test conditions this can mislead. A more reliable phrasing is:
Under a controlled test set on specified hardware, with a specified batch size, a specified input/output length range, and continuous batching plus the corresponding cache strategy enabled, decoding speed can reach the order of 150 TPS; actual deployment is affected by hardware model, batch size, sequence length, quantization implementation, KV cache strategy, and network overhead.
Cost comparisons should also use the same conditions — for example, the same task set, the same input-length distribution, the same output-length cap, the same concurrency configuration, and the same timing method. Otherwise, different systems are not comparable.
6. Pre-launch Checklist
Before a model goes live, at least the following checks are recommended:
- Accuracy regression: compare the original and quantized models on core tasks.
- Long-context retrieval: verify information recall at 128K or longer context.
- Perplexity change: check whether perplexity is acceptable after W4A16 and KV quantization.
- TTFT and TPS: measure first-token latency and generation throughput separately.
- Cache hit rate: observe whether prefix caching and KV reuse actually reduce repeated computation.
- Concurrency stress testing: verify stability under long-tail requests, traffic spikes, and tool-call chains.
- Rollback plan: prepare fast rollback mechanisms for model versions, quantization configurations, and routing policies.
7. Conclusion
Inference optimization in the agent era is not a pile of single-point techniques, but coordination across architecture, training, evaluation, and inference engineering. A reasonable path is: first control token consumption, then compress the KV cache, then do low-bit decoding and operator optimization, and finally validate the results with task-level evaluation.
For developers, choosing a model should not depend only on parameter scale or leaderboard ranking, but should focus on decoding speed, long-context ability, tool-call stability, and per-task cost. For enterprises, building an internal task set, defining measurement conditions clearly, and continuously tracking TTFT, TPS, and cost is more valuable than chasing a single metric.
