Decode Speed and Cost Per Task: Engineering MoE Sparse Architecture, Hybrid Attention, and Output-Length Compression
Model selection is shifting away from leaderboard scores toward the metric engineering teams actually optimize: cost per task. This article breaks down where that cost comes from and how to compress it. It covers MoE sparse architectures that decouple total parameters from per-token compute, hybrid attention that shrinks KV cache to cut long-context memory and latency, self-distillation plus reinforcement learning that buys capability while shortening output, and inference-serving techniques such as continuous batching and prefix caching that raise decode throughput. It also provides a reusable table for translating technical choices into cost impact, plus five steps developers can apply immediately.
1. A Shift in Perspective: From “Cost per Million Tokens” to “Cost per Task”
Over the past two years, the first metric for choosing a large model was the leaderboard score, the second was context length, and price often came last. In 2025, this order is reversing.
Take the wave of “Flash”-style compact models as an example. In 2025, many providers rolled out lightweight, high-speed, low-cost variants within their existing model families. The typical playbook is sparsification, reducing context overhead, shortening output length, and evaluating by real-world task performance as the primary criterion. This signals that the competitive focus has moved from the “capability ceiling” to “cost-effectiveness”: when capabilities are close to top-tier models, whoever can keep each task to a few cents is more likely to make it into production systems.
The per-token price and the cost per task are two different things. The per-token price is a vendor’s pricing label; the cost per task is what an engineering team actually optimizes. The end-to-end cost of a task can be written as:
Cost_task = Σ_i (N_in_i × P_in_i + N_out_i × P_out_i) / N_successwhere: N_in_i / N_out_i —— input / output tokens of the i-th model call (including failed retries)P_in_i / P_out_i —— the corresponding input / output unit pricesN_success —— number of tasks that met the expected goal
This formula immediately exposes a common misconception. When you look only at unit price, many people believe a price cut alone can drive cost down by an order of magnitude. But plug into the formula above: halving the input only halves the input portion; estimating input and output as roughly half each, total cost drops by just 25%. Only when both input and output are halved does total cost fall by 50%. To compress to roughly one-twentieth, unit price, token count, cache hit rate, and task success rate must all be optimized together.
Different product choices across the industry reinforce this point. High-speed tiers in cloud services usually lead with low price, while self-hosted platforms often use time-limited free access to grow usage. This shows that “cost per task” is simultaneously an engineering metric and a product metric. The following sections break down its sources across five areas.
2. Sparsification: Decoupling “Total Parameters” from “Per-Token Compute”
A single forward pass of a large language model involves a large number of matrix multiplications, and the main cost is “activated weights × token count.” In a dense model, all parameters participate in every computation, so the parameter count directly determines compute consumption.
Mixture of Experts (MoE) splits the feed-forward layer into multiple “experts,” and each token is processed by only the few experts selected by a gating network. Taking the common setup of “several experts, activating Top-K per token” as an example:
def moe_forward(x, experts, gate_proj, k=2):# x: [B*T, d], one d-dimensional vector per token# experts: several feed-forward experts, each with a weight size comparable to a dense model's FFN# gate_proj: the gating network that maps each token to a score for each expertlogits = x @ gate_proj.weight # [B*T, num_experts]topk_ids, topk_weights = logits.topk(k, dim=-1)topk_weights = torch.softmax(topk_weights, dim=-1) # normalize only across the selected expertsy = torch.zeros_like(x)for e in range(len(experts)):# only process tokens routed to expert e; compute cost ∝ number of tokens hittoken_ids = (topk_ids == e).nonzero(as_tuple=True)[0]if token_ids.numel() == 0:continuey[token_ids] += topk_weights[token_ids, 0] * experts[e](x[token_ids])return y
The classic early criticism of MoE in the community was “too many parameters yet slow to compute, with heavy memory usage.” The root cause is exactly that the implementation above runs each expert one by one with a for loop. Modern MoE inference frameworks first sort tokens by their target expert, then group and batch the computation, so the loop overhead becomes negligible. The only irreducible part is the matrix multiplication of the selected experts. Therefore:
- The activation ratio determines compute, and can be approximated directly as the multiply-accumulate work of the forward pass;
- The total parameter count determines memory footprint, and indirectly determines how many accelerators a deployed inference service needs.
So the accurate profile of a sparse model is “large memory demand, low compute consumption,” not the vague “both cheap and fast.”
Comparison with dense models. Using activation ratios common in the industry of 5%–20%:
- A 1000B-class MoE model with a 5% activation ratio activates about 50B per token;
- A 300B-class MoE model with a 20% activation ratio activates about 60B per token;
- The former needs roughly 3× the total memory of the latter, yet consumes less compute.
The former is bottlenecked by memory capacity, making it better suited for expert sharing across similar models and high-concurrency consolidated deployment; the latter is bottlenecked by compute, making it better suited for small-to-medium scale or single-card deployment.
Sparsification is not free. It comes with at least three costs:
- Training stability: routing makes gradients sparse, requiring load-balancing loss or a lossless balancing strategy to prevent a few experts from being overused while the rest are “starved”;
- Fine-tuning support: parameter-efficient fine-tuning updates only the attention and expert submodules; if the wrapper layer does not support sharding expert parameters, training will error out, or implicitly fall back to full training and the memory cost spikes;
- Quantization difficulty: expert weights are numerous and scattered; when quantizing expert by expert, experts with low activation lack sufficient calibration data, making the error hard to control.
Takeaways for developers: estimate speed and concurrency with the “activated parameter count” rather than the “total parameter count”; use memory to estimate deployment scale; and before fine-tuning, confirm whether the toolchain supports the expert structure.
3. Hybrid Attention: The Real Cost Is Not Token Count, but the KV Cache
During autoregressive generation, each generated token must attend over all previous tokens, so inference frameworks cache the keys and values of prior tokens for this purpose. This is the KV cache:
KV_per_token = 2 × num_layers × num_kv_heads × head_dim × bytes_per_elementKV_total = KV_per_token × concurrent_requests × sequence_length
Take a typical dense structure as an example: 32 layers, 8 KV heads, head dimension 128, FP16 storage:
- Each token occupies 2 × 32 × 8 × 128 × 2 = 128 KiB;
- A long-context request of 128K tokens (128K) needs about 16 GiB for the KV cache alone;
- If a long context carries multiple concurrent requests at once, the memory required grows proportionally.
And for each generated token, the data that must be read from memory includes all activated weights plus the entire KV cache. At a 128K context, the KV cache read volume already reaches several times the weight read volume, so latency and power are governed mainly by memory access rather than compute.
This is why long contexts are expensive: the cost bottleneck is not the number of input tokens, but the KV cache. The longer the context, the more data each decode step must move, the fewer concurrent requests one card can hold, and the lower the per-card throughput. Ultimately, the cost amortized onto each task rises sharply. Compressing the KV cache affects all three metrics at once: latency, throughput, and cost.
The mainstream approaches in the industry fall into three categories:
- Fewer heads: Multi-Query Attention (MQA) lets all query heads share one set of K/V, while Grouped-Query Attention (GQA) shares them per group. Many mainstream open-source models adopt GQA, partly to reduce this overhead;
- Compressed representation: compressing K/V into a smaller latent vector in a low-rank way, with DeepSeek’s MLA (Multi-head Latent Attention) as a typical example; recent work such as tensor-product attention continues compressing along the same line of thinking;
- Hybrid attention: most layers use a low-cost local attention such as sliding window; a few layers keep global attention responsible for long-range retrieval; the two are stacked in a fixed-ratio alternating pattern.
Hybrid attention is often understood as “saving compute by computing attention less often,” which is inaccurate. The scaled dot-product portion of full causal attention needs about 2 × n × d multiply-accumulates per token, a small fraction relative to the weight matrix multiplications (about 2 × P_act) at a 32K context with tens of billions of active parameters. Its real value is in memory: for a local layer with a limited window, the KV cache does not grow linearly with context, so the whole model’s KV cache is reduced proportionally by the “share of global layers.”
When the per-token KV cache overhead drops to one-sixth of the dense baseline, the same memory can:
- carry about 6× the long-context concurrency;
- or substantially raise the number of users a single card can serve;
- and markedly shorten decode latency under long contexts.
Hybrid attention is not without cost: information from tokens that fall entirely outside the window cannot be retrieved directly afterward, so it must be compensated for by global layers, an extra retrieval mechanism, or cache compression and summarization. For long-context applications, this should be validated specifically against scenarios such as “needle retrieval” and “multi-document QA.”
Two engineering notes:
- Paged KV cache management: the core idea of PagedAttention is to split the KV cache into fixed-size blocks, allocated on demand and stored non-contiguously, greatly reducing fragmentation. It is now widespread in mainstream open-source inference frameworks such as vLLM and SGLang, with essentially no extra hardware overhead. As a result, the memory bottleneck of long context is almost entirely determined by the “per-token KV size dictated by the model structure”; engineering optimization can only reduce waste;
- prefix caching with automatic prefix reuse: its hit depends on block-level prefix matching, block by block, and it can hit only on complete blocks. Therefore, if any token in the prefix changes, all subsequent blocks are invalidated. This also explains why input-side caching usually offers only a “partial hit” discount.
4. Post-Training: “Buying Capability and Saving Length” with Self-Distillation and Reinforcement Learning
Pre-training sets the capability ceiling of a model; post-training determines how it uses that capability. For cost-conscious small models, post-training must resolve one core tension: being close to a large model’s intelligence without buying correctness through “writing more.”
1. Self-distillation: capability standing on the shoulders of a large model
“Flash”-style small models usually do not complete the full from-scratch pre-training pipeline independently; instead, they do post-training on top of the training results already achieved by a larger model in the same family. A common approach uses a strong teacher’s full hidden-layer states for intermediate-layer distillation, transferring knowledge to models of a different structure.
A sound training recipe is usually staged:
| Stage | Training objective | Main effect | Data characteristics |
|---|---|---|---|
| Stage 1 | Supervised fine-tuning (SFT) only | Learn basic dialogue and instruction format | Diverse, broad coverage |
| Stage 2 | Response-level distillation from the teacher | Fit the teacher’s output distribution, improve general capability | Teacher samples + reward filtering |
| Stage 3 | Reinforcement learning (preference optimization) | Align with user preferences, compress output length | Preference pairs, verifiable rewards |
A common objective for response-level distillation is to minimize reverse KL divergence over the student’s outputs:
# Score the student-sampled sequences with the teacher and compute the per-token reverse KL# Reverse KL penalizes tokens where "the student assigns high probability but the teacher thinks otherwise"; overall it is mode-seekingteacher_logp = teacher(input_ids, labels=input_ids).logits.log_softmax(-1).gather(...)student_logp = student(input_ids, labels=input_ids).logits.log_softmax(-1).gather(...)kl = (student_logp.exp() * (student_logp - teacher_logp)).sum(-1).mean()loss = sft_loss + alpha * kl # alpha controls distillation strength
Compared with ordinary supervised fine-tuning that only learns a “standard answer,” distillation directly approaches the teacher’s distribution, so the student picks up more of the preferences embedded in the probability distribution and reaches a higher quality ceiling at the same size.
2. Reinforcement learning: making the model “say it clearly in one pass” instead of “thinking as it speaks”
Output length is often overlooked, yet it accounts for half of the inference cost. In the generation phase, every output token requires a full forward pass, so output tokens are usually several times more expensive than input tokens. If a model tends to answer with “first build up context, then summarize, then confirm,” the token count can easily swell to two or three times what is actually needed.
Reinforcement Learning with Verifiable Rewards (RLVR) is the mainstream solution: use “whether the task was truly completed” as the reward signal, then add a length penalty to guide the model toward concise expression. The training loop can be written as:
for prompt in prompts:responses = [policy.generate(prompt) for _ in range(G)] # sample G responses per questionrewards = [task_reward(prompt, r, len(r)) for r in responses]advantages = normalize(rewards) # compare within the same question group, replacing the value networkpolicy.update(prompt, responses, advantages)
Here, the key design of task_reward is “reward correctness, penalize redundancy”:
def task_reward(prompt, response, token_len):correct = verifier(prompt, response) # verifiable signal such as rule checks or unit testsr_correct = 1.0 if correct else 0.0# given "correct", shorter is better; penalize the portion over budget at extra costr_length = -beta * max(0, token_len - budget) / budgetreturn r_correct + r_length
The core of this design is turning “done” and “concise” into measurable signals. In code-generation tasks, unit-test pass rate is a natural reward; in summarization and information-extraction tasks, you can rely on rule checks or a preference model. Length compression works because the model finds a better path between “passing the tests” and “expressing concisely.” Public reinforcement-learning practice also shows that such methods often substantially reduce the number of inference tokens without sacrificing accuracy.
3. Prioritize artifact quality: don’t make decisions on leaderboard scores
The last easily overlooked piece is evaluation. Vendors’ public benchmarks are usually dominated by “public leaderboard scores,” but what an engineering team really needs is: given a real work artifact (code, a report, a spreadsheet, a reply), is the output actually usable?
A sound approach is to replace the evaluation set with “task-level” metrics:
- Artifact usability rate: the proportion of outputs judged usable by humans or rules;
- Average output length: a measure of real cost;
- End-to-end latency P95: the user experience;
- Cost per task: the realized value of Cost_task.
5. Inference Serving: Decode Speed Determines “How Much Concurrency You Can Sell”
Pre-training and post-training set the model’s ceiling; inference optimization determines “how many users one card can serve.” The optimization paths of mainstream inference frameworks are relatively mature:
- Continuous batching: rather than waiting for the whole batch of requests to finish, it immediately admits a new request as soon as one completes, greatly boosting GPU utilization;
- Prefix caching: the KV cache for identical system prompts and few-shot examples is computed only once, and subsequent requests hit it directly, saving a large amount of prefill compute;
- Quantization: INT8/FP8/INT4 weight quantization reduces memory-bandwidth pressure, and KV cache quantization (such as FP8) further lowers long-context overhead;
- Speculative decoding: a small model first “guesses” several tokens, a large model verifies them in parallel, and if accepted, it advances multiple tokens at once, giving an effective several-fold speedup in decode;
- MoE expert parallelism: dispatch tokens by the device where each expert lives, suited to deploying models with a large total parameter count and small active parameter count across multiple cards;
- Request scheduling and prioritization: schedule long and short requests separately to avoid long requests blocking the queue.
“Flash”-style models commonly claim “decode speed can reach over 150 tps.” What does this number mean in engineering terms?
- 150 tps is about 6.7 ms per token. Compared with human reading speed (tens of characters per second), this is already close to an “instant generation” experience;
- More importantly, decoding is a memory-bound process, where throughput ≈ memory bandwidth ÷ the number of weight and KV bytes that must be read per token. Reducing active parameters from tens of billions to the ten-billion range lowers the per-token read volume proportionally, so decode speed rises proportionally and the number of requests a unit of compute can serve rises with it.
This is exactly the final link in the sparse architecture’s “cost reduction”: it is not just a lower unit price, but that the same hardware can carry several times the number of users.
6. Translating Technical Choices into Cost: A Reusable Estimation Table
When an engineering team makes a selection, it can directly apply the following approach, converting technical differences into cost differences (parameters are illustrative; actual measurements should prevail):
| Stage | Technical choice | Impact on cost per task |
|---|---|---|
| Model structure | Dense vs. sparse MoE | Fewer active parameters → lower per-token compute, faster decoding |
| Attention structure | Standard attention vs. hybrid attention | KV cache down to 1/6 → several-fold higher long-context concurrency |
| Post-training | Plain SFT vs. self-distillation + RLVR | Stronger capability at the same size, shorter output length |
| Evaluation | Leaderboard score vs. artifact quality | Selection closer to real scenarios, less rework after launch |
| Inference optimization | Naive inference vs. continuous batching + prefix caching | Higher throughput, lower amortized cost per token |
Plug this table into the formula: lowering the per-token cost on the model side, halving the output length, and raising the cache hit rate stack together so that the cost per task can easily reach the “one-twentieth of before” range. This is not achieved by any single technology alone, but is the result of system-level optimization.
7. Landing Recommendations: Five Things Developers Can Do Right Now
- Build your own evaluation set first: collect input/output samples from 20–50 real business tasks, get “artifact usability rate + average output length” working, and only then discuss selection;
- Enable prefix caching first: put the system prompt and few-shot examples at the very front of the request, ensuring they are byte-for-byte identical across different requests;
- Structured output: constrain the output format with a JSON Schema or grammar constraints to reduce invalid tokens and the retries caused by format correction;
- Tiered routing: send simple tasks to a lightweight model and escalate complex tasks to a strong model, finding the inflection point along the cost/quality curve;
- Observability first: bring token usage, cache hit rate, and P95 latency into logs and monitoring so that optimization is grounded in data.
8. Conclusion: Model Competition Enters the “Cost per Unit Task” Era
Back to the opening question: why is model selection shifting from “leaderboard score” to “cost per task”? Because what the business is buying has never been the model itself, but the outcome of “getting the task done.” When capabilities are close, commercial success is decided by who can deliver results at lower cost, faster, and more reliably.
From a technical standpoint, MoE sparse architecture compresses compute, hybrid attention compresses the KV cache, self-distillation plus reinforcement learning compresses output length, and inference optimization compresses the hardware cost per token. These four technologies stack together to form the entire source of the cost-effectiveness advantage of “Flash”-style models. None of them is a “single-point black magic”; all are combined optimization of the engineering system.
For developers, this means two things:
- When selecting, don’t look only at parameters and leaderboards; ask “on my task, how much does each task cost and how long does it take to get a result?”;
- When building, don’t pin all optimization hopes on the model vendor; engineering measures such as caching, routing, output constraints, and observability can equally deliver several-fold cost gains.
As model capabilities gradually converge, engineering efficiency becomes the new moat.
