Agentic AI Architecture in Open-Weight LLMs: Key Design Choices and Trade-offs

    This article explains how the architecture of open-weight large language models affects agentic workloads. Using the DeepSeek-V3 technical report as the main reference, it covers sparse mixture-of-experts design, the difference between total and active parameters, latent attention for KV cache reduction, sparse attention, multi-token prediction, and engineering checks for deployment. It also notes which conclusions depend on a specific model and must be verified for each one.

    1. Introduction: New Requirements Agentic AI Places on Models

    An agent differs from a standard chat model in how it executes. An agent usually completes several steps: it reasons, calls tools, reads the returned results, decides whether to retry, and finally summarizes the outcome. Each step consumes inference time and GPU memory. The model therefore needs to be accurate, and it also needs to produce structured decisions reliably within a limited latency and resource budget.

    This is why open-source models are valuable. Open weights let teams deploy models privately, adapt them to specific domains, tune their inference services, and design their own safety and governance policies. A group of open-source large models released in the past two years follows a similar direction. Their total parameter counts are growing, only a subset of parameters is activated for each inference pass, the attention mechanism uses less cache, and they include built-in multi-token prediction.

    This article uses the public technical report for DeepSeek-V3 as its main reference and analyzes the architectural features of this type of sparse large model. All figures in the article come from that report. Other open-source models may be designed differently, so each one needs to be checked separately when making a selection.

    2. Overall Architecture: Consider Total and Active Parameters Separately

    DeepSeek-V3 has about 671B total parameters, and about 37B parameters are activated for each token during the forward pass. The activated parameters include the selected routed experts and the shared experts, but not the experts that were not selected. The gap between the two numbers comes from the sparse structure. The model stores the weights of all experts, but each token passes through only a small subset of them.

    This design has two direct effects. First, the total parameter count determines how much knowledge and how many patterns the model can store, which helps with understanding complex instructions, selecting tools, and interpreting intermediate results. Second, the active parameter count determines the compute cost per token, which is directly tied to latency and throughput.

    Sparse models reduce the compute per token, but they do not reduce the weight storage needed for deployment. Memory planning should therefore still be based on the total parameter count. The parallelism strategy then determines how the weights are split across multiple GPUs.

    3. Mixture of Experts: Routed Experts and Shared Experts

    The mixture-of-experts structure splits the feed-forward network into multiple experts. In DeepSeek-V3, each sparse layer contains 256 routed experts. For each token, a router selects the 8 routed experts with the highest scores to take part in the computation, and their outputs are summed using gating weights. Shared experts are always active for every token. The logic is illustrated below, with normalization and gating details omitted:

    1. score = router(hidden_state)
    2. selected = top_k(score, k=8)
    3. routed_out = sum(w_i * expert_i(hidden_state) for i in selected)
    4. output = shared_expert(hidden_state) + routed_out

    The main difficulty in routing is load balancing. If many tokens concentrate on a few experts, those experts become overloaded while others sit idle, and both training and inference efficiency drop. DeepSeek-V3 adjusts expert selection with a dynamic bias, which reduces its reliance on auxiliary loss.

    Shared experts handle general language ability and format control, while routed experts specialize in finer-grained patterns. For agents, if shared experts provide a fallback for the JSON formatting and instruction-following needed for tool calls, output inconsistencies caused by routing fluctuations should decrease. This is the intended design, but its effect still needs to be confirmed through evaluation.

    The first three layers of DeepSeek-V3 are dense layers that do not use routing. Dense shallow layers can first form a stable base representation before the sparse layers process it. Other models may have different configurations, so their own technical documentation should be consulted.

    4. Attention: MLA and Sparse Attention Solve Different Problems

    The main cost of long context comes from the key-value (KV) cache. Each token that has been generated must store its keys and values, so the cache grows as the context gets longer. Multi-head Latent Attention (MLA), used in DeepSeek-V2 and V3, targets the size of this cache.

    MLA first projects the hidden state into a low-dimensional latent vector. During caching, only this compressed representation is stored. When needed for computation, the required keys and values are reconstructed. The key idea can be summarized as follows:

    1. latent = down_proj(hidden_state) # cache latent
    2. k, v = up_proj(latent) # recover at compute time

    This approach reduces the cache cost per token, but it does not reduce the range of attention computation. Each query still computes attention against all historical positions.

    Sparse attention targets the range of computation instead. Each query attends to only a small set of candidate positions, which reduces the attention compute. DeepSeek Sparse Attention (DSA), used in later DeepSeek versions, uses a lightweight indexer to select a subset of historical tokens for each query. Native Sparse Attention (NSA) is another approach, which combines three branches: compression, selection, and a sliding window. The implementation details of these methods differ considerably, so they should not be grouped together as a single type of sparse attention.

    MLA and sparse attention can be combined, but their benefits come from different sources. MLA mainly saves cache memory, and sparse attention mainly saves compute. Evaluations should measure memory usage and attention computation time separately.

    5. Multi-Token Prediction: A Training Signal That Can Also Speed Up Inference

    Multi-token prediction (MTP) lets the model predict several subsequent tokens at each position at the same time. In DeepSeek-V3, the MTP module provides an additional learning signal during training, which helps the model plan further ahead.

    During inference, the MTP module can act as a draft model for speculative decoding. It proposes candidate tokens, and the main model verifies them in a single pass. Tokens that pass verification are accepted directly. When verification fails, decoding falls back from the point of failure. When a rejection sampling rule is used, the output distribution matches standard decoding.

    The value of MTP is most visible in structured output. Content such as tool-call arguments, JSON field names, and fixed-format plan lists is relatively easy to predict, so candidate tokens are usually accepted at a high rate. Acceptance rates for open-ended writing may be noticeably lower. MTP may therefore deliver significant gains for structured output tasks, but the benefit depends on the task type and should be measured.

    The validation method is straightforward. Under the same hardware, concurrency, and input distribution, run the model with MTP enabled and disabled, then compare first-token latency, output tokens per second, and format compliance rate.

    The model architecture sets the upper limit of capability and cost, but business results also depend on the engineering implementation. The following two checks relate directly to the architectural choices discussed in this article.

    1. Expert load monitoring. Record the number of tokens each expert receives within a fixed time window, and track the ratio of maximum load to average load. If this ratio stays high over time, routing is concentrating on a few experts, and the parallel partitioning or routing parameters need adjustment.
    2. Tiered KV cache management. Divide the cache into tiers based on context length and session activity. Active sessions stay in GPU memory, while historical sessions are offloaded to…