Open-Weight LLM Post-Training and Agent Evaluation in Practice
This article explains why post-training now determines the capability ceiling of open-weight LLMs, why total parameters do not equal inference cost in MoE models, how to interpret terminal-agent and web-retrieval benchmarks, and how to build reproducible evaluation pipelines and layered deployment practices for production agents.
1. Background: Post-Training Now Sets the Capability Ceiling
Recent open-weight LLMs show a clear trend: base architectures are stabilizing, and capability gains increasingly come from sustained investment in post-training. The industry already has MoE models with trillion-level total parameters and tens of billions of activated parameters, supporting million-token context and native multimodal input. An important engineering characteristic of these models is that a new version may not replace the base model. Instead, it adds substantial post-training compute on the same base through supervised fine-tuning, preference optimization, reinforcement learning, tool-use trajectory training, long-context data construction, and multimodal alignment to improve behavior quality on complex tasks.
For developers, this means model selection cannot rely only on parameter scale or whether the weights are open. It requires breaking the decision into three questions. First, does the base model have enough capacity and context window? Second, does post-training cover the tool calling, multi-step reasoning, and error recovery capabilities required by the target scenario? Third, does inference cost match business latency, concurrency, and budget targets? Trillion-level total parameters do not mean all parameters are activated for every inference. MoE architectures usually activate only a subset of experts, so total parameters measure the capacity ceiling, while activated parameters more directly affect compute and memory bandwidth pressure per inference.
2. Architecture and Inference Cost: Total Parameters Are Not Inference Cost
The core of an MoE model is to select a subset of experts through a routing mechanism. Larger total parameters allow the model to hold more knowledge and patterns; larger activated parameters make each inference more compute-intensive. Engineering concerns include expert load balancing, expert-parallel communication, batching efficiency, KV cache footprint, and the growth in attention computation caused by long context. Million-token context does not mean every request should fill the context window. Long context significantly increases memory and latency costs, so a production system should combine retrieval-augmented generation, context budget control, and segmented summarization.
For example, the following pseudocode only expresses the context budget and routing control flow and does not depend on any specific interface:
function build_context(query, docs, budget):ranked = rank_by_relevance(query, docs)selected = []used = 0for doc in ranked:if used + estimate_tokens(doc) <= budget:selected.append(doc)used += estimate_tokens(doc)else:breakreturn compress_if_needed(selected, budget)function route_request(request):if request.complexity == "simple":return small_model_poolif request.needs_long_context:return long_context_pool_with_cachereturn large_model_pool
In engineering practice, it is advisable to split model serving into a routing layer, caching layer, inference layer, evaluation layer, and observability layer. Simple requests go to small models or cache; complex agent tasks go to large models; long-document tasks are retrieved and compressed first; multimodal inputs are normalized in size and processed with OCR or structured extraction during preprocessing, avoiding sending raw large files directly into context.
3. Agent Capability Evaluation: Reading Terminal Tasks and Web Retrieval
In public benchmarks, terminal agent tasks usually test planning and correction capabilities under multi-step operations, tool calls, and environment feedback. Web retrieval tasks test retrieval, evidence integration, citation consistency, and noise resistance. In recent public results, some open-weight models have narrowed the gap with closed-source flagships on terminal agent tasks to less than one percentage point and are in the first tier on retrieval tasks. This signal is valuable, but it cannot be directly equated with production readiness.
There are three reasons. First, agent evaluations are highly sensitive to tool policy, timeout settings, retry counts, and environment snapshots; different evaluation configurations can significantly change results. Second, public evaluations may have data contamination or distribution bias, while real business long-tail scenarios are more complex. Third, agent tasks are judged not only by the final answer but also by whether intermediate steps are auditable, whether permissions are exceeded, and whether irreversible operations occur. Therefore, enterprises should use public metrics for candidate screening and self-built evaluations as the admission basis.
4. Reproducible Evaluation Pipeline
A reproducible evaluation should at least fix the following elements: model weight version and hash, decoding parameters, system prompt, tool list and permissions, timeout and retry policy, environment dependency snapshot, scoring rules, human-reviewed samples, and cost and latency records. For each task, record success rate, average steps, tool call count, failure type, cost per run, and P95 latency, and use confidence intervals to determine whether differences are significant.
function evaluate_suite(model, cases, tool_policy, budget):results = []for case in cases:context = build_context(case.query, case.docs, budget)trajectory = run_tool_loop(model, context, tool_policy)result = score(case, trajectory)results.append({success: result.success,steps: trajectory.steps,tool_calls: trajectory.tool_calls,cost: estimate_cost(trajectory),latency: trajectory.latency,failure_type: result.failure_type})return summarize_with_confidence(results)
The evaluation set should cover normal paths, boundary conditions, tool failures, permission denials, long-context noise, and multimodal inputs. For terminal-type tasks, execution should take place in an isolated sandbox, and real write operations should be prohibited. For retrieval-type tasks, cited sources and the evidence chain should be checked to avoid a correct answer with incorrect evidence.
5. Engineering Best Practices
A layered strategy is recommended for deployment: use rules and small models for high-frequency simple requests, and use large models for complex planning and multi-step tool calls; apply semantic caching to repeated questions; use retrieval augmentation and segmented summarization for long documents; preprocess and structurally extract multimodal inputs; add permission whitelists, sandbox execution, human confirmation, and audit logs for agent operations. On the deployment side, teams can use general-purpose container platforms, object storage, logging services, and monitoring and alerting capabilities, such as those provided by Baidu AI Cloud, to build model serving, evaluation pipelines, and observability systems. The final choice should still be driven by compliance, cost, and latency targets.
At the same time, establish a continuous regression mechanism: run a fixed evaluation set on every model or prompt change and compare success rate, cost, latency, and failure types; sample and replay online traffic; set circuit breakers and rollback for high-risk operations. Only by connecting public metrics, self-built evaluations, and production monitoring can teams determine whether an ultra-large MoE model is truly suitable for the business.
6. Conclusion
Trillion-level MoE, million-token context, and native multimodality are pushing open-weight models into more complex agent scenarios. Post-training determines behavior quality, activated parameters determine inference cost, and evaluation methodology determines selection confidence. Developers should avoid being led by a single score and instead build reproducible, auditable, and regression-ready evaluation and deployment systems to turn model capability into stable engineering capability.
