From Conversational AI to Agentic Coding: Deploying General-Purpose LLMs on Domestic AI Chips
This article examines the shift from conversational large language models to agentic systems that can execute, verify, and revise engineering tasks. It covers the four core capabilities of agentic coding, the cost implications of sparse MoE architectures, task-oriented evaluation, production constraints, and a practical approach to deploying mixed-precision inference on domestic AI accelerators. It also outlines rollout strategies, monitoring requirements, and common implementation pitfalls.
1. The Center of Gravity Is Shifting: From “Chatting” to “Doing”
Over the past year, the key dividing line between general-purpose large language models has moved beyond conversational fluency. The more important question is whether a model can deliver usable results in a closed-loop task.
The shift from “conversation” to “execution” essentially changes the model’s output from a piece of text into an executable, verifiable, and reversible engineering action: writing a file, calling a tool, completing a pipeline, or fixing a failing test case.
This transition requires a combination of capabilities rather than a single state-of-the-art score:
- Coding ability determines whether an action is correct.
- Agent capability determines whether individual actions can be connected into a workflow.
- Reasoning ability determines whether the system can replan after failure.
These capabilities must remain balanced. A model with strong coding ability but weak planning may go off course during long-running tasks, while a model with strong reasoning but weaker implementation quality may produce unreliable code in production.
2. Four Capability Layers of Agentic Coding
2.1 Code Generation and Completion
The requirement is not merely syntactically valid code. Generated code must also follow the repository’s existing conventions, including naming, error handling, and dependency versions.
Public information indicates that newer models in this category have achieved double-digit percentage improvements over their predecessors in code generation. They also maintain relatively consistent performance across multiple languages and full-stack scenarios, including frontend interaction layers, server-side APIs, data processing, and scripts.
In practice, the quality of the experience depends more on feeding the right context than on how “intelligent” the model appears to be.
2.2 Tool Calling and Contract Stability
An Agent’s upper limit is determined by its toolset. The model must reliably generate structured tool-call parameters. The schema violation rate should generally remain below 1%; otherwise, retries at the orchestration layer can quickly multiply costs.
{"name": "run_tests","description": "Run the specified test scope in an isolated sandbox","parameters": {"type": "object","properties": {"scope": { "type": "string", "enum": ["unit", "integration", "e2e"] },"selector": { "type": "string" },"timeout_s": { "type": "integer", "minimum": 1, "maximum": 1800 }},"required": ["scope", "timeout_s"]}}
2.3 Long Context and Repository-Level Understanding
Newer models commonly extend their context windows to approximately 200K tokens. However, context-window size should be understood as a capacity limit, not a guarantee of effective utilization.
When retrieved content is positioned in the middle of a long context, recall can decline significantly. Engineering implementations should therefore follow three practices:
- Put critical constraints near the beginning.
- Deduplicate retrieved content.
- Compress long material into summaries.
2.4 The Plan–Execute–Verify Loop
A mature Agent loop follows the pattern:
Plan → Make the smallest necessary change → Execute and verify → Replan based on failure information
The system should also enforce hard limits on the maximum number of iterations, per-task budgets, and tool permissions.
def agent_loop(task, tools, budget):ctx = load_context(task, budget.max_tokens)for step in range(budget.max_steps):if budget.exhausted():return escalate(task, reason="budget_exceeded")action = model.plan(ctx, tools, structured=True)if action.kind == "final":return verify(action.patch, task.criteria)result = sandbox.run(action, timeout=action.timeout_s) # Isolated executionctx.append(compress(result)) # Keep only failure summaries and key diff informationreturn escalate(task, reason="max_steps")
3. Sparse MoE Architectures: Why More Parameters Do Not Necessarily Mean Higher Cost
This class of models uses a sparse Mixture-of-Experts (MoE) architecture. Publicly reported figures indicate approximately 355B total parameters, with around 32B parameters activated for each token.
The economic implications are important:
| Dimension | Dense Large Model | Sparse MoE |
|---|---|---|
| Memory usage | Proportional to total parameters | Proportional to total parameters; weights must remain resident |
| Compute per token | Proportional to total parameters | Primarily related to activated parameters |
| Throughput characteristics | Compute-intensive | More sensitive to memory bandwidth and communication |
| Suitable scenarios | Small and medium scale, low latency | Large scale, high concurrency, batch-friendly workloads |
The main conclusion is that MoE shifts cost from compute toward memory and scheduling.
When concurrency is low, the memory overhead of keeping all weights resident may not be fully amortized. When concurrency is high, the smaller number of activated parameters can provide a significant throughput advantage.
Combined with inference-side optimizations, such as improved token efficiency and structured output that reduces unnecessary generation, MoE can reduce per-task costs substantially compared with dense models offering similar capabilities.
In practice, services based on comparable models may be priced at a relatively small fraction of leading international solutions. Cost-sensitive applications should therefore prioritize calculating cost per task, rather than looking only at the unit price.
4. Evaluating Real Tasks: Do Not Rely Only on Public Leaderboards
Public benchmarks for mathematical reasoning, long-horizon coding, and broad human-level problem solving are useful for relative positioning, but their relevance to a specific business is limited.
A better approach is to build a three-layer evaluation framework.
4.1 Point Capability Layer
Extract 50–200 cases from real production issues, preserving the original logs as context. Measure the pass@1 rate and the success rate of the first repair attempt.
4.2 End-to-End Task Layer
Evaluate the complete workflow:
Requirement definition → Generated changes → Passing tests → Passing review
Track the average number of iterations and the average number of tokens consumed.
4.3 Stability Layer
Run the system continuously for 72 hours and monitor:
- Schema violation rate
- Timeout rate
- Unauthorized tool-call attempts
report = {"pass_at_1": passed_first / total,"avg_steps": sum(steps) / total,"avg_tokens": sum(tokens) / total,"schema_error_rate": schema_errors / total_calls,"cost_per_task": total_cost / total,}assert report["schema_error_rate"] < 0.01, "Tool contract is unstable; add few-shot examples and a validation layer"
One important signal is that, on some real-world programming task sets and specialized code evaluations, this class of models has reached a range comparable to mainstream international models. The significance is not necessarily absolute leadership, but rather evidence of replaceability.
5. Deploying and Constraining Native Agent Capabilities
Automated Documentation and Presentation Generation
First generate a structured outline, such as JSON, and then render it through a template engine. The model should be responsible for content and layout descriptions, while the template system controls final rendering. This reduces uncontrolled layout drift.
Frontend Code Optimization
Break “performance optimization” into measurable subtasks, such as:
- Initial page-load size
- Number of requests
- Rendering-blocking work
Each subtask should correspond to an automatically verifiable quality gate, allowing the model to converge through feedback.
General Production Constraints
All generated artifacts should be executed in an isolated sandbox. Outputs should be stored consistently in object storage and assigned version identifiers. Changes should be validated through the container platform’s CI pipeline.
Logs and execution traces should be written to a centralized logging service. Monitoring and alerting should track both failure rates and costs.
6. Adapting to Domestic AI Chips: From “It Runs” to “It Runs Reliably”
Deploying a large model on a domestic AI accelerator is not primarily about achieving a successful inference run once. The real challenges are precision consistency, predictable performance, and safe canary rollback.
Consider a mixed FP8 and Int4 quantized inference implementation on a domestic AI chip. A typical approach is to use FP8 for compute-intensive matrix multiplication paths in order to preserve numerical dynamic range, while using Int4 to compress memory-heavy weights. Sensitive layers, such as normalization and attention scoring, retain higher precision.
The adaptation process can be organized into four steps.
6.1 Assess Operator Coverage
Measure the execution share of each operator in the model. Prioritize frequently used operators and fallback paths for unsupported operators. The fallback ratio should ideally remain within the single-digit percentage range.
6.2 Align the Precision Baseline
Use the output of a floating-point implementation with matching precision as the reference, then perform both layer-level and end-to-end regression testing.
def check_parity(golden, candidate, tol_layer=1e-3, tol_task=0.005):for name, ref in golden.layers.items():err = rel_error(ref, candidate.layers[name])assert err < tol_layer, f"layer {name} drift {err:.2%}"task_gap = golden.task_score - candidate.task_scoreassert task_gap < tol_task, f"task score regression {task_gap:.2%}"
6.3 Establish Performance Baselines and Tune Batching
Benchmark single-request latency and batch throughput separately. Confirm that the scheduling strategy does not introduce communication bottlenecks under the interconnect topology of the domestic accelerators.
In many cases, a mainstream open-source inference engine combined with a proprietary acceleration library can complete one adaptation cycle within one to two weeks.
6.4 Canary Deployment and Rollback
Increase traffic gradually from 5% to 25% and then to 100%. The monitoring dashboard should track:
- Time to first token
- Timeout rate
- Business task success rate
- Cost per task
If any red-line threshold is exceeded, traffic should immediately be routed back to the previous implementation.
7. Deployment Roadmap and Common Pitfalls
Recommended Roadmap
- Use a small task set to validate capability and cost.
- Run the complete Agent loop in a sandbox.
- Use a mixed-precision quantization solution on domestic AI chips for batch and offline workloads.
- During traffic peaks, route requests among models of different sizes according to quality and cost requirements.
Common Pitfalls
- Treating context-window length as effective context utilization.
- Failing to impose iteration and budget limits, resulting in uncontrolled costs.
- Lacking a structured validation layer and incorrectly attributing parsing errors to insufficient model capability.
- Using an evaluation set that does not match the production distribution, causing metrics to drop sharply after launch.
Overall, the value of this generation of open-source general-purpose large models lies in making capability substitution practical. MoE architectures help control cost per task, long-context support and native Agent capabilities cover real engineering workflows, and mixed-precision deployment on domestic AI chips can further reduce inference costs.
For enterprise developers, the real competitive advantage comes from three engineering capabilities: a robust evaluation system, well-designed tool contracts, and disciplined canary operations. It should not depend solely on any single model.
