Benchmarking Chinese LLM Agents in Production: Three Trade-offs That Drive Model Selection

    Hands-on testing of several mainstream Chinese LLMs and coding frameworks reveals how reasoning depth, cost structures, and framework compatibility affect Agent performance. This article covers a reproducible evaluation methodology, observed capability gaps in compound instruction handling and cross-task transfer, token consumption patterns, and a practical selection framework based on task complexity, call frequency, and budget.

    Why Revisit Agent Model Selection Now

    Over the past six months, LLM-based Agent development has moved from experiments into production. A typical Agent task involves four stages: intent parsing, tool invocation, result verification, and error recovery. In practice, developers frequently run into one problem: models are updated faster than engineering frameworks can adapt.

    While testing combinations of mainstream models and coding frameworks in a recent project, the author found that native model-plus-native-framework pairings generally performed best. However, domestic model versions are updated on a weekly basis. Sticking with native frameworks means migrating toolchains, reconfiguring APIs, and adjusting prompt templates with every model upgrade — an expensive maintenance burden.

    For this reason, the author adopted an architecture of “a general-purpose coding framework with multi-model API adapters.” This sacrifices some performance headroom in exchange for the flexibility to switch models. All observations below are based on this architecture.

    Designing a Reproducible Agent Evaluation

    Comparative benchmarks are only meaningful when conditions can be reproduced. Before publishing conclusions or making selection decisions, record the following:

    • Test task set: the number of tasks and the difficulty distribution across task types.
    • Repeat count: how many times each task runs. Report success rates, not single-run results.
    • Scoring criteria: what counts as success, who judges it, and whether the criteria were fixed before testing.
    • Version information: model version numbers, framework version, API endpoints called, and test dates.
    • Runtime environment: network conditions, concurrency levels, whether rate limiting was encountered, and the retry strategy used.
    • Parameter settings: sampling parameters. Lowering temperature reduces output randomness but does not guarantee identical outputs. For more stable results, pin the model version, run multiple trials, and analyze the result distribution.

    Testing covered four dimensions:

    1. Instruction understanding: provide task descriptions with multiple nested conditions and check whether the model decomposes steps accurately.
    2. Plan generation: for open-ended problems — e.g., “analyze the sentiment distribution of a text dataset and generate a visualization report” — evaluate whether the model’s decomposition is reasonable.
    3. Self-correction: when unexpected inputs or errors occur, can the model locate the problem and adjust its strategy?
    4. Cost and efficiency: API fees, token consumption rate, and per-task response latency.

    Test Observations: Gaps Concentrate in Compound Instructions and Transferability

    Instruction Understanding: Compound Instructions Are the Main Divider

    Differences between the three models were pronounced in instruction understanding tests. The strongest performer, model K3, could infer necessary processing steps that were not explicitly stated in the instructions, and proactively asked for clarification when parameters were missing. The other two models each had distinct weaknesses:

    • Model A (an open-source model with hundreds of billions of parameters): code generation was acceptable, but with sequential logic like “process A first, then check B, finally aggregate C,” it often scrambled the order, feeding incorrect inputs to downstream tools. Static testing rarely catches this — it only surfaces in end-to-end Agent pipelines.
    • Model B (a balanced commercial model): solid across dimensions, but weak in transfer. When the same task was rephrased, it treated it as a new problem rather than recognizing that the underlying logic was identical. The Agent consequently burned significant tokens re-reasoning from scratch.

    K3 also performed well in self-correction: it detected anomalies in test data and adjusted its analysis strategy. The author believes this capability reduces the need for manual intervention in long-chain tasks.

    Cost: Billing Structure Matters More Than Unit Price

    The numbers below come from the author’s personal testing. Billing structures differ across vendors, so treat them as directional references only — actual prices vary with plans and time.

    • Model A offers a coding subscription plan, but quota is hard to obtain and peak-hour queue times are unpredictable. Once the plan’s allowance is exhausted, pay-as-you-go rates can reach several times the standard price.
    • Model B bills by token only. The author measured a mid-complexity task with roughly 10 minutes of interaction at about ¥1. A full Agent lifecycle test (including environment setup, multiple iterations, and error recovery) ran 5 hours and cost more than ¥30.
    • K3 was accessed through a cloud vendor’s Agent plan that bundles usage rights to several mainstream models. Completing the same tasks cost about one-third of Model B. However, its token consumption is fast: an intensive 90-minute test consumed roughly 2 million tokens of quota.

    Token consumption rate does not necessarily correlate with model parameter count. It is also affected by reasoning mode, output length, context caching, and batching. Looking at token unit price or per-request consumption alone therefore cannot accurately determine cost. A more practical metric is total cost per completed task, including failed retries, multi-turn corrections, and repeated context occupancy.

    When comparing costs, record unit prices, input/output token counts, plan allowances, queueing and rate-limiting behavior, and the number of task executions.

    Deeper Analysis: Trading Reasoning Depth Against Execution Efficiency

    Why Reasoning-Heavy Models Consume More Tokens

    Output logs revealed two reasoning patterns. Competing models favor “fast think, fast answer”: first-token latency is low, but errors often require multiple follow-up prompts to fix. K3 favors “slow think, slow answer”: after receiving a task, it runs a longer internal deliberation, produces a chain of thought with multi-step verification, then delivers the result in one pass.

    For complex tasks, the latter reduces interaction rounds and limits repeated context-window occupancy. Its downside is equally clear: for simple tasks like “extract the dates from this text,” the reasoning process is redundant, adding latency and token consumption. This conclusion comes from the author’s log observations and has not yet been validated with large-sample statistics.

    Framework Compatibility: An Underrated Variable

    The general-purpose coding framework the author tested has a reasonably designed tool-invocation protocol and supports multiple models. But in non-native combinations, the following issues still appeared:

    • Structured parsing of tool return values occasionally failed, requiring an extra format-conversion layer.
    • The model’s system prompt could conflict with the framework’s default prompt, causing role definition drift.
    • Under concurrent scheduling, the framework’s handling of model API rate limits was not smart enough, easily triggering throttling.

    Before going live, add these safeguards:

    • Set a timeout for every model call, with a fallback or retry flow on timeout.
    • Use exponential backoff for rate-limit errors, with a maximum retry count.
    • Never log API keys or sensitive user data; sanitize tool return content before writing it to logs.
    • Set a token cap per task to prevent runaway costs from looped calls.

    When using such “grafted” setups, reserve at least a week of debugging time to tune prompt templates and framework parameters.

    Selection Advice: Weigh Three Dimensions

    Model selection should be driven primarily by task complexity, call frequency, and budget. Compliance requirements and fallback plans should also factor in.

    Scenario Recommended approach
    High task complexity, long-chain workflows Prioritize reasoning depth and self-correction; a reasoning-heavy model reduces manual intervention
    High call frequency with simple tasks Prioritize response latency and cost; avoid paying for redundant reasoning
    Tight budget, experimental stage Choose plans with bundled quotas, track total cost per task rather than unit price

    Three trade-offs ultimately drive the decision:

    1. Reasoning depth vs. latency: reasoning-heavy models win on complex tasks but are wasteful for simple ones.
    2. Unit price vs. total cost: billing structure, quota, and retry overhead matter more than headline token prices.
    3. Native pairings vs. flexible adapters: native frameworks offer higher ceilings, while a general framework with multi-model adapters trades some performance for maintainability and switching flexibility.

    Before committing to a choice, run a reproducible evaluation under your own conditions — the capability gaps that matter most tend to appear only in real end-to-end Agent workflows.