How to Choose a Domestic Coding LLM: Evaluation Criteria and a Practical Testing Method

Start with the conclusion: there is no “strongest” model, only the “most suitable” one.

If you are choosing a domestic coding LLM for daily development, do not ask “which one is the strongest.” Ask “which one performs best across my actual task distribution.” Coding tasks are highly heterogeneous: real-time IDE completion, error diagnosis, unit test generation, and cross-file refactoring demand very different capabilities. The same model may respond quickly in lightweight Q&A but require repeated rework in complex refactoring.

There is another fact that reputation often hides: fast does not mean good. A model with very low first-token latency can still be slow in effective speed if simple requests need two rounds and complex requests need many rounds of back-and-forth. Every additional rework round means rewriting the prompt, waiting again, and manually checking again, which multiplies total time and total cost. By contrast, a slightly slower model with a high first-pass success rate often provides a smoother end-to-end experience.

Therefore, the recommended selection path is: define evaluation dimensions → build a private evaluation set → run quantitative tests → assign models by scenario. The rest of this article explains each step.

1. Six dimensions you must evaluate

1.1 Response latency: time to first token and output speed

• Time to first token (TTFT) determines how responsive the model feels. Output speed determines how long users wait for long generations. Evaluate them separately.
• As a rough reference only, simple Q&A tasks usually have TTFT around 1 second, code generation around 2–3 seconds, and long-form writing may take 5–10 seconds overall because of the large output volume. Test this yourself rather than relying on these numbers.
• Latency is strongly affected by network quality and time of day. Fluctuations during peak hours are normal. Always sample latency across multiple time periods instead of drawing conclusions from a single test.

1.2 Generation quality and rework rate: the largest hidden cost

• The key metrics are first-pass success rate and average rework rounds. You can evaluate quality in layers: syntactically correct → logically correct → architecturally reasonable → compliant with team conventions. The higher the layer at which an error occurs, the more expensive the rework.
• Total cost can be approximated as: token price × tokens per round × average number of rounds + manual communication and review time. A model with a low unit price but frequent rework may end up costing more overall.
• Record “rework rounds” as a metric with the same weight as latency, rather than focusing only on how fast the model generates.

1.3 Context length and codebase understanding

• Distinguish between nominal context and effective context. Many models show noticeable recall degradation for information near the beginning of the window as they approach the context limit. This matters especially for cross-file tasks.
• For repository-level Q&A and large-scale refactoring, in addition to window length, check whether the model can work reliably with common engineering techniques such as retrieval and workspace pruning.

1.4 Instruction following and tool calling

• In intelligent coding assistants and Agent workflows, the model must consistently output agreed formats, such as unified diffs or structured JSON, and correctly initiate tool calls. A single format drift can break the pipeline.
• Design specific tasks that require output in a specified format and measure format compliance rate.

1.5 Cost and billing model

• Common models include per-token billing, subscriptions, and resource packages. Also check whether prompt caching, batch calls, and other standard cost-reduction mechanisms are available.
• Looking only at unit price is misleading. The correct approach is to measure how much it costs to complete one standard task, then compare at the task level.

1.6 Stability and rate limiting

• Latency fluctuations during peak hours, concurrency limits, rate limits, and error rates all directly affect whether a model can be integrated into a team pipeline.
• From an engineering perspective, design retries, timeouts, and fallback strategies by default, such as switching to a backup model when the primary model fails, instead of assuming the service will always run smoothly.

2. How to run a credible evaluation

Reputation can mislead. Data will not. It is worth spending one or two days setting up a lightweight evaluation.

2.1 Build a private evaluation set

Extract 20–50 tasks from your real work and stratify them by type:

• Bug fixing: provide code with an edge-case problem and ask the model to fix it and explain the cause.
• Feature implementation: provide a clear requirement and ask for a runnable function.
• Unit test generation: ask the model to generate tests covering boundary conditions for a given function.
• Code explanation and review: ask the model to identify code smells and suggest improvements.
• Cross-file refactoring: provide context from multiple files and ask for consistent changes.

Define clear pass criteria for each task type in advance so that evaluation does not become ad hoc.

2.2 Define quantitative metrics

• Time to first token P50 / P95, recorded at different times of day.
• First-pass success rate = number of tasks that meet the standard in a single interaction ÷ total number of tasks.
• Average rework rounds.
• Task-level total cost in tokens × rounds.
• Format compliance rate for Agent scenarios.

2.3 Use scripts for automatic measurement instead of manual notes

The example below needs adaptation according to the API documentation of the platform you use. For accurate TTFT, switch to streaming output and record the arrival time of the first chunk.

import os, time, requests, statistics
EVAL_SET = [
    {"task": "fix_bug",   "prompt": "The following function throws an index error when the input is an empty list. Fix it and explain the cause: ..."},
    {"task": "gen_tests", "prompt": "Generate unit tests covering boundary conditions for the utility function below: ..."},
    # ... aim for 20–50 real tasks
]
def call_llm(prompt: str) -> float:
    start = time.perf_counter()
    resp = requests.post(
        os.environ["LLM_ENDPOINT"],  # Chat endpoint URL provided by the platform
        headers={"Authorization": f"Bearer {os.environ['LLM_API_KEY']}"},
        json={"model": os.environ["LLM_MODEL"],
              "messages": [{"role": "user", "content": prompt}],
              "temperature": 0.2},
        timeout=120,
    )
    resp.raise_for_status()
    return time.perf_counter() - start  # Simplified: whole-request time approximates TTFT
latencies = [call_llm(t["prompt"]) for t in EVAL_SET]
print("P50:", statistics.median(latencies),
      "P95:", sorted(latencies)[int(len(latencies) * 0.95)])

2.4 Control variables and blind evaluation

• Use the same prompt, the same temperature, and the same coding assistant configuration for each candidate model.
• Sample each task at least 2–3 times and take the median to reduce randomness.
• Ask colleagues who are not involved in the selection process to blindly score code quality, avoiding bias from preconceived preferences for a particular vendor.

3. Route models by scenario instead of using one model for everything

After testing, most teams conclude not “use only one model,” but “configure models by scenario.”

• IDE real-time completion: most latency-sensitive. Choose the fastest tier, even if capability is slightly lower.
• Daily Q&A and explanation: a balanced model is enough. Focus on first-pass success rate.
• Complex refactoring and architecture tasks: prioritize quality and accept higher latency and cost. The cost of one rework in these tasks is far greater than waiting a few extra seconds.
• Batch tasks such as bulk unit test generation, comments, or documentation: prioritize cost and use batching and caching to reduce expenses.
• Agent automation pipelines: prioritize tool-calling stability and format compliance.

4. Three suggestions for engineering integration

4.1 Abstract a thin adapter layer

No matter which provider you choose, for example major cloud vendors such as Baidu AI Cloud or direct model APIs, build a thin adaptation layer between your business logic and the model. Unify chat, streaming, and retry interfaces so that A/B switching and gradual rollout become simple configuration changes:

class LLMClient:
    """Unified interface: complete(messages) -> str; concrete implementations connect to different model services"""
    def complete(self, messages: list[dict]) -> str:
        raise NotImplementedError

4.2 Pay attention to prompt and context engineering

For the same model, providing clear constraints and pruning irrelevant context can make a difference comparable to switching models. Improve prompts first, then consider changing models.

4.3 Design for instability

Peak-hour latency jitter and occasional rate limiting are normal. The client should have timeout, retry, and backup-model fallback paths, and usage and error rates should be connected to monitoring and alerting for continuous review.

5. A decision checklist you can execute directly

• Clarify your primary scenarios: completion, Q&A, refactoring, batch, or Agent.
• Select 2–3 candidate models and integrate them through a unified interface.
• Run a quantitative evaluation with 20–50 real tasks, recording latency, pass rate, rework rounds, and task-level cost.
• Retest latency at different times of day to confirm peak-hour behavior.
• Assign models by scenario and prepare retry and fallback paths for production.
• Regress against the same evaluation set monthly. Model iterations are fast, so conclusions need to stay fresh.

Conclusion

“I heard model X is the strongest” is the least reliable source of information in model selection. Translate subjective reputation into four numbers: latency, first-pass success rate, rework rounds, and task-level cost. Then spend one or two days running your own test. The conclusion you get will be far more relevant to your actual workflow than any review article. After all, your codebase is the best judge of whether a model is good.