Evaluating Chinese LLMs: Model Selection Criteria and Engineering Best Practices

    A practical framework for evaluating Chinese large language models based on effective context length, code generation, deployment efficiency, and multimodal capabilities. This guide explains how to build reproducible benchmarks, measure quality, latency, and cost, map business scenarios to model capabilities, and design routing, observability, and governance strategies for production use.

    1. Why “Which Model Is Stronger?” Is the Wrong Question

    Discussions about large language model capabilities often take the form of a comparison table: one vendor highlights context length, another emphasizes code generation, while others focus on model size, inference cost, or multimodal capabilities. The problem is that these figures are often not directly comparable.

    Context windows may be measured in tokens, Chinese characters, or general characters, and the conversion ratios are not consistent. Code capability may be reported as function-level completion accuracy by one provider and end-to-end task pass rates by another. The evaluation datasets and difficulty levels may also differ. “Lightweight” is even more ambiguous: it might refer to a small parameter count, low GPU memory usage after quantization, or high concurrency.

    Putting metrics with different definitions into one table replaces engineering decision-making with marketing language.

    This article does not compare the strengths or weaknesses of specific products. Instead, it addresses a more general and durable question: How can a business requirement be translated into measurable model capabilities, and how can a reproducible process be used for model selection and validation? This methodology applies to any model service, including services accessed through cloud platforms such as Baidu AI Cloud.

    2. Breaking Model Capability into Four Dimensions

    2.1 Long Context: Maximum Context Is Not Effective Context

    A model’s advertised context window is a structural limit. It does not guarantee that the model can use information reliably at that length. Three engineering constraints are especially important:

    • Attention computation grows approximately quadratically with sequence length. Very long contexts often lead to higher time-to-first-token (TTFT) latency and lower throughput.
    • KV cache usage grows approximately linearly with context length. This directly affects the number of concurrent requests a single GPU can support and is a major driver of cost in long-context workloads.
    • Information position matters. Models generally retrieve information more reliably from the beginning and end of a prompt, while details in the middle are easier to overlook. As more retrieved passages are added, interference between them can further reduce accuracy.

    The practical conclusion is that long-context applications must measure effective context length, not just the advertised maximum. A recommended method is the needle-in-a-haystack test: insert several independent “needles” into long filler text at different relative positions, ask for the inserted information, and measure how recall changes with position. Then evaluate aggregation tasks that require reasoning across multiple passages, such as “Summarize all budget-related clauses and identify conflicts.” These tasks more closely resemble real business workloads.

    1. # Needle retrieval test: measure effective context length and position robustness
    2. POSITIONS = [0.02, 0.2, 0.5, 0.8, 0.98] # Relative needle positions in the context
    3. def build_prompt(filler_units, needle, position):
    4. idx = int(len(filler_units) * position)
    5. units = list(filler_units)
    6. units.insert(idx, f"[Record] The verification code is {needle}")
    7. body = "\n".join(units)
    8. return body + "\n\nQuestion: What is the verification code above? Answer with the code only."
    9. def eval_effective_context(client, max_units=32000, needle="7F3A-9K"):
    10. rows, filler = [], [f"Line {i}: This is irrelevant background filler text." for i in range(max_units)]
    11. for n in (2000, 8000, 16000, 32000):
    12. for p in POSITIONS:
    13. prompt = build_prompt(filler[:n], needle, p)
    14. out = client.chat(prompt) # Adapter layer hides API differences
    15. rows.append({"context_units": n, "position": p,
    16. "hit": needle in out}) # A match counts as successful retrieval
    17. return rows # Calculate hit rates to build the model's effective-context curve

    2.2 Code Capability: From Completion to Repository-Level Understanding

    Code capability is not a single metric. It is a difficulty ladder that becomes increasingly close to real software development challenges:

    1. Single-line or snippet completion: Measures syntax correctness and continuation quality. Latency and acceptance rate are the primary metrics.
    2. Function-level generation: The model receives a signature and documentation and generates an implementation. Unit-test pass rate can provide an objective score.
    3. Cross-file understanding and refactoring: This requires retrieval, dependency analysis, and interface consistency. It evaluates context management rather than generation alone.
    4. Bug localization and repair: Given an error log and a minimal reproduction, the model must identify the root cause and produce a compilable patch.

    Evaluation should follow three rules:

    • Keep test data separate from business code to prevent inflated results caused by benchmark samples appearing in training data.
    • Prioritize objective automated metrics, including compilation success, unit-test pass rate, and the number of new static-analysis warnings.
    • Track human acceptance rate to determine whether developers are willing to merge generated code directly.

    Security should also be measured separately. Check whether generated code introduces injection vulnerabilities, hardcoded credentials, unsafe deserialization, or similar patterns. In CI pipelines, these checks may be more important as release gates than functional correctness alone.

    2.3 Lightweight Deployment: Translate “Light” into Billable Engineering Metrics

    In engineering terms, “lightweight” should be divided into two independent questions: Does a single request consume fewer resources? And Is the cost per unit of throughput lower?

    Key measurable metrics include:

    Metric Meaning Business impact
    TTFT Time from request submission to the first token Determines interactive responsiveness, especially for streaming conversations
    TPOT / per-token latency Time required for each token after the first token Determines the smoothness of long streaming responses
    Throughput (tokens/s) Overall processing capacity of a service instance Influences cluster size and billing
    Memory usage and concurrency Number of requests supported by one instance Determines autoscaling strategy and cost curve
    Quality loss after quantization Capability reduction caused by compression Determines whether a lower-cost configuration is suitable for production

    Common optimization techniques include weight quantization, KV-cache quantization, continuous batching, paged memory management, and speculative decoding. These techniques can improve cost and latency substantially, but they must be accompanied by quality regression testing. Rerun the evaluation set with the compressed configuration, measure the decline in key task metrics, and switch configurations only if the reduction is acceptable for the business.

    A simplified cost estimate is:

    1. Monthly cost ≈ Monthly request count × (Average input length × Input price
    2. + Average output length × Output price)
    3. × (1 - Cache hit rate) + Fixed inference resource cost

    Average output length is one of the variables most likely to be underestimated. Limiting response length can often reduce costs more directly than switching models.

    2.4 Multimodal Capability: Separate Understanding, Recognition, and Generation

    Multimodal capability is also not a single function. It includes at least four different task categories:

    • Image understanding and visual question answering: Description, reasoning, and image-text relationship analysis.
    • Text recognition and layout parsing: Accuracy on small text, skewed images, low-resolution inputs, and complex tables.
    • Chart and document structuring: Whether the model can produce stable structured output and preserve table row and column alignment.
    • Video and temporal understanding: Locating events in long sequences and aligning them with timestamps.

    A unique challenge in multimodal evaluation is structured-output stability. Across repeated calls with the same input, fields may be missing or the output format may drift. Enforce an output format such as a JSON Schema, and implement tolerant parsing and retries at the application layer instead of assuming that every response will be immediately parseable.

    3. Building a Reproducible Evaluation System

    Model selection should not depend on demonstration videos. It should rely on an evaluation pipeline that your team controls.

    1. Build the evaluation set

    Sample and stratify real business logs to cover frequent requests, long-tail cases, and edge cases, including extremely long inputs, ambiguous questions, and out-of-scope requests that should be refused. Create human-labeled gold answers. Keep an independent test set that is never used during tuning to prevent overfitting.

    2. Create a unified adapter layer

    Wrap all candidate models behind the same interface. Only then can latency, token usage, and output structure be compared consistently.

    1. import time
    2. from dataclasses import dataclass, asdict
    3. @dataclass
    4. class CallMetrics:
    5. ttft_ms: float | None = None
    6. total_ms: float = 0.0
    7. prompt_tokens: int = 0
    8. completion_tokens: int = 0
    9. def chat_with_metrics(client, messages):
    10. """Unified adapter: hides service differences and collects streaming metrics."""
    11. m, t0, first, chunks = CallMetrics(), time.perf_counter(), None, []
    12. for delta in client.stream_chat(messages):
    13. if first is None:
    14. first = time.perf_counter()
    15. m.ttft_ms = round((first - t0) * 1000, 1)
    16. chunks.append(delta)
    17. m.total_ms = round((time.perf_counter() - t0) * 1000, 1)
    18. m.completion_tokens = len("".join(chunks)) # Use actual usage data in production
    19. return "".join(chunks), asdict(m)
    20. def score_case(pred, gold, case_type):
    21. if case_type == "objective": # Structured or numeric tasks: exact or rule-based match
    22. return float(pred.strip() == gold.strip())
    23. return judge_pairwise(pred, gold) # Pairwise comparison plus calibrated human sampling

    3. Score and run regression tests

    Use automated grading for objective tasks. For subjective tasks, use pairwise comparison—asking which of two answers is better for the same prompt—to reduce drift in absolute scoring. Run the full regression suite after every model or prompt change. Write metrics to a logging and monitoring system, and block releases when key metrics decline.

    4. Route requests by difficulty

    Routing requests with different difficulty levels to models of different sizes is one of the most effective ways to optimize both user experience and cost. Rules or a lightweight classifier can perform initial classification at the request entry point.

    4. Mapping Business Scenarios to Model Capabilities

    Business scenario Key capabilities Primary metrics Common pitfalls
    Intelligent customer service and knowledge Q&A Instruction following, long context, or retrieval augmentation Answer accuracy, correct refusal rate Adding more context can create interference
    Coding assistants Code generation, repository-level context Unit-test pass rate, acceptance rate Evaluating only snippets hides cross-file weaknesses
    Document parsing and ingestion Multimodal recognition, structured output Table reconstruction accuracy, field completeness Ignoring skewed images, low resolution, and complex layouts
    Content generation Long-form coherence, style control Human ratings, rework rate Measuring fluency without checking factual consistency
    Edge or high-concurrency applications Lightweight deployment, low TTFT Throughput, unit cost Skipping quality regression after quantization

    5. Four Engineering Decisions

    Long context or retrieval augmentation?

    This is not an either-or decision. Long context is simple to implement and can preserve global semantics, but costs increase with context length and information in the middle of the prompt may be lost. Retrieval augmentation offers more predictable costs, supports immediate knowledge updates, and makes permission isolation easier, but depends on retrieval quality.

    A mature approach is usually a hybrid design: use retrieval to narrow the scope, rerank the candidate passages, and then provide a concise context to the model. This balances cost and accuracy.

    Cost and capacity planning

    Make output-length limits, concurrency limits, timeouts, and retry counts configurable. Enable result caching for highly repetitive requests. Deploy batch offline workloads and online interactive workloads in separate resource pools to prevent them from competing for capacity.

    Observability and governance

    Record request metadata and metrics across the full pipeline. Input and output content should be de-identified, and sensitive fields should not be stored in plaintext. Maintain separate dashboards for quality, latency, and cost. Release model-version changes through staged rollouts and A/B comparisons.

    Architectural decoupling

    Encapsulate model calls behind an independent adapter layer and a unified gateway so that business code does not depend on a specific API format. Platform capabilities such as object storage, message queues, container platforms, logging, monitoring, and alerting can provide the storage, asynchronous decoupling, deployment, and operational foundation.

    This makes it easier to switch between models, route requests by scenario, and implement graceful degradation.

    6. Model Selection Checklist

    1. Did you evaluate using your own business samples rather than vendor-provided examples?
    2. For long-context workloads, did you measure the effective-context curve and retrieval accuracy at middle positions?
    3. For coding workloads, did you test cross-file understanding and bug repair, and add security gates to CI?
    4. Did you translate “lightweight” into four measurable metrics: TTFT, throughput, memory usage, and unit cost?
    5. Did quantized or reduced-capacity configurations pass quality regression testing?
    6. Do multimodal tasks have validation and retry mechanisms for structured-output stability?
    7. Do you have tiered routing and fallback capabilities, along with continuous quality, latency, and cost dashboards?
    8. Is the architecture replaceable and compatible with staged rollouts through an adapter layer?

    The reliable way to turn model capability into business value is to replace the comparison of promotional specifications with a reproducible evaluation process and a realistic cost model.