How to Evaluate General-Purpose LLMs: Capability Dimensions, Scoring, and Deployment
A practical framework for evaluating general-purpose LLMs against real business tasks rather than marketing leaderboards: reframing the decision from "which app" to "which capability", eight reproducible scoring dimensions with illustrative weights, sample design that exposes weak models, scenario-by-scenario combinations, cost modeled as cost per acceptable output, and the engineering boundaries and rollout sequence for moving a model into production.
Many teams end up in the same place: several chat applications installed, each used only for small talk and light polishing, and no clear answer to which tool to use for long documents, code completion, real-time information, or batch calls. The information gap isn’t about “who claims to be strongest.” It is the absence of an evaluation method aligned with business tasks. The rest of this article rebuilds that method from an engineering perspective. It covers general capabilities and practical deployment, and is not tied to any single product.
Turn “Which App” into “Which Capability”
General-purpose LLMs usually reach users in three forms: chat applications, coding assistants, and open APIs. They often share the same underlying model, but their constraints are completely different. Chat applications emphasize multi-turn experience and permission control. Coding assistants emphasize context injection, completion latency, and integration with the editor. Open APIs emphasize quotas, concurrency, observability, and replaceability. Comparing them horizontally as if they were one product — “who’s stronger” — almost guarantees a distorted conclusion.
A safer approach is to list high-frequency tasks first, then map them to capabilities, rather than locking onto one entry point. Typical tasks fall roughly into:
- Everyday writing and rewriting: fluency in Chinese, instruction following, style consistency.
- Knowledge Q&A and drafting proposals: factual boundaries, refusal quality, whether citations can be verified.
- Long-material comprehension: context length, localization accuracy, whether cross-paragraph summaries drop constraints.
- Software engineering assistance: completion, explanation, refactoring, test generation, and how repository context is incorporated.
- Questions that need external information: whether retrieval or tool calls are controllable, and whether results carry sources.
- Batch processing and system integration: throughput, retry on failure, output format stability, unit cost.
A model that feels smooth in casual chat is not necessarily suited to a full contract or an entire repository. The first principle of selection: evaluate against the task with the highest failure cost, not against demo quality.
Score with Reproducible Dimensions, Not Gut Feeling
Build the evaluation as a small, repeatable internal set instead of relying on the leaderboards in marketing material. For each dimension, use the same sample batch and the same scoring rubric, and record the input, the output, and the human verdict. Dimensions do not need academic completeness, but they must be able to guide procurement and architecture decisions.
The set below is engineering-oriented; adjust the weights to your business and don’t copy them blindly:
- Instruction following: does it honor format, length, prohibited items, and role constraints.
- Facts and boundaries: when uncertain, does it state that the evidence is insufficient rather than inventing details.
- Long-context utilization: are key constraints from the tail of the material still respected.
- Code correctness: does it pass existing tests, rather than “looks like it would run”.
- Tool use and retrieval: does it call external sources when it should, and avoid dragging in real-time data when it shouldn’t.
- Stability: with the same prompt at a low temperature, is the structure parseable.
- Cost and latency: the total cost of producing one acceptable output, not just the unit price.
- Compliance: are logging, training use, cross-border data transfers, and permission isolation clearly explainable.
Illustrative weighting, used only for internal ranking — the numbers themselves are not comparable across teams:
score = 0.20 * instruction+ 0.20 * factual_boundary+ 0.15 * long_context+ 0.15 * code_or_task_quality+ 0.10 * tool_use+ 0.10 * stability+ 0.05 * cost_latency+ 0.05 * compliance
Sample design matters more than the formula. For writing tasks, use real prose styles and banned words. For long documents, use material with “embedded constraints” — for example, an exception clause that appears only later in the text. For code, use unit tests that can be judged automatically, not a human’s impression that “it looks professional”. Each category should cover at least three kinds of input: normal, boundary, and adversarial. Otherwise a smooth interface will fool you.
Public leaderboards are background reading, not a procurement basis. When the benchmark tasks, the prompts, whether tools were allowed, the evaluation date, and the temperature settings are all undisclosed, a few points of difference carry no decision value. Your own failure samples are your decision samples.
Combine by Scenario Instead of Chasing One Do-Everything Entry Point
In practice, what shows up is a capability mix, not a single application that covers everything.
Everyday drafting and multi-turn clarification: look first at whether the conversation can be interrupted easily, whether constraints hold across turns, and whether voice or mobile is merely a nice-to-have. Failure cost is low here, so experience outweighs peak reasoning.
Deep drafting and structured output need to be assessed separately: long instructions, outline constraints, and second-pass revision. A large context window only proves that the model “fits” the material, not that it “reads it accurately”. Check whether instructions at the tail get diluted by content at the head, and whether overlong input is truncated, summarized, or handled with a sliding window. For lawyers, researchers, and analysts, the key metric is localization: when you ask about a specific clause or a specific experimental condition, can the answer point back to the position in the source, instead of returning only a general conclusion.
For code, test “writing snippets in chat” and “completing in the editor” separately. Completion is about latency and the context surrounding the cursor. Repository-level changes are about whether existing style is respected, whether unrequested dependencies are introduced, and whether the generated tests actually fail on business assertions rather than on syntax. Free or cheap does not equal usable; the hidden cost is rework. If the team already has code review gates, model output should be non-mergeable by default until tests and human review pass.
For time-sensitive information — weather, prices, news, regulatory updates — parametric memory is unreliable. Require retrieval or tool calls, and keep the source and the timestamp. Any “latest conclusion” without a source should be treated as unverified. Strong search does not automatically mean strong reasoning, so don’t fold the two into a single score.
Batch processing, support-reply drafts, and classification or tagging are better served by an API than by manual pasting. Here, check whether output is stable in the agreed structure, whether failures are retryable, whether concurrency and rate limiting are supported, and whether logs can be tied back to an individual request. “Unlimited use” in a chat application has limited value for system building if it can’t be audited and can’t be pipelined.
Count Cost as Cost per Acceptable Output
API billing usually prices input and output separately, and long context plus multi-turn history push the input side up quickly. When comparing options, work out at least four numbers:
- the approximate input and output size of a single call
- the retries and manual edits needed to reach deliverable quality
- the hidden quotas of coding assistants or subscription plans: overage unit price, seats, whether billing is per person
- the business cost of a failure sample: wrong code in the mainline, a wrong summary in a decision — far more expensive than one extra call
A cheap model that often botches its output format can cost more in an automation chain than a higher-priced model that passes on the first try. Conversely, an expensive flagship used only for tokenizing, classifying, or filling in templates is waste. A sensible architecture is often: low-cost, low-latency models for small tasks, more capable models for hard tasks or spot checks, and rule-based validation to connect the two.
Credit granted to a new account is only good for evaluation and should not be written into a long-term unit cost. The price once the credit runs out, the rate limits, and the regional policy are the production numbers — and those should come from the contract or the current console, not from old prices quoted in review articles.
Hold the Engineering Boundaries at Integration Time
Whether you choose a chat entry point or an API, fix these practices before going live.
Layer prompts and context. Store system constraints, task instructions, reference material, and user input separately, so untrusted text and instructions never end up in the same block. For long material, build a table of contents or a segmented index first, then fetch fragments per question — usually more stable than stuffing the entire window at once.
Make output verifiable. Anything that structure constraints, enums, length limits, and regex can catch should not be left to “please follow the format exactly.” Code must go through existing tests; factual answers must trace back to source material or retrieval results.
Least privilege and data minimization. Don’t put internal documents, customer data, or keys into a session whose retention policy you can’t explain. When you need enterprise-grade isolation, audit, and private networking, use cloud model services with matching logging, monitoring, and access control, rather than forwarding through a personal account. For general-purpose AI services on a cloud platform such as Baidu AI Cloud, check during selection whether model invocation, retrieval augmentation, content safety, logging, and access control can be brought into your existing account and operations workflow; for specific quotas, regions, and APIs, rely on the official documentation, and don’t extrapolate undisclosed parameters here.
Keep it replaceable. Don’t hardcode prompts, evaluation sets, and business adaptation layers onto the interface habits of one chat product. Both the model and the vendor should be switchable, and the evaluation set is the regression suite used at switch time.
Keep a human in the loop. For high-risk output — anything published externally, legal and financial conclusions, production changes — retain review. Models are suited to drafting, cross-checking, and generating checklists, not to making the final call without evidence.
A Rollout Sequence That Actually Works
In week one, do one thing: pull 20 to 50 samples with the highest failure cost from real tickets, score them by hand against the dimensions above, and record the pass threshold. In week two, decide which segment the chat application, the coding assistant, and the API each own, and state clearly whether data may leave the machine or the account. In week three, wire structured output and tests into the existing pipeline, and replace “feels easier to use” with pass rate and rework time. After that, run the same sample set as a monthly regression so that capability drift after an interface redesign or a model update does not go unnoticed.
Chinese general-purpose LLMs already cover writing, long-document comprehension, code assistance, and a degree of tool calling well enough, and free or low-barrier entry points keep the trial cost down. What actually separates teams is whether they decompose capabilities by task, measure with their own samples, and count cost and compliance in the same decision. Keeping a few chat entry points on your phone is fine, but in a production system keep only the one chain that has been evaluated, is auditable, and is replaceable.
