Evaluating and Selecting General-Purpose Large Language Models: A Six-Dimension Capability Framework

    This article provides a practical framework for evaluating and selecting general-purpose large language models. It breaks model capability into six dimensions—language quality, logical reasoning, code generation, tool calling, context handling, and multimodal perception—and then covers sparse Mixture-of-Experts architectures, tiered model routing, cost modeling, evaluation system design, production architecture, and common selection pitfalls.

    From Conversational Models to Models That Complete Work

    Over the past two years, the product form of large language models has shifted from “models that can chat” to “models that can do work.” Early models were mainly judged by whether their replies sounded natural and human-like. Today, the more important question is whether a model can complete a multi-step task end to end.

    This shift creates different technical requirements. Conversational ability focuses on single-turn semantic coherence, while task execution depends on planning, error correction, state retention, and verifiable results.

    A notable signal is that new general-purpose models are being promoted around combined capabilities rather than isolated features: logical reasoning + code generation + tool calling. This article breaks the broad idea of an “all-around” model into six dimensions and provides practical guidance for evaluation and integration.

    Capability Dimensions

    1. Language understanding and generation quality

    This is the foundational capability, covering long-form summarization, structured rewriting, and multi-turn instruction following. In engineering practice, what often differentiates models is not rhetorical quality but format stability: whether the model can reliably output valid JSON, follow length and hierarchy constraints, and avoid dropping constraints in long instructions. This directly determines whether the output can be consumed by downstream systems.

    2. Logical reasoning

    Logical reasoning includes multi-hop reasoning, numerical calculation, and constraint-satisfaction problems. A key evaluation consideration is that errors accumulate during long-chain reasoning. If each step is 95% correct, performance can degrade significantly after ten steps. In practice, it is often better to design interactions that break a large problem into smaller, independently verifiable steps rather than relying on a model to complete a long reasoning chain in one pass.

    3. Code generation

    Modern general-purpose models usually provide strong capabilities for code generation, completion, explanation, and refactoring. The key metric here is not whether the model can produce a demo, but how well it performs on patch-style tasks, such as making localized changes in an existing codebase, understanding cross-file dependencies, and generating code that passes tests. Using executable unit tests as a feedback signal is currently one of the most reliable ways to improve code-generation quality.

    4. Tool calling and task execution

    Tool calling is the core dividing line between conversation and productive work. The model needs to select tools based on the task, generate structured call parameters, parse returned results, retry or switch strategies when a call fails, and maintain state across multiple turns. In engineering practice, this is usually constrained by unified capability descriptions, such as tool name, purpose, and parameter schema, while an external orchestration layer performs execution and validation.

    5. Context capacity

    Long-context support is not only about having a larger context window. It creates two outcomes. First, long documents such as contracts, reports, and formal materials can be modeled in a single pass. Second, memory usage and latency costs increase. In practice, it is often necessary to control context size with techniques such as retrieval augmentation, which injects relevant passages ordered by relevance, and hierarchical summarization, which summarizes sections first and then aggregates the results.

    6. Multimodal perception

    Understanding images, tables, audio, and video is becoming a standard capability for general-purpose models. This is especially valuable in document-intensive scenarios, such as scanned documents, charts, and complex reports with intricate layouts, because it can reduce a large amount of manual annotation and structuring work.

    Why Sparse Mixture-of-Experts Architectures Can Balance Performance and Cost

    New-generation large models commonly adopt sparse Mixture-of-Experts architectures. The core idea is straightforward: the total number of model parameters is large to provide knowledge capacity, but only a small subset of experts is activated during each forward pass. As a result, the compute required per token is significantly lower than that of a dense model of the same nominal size.

    This design brings three engineering benefits:

    • Capacity and compute are decoupled. Knowledge capacity can scale with the number of experts, while inference compute is mainly related to the number of activated parameters.
    • It is throughput-friendly. Lower activation allows higher generation speed under the same hardware conditions.
    • Costs are more controllable. Lower compute per token makes high-frequency scenarios, such as batch document processing, more practical.

    The tradeoff is increased engineering complexity. Teams must handle load imbalance in expert routing, communication overhead for expert parallelism across devices, memory capacity pressure because the total weight size is still large, and accuracy tradeoffs during quantized deployment. This also explains why such models often require managed inference from a service provider rather than self-deployment on the business side.

    Flagship and Lightweight Model Tiers: Selection and Cascade Routing

    Vendors commonly provide multiple model tiers with different parameter sizes. This is not simply a choice between “better” and “worse”; it is a choice between different balances of performance and cost. A sound engineering approach is to route tasks by difficulty: send most simple requests, such as format rewriting, field extraction, and short question answering, to a lightweight tier, and escalate complex reasoning and multi-step execution requests to the flagship tier.

    1. class TaskRouter:
    2. """Illustrative: choose a model tier based on task features; thresholds must be calibrated with business evaluation data."""
    3. LIGHT_RULES = ("extract", "rewrite", "classify", "format", "short QA")
    4. HEAVY_RULES = ("reason", "plan", "debug", "refactor", "multi-step")
    5. def pick(self, task_text: str, tools_available: bool) -> str:
    6. heavy = any(k in task_text for k in self.HEAVY_RULES)
    7. light = any(k in task_text for k in self.LIGHT_RULES)
    8. if heavy or (tools_available and len(tools_available) > 5):
    9. return "flagship" # Complex planning and tool orchestration
    10. if light:
    11. return "light" # High-frequency, low-complexity tasks
    12. return "flagship"

    The cascade strategy can be further upgraded to “answer with the lightweight tier first, check confidence, and escalate when necessary.” In scenarios with historical feedback data, this approach can often keep most requests on low-cost tiers while keeping quality loss on complex requests within an acceptable range. One important caveat is that routing thresholds must be calibrated using your own business evaluation set rather than copied from someone else’s experience.

    Measuring Performance and Cost

    Statements such as “fast” and “low cost” are relative. In engineering practice, they must be translated into measurable metrics:

    • TTFT (time to first token): This affects perceived responsiveness and is strongly related to queuing and prefill length.
    • TPOT / generation speed (tokens per second): This determines the total time required for long-form generation.
    • Concurrency and throughput: These determine the overall cost of batch workloads.
    • Effective context: This must be validated against the document-length distribution of your business, not judged only by the advertised maximum context size.

    Costs should be modeled rather than estimated from unit price alone:

    1. Task cost = input tokens × input price + output tokens × output price
    2. Total cost = Σ task cost × request volume × (1 + retry amplification factor)

    The retry amplification factor is often ignored. If the failure rate of tool calls is high, actual consumption may be 1.3 to 2 times the theoretical value. In addition, retrieval augmentation can improve accuracy, but it also increases input tokens, so a quantitative tradeoff between accuracy gains and cost increases is required. For specific pricing and quotas, refer to the official billing page of the service platform you use.

    Building an Evaluation System: Do Not Treat Leaderboards as Conclusions

    Public benchmark suites for mathematics, coding, knowledge question answering, and other domains can serve as an initial screening reference, but they have clear limitations: test items may be covered by training data, their distribution may not match your business scenarios, and single-run results often lack confidence intervals.

    A three-layer evaluation system is recommended:

    1. General capability regression set: Hundreds to thousands of examples covering reasoning, code generation, and instruction following, used to prevent quality regressions during version upgrades.
    2. Business scenario set: Input-output pairs from real business cases, including typical difficult cases and edge cases, used as the main basis for model selection and tuning.
    3. Online metrics: Task completion rate, human acceptance rate, rework rate, and average number of turns, used to verify consistency between offline evaluation and online behavior.
    1. def evaluate(cases, model_client, judge=None):
    2. """Illustrative skeleton for a reproducible offline evaluation; judge may be a rule-based function or a model."""
    3. results = []
    4. for case in cases:
    5. out = model_client.chat_stream(
    6. messages=case["messages"],
    7. temperature=0.0, # Keep evaluation deterministic
    8. max_tokens=case["max_tokens"],
    9. )
    10. passed = check_schema(out, case["schema"]) if case["schema"] else judge(out, case["gold"])
    11. results.append({"id": case["id"], "passed": passed, "tokens": estimate_tokens(out)})
    12. total = len(results)
    13. pass_rate = sum(r["passed"] for r in results) / max(total, 1)
    14. return {"pass_rate": pass_rate, "detail": results}

    There are three key points. First, evaluation must fix random factors such as temperature and repeat runs to compute averages. Second, it must distinguish between two types of failure: format errors and content errors. Format errors can often be solved through prompt design and constrained decoding. Third, evaluation results should be stored for traceability so that version-to-version comparison curves can be formed.

    Implementation Notes for Document-Heavy Scenarios: Formal Documents and Report Writing

    Document-writing scenarios place combined demands on general-purpose models: the model must understand long materials, follow formatting rules, and avoid introducing facts beyond the provided sources. Reusable engineering patterns include:

    • Template constraints: Express document type, heading hierarchy, and required elements as structured templates, and let software validate the structure rather than relying on the model’s discretion.
    • Retrieval-augmented source material: Retrieve similar materials from knowledge bases and historical documents as references, which significantly reduces the probability of fabricated content.
    • Segmented generation with global consistency checks: First generate and confirm an outline, then expand each section, and finally check terminology and style consistency.
    • Human review loop: Feed review edits back into evaluation samples to form a data flywheel for continuous improvement.

    Access Architecture and Observability

    It is recommended to add an abstraction gateway between business applications and the model. This layer can uniformly handle routing, retries, fallbacks, logging, and metrics.

    1. # Illustrative: key responsibilities of the model access layer
    2. class ModelGateway:
    3. def call(self, task, messages, tools=None):
    4. tier = self.router.pick(task, tools)
    5. try:
    6. return self.client(tier).chat(messages=messages, tools=tools, stream=True)
    7. except RateLimitError:
    8. return self.retry_with_backoff(task, messages, tools)
    9. except UpstreamError:
    10. self.metrics.inc("fallback", {"tier": tier})
    11. return self.client("light").chat(messages=messages, tools=tools)
    12. finally:
    13. self.log.trace(task_id=task.id, tier=tier,
    14. prompt_tokens=..., completion_tokens=...)

    Key capabilities to build include streaming output to improve perceived responsiveness, timeout and retry backoff, degradation switching across model tiers, token and latency monitoring by business dimension, sensitive information masking, and data retention policies. If a cloud-hosted approach is used, model inference can often be combined with common capabilities such as object storage, retrieval services, message queues, logging services, and monitoring/alerting to build a complete pipeline from data ingestion to performance observation. For scenarios with strict data compliance requirements, the retention scope and access permissions of request logs must also be clearly defined.

    Common Misconceptions

    • Relying only on leaderboards instead of business datasets: A high leaderboard score does not necessarily mean good performance on your tasks. Scenario differences are often larger than differences between models.
    • Treating prompt engineering as a long-term solution: The marginal benefit of prompt tuning diminishes quickly. Retrieval augmentation, tool orchestration, and workflow redesign have higher ceilings.
    • Ignoring non-linear growth in inference costs: Output length, number of tool calls, and retry count can all multiply costs, so these factors must be modeled during budget planning.
    • Trying to solve everything with one call: Multi-stage pipelines, such as extraction → planning → generation → validation, are usually more stable and controllable than a single oversized prompt.

    Conclusion

    To judge whether a general-purpose large language model is truly “all-around,” it is better to rely on a verifiable framework than on marketing claims. Capability coverage should span language, reasoning, code, tool calling, context handling, and multimodality. Architecture understanding should include the capacity-compute decoupling provided by sparse expert designs. Engineering work should include tiered routing, cost modeling, evaluation systems, and observability.

    This approach is independent of any specific vendor implementation and can be transferred directly to future model upgrades or replacements. What needs to be accumulated over time is your evaluation set, orchestration layer, and data feedback loop, rather than the invocation method of any single model.