General-Purpose LLM Evaluation and Selection Guide: Capability, Stability, and Cost

    This article provides a practical framework for developers and enterprises evaluating general-purpose LLMs beyond public leaderboards. It covers scenario-based metrics, correctness and robustness, long-context retention, tool calling, structured output, code engineering, creative writing, multimodal and ecosystem fit, API stability, version drift, cost and billing, and routing with fallback strategies. Readers will learn how to build reproducible evaluations and select models based on task distribution, engineering constraints, and total cost of ownership.

    Model versions, service policies, and pricing change over time. This is a basic premise when evaluating general-purpose LLMs. For developers and enterprises, “overall performance” is not a single score. It is a combination of capability balance, engineering usability, cost control, and service stability. The current general-purpose LLM market shows clear capability divergence. Some models are broadly balanced and fit many use cases, but their expression tends to be conservative. Others have advantages in creative writing or complex code tasks, but may show volatility in toolchains, billing policies, or version stability. Therefore, model selection should not stop at leaderboard thinking about “who is the strongest.” It should return to specific scenarios, reproducible evaluations, and engineering constraints.

    1. Why “Overall Performance” Cannot Rely Only on Leaderboards

    Public leaderboards usually cover standardized tasks such as knowledge Q&A, mathematical reasoning, and code snippet generation. They are suitable for initial screening, but they rarely reflect long-chain tasks in real production environments. In practice, a single request may involve long-context reading, tool calls, structured output, multi-turn correction, and cost budget control. A model that performs well in single-turn Q&A does not necessarily complete code reading, localization, modification, and verification reliably in complex engineering tasks. More importantly, model versions, rate-limit policies, subscription coverage, and usage-based billing rules may all change. Yesterday’s conclusion may not apply today.

    Evaluation should therefore be built on a three-layer framework: scenario metrics, stability metrics, and cost metrics. Scenario metrics focus on task completion rate, correctness, maintainability, and manual rework rate. Stability metrics focus on output variation for the same prompt across different times and versions. Cost metrics include not only invocation fees but also retries, manual review, toolchain migration, and lock-in risk.

    2. Core Evaluation Dimensions: From Single-Point Capability to Engineering Usability

    1. Correctness and robustness. Correctness is not only about whether an answer is right or wrong. It also includes handling of boundary conditions, responses to abnormal inputs, and compliance rate for structured output. Robustness requires the model to remain usable when prompts change slightly or context noise increases.

    2. Long context and information retention. Long-context capability should not be judged only by window length. It should also be evaluated by key-information recall, cross-section reference consistency, and factual retention in long-document summarization. For scenarios such as codebase analysis, contract review, and log troubleshooting, information retention matters more than how many tokens can be inserted.

    3. Tool calling and structured output. Production systems often require models to return results in a fixed structure and to call general tools such as retrieval, calculation, and database queries. Evaluate tool-selection accuracy, parameter validity, failure retry strategy, and output parseability rate.

    4. Code engineering capability. Code generation is only a starting point. More important are code reading, dependency understanding, minimal modification, test completion, and regression verification. In complex engineering tasks, whether a model is willing to read enough context and whether it frequently makes ineffective changes are important observations.

    5. Creative writing and expression quality. Creative writing evaluation is highly subjective, but relative comparison can be made through style consistency, structural completeness, detail density, and the proportion of template-like phrasing. Some balanced models are stable in general Q&A but easily turn creative content into flat explanatory text. Such differences need to be identified through blind evaluation.

    6. Multimodal and ecosystem capability. For multimodality, focus on image-text understanding, document parsing, chart reading, and cross-modal consistency. Ecosystem capability includes plugins, tool integration, SDK maturity, documentation quality, and community practices. However, ecosystem scale should not be directly equated with task effectiveness.

    7. API stability and version drift. API availability, latency distribution, error-code consistency, and version-change notification mechanisms directly affect online systems. Version drift is an easily overlooked risk. After a service policy adjustment, the same model may show changes in output style, fluctuations in complex-task performance, or changes in tool-calling behavior.

    3. Capability Profiles: Balanced and Specialized Models Coexist

    From an engineering perspective, general-purpose LLMs can be broadly divided into balanced and specialized types. Balanced models are more comprehensive in Q&A, multimodality, API stability, and ecosystem fit. They are suitable as a default entry point or fallback model. Their drawbacks are that expression may sound official or boilerplate, and they may offer fewer surprises in creative writing and highly customized tasks. Specialized models may be stronger in code, long text, or creative expression, but often have clear weaknesses, such as insufficient stability in complex tasks, changing billing policies, limited toolchain coverage, or fluctuating version behavior.

    This is not a question of “who replaces whom,” but of “how tasks are allocated.” For most teams, the benefit of using a single model for all scenarios is declining. A combination of “default model + specialized model + fallback model” is more consistent with engineering reality. The default model handles general Q&A and low-risk tasks. The specialized model handles high-value scenarios. The fallback model takes over when there is a timeout, failure, or quality shortfall.

    4. Quality Fluctuation and How to Identify “Getting Dumber”

    Community feedback often includes reports that a model has “recently become dumber” or “lazier.” Such phenomena may come from model version updates, system policy adjustments, rate limits, or billing changes. They may also be subjective impressions caused by changes in prompts and task distribution. Engineering teams should not rely only on perception. They should establish version-drift monitoring: fix a representative set of tasks and periodically record task completion rate, average edit count, invalid modification ratio, tool-call failure rate, and manual rework rate. If complex engineering tasks show fewer context reads, wider modification scope, or lower test pass rates, a review should be triggered.

    It should be noted that a service provider may attribute abnormal behavior to “system misjudgment” or policy adjustment, but users should rely on reproducible data. For critical business processes, it is advisable to lock the model version or keep switchable interfaces, so that core workflows are not bound to the behavior of a single version.

    5. Creative Writing and Code Engineering: Selection Logic for Two High-Frequency Scenarios

    For creative writing scenarios, focus on style controllability, detail richness, and instruction-following for revisions. If a model tends to output bland explanatory text, few-shot examples, style constraints, and segmented generation can improve results, but fundamentally a candidate model with more flexible expression should still be selected. For code engineering scenarios, focus on repository-level understanding, dependency analysis, minimal diff, test generation, and error-fixing capability. For complex engineering tasks, consider adding a “read before modify” constraint and requiring change notes and verification steps.

    6. Cost and Billing: Do Not Look Only at Unit Price

    Cost evaluation should cover subscription plans, usage-based billing, third-party tool coverage, and overage policies. Subscriptions suit high-frequency interaction and fixed teams. Usage-based billing suits fluctuating workloads and toolchain integration. If a subscription no longer covers certain general tool calls, the actual cost may shift from a fixed monthly fee to usage-based add-ons. Enterprises should calculate total cost of ownership, including invocation fees, retry fees, manual review fees, migration costs, and rework costs caused by version fluctuations.

    7. Implementation: Reproducible Evaluation and Routing with Fallback

    A lightweight evaluation pipeline is recommended: maintain a task set, define metrics, replay periodically, and record versions and costs. At the infrastructure level, generic cloud services such as object storage, logging, and monitoring—for example, Baidu AI Cloud where available—can support evaluation data and result retention, but evaluation metrics and conclusions should remain decoupled from any specific vendor.

    Illustrative pseudocode:

    1. tasks = load_eval_tasks()
    2. for model in candidate_models:
    3. for task in tasks:
    4. result = call_model(model, task.prompt, budget=task.budget)
    5. score = evaluate(result, task.rubric)
    6. record(model.version, task.id, score, result.cost, result.latency)
    7. router(task):
    8. if task.type == "creative_writing":
    9. return fallback_chain(["specialized_candidate", "balanced_candidate"])
    10. if task.type == "complex_code":
    11. return fallback_chain(["code_candidate", "balanced_candidate"])
    12. return fallback_chain(["balanced_candidate", "low_cost_candidate"])

    Routing strategies should include quality thresholds, cost ceilings, timeout fallback, and manual takeover. For high-risk tasks, outputs must go through structured validation and human review.

    8. Conclusion

    No single general-purpose LLM currently wins in every scenario. Overall performance should be understood as “stable usability in target scenarios.” When selecting models, first clarify task distribution and risk levels, then use reproducible evaluations to verify correctness, stability, tool calling, and cost. Use specialized candidates for high-value scenarios such as creative writing and complex code, use balanced candidates for general Q&A, and retain fallback and version-locking mechanisms. Only by incorporating model evaluation into engineering governance can teams avoid being misled by short-term leaderboards and subjective fluctuations.