Large Language Model Selection: A Reproducible Framework for Evaluation, Cost, and Deployment

    This article argues that asking which large language model is "the strongest" is the wrong starting point, because model selection is a constrained engineering optimization problem shaped by quality, latency, cost, data compliance, and operations. It presents a reproducible method for evaluating candidate models: defining business constraints up front, building golden-set evaluations for code, long-context, and multimodal capabilities, measuring agent tool-calling stability, and calculating true cost per task. It also covers deployment options and compliance, a six-step selection workflow with weighting and canary testing, and a scenario-to-capability reference table, helping teams choose the model that best fits their workloads instead of chasing leaderboard rankings.

    When choosing a large language model, many teams ask the same first question: “Which model is the strongest right now?”

    That question is a false premise. Model selection is not a contest of leaderboard rankings; it is a constrained engineering optimization problem: finding the best overall solution for a specific business under multiple constraints—quality, latency, cost, data compliance, and operational capability. Strengths such as strong code generation, reliable tool calling, long context, fast output, and low price rarely converge in a single model. When the scenario changes, the weights change, and so does the conclusion.

    This article does not restate vendor specifications. Instead, it provides a reproducible selection method that teams can use to break down requirements, build evaluations, and account for costs, ultimately arriving at their own answer.

    1. Why “the strongest model” cannot be answered directly

    1.1 Public benchmarks diverge systematically from real business needs

    Public evaluations typically measure “general capability,” while businesses need “stable output on specific tasks.” Leaderboard-style benchmarks have the following limitations:

    • Data contamination: Test questions may already be in the training corpus, so scores are systematically inflated.
    • Generic questions: Real business involves implicit constraints, internal terminology, and long-tail edge cases, whereas benchmark questions tend to be clean and idealized.
    • Single-metric focus: Leaderboards mostly track the correctness of a single answer and do not reflect format compliance rate, long-chain success rate, P95 latency, or cost per task.

    A more reliable approach is to run regression tests against a business-owned “golden dataset.” Because each team’s requirement distribution differs, conclusions drawn from your own data are what carry decision-making value.

    1.2 Capability gaps tend to appear on the critical path

    Suppose a model performs excellently in single-turn Q&A, but the core of the business is a 20-step approval-flow agent where every step depends on structured tool calls. Here, success is not determined by the average score but by:

    • At a 98% single-step success rate, the ideal end-to-end success rate is 0.98²⁰ ≈ 67%;
    • With automatic retries, if the single-step success rate rises to 99.9%, the ideal chain success rate is about 0.999²⁰ ≈ 98%.

    Two models averaging 90 and 85 can differ by more than twofold in output yield on long-chain tasks. Selection must focus on the weakest link on the critical path.

    2. Prerequisites: write down your business constraints

    Before writing any code, it helps to answer these six questions on a single page:

    1. Task profile: Is the work mainly generation, extraction/classification, code generation, document Q&A, or multi-step tool calling? What proportion does each represent?
    2. Data classification: Does the input contain personal information, trade secrets, or classified content? Can it leave the intranet?
    3. Performance SLA: What is the ceiling on end-to-end P95 latency? How long a time-to-first-token (TTFT) is acceptable? What is the peak concurrency?
    4. Output format: Is it streaming output for end users, or structured results for systems (e.g., JSON)? What are the format-compliance requirements?
    5. Budget basis: Is a lower unit price always better, or is lower “cost per task + rework cost” better? What is the scale—thousands of calls per day, millions, or hundreds of millions of tokens?
    6. Engineering constraints: Can the existing tech stack and calling protocols integrate smoothly? Is fine-tuning or private deployment required? Does the team have inference-ops capability?

    Once constraints are quantified, selection shifts from “comparing by feel” to “filtering by conditions.”

    3. Capability dimension 1: Code and engineering capability

    3.1 It is not just about “can it write code”

    Coding ability can be broken into four layers:

    • Local generation: Completing functions, generating implementations from comments.
    • Cross-file understanding: Locating relevant code in a repository with multiple modules while keeping naming style, error handling, and directory structure consistent.
    • Review and refactoring: Identifying potential defects (e.g., null pointers, missed boundary conditions, concurrency races) and giving convention-compliant fix suggestions.
    • Business logic explanation: Understanding domain code and explaining the call chain and data flow in natural language, helping newcomers grasp legacy systems quickly.

    The last two better reflect a model’s engineering depth, and they are exactly what scenarios like code review tools and R&D knowledge-base Q&A need most.

    3.2 Evaluate with your own repositories

    The evaluation set can be accumulated from the team’s real code repositories:

    • Take diffs from historical commits as samples, have the model complete them before the change, then compare against the human result.
    • Cover tasks such as unit-test generation, defect localization, API documentation generation, and legacy-code interpretation.
    • Track results by language and framework separately, so a strength in one language does not mask weaknesses in others.

    A minimal runnable evaluation skeleton is shown below (the client is pseudocode representing the model service the team actually integrates with):

    1. # eval_runner.py: Compare candidate models side by side using an internal evaluation set
    2. import json
    3. import statistics
    4. import time
    5. from concurrent.futures import ThreadPoolExecutor
    6. def load_cases(path="golden_set.json"):
    7. """golden_set.json is built from real business samples;
    8. each item looks like {"id": ..., "type": ..., "prompt": ..., "expected": ...}"""
    9. with open(path, encoding="utf-8") as f:
    10. return json.load(f)
    11. # The registry maps each task type to a checker: unit-test execution, diff similarity, JSON validation, rule matching, etc.
    12. CASE_CHECKERS = {}
    13. def case_checker(case_type):
    14. def decorator(fn):
    15. CASE_CHECKERS[case_type] = fn
    16. return fn
    17. return decorator
    18. def run_case(client, case):
    19. t0 = time.perf_counter()
    20. output = client.complete(
    21. messages=[{"role": "user", "content": case["prompt"]}],
    22. temperature=0.0, # Fixed sampling during evaluation to reduce variance
    23. )
    24. latency = time.perf_counter() - t0
    25. passed, score = CASE_CHECKERS[case["type"]](output, case)
    26. return {"id": case["id"], "pass": passed, "score": score, "latency_s": latency}
    27. def evaluate(client, cases, workers=8):
    28. with ThreadPoolExecutor(max_workers=workers) as pool:
    29. results = list(pool.map(lambda c: run_case(client, c), cases))
    30. n = len(results)
    31. return {
    32. "pass_rate": sum(r["pass"] for r in results) / n,
    33. "avg_score": statistics.mean(r["score"] for r in results),
    34. "p95_latency_s": sorted(r["latency_s"] for r in results)[int(n * 0.95)],
    35. "details": results,
    36. }

    To spread the candidate models apart, prompts should deliberately include engineering constraints, for example:

    1. Modify only src/order/service.py; do not change the interface signatures;
    2. preserve the existing logging and exception-handling style;
    3. new logic must cover both boundary cases where the order amount is 0 or negative;
    4. output in unified diff format.

    Only a model that satisfies all of these constraints at once is truly engineering-ready.

    4. Capability dimension 2: Long-context and multimodal understanding

    4.1 “Supporting long context” is not the same as “using it effectively”

    The advertised context length is usually a maximum value; the usable length depends on three more things:

    • The “lost in the middle” phenomenon: When key information sits in the middle of the context, the model tends to ignore it.
    • Attention overhead: Attention computation grows quadratically with sequence length, so long inputs noticeably increase latency and cost.
    • Accuracy decay: Under very long inputs, detail recall and numerical computation become error-prone.

    Classic industry research has confirmed that even when key information is within the context, a model may answer incorrectly because of its position, which shows that “fitting it in” is not the same as “reading it clearly.”

    Long-context capability deserves its own evaluation questions:

    • Needle in a haystack: Embed a fact at a random position in a tens-of-thousands-token document and require an exact restatement.
    • Cross-paragraph reasoning: Conditions scattered across multiple sections of a document must be combined to reach a conclusion.
    • Structured extraction: Extract hierarchical table data from a long PDF report.
    • Summary consistency: The summary must not contain information absent from the original.

    Another important judgment is the division of labor between long context and retrieval-augmented generation (RAG):

    • Long context suits single tasks with predictable input size that need full-text cross-analysis, such as contract comparison, industry-report study, and reviewing a full year of meeting notes.
    • RAG suits scenarios where the knowledge base far exceeds the context window, needs frequent updates, and requires citation traceability, such as enterprise-policy Q&A and customer-service knowledge bases.

    In practice, the two are often combined: retrieve to narrow the scope first, then feed the whole passage into long context, balancing coverage and cost.

    4.2 The value of native multimodality lies in “removing conversions”

    When a model can directly understand images, screenshots, scanned documents, PDFs, and audio/video, it can eliminate multiple intermediate steps such as OCR, layout reconstruction, and format conversion. The benefits include:

    • Reduced conversion loss: Complex tables and layouts are easily misaligned after OCR, whereas native understanding preserves spatial structure.
    • Shorter pipelines: One fewer step means one fewer failure point and one less data transfer.
    • Lower maintenance cost: No need to maintain upgrades and adaptations for multiple parsing services.

    Typical scenarios include turning screenshots into page code, reconstructing design mockups, extracting information from paper documents, and summarizing video content. During evaluation, focus on error-prone aspects such as fine-grained localization (the relative position of a button in an image), dense-text recognition, and detecting contradictions between text and images.

    5. Capability dimension 3: Agent capability and long-chain tool calling

    An agent’s reliability depends mainly on the following engineering metrics rather than the model’s overall “cleverness”:

    Metric Meaning Suggested acceptance criteria (example)
    Argument schema compliance Share of tool-call arguments that conform to the JSON Schema ≥99.5%
    Per-step tool selection accuracy Share of cases where the correct tool is chosen from the candidate set ≥95%
    Long-chain completion rate Share of multi-step tasks completed within the maximum step count ≥90%
    Error recovery rate Share of cases where the model self-corrects and continues after a tool error ≥80%
    Prompt-injection resistance External content returned by tools cannot hijack model behavior 100% blocked

    Engineering should also hold to three principles:

    1. The model decides, the code validates: All tool arguments must be validated server-side before entering a real system; model output is always treated as untrusted input.
    2. Make implicit rules explicit: In tool descriptions, clearly state the meaning of each argument, its value range, its side effects (read-only or write), and any requirement for secondary confirmation on high-risk actions.
    3. Externalize state and make failures traceable: Persist conversation history, executed steps, and intermediate results, and support resuming from breakpoints and human takeover.

    An example of a tool-call wrapper with validation and retry:

    1. import json
    2. TOOL_SCHEMAS = {
    3. "query_order": {
    4. "required": {"order_id"},
    5. "types": {"order_id": str, "fields": list},
    6. },
    7. "create_refund": {
    8. "required": {"order_id", "amount", "reason"},
    9. "types": {"order_id": str, "amount": (int, float), "reason": str},
    10. },
    11. }
    12. def validate_args(name, args):
    13. schema = TOOL_SCHEMAS[name]
    14. if not isinstance(args, dict) or not schema["required"].issubset(args):
    15. return False
    16. return all(isinstance(args[k], t) for k, t in schema["types"].items() if k in args)
    17. def call_tool_with_retry(client, messages, tool_choice, max_retry=3):
    18. """Model output is always treated as untrusted input; arguments may only be
    19. executed after passing schema validation."""
    20. for attempt in range(max_retry):
    21. raw = client.complete(messages, tools=TOOL_SCHEMAS, tool_choice=tool_choice)
    22. try:
    23. args = json.loads(raw)
    24. except json.JSONDecodeError:
    25. args = None
    26. if validate_args(tool_choice, args):
    27. return args
    28. messages = messages + [
    29. {"role": "assistant", "content": raw},
    30. {"role": "user", "content": "The previous arguments were invalid. Please output valid JSON strictly following the schema."},
    31. ]
    32. raise RuntimeError(f"Arguments for tool {tool_choice} were still invalid after {max_retry} attempts")

    In addition, high-risk write operations (such as refunds, transfers, and data deletion) should not be triggered directly by the model. The safer approach is “the model proposes arguments, and the business system re-validates them and requires human confirmation.”

    6. Performance and cost: the underestimated hidden overhead

    6.1 Look at latency in two parts

    The speed users perceive is determined jointly by two metrics:

    • Time to first token (TTFT): The time from when input enters the inference queue to when the first token is emitted; it determines “how long after clicking before text starts appearing.”
    • Output speed (TPS): The number of tokens generated per second; it determines how long a complete answer takes to read.

    Long inputs, queue waiting, and model size all raise TTFT. Consumer-facing chat products are especially sensitive to output speed: when it is noticeably slow, users keep refreshing or resubmitting, which worsens backend queuing.

    During evaluation, record P50 and P95 metrics under different input lengths, and cover real traffic periods such as morning and evening peaks, rather than looking only at averages.

    6.2 Account for cost as “cost per task”

    The price list is only part of the cost. A genuinely comparable basis is:

    1. cost per task = input token cost + output token cost - cache savings + retry overhead + rework overhead
    2. input token cost = (system prompt + conversation history + current input + retrieved snippets) / tokenizer conversion factor × input price
    3. output token cost = output tokens / tokenizer conversion factor × output price
    4. cache savings = reusable prefix tokens / tokenizer conversion factor × input price × discount rate
    5. retry overhead = cost per task × (1 / first-attempt success rate - 1)

    The tokenizer conversion factor varies by language: a 200-character Chinese text roughly corresponds to somewhere between one hundred and a couple hundred tokens, depending on the tokenizer implementation.

    A cheap but low-accuracy model can push costs higher through frequent retries and human rework; an expensive model that gets it right the first time may actually be cheaper. Practical cost-reduction measures include:

    • Switching simple tasks such as classification, extraction, and format conversion to smaller-parameter models;
    • Enabling prefix caching so that fixed system prompts and knowledge snippets reuse computation results;
    • Controlling output length, agreeing that structured output returns only necessary fields;
    • Summarizing and compressing conversation history to avoid unbounded context growth.

    6.3 Concurrency determines capacity planning

    Under high concurrency, two more things matter:

    • Queuing and rate limiting: The server’s rate limits and concurrency ceiling determine whether it can hold up under peak traffic.
    • Per-resource throughput: In private deployment, the number of concurrent requests a single card can support directly determines the amount of hardware to buy.

    It is advisable to run load tests with requests that match the production distribution, measure P95 latency under light load and full load separately, and identify at what concurrency level the inflection point appears.

    7. Deployment forms and data compliance

    Common deployment forms fall into four categories:

    Deployment form Suitable scenarios Key concerns
    Public API calls Fast prototyping, internet businesses with elastic traffic Price, rate limits, SLA, whether data is used for training
    Dedicated inference resources Production systems with stable latency and concurrency requirements Capacity planning, elastic scaling, cost control
    Private deployment Confidential data, heavily regulated industries, intranet environments Hardware investment, inference framework selection, ops headcount
    Self-hosted open weights Deep fine-tuning needs, full control over model behavior Fine-tuning data quality, version upgrades and regression testing

    Open weights do make private deployment convenient, but this is not a cost-free option:

    • Hardware investment: Purchasing or renting GPUs, plus data-center power and cooling.
    • Inference optimization: Quantization, batch scheduling, and GPU-memory management all require dedicated tuning.
    • Operations system: Version upgrades, monitoring and alerting, and failure recovery all need dedicated owners.
    • Capability gap: Self-built versions usually lag top closed-source models in capability, and this gap must be closed through business-specific fine-tuning.

    On the compliance side, it is advisable to establish a data-classification policy that specifies which data may leave the intranet, which must be processed locally, and which must be desensitized before calls, and to retain desensitization records in call logs to support audit traceability.

    For example, on Baidu AI Cloud, enterprises can use the Baidu AI Cloud Qianfan model service platform to onboard models of different sizes into a unified application development and invocation system, balancing quality and cost; the specific models and capabilities supported are subject to the official documentation. Baidu AI Cloud also offers general-purpose cloud products such as containers, object storage, log services, and monitoring/alerting, which can respectively host the deployment and runtime of inference services, storage of datasets and documents, tracing of invocation chains, and observation of capacity and cost—helping enterprises build the infrastructure that follows model selection.

    8. Turning selection into a reproducible engineering process

    It is recommended to proceed in six steps:

    1. Narrow the candidate set: Filter 3–5 candidates from three angles—quality, form (open/closed source), and compliance—to avoid an overly broad scope that dilutes the evaluation.
    2. Build the golden set: Sample 200–500 items from real business, covering high-frequency, long-tail, and error-prone scenarios, and fix the expected answers or acceptance rules.
    3. Unify the access layer: Use an adapter layer to hide protocol differences between services, so evaluation and business code share the same calling logic:
    1. class LLMClient(Protocol):
    2. def complete(self, messages: list, **options) -> str: ...
    3. def stream(self, messages: list, **options) -> Iterator[str]: ...
    4. class RoutingClient:
    5. """Routes tasks to different models by task characteristics, making it easy
    6. to do gradual rollouts and cost optimization."""
    7. def __init__(self, routes: dict):
    8. self.routes = routes
    9. def select(self, task: dict) -> LLMClient:
    10. if task.get("input_tokens", 0) > 120_000:
    11. return self.routes["long_context"]
    12. if task.get("tool_steps", 0) >= 5:
    13. return self.routes["agent"]
    14. if task.get("difficulty") == "simple":
    15. return self.routes["light"]
    16. return self.routes["general"]
    1. Score across dimensions: Set weights for each dimension, quantify scores from evaluation results, then aggregate them by weight:
    Evaluation dimension Suggested weight Key metrics
    Task quality 35% Golden-set pass rate, human rating
    Critical-path stability 20% Format compliance rate, long-chain completion rate
    Performance 15% P95 latency, TTFT, output speed (tokens/sec)
    Cost 15% Cost per task (including retries and rework)
    Deployment & compliance 15% Deployment form, data boundaries, ecosystem compatibility

    Weights can be adjusted by business stage: in the MVP stage, raise the weight on quality and speed; at scale, raise the weight on cost and stability.

    1. Canary validation: Let the new model take 5%–10% of production traffic first, run an A/B comparison against the baseline model, and track success rate, latency, human-takeover rate, and cost per unit.
    2. Continuous regression: Model services iterate continuously, so the golden set should be integrated into CI; after each version switch or prompt change, run automatic regression to prevent silent capability degradation.

    9. Scenario-to-capability mapping: a quick reference table

    Business scenario Primary capability Secondary consideration Suggested form
    Complex business code generation and review Code understanding and adherence to engineering conventions Cross-file consistency API or dedicated resources
    Large-scale document reading and comparison Effective long-context utilization Degree of structured output Long-context model
    Design mockups, screenshots, document understanding Native multimodality Fine-grained localization Multimodal model
    Multi-step approval / ops agents Tool-call compliance and error recovery Chain observability High-stability model + state machine
    High-frequency consumer-facing chat Output speed and concurrency capacity Cost per request Lightweight, high-speed model
    Confidential intranet systems Data must not leave the domain Private-deployment ops cost Self-hosted open weights
    Massive simple classification and extraction Throughput and unit price Structured-output stability Small-parameter model

    Conclusion

    The answer to “which is strongest” changes over time and by task, but the selection method does not go out of date. What truly determines a project’s success is often three things:

    • Evaluate with your own data rather than public leaderboards;
    • Focus on the weakest link on the critical path;
    • Account for cost down to the “per task” granularity.

    Going further, mature teams usually do not bet the business on a single model. Through a unified access layer and routing strategy, they orchestrate multiple models—large models tackling complex tasks, small models handling high-frequency simple requests, and long-context models processing very long inputs—to achieve a long-term balance among quality, cost, and stability.

    If you remember just one sentence: Don’t ask which model is strongest; ask which model best fits the task in front of you.