LLM Selection for Engineering Projects: A Reusable Evaluation Method

    With so many capable large language models on the market, "which one is best?" has no universal answer. This article presents a reusable engineering method for choosing an LLM: translate business needs into measurable constraints, evaluate candidates across six dimensions (long context, coding and reasoning, agent workflows, inference performance and concurrency, multimodality, and deployment and total cost), then follow a five-step process from candidate shortlist to production launch. It also includes a scenario-to-focus mapping, common pitfalls to avoid, and architectural practices that keep models swappable—turning selection from a gut-feeling question into a quantifiable, verifiable, and repeatable engineering decision.

    Introduction: Why “Which One Should I Choose?” Is Hard to Answer

    “With so many large language models on the market, and the leading options from each provider having their own strengths, which one should we actually pick?” This is a question almost every team that is about to put an LLM into production runs into. After several rounds of iteration, the leading models are now fairly close in capability; their differences show up more in capability focus, engineering characteristics, and cost structure.

    Because of this, “which is the best” has no single answer. The right question to ask in engineering is: Under your business constraints, which model minimizes total cost × failure risk? Here, “total cost” includes not only API usage fees but also labor, operations, and trial-and-error costs; “failure risk” refers to the likelihood of wrong answers, interrupted tasks, budget overruns, and violations of data-compliance requirements.

    This article does not run a head-to-head benchmark of any specific product. Instead, it starts from a general engineering methodology and breaks the selection process into four stages:

    1. Formalizing requirements: Translating business needs into measurable metrics.
    2. Dimension-based evaluation: Assessing candidates across six dimensions—context, coding, agents, performance, multimodality, and deployment and cost.
    3. Process-driven decision-making: Using a five-step method to close the loop from candidate pool to launch.
    4. Engineering the mechanism: Preserving the ability to switch models at the architecture level.

    If you only care about the ranking on some generic leaderboard, reading that leaderboard is enough. But if you need to choose a model for a real business, I suggest spending half a day mapping out your scenario constraints before you start benchmarking. That upfront investment often saves weeks of rework later.

    1. Start by Building a Scenario Profile: Know What You Actually Need

    Many selection efforts get bogged down in “analysis paralysis” because they start by comparing product specs side by side, without first answering “what kind of model does my business actually need?” A good approach is to structure your requirements with a “scenario profile table”:

    Profile dimension Key question to answer Impact on selection
    Input form Plain text, images/scans, or audio/video? Determines whether you need multimodal capability or a two-stage “recognition + text” approach
    Context size How long is a single document? How many documents at once? Determines the required context window and its effective usability
    Task chain Single-turn Q&A, or a long workflow that calls multiple tools? Determines requirements for agent capability and structured output
    Reasoning depth Summarization/extraction, or cross-document reasoning and solution design? Determines requirements for complex reasoning and coding
    Traffic profile Real-time consumer-facing interaction, or internal batch processing? Determines sensitivity to latency, throughput, and concurrency
    Data boundary Can data leave the internal network? Are there compliance-audit requirements? Determines whether a public cloud API is acceptable or private deployment is mandatory
    Cost structure Pay-as-you-go, or existing in-house compute? Affects the trade-off between unit price and total cost of ownership
    Iteration cadence Rapid experimentation, or long-term stable operation? Affects whether you need a unified access layer and continuous evaluation

    After completing the profile, you will find the candidate range narrows quickly. Here are two examples:

    • If the business is high-concurrency Q&A for the public, data can go to the cloud, and the context usually does not exceed tens of thousands of tokens, then selection priorities land on time-to-first-token, concurrent throughput, and unit price; whether the context window reaches hundreds of thousands of tokens is a secondary factor.
    • If the business is an internal R&D knowledge base and code assistant that requires documents and code not to leave the domain, then the private deployment capability of open-weight models is often a hard prerequisite, and no matter how convenient a cloud API is, it is out of consideration.

    Conclusion: constraints determine the candidate pool; capability determines the final choice. Without this step, any comparison is just “comparing for the sake of comparing.”

    2. Six Evaluation Dimensions: See Clearly Where “Strong” Actually Comes From

    Below I break down the six most important dimensions for selection. Each one covers what to observe, how to verify, and what to watch out for.

    2.1 Long Context: Window Length ≠ Effective Comprehension

    What to observe

    The context window determines how much input a single request can hold, directly affecting “can I hand the model a whole contract, an entire code repository, or a year’s worth of meeting notes at once.” But a longer window does not mean more reliable understanding, for three reasons:

    • Attention dilution: As input length grows, the model’s ability to locate key information often declines. It is commonly observed that details buried in the middle of a long text are more likely to be overlooked.
    • Amplified cost and latency: Long context drives up GPU memory usage and time-to-first-token for inference, and the cost per request rises accordingly.
    • Irrelevant information interference: Dumping everything in may seem convenient, but large amounts of irrelevant paragraphs dilute the useful signal and can actually reduce answer quality.

    How to verify

    Do not just look at the advertised window; run three kinds of tests with your own data:

    1. Location test: Place key information at the beginning, middle, and end of a long document, and check whether the model can accurately recall it.
    2. Cross-section association test: Require the answer to synthesize multiple sections that are far apart, testing multi-hop reasoning.
    3. Long-task retention test: After dozens of consecutive dialogue turns, check whether the model still remembers the originally specified format and rules.

    “Dump everything into a large window” and “read a medium window deeply” are two different technical approaches: the former excels at quickly locating information in massive corpora and covering broad archives, while the latter is better at fine-grained cross-referencing of very long text. Which one fits better is ultimately decided by evaluation results, not by the window number.

    Engineering advice

    For most businesses, a more reliable approach is a “retrieval + model” combination: retrieve relevant fragments first, then hand them to the model. This controls cost while reducing noise. Here is an illustrative context-budget assembly function:

    1. MAX_INPUT_TOKENS = 200_000 # Adjust to your business and the model's actual capability
    2. OUTPUT_RESERVE = 4_000 # Reserve space for the model's output
    3. def build_context(prompt: str, hits: list[dict], limit: int = 200_000):
    4. """
    5. Assemble document fragments within the context budget.
    6. hits: retrieval results sorted by relevance (high to low), each with text and token count
    7. """
    8. used = count_tokens(prompt) + OUTPUT_RESERVE
    9. selected = []
    10. for hit in hits:
    11. if used + hit["tokens"] > limit:
    12. break # Truncate when the budget runs out to keep input within the limit
    13. selected.append(hit["text"])
    14. used += hit["tokens"]
    15. return "\n\n".join(selected)

    2.2 Coding and Reasoning: Use Your Own Test Set Instead of “Grinding the Leaderboard”

    What to observe

    In coding scenarios, model capability can be broken into five progressive levels:

    Capability level Typical task Main difficulty
    Single-point code generation Writing utility functions, SQL, regular expressions Correct syntax, idiomatic style
    Cross-file understanding Locating issues along the call chain, explaining legacy modules Requires understanding project structure
    Refactoring and optimization Splitting overly long functions, replacing deprecated usage Not breaking existing behavior
    Architecture design Designing service boundaries, data models, interface contracts Trade-offs, extensibility
    Code review and business alignment Finding edge-case defects, explaining business-logic intent Understanding both code and business context

    There are also two often-overlooked points:

    • Output length and structural stability: Whether the model keeps consistent indentation, naming style, and file structure when generating large blocks of code directly determines the usability rate of the output.
    • Business-context understanding: Beyond writing code “correctly,” whether the model can explain “why it is written this way” also affects its value in code review and cross-team collaboration.

    How to verify

    Most public benchmark problems skew toward standardized algorithm problems, which differ greatly from real enterprise legacy code, internal frameworks, and coding standards. A more effective approach is to build your own test set:

    1. Real sources: Select 100–300 real cases from your team’s historical issues, PRs (pull requests), and technical review comments.
    2. Task diversity: Cover all five capability levels above, and retain a number of “long-tail” cases (e.g., old tech stacks, complex edge cases).
    3. Automated judging:
      • For generation tasks, judge automatically with “compile + unit tests”;
      • For review tasks, compute accuracy by “manually annotating whether known defects are caught”;
      • For explanation tasks, use multi-person blind scoring.
    4. Quantifiable metrics: For example, first-pass rate, number of repair rounds, human adoption rate, and average hours saved.

    Code snippet:

    1. def eval_code_case(case, model):
    2. """
    3. Evaluate a single code case.
    4. case: {prompt, run_tests(), expected_issues(optional, for review tasks)}
    5. """
    6. answer = model.generate(case["prompt"])
    7. passed = case["run_tests"](answer) # Automatically run unit tests
    8. issues = 0
    9. if case.get("expected_issues"):
    10. # Review task: count how many known defects were caught
    11. issues = len(set(answer.issues) & set(case["expected_issues"]))
    12. return {"passed": passed, "hit_issues": issues}

    2.3 Agent Engineering: Long-Chain Tasks Depend on “Reliable Completion Rate”

    What to observe

    Agent applications require the model to repeatedly “think—call a tool—read the result—decide again” until the task is done. Its requirements differ fundamentally from single-turn Q&A, and the core ones include:

    • Instruction following: Strictly adhering to call order, boundary conditions, and output format.
    • Structured output: Tool-call arguments given as JSON must conform to the agreed schema, avoiding mixing “thinking text” with call arguments.
    • Error recovery: When a tool returns an error, being able to read the message, adjust the strategy, and retry, rather than spinning in place.
    • Chain stability: The more steps a task has, the greater the risk of accumulated errors. In long chains of a dozen or even dozens of steps, whether the model stays aligned with the original constraints is key to whether an agent can make it to production.

    How to verify

    Agent evaluation should center on end-to-end task success rate. I recommend building the following metrics card:

    Metric Definition Reference significance
    Task completion rate Proportion of tasks that fully achieve the goal The most core usability metric
    Argument parsing failure rate Proportion of tool calls whose arguments cannot be parsed Reflects the stability of structured output
    Average call steps Average number of tool-call rounds needed to complete a task More steps mean longer chains, higher cost, and higher error probability
    Human takeover rate Proportion of tasks that require human intervention to finish Measures the degree of automation
    Average cost per task Total tokens and fees consumed per task Assesses economic feasibility at scale

    Engineering advice

    • Set a step limit: Prevent the model from getting stuck in an infinite loop; after exceeding the limit, hand off to a human or fall back to a degraded solution.
    • Control temperature: Task-orchestration scenarios usually use a lower sampling temperature to improve output consistency.
    • Tool design: Clear tool descriptions, minimal parameters, and stable return structures—the benefits of these design choices often outweigh switching models.

    The illustrative code below shows the main loop framework of an agent (model, TOOLS, finish, fail are illustrative definitions; in a real implementation, replace them with the interface of the chosen model):

    1. MAX_STEPS = 25 # Safety cap to prevent infinite loops
    2. def run_agent(task: str, model, TOOLS, finish, fail):
    3. messages = [{"role": "system", "content": TOOLS.description},
    4. {"role": "user", "content": task}]
    5. for step in range(1, MAX_STEPS + 1):
    6. # Use a low temperature to improve determinism in task orchestration
    7. reply = model.chat(messages, tools=TOOLS, temperature=0.1)
    8. if reply.tool_calls:
    9. messages.append(reply.message)
    10. # Execute tools one by one and feed structured results back to the model
    11. for call in reply.tool_calls:
    12. result = TOOLS.run(call.name, call.arguments)
    13. messages.append({"role": "tool", "content": str(result)})
    14. continue
    15. if reply.structured_output:
    16. return finish(reply.structured_output) # Final result in the agreed format
    17. messages.append(reply.message)
    18. # Step limit exceeded; hand off to a human or degrade, avoiding unbounded runs
    19. return fail("exceeded max steps")

    2.4 Inference Performance and Concurrency: Speed Is Not Just “Tokens/Second”

    What to observe

    LLM service performance usually includes three core metrics:

    • Time-to-first-token (TTFT): The time from sending a request to receiving the first token, which directly affects user perception. If a user clicks “send” and waits several seconds before seeing the first character, the experience clearly degrades.
    • Output speed (TPOT): The average time to generate each token in the output phase. Its reciprocal is often loosely understood as “characters per second.” Most mainstream flagship models output at around 30–40 tokens per second; some throughput-focused models can reach around 100 tokens per second.
    • System concurrency: In high-concurrency scenarios, whether the system maintains stable latency as request volume rises.

    Things to note

    • Averages ≠ experience: You must look at the P95 and P99 percentiles. Good averages with a severe long tail still cause widespread stuttering.
    • Complex tasks slow down: With long input, many tool calls, and long output, per-request resource usage is higher, and speed can drop noticeably. Speed tests should cover both “short input, short output” and “long input, long output” scenarios.
    • Batch processing vs. online service trade-off: High throughput often means stronger batch scheduling, which can bring longer waits for the first token. Offline batch tasks and online interactive tasks should be evaluated separately.

    How to verify

    I recommend load-testing with real, high-frequency requests from your business, covering low, medium, and high concurrency levels, recording the percentile curves of TTFT and TPOT, and then judging the stable operating range together with the timeout rate.

    Scenario mapping

    Scenario Typical pattern Focus
    Code completion Short input, short output, high-frequency calls Time-to-first-token and unit cost
    Chat and customer service Medium input, streaming output Time-to-first-token, output speed, concurrency stability
    Document summarization/report generation Long input, long output Output speed, context-handling capability
    Automated agent tasks Multi-turn tool calls End-to-end completion time, cost per task

    2.5 Multimodality and Document Parsing: Native Input or a “Bolted-On Pipeline”

    What to observe

    Enterprise documents are not all plain text; there are also design-mockup screenshots, scanned PDFs, table images, PPTs, and screen recordings. There are usually two ways to handle them:

    • Bolted-on pipeline: First convert images and scans to text via OCR or format conversion, then hand them to a text model. The chain is longer and tends to lose information with complex layouts, tables, and mixed text-image content.
    • Native multimodality: The model reads images, pages, and other inputs directly, reducing intermediate conversion loss. In scenarios like screenshot understanding and converting design mockups to front-end code, this saves steps.

    How to verify

    Select real document samples from your business (at least 30 per category is recommended), and focus on checking:

    • Whether information in scans, tables, and captions is fully preserved;
    • Whether the layout order of mixed text-image pages is correct;
    • After converting a design mockup to code, how well the layout is reproduced and how maintainable the code is;
    • Whether end-to-end latency and per-document cost are acceptable.

    Engineering advice

    For simple form-type documents, a bolted-on OCR is often more stable and cheaper; for complex layouts and visual tasks, prioritize native multimodality. You can also combine the two: first identify the document type with rules, then route to the most suitable processing pipeline.

    2.6 Deployment Model and Total Cost of Ownership: Unit Price ≠ Total Cost

    What to observe

    Deployment approaches fall into three categories:

    Deployment approach Advantages Costs
    Public cloud API Fast integration, no ops, capabilities keep updating Data must leave the internal network; the cost of long-term large-scale calls grows linearly with usage
    Open weights + private deployment Data stays in-domain, self-service tuning and fine-tuning, easy to meet audit requirements Requires GPU resources and an ops team; hardware procurement and cluster scheduling costs are relatively high
    Hybrid deployment Sensitive data stays internal, non-sensitive high-concurrency goes to public cloud Architecture complexity rises; requires unified routing and observability

    Cost estimation should not look at unit price alone. I suggest a simple formula to estimate the total cost of a single request:

    1. Total cost per request = input-token cost + output-token cost + retry cost + human-fallback cost
    2. Monthly total cost = Σ(traffic per business line × total cost per request) + fixed costs (GPU depreciation, ops labor, etc.)

    The cost of long-output, multi-turn agent tasks is often underestimated: a task that goes through a dozen tool calls can consume tens of times the tokens of a single simple Q&A. Only by measuring against your business’s real call structure can you draw meaningful conclusions.

    3. The Five-Step Selection Method: From Candidate Pool to Production Launch

    Step 1: Define constraints

    Use the scenario profile table from Part 1 to produce a “requirements list” that includes hard thresholds and preferences. For example:

    • Hard thresholds: data must not leave the internal network, context must support a single 500-page document, tool-call arguments must parse reliably.
    • Preferences: time-to-first-token below 1 second, support for direct design-mockup input, controllable per-call cost.

    Step 2: Build the candidate pool

    Filter out 3–5 candidates based on the hard thresholds, and avoid “looking at only one or two.” Leading models each have their strengths, and information from a single source easily introduces a biased view.

    Step 3: Build the evaluation set

    Build the evaluation set around your business’s real data, at a recommended scale of:

    • General capability set: 100–200 questions;
    • Long-context set: 20–50 questions;
    • Agent task set: 20–50 tasks;
    • Long-tail/adversarial samples: 20–50 examples.

    Keep a “reference answer” or “acceptance rule” for each type of sample to ensure evaluations are repeatable and comparable.

    Step 4: Score with a weighted scorecard

    The weight of each dimension varies by business. The table below is only a reference configuration and can be adjusted based on your profile:

    Evaluation dimension R&D assistant Consumer-facing customer service Document analysis platform
    Coding and reasoning 40% 10% 15%
    Agent engineering 20% 15% 20%
    Long context 10% 5% 30%
    Performance and concurrency 10% 40% 10%
    Multimodality 5% 10% 15%
    Deployment and cost 15% 20% 10%

    Step 5: Canary validation and continuous iteration

    Evaluation scores only reflect lab performance. Before launch, I recommend proceeding at the pace of “small internal traffic → 5%–10% canary → full rollout,” and monitoring task success rate, user satisfaction, P95 latency, and cost consumption. Models iterate quickly, and selection is not a one-time decision; I suggest re-evaluating once per quarter with an updated evaluation set.

    4. Typical Scenario Mapping: Where to Focus Under Different Needs

    Business need Key constraint Selection focus Recommended validation
    Complex business code generation, architecture design, code review Requires cross-file understanding, large output volume Coding and reasoning, agent stability Real PR test set, track adoption rate
    Mass document archiving, long-form deep reading Many documents, strong cross-document association Long-context effectiveness, cross-section reasoning Location test + cross-section association test
    Multimodal document processing, design-mockup-to-code Scans, images, complex layouts Native multimodality, quality of converted code End-to-end evaluation on sample documents
    Consumer-facing real-time interaction, batch automation tasks High concurrency, low latency, low cost Throughput and concurrency, cost per task Multi-level concurrency load test + cost accounting
    Internal knowledge base, confidential document handling Data must not leave the internal network Open weights and private deployment capability Deployment validation in a private environment

    It is worth emphasizing that the table gives “focus points,” not “the only answer.” For the same need, multiple models can usually handle the job, and the final choice should be decided jointly by weighted scores and canary results.

    5. Common Pitfalls: Don’t Be Led Astray by a Single Metric

    1. Leaderboard-only thinking: Public benchmarks differ greatly from real business distributions in problem type and scoring rules; a high leaderboard ranking does not equal good business performance.
    2. Window-only thinking: A long window only means “it fits,” not “it understands”; effective context and cross-section reasoning are what matter.
    3. Speed-only thinking: High output speed does not mean low end-to-end latency; stability under complex tasks and percentile performance matter just as much.
    4. Open-source-only thinking: Open weights bring self-control, but compute procurement, inference-cluster operations, and tuning labor are all real costs.
    5. Single-trial thinking: Results from trying a few prompts have high randomness; formal selection must rely on an evaluation set that covers the long tail.
    6. One-and-done thinking: Model capabilities evolve rapidly, and today’s best choice may not be the best in six months. The architecture should preserve the ability to switch, rather than hard-coding the business to a single model.

    6. Architecture Advice: Turn “Selection” into a Sustainable Engineering Mechanism

    1. Unified access layer

    Use a unified interface to hide the differences between models, so business code does not depend on a specific implementation. Illustrative code:

    1. from dataclasses import dataclass
    2. @dataclass
    3. class ChatRequest:
    4. prompt: str
    5. max_tokens: int = 1024
    6. temperature: float = 0.2
    7. tools: list | None = None
    8. class ChatModel:
    9. """Unified model access interface: business code needs no changes when switching models"""
    10. def chat(self, req: ChatRequest) -> str:
    11. raise NotImplementedError

    2. Configurable routing

    Dispatch requests to different models by task type: simple, high-frequency tasks (such as classification and extraction) go to a lightweight model to control cost, while complex, low-frequency tasks (such as architecture design and long-document analysis) go to a more capable model. Put the routing rules in configuration to make them easy to adjust dynamically.

    3. Platform-based governance

    Enterprises can use model services and compute resources from cloud platforms such as Baidu AI Cloud to handle model integration, evaluation, and launch, and can leverage general cloud product capabilities such as object storage, message queues, logging services, and monitoring and alerting to archive evaluation data, decouple asynchronous tasks, and achieve unified monitoring of latency, cost, and success rate.

    4. Observability and an evaluation loop

    Record the model version, input and output, latency, and cost of each call to form traceable logs; periodically feed online badcases back into the evaluation set to drive the next round of selection or tuning.

    Conclusion

    Back to the original question: among first-tier models like GLM-5, Kimi 2.5, and Minimax M2.7, which one should you actually choose? The right answer is not in any head-to-head benchmark, but in your own business constraints and evaluation data. Start by building a scenario profile, use the six-dimension evaluation to see capability boundaries clearly, and finally complete a canary launch through the five-step selection method—only then can you turn “which one to choose” from an open-ended question answered by gut feeling into a quantifiable, verifiable, and repeatable engineering decision process. Models will keep iterating, but this method will always hold.