From Evaluation to Production: A Practical Guide to Coding LLM Agent Selection

Introduction: The Focus of Competition Has Shifted

Two recent industry trends are worth noting. First, leading open-source models have achieved completion rates of over 60% on terminal task benchmarks from authoritative third-party evaluation institutions, showing a cross-over trend with closed-source flagship models. Second, multiple model vendors releasing new generation models have committed to opening full weights while reducing API prices by about one-third compared to the previous generation. Simultaneously, resident agents for individual users have begun to emerge, shifting model capabilities from “passive response” to “active execution.”

These trends point to a clear fact: the competition among large models has shifted from “parameter scale” to a comprehensive contest of “agent practical capability + price + openness.” For developers, the question is no longer “which model is the strongest,” but “how to use engineering methods to select a model suitable for your specific scenario.” This article explores a practical selection methodology across three dimensions: evaluation, cost, and open-source.

1. Evaluation Dimension: Determining if a Model “Can Work”

1. From Static Evaluation to Dynamic Tasks

Traditional code evaluations (completing functions, fixing single-point bugs) measure a model’s “knowledge reproduction” capability. Agent evaluation measures “task execution” capability: the model must autonomously complete an engineering task in a real terminal environment through multi-round tool calls (reading/writing files, executing commands, searching logs, running tests). The difference is similar to a “written exam” versus a “practical operation.”

The recent rise of terminal task benchmarks in the industry reflects this trend: connecting the model to an isolated command-line environment, assigning a real engineering goal (such as fixing a failing test or completing an environment configuration), and calculating the proportion of tasks the model autonomously completes within a limited number of steps. The fact that leading models have surpassed a 60% completion rate means that “AI autonomously completing moderately complex engineering tasks” is becoming engineeringly feasible.

2. Why Prioritize Third-Party Independent Evaluations

Vendor-reported evaluation scores carry three typical risks: the evaluation set might have data contamination (highly similar samples seen during training); comparison objects might use weaker configurations; and the scoring pipeline might favor their own output format. The value of independent evaluation institutions lies in:

• Horizontal comparability under the same evaluation set and scoring pipeline;
• Usually retaining a private evaluation subset, which is less prone to contamination;
• Marking confidence intervals to avoid over-interpreting minor leads.

3. Build Your Own Evaluation Matrix

General evaluations serve only as an initial screening. The final decision must be based on proprietary business samples. It is recommended to build an evaluation matrix across four dimensions:

• Task completion rate: The ratio of end-to-end task completion in a real repository/environment;
• Process quality: Average steps, proportion of invalid operations, error recovery capability (whether the model can self-diagnose and adjust strategy after a command failure);
• Context processing: Long log reading, cross-file understanding, memory consistency in multi-turn sessions;
• Safety: Whether it executes destructive operations without confirmation (deleting files, overwriting configurations, sending external requests).

During evaluation, the complete execution trajectory should be recorded rather than just the final result, facilitating the attribution of failure modes. A usable evaluation scaffold pseudo-code is as follows:

def run_agent_eval(model, task_suite, max_steps=30):
    results = []
    for task in task_suite:
        env = SandboxEnv(repo=task.repo, goal=task.goal)  # Isolated sandbox environment
        trajectory = []
        for _ in range(max_steps):
            action = model.decide(env.observation())
            trajectory.append(action)
            if env.is_dangerous(action):        # Safety red line check
                results.append(Record(task, "aborted", trajectory))
                break
            env.execute(action)
            if env.is_success():
                results.append(Record(task, "success", trajectory))
                break
        else:
            results.append(Record(task, "timeout", trajectory))
    return summarize(results)  # Completion rate, average steps, dangerous operation count

4. Two Common Evaluation Pitfalls

• Data contamination: If evaluation tasks are too similar to public cases, scores will be artificially high. The countermeasure is to maintain a private evaluation set and periodically rotate samples.
• Single metric reliance: Any benchmark only covers a portion of the capability spectrum. It is recommended to combine at least two types of benchmarks (code understanding + agent execution), plus a round of proprietary business evaluation for cross-validation.

2. Cost Accounting: Cheap Unit Price Does Not Mean Cheap Total Bill

1. Understand the Billing Structure

API costs consist of four parts: input token price, output token price, cache hit discount, and surcharges per request. The scissor gap of “performance upgrade, unit price reduction by about one-third” seen in recent model iterations means that for the same task load, the bill might decrease rather than increase—provided your calling method is economical enough.

2. Establish a “Unit Task Cost” Metric

Do not just look at the nominal price per million tokens; look at the “expected cost to complete a standard task.” For the same task, a weaker model might consume more rounds due to repeated trial and error. This can be estimated as follows:

Expected task cost = (initial input + retries × incremental per round) × input unit price + cumulative output tokens × output unit price

An intuitive magnitude demonstration (numbers are illustrative only): Suppose Model A’s nominal unit price is three times that of Model B, but it has a high first-pass success rate and rarely needs retries; Model B requires an average of two retries. The actual gap in unit task cost will be far less than threefold. During stress testing, both “success rate” and “token consumption” sequences must be recorded simultaneously; neither can be omitted.

3. Three Engineering Optimization Methods

• Context trimming: In agent scenarios, logs and file contents make up the bulk of input tokens. Summarize, truncate, and read on-demand instead of stuffing everything into the context;
• Cache utilization: Place stable system prompts and tool definitions at the request prefix to hit the cache. Most vendors offer significant discounts for cache-hit portions;
• Tiered routing: Offline migrations, batch refactoring, and other latency-tolerant tasks go through batch processing channels; online scenarios route to different model tiers based on task difficulty.

4. Hidden Costs of Self-Deployment

Open-source weights are free, but inference computing power is not. You need to account for: GPU instance costs, multi-replica high-availability redundancy, inference service operations personnel, and reserved redundant computing power for throughput peaks. A common rule of thumb: at lower daily call volumes, the total cost of the API solution is usually lower; when exceeding a certain volume and having hard requirements for data sovereignty, self-deployment begins to possess both cost and compliance advantages simultaneously.

3. Open-Source Weights: From “Usable” to “Controllable”

1. Difference Between Full Weights and Quantized Weights

A key detail in this open-source trend is that multiple vendors have committed to opening full weights, rather than just quantized versions for trial. The difference is: full weights retain original precision and can serve as a base for continued fine-tuning, distillation, or quantization; quantized versions have lower deployment barriers but irreversible precision loss, making them unsuitable for secondary development. When selecting, strictly verify license terms: whether commercial use is allowed, whether there are revenue scale limits, and whether derivative models have attribution or open-source obligations—these terms directly determine whether the model can enter the production pipeline.

2. Basic Estimation Before Deployment

The most basic estimation is VRAM requirements:

Inference VRAM ≈ Weight volume + KV cache + Activation overhead

Weight volume is approximately “parameter count × precision bytes” (e.g., 70B parameters at half-precision requires about 140GB of VRAM). KV cache is proportional to concurrency and context length. In practice, quantization is often used to compress weight volume, trading KV cache reuse for concurrency capability.

3. General Path for Cloud Deployment

Self-deployment does not mean building a data center from scratch. The industry’s common practice is to combine standardized cloud capabilities:

• Container platform + GPU computing power: Hosts the inference service, elastically scaling with business peaks and valleys;
• Object storage: Stores model weight files, ensuring distribution consistency with content verification;
• Message queue: Handles asynchronous inference requests, peaking shaving and valley filling;
• Logging service and monitoring alerts: Collects inference latency, throughput, and VRAM water levels to establish capacity alerts.

Cloud providers like Baidu AI Cloud have encapsulated these capabilities into general services, allowing teams to focus their energy on evaluation and business orchestration rather than infrastructure operations.

4. Criteria for Deciding on Fine-Tuning

Should you fine-tune after obtaining the weights? It is recommended to do baseline testing first: if prompt engineering plus retrieval-augmented generation (RAG) can solve it, do not fine-tune. Only when the task involves strong domain terminology, fixed output structures, or specific style constraints should you consider parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA). Furthermore, you must build an evaluation set first to prove that the benefit outweighs the risk of overfitting.

4. Selection Decision Framework

Combining the three dimensions, the process can be abstracted into a four-step decision checklist:

1. Can data leave the domain? If not, only consider self-hosting open-source weights (and verify the license allows commercial use). If yes, proceed to step 2.
2. What are the call volume and latency requirements? For low volume, strong elasticity, and rapid validation phases, prioritize APIs, focusing on cost-performance ratio and stability. For high volume with predictable loads, calculate the cost crossover point between API and self-deployment.
3. Is deep customization required? If domain fine-tuning or a private toolchain is needed, select open-source weights as the base. If not, rely primarily on APIs with self-deployment as a fallback.
4. How to track continuously? Establish a quarterly evaluation mechanism: maintain a private evaluation set + track independent evaluation reports + record unit task cost trends. At the pace of “quarterly iterations,” a one-time selection will quickly become obsolete. Continuous evaluation is more important than any single-point conclusion.

Conclusion

The model layer is converging into three composable assets: evaluable capability, accountable cost, and controllable weights. A developer’s true moat is not “backing the right model,” but establishing an evaluation pipeline, cost accounting caliber, and deployment switching capability that is decoupled from specific models—no matter how the underlying model iterates, this system allows you to complete validation and switching within a week. This holds more long-term value than chasing any benchmark leaderboard ranking.