LLM Notes (Part 2): A Practical Guide to Model Selection in the Agent Era

    This second entry in an LLM learning-notes series looks at model selection in the Agent era. It explains how reasoning capability has settled into a baseline while reliable execution has become the real dividing line, why small and mid-size models deserve a second look, and how to choose models through measurement rather than gut feeling. It walks through a four-step evaluation methodology with a ready-to-use Python benchmarking script, covers VRAM and cost estimation for deployment, and describes the engineering practices—tool contracts, structured output, tiered routing, and fallbacks—that keep Agents reliable in production.

    1. Introduction: From “Deep Thinking” to “Reliable Execution”

    Looking back at 2026, it is hard for the LLM field to reproduce the kind of shock the previous year’s “deep thinking” paradigm delivered. Reasoning models—driven by reinforcement learning to produce long chains of thought and to “think a little longer” before answering—profoundly reshaped developers’ perception of the ceiling of model capability. Today these capabilities have gradually become an industry default, folded into the training recipes of mainstream models.

    This year’s spotlight mostly falls on AI Agents. But for developers, what matters more than the hype around a concept is a real shift: the evaluation standard is moving from “how impressive is the answer” to “can the task be completed reliably.”

    Building on the previous entry in this series, this article focuses on model selection and engineering practice in the Agent era, and answers four questions:

    • What has changed in the overall capability landscape;
    • Why small and mid-size models are worth re-evaluating;
    • How to select models scientifically and quantify the payoff;
    • How to ensure reliability when Agents go into production.

    2. The Capability Landscape: Reasoning Settles In, Execution Becomes the Dividing Line

    2.1 The Aftermath of Deep Thinking

    The value of the deep-thinking paradigm has not faded; it has been “infrastructuralized.” Mainstream models now generally handle chain-of-thought reasoning, multi-step decomposition, and self-verification. Simple tasks no longer require lengthy internal deliberation, and complex tasks can raise accuracy through step-by-step reasoning. Reasoning has shifted from a “scarce selling point” to a “foundational layer,” and the competitive differentiation between models has moved toward execution: the stability of tool calls, the accuracy of tool invocation, and the ability to self-correct over long workflows.

    2.2 Why Agents Take Center Stage

    An Agent is essentially a combination of “large model + loop + tools + state.” It no longer stops at emitting a passage of text; instead, it can:

    • decompose a goal;
    • call external tools such as search, code execution, databases, and business APIs;
    • adjust its strategy dynamically based on the results returned;
    • retry or switch paths on failure;
    • ultimately deliver a usable result.

    The value ceiling of a single-turn Q&A is a piece of text; the value ceiling of an Agent is a task that can be verified. That is the fundamental reason it keeps drawing attention.

    2.3 A Realistic View of the Capability Gap

    Across the overall capability distribution, a gap still exists between the open-source camp and the world’s top closed-source models, especially on high-difficulty composite tasks—complex reasoning, consistency over very long contexts, and the stability of multi-tool coordination. The gap is mainly at the ceiling, not the floor.

    For most business scenarios, open-source models are already good enough for general Q&A, document processing, and common coding tasks. They are sufficient to support production deployment, while offering advantages that closed-source options struggle to match: data sovereignty, on-premises deployment, custom fine-tuning, and invocation cost. So the more pragmatic stance is to “allocate models by task difficulty” rather than to chase “one strongest model that solves everything.”

    2.4 A Shifting Division of Labor for Long-Tail Tasks

    The relative value of top-tier models is also changing. Once foundational capabilities become commoditized, these models are better suited to a long-tail and fallback role—low-frequency, high-difficulty tasks with high trial-and-error costs. High-frequency, standardized tasks can be pushed down to lighter models and more deterministic engineering pipelines. This division of labor is the basis for the “tiered routing” strategy discussed later.

    3. The Rise of Small and Mid-Size Models: From “Workable” to “Good”

    3.1 Where the Progress Comes From

    In the past, small-parameter models were often dismissed as “toys” that could only handle narrow tasks like classification and extraction, and struggled once instructions grew complex. That impression has now been broken, for roughly four reasons.

    1. Mature training recipes: High-quality instruction data, preference alignment, and distillation techniques are widely applied, allowing small models to inherit part of the reasoning and instruction-following ability of larger ones.
    2. Externalized reasoning paradigms: Capabilities such as chain-of-thought, tool calling, and self-verification are written into training data and system-prompt templates, so even a model with a limited “brain” can achieve more reliable results by leaning on external steps.
    3. Architectural and inference optimization: Mainstream models commonly adopt grouped-query attention to reduce key-value cache overhead. Engineering optimizations in inference frameworks—VRAM management, continuous batching, prefix-cache reuse, and speculative decoding—also give small and mid-size models respectable throughput on a single GPU or even consumer-grade hardware.
    4. Mature quantization and deployment: 4-bit weight quantization is now fairly common, with limited accuracy loss after quantization. A mature deployment toolchain makes “download and run” a reality.

    3.2 Parameter Scale and Capability Boundaries

    Parameter count does not equal capability. Model architecture, data quality, training stages, and post-training strategy are just as important. For selection purposes, capability can be tiered by parameter scale:

    Parameter scale Typical runtime environment Suitable tasks Key concerns
    Below tens of billions (e.g., 4B) Edge devices, lightweight servers Classification, information extraction, intent recognition, simple multi-turn dialogue Limited instruction-following; degrades on long context
    Around ten billion (e.g., 9B) A single datacenter GPU, or a consumer GPU with large VRAM (after quantization) Local knowledge Q&A, code completion, short subtasks within an Agent Watch recall and consistency under long context
    Tens to hundreds of billions (e.g., 27B, 31B) Single or dual GPU (after quantization) Complex instruction-following, document understanding and summarization, medium-difficulty coding tasks Approaches the earlier closed-source API performance range; well suited to private deployments as a mainstay
    One hundred billion and above Multi-GPU clusters or cloud services Complex planning, cross-tool long-horizon tasks, high-difficulty reasoning Cost and stability must be weighed together

    From personal experience, some open-source models in the tens-to-hundreds-of-billions range already perform remarkably well on general conversation and coding tasks, and the high latency reported in earlier evaluations has eased somewhat with inference optimization. Models around ten billion parameters perform steadily on most everyday tasks, while lightweight models of a few billion to tens of billions are already enough for basic conversational Q&A.

    It is worth stressing that “can run” and “suited to production” are two different things. The former depends on whether it fits in VRAM; the latter depends on whether task pass rate, tail latency, and unit cost meet the bar—which is exactly what evaluation is for.

    4. Selection Methodology: From Subjective Impression to Quantifiable Decisions

    4.1 The Four-Step Selection Process

    Step 1: Profile the task. Pin down these dimensions:

    • the length distribution of inputs and outputs;
    • latency requirements (time to first token and total completion time);
    • concurrency scale;
    • whether data can leave your environment;
    • whether structured output or tool calling is required.

    Step 2: Build a business evaluation set. Sample from real logs, covering typical, edge, and failure cases. Aim for no fewer than 200 cases, and keep human-annotated reference answers or acceptance criteria. Comparing different models and parameter sizes on the same set of cases is the only way to make conclusions comparable.

    Step 3: Run a small-traffic canary. Run candidate solutions in parallel on 5%–10% of traffic, and compare against real online metrics rather than evaluating only in an offline environment.

    Step 4: Establish a regression mechanism. Bake the evaluation set into the CI pipeline. Automatically trigger a regression whenever you swap models, upgrade the inference engine, or adjust prompts, to prevent “upgrade means regression.”

    4.2 Quantifying Quality and Performance with a Script

    Below is a ready-to-adapt offline evaluation script skeleton. It simulates real requests concurrently and reports pass rate and tail latency:

    1. import json
    2. import statistics
    3. import time
    4. from concurrent.futures import ThreadPoolExecutor
    5. from dataclasses import asdict, dataclass
    6. from typing import Callable, Optional
    7. @dataclass
    8. class Case:
    9. id: str
    10. prompt: str
    11. expected: Optional[str] = None # Human-annotated acceptance criteria
    12. @dataclass
    13. class Result:
    14. case_id: str
    15. latency_ms: float
    16. output_tokens: int
    17. passed: bool
    18. error: Optional[str] = None
    19. def run_case(generate: Callable[[str], str], case: Case) -> Result:
    20. started = time.perf_counter()
    21. try:
    22. text = generate(case.prompt)
    23. passed = (case.expected is None) or (case.expected in text)
    24. # Rough token estimate; in production, use the model's tokenizer
    25. return Result(case.id, (time.perf_counter() - started) * 1000,
    26. len(text) // 4, passed)
    27. except Exception as exc:
    28. return Result(case.id, (time.perf_counter() - started) * 1000,
    29. 0, False, repr(exc))
    30. def evaluate(generate: Callable[[str], str], cases: list[Case],
    31. concurrency: int = 8) -> dict:
    32. # Warm up first so the initial requests are not skewed by
    33. # model loading and cold caches
    34. generate("warmup request")
    35. with ThreadPoolExecutor(max_workers=concurrency) as pool:
    36. results = list(pool.map(lambda c: run_case(generate, c), cases))
    37. ok = [r for r in results if r.passed and r.error is None]
    38. lat = sorted(r.latency_ms for r in ok)
    39. passed_ids = {r.case_id for r in ok}
    40. per_case = {} # Per-case latency, to help locate "slow" and "wrong"
    41. for r in results:
    42. per_case.setdefault(r.case_id, []).append(r.latency_ms)
    43. return {
    44. "total": len(cases),
    45. "pass_rate": round(len(passed_ids) / max(len(cases), 1), 3),
    46. "p50_ms": round(statistics.median(lat), 1) if lat else None,
    47. "p95_ms": lat[int(len(lat) * 0.95)] if lat else None,
    48. "per_case_ms": {k: round(statistics.median(v), 1)
    49. for k, v in per_case.items()},
    50. "errors": sum(1 for r in results if r.error),
    51. "details": [asdict(r) for r in results],
    52. }
    53. if __name__ == "__main__":
    54. def generate(prompt: str) -> str:
    55. # Replace with your actual call: a local inference service,
    56. # a self-hosted service, or a cloud model service API
    57. return "sample output"
    58. cases = [Case("001", "Explain in one sentence what a vector database is", "vector database"),
    59. Case("002", "Rewrite the following in formal written English: don't forget the meeting tomorrow",
    60. "tomorrow's meeting")]
    61. report = evaluate(generate, cases, concurrency=4)
    62. with open("report.json", "w", encoding="utf-8") as fh:
    63. json.dump(report, fh, ensure_ascii=False, indent=2)
    64. print(report["pass_rate"], report["p50_ms"], report["p95_ms"])

    Two things to keep in mind when using this script:

    • Looking only at averages hides problems: Average latency is smoothed, but what users actually perceive is tail latency. Track P95, P99, and the overlap between “slow” and “wrong”—for example, the task completion rate after timeouts and retries.
    • Scoring should be automated: Do the acceptance judgment programmatically inside run_case, such as checking whether key fields are present, whether structured output can be parsed, and whether code passes unit tests. Human annotation is only for reference answers.

    5. Deployment and Cost: Get the VRAM Math Right

    5.1 The VRAM Estimation Formula

    When deploying small and mid-size models, VRAM is the first resource to calculate. You can roughly estimate it as:

    1. VRAM ≈ weight footprint + KV cache + activations and framework overhead
    2. Weight footprint ≈ parameter count × bytes per parameter (FP16 ≈ 2 bytes, INT8 ≈ 1 byte, INT4 ≈ 0.5 byte)
    3. KV cache per sequence ≈ 2 × num_layers × num_KV_heads × head_dim × sequence_length × bytes per element

    Take a model of roughly thirty billion parameters that uses grouped-query attention as an example:

    • FP16 weights need about 60 GB, and about 15 GB after INT4 quantization;
    • if the model has 60 layers, 8 KV heads per attention block, and a head dimension of 128, then at FP16 precision each token takes about 240 KB of KV cache;
    • at a 32K-token context, a single sequence’s KV cache alone approaches 7.5 GB.

    The VRAM budget must also leave headroom for the KV cache of concurrent requests and for the framework’s own activation and communication buffers. This is a common reason behind “the model runs, but concurrency doesn’t scale.”

    5.2 Quantization, Context, and Concurrency

    • Weight quantization: INT8 and INT4 are the two most common tiers today. The gains are clear, but it is worth re-checking accuracy with your business evaluation set, focusing on sensitive scenarios such as numeric computation, structured output, and long-text recall.
    • KV cache compression: You can apply more aggressive KV cache quantization (such as 8-bit or even 4-bit) and set a reasonable maximum context length. Reuse the same system prompt through prefix caching to avoid redundant computation.
    • Batching and speculative decoding: Continuous batching improves throughput, while speculative decoding has a small model “guess” and a large model “verify” to reduce latency. The two work well together.
    • Hardware and deployment form factor: VRAM bandwidth has a significant impact on decode-phase latency, so weigh GPU models when selecting. For high-concurrency scenarios, use a container platform to enable elastic scaling.

    5.3 Cloud or Self-Hosted

    • Cloud model services: Good elasticity and no operations burden; suitable for the phases of fluctuating traffic or rapid validation.
    • Self-hosted inference: Under high-frequency, stable loads, the unit cost is usually more controllable, and data is easier to keep in-house. But it demands more in terms of VRAM capacity, concurrency scaling, and failure recovery.

    If you go with a cloud approach, you can build a closed loop of “evaluate—deploy—monitor—regress” using general-purpose cloud capabilities such as elastic compute, container platforms, object storage, logging, and monitoring with alerting. On cloud platforms such as Baidu AI Cloud, these capabilities are typically offered as general-purpose cloud products; refer to the official documentation for specific specifications and pricing.

    In practice, cost should not be compared only by “price per million tokens.” There are three additional accounts to settle:

    1. the number of calls required to reach the same pass rate, including retries and follow-up questions;
    2. the engineering effort your team invests in deployment and operations;
    3. the experience loss caused by latency fluctuations.

    6. Agent Engineering Practice: Key Variables Beyond the Model

    6.1 The Reliability Checklist

    An Agent’s stability is largely determined by engineering, not solely by model capability. Check each of the following:

    • Clear tool contracts: Define each tool’s name, purpose, parameter structure, and return values, and describe them clearly in the prompt to reduce wrong tool selection by the model.
    • Enforce structured output: For scenarios that require tool calls, use constrained decoding so the output strictly follows a predefined JSON schema, cutting parsing failures at the source.
    • Parameter validation and idempotency: Validate parameters before execution; for side-effecting operations such as placing orders or writing to the database, introduce idempotency keys to avoid duplicate execution caused by retries.
    • Timeouts, retries, and budgets: Set a timeout per step, retry a limited number of times and switch strategies on failure, and cap the maximum number of steps and total execution time to prevent the Agent from looping.
    • Context management: Trim and summarize long tool outputs, and write intermediate conclusions into structured state so key information is not diluted within a long context.
    • Least privilege: Grant tool permissions at task granularity, route high-risk actions to human confirmation, and retain complete execution logs for auditing.
    • Determinism first: Fix sampling parameters such as the random seed and temperature during evaluation. If the business needs diversity, sample multiple times and evaluate by pass rate rather than a single result.

    6.2 Tiered Routing and Fallback

    Tasks of different difficulty place very different demands on model capability, so you can organize models into a “tiered routing” structure that balances quality and cost:

    1. User request
    2. │
    3. ▼
    4. Lightweight classifier ──► simple Q&A / chitchat ──► answer directly with a multi-billion- to ten-billion-parameter model
    5. │
    6. ▼ (complex request)
    7. Local ~10B-parameter model ──► verifier assesses confidence
    8. │ │
    9. │ pass ─┴─ fail ──► tens-of-billions-parameter local model / cloud model service
    10. │
    11. ▼
    12. Output result ──► on failure ──► fall back to a templated reply or hand off to a human

    The verifier can be rule-based (can the structured output be parsed, are fields complete), a retrieval-evidence consistency check, or a lightweight model self-check. The fallback path matters just as much: it determines the experience floor when an Agent hits an extreme case.

    7. Conclusion: Selection Is an Iteratable Engineering Capability

    The 2026 LLM competition has moved from a “single-shot capability contest” to a “systematic deployment contest.” Deep-thinking capability has settled into a foundation, Agents push value toward a closed task loop, and small and mid-size models make “completing the right task at the right cost” a reality.

    For developers, what is more worth investing in than chasing leaderboard rankings is building three long-term capabilities:

    • replace subjective impressions with a business evaluation set;
    • replace the vague “can it run” judgment with VRAM and concurrency ledgers;
    • replace the assumption that the model is “always right” with engineering constraints.

    On that basis, consider the following checklist:

    • Have you built a business evaluation set covering typical, edge, and failure cases;
    • Are you tracking both pass rate and P95/P99 tail latency;
    • Have you run the VRAM math for weights, KV cache, and concurrency headroom;
    • Have you designed tiered routing and fallback/degradation;
    • Have you brought evaluation regression into the process for model or inference-engine upgrades.

    The next note in this series plans to continue with multi-tool coordination and evaluation systems for Agents. Feedback and discussion are welcome.