Large Language Model Selection: Why Intent Understanding Beats Benchmark Scores

    Public benchmark scores don't reveal whether an LLM can handle real production work. This article breaks model quality into three observable dimensions — intent completion, contract adherence, and trade-off reasoning — and uses a concrete concurrency-control case study to show how a single missing constraint can derail an implementation. It explains where small-parameter models tend to fall short, offers engineering countermeasures such as structured contracts, shift-left acceptance testing, and layered model routing, and describes how to build a private evaluation set. The takeaway: select models by rational fit to your actual tasks and by measured data, not by leaderboard rankings alone.

    1. Introduction: Beyond Benchmarks, Projects Need “Intent Understanding”

    Many developers have had this experience: the same vague requirement description, handed to different AI coding tools or large language models, produces drastically different results. Some tools generate code that passes tests but isn’t the design you actually wanted; others look unremarkable yet quickly grasp your real intent by working with the context.

    In “wish-style, quick prototype” scenarios, you only need a model that “gets the work done.” But once you move into a production project, things change. Beyond delivering functionality, the model must understand the project’s purpose, follow team conventions, and advance along the developer’s existing design direction. These capabilities are hard to reflect directly on public leaderboards.

    So when evaluating an LLM, it helps to break “whether a model is genuinely good to use” into three observable dimensions:

    1. Intent completion: Can it infer real needs from fragmented descriptions and proactively spot missing edge cases?
    2. Contract adherence: Can it consistently follow explicit rules and conventions across complex changes?
    3. Trade-off and expression: Faced with multiple solutions, can it analyze pros and cons, explain the underlying principles and boundaries, rather than always compromising or nitpicking?

    The following sections analyze these three dimensions layer by layer, combine them with real enterprise R&D cases, and lay out a practical LLM selection and evaluation method.

    2. Three Dimensions: The Real Gap in Model Capabilities

    2.1 Intent Completion: From “Literal Execution” to “Inferring the Real Purpose”

    Users who focus mainly on trying out tools often adopt “wish-style programming”: the shorter the requirement description, the better, leaving the rest to repeated trial and error. In this mode, the model only needs to “make it work.”

    Production projects face a different set of constraints:

    • Many implicit premises: project background, team norms, and historical decisions often exist only in people’s heads or scattered across documents;
    • Boundaries must be explicit: conditions like concurrency, exceptions, permissions, and rollback directly determine the system’s correctness and security;
    • Design over implementation: sometimes a theoretically superior solution exists, but developers deliberately choose another path based on readability, maintenance cost, and performance trade-offs.

    This demands intent completion ability: when instructions are ambiguous, the model can infer the real purpose from context and proactively spot missing edge cases before it acts. Small-parameter models may not be incapable of “understanding,” but they rely more heavily on how explicitly the prompt is stated. If the edge cases are clearly specified, they can work fast and well; if the prompt is slightly ambiguous, they tend to “make their own decisions,” and often pass tests by luck, so problems get hidden instead of surfaced. Larger parameter counts usually mean stronger context association and multi-step reasoning, so these models are more comfortable with intent completion. But a model’s ability to “guess” is an advantage — it doesn’t mean you can rely on it to guess. Precisely because larger models will “decide for you” when a prompt is ambiguous, developers should write the key constraints clearly.

    2.2 Contract Adherence: The Hidden Test of Production Projects

    Borrowing the concept of “contracts” from software engineering, we can view a project’s requirements on the model as a series of explicit and implicit contracts:

    Contract type Example content
    Structural contract Module division, interface signatures, data structures and return formats
    Behavioral contract Exception-handling strategy, concurrency semantics, idempotency and transaction boundaries
    Style contract Naming conventions, comment granularity, error-code systems
    Evolution contract Changes must be backward compatible and must not break existing callers

    The larger the parameter count, the more it helps to hold multiple constraints simultaneously in a long context. When the context grows longer and constraints pile up, small-parameter models are more prone to fixing one thing while breaking another: a step satisfies the interface format but crosses the line on concurrency semantics; the current test case is fixed but the public behavior of an unrelated module is changed.

    It’s worth stressing that “wish-style programming” and contract adherence are not contradictory. The former solves “getting it built from 0 to 1”; the latter solves “getting it right and stable from 1 to N.” The prototype stage can tolerate ambiguity; production projects must treat contracts as first-class citizens.

    2.3 Trade-offs and Expression: No Compromise, No Nitpicking

    Developers’ discussions of technical solutions often revolve around arguing “whether a judgment is sound, what the principle is, and where the boundaries lie.” This tests the model’s analytical ability:

    • Can it distill the core argument from a lengthy discussion?
    • Can it judge whether that argument holds in a specific scenario?
    • Can it explain the underlying principle and its applicability boundary?
    • When disagreement arises, can it give an evidence-based rebuttal rather than vaguely saying “both have their merits”?

    For example, in distributed systems there is a trade-off between “strong consistency” and “high availability,” as revealed by the CAP theorem: when network partitions are unavoidable, you can only choose between consistency and availability, and which side to pick depends on the business’s tolerance. An experienced engineer will point out in context: “This solution amplifies synchronization latency in cross-region deployments; we need to confirm whether the consistency requirement can be relaxed.” That is far more valuable than a flat “either works.”

    3. Case Study: How One Missing Constraint Derailed the Implementation

    The following uses an abstracted, enterprise-level system case to illustrate.

    Scenario: An organization is building an archive management system in which business data and archives are interrelated: one business record (such as a contract or an approval record) may correspond to multiple archive files, and the archives exist as attachments to the data, with the data’s lifecycle determining the visibility scope of the archives.

    Requirement: Both business data and archive files need to support operations such as editing and deletion, and concurrent modification of the same data and its archives must be prevented.

    Developer’s design intent: Use the data primary key as the granularity of concurrency control, exclusively locking all archives of the same data record together. CRUD operations on the business data and the archive operations share the same “lock,” so a background cleanup task deleting data can also be made mutually exclusive with ongoing archive operations.

    The model’s incorrect implementation: The model added a lock keyed by the archive directory identifier for “archive operations.” This implementation passed all test cases but deviated from the design intent.

    Sample code:

    1. # Counter-example: locking on the archive directory identifier,
    2. # with the lock object decoupled from the data entity
    3. def update_file(file_id, content):
    4. # Locks the archive's directory, not the business data
    5. with exclusive_lock(f"dir:{get_dir_id(file_id)}"):
    6. storage.write(file_id, content)
    7. def delete_record(record_id):
    8. # No data lock acquired; deletion may happen concurrently with archive writes
    9. db.delete_record(record_id)
    1. # Correct example: locking on the data primary key, with data and its
    2. # archives under the same protection scope
    3. def update_file(file_id, content):
    4. # Look up the owning data from the archive, then take an exclusive
    5. # lock on the data primary key
    6. record_id = db.get_record_id_by_file(file_id)
    7. with exclusive_lock(f"record:{record_id}"):
    8. storage.write(file_id, content)
    9. def delete_record(record_id):
    10. # Shares the same lock as archive operations, inherently mutually exclusive
    11. with exclusive_lock(f"record:{record_id}"):
    12. db.delete_record(record_id)

    Risks of the incorrect implementation:

    1. Misplaced lock protection target: The lock is applied at the directory level and cannot cover the race between other entities under that directory and data changes; the protection scope doesn’t match the business semantics;
    2. Lifecycle race: If a background cleanup task deleting data doesn’t participate in the same lock, a “data already deleted, user still writing archives” situation can occur, producing orphaned data;
    3. Complexity explosion: With the lock decoupled from the business entity, every new operation must separately consider “what to lock and what it’s mutually exclusive with,” and code branches and test cases grow rapidly;
    4. False-negative validation: The unit tests only verified “functionality works” and didn’t cover concurrency timing or data-deletion scenarios, so the error was masked.

    Root cause: The prompt’s requirements for “concurrency safety” and “exclusive lock” were explicit, but the key information — “use the data primary key as the lock granularity” — was omitted. For small-parameter models that rely more on explicit instructions, it is easy to “understand literally” and add a lock to the archives, the most frequently occurring objects. Larger-parameter models are more likely to combine the background that “archives attach to data” and infer that the lock should land on the data entity.

    Improved prompt:

    1. 【Task】Add concurrency control for read/write operations on archive data.
    2. 【Constraints】
    3. 1. Use the "data primary key" as the sole lock granularity; all archives under the same data record share the same lock.
    4. 2. Archive read/write operations must first look up the owning data via the archive, then request an exclusive lock on the data primary key.
    5. 3. Data edit and delete operations must hold the same exclusive lock.
    6. 4. Using a directory, file path, or the archive's own identifier as the lock key is forbidden.
    7. 5. Batch tasks such as background cleanup must follow the same locking rules.
    8. 【Acceptance】Add integration tests for concurrency and data-deletion scenarios, simulating "archive writes happening simultaneously with data deletion."

    From an engineering perspective, this requirement should be elevated to a team structural contract: concurrency control granularity must stay consistent with the business entity’s lifecycle boundary. Once an entity is created, modified, or deleted, all resources around it should fall under the same protection scope.

    4. Why It Fails: Three Capability Gaps of Small-Parameter Models

    Combining the case above, we can summarize the typical failures of small-parameter models in production projects into three categories:

    1. Insufficient association of implicit information: The model struggles to derive the implicit conclusion that “the lock should be on the data” from background information like “archives attach to data.” It won’t proactively fill in what the prompt didn’t state.
    2. Weak constraint retention in long contexts: When the prompt contains multiple constraints — interface definitions, naming conventions, exception strategies — on top of a large amount of code context, the model is more likely to lose or confuse some of them.
    3. Insufficient self-checking awareness: It tends toward “if it runs, it’s correct,” rarely questioning whether its own assumptions hold, and seldom proactively pointing out “there’s a concurrency risk here.” By contrast, more capable models more often raise counter-questions like “is this assumption of mine reasonable?”

    It should be stressed that this does not negate the value of small-parameter models. They respond fast and cost little, and suit tasks with clear goals and well-defined scope; when the prompt is complete and the constraints are explicit, their execution efficiency is often quite impressive. The core issue is matching capability to the task.

    At the same time, larger-parameter models can also “fail confidently,” so “being better at inferring intent” cannot replace rigorous validation.

    5. Engineering Countermeasures: Lowering the Cost of “Guessing”

    5.1 Explicitness: Structured Contracts Instead of Verbal Agreements

    For key tasks, rather than repeatedly adding clarifications in conversation, it is better to solidify the requirements in a structured task-description file. It is convenient for version control and review, and serves as a common input across different tools:

    1. 【Task】Add concurrency control for read/write operations on archive data.
    2. 【Scope】Only modify the data access layer of the archive service; do not touch the interface protocol or frontend calls.
    3. 【Constraints】
    4. 1. Use the data primary key as lock granularity; archives under the same data record share the same lock.
    5. 2. Archive read/write must first look up the owning data, then request an exclusive lock.
    6. 3. Data edit and delete must hold the same lock.
    7. 4. Using a directory or file path as the lock key is forbidden.
    8. 【Prohibited】Do not add new common dependencies; do not modify existing interface signatures.
    9. 【Acceptance criteria】Integration tests for concurrency and deletion scenarios all pass, with no new warnings from static checks.

    5.2 Context Management: Give What’s Accurate, Not What’s Abundant

    • Trim, don’t pile up: Provide only relevant modules, interface definitions, and key background, avoiding unrelated code that dilutes attention;
    • Top-load key information: Put the most important constraints at the front or in a separate block to reduce the chance they get buried;
    • Incremental iteration: Break complex changes into small steps, confirming each before proceeding, to avoid generating large-scope changes all at once.

    5.3 Shift-Left Acceptance: Let Errors Be Caught in Tests

    • Prompt as acceptance criteria: Write requirements such as edge cases, error codes, and concurrency semantics directly as test cases;
    • Layered testing:
      • Unit tests verify local logic;
      • Integration tests cover module interactions;
      • Concurrency and fault tests verify timing behavior;
    • Automated gating: Add static checks, contract tests, and regression tests to the CI pipeline, with code-review gates.

    For the concurrency issue in the case, we can add a sample test like:

    1. def test_delete_during_file_update(record_id, file_id):
    2. # Scenario 1: data is deleted concurrently while an archive write is in progress
    3. with concurrent.futures.ThreadPoolExecutor(max_workers=2) as pool:
    4. write_task = pool.submit(update_file, file_id, "v2")
    5. delete_task = pool.submit(delete_record, record_id)
    6. write_task.result()
    7. delete_task.result()
    8. # Assertion: no archive files may remain after deletion
    9. assert not storage.exists(record_id)
    10. assert db.get_record(record_id) is None

    If the implementation uses the “directory lock” version, the above test case will likely expose the race between writing and deletion.

    5.4 Layered Routing: Exploration to Large Models, Execution to Small Models

    A pragmatic strategy is “layered routing”: give exploratory and high-risk tasks to large-parameter models, and tasks with clear boundaries to small-parameter models.

    Task type Typical characteristics Recommended strategy
    Exploration / prototype Vague requirements, need multiple rounds of trial and error Use large-parameter models intensively, controlling per-session duration and context length
    Design & refactoring Broad impact, involves architecture trade-offs Large-parameter models lead; small-parameter models assist with search and organization
    Contract-clear implementation Clear interfaces and boundaries, localized changes Small-parameter models execute, backed by structured contracts and automated validation
    Mechanical tasks Format conversion, comment completion, unit-test generation Small-parameter models batch-process, with cost caps set

    Sample routing logic:

    1. def select_model(task):
    2. if task.risk == "high" or task.scope == "cross_module":
    3. return LARGE_MODEL
    4. if task.retry_budget <= 0:
    5. return LARGE_MODEL
    6. return SMALL_MODEL

    At the same time, establish quota and cost-monitoring mechanisms — for example, setting token-usage caps and retry-count caps for each task category. Once a threshold is exceeded, a human should intervene, rather than letting the model keep burning budget in the wrong direction.

    6. Evaluation Methods: Build Your Own Private Evaluation Set

    Public leaderboard scores can serve as a reference for initial screening, but they cannot test your team’s private codebase, business background, or historical decisions. A more reliable approach is to build a private evaluation set:

    1. Collect real tasks: Select 30–50 representative tasks from historical requirements, defect tickets, and code-review records, covering types such as “vague requirements,” “multi-constraint changes,” “concurrency and exceptions,” and “legacy system maintenance”;
    2. Fix the judging criteria: Predefine pass conditions for each task — including unit tests, integration tests, and human review points — so that different models are compared under the same standard;
    3. Sample multiple times: Run each task several times to observe output stability, not just a single result;
    4. Quantify key metrics:
      • Intent hit rate: the proportion of outputs that match the design intent on the first try;
      • Contract adherence rate: the proportion of outputs that don’t violate explicit constraints;
      • First-pass rate: the proportion that can be merged without human correction;
      • Rework cost: the time, tokens, and retries spent to reach an acceptable result;
      • Stability: output variance across similar tasks;
    5. Iterate and update: Update the evaluation set as project norms evolve, so the criteria don’t become outdated.

    In addition, you can design “argument analysis” questions: give a technical viewpoint with hidden flaws or over-generalizations, and test whether the model can pinpoint the problem, explain the principle, and state the boundaries. Such questions are hard to score well on through memorization and better reflect a model’s real analytical ability.

    7. Comprehensive Recommendations: Rational Selection, Not Score Worship

    Returning to the question “how to evaluate a model,” we can form the following decision framework:

    1. Define the task profile first: Clarify whether the team’s tasks are mostly exploration or mostly execution, along with the risk level and compliance requirements;
    2. Then check capability match: Exploration and design value intent completion and trade-off analysis; execution places more weight on stability and cost;
    3. Validate with the private evaluation set: Use the team’s “first-pass rate” and “rework cost” as core metrics;
    4. Back it up with engineering means: Whichever model you choose, structured contracts, automated testing, and code review are indispensable safety nets;
    5. Keep reviewing: As versions iterate and projects evolve, periodically re-evaluate and dynamically adjust the mix strategy.

    Finally, it is especially important to note that many regulated enterprises and public-sector organizations restrict the use of external or third-party services and tools due to requirements around data security, supply-chain control, and regulatory compliance, and their code and data must be processed within a controlled environment. This means the available options are themselves constrained, and selection must also account for requirements such as private deployment, data residency, and auditability. Within compliance boundaries, systematically improving productivity by building private evaluation sets and refining engineering standards is often more valuable than fixating on any single benchmark score. Technology evolves rapidly, and the capability gaps between models are not fixed; staying rational and letting measured data speak is the sound way to meet the tide of technology.