Large Language Model Evaluation: Capability Boundaries, Metrics, and Selection Criteria

    This article explains why large language models often feel reliable when instructions are clear but "not smart enough" when requirements are vague. It breaks capability down into execution, reasoning, and intent layers, introduces measurable evaluation dimensions, and shows how to build a minimal benchmark. It also examines the limits of post-training, inference-time compute, and agent engineering, while offering practical guidance on structured prompts, self-check loops, context engineering, cost control, and stopping criteria.

    1. A Common Pattern: Clear Instructions Work Well, Vague Requirements Do Not

    Many developers have had the same experience. When requirements are written clearly enough and the task is to generate a piece of business code, clean a dataset, or polish a document, the model can often produce usable output on the first try. This avoids much of the usual “write it once, revise it three times” cycle. But once the goal becomes vague, such as “help me do an analysis” or “write a good-looking script,” the experience changes completely. The model either follows the literal request too mechanically or makes decisions on the user’s behalf at critical points. In both cases, the output usually requires repeated human correction.

    If we abstract this experience, the problem is not simply “can the model write code?” The more important point is:

    The execution capability of general-purpose large models has clearly moved ahead of their ability to understand requirements.

    The more specific the instruction, the more capable the model appears. The more ambiguous the instruction, the more humans need to make judgments on its behalf.

    Therefore, evaluating a large model should not be reduced to a single question such as “can it write code?” A more systematic approach is:

    • Break capability into observable and verifiable dimensions.
    • Distinguish which limits are ceilings determined by pretraining scale and which gaps can be narrowed through post-training and inference-time compute.
    • Use engineering methods to turn repeated human tuning into reusable assets, then decide when to invest more effort and when to stop.

    This article follows that line of reasoning.

    2. Three Sources of the Capability Gap

    1. Task verifiability shapes the first impression

    Programming tasks have a natural advantage: their results can be verified quickly. Whether code compiles, whether unit tests pass, whether an API response matches expectations, and whether logs show anomalies are all objective signals. When task boundaries are clear, the model is effectively working in an environment with a built-in scorer. Having such a scorer is itself an amplifier of capability.

    2. Following clear instructions is not the same as understanding intent

    Tasks in real enterprise environments usually contain many implicit constraints:

    • Why a piece of logic must be written a certain way.
    • The business meaning behind a metric.
    • How boundary conditions should be handled and whether exception branches should be silently handled.

    When instructions are vague, the model can only provide a “literal best solution,” not a “business best solution.” This is the root cause of the feeling that model behavior is mechanical.

    3. The capability ceiling is determined by pretraining; engineering can only adapt it

    A key judgment is needed here: large-scale, high-quality pretraining determines the upper limits of model knowledge breadth, reasoning depth, and language quality. In this sense, parameter scale can be viewed as a rough approximation of an “intelligence ceiling.”

    Post-training methods, such as supervised fine-tuning and reinforcement learning from human feedback, can significantly improve instruction following, format compliance, safety alignment, and task style. Inference-time compute, such as longer chains of thought, multiple rounds of self-verification, and multi-candidate voting, can improve the success rate on difficult tasks. Agent engineering, such as tool calling, task decomposition, and memory management, can combine individual capabilities into workflow-level capability.

    However, all of these methods adapt the model within existing limits. They cannot create emergent capabilities that the model did not originally have.

    3. Why It “Doesn’t Feel That Different”: Turning Feelings into Measurable Metrics

    The feeling that a model is “not that impressive” is a valuable subjective signal, but it is not reproducible, comparable, or optimizable. To turn it into a manageable engineering problem, we need layered evaluation.

    1. Layer one: Automatically verifiable metrics

    Metric Meaning Notes
    First-pass usability The percentage of first-round outputs that pass acceptance without human modification Best reflects real efficiency
    Average revision rounds The number of iterations needed to turn the first output into a usable result Reflects stability and communication cost
    Implicit requirement hit rate The percentage of unstated constraints proactively covered by the output Reflects intent understanding
    Specification compliance Deviation from required formats, fields, lengths, styles, and constraints Can be significantly improved through post-training
    Fallback rate The percentage of cases requiring major human rewrites or abandonment Reflects capability boundaries
    Resource consumption per task Token usage, invocation time, and human intervention time Cost must be included in evaluation

    A minimal but executable evaluation set can look like this:

    1. # Pseudocode: build a minimal but repeatable evaluation set.
    2. # Each case contains: task description, implicit requirements, checker script, and expected output structure.
    3. BENCHMARK = [
    4. {
    5. "id": "biz-001",
    6. "task": "Summarize order details by region and output gross margin rankings",
    7. "implicit_needs": [
    8. "Exclude refunded orders",
    9. "Keep amounts to two decimal places",
    10. "When ranks tie, sort by sales descending",
    11. ],
    12. "checker": "scripts/check_biz_001.py",
    13. "spec": {
    14. "format": "csv",
    15. "columns": ["region", "gross_margin", "rank"],
    16. },
    17. },
    18. ]
    19. def evaluate(model, cases=BENCHMARK):
    20. report = []
    21. for case in cases:
    22. out = model.generate(case["task"]) # First-round output
    23. passes = run_checker(case, out) # Automated acceptance check
    24. implicit_hit = [n for n in case["implicit_needs"] if n in str(out)]
    25. report.append({
    26. "case": case["id"],
    27. "first_pass": passes,
    28. "implicit_coverage": len(implicit_hit) / len(case["implicit_needs"]),
    29. "needs_revision": not (passes and implicit_hit == case["implicit_needs"]),
    30. })
    31. return summarize(report) # Aggregate metrics and form a comparable baseline

    With this baseline, the question is no longer “do I think it works?” It becomes “on my real task distribution, is the first-pass usability rate 42% or 78%?” This distinction determines whether later investment decisions are rational.

    2. Layer two: Cross-task consistency

    The same model can vary greatly across tasks. It is advisable to cover at least four categories because their verifiability differs significantly:

    1. Structured programming tasks with tests and specifications.
    2. Long-tail business logic tasks with complex rules and missing context.
    3. Open-ended writing and content creation tasks with subjective standards and style sensitivity.
    4. Professional reasoning tasks that require multi-step derivation and external knowledge verification.

    Strong performance on the first category does not mean the model is equally usable on the second or fourth.

    3. Layer three: The “verification gap” trap in long-tail domains

    Tasks such as financial analysis, content scripts, and professional writing share a common feature:

    The correctness of the result is difficult to verify automatically.

    • In financial scenarios, a model conclusion may appear reasonable, but it may imply incorrect assumptions about metric definitions, outdated data ranges, or expressions that do not meet regulatory requirements.
    • In content creation scenarios, standards for “well written” are highly subjective. Models tend to produce templated structures, and plot tension and character motivation are often flattened into “correct but mediocre” paragraphs.

    In such tasks, the model’s weakness is not that it “cannot write.” It is that it “does not question its own assumptions.” It rarely proactively asks questions such as “Should this metric definition be adjusted?” or “Is this character motivation internally consistent?” These are precisely the most valuable aspects of experienced professional work.

    4. Locating Gaps with a Three-Layer Capability Model

    A clearer way to classify capability is as follows:

    Layer Typical tasks Main determining factors Optimization methods Common failure modes
    L1: Execution layer Formatting, refactoring, completion, unit test generation Post-training, context quality Structured prompts, example constraints, automated validation Format drift, missing boundaries
    L2: Reasoning layer Multi-step derivation, complex algorithms, professional analysis Pretraining scale, inference-time compute Chain-of-thought guidance, self-checking, multiple candidates Confident mid-way errors, unrecoverable mistakes
    L3: Intent layer Requirement clarification, implicit constraint discovery, goal setting Pretraining scale and world knowledge Clarifying questions, context enrichment Mechanical execution, making decisions for the user

    Deciding which layer a gap belongs to is crucial for later decisions:

    • Most L1 problems can be converged through engineering methods and are not worth excessive anxiety.
    • L2 problems can be partially mitigated through inference-time compute and external tools.
    • L3 problems currently still rely mainly on human experience and are difficult to remove through “tuning.”

    5. Post-Training, Inference-Time Compute, and Agent Engineering: Where the Boundaries Are

    1. Post-training: Improving usability and stability

    The core value of post-training is to make models more compliant, more stable, and more aligned with human preferences. It can help models follow formats better, reduce refusals, and produce more natural expressions. However, it cannot fill knowledge gaps in unfamiliar domains. Nor can it make the model spontaneously produce the tendency to think, “I should ask a clarifying question first,” unless that tendency is explicitly trained into the model.

    2. Inference-time compute: Trading more computation for fewer errors

    Longer chains of thought, multiple sampling and voting, and self-verification loops can indeed improve accuracy on complex tasks. But the costs are higher latency, higher cost, and diminishing marginal returns. For “vague requirement” tasks, a longer chain of thought often means spending more time thinking along the same wrong path, not discovering a wrong assumption.

    3. Agent engineering: Combining individual capabilities into workflows

    Tool calling, retrieval augmentation, code execution environments, task decomposition, and memory management can significantly expand the set of tasks a model can complete. But the essence of agent engineering is to make human workflows explicit. It amplifies capabilities the model already has; it does not create new ones. If a model lacks intent understanding at L3, an agent will only help it run faster in the wrong workflow.

    A pragmatic judgment is:

    First use engineering methods to handle L1 well, then use inference-time compute to tackle L2, keep humans in control of L3, and do not expect the model to “grow up” on its own.

    6. Turning Manual, Hands-On Tuning into Reusable Engineering

    Much developer fatigue comes from “having to teach the model again every time.” This kind of repetitive work can be engineered.

    1. Use structured prompts to make implicit requirements explicit

    Do not let the model guess. Turn guessing into an explicit list:

    1. [Task] Summarize sales details by region and output gross margin rankings
    2. [Constraints]
    3. 1) Exclude refunded orders
    4. 2) Keep amounts to two decimal places
    5. 3) When ranks tie, sort by sales descending
    6. [Output format] CSV, column order: region, gross_margin, rank
    7. [Self-check] After completion, run a self-check:
    8. - Does the row count match the deduplicated details?
    9. - Are ranks continuous and free of duplicates?

    The key is the final “self-check” item. It moves L1 verification responsibility from humans to the process. This is one of the highest-value changes available.

    2. Replace repeated human correction with a “self-check and revise” loop

    1. def build_prompt(task, context="", role="Executor"):
    2. return f"""
    3. You are acting as {role}. Task: {task}
    4. Context: {context}
    5. Requirements:
    6. 1. First restate your understanding of the task and list any ambiguities you see.
    7. 2. If an ambiguity affects the result, state your assumption and label it.
    8. 3. After producing output, self-check: verify item by item that all constraints are satisfied.
    9. """
    10. def run(model, task):
    11. draft = model.generate(build_prompt(task, role="Executor"))
    12. review = model.generate(build_prompt(draft, context=task, role="Reviewer"))
    13. if "passed" not in review:
    14. draft = model.generate(build_prompt(draft, context=review, role="Reviser"))
    15. return draft

    This loop solves the “not acceptable on the first try” problem. It does not solve the “wrong direction” problem. That still requires humans to set the direction.

    3. Use context engineering to reduce the model’s need to guess

    One major reason models perform poorly is insufficient information. Putting business background, data definitions, historical decisions, and glossaries into the context is often more effective than repeatedly tuning the prompt. For long-document tasks, prioritize structured chunking and summarization rather than stuffing the entire source material into a single pass.

    7. Cost and Stopping: When to Give Up Decisively

    “Simplicity goes a long way” is a rational resource allocation strategy and should be institutionalized:

    • Set clear usage thresholds: Use new models when they are usable. Archive them when their limitations clearly fail to meet requirements. Avoid emotional investment.
    • Set cost ceilings: Define token budgets and human intervention time limits for individual tasks. Trigger human takeover when limits are exceeded.
    • Set abandonment criteria: When tuning cost × usage frequency > expected benefit, switch to another solution instead of continuing to invest.
    • Turn tuning results into assets: Prompt templates, evaluation sets, validation scripts, and domain knowledge bases remain valuable even when models are replaced.

    The sustainable approach is not to wait for a perfect model. It is to use engineering to make imperfection manageable.

    8. Common Evaluation Mistakes

    1. Drawing conclusions from a single experience. Model performance varies greatly across tasks. One experience is not enough to support a judgment.
    2. Treating “no strong impression” as “poor capability.” Subjective experience often comes from insufficient prompting or vague task descriptions. Check input quality first.
    3. Looking only at output quality, not interaction cost. A result that takes three rounds of back-and-forth to become usable may be less efficient than writing it yourself.
    4. Treating L1 defects as fundamental model problems. Format, field, and style issues can almost always be solved through structured prompts.
    5. Ignoring domain verification. When professional domains lack objective scorers, responsibility for verification must be explicitly left to humans or rule engines.

    9. Action Checklist

    1. Within this week, build a minimal evaluation set of 20 to 30 cases covering programming, business logic, writing, and professional reasoning. Each case should have an acceptance script or acceptance criteria.
    2. Write structured prompt templates for high-frequency tasks. Include implicit constraints, output formats, and self-check requirements in the templates.
    3. Add “self-check and revise” loops and rule engine validation to key workflows. Let automation handle correctness at the L1 layer.
    4. Organize business background, glossaries, and historical decisions into reusable context assets.
    5. Set cost thresholds and abandonment criteria. Retest new models with the evaluation set every quarter, and use data to decide whether to switch.

    10. Conclusion

    The current capability landscape of large models can be summarized as follows:

    L1 is already quite reliable. L2 is continuously improving. L3 still mainly depends on human experience.

    The experience that “clear instructions work well in programming tasks” is real. So are the gaps in financial analysis, content creation, and mathematical reasoning. These observations do not contradict each other.

    For developers and enterprise users, a more pragmatic strategy is to acknowledge capability boundaries, apply engineering effort within those boundaries, and reserve human judgment for outside them. When evaluating a model, do not only ask, “How smart is it?” Ask:

    • Where is the boundary on my task distribution?
    • What is the cost?
    • What methods can I use to compensate for its gaps?

    Once these three questions are answered clearly, the anxiety around model selection will naturally fade.