LLM Capability Evaluation: Complex Reasoning, Long-Horizon Tasks, and a Reproducible Benchmark Method

1. Ask the Right Question First: Evaluation Is Not a Ranking Contest

“Which model is stronger?” is a question that cannot be answered, because its subject is missing. The meaningful version is: given a specific task distribution, latency budget, cost ceiling, and compliance constraints, which model (or which model route) performs better? The same model can lead on “function-level completion” yet lose points on “repository-level cross-file refactoring” because of how context is organized; it can be stable on single-turn long-form writing yet lose the goal somewhere after the tenth step of a multi-turn tool-calling task.

Three categories of misjudgment show up often in practice:

• Benchmark transferability misjudgment. The corpora, prompt templates, and scoring methods behind public leaderboards usually differ from your real business distribution. Leaderboard rank is a screening signal only, never a substitute for internal validation.
• Single-sample quality misjudgment. A single generation can look impressive, but run the same prompt ten times and the pass rate may swing widely. Production cares about the stable pass rate, not the best single output.
• Feature-checklist misjudgment. Capabilities such as context length, tool calling, and structured output only prove “supported,” not “sufficient.” A model that claims very long context support still has to be measured on how well it actually recalls and integrates facts across a long document.

So the first step in evaluation is to cut business work into decidable, reproducible task units.

2. Four Core Capability Dimensions and What to Observe

2.1 Complex Reasoning: Multiple Hops, Constraints, Verifiability

Complex reasoning is not “high scores on hard problems.” It is the absence of logical collapse under multiple simultaneous constraints. Recommended observable signals:

• Constraint satisfaction rate: the proportion of the N explicitly stated constraints (format, boundaries, prohibitions, field definitions) that are all satisfied at once. Most failures are not arithmetic errors; the model satisfies 8 constraints and misses the 9th.
• Multi-hop consistency: whether the intermediate facts a conclusion depends on are stated explicitly, and whether they contradict each other.
• Verifiability: whether the output can be judged automatically by a program or a rule. Any task decidable by tests, schema validation, or SQL execution results should be turned into automatic judgment first, to avoid the noise of subjective scoring.

2.2 Code and Software Engineering: From Completion to Repository-Level Change

Code capability is layered, and evaluation should be layered too:

Tier	Task shape	Main bottleneck
L0 single point	Single-function generation, unit test completion, error explanation	Syntactic and semantic correctness
L1 multi-step	Defect localization, small cross-file edits, dependency upgrades	Retrieval and context pruning
L2 repository-level	Modular refactoring, bulk interface migration, enforcing architectural constraints	Context organization, state retention, regression verification

L2 is where models separate most clearly, because it simultaneously tests retrieval quality, the ability to keep changes minimal, and the discipline of “not breaking existing behavior.” A useful scoring combination is: repository snapshot + hidden tests + change-scope scoring (is the diff minimal, did it touch unrelated files).

2.3 Long Documents and Knowledge Work: Recall, Integration, Attribution

The real metric for long context is position-independent recall and cross-section integration. The evaluation set should deliberately place key facts at the beginning, middle, and end of a document, and require outputs to carry reference locations (section numbers or chunk IDs). For structured outputs (requirement breakdown, review comments, metric definitions), enforce schema validation:

{
  "type": "object",
  "required": ["requirement_id", "acceptance_criteria", "risks"],
  "properties": {
    "requirement_id": {"type": "string"},
    "acceptance_criteria": {"type": "array", "minItems": 1},
    "risks": {"type": "array"}
  }
}

A failed validation counts as a failure. These hard failures reveal real usability far better than style scoring.

2.4 Long-Horizon Tasks: Planning, Tool Calls, Error Recovery

For long tasks (agentic execution), the object of evaluation is the entire trajectory, not the final answer. Key signals: task completion rate, average step count, share of ineffective tool calls, error recovery rate (can the model self-correct after a failed tool return), number of context overflows, and per-task token and cost. A model that can self-correct within thirty steps is usually worth more than a “lucky” model that gets it right in five, because the failure cost of the latter is uncontrollable.

2.5 Stability and Engineering Metrics: Variance, Latency, Cost, Observability

Run the same task k times and record first-attempt pass rate, pass-within-k rate, and pass-rate variance. Also track time to first token, end-to-end latency percentiles (P50/P95), per-task cost, and rate-limit hits. A high-variance model is often more expensive in production than a lower-mean one, because retries and human fallback costs do not appear on the model bill.

3. Building a Reproducible Internal Benchmark Set

The internal benchmark set is the only trustworthy basis for decisions. Build it as follows:

1. Task tiers and proportions. Set the number of L0/L1/L2 tasks according to real business share. For a team whose main work is code, push repository-level (L2) tasks above roughly thirty percent; otherwise evaluation results decouple from real returns.
2. Corpus sources and redaction. Draw from historical production tickets, review records, code repositories, and documents; redact and freeze snapshots. Do not casually swap samples during iteration, or historical data stops being comparable.
3. A three-part scorer stack: automatic judgment (tests / rules / schema) → model judging (for style and completeness only, with fixed judging prompts and scoring anchors) → human spot checks (sampled by ratio, to calibrate the first two).
4. Contamination control and rotation. Keep the core set private and versioned, rotate a portion of samples periodically, and run perturbation tests on prompt templates (paraphrasing, order shuffling). If scores drop sharply, the model is memorizing templates rather than understanding the task.
5. Regression as routine. After every model swap, prompt change, or retrieval strategy change, run the same benchmark set to build comparable curves.

A sketch of the evaluation pipeline:

TASK_SET = load_tasks("internal-bench/v3")   # private benchmark set, versioned
def evaluate(route, task, k=3):
    runs = []
    for _ in range(k):
        trace = runner.execute(task, model=route, tools=task.allowed_tools)
        runs.append(scorer.score(task, trace))   # automatic first, then human sampling
    return runs
report = []
for task in TASK_SET:
    runs = evaluate(router.pick(task.profile), task, k=3)
    report.append({
        "task_id": task.id,
        "tier": task.tier,                          # L0 / L1 / L2
        "pass_at_k": any(r.passed for r in runs),
        "first_try_pass": runs[0].passed,
        "p95_latency_ms": percentile([r.latency_ms for r in runs], 95),
        "mean_cost": mean(r.cost for r in runs),
        "failure_classes": classify_failures(runs),  # retrieval gap / missed constraint / regression / timeout
    })

Failure classification is usually more valuable than the total score: a total of 0.82 helps no decision, while “62% of failures come from missing cross-file retrieval” points straight at improving repository indexing and retrieval strategy rather than swapping the model.

4. Statistical Discipline: Don’t Let Averages Fool You

• Report two lines: pass-within-k rate and first-attempt pass rate. The gap between them is the retry cost.
• Give confidence intervals on small samples. A difference of a few dozen samples inside the noise band is not a basis for selection.
• Describe latency with percentiles, not means; describe quality structure with a distribution of error classes.
• Fix and record sampling parameters that affect randomness, so results stay comparable across versions.

5. Scenario-Specific Practices

Code production. Invest heavily in context engineering: repository indexing and symbol retrieval, pruning relevant files along the call graph, putting only necessary snippets into the window. Constrain changes (no unrelated refactoring, tests required, a whitelist of files that may be touched) and use hidden tests plus static checks as gates. Here, the marginal gain from more model capability is often smaller than the gain from better retrieval and gating.

Long documents and knowledge work. Use a chunk → summarize → attributed-output pipeline, with outputs carrying chunk locations; apply strict schema validation with retry on failure for structured outputs. For very long material, compress at the table-of-contents level first, then read section by section, instead of dumping everything into the window at once and letting middle-section recall decay.

Complex logic and requirements analysis. Require the model to list constraints and assumptions explicitly, produce several compared options (benefit / risk / cost), then enter a critique stage (the same or a different model hunts for errors in the draft), and finish with human confirmation. This “generate — self-check — critique” structure usually pays off more on complex requirements than switching to a stronger base model.

Long-horizon tasks. Externalize state, set checkpoints, restrict tool permissions and step counts, and define verifiable completion conditions for each subgoal:

def run_long_task(goal, max_steps=40):
    state = {"plan": decompose(goal), "step": 0, "green": None}
    while state["step"] < max_steps and not is_done(state):
        step = next_action(state)
        result = tool_call(step.tool, step.args)      # least-privilege tools
        state["green"] = snapshot(state) if verify(step, result) else state["green"]
        if not verify(step, result):
            state = rollback(state["green"])          # roll back to last green state on failure
        state["step"] += 1
    return summarize(state, evidence=state["green"])

Externalizing state makes long tasks recoverable and auditable; the rollback mechanism contains the risk of “getting worse with every change” to a single step.

6. Enterprise-Side Engineering Requirements

Turning model capability into a stable service needs general-purpose infrastructure: a unified model service gateway handling routing, fallback, rate limiting, and canary releases; a message queue for asynchronous long tasks and batch processing; object storage for evaluation corpora, traces, and artifacts; a container platform for evaluation and batch inference jobs; logging plus monitoring and alerting covering latency, failure rate, cost, and content-safety signals; and a key and permission system implementing least privilege with end-to-end audit trails. Baidu AI Cloud can support enterprises on such general cloud capabilities as model service access, storage, containers, logging, and monitoring, turning evaluation and regression into a routine engineering process rather than a one-off selection exercise.

Cost governance is also an engineering concern: tiered routing (simple tasks to small models, complex tasks to strong models), result caching and semantic caching, moving batch work offline, and attributing token cost per business line with quotas and alerts.

7. Anti-Patterns to Avoid

• Using one general leaderboard to select models for the whole company;
• Evaluating only the final answer and never looking at the execution trajectory;
• Hard-coding prompts, swapping the model, and concluding that “the new model is worse”;
• Never rotating the benchmark set, so metrics drift upward;
• Treating long context as equivalent to long-document capability;
• Lacking regression, so a silent model-side upgrade degrades production behavior with no warning.

8. Rollout Roadmap

1. Two weeks: sample 100–200 real tasks from production history, label them across the three tiers, and wire up automatic judgment.
2. Within one month: get the evaluation pipeline running and produce four tables: first-attempt pass rate, pass-within-k rate, latency percentiles, and failure classification.
3. Next: establish tiered routing and gates (tests + schema + static checks), making model choice a configuration item rather than hard-coded logic.
4. Ongoing: trigger regression on every model, prompt, or retrieval strategy change; compare on the same curve and decide from data, not impressions.

The conclusion is straightforward: there is no universally strongest model, only the combination that best matches your task distribution, verification means, and cost constraints. Rewrite “who is stronger” into “on my task set, by my criteria, how often does it pass stably,” and model selection moves from argument to reproducible engineering conclusion.