Same Base Model, Different Tiers: Evaluating Long-Horizon LLM Capability and Cost-Effectiveness

1. Why long-horizon tasks rewrite delivery timelines

In software engineering and research assistance, there is a class of “long-horizon tasks”: formal specification and proof, cross-module refactoring, vulnerability reproduction, complex data-pipeline construction, and de novo design of materials or protein-binding compounds. Their common characteristics are many steps, many states, seemingly simple individual steps, and errors that accumulate over time. When humans complete these tasks, they are usually measured in days or even weeks.

When a model compresses this class of tasks from “weeks” to “days”, what really changes is not a single benchmark score but the economic boundary of the task: engineering items that were previously not worth investing in become schedulable. This also creates a new judgment problem: can the capability numbers in marketing materials actually be used to decide whether a high-risk long task should be assigned to the model?

2. Key premise: capability results have ownership

A common misjudgment needs to be clarified. The industry does have practices of splitting one underlying model into multiple product tiers according to guardrail and admission rules: the base weights and inference capabilities are similar, while the differences are reflected in what is allowed, what is refused, and in which environments it can run.

This leads to two rules:

1) Fewer refusals does not mean stronger capability. A lower refusal or interception rate indicates that the admission threshold has been relaxed, not that task completion quality has improved. If the scenario you care about is already within the guardrail scope—such as high-risk operations, sensitive domains, or real-world side effects—relaxing the threshold may improve usability, but it may also widen risk exposure. High-risk tasks must be confirmed item by item according to guardrail scope, rather than based on overall interception statistics.

2) Evaluation scores cannot be transferred across tiers. High-difficulty items in official benchmarks—such as complex scenario reasoning, vulnerability reproduction, and de novo protein-binding compound design—are often tied to specific tiers. When evaluating another tier, these scores are non-transferable:

Information type	Transferable?	How to handle it
Base architecture, context capability, tool-calling paradigm	Can serve as prior knowledge	Cite it, but calibrate with your own tests
Benchmark scores for a specific tier	Not transferable	Reproduce them with your own task set
Refusal or interception statistics	Not transferable	Measure them according to your own risk profile
Unit price, quota, latency	Not transferable	Use contracts and actual stress tests

The conclusion is direct: tier differences mean evaluation must be tier-specific. Building your own evaluation pipeline is not optional; it is a prerequisite for procurement decisions.

3. Breaking “can it deliver” into measurable dimensions

Asking vaguely whether a model is “capable enough” cannot guide decisions. It is better to break this into five layers, each with observable indicators:

1. Single-step correctness: accuracy of individual operations, which determines subsequent rework.
2. State retention: after many tool calls, context compression, or session recovery, whether the model still remembers constraints, naming rules, and completed items.
3. Error self-recovery: when commands fail, dependencies conflict, or tests fail, whether it can read logs, locate the issue, and fix it, rather than repeating the same action.
4. Verifiability of deliverables: whether outputs can pass existing acceptance tools—compilation, tests, static checks, proof checkers—rather than merely “looking right”.
5. Consistency of admission for risky tasks: whether policy decisions remain stable for the same input, avoiding “allowed this time, blocked next time” jitter that can break a pipeline.

Items 3 and 5 are the easiest to ignore, but they have the largest impact on end-to-end success rate for long tasks: one mid-task interruption can invalidate all prior investment.

4. Evaluation set and automated judgment: making long tasks scoreable

Long tasks cannot be scored manually item by item; the acceptance oracle must be engineered.

Task layering can be divided into three levels by time span and state complexity:

• L1 single-step: judged with one tool call, such as generating a single file or fixing a clearly defined bug.
• L2 multi-step: 5–30 steps, requiring reading and writing repositories and self-testing.
• L3 long-horizon: spanning sessions, requiring checkpoint resumption, with output judged by external validators (build + full tests + proof checking).

Example case structure:

CASE = {
    "id": "L3-refactor-017",
    "tier": "L3",                        # Determines budget and acceptance rigor
    "goal": "Perform a behavior-preserving refactor in an existing codebase",
    "budget": {"max_steps": 120, "wall_clock_s": 3600},
    "oracle": ["build", "unit_tests", "linter"],   # Fully automated acceptance
    "partial": ["compilable", "all unit tests green", "no new warnings"],  # Supports partial scoring
    "risk": "medium",                    # Expected admission tier to trigger
    "resume_from": "checkpoint://step-40"        # Supports resuming from a checkpoint
}

Judgment principles:

• Use only objective pass/fail results as the primary metric, not model self-assessment or manual impressions.
• Keep partial scoring to distinguish “complete failure” from “nearly usable” and avoid compressing long tasks into a 0/1 score that loses information.
• Fix the randomness policy for each case—temperature, retry count, tool whitelist—otherwise cross-tier comparison is not valid.

5. Admission and guardrails: false refusals are more damaging in long tasks

In a short task, if the request is refused, you can ask again. In a long task, being refused at step 80 means the entire run is wasted. Therefore, interceptions should be treated as a class of failure that can be classified, counted, and mitigated:

• Interception location: input side, output side, tool-call side, environment-action side.
• Interception nature: real-risk interception vs. false refusal.
• Mitigation methods: action-level authorization, sandbox isolation, idempotent replay, checkpoint recovery, and fallback to human intervention.

Example of action-level authorization:

POLICY = [
    {"action": "read_repo",       "risk": "low",      "approval": "none"},
    {"action": "run_tests",       "risk": "low",      "approval": "none"},
    {"action": "write_file",      "risk": "medium",   "approval": "auto+audit"},
    {"action": "push_branch",     "risk": "high",     "approval": "human"},
    {"action": "delete_resource", "risk": "critical", "approval": "human"},
]

Principle: least privilege + rollbackable side effects + human confirmation gates for high-risk actions. This way, even if a tier has looser admission rules, the risk is still constrained by your own policy layer rather than relying entirely on model-side guardrails.

6. Cost model: unit price is not effective cost

Cost-effectiveness should be calculated by the cost per usable deliverable, not by price per million tokens. A failed long task incurs sunk costs such as compute, retries, human review, and waiting time.

def effective_cost(run):
    passed = sum(1 for r in run if r.oracle_passed)
    return {
        "pass_rate":        passed / len(run),
        "first_pass_rate":  sum(1 for r in run if r.first_try_passed) / len(run),
        "cost_per_delivery": run.total_cost / max(passed, 1),
        "human_min_per_delivery": run.human_minutes / max(passed, 1),
        "mid_task_block_rate": sum(1 for r in run if r.blocked_midway) / len(run),
        "resume_success_rate": sum(1 for r in run if r.resumed_ok) / len(run),
    }

A common result is that a tier with a lower unit price may have a higher cost_per_delivery because of more rework and human intervention; meanwhile, a tier with looser admission may require additional audit effort in compliance-sensitive scenarios. These hidden costs become visible only through actual measurement.

7. Production pipeline: concurrency, isolation, and observability

Make evaluation a continuous pipeline rather than a one-off PoC:

1. Environment isolation: give each case an independent container or independent workspace to avoid state pollution; perform all destructive actions inside a sandbox.
2. Concurrent orchestration: L3 cases take a long time, so use task queues to smooth peaks and container platforms to scale elastically, avoiding evaluation work that squeezes production resources.
3. Data retention: store case corpora, trajectory logs, and acceptance results in object storage buckets to support regression comparison.
4. Observability: aggregate trajectories with a log service, and use monitoring alerts to track drift in pass rate, average steps, interception rate, and cost per deliverable, forming version-level dashboards.
5. Regression baseline: every time the base model or tier policy changes, run a full baseline and compare only the two-dimensional point of “pass rate vs. effective cost”, rather than looking only at a single benchmark score.

When building this pipeline on Baidu AI Cloud, the above capabilities can map to general product categories such as object storage, message queue, container platform, log service, and monitoring alerts. Evaluation and production can share the same operational system, so there is no need to build separate infrastructure solely for validation.

8. Decision checklist

Before handing a long task or high-risk task to a specific model tier, confirm the following item by item:

• Do the capability items I care about have actual measurement data for that tier itself, rather than borrowed scores from other tiers?
• Is the acceptance oracle for the target task automated and able to provide an objective pass/fail judgment?
• Have you measured the mid-task interception rate and checkpoint recovery success rate?
• Is the decision based on cost_per_delivery and human minutes, rather than unit price?
• Do high-risk actions already have action-level authorization and human confirmation gates?
• For every version or policy change, is there a regression baseline available for comparison?

One-sentence summary: the same base model does not mean the same capability surface, and capability scores do not equal cost-effectiveness. Only when “can it deliver” becomes an automatically judgeable metric and “cost-effectiveness” becomes the total cost per usable deliverable can the evaluation conclusion truly be used for scheduling and procurement.