LLM Cost-Effectiveness Evaluation and the Capability Boundaries of General-Purpose Models

1. First, correct a common evaluation misconception: a one-dimensional “kill line” systematically misleads model selection

The industry often uses a “kill line” to summarize a model’s capability range: any model with lower capability and higher price is considered outside the acceptable range. This view implicitly assumes that capabilities are comparable, prices are comparable, and tasks are comparable. In real engineering, however, these assumptions often do not hold.

Common sources of distortion include the following:

1. Benchmark distributions do not match business traffic. Public benchmarks have fixed task distributions, while real enterprise traffic is long-tailed. A model that leads by 5 percentage points on a math reasoning benchmark may show no perceptible difference on high-frequency short-text tasks such as ticket classification.

2. Average latency hides tail latency. Published throughput data is usually measured under ideal concurrency, while the real user experience is determined by P95/P99. This is especially obvious in long-output scenarios. A 30% improvement in generation speed can be largely offset if queueing time worsens when concurrency saturates.

3. Unit price is decoupled from the real bill. Price per million tokens is only one price dimension. Real cost also depends on retry rate, fallback calls after failures, context expansion ratio, and the extra length of system prompts added to control output.

4. Capability strength is not the same as usability. Whether a model can reliably produce schema-compliant structured output, maintain stable recall under long context, and behave predictably when stopping—these engineering capabilities often matter more than benchmark scores.

A more reliable approach is therefore to move from a one-dimensional ranking to a multi-dimensional joint evaluation, and explicitly define the constraints that actually matter.

2. Quantifying cost-effectiveness: from unit price to cost per effective output

It is useful to introduce a computable metric: the cost paid to obtain one business-effective output. The basic form is:

def cost_per_effective_output(
    in_tokens: int,
    out_tokens: int,
    price_in_per_mtok: float,   # input price per million tokens
    price_out_per_mtok: float,  # output price per million tokens
    success_rate: float,        # fraction of calls that pass validation on the first try
    retry_rounds: int = 1,      # average number of retry rounds
) -> float:
    """Estimate the cost of one successful business outcome (illustrative; excludes queueing and network overhead)."""
    if success_rate <= 0:
        return float("inf")
    cost_per_call = (
        in_tokens * price_in_per_mtok
        + out_tokens * price_out_per_mtok
    ) / 1_000_000
    return cost_per_call * retry_rounds / success_rate

This formula reveals several counterintuitive conclusions:

• Success rate is a multiplier, not a bonus. Improving success rate from 90% to 99% directly reduces effective cost by about 9%. For agent scenarios that require multiple tool calls, a small improvement in per-call success rate compounds along the call chain.

• The value of a lower input price depends on the context expansion coefficient. If input tokens are 20 times the number of output tokens, then the real bill difference between “input price down 20%, output price down 0%” and “input price down 0%, output price down 20%” is about 20:1, not the intuitive 1:1.

• Output price is the main cost driver for long-text tasks. When generating long reports, long code, or complete test suites, the weight of output unit price significantly exceeds that of input unit price.

On this basis, three engineering constraints must be added to form a deployable decision framework: a per-call latency limit, the effective working area size of the context window rather than its nominal value, and reliable structured-output capability.

3. Long context and maximum output: real constraints beyond parameters

Current-generation flagship models commonly advertise context windows in the million-token range and maximum outputs in the hundred-thousand-token range. These numbers have limited meaning on their own. What matters are the following engineering implications.

A context window is capacity, not a working area. Putting 1 million tokens into context is not the same as effectively using 1 million tokens. The real quality of long-context usage depends on “needle-in-a-haystack” information retrieval and whether instruction consistency can survive across multiple rounds of tool results. A safer engineering approach is to treat a long window as a cache and context-reuse layer, rather than as unlimited memory:

• Write retrieval results, code snippets, and historical decisions into context to avoid repeated retrieval.
• Periodically perform context compression: keep conclusions and key constraints, and discard intermediate reasoning and redundant logs.
• For content that must be exact, such as API definitions, field names, and version numbers, verify against an external authoritative source instead of relying on model memory.

Maximum output length determines the upper bound of task granularity. The significance of increasing the output limit from the 32k level to the 128k level is not that the model can write longer code in one pass, but that a single atomic task can carry a larger deliverable: generate a complete module and its unit tests, generate a database change with a full migration script, or generate a structurally complete configuration template. The larger the atomic task, the less state needs to be passed across tasks, and the lower the overall failure rate.

The following code shows how structured constraints make long output verifiable, which is the key to making long output deployable:

from typing import Literal
class CodeChange:
    files: list[str]
    diff: str
    tests_added: int
    risk_level: Literal["low", "medium", "high"]
    rollback_plan: str
    must_review_by_human: bool
# A change is considered failed if schema validation fails, and it is counted toward retry_rounds.
def apply_change(change: CodeChange) -> bool:
    if change.must_review_by_human or change.risk_level == "high":
        return False          # route to the human gate
    return run_ci(change.diff)  # CI must pass before the change is considered successful

4. General-purpose models versus specialized models

One of the most notable observations from the original discussion is this: on a public benchmark oriented toward the physical world or robotics, general-purpose flagship models have approached specialized agents optimized specifically for leaderboard performance. This leads to a commonly misunderstood conclusion: “general-purpose models will sweep specialized models.” A more precise formulation is:

• For data-intensive, semantically complex tasks that require cross-domain transfer, the advantage of general-purpose models keeps expanding. The reason is that these tasks do not rely on specific priors but on strong generalizable representation capabilities. Specialized models depend on hand-crafted features and narrow-domain data; once the task definition changes, their gains decay quickly.

• For data-sparse tasks with clear physical constraints, expensive feedback signals, or extremely delayed feedback, specialized solutions still have value. They encode domain priors into the model or environment, trading priors for sample efficiency.

• The sustainable form of specialized solutions is shifting: from “training a specialized small model” to “general-purpose large model + specialized tool layer / simulation environment / reward function.” The moat is moving from model weights to data loops and environmental fidelity.

For enterprises, this means that the economics of self-built specialized models is declining, while the economics of evaluation systems, data loops, tool chains, and domain-knowledge accumulation is rising. This is also why many teams should shift their investment focus from “training models” to “building evaluations and environments.”

5. From conversation to engineering: long-horizon multi-agent tasks

The original material mentions an internal experiment: starting from an empty directory and developing a prototype operating system, the model coordinated more than 1,000 sub-agents over eight consecutive days of coding, testing, and code review. The final artifact could boot on a laptop and included its own kernel, disk and network drivers, and filesystem.

Beyond the specific numbers, such experiments reveal the organization method of long-horizon autonomous tasks. The reusable engineering structure can be summarized as follows:

1. Task decomposition must have stable contracts. An open-ended goal is first decomposed into independently verifiable subtasks, such as kernel boot, memory management, drivers, filesystem, and build system. The input/output contract of each subtask must be determined before implementation. If contracts are unclear, sub-agents will pollute each other’s state.

2. Verifiers matter more than generators. Whether the system can advance autonomously depends on whether correctness can be machine-judged. Compile success, unit-test success, and successful boot in a simulator are automatable objective signals and prerequisites for long-horizon execution. Tasks without verifiers quickly collapse to a few conversational turns.

3. The orchestration layer must be stateful and support resuming from checkpoints. More than 1,000 sub-agents means that failures and drift are inevitable. It is necessary to persist task state, support failure retries and human intervention points, and make context compression policies explicit.

4. Review and generation must be separated. Having the same context write code and review its own code systematically amplifies blind spots. An independent-context review agent or rule-based static checks are the guarantee of the quality floor.

5. Artifacts must be regressed in real environments. Passing a simulator does not mean the artifact can boot on real hardware. In enterprise deployment, real-device or pre-release environment regression should be a mandatory gate, not an optional step.

A compact orchestration skeleton is shown below. This is pseudocode intended only to express structure:

while not goal.done() and budget.remaining():
    task = planner.next_task(contracts, state)      # only claim tasks with clear contracts
    ctx = context.build(task, compress=policy)      # explicit context compression
    patch = worker.execute(task, ctx, schema=Contract)
    verdict = verifier.run(patch)                   # compile / unit tests / simulation / static checks
    if verdict.passed:
        state.commit(patch, evidence=verdict.logs)
    else:
        state.record_failure(patch, verdict.logs)   # failures are also reusable context
    reviewer.review(state.recent_commits)           # independent-context review

6. A practical model selection and evaluation workflow

The above analysis can be condensed into a workflow that can be used directly for enterprise model selection:

Step 1: Build your own evaluation set. Sample from real traffic and cover stratified categories: high-frequency simple tasks, long-tail difficult tasks, long-context tasks, and structured-output tasks. Include at least several hundred cases per category, and pre-define automatically checkable success criteria. Do not use generic benchmarks as a replacement for a business evaluation set.

Step 2: Define hard constraints. Specify the maximum latency, maximum cost per call, minimum context window, minimum output length, and minimum structured-output success rate. Models that fail the hard constraints are eliminated and do not participate in further comparison.

Step 3: Rank by cost per effective output. Among candidates that satisfy the hard constraints, calculate effective cost using the formula above, and perform sensitivity analysis: if success rate drops by 5 percentage points or output tokens increase by 30%, does the ranking remain stable? Models with unstable rankings are not suitable as long-term base models.

Step 4: Use tiered routing rather than a single model. In practice, the goal is usually not to “choose one model” but to layer: high-frequency simple tasks go to low-cost models, complex tasks escalate to flagship models, and deterministic rules handle edge cases. The routing strategy itself should be evaluated and continuously tuned.

Step 5: Industrialize the evaluation pipeline. For example, on Baidu AI Cloud, general-purpose cloud services can be used: container platforms can host concurrent load testing and replay, object storage can store evaluation sets and call samples, message queues can smooth batch evaluation scheduling, logging services can retain auditable request and response records, and monitoring/alerting can track success rate, latency percentiles, and cost drift. These capabilities are general cloud-service categories, are not tied to a specific model, and do not lock evaluation results to a single vendor.

7. Conclusion

First, do not choose models by a one-dimensional ranking. Using “cost per effective output” as the core, together with three constraints—latency, effective context, and structured-output reliability—forms a decision-ready evaluation framework.

Second, long context and long output change task granularity rather than merely meaning “more can fit.” Their real value lies in reducing cross-task state transfer and thereby systematically lowering the failure rate of complex tasks.

Third, the boundary between general-purpose models and specialized models is moving, but the conclusion is not that “specialized models have no value.” A more accurate statement is: the moat is shifting from model weights to data loops, verification environments, and domain-knowledge accumulation.

Fourth, the success of long-horizon autonomous tasks depends on verifiers and orchestration, not on the strength of a single model generation. Without automatable success criteria, even the strongest model cannot run autonomously for a long time.

By solidifying this approach into your own evaluation and routing system, you can keep cost controllable, quality predictable, and migration cost low amid rapid model iteration.