After the Benchmark: AI Competition Shifts to Unit Cost and Long-Horizon Completion

1. Introduction: The Benchmark Is Only the Starting Gun

Recent public developments are dense:

• A model from a leading vendor, built on the same base model, topped an independent third-party evaluation organization’s overall capability index with 65 points, about 8% ahead of the runner-up.
• A vendor expanded its in-house accelerator product line into four system configurations within a year.
• A vendor released a small-size model with 1 million-token context.
• A vulnerability was disclosed involving an AI coding tool writing credentials in plaintext into a project configuration file.
• An open tactile foundation model was released.
• An automated research system was introduced.

Viewed side by side, these developments point to a main thread with more engineering value than “who is stronger”: competition is shifting from single-turn benchmark scores to cost per task and long-horizon completion.

Supporting evidence comes from a set of comparison data:

• A higher-tier model built on the same base model completed only about one quarter of a certain class of long-horizon formal tasks within an 18-hour budget.
• In another setting, 1,000 agents working in parallel compressed an 11-day formal proof effort into 30 hours, while producing only one third of the original output tokens.

This is not merely “faster.” It is the same compute being organized more effectively. The latter is the reusable engineering conclusion.

2. The Evaluation Coordinate System Is Moving: Three Metrics, Three Blind Spots

A single overall capability index is no longer sufficient, but it still has value: it is a quick probe of the upper bound of capability. What needs to be added are two more measuring sticks.

Metric	What it measures	Main blind spot	Decision it supports
Overall capability score	Upper-bound capability on single-turn, short-horizon tasks	Long-horizon stability, failure recovery, real cost	Initial model screening
Cost per task	Total cost of completing one acceptable task	Long-tail quality distribution	Capacity planning and pricing
Long-horizon completion	Acceptance pass rate under a given time/token budget	Requires building a task set	Whether critical workflows can be handed to agents

A 5-point gap between capability scores is far less meaningful in real business than a 5-percentage-point gap in pass rate. The former affects “what can be done.” The latter directly determines “whether a human needs to watch over it.”

Actionable metric definitions — recommended for inclusion in team standards:

• First-pass acceptance rate: the proportion of tasks that pass acceptance without manual modification.
• Interruption recovery rate: the proportion of tasks that can continue from a checkpoint and eventually complete after context overflow or process termination.
• Effective token rate: tokens that enter the final artifact divided by total output tokens. For formal-proof tasks, this ratio is usually very low because many candidates are rejected by the verifier. Therefore, “fewer output tokens” often equals “better retrieval and pruning.”

3. Cost per Task: Break the Bill into Readable Components

Do not focus only on unit price. The real cost formula is:

Total task cost ≈ model invocation fees + retry amplification + human review + infrastructure
          = Σ(token_in × p_in + token_out × p_out)
            + (1 / first-pass acceptance rate − 1) × single invocation cost
            + review hours × labor unit price
            + runtime × accelerator-card hourly price

Retry amplification is the most easily underestimated part. Raising the first-pass acceptance rate from 40% to 60% reduces the average number of attempts from 2.5 to 1.67. Cost drops by about one third without changing the unit price at all.

This explains the pricing strategy of small models. At a price of $0.10 / $0.50 per million input / output tokens, a small model scored 36 points in evaluation, comparable to mainstream small models, while its cost per task is about one third. It is not competing for the capability ceiling. It is taking over “repetitive but high-volume” tasks: tool calling, code subtasks, and long-document extraction. This is not a downgrade. It is routing.

Production Form of Layered Routing

# Pseudocode: three-tier routing by task risk and difficulty
def route(task, ctx):
    if task.verifiable and task.est_tokens < 200_000:
        return SMALL_MODEL          # Low cost; suitable for large-scale parallel sampling
    if task.risk == "high" or task.requires_reasoning_depth:
        return LARGE_MODEL          # Used only for planning, review, and final synthesis
    return MID_MODEL
# Key: expensive models should appear at decision points, not in execution loops
plan = route(planning_task, ctx)(task)          # once
edits = parallel_map(route(small_task, ctx), plan.steps, concurrency=256)  # many times
final = route(review_task, ctx)(review(edits))  # once

Three supporting cost-reduction techniques have stable effects:

• Prefix caching: keep system prompts, repository structure, and specification documents fixed at the head of the prompt. Cached input tokens are usually billed at a significantly lower rate.
• Batch processing: send offline tasks through asynchronous batch interfaces, where unit prices usually have clear discounts.
• Semantic caching: reuse conclusions for similar requests. A similarity threshold is needed to prevent cross-contamination.

4. Why Long-Horizon Tasks Fail: Three Failure Points

Long-run failures usually occur in the same places, not because the model is “not smart enough.”

1. Context walls: after tens of thousands of tokens of repositories, logs, and papers are stacked together, key information is drowned out, and the model starts repeating completed work. The response is layered memory: working memory for the original text of the current segment, episodic memory for structured conclusion cards, and semantic memory for definitions, interfaces, and theorem libraries. Retrieve on demand instead of stuffing everything in.

2. State is not persisted: if the process crashes, work starts from scratch and the budget is wasted. The response is explicit checkpoints: the task tree, verified conclusions, and failure reasons should be written into external storage — object storage plus structured logs — so execution can resume after restart.

3. Verification is missing: in long chains of reasoning, one wrong step invalidates the whole chain. The response is make verification cheap.

Why Formal Tasks Can Speed Up by an Order of Magnitude

Mathematical-proof tasks have a special property: verification cost is far lower than generation cost. This property makes large-scale parallelism extremely economical. Parallel sampling generates many candidate proofs, a verifier cheaply eliminates incorrect ones, and only the survivors are passed to expensive models.

# Pseudocode: parallel sampling and early-stop pruning for verifiable tasks
frontier = [initial_goal]
while not verifier.check(frontier, target):
    candidates = parallel_map(worker, expand(frontier), concurrency=N)
    survivors = [c for c in candidates if verifier.check_step(c)]   # cheap rejection
    frontier = dedup_and_rank(survivors, by=proof_state_signature)  # state deduplication
    if len(frontier) > cap:
        frontier = ranker.keep_top(frontier, cap)                   # budget-constrained pruning

Three engineering points:

• State-signature deduplication: avoid thousands of agents repeatedly walking the same dead end.
• Verifier first: do not make expensive models judge correctness.
• Budget circuit breaker: downgrade or hand over to a human when the threshold is exceeded.

The same pattern applies to code repair with tests as verifiers, data cleaning with rule validation, and document generation with link and citation validation.

5. The Foundation for Long-Horizon Tasks: Long Context and Confidential Inference

Long context is not free. It consumes three things at the same time: compute, on-chip SRAM for attention intermediate results, and HBM bandwidth for repeated reads and writes of the KV cache. This is the core logic behind this round of hardware expansion.

One vendor’s two new system configurations are relevant:

• One packages two accelerators into the same package, a dual-die package, doubling compute, on-chip SRAM, and HBM bandwidth at the same time, directly targeting very large contexts.
• The other is aimed at security and confidential inference, keeping data and weights encrypted and isolated while in use.

The wording matters: both are system configurations, not new chip architectures. This reflects a pragmatic industry pace. Architecture iteration cycles are long, while adjustments to packaging combinations, interconnect topology, and memory configuration can be delivered multiple times within a year. Fixing the bottleneck first is the more realistic path.

Implications for platform teams:

• Long context requires budget discipline. The longer the context, the more likely latency and cost grow superlinearly. Filter content by “whether it affects the current decision” instead of blindly filling the context window.
• The KV cache is a cost center. Reusable prefixes should be stably ordered: system prompt → specifications → code skeleton → change set, avoiding cold-start warming every time.
• Confidential inference solves compliance, not performance. When inference workloads involve undisclosed code, medical data, or financial data, “encryption in use + remote attestation + customer-managed keys” is usually more practical than building an isolated environment. The deployment form is a combination of general cloud capabilities: a container platform hosts inference services, a key management service stores master keys, object storage handles cold data, and logging, monitoring, and alerting handle audit and anomaly detection.

6. A Security Incident That Must Be Taken Seriously: Why an Approved Command Wrote Keys to Disk

The disclosed vulnerability chain is typical and worth a full review:

1. An AI coding tool, to support a “pre-approved commands” experience, wrote the raw text of approved commands into a settings file inside the project.
2. If the command contained export DB_PASSWORD=... or curl -H "Authorization: Bearer ...", credentials were persisted to disk along with the command.
3. The settings file was committed to version control.
4. Even if the file was later deleted, plaintext remained in historical commits.
5. More than 100,000 such files were already present on public hosting platforms, and about 1,100 contained keys. After merging Git history with shared settings, researchers identified 2,539 real credentials across 1,994 code repositories and 63 organizations.

The essence of the problem is not tool negligence. It is the structural conflict between the convenience feature “pre-approval” and the basic principle “secrets must not enter disk.” Any design that persists command-line text as-is is inherently dangerous.

Four Engineering Defenses

Defense 1: Whitelist commands as structure, not as text. Store only a structured description of the command. Never store the raw string.

// Counter-example: plaintext in the database
{ "command": "psql $URL -c \"select 1\" --password=s3cr3t" }
// Correct example: structure plus credential references
{
  "tool": "database_query",
  "args": { "dsn_ref": "secret://prod/pg-main", "statement_ref": "queries/health.sql" },
  "credential_mode": "short_lived_token",
  "ttl_seconds": 900
}

Defense 2: Short-lived credentials. Replace long-lived keys with short-lived tokens:

• In the cloud, use temporary security credentials that expire automatically and are issued per session.
• In CI environments, use OIDC federation to obtain short-lived tokens.
• For databases, use IAM integration or passwords rotated on a minute-level schedule.

As long as a credential’s lifetime is shorter than the time required to discover a leak, the harm from leakage is greatly reduced.

Defense 3: Intercept before commit. Scan locally and in the pipeline. Block on a hit.

# Key excerpt from .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: secret-scan
        name: secret-scan
        entry: ./scripts/scan-secrets.sh
        language: script
        files: \.(json|yaml|yml|toml|env|sh|md)$
        pass_filenames: true

# Equivalent logic for scripts/scan-secrets.sh: high-entropy strings plus known patterns
import re, math, sys
PATTERNS = [r"(?i)(password|passwd|secret|token|api[_-]?key)\s*[:=]",
            r"AKIA[0-9A-Z]{16}", r"gh[pousr]_[A-Za-z0-9]{20,}",
            r"-----BEGIN (RSA |EC |OPENSSH )?PRIVATE KEY-----"]
def entropy(s: str) -> float:
    if not s: return 0.0
    freq = {c: s.count(c) / len(s) for c in set(s)}
    return -sum(p * math.log2(p) for p in freq.values())
def is_secret(line: str) -> bool:
    if any(re.search(p, line) for p in PATTERNS):
        return True
    for tok in re.findall(r"[A-Za-z0-9_\-]{20,}", line):
        if entropy(tok) > 4.0 and not tok.startswith(("sha256:", "urn:")):
            return True
    return False
bad = [f"{i}:{l.strip()}" for i, l in enumerate(open(sys.argv[1]), 1)
       if is_secret(l)]
if bad:
    print("Suspected credentials detected. Commit aborted:\n" + "\n".join(bad)); sys.exit(1)

Defense 4: Rotate on discovery, not delete on discovery. Once a key is confirmed to have entered history, assume it is already leaked. Immediately revoke and rotate it, audit all call records for that key, and then clean history. Deleting the file solves nothing.

Supporting management rules are equally important:

• Put the set of commands an agent can execute into a least-privilege list.
• Block out-of-band communication, such as webhooks and reverse-shell-like instructions, in the sandbox by default.
• Keep auditable logs for every tool call made by an agent.
• Add explicit exclusion rules for project configuration directories, forbidding .env files, key files, and local settings from entering version control.

7. Embodied Intelligence: The Place of a Unified Tactile Representation Layer

Tactile perception has long been the most overlooked part of robot perception. Vision and language already have large-scale pre-training paradigms, while tactile perception remains stuck in sensor fragmentation. Piezoresistive, magnetic, and vision-based tactile sensors differ completely in output units, sampling rates, and spatial resolution. As a result, data cannot be pooled, and models cannot be reused across devices.

The value of an open tactile foundation model lies in inserting a unified tactile representation between the sensor abstraction layer and the action model. Public information indicates that its pre-training covers more than 460,000 interaction segments, about 1.33 billion frames, and three sensing modalities. It was evaluated on 26 real-robot tasks.

From an engineering perspective, this layer must solve four problems:

1. Alignment: timestamp synchronization and spatial coordinate calibration across different sensors are prerequisites for cross-modal pre-training. They usually take more time than model structure.
2. Pre-training objectives: mainstream approaches are self-supervised, including masked segment reconstruction, cross-modal alignment between tactile, visual, and proprioceptive signals, and prediction of future contact forces.
3. Downstream adaptation: grasp force control, slip detection, assembly alignment, and material recognition should each fine-tune lightweight heads rather than retraining the backbone.
4. Real-robot evaluation protocol: it must include out-of-distribution objects, occlusion and lighting changes, and sensor drift. Otherwise, simulation metrics have little reference value.

The pragmatic deployment point is to treat tactile representation as a pluggable module. Upstream connects to a sensor abstraction layer that uniformly defines sampling rate, units, and coordinate system. Downstream connects to the policy network. Changing sensors then requires only driver and calibration changes, not model changes.

8. Automated Research Systems: Two-Layer Loop and Role Division

One automated research system organizes work with a two-layer loop:

• The outer layer handles research direction selection and budget allocation.
• The inner layer handles experiment design, execution, and analysis.

Five roles advance the work in parallel: topic proposal, implementation, execution, review, and archiving. According to the official description, the system exceeded previously publicly verified records on three metrics for two public tasks, and the optimization was applied to the inference chain of the vendor’s in-house reasoning model.

This structure is structurally similar to the orchestration in Section 4 and can be reused for engineering R&D:

• The outer layer is combinatorial optimization: under a finite budget, candidate hypotheses are ranked by “expected benefit / verification cost.” Resources are allocated round by round, and a portion of the budget is reserved for exploration to avoid premature convergence to a local optimum.
• The inner layer is an experimental loop: change one place → run once → record → analyze → decide whether to keep or roll back.
• The essence of role division is separation of duties: the implementer should not be the evaluator. Letting the review role independently hold the retained set and acceptance criteria is the only effective way to prevent “grading one’s own work as passing.”

The most dangerous aspect of automated research is metric overfitting: the system instinctively optimizes the number you are measuring. Three defenses:

• Pre-register metrics: fix the evaluation criteria and primary metric before the experiment.
• Hold out a validation set: the evaluation set should be read-only and invisible to the agent.
• Record all failed experiments: negative results are data too. Otherwise, the system repeatedly makes the same mistake.

Skepticism is also necessary. There is a gap between vendor self-reported claims of “beating public records” and third-party reproducible verification. Before deployment, request reproducible materials: data versions, environment images, random seeds, and evaluation scripts.

9. Action Checklist for Technical Teams

• Build a cost dashboard: track tokens, duration, first-pass acceptance rate, and human intervention rate by task type. Measure before optimizing.
• Set a routing policy: small models handle high-frequency execution; expensive models appear only at planning and review nodes. Stable prefixes go through cache; offline batch work goes through asynchronous batch interfaces.
• Add checkpoints to long tasks: persist the task tree, verified conclusions, and failure reasons externally, so processes can restart and resume at any time.
• Treat verifiers as first-class citizens: write tests or validation rules first, then let the model act. Cheap verification is the prerequisite for parallelization.
• Start credential governance immediately: replace long-lived keys with short-lived tokens, centralize key management and rotate regularly, scan before commit, run a second scan in the pipeline, and revoke immediately once a key is found to have entered disk.
• Review persistence behavior in tools: check local settings files, cache directories, and logs. Ensure raw command text does not contain credentials, and add sensitive paths to version-control exclusion rules.
• Load long context on demand: use stable prefixes plus layered retrieval. Avoid the cost and attention decay caused by full stuffing.
• Prefer confidential inference for sensitive data: encryption in use + remote attestation + customer-managed keys is less effort than building an isolated environment.
• Do data alignment first for embodied scenarios: sensor abstraction layer, time synchronization, and coordinate calibration should come before model selection.
• Enforce separation of duties in automated research: separate implementation from review, pre-register metrics, and make the evaluation set read-only for agents.

10. Conclusion

65 is a beautiful number, but it measures what a model can do. Enterprises care about two other things: how much one task costs, and whether the system can run continuously for 30 hours with acceptable results.

The first is determined by routing strategy, caching, and verification cost. The second is determined by checkpoints, verifiers, and failure-recovery capability.

The practical implication of this main thread is clear: spending the budget on verification and orchestration usually yields more than spending it on a larger model. The difference between advancing only one quarter of the tasks in 18 hours and completing 11 days’ worth of work in 30 hours is not model intelligence. It is whether there is a cheap, reliable, parallelizable verifier, and an engineering structure that allows failure and retry.