Why the Same Model Scores 99.9 With a Custom Framework but 62.7 Under a Unified Standard

1. Breaking Down the Gap: Why the Two Scores Differ by 37 Points

A vendor’s new-generation model posted strong results on several benchmarks. It scored 72.6% on a computer operation test, nearly 7 percentage points higher than the previous generation. Its score on a professional work automation test rose from 18.1% to 41.4%. On a terminal interaction benchmark, it reached 57.9%, ahead of a competing model. However, the same model scored 99.9 under a “custom-adapted framework” and only 62.7 under a “unified standard.” These numbers are not contradictory. They point to an overlooked problem: the height of an evaluation score often depends on how much scaffolding the evaluation environment provides.

To understand the gap, we first need to clarify the difference between the two types of evaluation:

• Custom-adapted framework: Tailors prompt templates, tool-calling protocols, retry policies, and result parsers to the characteristics of the model under test. It may even let the model self-correct across multiple interaction turns. The framework takes on much of the fallback engineering, so the score reflects the combined capability of “model + framework.”
• Unified standard: Uses the same prompts, tool interfaces, and scoring scripts for every participating model, with no model-specific adaptation. The score is closer to a horizontally comparable measure of the model’s bare capability.

The two scores differ by 37.2 points, which shows that this model depends heavily on framework adaptation. This is not a flaw in itself. In engineering practice, building scaffolding around a model is normal. However, when “99.9” is presented as a general capability claim, the evaluation conditions behind it need to be scrutinized.

2. Three Technical Variables Behind the Scores

1. Sensitivity to Prompts and Output Formats

The unified standard typically enforces fixed format constraints, such as requiring output in a specific JSON structure. If the model does not follow formatting instructions well, the scoring script marks the answer wrong outright. A custom framework can recover these cases through post-processing, tolerant parsing, and retries. The difference is especially visible in code generation and structured extraction tasks.

# Strict parsing under the unified standard (illustrative)
def strict_parse(output: str) -> dict:
    data = json.loads(output)  # Malformed output raises an exception and scores 0
    assert set(data.keys()) == {"action", "args"}
    return data
# Tolerant parsing under the custom framework (illustrative)
def tolerant_parse(output: str) -> dict:
    try:
        return strict_parse(output)
    except Exception:
        # Extract code blocks, fall back to regex, follow up with the model
        return repair_with_retry(output)

The same model output scores 0 under strict parsing and may score full marks under tolerant parsing. The difference lies in the evaluation pipeline, not the model.

2. Fit of the Tool-Calling Protocol

Computer operation and terminal interaction tasks depend heavily on tool calling. The unified standard defines a common set of function signatures, and the model must understand them zero-shot and fill in the parameters correctly. A custom framework can inject few-shot examples in advance, constrain parameter enumerations, and feed error messages back to the model so it can retry after a failure. The effect of multi-turn retries on final success rates is amplified in long-chain tasks.

3. Differences in Task Decomposition Granularity

Professional work automation rose from 18.1% to 41.4%. The absolute value is still low, which suggests that completion rates for long, multi-step tasks remain the main bottleneck. Custom frameworks often include a task planner that breaks a complex goal into sub-steps and calls the model step by step. The unified standard requires the model to plan and execute in a single pass. A significant part of the score gap comes from this architectural choice: who breaks down the task.

3. How to Judge the Credibility of Benchmark Scores

When you see any “XX points” capability claim, check the following:

1. Is the evaluation methodology public? Are the prompt templates, tool interfaces, and scoring scripts reproducible? Publishing scores without the pipeline offers limited reference value.
2. Does the report separate “model score” from “system score”? An honest report should give both the bare model score and the score with its paired framework, and explain what work the framework does.
3. Is the baseline measured under the same conditions? Were competitor scores obtained under the same unified standard? Comparing results across methodologies, with one side using a custom framework and the other using a unified standard, is not valid.
4. Is the task set saturated? If a benchmark has been pushed close to full marks, its ability to distinguish models has declined, and a harder replacement set is needed.
5. Are failure cases disclosed? Reporting only averages, without variance or the distribution of failure samples, can hide long-tail problems.

4. Engineering in Practice: Turning Framework Scores into Production Value

For enterprise users, what matters is not leaderboard scores but success rates that can be reliably reproduced on their own business pipelines. Build along the following path:

Step 1: Build an internal unified evaluation set. Sample tasks from real business logs, and fix the prompts and tool interfaces to form a regression baseline. Run this baseline before any model selection or framework upgrade.

Step 2: Treat scaffolding as a first-class component. Recognize that “model + framework” is the unit of delivery, and explicitly separate the architecture into layers: model access, tool adaptation, retry and fault tolerance, and result validation. The contribution of each layer can be measured independently.

Step 3: Focus on end-to-end success on long chains, not single-step accuracy. The end-to-end success rate of a multi-step task is usually the product of the success rates of each step, so improving a single step by 1% can have an outsized effect on the overall result. Prioritize retries on failure, error feedback, and task decomposition.

Step 4: Use monitoring, alerting, and logging to track production performance. Offline scores drift, and online distributions change. Track task success rate, retry count, and parse failure rate, set threshold alerts, and close the loop.

Step 5: Roll out gradually and prepare rollback plans. Validate new models or frameworks on a small share of traffic first. Compare results against baseline metrics before a full rollout, so that high evaluation scores do not turn into production failures.

5. Conclusion

The gap between 99.9 and 62.7 reflects less about whether the model’s capability is genuine and more about how transparent the evaluation methodology is. The high score shows the value of engineering support around a model. The low score marks the limits of bare capability. For developers and enterprises, the right approach is to avoid trusting any single score, build your own evaluation under consistent conditions, count framework capability as part of the delivered unit, and use online metrics as the final judge. The real worth of AGI deployment is not written on leaderboards. It shows up in business pipelines that run reliably.