Benchmarking LLM Reasoning and Coding in Practice: Methods and Lessons

1. From Capability Checklists to Engineering Performance: Why Build Your Own Benchmark

When a new model ships, official materials typically include a capability checklist: long-context input, file understanding, content generation, data analysis, general user orientation, and multiple access options such as web, API, and SDK. These descriptions help everyday users set expectations, but they are rarely sufficient for developers. What actually determines whether a model can ship in production is: on my task distribution, what are its accuracy, token consumption, and cost?

General-purpose public benchmarks have two long-standing problems. First, scores from leading models are increasingly saturated, reducing differentiation. Second, public test items risk appearing in training data, inflating results. A self-built, reproducible, and continuously updated evaluation framework therefore has more engineering value than a one-off leaderboard run. It answers three questions: Where is this generation stronger than the previous one? On which task types? And at what additional cost?

Recent observations from benchmarking a new-generation model show a clear industry trend: models are becoming more like reliable collaborators on long-horizon tasks and multi-step reasoning. If the previous generation was nearly usable on practical problems, the new generation is starting to be genuinely usable—but behind that usability lie many details worth examining.

2. Evaluation Framework Design: Tasks, Process, and Scoring Rules

2.1 Task Set Design

Benchmark tasks should focus on multi-step logical reasoning, constraint satisfaction, and multi-solution enumeration problems, with difficulty tiers:

• Basic tasks: test output stability and format compliance.
• Intermediate tasks: examine completeness of multi-step reasoning chains.
• Hard tasks: especially problems with multiple valid solutions, to test whether the model explores the solution space thoroughly.

2.2 Test Process

• Fix prompts and sampling parameters to ensure fair comparison.
• Run each task multiple times and take the median score to reduce random variance.
• Label reasoning mode (slow thinking) and non-reasoning mode (fast thinking) separately. Do not mix them in aggregate results. In practice, you can use color or tags—for example, red for reasoning mode and black for non-reasoning mode in reports.

2.3 Two Critical Scoring Rules

Output limits and truncation scoring are the most easily overlooked but most influential part of benchmark design. In one recent evaluation, the non-reasoning mode had a 15K output token limit, while the reasoning mode had an 80K limit. Any sample whose output was truncated received a score of 0. The rationale is straightforward: a truncated answer is not deliverable, and it also means cost is uncontrollable.

This rule produced two important observations:

• Adaptive expansion in non-reasoning mode: this generation of non-reasoning mode behaves more like an adaptive reasoning model with a compressed thinking budget. Simple problems can be solved within a few thousand tokens, but complex tasks may expand to 20K or even 30K tokens of output. With the limit set at 15K, nearly half of the samples exceeded the cap and could not be scored, making full testing temporarily impossible.
• Truncation risk for long chain-of-thought: the reasoning mode was also affected. A small number of tasks exceeded the 80K output limit across multiple runs and had to be scored as 0. This is part of the reason why the median reasoning-mode score for some models was lower than expected.

The takeaway for evaluators: any leaderboard must state its output limits and truncation scoring rules explicitly. Otherwise, ranking differences may simply be artifacts of different settings.

2.4 Example Code: Evaluation Skeleton

def run_eval(model, mode, max_output_tokens, rounds=3):
    results = []
    for task in TASKS:
        per_round = []
        for _ in range(rounds):
            resp = model.generate(
                prompt=task.prompt,
                mode=mode,                       # reasoning / standard
                max_output_tokens=max_output_tokens,
            )
            truncated = resp.finish_reason == "length"
            per_round.append(0 if truncated else grade(task, resp.text))
        results.append({
            "task_id": task.id,
            "score": median(per_round),          # median across rounds
            "tokens": max(r.output_tokens for r in per_round),
        })
    return summarize(results)                    # total score, distribution, token usage

3. Reasoning vs. Non-Reasoning Modes: Behavioral Differences and Engineering Implications

The difference between the two modes is not just “fast vs. slow.” It is a fundamental difference in cost structure:

• Reasoning mode: chain-of-thought is explicit and relatively stable in length. As long as the reasoning chain does not change dramatically, total per-task cost remains manageable even if the unit price increases modestly with a new generation—a common industry pattern and a key reason vendors can justify price increases.
• Non-reasoning mode: superficially “fast thinking,” but it actually exhibits adaptive behavior. Token consumption variance is extremely high. Simple and complex tasks can differ by an order of magnitude.

Three engineering recommendations follow:

1. Cost estimation should look at distributions, not just averages. Budgeting based on average consumption will break badly on long-tail tasks.
2. Set a budget cap and fallback strategy for non-reasoning mode instead of allowing unbounded output expansion:

def ask(model, prompt, budget=15_000):
    resp = model.generate(prompt, mode="standard",
                          max_output_tokens=budget)
    if resp.finish_reason == "length":
        # Truncated result is not deliverable; retry in reasoning mode
        return model.generate(prompt, mode="reasoning",
                              max_output_tokens=budget * 2)
    return resp

1. Monitor finish_reason to distinguish truncation from normal completion, and treat truncation rate as an independent health metric. On cloud platforms such as Baidu AI Cloud, developers can use model services, logging, and monitoring/alarming capabilities to observe token consumption, truncation rate, and error rate in a unified way.

4. Key Findings in Long-Chain Reasoning: Solution Space Exploration and Thinking Efficiency

4.1 Generational Improvement on Multi-Solution Problems

The previous generation did not score poorly on multi-solution logic problems, but it had a clear weakness: insufficient solution space exploration. It could find some solutions but often missed other valid ones. The new generation largely solves this problem, exploring possibilities thoroughly and finding as many solutions as possible. This matters significantly for planning, scheduling, and enumeration tasks in real business scenarios: the cost of a missed solution is not “wrong,” but “incomplete.”

4.2 The Price: Running Against the Budget

Progress is not free. While overall token consumption remained comparable to the previous generation, consumption on complex logic problems could reach more than 3x that of the prior generation—because the model keeps reasoning until it hits its thinking budget. Meanwhile, leading models in the industry can solve the same problems with the same perfect score using only a fraction of the tokens. Improving thinking efficiency—achieving the same accuracy with shorter reasoning chains—will be the next competitive battleground.

4.3 Recommended Metrics

A benchmark report should not report accuracy alone. At minimum, it should present side by side:

• Average output tokens per task and P95 consumption.
• “Cost per unit of accuracy”: how many tokens are needed to earn one point.
• Reasoning and non-reasoning modes reported separately, never mixed.

5. Coding Ability: High One-Shot Scores Do Not Mean Agent Usability

The most common mistake in coding evaluation is using one-shot generation scores as a proxy for coding ability. Experience shows a systematic gap between the two. The previous generation significantly improved one-shot coding performance, but performed poorly on real agents with real projects. The core deficiency was a lack of effective debugging capability—the model could write code, but when it failed to run, it could not locate or fix the problem.

Recently previewed programming models show that agentic coding capability has improved substantially. For standard frontend and backend development, performance has entered the “usable” range. The root cause of the difference lies in task structure:

• One-shot is “read the problem and answer”: a single generation with no environmental feedback.
• Agent mode is a “run–observe–fix” loop: it requires tool use, error localization, and cross-turn context management.

Therefore, coding evaluation should set up real repository tasks, allow multiple rounds of tool calls, and measure fix success rate and average rounds:

while steps < MAX_STEPS:
    observation = run_tests(repo)          # run tests and collect errors
    if observation.all_passed:
        break
    action = model.decide(history + observation,
                          tools=["read_file", "edit_file", "run_cmd"])
    history.append(execute(action))        # execute and append observation

6. Cost and Model Selection: Separate Unit Price from Total Cost

For model selection, adopt a scenario-based strategy:

• High-frequency simple tasks (classification, extraction, summarization): prefer non-reasoning mode with a strict budget cap to control long-tail consumption.
• Complex logic and planning tasks: use reasoning mode, with thinking budgets and retry-on-truncation mechanisms.
• Coding tasks: always validate agent performance on real projects, not demo-level one-shot results.
• General capability integration: new-generation models generally continue to strengthen long-context input, file understanding, content generation, and data analysis. They can be accessed through web, API, and SDK options, and teams can choose according to their tech stack.

It is worth repeating: your bill is determined by the product of “chain-of-thought length × unit price × call volume,” not by unit price alone. If a price increase brings lower truncation rates and higher accuracy, total cost of ownership may actually decrease.

7. An Engineering Checklist for Reproducible Benchmarks

A practical checklist for your own evaluation framework:

1. Pin the model version and invocation date, and record them in the report.
2. Fix prompts and sampling parameters, and document sources of randomness.
3. State output limits and truncation scoring rules explicitly.
4. Run multiple rounds and take the median.
5. Label reasoning and non-reasoning modes separately.
6. Only list comparable samples on leaderboards, and note when rankings are not complete.
7. Update monthly to track generational changes over time.

Conclusion

Every model generation comes with similar marketing highlights: long context, strong reasoning, coding ability. But the real differences hide in long-horizon task performance, token consumption distributions, and truncation rates—engineering details that matter in production. For developers, a capability checklist is only a starting point. Building your own benchmark with explicit scoring rules, cost observability, and mode labels is the key step toward turning model capability into business value.