Evaluating New LLM Releases: Real-World Capability and Cost Effectiveness

After a new large language model is released, developers usually care less about “how high the benchmark scores are” and more about whether the model can complete real work reliably, and what it costs to finish an acceptably complete task. Looking only at per-call pricing or the release notes makes it easy to miss quota limits, rework, context consumption, and engineering integration cost. A more useful approach is a reproducible evaluation framework: probe capability boundaries with small tasks, test engineering stability with complex project samples, then convert subscription fees, API fees, human time, and rework into effective cost. This article follows that approach and includes a prompt-driven case of building a 2.5D beat-em-up action RPG from an empty directory.

Why release notes are not enough

Release notes typically highlight gains on general benchmarks. Developer work is often long-chain, multi-constraint, and strongly acceptance-driven. A model may look strong in single-turn Q&A yet fail frequently on multi-file code generation, state-machine decomposition, compile-error repair, and cross-module consistency. Evaluation should cover at least three scenario types: short tasks (function completion, error explanation, config generation); medium tasks (module design, interface alignment, test-case completion); and complex tasks (building a runnable small project from an empty directory). Only when pass rate, human edits, and end-to-end time are recorded for all three can you judge real capability.

Six dimensions of capability evaluation

1. Reasoning and instruction following. Check whether the model keeps constraints from drifting in long prompts. For example, if you require “no full 3D rendering module,” “placeholder art with geometric shapes,” and “data config decoupled from logic,” does later code still obey those rules. Conflicting constraints can be used to see whether trade-offs are reasonable.

2. Code generation and compile pass rate. Do not judge snippets that merely “look like code.” Check whether they form a compilable minimal project. Record first-pass compile rate, runtime crash rate, and interaction rounds needed to fix. For statically typed projects, also check type definitions, module imports, and build config completeness.

3. Long context and cross-file consistency. Complex projects produce many entities, states, and configs. Whether the model remembers earlier coordinate systems, collision rules, and state-machine names affects later maintenance. Fix a glossary and architecture constraints in the prompt, and require a change summary after each iteration.

4. Tool use and engineering integration. Behavior in an IDE, CLI builds, log analysis, and test execution often matters more than chat. Record whether the model can locate files from compile errors, converge after multiple failures, and avoid useless retries.

5. Multi-turn stability and recoverability. Real development is not one-shot generation. Watch whether structure stays stable under requirement changes, error fixes, and module expansion. An “add-on requirement” test helps: after basic combat works, add weapon throwing and downed follow-up attacks, and check whether existing systems break.

6. Safety and compliance boundaries. When user data, logs, secrets, and external dependencies are involved, the model should not emit hardcoded secrets, dangerous commands, or non-compliant dependencies. Include safety prompts and human review.

Cost effectiveness: from unit price to effective cost

Cost effectiveness is not simply “price per million tokens.” A better metric is effective cost:

effective cost = (subscription or API fees + human time cost + rework cost + infrastructure cost) / number of accepted tasks

Subscription or API fees should include quota conversion. Some plans use weekly limits; higher-tier features may open only part of the weekly quota by default and may offer a one-time quota reset. The same capability called from an IDE may yield a much lower weekly usable quota than the main entry point. Record “nominal quota” and “tasks actually completed” separately, or cost will be underestimated.

Human time includes prompt writing, review, compile fixes, and test verification. Rework includes throwaway work from architectural mistakes. Infrastructure includes isolated runtimes, log storage, monitoring/alerting, and CI resources. For teams, also count knowledge-capture cost: prompt templates, acceptance checklists, and failure case libraries.

Complex project practice: a 2.5D beat-em-up action RPG

To test complex-task capability, use a from-empty-folder sample: a small 2.5D game similar to a classic beat-em-up action RPG. It includes a coordinate system, state machine, collision, input, weapons, projectiles, config data, and placeholder art, which covers engineering constraints well.

Prompt design principles

The prompt should include goal, stack constraints, art placeholder strategy, data-driven requirements, core architecture, feature modules, acceptance criteria, and iteration order. Example template:

Project goal: From an empty directory, build a runnable 2.5D beat-em-up action RPG prototype.
Tech stack: Use a statically typed language with cross-platform compilation and a lightweight 2D rendering framework; prefer bytecode or web targets.
Rendering constraint: Simulate 2.5D with 2D drawing; do not use a full 3D rendering module.
Art placeholders: Draw all characters, terrain, weapons, effects, and UI with geometric shapes and text.
Data-driven: Decouple character stats, shop items, map config, and skill data via JSON or static config classes.
Core architecture: Entities have (x,y,z) logical coordinates; screenX=x, screenY=y-z; each entity draws a translucent ellipse shadow; sort the scene by y each frame; collision uses AABB + Z height + Y depth thickness.
Input: Arrow keys to move; double-tap left/right to run; A for punch/pick up/use weapon; B for kick/throw weapon; A+B or a dedicated jump key to jump.
Feature modules: state machine, weapons and projectiles, enemy AI, shop and skill unlocks, level config, save and debug panel.
Acceptance: Project compiles, runs, completes a basic combat loop, and outputs module notes plus known limitations.
Iteration order: coordinates and rendering first, then input and state machine, then combat collision, then weapons, then config and UI.

Core architecture pseudocode

Keep coordinate-to-screen mapping simple:

type Entity = {
  x: Float; y: Float; z: Float;
  vx: Float; vy: Float; vz: Float;
  hp: Int; state: String;
}
screenX = entity.x
screenY = entity.y - entity.z
shadowX = entity.x
shadowY = entity.y

Collision should consider horizontal overlap, height, and depth:

hit = overlapX(attackerBox, defenderBox)
   && overlapZ(attackerBox, defenderBox)
   && abs(attacker.y - defender.y) <= depthTolerance

A useful state machine includes Idle, Walk, Run, Jump, Attack, Hurt, Panting, Downed, and Grabbed. Accumulated damage above a threshold enters panting. After an enemy is downed, the player can follow up or lift and throw when close. Weapons should support random scene drops, pickup, durability or single use, projectile flight, and recovery on landing. Put all config in JSON or static config classes; logic only reads config and avoids hardcoding.

Feature modules and acceptance

Accept by module: stable coordinates and shadows; correct Y-sorting when characters overlap; double-tap run without false triggers; no illegal state jumps; attack and hurt boxes affected by Z; pickup and throw consistent with the state machine; enemy AI that can chase, attack, and go down; shop and skill unlocks that are data-driven. Record first-pass rate, human-edited lines, and repair rounds per module, then aggregate a complex-task score.

Evaluation pipeline and log sheet

Keep a fixed benchmark set and complex project sample. Run in an isolated environment and save prompt versions, model output, build logs, and test results. You can use object storage, a container platform, logging, and monitoring/alerting from Baidu AI Cloud to host the pipeline, but the evaluation logic should stay neutral and reproducible. The log sheet should at least include: task ID, task type, first-pass success, compile pass, run pass, human-edited lines, interaction rounds, end-to-end time, quota consumed, effective cost, and failure reason.

Cost-reduction practices

First, small then large: establish capability boundaries with short tasks before investing in complex projects. Second, pin prompt versions so results stay comparable. Third, prefer automated tests for compile, run, and key behaviors to reduce manual review. Fourth, route by difficulty: cheap models for simple tasks, stronger models for complex ones. Fifth, cache and reuse stable configs, templates, and test cases. Sixth, monitor quotas weekly and adjust when conversion looks off. Seventh, compliance review of generated code for dependencies, security, and licenses.

Conclusion

Real capability and cost effectiveness of a new LLM must be measured on real task chains. Capability evaluation should cover instruction following, code generation, long context, tool use, and multi-turn stability. Cost evaluation should fold subscription quotas, API fees, human time, and rework into effective cost. Building a 2.5D beat-em-up action RPG from scratch is a strong complex sample for coordinates, state machines, collision, input, weapons, and data-driven design. With a reproducible evaluation pipeline, teams can choose models rationally as versions iterate, instead of being driven by a single demo or a leaderboard score.