Evaluating LLM Code Capability Through the AGI Five-Stage Roadmap

    An increasing number of teams gauge general-purpose AI through code capability, yet benchmark scores alone can easily mislead. Using the widely referenced five-stage AGI roadmap—pretraining, alignment and reasoning, autonomous learning, self-awareness, and conscious intelligence—this article offers a practical lens for judging where models realistically stand. It explains why code is an effective capability probe, breaks down three categories of evaluation metrics (pairwise blind tests, automated benchmarks, and proprietary end-to-end tests) alongside four common pitfalls such as data contamination and the pass@1/pass@k mix-up, and shows how to evaluate interface-operating agents. It closes with four recommendations for turning evaluation into a repeatable enterprise asset.

    1. Two Paths, One AGI Timeline

    The current LLM competitive landscape can roughly be abstracted into two types of teams. The first originates from university labs focused on knowledge engineering and natural language processing, having moved from established disciplines such as semantic understanding and knowledge graphs all the way to large-scale pretraining—teams with deep academic accumulation and a high degree of self-developed foundation models. The second consists of product-oriented, engineering-driven teams that use engineering efficiency, compute output per unit, and overseas markets as their entry points, trading a faster iteration cadence for advantages in product experience.

    For developers and technical decision-makers, the valuable observation is not “who resembles whom,” but what reusable engineering experience each path has accumulated. The academic camp has proven that self-developed foundation models are the key to controlling long-term cost. The product camp has proven that post-training and inference-side optimization can substantially raise performance per unit cost without swapping out the base model. Both point to the same conclusion: AGI is not a single breakthrough, but a phased roadmap of capability build-out.

    2. An Engineering Reading of the Five-Stage AGI Roadmap

    A common grading scheme divides capability evolution into five stages. Here is what each stage means from an engineering perspective:

    1. Pretraining stage: teaching machines language. The core work is corpus cleaning and deduplication, data-mix design, tokenizer selection, and context-length planning. This stage is the most expensive, and once the foundation is set, later work can only “correct” it through post-training rather than rebuild it. Engineering takeaway: data-pipeline quality determines the ceiling, while model architecture only determines the cost curve.
    2. Alignment and reasoning stage: aligning the machine with human intent. Through supervised fine-tuning, preference optimization, chain-of-thought elicitation, and tool-use training, the model gains instruction following, constraint satisfaction, and task planning. The gains here are the most “visible” and are currently the main source of differences in product experience.
    3. Autonomous learning stage: teaching the machine self-reflection. The key is not “knowing how to reflect,” but whether a verifiable signal exists—unit tests, compilers, type checking, deterministic rules, or a reliable reward model. Without a verifier, self-reflection degrades into self-reinforcing hallucination.
    4. Self-awareness stage: forming a stable autonomous stance. In engineering terms, this shows up as consistency of persona and value orientation across long conversations, memory consistency, and how boundaries are expressed.
    5. Conscious intelligence stage: a conceptual phase. The industry currently has no agreed definition or quantifiable metrics, so it is not suitable as a basis for procurement or architecture decisions.

    In practice, most vendors have already brought the first two stages to a deliverable level, while the third sits somewhere between pilot and partial availability. When writing technical plans, treating “capability stages” as a roadmap rather than as marketing language leads to far more accurate judgments.

    3. The Capability Triangle: Foundation, Multimodality, and Agents

    A complete capability matrix should cover at least three areas:

    • Foundation models, which determine the ceiling for language understanding, reasoning, and code generation;
    • Multimodal models, which bring images, video, and audio into a unified representation and solve the “can see” problem;
    • Agent models, which chain perception and decision-making into action sequences and solve the “can do” problem.

    The three do not replace one another. Without multimodality, a model cannot operate graphical interfaces; without agent capability, a model can only generate text and cannot complete tasks. When assessing a platform, test these three areas separately rather than inferring the whole from a single leaderboard score.

    4. Why Code Is the Best “Capability Probe”

    Among all capability dimensions, code ability is repeatedly used as a proxy for general capability, for very practical reasons:

    • Automatically verifiable: whether it compiles and whether it passes tests are among the few objective signals that require no human judgment;
    • Strong structural constraints: syntax and type systems inherently penalize vague expression and expose instruction-following flaws;
    • Long-context pressure: cross-file understanding and dependency tracing place heavy demands on how well context is used;
    • High tool dependency: real coding tasks require reading files, running commands, inspecting errors, and then revising—nearly equivalent to an agent-capability test.

    As a result, what code ability measures is often not merely “whether the model can write code,” but its combined level of reasoning, planning, tool use, and error correction.

    5. Evaluation Methodology: Three Metric Types and Four Pitfalls

    Three metric types

    1. Pairwise blind battles: large numbers of users vote on two sets of outputs with model identities hidden, and rankings are derived in an Elo-like manner. The strength is that it stays close to real preferences; the weakness is that you must control for prompt-distribution bias and presentation-order bias, or the ranking gets polluted by irrelevant factors such as “which side produced a longer answer.”
    2. Automated benchmarks: such as repository-level defect fixing, multilingual code completion, and terminal command-execution evaluations. The strength is that they are objective and reproducible; the weakness is that they are easily contaminated by training data and may differ greatly from your actual business distribution.
    3. Proprietary end-to-end business evaluation: measuring task success rate on real repositories, real defects, and real acceptance criteria. This is the only data you can use for go-live decisions.

    Four common pitfalls

    • Data contamination: benchmark questions or their solutions have already appeared in the training corpus, inflating scores;
    • Mixing up pass@1 and pass@k: single-attempt pass rate and at-least-one-pass rate after multiple samples are two different things, and the sampling strategy must be stated clearly when disclosed externally;
    • Looking only at pass rate, not cost: ignoring token consumption, retry counts, and latency, which can make online costs differ by several times;
    • Insufficient language and scenario coverage: a high score in a single language does not mean cross-language usability, and vertical domains (such as embedded or finance-compliance code) often require a separate dataset.

    A minimal viable evaluation scoring logic looks roughly like this:

    1. def evaluate(case, model, k=1):
    2. results = []
    3. for _ in range(k):
    4. out = model(case.prompt)
    5. results.append({
    6. "passed": run_tests(case.repo, out.patch),
    7. "tokens": out.usage.total_tokens,
    8. "latency_ms": out.latency_ms,
    9. "turns": out.turns, # tool-call turns
    10. })
    11. passed = sum(r["passed"] for r in results)
    12. return {
    13. "pass@k": float(passed > 0),
    14. "pass@1": float(results[0]["passed"]),
    15. "avg_tokens": mean(r["tokens"] for r in results),
    16. "avg_turns": mean(r["turns"] for r in results),
    17. }

    The key is to record “effectiveness” and “cost” in the same entry. Looking only at effectiveness selects models you cannot support at scale; looking only at cost selects models you dare not put into production.

    6. Agent Deployment: Interface Operation as an Example

    Capabilities like “an agent that can autonomously operate a phone” are, from an engineering standpoint, made up of four parts:

    • Perception: combining the accessibility element tree with visual screen understanding for element localization; the two are complementary and avoid failure from a single signal;
    • Action space: a finite set of actions such as tap, swipe, type, wait, and back; the larger the action set, the faster errors accumulate;
    • State management: setting checkpoints and supporting rollback and idempotent operations, so failures are recoverable rather than requiring a full restart;
    • Security and compliance: enforcing a second confirmation for irreversible actions such as payments, deletions, and sends, keeping an audit log throughout, and following the principle of least privilege.
    1. while not task.done and step < max_steps:
    2. obs = capture_screen() # screenshot + element tree
    3. plan = planner(obs, task.goal) # output structured action
    4. if plan.action in HIGH_RISK_ACTIONS:
    5. if not confirm(plan): # human gate
    6. break
    7. result = executor(plan)
    8. if result.confidence < THRESH:
    9. rollback(last_checkpoint()) # recoverable failure
    10. step += 1

    Evaluating this kind of agent should track at least task success rate, average number of steps, failure recovery rate, and the number of unauthorized-action triggers—the last of which should always be zero.

    7. Turning “Capability” into an Enterprise Asset: Four Practical Recommendations

    1. Build your own layered evaluation set: L1 single-point capability, L2 repository-level tasks, L3 business end-to-end flows. All three layers are needed, and they should be continuously updated to guard against contamination.
    2. Connect the data loop: capture request logs, execution traces, and failure samples into object storage, and use logging services with monitoring alerts to establish regression gates, so that a model version change automatically triggers evaluation.
    3. Do cost engineering: adopt large/small model cascading routing, semantic caching, batching, and context compression—reserving high-value tasks for strong models and handing repetitive tasks to lightweight ones.
    4. Ensure deployment elasticity: leverage the elastic scaling and canary-release capabilities of a container platform so that model upgrades and traffic switches are rollback-able and comparable.

    When making technology choices, ask only three questions: How does this capability perform on your own data distribution? What are the per-task cost and latency? What is the fallback path when the capability fails? Answering these three clearly supports decisions far better than checking against any leaderboard.

    Baidu AI Cloud offers a complete set of supporting capabilities in general categories such as model development, compute scheduling, object storage, and monitoring and alerting, making it easier for teams to engineer their evaluation and canary-release workflows and shorten the path from validation to launch.