Coding Plan: A Practical Guide to Selecting and Evaluating Coding Models

    This article explains why coding performance should be evaluated as a system-level capability rather than as a property of a single model. It presents a practical framework for building a coding plan: classifying tasks from L0 snippets to L3 operations, constructing private evaluation sets with executable verifiers, measuring correctness and mergeability, using sandboxes for reproducible evaluation, routing across models through an abstraction layer, assembling context effectively, enforcing security and merge gates, and continuously operating the system with observability and cost controls.

    1. Coding Plan Is About Building a Workbench, Not Just Picking a Model

    In demo environments, coding models often look impressive: a function, a regular expression, or a simple refactor can be completed in seconds with usable output. However, once the same model is placed inside a real repository, the situation changes significantly.

    Tasks may require understanding call relationships across a dozen files, following internal framework conventions and coding standards, compiling and running tests after changes, and recovering from failures by reading logs, revising assumptions, and retrying.

    This leads to a key judgment: coding capability is a systemic capability, not an attribute of a single model. It is shaped by at least four factors:

    1. The model’s coding and reasoning ability, including code understanding, long-context retention, and stable tool calling.
    2. The quality of context supply, meaning whether the model is given the code and conventions it actually needs.
    3. The strength of the verification loop, meaning whether there are automatically executable compilation, tests, and static checks that provide feedback.
    4. Orchestration and governance mechanisms, including routing, retries, fallback, cost control, and safety boundaries.

    A coding plan is an engineering approach that defines, measures, and continuously optimizes these four factors. It does not answer “which model is strongest today.” Instead, it answers: “Given our codebase and task distribution, which combination can reliably produce mergeable changes at acceptable cost and latency?”

    2. Break Coding Capability into Measurable Task Levels

    Any model selection or evaluation effort depends on task classification. It is not meaningful to ask only “how good is the model at writing code,” because different task levels have very different sensitivity to context window, autonomy, and verification cost.

    A practical classification is:

    Level Typical tasks Key constraints Verifiability
    L0: snippet-level Function completion, type annotations, unit test drafts, comments and documentation Single-file context, short output Strong; compilation can verify
    L1: file-level Single-file module implementation, local refactoring, fixing failing tests Requires understanding file structure Strong; tests can decide
    L2: repository-level Cross-file feature implementation, dependency upgrades, locating and fixing defects Requires retrieval, reading multiple files, may break implicit contracts Medium; needs regression tests
    L3: engineering operations Build and pipeline configuration, deployment orchestration, production troubleshooting Requires running commands, reading logs, judging environment differences Weak; often needs human confirmation

    The direct value of this classification is matching and loss control.

    L0 tasks are extremely sensitive to latency and cost. Using the strongest long-context flagship model for such tasks is wasteful. L2 and L3 tasks are sensitive to model autonomy, tool-calling reliability, and context length. Using a lightweight model for these tasks often leads to lost constraints or unrelated changes during multi-round edits.

    Only after tasks are classified can routing strategies, evaluation set ratios, and cost budgets be balanced meaningfully.

    3. Build a Private Evaluation Set; Public Leaderboards Are Only for Early Screening

    Public benchmarks are useful for quickly eliminating obviously unsuitable candidates. However, they cannot answer the most important question: how does the model perform on your codebase?

    There are three reasons:

    1. Private coding styles, internal frameworks, and proprietary protocols create many implicit conventions that do not exist in public datasets.
    2. Source code often cannot be sent externally, so external evaluation services cannot be relied upon.
    3. Public tasks may be contaminated by training data, which can inflate scores.

    3.1 Where Samples Come From

    The best samples usually come from the team’s own history:

    • Merged bug-fix commits: problem description + context before the change + the final human patch.
    • Merged feature development requests: requirement description + related modules + acceptance tests.
    • Incident reviews and on-call records: symptoms + log snippets + root cause and fix.
    • High-frequency rework points in code review: these reveal which constraints models are most likely to ignore.

    3.2 The Three Required Components of Each Sample

    A usable evaluation sample must include:

    • Input: task description + the allowed context scope, usually a list of files rather than the entire repository.
    • Expected result: a merged human patch, or, more reliably, an acceptance test command.
    • Verifier: an automatically executable command with deterministic results, such as compilation + unit tests + static checks.

    A sample with a verifier is better than one that only provides an expected answer. It can pass a model output that differs from the human implementation but is still correct.

    3.3 Contamination Control and Reproducibility

    • Time-based splitting: collect commits after a specific cutoff date to reduce overlap with training corpora.
    • Private sample ratio: at least half of the samples should come from internal repositories, with secrets, internal domains, and personal information removed.
    • Fixed environment: pin dependency versions, base images, and random seeds so that the same patch does not drift across environments.
    • Balanced distribution: L0 through L3 should follow the team’s real task distribution. In addition, maintain a separate high-difficulty holdout set that is not used for tuning and is reserved for final acceptance.

    4. Metrics: From “Looks Right” to “Can Be Merged”

    Judging output quality only by human inspection cannot support large-scale model selection. It is better to build a dashboard around four metric groups.

    4.1 Correctness Metrics

    • Unit test pass rate and compilation pass rate.
    • pass@1: success rate from a single attempt.
    • pass@k: probability that at least one attempt succeeds across multiple samples. This is useful for evaluating whether retries can recover failures.
    • Regression introduction rate: the proportion of model changes that cause existing tests to fail. This metric is often underestimated but is critical.

    4.2 Engineering Usability Metrics

    • First-pass merge rate: the proportion of outputs that can be merged without manual modification.
    • Human rework ratio: human-edited lines divided by model-generated lines. Lower is better.
    • Change size: patch line count and number of affected files. Abnormal expansion often means the model is guessing.
    • Review comment density: the average number of review rounds required before merge.

    4.3 Process and Experience Metrics

    • Time to first token and end-to-end latency.
    • Number of tool calls and failed retries.
    • Interruption rate under streaming output.

    4.4 Economic Metrics

    The cost of a single task can be estimated as follows. The actual prices should be filled in according to the team’s subscription or billing terms. The formula below only describes the structure:

    1. cost(task) = in_tokens * price_in
    2. + out_tokens * price_out
    3. + cache_hit_tokens * price_cache
    4. cost(task) *= (1 + retry_rate) # Retry amplification effect

    The retry_rate is often ignored, but for L2 and L3 tasks, retries may double the actual spend. It is also important to remember that output tokens are usually the most expensive part. Controlling verbose output and using structured diff output are often more effective than simply switching to a cheaper model.

    4.5 Statistical Rigor

    Do not draw conclusions from very small samples. If the sample size is below 30, differences in success rates may simply be noise. It is advisable to report confidence intervals and clearly state sample size and confidence level in the evaluation report.

    5. Evaluation Execution Framework: A Reproducible Sandbox

    Evaluation must run in an isolated, repeatable, and auditable environment. The core data structures can be lightweight:

    1. from dataclasses import dataclass
    2. @dataclass
    3. class TaskSpec:
    4. task_id: str
    5. level: str # L0 / L1 / L2 / L3
    6. prompt: str # Task description: bug ticket, requirement, or failing test output
    7. context_files: list[str] # Files allowed to be exposed to the model
    8. verify_cmd: list[str] # For example: ["make", "test"]
    9. golden_patch: str | None # Human patch, used for comparison only, not as the sole criterion
    10. @dataclass
    11. class CaseResult:
    12. task_id: str
    13. passed: bool
    14. latency_ms: float
    15. tokens_in: int
    16. tokens_out: int
    17. tool_calls: int
    18. human_edit_ratio: float # Human-edited lines / model-generated lines

    The execution loop follows a fixed process to ensure replayability:

    1. def run_case(task: TaskSpec, backend) -> CaseResult:
    2. sandbox = create_sandbox(image=PINNED_IMAGE, network="restricted")
    3. try:
    4. ctx = assemble_context(task) # See section 7
    5. resp = backend.complete(ctx, budget=BUDGETS[task.level])
    6. sandbox.apply_patch(resp.patch)
    7. ok = sandbox.run(task.verify_cmd, timeout=TASK_TIMEOUT) == 0
    8. return CaseResult(task.task_id, ok, resp.latency_ms,
    9. resp.tokens_in, resp.tokens_out,
    10. resp.tool_calls, 0.0)
    11. finally:
    12. sandbox.destroy() # Reclaim resources regardless of success or failure

    Key sandbox design points:

    • Containers should be created and destroyed per run.
    • External network access should be disabled, with dependencies preloaded into the image.
    • Commands should be restricted to a whitelist.
    • CPU, memory, and timeout limits should be enforced.
    • All inputs and outputs should be persisted so that results can be audited and reproduced later.

    Without this step, evaluation conclusions cannot be trusted.

    6. Model Routing and Abstraction: Avoid Tying Business Code to One Interface

    Model and coding-tool supply will continue to change. Therefore, business code should depend only on an abstract interface, while concrete backends are selected by configuration. This is one of the most important designs for reducing long-term costs in a coding plan.

    1. from typing import Protocol
    2. class CodingBackend(Protocol):
    3. name: str
    4. context_window: int
    5. price_in: float
    6. price_out: float
    7. def complete(self, ctx: Context, budget: Budget) -> Response: ...

    On top of this abstraction, four routing strategies can be implemented:

    • Capability routing: L0 tasks use low-latency, smaller models; L2 and L3 tasks use long-context, strong-reasoning models.
    • Cascade routing: try a cheaper model first, then escalate to a stronger model if verification fails, reducing average cost.
    • Constraint routing: enforce both budget and SLA constraints, automatically downgrading when the budget is exceeded or timeout is approaching.
    • Canary routing: new models first receive 1%–5% of traffic, split by repository or team. Traffic is increased only after metrics meet targets, and a one-click rollback should be available.
    1. def solve(task: TaskSpec, backends: list[CodingBackend]):
    2. for be in backends: # Ordered from cheaper to stronger
    3. resp = be.complete(build_context(task), budget=BUDGETS[task.level])
    4. if verify_and_score(task, resp):
    5. return resp
    6. metrics.record_failure(be.name, task.task_id)
    7. return escalate_to_human(task) # The fallback is always human escalation

    A caveat is needed: cascading does not always save money. If verification is expensive, for example when every attempt requires running the full regression suite, trying multiple times may cost more than using a stronger model once. Whether to enable cascading should be decided based on measured results from the private evaluation set described in section 3, rather than copying general assumptions.

    7. Context Assembly: What You Feed Often Determines the Ceiling

    In practice, many failures are not caused by insufficient model capability, but by the model not seeing key constraints. Context assembly should proceed according to the following priority:

    1. Repository indexing and retrieval: build symbol indexes and call graphs, and combine keyword retrieval with vector retrieval. Locate relevant modules first, then worry about context length.
    2. Convention injection: place stable content such as directory structure, dependency summaries, coding standards, and internal framework usage at the front of the prompt. A stable prefix can improve quality and also increase prefix-cache hit rates, directly reducing costs.
    3. Layered context: order content by importance, for example task description → related code → interface signatures → errors and logs → historical discussion. When the budget is exceeded, trim from the tail first.
    4. Correct use of long context: a long context window does not mean effective context. If information density is low, important content in the middle may be ignored. For very large repositories, summarize and trim before feeding content to the model.
    5. Tools over stuffing: allowing the model to search, read files, run tests, and inspect outputs autonomously is often more effective than dumping a large amount of code into a single prompt. It also makes it easier to locate the real root cause.

    8. Security, Compliance, and Merge Gates

    Once coding workflows connect to models, the attack surface expands significantly and must be handled systematically:

    • Source code should not leave the controlled environment: use dedicated deployment or private environments for sensitive repositories. Logging systems must not persist source code content. Before model invocation, scan for secrets, credentials, and customer data.
    • Treat untrusted input as untrusted: repository content, bug descriptions, and external dependency documentation should all be treated as untrusted input. Defend against prompt injection that could lead to unauthorized operations.
    • Least privilege for tools: credentials should be read-only, writable directories should be restricted, commands should be whitelisted, and external network access should be disabled. Any destructive operation must require human confirmation.
    • Supply chain risk: dependencies introduced by model output should be checked for source and license. Generated content should also pass license scanning.
    • Merge gates: compilation, unit tests, static checks, secret scanning, and license scanning must all pass before code can enter review. Model-generated output receives no exemptions.
    • Auditability: retain prompt versions, model identifiers and versions, patch diffs, and evaluation conclusions to form a traceable chain.

    9. Operations Loop: Keep the Coding Plan Improving

    A one-time model selection will quickly become stale because models, dependencies, and codebases all change. A fixed operating rhythm is recommended:

    • Regression evaluation: any change to prompts, retrieval strategies, or model versions should run the same fixed evaluation set. Set thresholds as gates; do not release if metrics do not pass.
    • Observability dashboards: display first-pass merge rate, rework ratio, regression introduction rate, per-task cost, and latency distribution by task level.
    • Shadow traffic and A/B testing: run new configurations in a bypass mode first, recording metrics without affecting production. Switch traffic only after metrics align.
    • Feedback loop: labels such as “directly accepted / partially modified / completely rejected” from developers are the cheapest and most realistic signal. These should be collected regularly and turned into new evaluation samples.
    • Cost review: calculate monthly cost distribution by task level and retry amplification factors, and identify wasted calls.

    10. Connecting the Framework to Cloud Capabilities

    The infrastructure requirements of this framework can be supported by general cloud product categories and do not require proprietary components:

    • Containers and compute resources can host evaluation sandboxes and batch regression jobs, scaling on demand.
    • Object storage can store desensitized evaluation sets, model outputs, and evaluation reports, making version management easier.
    • Message queues can decouple long-running tasks and support asynchronous batch evaluation and retries.
    • Logging, monitoring, and alerting services can observe latency, failure rates, token consumption, and cost anomalies.
    • Unified model gateways can hide backend differences and support configurable routing, rate limiting, and canary releases. This is also the production form of the abstraction interface described in section 6.

    Baidu AI Cloud provides corresponding product categories in these general capability areas and can be considered as one reference option when teams build coding-plan infrastructure. The final choice should still depend on the team’s scale, compliance requirements, and measured evaluation results.

    Conclusion

    The core of a coding plan can be summarized in four points:

    • Classify by task level, not by leaderboard rank.
    • Build a private evaluation set with verifiers; use public leaderboards only for early screening.
    • Measure real value through first-pass merge rate, rework ratio, regression introduction rate, and per-task cost.
    • Turn routing, context assembly, sandboxing, and merge gates into configurable and rollbackable engineering capabilities.

    With these four points in place, volatility caused by model iteration changes from passive adaptation into measurable parameter tuning. Coding intelligence can then enter the main development workflow in a stable and reliable way.