A Practical Framework for Choosing Models in AI Coding Tools: Input Screening, Engineering Constraints, and Custom Evaluation

    Switching models is a recurring debate on most engineering teams, and the real difficulty is not a shortage of options but the absence of a common yardstick. This article lays out a three-layer model selection framework for agentic coding tools: coarse screening by input modality (visual material, large-scale or multi-file context, structured instructions and code), fine screening with engineering constraints (latency, cost and retry overhead, tool calling reliability, deployment and compliance, repeatability), and final selection against a self-built evaluation set scored on build pass rate, unit test pass rate, review rework rate, change locality, token cost, and P95 latency. It also includes a routing configuration example, a cascade routing strategy, common pitfalls, and governance recommendations for making model choices measurable, reproducible, and regression-testable.

    1. The Starting Problem: Selection Is Hard Because There Is No Shared Yardstick

    In agentic coding tools, “which model should we switch to” comes up almost every week. The difficulty is not that there are too few options. It is that there is no common yardstick. Each model’s strengths sit on non-overlapping dimensions: one turns UI screenshots into clean, well-structured component code, another reliably locates relevant files across a repository context of hundreds of thousands of tokens, and another produces fewer rework cycles on strictly specified script and config generation.

    If you decide purely from a “composite leaderboard ranking,” the conclusion often has little to do with your team’s actual workload.

    So model selection should shift from comparing models to characterizing tasks. The framework below has three layers, from shallow to deep: coarse screening by input modality, fine screening by engineering constraints, and final selection with a self-built evaluation set.

    2. Layer 1: Coarse Screening by Input Modality

    How a model ingests input largely determines whether it can handle the task at all. Three questions asked up front can compress the candidate set into a manageable size.

    Question 1: Is the core input visual material?

    If the main entry point of your daily workflow is UI design mockups, page screenshots, hand-drawn wireframes, or table images, then candidates must have dependable multimodal understanding and the ability to map visual semantics onto component structure: recognizing layout hierarchy, inferring spacing and grids, and reproducing interaction states such as hover, disabled, and empty states.

    For this class of task, the key metric is not “does the code run.” It is visual fidelity and structural soundness. When evaluating, focus on supported image input resolution, batch handling of long images and multiple images, and whether the generated output tends toward stacked inline styles rather than a maintainable component-based implementation.

    Question 2: Is the core input large-scale text or multi-file context?

    When the input is dozens of requirement documents, a full set of historical design specs, or a codebase spanning hundreds of files, the bottleneck moves from understanding to retrieval and recall. A large context window is only a necessary condition. What actually decides success is mid-context utilization: many models perform well at the beginning and end of a long context but degrade noticeably on information in the middle.

    Evaluate this deliberately: bury the critical information in the middle of the test input and check whether the model can cite it accurately. Also look at the context compression strategy: sliding-window truncation, hierarchical summarization, or retrieval-augmented generation (RAG) with reranking. In engineering practice, the stable approach is usually “retrieval + summarization + layered injection” rather than stuffing an entire repository into the prompt at once.

    Question 3: Is the core input structured instructions and code?

    If the input takes the form of clear development requirements, explicit interface contracts, shell commands, or config snippets, then capability shifts toward instruction following and output format stability: does the model strictly respect naming conventions, directory structure, and error-handling conventions, and does it stay inside hard constraints such as “no new dependencies allowed” and “existing function signatures must be preserved”?

    These scenarios are the best fit for automated assertion-based acceptance and the easiest to quantify, so it is worth building automated evaluation here first.

    3. Layer 2: Refining Candidates with Engineering Constraints

    After coarse screening, a few candidates usually remain. Now bring in constraints that are not about raw capability.

    Latency and throughput. Interactive completion typically needs first-token latency in the hundreds of milliseconds, while long autonomous tasks can tolerate more inference time but require stable streaming output and observable progress. Writing the latency budget into your selection criteria avoids picking a solution that is strong on a single axis and unusable in practice.

    Cost and billing structure. Token consumption in long-context scenarios is often non-linear: one cross-repository analysis can consume several times the input volume of a single chat turn. Estimate it as average consumption per task × average daily task volume × retry factor, where the retry factor is routinely underestimated. Once failure retries and self-repair loops are in play, actual consumption can reach 1.5–3× the ideal value.

    Tool calling and structured output. Agentic modes depend on the model reliably producing schema-conformant function calls and patches. The question is not “can it call tools” but parameter extraction accuracy, consistency across multi-turn calls, and whether the model corrects itself based on error messages after a tool fails instead of looping on repeated calls.

    Deployment and compliance. When internal code and business data are involved, confirm whether private or dedicated deployment is supported, whether data is used for training, and whether log redaction and audit capabilities are complete. This dimension is often a single-veto item.

    Repeatability. Run the same input multiple times; variance in output quality should stay controlled. A model with high variance is hard to keep stable in an automated pipeline.

    A minimal illustration that makes these constraints explicit:

    1. # Route on "task profile + engineering constraints", not on model names
    2. TASK_PROFILES = {
    3. "vision_to_ui": {"ctx_tokens": 64_000, "modality": {"image"}, "p95_latency_ms": 8_000},
    4. "repo_level": {"ctx_tokens": 512_000, "modality": {"text"}, "p95_latency_ms": 30_000},
    5. "codegen": {"ctx_tokens": 32_000, "modality": {"text"}, "p95_latency_ms": 5_000},
    6. }
    7. def route(task: str, registry: list[ModelProfile]) -> ModelProfile:
    8. p = TASK_PROFILES[task]
    9. candidates = [
    10. m for m in registry
    11. if p["modality"] <= m.modalities # modality must be covered
    12. and m.ctx_tokens >= p["ctx_tokens"] # context capacity must be sufficient
    13. and m.p95_latency_ms <= p["p95_latency_ms"] # within the latency budget
    14. and m.compliant # compliance / deployment mode acceptable
    15. ]
    16. # prefer lower cost at equal capability, higher quality score at equal cost
    17. return min(candidates, key=lambda m: (m.cost_per_1k, -m.eval_score))

    The value of routing is that it turns “model selection” into configuration: when a new model is onboarded, you only update the registry and evaluation scores, not business code.

    4. Layer 3: Final Selection with a Self-Built Evaluation Set

    Public leaderboards are useful only as a first-pass reference. The final decision has to rest on an internal evaluation set that reflects your own code style. One way to structure it:

    1. eval_set:
    2. - id: repo-refactor-07
    3. category: cross-repo refactor
    4. input: {files: 128, prompt: prompts/refactor.md}
    5. assertions:
    6. - type: build # build must pass
    7. - type: unit_test # existing unit test pass rate >= 95%
    8. - type: diff_locality # changed files <= 30% of input files
    9. budget: {max_tokens: 400k, max_latency_s: 120}
    10. - id: ui-screenshot-12
    11. category: design-to-code
    12. input: {images: 3, prompt: prompts/ui2code.md}
    13. assertions:
    14. - type: visual_diff # structural similarity threshold
    15. - type: a11y_lint # zero accessibility rule violations

    Six dimensions are worth covering: build pass rate, unit test pass rate, human review rework rate, change locality (whether edits stay within the necessary scope), average token cost, and P95 latency.

    Change locality is the one most often overlooked, yet it directly drives code review cost: an implementation that is functionally correct but spreads changes across thirty files costs far more to maintain over time than one failed retry.

    On top of evaluation, you can add cascade routing: handle routine tasks with a low-latency, low-cost model first, and escalate to a stronger model if automated assertions fail, keeping average cost down without giving up quality.

    5. Common Pitfalls

    • Leaderboard-only thinking. Leaderboard task distributions differ substantially from real workloads, especially for long-context and repository-level refactoring scenarios.
    • Looking only at context length. Advertised length is not effective recall. Always run a mid-context retrieval check.
    • Underestimating retry cost. Actual token consumption without self-repair loops accounted for often deviates sharply from budget.
    • Ignoring prompt–model coupling. Stuffing model-specific tricks into prompts means quality collapses when you switch models. Keep prompts as a model-agnostic template layer under version control.
    • Switching everything at once. Go through gradual rollout and regression gates, and keep a rollback path.

    6. Engineering Rollout and Governance Recommendations

    1. Configuration and observability. Externalize model identifiers, parameters, and routing rules as configuration. Log task category, token consumption, latency, assertion results, and whether the output was ultimately adopted for every call, building a sample library that supports ongoing analysis.
    2. CI gates. Wire the self-built evaluation set into the pipeline. When a model version or prompt changes, run regression automatically and block the release if metrics degrade.
    3. Prioritize context engineering. A large share of quality problems comes from context organization rather than model capability. Invest first in retrieval quality, hierarchical summarization, dependency graphs, and the injection strategy for standards and spec files.
    4. Security and compliance. Redact sensitive information before it reaches the model, keep an audit trail of calls, and automatically scan generated dependencies and licenses to avoid pulling in non-compliant components.
    5. Continuous review. Report adoption and rework rates per model against the real task distribution each month, and adjust routing weights accordingly.

    7. Conclusion

    The essence of model selection is converting vague subjective impressions into measurable, reproducible, regression-testable engineering decisions. Using the three-layer framework of input modality — engineering constraints — self-built evaluation as the main thread, teams can keep making stable choices in an environment where models iterate quickly. General practice around model services and AI engineering toolchains at Baidu AI Cloud reflects the same logic: what really sets the ceiling on results is usually not the name of a single model, but the overall maturity of context organization, evaluation systems, and governance processes.