1. Why “Pick One Model” Is No Longer a Valid Question

Over the past year, releases of leading general-purpose large models have become much more frequent, often arriving every few months. Model rankings change quickly: one model may lead in code repair this round, while another may gain an advantage in long-document understanding or structured output next round.

This creates a very practical engineering problem: any selection document that concludes with “which model is the strongest” usually remains valid for only one model release cycle.

More importantly, the capability structure of large models is uneven. There is no single model that is simultaneously best across all dimensions:

• Function-level code completion cares about latency and cost per token.
• Repository-level defect fixing cares about cross-file reasoning and the ability to make minimal changes.
• Long-document summarization and cross-file retrieval care about how effectively the context window is used.
• Structured output and tool calling care about stable format compliance and failure modes.
• Batch repetitive tasks, such as log classification, field extraction, and unit-test generation, care about throughput and cost.
• For data subject to compliance constraints, the first question is where the data can flow before evaluating capability.

Therefore, the more reasonable question is not “Which model should we choose?” but rather:

How do we continuously evaluate models, orchestrate them by task, and bring their outputs into a controlled engineering loop?

The rest of this article follows that logic.

2. Evaluation: Build the Ruler Before Choosing Models

2.1 Public benchmarks are only for initial screening

Public benchmarks are useful because they provide horizontal comparability. However, three limitations must be understood:

1. Data contamination: benchmark questions may already exist in training corpora, causing scores to exceed real-world performance.
2. Prompt sensitivity: the same model can show meaningful score differences under different prompt templates and grading scripts.
3. Metric differences: some evaluations only check whether unit tests pass, while others also constrain the number of changed lines or prohibited edit areas.

Public benchmarks are therefore suitable for coarse filtering, narrowing candidates down to two or three options. They are not suitable for final decision-making.

2.2 Two task types that are closer to engineering value

More discriminating than “write a function” are these two categories:

1. Real repository-level defect fixing: Given an issue description and a codebase, the model must locate the correct files, produce a minimal change, and pass existing tests. This evaluates retrieval, cross-file reasoning, and self-verification at the same time.
2. Information extraction under noisy input: Given a messy log, error stack trace, or semi-structured document, the model must output results that strictly follow a schema. This evaluates instruction following and whether the model fabricates answers when uncertain.

2.3 Build a small gold evaluation set

One of the most effective steps is to extract 20 to 50 samples from your own codebase and business documents and turn them into a repeatable evaluation set:

evals/
  cases/
    001_bug_null_check.yaml      # Bug fix: repository snippet + expected behavior
    002_refactor_split_func.yaml # Refactor: external behavior must remain unchanged
    003_test_gen.yaml            # Test generation: must cover boundary conditions
    004_log_triage.yaml          # Log triage: output structured JSON
    005_sql_analysis.yaml        # Data analysis: provide table schema, produce queries and explanations
  rubric.yaml                    # Scoring rules
  run.py                         # Runner: executes the same cases against multiple candidate implementations

It is recommended to fix six scoring dimensions, each scored from 0 to 5, and to treat rework rounds as an independent metric:

Dimension	Focus
Functional correctness	Whether existing tests and new assertions pass
Minimal change	Whether the diff is restrained and avoids opportunistic refactoring
Instruction following	Whether schema, naming, and prohibited actions are respected
Explainability	Whether assumptions, risks, and unverified parts are stated
Rework rounds	How many human iterations are needed to converge
Resource consumption	Token usage, tool-call count, and wall-clock time

2.4 Evaluate the entire chain, not single-turn answers

In agent mode, one task includes multiple rounds of tool calls and self-correction. The same model can behave very differently in single-turn completion and long autonomous execution. Some models converge early and admit when they cannot solve a problem, while others repeatedly retry in the wrong direction.

Therefore, evaluation scripts must retain the full trajectory and classify failure modes, such as:

• Locating the wrong file
• Misjudging the root cause
• Over-editing
• Infinite retry loops
• Broken output format

This is more actionable than a single total score.

def run_case(case, agent, rubric):
    trace = agent.run(case["prompt"], tools=case["allowed_tools"],
                      max_steps=case.get("max_steps", 20))
    result = {
        "case": case["id"],
        "tests_passed": run_tests(case["verify_cmd"]),
        "diff_lines": count_changed_lines(trace.workdir),
        "steps": len(trace.steps),
        "tokens": trace.usage.total_tokens,
        "latency_s": round(trace.elapsed, 1),
        "schema_ok": validate_schema(trace.final_answer, case["schema"]),
    }
    result["score"] = score(result, rubric)
    return result

3. Task-Based Model Assignment: Use the Right Model for the Right Job

Once the ruler is built, orchestration becomes straightforward. A common industry practice is to layer responsibilities instead of forcing one model to handle the entire workflow:

• Planning layer (strong reasoning): reads requirements, decomposes tasks, and defines change boundaries. Call frequency is low, so a higher-cost model is acceptable.
• Implementation layer (low latency, controlled cost): writes code in batches, completes functions, and generates tests. This layer has the highest call frequency and is the main source of cost.
• Review layer (different model family from implementation): use a different model family for code review and defect rechecking to avoid shared blind spots of the same model.
• Long-document layer (long context): handles cross-file retrieval, design documents, and historical issues.
• Batch extraction layer (small models): handles repetitive tasks such as log classification, field normalization, and format conversion.

On top of this division of labor, three engineering safeguards are needed:

• Routing: select models based on task type, file size, and sensitivity.
• Fallback: automatically switch to a backup when the primary model times out or is rate-limited.
• Convergence: set maximum step counts and budget limits to prevent endless retries and excessive spending.

class Router:
    def __init__(self, providers, policy):
        self.providers = providers      # Abstract interface: invoke(prompt, tools, budget)
        self.policy = policy            # Task type -> primary / fallbacks / budget
    def dispatch(self, task_type, prompt, ctx):
        plan = self.policy[task_type]
        last_err = None
        for name in [plan["primary"]] + plan.get("fallbacks", []):
            try:
                return self.providers[name].invoke(
                    prompt, tools=plan.get("tools", []),
                    max_steps=plan.get("max_steps", 20),
                    budget=plan.get("budget"),
                    meta={"model": name, "task": task_type},
                )
            except (Timeout, RateLimited, ProviderError) as e:
                last_err = e
                continue
        raise AllProvidersFailed(task_type, last_err)

This example intentionally uses a unified abstraction interface such as providers[name].invoke(...): model differences are consolidated into one adapter layer, so upper-level business code does not depend on a specific implementation. This is key to maintaining a multi-model system over time. When models are replaced, only configuration and the adapter layer need to change; business logic remains untouched.

4. Turning Prompt Engineering into Engineering Assets: Four Extension Mechanisms

AI coding tools become productivity tools not because of the chat window, but because of four extensible mechanisms. Understanding their roles matters more than memorizing any specific feature.

4.1 Custom commands: turn high-frequency instructions into templates

Handwritten prompts lead to “prompt drift”: the same requirement may be written with different constraints by different people on different days, making results difficult to compare. Turning high-frequency instructions into templates and checking them into version control gives the team an auditable operating manual:

.claude-ish/
  commands/
    fix-issue.md     # Input: issue ID; steps: locate -> reproduce -> minimal fix -> add tests -> report risks
    review-diff.md   # Input: branch name; output: issues grouped by severity
    gen-migration.md # Input: table schema change; requirement: rollback script + data validation SQL
  agents/
    test-writer.md   # Read-only source code; specializes in boundary tests
    doc-writer.md    # Writes docs only from code facts; prohibits speculation about unimplemented capabilities
  skills/
    sql-style.md     # Team SQL conventions and common templates

4.2 Subagents: context isolation and parallelism

Once the main context is filled with irrelevant content, model attention becomes diluted, especially in the later stages of long tasks. The value of subagents is context isolation: they can read dozens of files for retrieval and return only a summary. Multiple subagents can also run in parallel on independent subtasks, with results merged later.

Practical point: the read-only or writable permissions of subagents should be explicitly declared, and write operations should be consolidated into the main agent as much as possible to simplify review.

4.3 Hooks: turn non-negotiable constraints from suggestions into enforcement

Prompts are suggestions; hooks are gates. Any rule that would cause an incident if violated should be enforced by hooks, including:

• Formatting
• Static checks
• Running tests before commits
• Commit message standards
• Sensitive information interception: preventing secrets, internal addresses, and customer data from being written into files or logs

{
  "hooks": {
    "post_edit":  ["formatter --write", "linter --fix"],
    "pre_commit": ["unit-tests --changed-only", "secret-scanner --staged"],
    "pre_tool":   ["policy-check --deny-network,deny-path:.env"]
  }
}

4.4 Skills and plugins: packaging and distributing domain knowledge

A skill packages domain knowledge, such as coding standards, framework conventions, internal component usage, and troubleshooting manuals, into a searchable and loadable capability package. Plugins go further by bundling these capabilities together with commands and hooks for distribution.

For teams, this means new members can obtain the same level of context as experienced members from day one, instead of relying on manual knowledge transfer.

5. Not Just a Code Generator: Four High-Value Use Cases

The capability boundary of these tools is much smaller than their actual usage range. The following four scenarios often produce higher returns.

1. Long-form writing and content production

Its strengths are structure building, material summarization, multi-version rewriting, and language polishing. Its weakness is factual accuracy.

The correct approach is to provide the material and viewpoints yourself, and let the tool handle organization and expression. All facts, data, and citations must be verified manually one by one.

2. Data analysis loops

Starting from “table schema + business question,” the tool can produce queries, execute them, inspect results, find anomalies, correct metric definitions, and then produce charts and conclusions.

It is recommended to fix the workflow and retain the SQL and intermediate results of each step for review:

-- Requirement: compare retention across channels over the last 30 days,
-- excluding test accounts and abnormal channels
WITH base AS (
  SELECT user_id, channel, MIN(dt) AS first_dt
  FROM events
  WHERE dt >= CURRENT_DATE - 30 AND is_test_account = false
  GROUP BY user_id, channel
)
SELECT channel, COUNT(*) AS new_users
FROM base GROUP BY channel ORDER BY new_users DESC;

3. Workflow automation

Repetitive operations, log aggregation, report generation, and batch configuration changes can all begin with the tool producing script drafts. After human review, the scripts can be incorporated into scheduling systems.

4. Rapid prototyping and visualization

The tool can build locally previewable pages or dashboards, which can then be published to static hosting. General cloud capabilities such as object storage and CDN are sufficient.

Two reminders:

• Credentials should always go through secret management or environment variables. Do not commit them to the repository.
• Before exposing a prototype externally, confirm whether access controls are enabled.

6. Engineering Rules: Keep AI Output in a Controlled Loop

The larger a multi-model system becomes, the larger its risk surface. The following six rules should be treated as an admission checklist:

1. No exemption from review and gates: AI-generated code goes through the same code review and CI process. Do not allow “AI wrote it, so merge it quickly.”
2. Least privilege and sandboxing: limit filesystem scope, network egress, and available credentials. Destructive operations, such as force push, database deletion, or permission changes, require manual confirmation.
3. Dependency source verification: do not blindly execute installation commands suggested by the model. Guard against dependency confusion and typosquatting. Lock versions and verify sources.
4. License and compliance scanning: generated code may reproduce fragments from training corpora, so license scanning should be included.
5. Reproducibility: version-control model versions, prompt template versions, and command, hook, and skill configurations. Otherwise, production issues cannot be traced back.
6. Observability: use logs, monitoring, and alerts to track call volume, cost, failure rate, and average rework rounds. Review these metrics weekly. This is the only reliable basis for deciding whether a model should be upgraded or down-weighted.

7. Rollout Roadmap

Phase	Duration	Key Actions	Deliverables
Pilot	1–2 weeks	Choose one high-frequency task domain and build a 20–50 sample evaluation set	Evaluation set + initial comparison report
Standardization	2–4 weeks	Establish command templates, hook gates, and team skill packages	Distributable capability packages
Multi-model orchestration	4–8 weeks	Implement routing, cross-review, fallback, and budget limits	Adapter layer and routing policy
Scale	Continuous	Cost and quality dashboards, admission checklist, quarterly re-evaluation	Model selection and governance mechanism

8. Conclusion

In a model landscape that changes every few months, treating model selection as a one-time decision is wrong. It should be treated as a continuous mechanism:

• Use self-built evaluation sets instead of intuition.
• Use layered orchestration instead of single-model dependence.
• Use hooks and gates instead of verbal agreements.
• Use skill packages instead of manual knowledge transfer.

For developers, the greatest value of these tools is not simply faster coding, but the fact that they substantially lower barriers such as “I cannot code,” “my English is not good enough,” and “I do not understand frontend engineering.” Data processing, tool building, visualization dashboards, and content production, which previously required collaboration across multiple specialized roles, can now be completed by one person from start to finish.

For teams, the real differentiator is who first turns prompts, constraints, and domain knowledge into reproducible, auditable, and transferable engineering assets.