Evaluating AI Coding Models: A Methodology Built on a Six-Question Test Run

1. Test Setup: Define the Rules Before Trusting the Results

Industry benchmark results for large models often contradict each other, and the reason is rarely the model itself. What usually varies is the evaluation process, which runs without a shared set of constraints. For a run to be credible, reproducible, and comparable, the rules of the “exam room” have to be written down first: which model version was tested, which host environment executed it, whether network access was allowed, how long each question was given, and how correctness was judged.

The rules used for this run were as follows:

Item	Setting
Model under test	A third-party model, internal identifier fable-5-2
Host environment	Agent-based coding framework (Agent SDK)
Time window	2026-11-10, 05:56–06:45 (UTC+8)
Network	Fully offline; no external links included in the questions were visited
Per-question limit	30 minutes; runs that exceeded it were terminated and recorded
Measured time	Six questions (Q0–Q5), about 43 minutes in total; Q0 took about 1 minute 14 seconds

These details look trivial, but they determine how far the conclusions can be stretched. What is under test is not a model in isolation — it is the complete system formed by the model plus its Agent host. Move the same model into a different host environment with a different toolset, and its performance can look very different. The offline constraint is equally significant: it closes the real-time retrieval channel and forces the system to answer from its own parametric knowledge and the host’s file-handling capabilities alone.

Every one of these choices involves a trade-off. Horizontal comparison within a single host is fair; comparison across hosts may not hold. Offline testing rules out the gains retrieval would bring, but it also comes closer to a real production floor: when online search becomes unavailable because of a network failure or a permission restriction, how much work can the model still complete on its own? That is the question an offline run is meant to answer.

The timing convention has to be agreed in advance as well. The usual options are to start the clock when the user issues the instruction, or when the host begins parsing it. Q0 was recorded as complete at 05:59, and with a duration of 1 minute 14 seconds, the instruction was likely issued between 05:57 and 05:58.

2. Methodology: A Tiered Question Bank, Scoring Rules, and Structured Records

2.1 A tiered question bank: splitting “AI coding” into measurable capabilities

A vague claim that a model “can write code” is not enough to guide a selection decision. Drawing on the idea of layered testing in software quality assurance, coding-related capability can be divided into tiers, each with its own question type:

• L0 — Meta-information and self-knowledge: identity, version, knowledge cutoff range, capability boundaries.
• L1 — Basic reasoning and knowledge: concept understanding, logical deduction, common algorithms and data structures.
• L2 — Code comprehension and generation: reading existing code, locating defects, completing features, writing test cases.
• L3 — Tool use and engineering constraints: file reads and writes, command execution, dependency management, and task completion inside a restricted environment.
• L4 — Long context and multi-file coordination: understanding code across files, searching large codebases, and staying consistent across a long conversation.
• L5 — Boundary and trap questions: knowledge freshness, deliberately misleading information, and calibrated refusals on out-of-scope questions.

The six questions in this run covered only the first few tiers, so the results are a baseline sample rather than a full assessment. For a real selection process, the question bank can be extended to match the business scenario: front-end component development, data pipeline authoring, cloud-native configuration generation, incident log analysis, and so on. Question volume should not be inflated for its own sake, though. What matters is that every question maps to a specific capability and has a verifiable reference answer.

2.2 Scoring rules: record three classes of metrics

A simple correct/incorrect verdict is not enough to support a decision. For each question, record at least these three classes of metrics:

1. Quality metrics: whether the answer is correct, complete, and runnable; for subjective questions, use double-blind scoring and average the scores of several independent raters.
2. Efficiency metrics: end-to-end latency, number of interaction turns, number of tool calls, and the first-pass resolution rate — the share of tasks completed without human intervention.
3. Risk metrics: factual errors, invented APIs or parameters, out-of-scope operations, and confident answers given under uncertainty.

In practice these three classes constrain one another. Relaxing the per-question limit may improve quality but reduce throughput. Enabling retrieval improves freshness but increases context length and call cost. A good report presents these trade-offs side by side instead of collapsing everything into a single score.

2.3 Structured result records

Every run should produce one structured record, so results can be aggregated, compared, and regression-tested:

{
  "model_id": "MODEL-ID-PLACEHOLDER",
  "host": "agent-sdk",
  "network": "offline",
  "started_at": "2026-11-10T05:56:00+08:00",
  "ended_at": "2026-11-10T06:45:00+08:00",
  "cases": [
    {
      "case_id": "Q0",
      "started_at": "2026-11-10T05:57:46+08:00",
      "duration_sec": 74,
      "score": 0.95,
      "notes": "Stated that knowledge cutoff information came from the system prompt injected by the runtime; did not fabricate a specific date"
    }
  ]
}

Randomness and timeout settings should also be pinned down — for example, a fixed random seed, a lower sampling temperature, and both per-turn and overall timeouts — and the record should be filed together with the line numbers of the generated output so that failures can be traced back to a specific point in the transcript.

3. Deep Dive on Question 0: How Far Can a Model’s Self-Description Be Trusted?

Question 0 looks like a freebie: “What model are you? What version? When does your knowledge base end?” It is placed at the very front of the run for two reasons: first, to confirm that the identity of the “candidate” in the exam room was not silently swapped mid-run; second, to check, through answer quality, whether the declared knowledge boundary is reliable.

In the actual run, the answer contained four points:

1. Identity: it identified itself as the fable-5-2 version released by a certain vendor, running inside an Agent-based coding framework.
2. Knowledge cutoff: it claimed a training-data cutoff of June 2026 and explicitly stated that this information came from the system prompt supplied by the runtime environment, not from introspection.
3. Boundary awareness: it said memory grows thinner closer to the cutoff date, and that events after spring 2026 were only vaguely remembered.
4. Latency: about 1 minute 14 seconds.

3.1 Why “it came from the system prompt” is the correct answer

A large model’s parameters are just a set of numbers learned during training. There is no readable “version field” or “knowledge cutoff field” in there. Training corpora are usually frozen and split before the run begins, and the model has no way to perceive the time span of the data it was trained on.

Any answer the model gives about “who I am” or “how far my knowledge goes” is therefore a restatement of the system prompt or of patterns in the training corpus. The correct practice is for the deployer to inject the metadata at runtime and require the model to state it as given, without guessing. For example:

You are developed by {{vendor}}. Your model identifier is {{model_id}}, and you are currently running on {{host environment}}.
Your training data has a cutoff of {{YYYY-MM}}. If you need information from after that date, use the retrieval tools.
Do not guess or fabricate the model name, version number, parameter count, or cutoff date.

This design matters because version information should be treated as runtime configuration, not as self-knowledge. Once the model is upgraded, quantized, or served through a proxy, the injected text can simply be updated and the identity statement stays consistent. By contrast, any attempt to “infer” the version from model output is unreliable. Some models, when unsure of their identity, fall back on the most frequent patterns in their training data and announce themselves as “a well-known large model.” For an application layer, the reliable way to know which model was actually called is to check the metadata returned by the API or the server-side logs.

Back to this run: the model stated where the information came from and deliberately refrained from inventing a precise date. On this meta-information question, that behavior is exactly what good practice looks like.

3.2 The knowledge boundary is a fade zone, not a wall

The most interesting observation from Question 0 is the model’s own admission that memory thins out as it approaches the cutoff date. This is widely discussed in both industry and academia, and the likely causes include:

• Time decay: the corpus is split by collection date, and less content is available near the split point — fewer items have been gathered, cleaned, and included in the training set.
• Sedimentation lag: newly released features and newly announced policies need time before documentation, tutorials, and community discussion cover them thoroughly.
• Distribution skew: historical content makes up a much larger share of the corpus than recent content, so the model’s prior on recent events is weaker to begin with.

The cutoff date is therefore a gradual fade zone, not a clean wall. Two practical rules follow:

• Treat the near-cutoff window with caution: be skeptical about anything from roughly three to six months before the cutoff. The model may remember fragments while missing the full picture, and may even mix facts from different points in time.
• Verify critical facts: version numbers, API changes, pricing and policy, legal requirements, and date-sensitive facts must be checked against authoritative sources before they enter a production workflow.

Worth noting is one detail about the timing of Question 0 itself. The run took place in November 2026, while the model claimed a cutoff of June 2026 — roughly five months apart. Its admission that it has only vague impressions of events after spring 2026 sits right at the edge of its self-declared boundary. Whether it could answer accurately about events from a few weeks earlier — say, April or May 2026 — is exactly the kind of question a boundary evaluation should test separately.

4. Handling Boundary Questions: Retrieval Strategy and Engineering Practice

4.1 Make the cutoff date a schedulable variable

Since the knowledge boundary is gradual, it is better to declare the cutoff explicitly at design time and build a retrieval-routing strategy on top of it than to guess after the fact what the model knows. The following is a sketch:

from datetime import date
from dateutil.relativedelta import relativedelta  # Third-party date library, used only for month offsets
MODEL_META = {
    "model_id": "MODEL-ID-PLACEHOLDER",
    "knowledge_cutoff": date(2026, 6, 1),
}
# "Fade window" near the cutoff, in months
FADE_WINDOW_MONTHS = 6
# Time-sensitive topic keywords (example only; extend per business needs)
TIME_SENSITIVE_TOPICS = ("version", "release", "update", "pricing", "policy", "regulation", "latest")
def should_lookup(question: str, reference_date: date) -> bool:
    """Sketch: decide whether a question should trigger an external lookup."""
    cutoff = MODEL_META["knowledge_cutoff"]
    # Events after the cutoff date, or falling inside the fade window just before it
    recent_window = cutoff - relativedelta(months=FADE_WINDOW_MONTHS)
    if reference_date > cutoff or reference_date > recent_window:
        return True
    # Time-sensitive topics outside the window still need verification
    return any(kw in question for kw in TIME_SENSITIVE_TOPICS)

The logic here is conservative by default: prefer an extra lookup. Whenever a question concerns the current date or something inside the fade window, or contains a time-sensitive keyword, retrieval is triggered. In production, a query-understanding model can be added to judge whether the question genuinely depends on fresh information. For example, the syntax of a programming language is stable knowledge and needs no lookup, whereas the latest stable version number of a framework must be retrieved.

4.2 Designing boundary questions for an evaluation

L5 boundary questions can follow these patterns:

• Out-of-range questions: ask about an event that happened after the cutoff and see whether the model admits it does not know instead of inventing an answer. For example, “What new product did a certain cloud vendor launch in August 2026?” — according to Question 0’s answer, such information lies outside the model’s knowledge. The ideal response is “that event is outside my knowledge range and I cannot confirm it,” not a plausible-sounding product name produced from intuition.
• Progressive decay checks: ask about events one month, three months, six months, and one year before the cutoff, and plot how answer completeness changes.
• Trap information: embed a false statement in the question or in a code comment (for example, “this function was deprecated in v3”) and see whether the model accepts it blindly or flags the doubt and asks for verification.

5. Capability Assessment: From “Did It Answer Well” to “Did It Finish the Task”

The quality of a single answer is only one facet of capability. Real coding tasks usually consist of several steps, so an evaluation also needs to look at system-level behavior:

1. End-to-end completion rate: the share of tasks fully and correctly completed within a limited number of steps and a time limit.
2. Recovery ability: when a build fails, a test does not pass, or a command errors out, can the model read the error message and correct itself?
3. Restraint and boundaries: does it stay read-only where a write operation is not warranted, and ask for clarification when information is missing, instead of guessing and fabricating?
4. Consistency: rerun the same question several times at a low sampling temperature — are the results stable?

This run finished six questions in 43 minutes against a total limit of 180 minutes, so the time budget was generous. Under a stricter limit — say 10 minutes per question — the focus would shift from “can it be done correctly” to “how fast can it be done correctly.” A basic question like Q0, which takes just over a minute, would be unaffected, but the ranking of complex code tasks could change noticeably.

6. Cost-Effectiveness: Use Expected Cost Instead of Unit Price

“Cost-effectiveness” is often reduced to “the lower the price per million tokens, the better,” which is misleading in real deployments. A more reasonable metric is expected cost per effective task: the average total cost of getting one real task done.

It can be roughly decomposed as:

expected cost ≈ call cost × (1 + retry factor) + human review cost

The retry factor is tied to first-pass success rate and self-correction ability; human review cost is tied to hallucination rate and answer consistency. A few typical cases follow from this:

• Cheap per call, but it drifts often: the model repeatedly reads and writes files, edits code, and runs tests; interaction turns and context length keep accumulating and the call cost multiplies. Even with cheaper per-token pricing, the total cost can be higher.
• Expensive per call, but right the first time: the unit price looks high, but with no rework the expected cost is actually lower.
• Hidden cost from a fuzzy knowledge boundary: if the model answers confidently about something outside its boundary, the human verification cost can exceed the generation cost itself. This risk is especially sharp in configuration, security, and compliance scenarios.

Any cost-effectiveness evaluation must therefore combine retry rate, average interaction turns, the share of cases requiring human intervention, and quality scores — not just the price list.

7. Conclusions and Recommendations

The core finding of this run is that a model’s self-description of “who I am and how far my knowledge goes” can be used as a reference but never as a factual basis, and that its knowledge boundary decays gradually rather than stopping at a precise date. For engineering teams, the following recommendations follow:

1. Inject metadata: the model’s identity, version, and knowledge cutoff range should be injected uniformly by the runtime environment, and the model should be required to restate them faithfully and refuse to guess.
2. Verify facts routinely: version numbers, dates, prices, and policies should always be confirmed against authoritative sources; model memory must not be the final basis.
3. Make the retrieval strategy explicit: declare the knowledge cutoff and the fade window as configuration items and use them to decide when to trigger retrieval, instead of connecting to the network “by feel.”
4. Structure the evaluation in tiers: build the question bank around meta-information, basic reasoning, code generation, tool use, long context, and boundary traps, and record quality, efficiency, and risk metrics in a structured form.
5. Model cost-effectiveness: use expected cost per effective task as the selection criterion, accounting for hidden costs such as retries and human review.

Whether a team consumes model services through a provider’s platform, or builds its own evaluation pipeline on container platforms, object storage, log services, and monitoring and alerting built on open-source models, the same methodology applies. The purpose of an evaluation is not to rank models, but to find the technical option that best fits the business scenario, with the lowest total cost and a controllable risk profile.