Evaluating Code Agent Capabilities in Restricted Offline Environments

    As intelligent coding assistants and code agents evolve rapidly, teams often struggle with accurate evaluation, relying either on marketing labels or quick speed tests. This article introduces a universal evaluation method based on a controlled testing environment—completely offline, time-limited per task, and covering six categories of tasks. It details how to design tasks, standardize execution, apply a multi-dimensional scoring system, calculate cost-effectiveness, and avoid common misinterpretations to assess the actual capabilities and value of code agents rationally.

    With the rapid evolution of intelligent coding assistants and code agents, teams often fall into two extremes during selection: relying solely on marketing “capability labels” or drawing conclusions based purely on completion speed from a short test. A more reliable approach is to place the subject in a unified, reproducible, and auditable controlled environment, using a set of tasks covering real development workflows, while observing correctness, verifiability, knowledge boundaries, time consumption, and manual review costs. This article abstracts a general evaluation method based on a controlled test featuring “fully offline, time-limited per task, and six categories of tasks,” focusing on how to determine actual capabilities and cost-effectiveness.

    I. Clarify the Evaluation Object and Boundaries

    The evaluation object is not the “model name” but the “comprehensive performance of the model within a specific development kit.” The same model can perform differently under varying runtime environments, context organization, and tool permissions. Therefore, evaluation records should at least include: the tested model or service identifier, host development kit type, runtime window, network status, per-task time limit, number of questions, total time, external link access, retry allowance, and manual review method.

    In the original test background, environmental conditions can be abstracted as: fully offline, external links provided in questions are inaccessible; 30-minute limit per question; six questions in total, taking approximately 43 minutes; the runtime environment provides model and knowledge cutoff information via system prompts. These conditions are valuable because they eliminate the “can it search the internet” variable, allowing for a purer observation of the model’s own knowledge, reasoning, code organization, and tool usage abilities. However, short completion time does not equal high quality, and an offline environment does not equal a real production environment. Evaluation conclusions must clearly state boundaries to avoid extrapolating controlled test results into general performance.

    II. Task Set Design: Six Categories Covering “Self-Description, Knowledge, Code, Verification”

    The six questions should not all focus on the same capability dimension. A more reasonable design is to have questions cover different stages and assign weights. The following six categories can be used as a reference:

    1. Environment Self-Description and Consistency Check: Ask the subject “What model are you, what version, and when is your knowledge cutoff?” The key is not the fluency of the answer, but whether the source is clear. If the answer explicitly states “information comes from the runtime environment system prompt, not introspection,” it should be considered good environment consistency; if it packages the system prompt as “self-memory,” points should be deducted. This question should not count toward the main knowledge score but serve as an environmental credibility check.

    2. Knowledge Cutoff and Timeliness Boundary Detection: Ask questions around information near the knowledge cutoff time to observe whether it admits “sparse memory” and distinguishes between certainty and uncertainty. The closer to the cutoff, the more incomplete the model’s memory of related events usually is. Therefore, such questions are suitable for assessing “boundary awareness” but not as the sole evidence of capability. Answers should be required to annotate uncertainty and avoid fabricating details.

    3. Code Understanding and Local Modification: Provide a medium-sized code snippet, requiring the location of a problem and a minimal fix. Scoring focuses include: whether it understands the call chain, maintains interface compatibility, introduces side effects, and provides verification methods.

    4. Cross-File Refactoring and Dependency Management: Require adjusting multiple modules without breaking existing conventions. Such questions can observe context organization, scope of change, and regression risk control capabilities.

    5. Test Generation and Bug Reproduction: Require constructing a failing test case first, then fixing the problem. Scoring should look at whether the test truly covers the defect, can stably reproduce it, and passes verification after the fix, rather than just looking at the final code.

    6. Tool Usage and Log Tracing in Restricted Environments: Under offline conditions, observe whether it reasonably uses local files, runs local tests, and records assumptions and failure reasons. If it cannot access external links, it should explicitly state the limitation rather than pretending to have looked them up.

    A 30-minute upper limit per question is a reasonable “hard budget,” but actual time spent, retry counts, and manual intervention counts should be recorded. The total of ~43 minutes for six questions in the original background only illustrates the time recording method for that execution and cannot directly represent general efficiency; it must be interpreted in conjunction with manual review and rework costs.

    III. Execution Specifications: Isolation, Timing, Tracing, Blind Evaluation

    To make results reproducible, it is recommended to fix the execution flow. Pseudocode is as follows:

    1. for case in cases:
    2. env = isolate(network=False, snapshot=clean)
    3. start = now()
    4. output = run_case(case.prompt, env, time_limit=30min)
    5. elapsed = now() - start
    6. log = collect(env.logs, output, elapsed, retries)
    7. score = judge(case, output, log, rubric)
    8. store(case.id, score, elapsed, log)

    During execution, at least four things must be done: First, use a unified snapshot to prevent residual effects from previous tasks. Second, uniformly enforce offline mode and permissions, clarifying which local resources are available. Third, completely record system prompts, tool calls, failed retries, and manual interventions. Fourth, adopt blind evaluation or dual-person review to avoid inflating scores just because something “looks fast.” For self-description questions, the “original system prompt” and the “model’s answer” should be saved separately to determine whether the information is an environmental declaration or a model inference.

    IV. Scoring Dimensions and Cost-Effectiveness Formula

    Scoring should not just look at “whether it is completed.” It is recommended to set at least six dimensions: Correctness, Maintainability, Verifiability, Boundary Awareness, Robustness, and Total Cost. Correctness checks if functionality is met; Maintainability checks if changes are local and naming is clear; Verifiability checks if tests or reproduction steps are provided; Boundary Awareness checks if it admits knowledge cutoffs and uncertainties; Robustness checks exception input and failure handling; Total Cost checks machine time, manual review, rework, and verification costs.

    Cost-effectiveness can be expressed with a general formula:

    1. Effective Output = Passed Tasks × Task Weight × Manual Review Coefficient
    2. Total Cost = Machine Time Cost + Manual Review Cost + Rework Cost + Verification Cost
    3. Cost-Effectiveness = Effective Output / Total Cost

    The manual review coefficient is used to penalize results that “appear complete but require significant rework.” If a test takes a very short time but manual review finds multiple hidden errors, the total cost is actually higher. Therefore, teams should record both the “first-pass rate” and “post-review pass rate” during selection, and separately track rework time. For knowledge cutoff questions, the proportion of “admitting uncertainty” should also be recorded to avoid mistaking fabricated details for broad knowledge.

    V. Common Misinterpretations in Results Analysis

    First, treating self-description as introspection. When the subject answers “what version I am, when is the knowledge cutoff,” it likely comes from the runtime environment system prompt, not the model’s actual internal memory. Evaluation reports should clearly annotate the source and not package system prompts as model capabilities.

    Second, treating short completion time as high efficiency. Under offline, time-limited, and small question set conditions, a model might quickly provide a seemingly complete answer but lack verification. Executable verification or manual review must be required, otherwise time consumption data has no decision-making significance.

    Third, treating time-sensitive questions as the main score. Memory of information near the knowledge cutoff is usually sparser, and the model might only remember scattered fragments. Such questions are suitable for boundary detection but not as the core capability score.

    Fourth, treating a single result as a general conclusion. Controlled tests should fix the question set, environment, and scoring rubric, and retain original logs. Only after multiple runs and cross-task retests can stability be discussed.

    VI. Implementation Recommendations Checklist

    1. Establish a fixed question set and weights, distinguishing between environment check questions, knowledge boundary questions, code implementation questions, and verification questions.
    2. Record system prompts, tool logs, time consumption, retries, and manual interventions throughout to ensure auditability.
    3. Adopt blind evaluation and dual-person review, checking verification results before reading explanatory text.
    4. Require certainty annotations for knowledge cutoff questions, encouraging “I don’t know” over fabrication.
    5. Include rework, review, and verification costs in cost-effectiveness, not just per-task time.
    6. Clearly state environmental boundaries in the report: offline, time limits, question volume, and whether external links are allowed.
    7. Desensitize and isolate tests involving production code to prevent sensitive information from entering logs.
    8. Use test results as one reference for selection, not the sole basis; make comprehensive decisions considering team processes, maintenance costs, and compliance requirements.

    Overall, the key to evaluating intelligent coding assistants is not “how strong it says it is,” but “whether it can stably produce verifiable, maintainable, and reviewable results under clear boundaries.” By using a unified environment, six categories of tasks, complete logs, and a comprehensive cost formula, a short test can be transformed into a reusable technical judgment basis, thereby more rationally assessing actual capabilities and cost-effectiveness.