How to Evaluate Large Language Model Coding Ability: Time, Cost, and Quality

    A structured framework for evaluating coding-focused large language models through execution time, token usage, code efficiency, delivery quality, and repeatable testing. The article also explains how to design composite engineering tasks, standardize data collection, and avoid common interpretation errors.

    1. Why Structured Evaluation Matters

    A common way to evaluate coding models is to ask them to write a piece of code and then check whether it runs. This only answers whether the model can complete the task. It does not show how quickly it works, how many resources it consumes, or how much rework is required. These factors are just as important as functional correctness when a team is selecting tools or estimating costs.

    The goal of structured evaluation is to break subjective impressions into measurable and reproducible metrics. Testers should define the task boundaries, standardize input conditions, record process data, and repeat the evaluation under the same conditions. Only then can the results be reproduced and independently verified.

    2. Four Core Metrics

    2.1 Task Duration and Generation Speed

    Task duration is the wall-clock time from issuing the instruction to obtaining a runnable result. It consists of three parts:

    • Model generation time
    • Waiting time
    • Correction time

    Looking only at total duration can hide the underlying cause. A slower model may have low single-pass throughput, or it may generate a correct result on the first attempt and require less rework. Therefore, average output speed, measured in tokens per second, should be recorded together with the number of rework rounds. These metrics help explain where the total time comes from.

    2.2 Token Usage Structure

    Total token processing usually includes input tokens, output tokens, and cached tokens read from context. Looking only at the total can lead to inaccurate cost estimates. In coding tasks, existing code is often included repeatedly in the context. When the cache hit rate is high, actual billing and computational overhead can decrease significantly.

    It is recommended to break token usage into the following categories:

    • Total processed tokens: The sum of input and output tokens, reflecting the overall scale of the task.
    • Cache ratio: The proportion of cache-hit tokens in the total processed tokens, reflecting context reuse efficiency.
    • Output tokens: Newly generated content from the model, which is most directly related to the delivered result.

    2.3 Effective Code Ratio

    The effective code ratio is the amount of code retained in the final deliverable as a percentage of all code generated by the model. It can be calculated as follows:

    1. Effective code ratio = Final retained lines of code ÷ Total generated lines of code × 100%

    This metric reflects how much generated code was later discarded. A high ratio generally indicates a clear development path and appropriately sized changes at each step. A low ratio is often associated with repeated trial and error or large-scale rewrites.

    However, deleted code is not necessarily useless. Refactoring may simplify an implementation and reduce the proportion of code retained from the original output. The effective code ratio should therefore be interpreted together with rework records.

    2.4 Delivery Quality

    Quality should be divided into individually verifiable criteria, such as:

    • Whether all required functionality is implemented
    • Whether edge cases are handled
    • Whether interaction feedback is smooth
    • Whether obvious defects remain

    Each criterion can be scored as passed, partially passed, or failed. It is also important to distinguish critical defects from acceptable imperfections. A deliverable missing a core feature should not be considered equivalent to one that has only minor visual inconsistencies.

    3. Designing Composite Test Tasks

    The choice of test task directly affects how well an evaluation differentiates between models. A single algorithm problem generally tests only syntax and logical reasoning. Composite tasks involving physical interaction, state management, and frontend animation can expose weaknesses in longer engineering workflows, such as missing state transitions, numerical instability, or inconsistencies between the interface and underlying logic.

    Consider a simulated claw-machine game. Typical requirements may include:

    1. A controllable claw mechanism that supports horizontal movement.
    2. State transitions for lowering the claw, grabbing an object, raising the claw, and returning it.
    3. A swinging suspended cable whose motion gradually decays over time.
    4. Collision detection and drop handling between the claw and objects.
    5. Scoring rules and an end condition.

    State transitions are one of the most error-prone parts. A state machine can first be used to define valid transitions and then serve as the acceptance baseline:

    1. State set: IDLE, DESCENDING, GRABBING, ASCENDING, RETURNING, SCORED
    2. Valid transitions:
    3. IDLE -> DESCENDING
    4. DESCENDING -> GRABBING
    5. GRABBING -> ASCENDING or IDLE (return to the ready state if grabbing fails)
    6. ASCENDING -> RETURNING
    7. RETURNING -> SCORED or IDLE
    8. SCORED -> IDLE
    9. Handling invalid transitions:
    10. Raise an exception and record a log entry. Do not perform a silent transition.

    Cable swinging can be described using a damped pendulum model. The following pseudocode illustrates the basic form of a single-step update:

    1. function step_pendulum(angle theta, angular velocity omega, time step dt,
    2. cable length length, damping coefficient damping):
    3. gravitational acceleration g = 9.8
    4. angular acceleration alpha = -(g / length) * sin(theta) - damping * omega
    5. omega = omega + alpha * dt
    6. theta = theta + omega * dt
    7. return theta, omega

    During acceptance testing, three aspects can be checked:

    1. Whether the swing amplitude decreases over time.
    2. Whether energy is approximately conserved when damping is zero.
    3. Whether the numerical values diverge under extreme inputs.

    These checks can be implemented as automated test cases to reduce the subjectivity of manual evaluation.

    4. Data Collection and Measurement Consistency

    Whether results can be compared depends on whether the measurement method is consistent. The following rules are recommended:

    • Use the same prompt and the same initial project state for each run of a given task.
    • Record the start and end timestamps, and specify whether manual waiting time is included.
    • Use server-returned token data or platform exports whenever possible. If the values are estimates, clearly label them as such in the report.
    • Subscription quotas, credits, and usage-based billing cannot be converted directly into one another. List the original units separately and explain the conversion method.
    • Repeat each experiment at least three times and use the median to reduce the impact of random variation.

    Results should be summarized in a standardized table for easier comparison:

    Approach Duration (minutes) Total Processed Tokens Cache Ratio Output Tokens Effective Code Ratio Delivery Score
    Approach A
    Approach B

    The values in the table should be filled in using actual test results. Data that has not been measured should not be entered in advance.

    5. Common Mistakes When Interpreting Results

    1. Treating a single result as stable capability. Model outputs are probabilistic, so one strong or weak result is not enough to establish a general conclusion.
    2. Equating slower speed with lower efficiency. A slower approach may require fewer rework rounds and therefore finish the task faster overall.
    3. Comparing only total token usage. Differences in caching can produce much larger cost differences than differences in total token volume.
    4. Replacing scoring criteria with subjective impressions. Without predefined standards, conclusions are difficult for others to accept or reproduce.
    5. Ignoring the effects of prompts and task difficulty. The same model may perform very differently under different prompts. The report should document the input conditions.

    6. Practical Recommendations

    Teams can formalize this method as an internal evaluation process.

    First, build a benchmark task set covering representative business scenarios, such as API development, data processing, and interface interaction. Second, repeat the evaluation whenever the model or service version changes to track performance trends. Third, place time, cost, and quality in the same table and calculate a weighted overall score based on business priorities, preventing any single metric from dominating the decision. Fourth, code involving financial transactions, permissions, or data security should always undergo human review, regardless of its evaluation score.

    7. Conclusion

    The core of evaluating coding-focused large language models is to turn the questions of “How fast is it?”, “How resource-efficient is it?”, and “How good is the result?” into data that can be collected and recalculated.

    Task duration, token usage structure, effective code ratio, and delivery quality provide complementary perspectives and produce a more reliable assessment than a single hands-on trial. Consistent measurement standards and repeated validation are essential for trustworthy conclusions.