Evaluating Large Language Models on Code and Long-Horizon Agent Tasks

    This article presents practical methods for evaluating LLMs that must complete code and long-horizon agent tasks rather than only answer questions. It covers end-to-end task completion, reproducibility and pass@k sampling, sandboxed terminal operations, state management, 3D and CAD script generation, financial analysis delivery standards, factual accuracy testing, safety filtering, and cost-latency tradeoffs, with an implementation checklist for building reliable internal benchmarks.

    Large language models are shifting from “answering questions” to “completing tasks.” In the past, model evaluation focused mainly on single-turn question-answering accuracy. Now, more teams expect models to read code repositories, call tools, execute dozens of steps in constrained environments, and deliver directly usable results. Evaluation methods need to change accordingly; a single leaderboard score is no longer sufficient to support model selection decisions.

    1. Evaluation Dimensions for Code and Long-Horizon Agent Tasks

    1.1 Task Completion Rate

    For agents, the core metric is end-to-end completion rate: whether the entire process from receiving a goal to producing a verifiable result succeeds. A correct single-step answer does not mean the overall task is usable, because errors accumulate across the step chain. Teams should split tasks into checkpoints and record whether each checkpoint passes, so they can identify where the model first deviates from the goal.

    1.2 Reproducibility and Multiple Sampling

    The same task may produce different results across runs. A common practice is to sample each task multiple times and calculate pass@k, the proportion of tasks where at least one of k attempts succeeds. A high pass@k indicates that the model has the potential to solve the problem. If the production environment allows only one output, teams should also monitor pass@1. Using both metrics together reflects the real risk.

    1.3 Terminal Operations in Constrained Environments

    Real development often takes place in sandboxes with restricted permissions, such as read-only file systems, no internet access, and execution timeouts. Evaluation should run under the same constraints and record the model’s ability to recover after command failures. Whether the model can read error information and correct parameters reflects engineering ability more than whether it can write a command correctly on the first attempt.

    1.4 State Management and Long Context

    Long-horizon tasks require the model to remember previous decisions and intermediate artifacts. Teams can design tasks that span multiple rounds of reading, modification, and validation to test whether the model forgets early constraints. A larger context window does not automatically solve this problem; summaries, external notes, or structured state files are still needed.

    Example: a simplified evaluation loop

    1. for task in task_set:
    2. for attempt in range(k):
    3. env = create_sandbox(task.image, timeout=600, network=False)
    4. result = run_agent(model, task.prompt, env, max_steps=50)
    5. passed = run_checks(env, task.checkers)
    6. record(task.id, attempt, passed, result.steps, result.cost)
    7. env.destroy()

    This pseudocode reflects three principles. First, each attempt uses a clean environment to avoid state contamination. Second, steps and timeouts are limited so that costs can be compared across models. Third, acceptance logic is implemented as independent checkers rather than relying on model self-evaluation.

    2. 3D Modeling and Engineering Script Generation

    In scenarios such as game development and industrial design, models are often asked to write scripts that drive 3D modeling software or CAD systems to generate parametric models. The acceptance criteria for these tasks are relatively clear: whether geometric dimensions meet constraints, whether the script runs without errors, and whether the model updates as expected after parameters are changed.

    A closed-loop process of “generate, execute, render, and compare” is recommended. After the model outputs a script, the system executes it in an isolated environment and exports geometry data, then calculates metrics such as volume, bounding box, and surface distance against a reference model. If execution fails, the error information should be passed back to the model for further correction. This evaluates end-to-end engineering ability, not just whether the syntax is correct.

    For game engine scenarios, teams also need to account for differences in resource paths, coordinate systems, and unit systems. The same dimension parameter may correspond to different units in different environments, so teams should specify conventions clearly in prompts and add unit validation to checkers.

    3. Delivery Standards for Financial Analysis Workflows

    Financial modeling, investment analysis, and industry research reports are high-risk outputs, and numerical errors can directly affect decisions. Delivery standards should include three levels: data sources must be traceable, calculations must be reviewable, and conclusions must include assumptions and sensitivity analysis.

    In practice, teams can require the model to label the source of each key number and generate an independent calculation table. Reviewers spot-check selected numbers against source materials and formulas item by item. If a metric depends on forecast assumptions, the report should list the assumption ranges and explain how changes in assumptions affect the conclusions.

    Benchmark scores only indicate model performance on specific questions. Teams should build internal test sets based on historical cases in their industry, covering common accounts, accounting standard differences, and anomalous data scenarios.

    4. Test Design for Factual Accuracy

    Factual errors are one of the main obstacles to deploying large models. Two types of metrics are commonly used in evaluation. The first is single-statement error rate, which measures the frequency of fine-grained errors. The second is the proportion of responses containing at least one error, which measures the probability that users actually encounter an error. The two have different meanings and cannot replace each other.

    Test sets should include three types of samples. The first is regular questions, reflecting everyday usage. The second is error-prone questions, such as confusions between similar concepts, unit conversions, and date calculations, which expose weak areas. The third is questions that require external retrieval to answer, which test whether the model fabricates content when evidence is lacking.

    Test sets designed specifically for error-prone cases usually produce higher error rates than everyday usage. When reporting results, teams should note how the test set was constructed to avoid treating stress-test numbers as general performance.

    5. Safety Governance and Data Filtering

    As model capabilities improve, misuse risk also increases. Common industry practices include identifying and removing high-risk content from training data, adding refusal policies during reinforcement learning, and setting access control and content auditing at the deployment layer.

    When evaluating safety measures, teams can compare the pass rate on the same test set before and after filtering. If the unfiltered version has a clearly higher pass rate, filtering has indeed reduced a certain capability. When selecting models, enterprises should focus on the performance of released versions in compliance scenarios and confirm whether the service provider offers audit logs, permission tiers, anomaly alerts, and other supporting mechanisms.

    6. Engineering Tradeoffs Between Cost and Latency

    Large model calls are usually billed by the number of input and output tokens. Long-horizon agents repeatedly read context, so actual costs are often much higher than single-turn question answering. Common cost-reduction methods include using context caching to reuse fixed prefixes, compressing verbose tool outputs, assigning simple subtasks to smaller models, and reserving stronger models for critical steps.

    Accelerated or priority queues usually have higher unit prices and are suitable for latency-sensitive interactive scenarios. Offline batch tasks can tolerate longer waits in exchange for lower unit prices. Teams should tier calls by task type and regularly adjust strategies based on cost reports.

    7. Implementation Checklist

    1. Define acceptance criteria and turn “looks correct” into automatically checkable conditions.
    2. Build an independent test set for each task type and update it regularly.
    3. Record the steps, elapsed time, token usage, and failure reasons for each run.
    4. Sample-check numbers, dates, and citations in outputs.
    5. Run model-generated code and scripts in a sandbox, with permissions and network access restricted.
    6. Set human review nodes for content involving funds, compliance, and external publication.
    7. Re-run the full evaluation whenever the model version changes, and do not use historical conclusions as a substitute for current testing.