LLM Coding and Vision Evaluation: A Practical In-House Framework

    This article explains how teams can build a practical evaluation system for large language model coding and vision/frontend tasks. It covers task definition, evaluation set construction, scoring dimensions for coding and visual tasks, statistical interpretation, cost and latency considerations, and operational recommendations for continuous evaluation.

    1. Why Build an In-House Evaluation System

    Model versions update frequently, and aggregate scores on public leaderboards reflect only general capabilities. They are difficult to map directly to specific business scenarios. Teams commonly face two problems when selecting models: looking only at a single score and ignoring differences in task distribution, or drawing conclusions from a few demo examples with insufficient sample size and unstable results.

    An effective evaluation should answer three questions: How does the model perform on our tasks? How wide is the performance variance? And how much engineering cost will increase if we switch to another model? The following sections address these three questions.

    2. Defining Tasks Before Evaluation

    The first step in evaluation is to break business requirements into measurable tasks. Common coding-related tasks include:

    • Generating functions or modules from natural language requirements
    • Locating and fixing defects in existing code
    • Adding unit tests for existing functions
    • Generating frontend pages from design mockups
    • Generating charts or data visualization code from data
    • Generating descriptions and implementation code for 3D scenes or visual effects

    Each task category needs a clear input format, output format, and success criterion. For example, the output of a function-generation task should be source code that can be executed directly and passes the agreed test cases. The output of a page-generation task should render in a standard browser and match the design mockup in layout and interaction.

    The more specific the success criterion, the more stable the subsequent scoring. If the criterion is written as “good code quality,” different reviewers will give very different scores. If it is written as “passes all test cases and runtime does not exceed the specified limit,” the result can be counted automatically.

    3. Building the Evaluation Set

    The quality of the evaluation set determines how credible the conclusions are. Samples usually come from three sources:

    1. Anonymized real tickets or requirements. These are closest to the business but require handling privacy and compliance issues.
    2. Manually written questions from the team. These can cover edge cases and scenarios that have caused problems in the past.
    3. Subsets of public benchmarks that are close to the team’s own tasks. License terms should be confirmed before use.

    The evaluation set should be stratified by difficulty, for example easy, medium, and hard. Stratification makes it possible to observe performance curves across difficulty levels instead of getting only an average. The number of samples in each category should preferably be at least dozens. When there are too few samples, random variation from a single run may exceed the true difference between models.

    Data contamination also needs attention. If evaluation questions have already appeared in a model’s training data, scores will be overestimated. Countermeasures include using recently written, unpublished questions; rewriting public questions while preserving the test logic; and periodically replacing some samples.

    4. Scoring Dimensions for Coding Tasks

    Coding tasks should be scored across several dimensions, with each dimension counted independently to avoid merging everything into a vague total score.

    Functional correctness is the most basic metric. It is usually measured by unit test pass rate, and it is necessary to distinguish between “runs” and “produces correct results.”

    Runnability means whether the generated code passes syntax checks and dependency resolution. Many failure cases come from referencing nonexistent functions or missing necessary import statements. Such errors should be counted separately.

    Edge-case handling means whether the model handles empty inputs, extreme values, and unexpected types reasonably. The test set should include cases designed specifically for this purpose.

    Modification cost means how much of the model output requires manual correction. It can be measured by the number of changed lines before and after correction, or by the time spent on manual fixes. Modification cost often has a more direct impact on real productivity than pass rate.

    Below is a simple execution framework for running model-generated code and corresponding tests in an isolated directory:

    1. import os
    2. import subprocess
    3. import tempfile
    4. def run_unit_tests(code: str, test_code: str, timeout: int = 10) -> bool:
    5. """Run model-generated code and its tests in a temporary directory; return whether all tests pass."""
    6. with tempfile.TemporaryDirectory() as workdir:
    7. src_path = os.path.join(workdir, "solution.py")
    8. test_path = os.path.join(workdir, "test_solution.py")
    9. with open(src_path, "w", encoding="utf-8") as f:
    10. f.write(code)
    11. with open(test_path, "w", encoding="utf-8") as f:
    12. f.write(test_code)
    13. try:
    14. result = subprocess.run(
    15. ["python", "-m", "pytest", "-q", test_path],
    16. cwd=workdir,
    17. capture_output=True,
    18. text=True,
    19. timeout=timeout,
    20. )
    21. except subprocess.TimeoutExpired:
    22. return False
    23. return result.returncode == 0

    It is important to emphasize that executing model-generated code carries security risks. In production environments, the execution environment should be placed in a container or sandbox, with external network access disabled, CPU, memory, and filesystem write scope limited, and the environment destroyed after execution. The example above only demonstrates the workflow and should not be used directly for untrusted code.

    5. Scoring Methods for Vision and Frontend Tasks

    Frontend page and visual effect tasks rely more on human judgment than coding tasks, but a standardized process can still reduce subjective bias.

    The recommended process is as follows:

    • Use a headless browser to render the page at a fixed viewport size and save a screenshot, avoiding differences caused by different environments.
    • Score each item against a rubric, such as layout fidelity, whether text and spacing match the design mockup, whether interactive controls are usable, and whether errors appear in the console.
    • Use blind review, hide model names, and randomize display order.
    • Have at least two reviewers score each sample independently. When disagreement is significant, use a third reviewer for arbitration and record the reason.

    “Looking good” is a subjective impression and difficult to compare directly. Only after impressions are broken down into checkable items do review results become useful. For example, “whether the alignment error between the button and the container is within an acceptable range” can be judged clearly, while “overall aesthetic quality” is difficult to standardize.

    3D scene and motion tasks also require additional technical checks, such as whether the geometry is valid, whether the frame rate meets requirements, and whether the resource size exceeds the budget. These metrics can be collected automatically by scripts.

    6. Result Statistics and Interpretation

    Large model outputs are stochastic, and a single run is not enough to represent a model’s stable performance. It is recommended to run each sample multiple times and report both the mean and standard deviation.

    When comparing different models, the following variables need to be fixed: prompt template, system instructions, sampling temperature, maximum output length, and context content. When these variables differ, comparison results are meaningless.

    The following code shows the process of running multiple times and calculating the mean and standard deviation of the pass rate:

    1. from statistics import mean, stdev
    2. def evaluate(model_call, cases, repeat: int = 3):
    3. """model_call is a function that calls the model and returns a code string, implemented by the user."""
    4. pass_rates = []
    5. for _ in range(repeat):
    6. passed = 0
    7. for case in cases:
    8. code = model_call(case["prompt"])
    9. if run_unit_tests(code, case["tests"]):
    10. passed += 1
    11. pass_rates.append(passed / len(cases))
    12. return mean(pass_rates), stdev(pass_rates)

    Two points should be noted when interpreting results. First, if the difference in means between two models is smaller than the standard deviation, it is usually not possible to conclude that one is clearly better. Second, results should be examined separately by difficulty tier. A model leading on easy tasks does not necessarily lead on hard tasks.

    7. Cost and Latency Considerations

    Model invocation cost is usually determined by both input and output token counts. During evaluation, both should be recorded, along with the response time of each call.

    A lower unit price does not mean lower total cost. If a model has a lower unit price but requires more retries to achieve the same correctness rate, or its output length is significantly longer, actual spending may be higher. It is better to use “average cost to complete one task” as the metric instead of comparing only unit prices.

    Latency also needs attention. For interactive scenarios, time to first token and total output time both affect user experience. For batch processing scenarios, throughput and failure retry strategy are more important.

    8. Implementation Recommendations

    It is recommended to make evaluation a continuously running pipeline rather than a one-time model selection effort. Specific practices include:

    • Put the evaluation set under version control and record the reason for every change.
    • Automatically trigger regression evaluation when model versions are updated or prompts change.
    • Use canary releases before launch, observing real performance first in a small amount of traffic or in internal scenarios.
    • Keep a human review channel, especially for scenarios involving code merges, production configuration, or user-visible content.
    • Save evaluation results together with decision records so that selection rationale can be traced later.

    Common pitfalls should also be avoided. Looking only at aggregate scores hides differences in task structure. Using individual cases to represent overall performance amplifies random factors. Ignoring prompt differences removes the basis for comparison. Evaluation set leakage inflates scores. These problems are common in initial evaluations, and establishing a standardized process can significantly reduce their impact.