Evaluating and Selecting Medium-Scale Open-Source Code Models with Dual-Axis Evaluation and Complex-Task Stress Testing
This article explains how to evaluate and select medium-scale open-source code models using two evaluation axes—functional completion and behavioral performance—combined with complex-task stress testing. It covers verifiable task sets, behavioral guardrails, 3D front-end acceptance criteria, performance measurement, failure classification, and reproducible evaluation pipelines.
Evaluating and Selecting Medium-Scale Open-Source Code Models with Dual-Axis Evaluation and Complex-Task Stress Testing
For open-source models in the billions-of-parameters range, code capability is often reduced to a leaderboard score or whether the model can generate a piece of runnable code. In real engineering selection, however, the question is broader. A model may produce a functional prototype while repeatedly failing on instruction following, format constraints, multi-turn correction, and complex scenario generation. It may perform well on simple function problems but be unable to support front-end tasks that involve interaction, rendering, and performance constraints. For this reason, evaluating such models should use a methodology of “dual-axis evaluation + complex-task stress testing” rather than relying on a single-point result.
1. Start with Clear Evaluation Goals: Functional Completion and Behavioral Evaluation Are Both Required
Functional completion measures whether a model can produce an implementation that meets acceptance criteria for a given task. It is usually composed of a set of task cases, explicit acceptance conditions, and weighted scoring. Behavioral evaluation focuses on whether the model works according to instructions, including format compliance, tool invocation, error correction, context retention, boundary-condition handling, and output maintainability. Only by combining the two can we judge both “whether it can be used” and “whether it is reliable.”
Public evaluation trends suggest that, for medium-scale models, functional completion often improves gradually from just over 60% to around 70%, while behavioral evaluation improves from below 50% to around 60%. This range is meaningful: when functional completion approaches 70% and behavioral evaluation approaches 60%, a model has usually just crossed the lower bound of “usable for assisted coding.” It can handle small tasks with clear boundaries, but still requires human review, supplementary tests, and architectural constraints. If the behavioral score lags clearly behind the functional score, the model can write code but is not controllable, and it is not suitable for direct use in a production pipeline.
Therefore, selection should not ask only “What is the functional completion rate?” It should also ask: Which types of failure cases are concentrated in? Does behavioral evaluation cover format, tools, and multi-turn correction? Can the scores be reproduced on a private task set?
2. Functional Completion: Replace Subjective Impressions with Verifiable Task Sets
The core of functional completion is reproducibility. A task should be divided into four parts: requirement description, input constraints, acceptance checklist, and weights. The acceptance checklist should be automated where possible, for example interface existence, return structure, boundary conditions, exception handling, and static checks. For front-end tasks, interaction behavior and performance sampling should also be included. Scoring should not count only whether code was generated; it should count whether the code passes acceptance.
A general evaluation record structure can be abstracted as follows:
Task specification:name: complex scenario generationacceptance:- The time parameter slider can control lighting and sky tint- The fog intensity slider has a valid range and does not produce pure white overexposure- The water surface has waves and specular reflection- Distant scenery and water surface blend naturally through distance fog- Frame sampling time is stable at a fixed resolutionweights:functionality: 0.45interaction: 0.20visual: 0.20performance: 0.15Evaluation process:Run fixed prompts and fixed initial context for each caseRecord passed items, failed items, and human review conclusionsAggregate weighted scores and retain failure samples
A functional completion rate of around 70% does not mean that 70% of scenarios can go directly into production. A more reasonable interpretation is that, in a controlled task set, about 70% of acceptance items can pass, while the remaining 30% require manual correction or additional context. For assisted coding scenarios, this level can support scaffold generation, local refactoring, test completion, and implementation of clearly defined interfaces. For complex 3D visualization, cross-module refactoring, and high-reliability backend logic, staged validation is still required.
3. Behavioral Evaluation: Determines Whether a Model Can Enter the Engineering Loop
Behavioral evaluation is often underestimated, but it directly determines whether a model can enter the engineering loop. At minimum, five types of behavior should be covered. First, instruction following, including output format, file structure, and prohibited modification scope. Second, multi-turn correction: whether the model can locate a problem based on failure information rather than repeatedly regenerating. Third, tool use: whether the model can invoke retrieval, compilation, testing, and log analysis capabilities as agreed. Fourth, boundary awareness: whether the model proactively clarifies missing dependencies or ambiguous requirements. Fifth, maintainability: whether generated code has clear naming, error handling, and necessary comments.
When behavioral evaluation approaches 60%, a model can usually complete multi-step tasks under explicit constraints, but it still tends to deviate when requirements are ambiguous. At this point, engineering guardrails should be added: fixed output templates, minimal runnable examples, requiring a plan before code changes, forcing tests to run, and returning failure logs to the model. If the behavioral score is much lower than the functional score, the main bottleneck is not insufficient knowledge but insufficient execution discipline. In that case, simply increasing prompt length has limited benefit; structured workflows and automated acceptance should be prioritized.
4. Complex Generation Tasks: Use a 3D Scene to Test Overall Capability
A complex front-end 3D scene is a good stress test. Take “a voxel-style simulation of a large-span suspension bridge” as an example. The task simultaneously includes scene modeling, lighting atmosphere, fog effects, water shading, interactive sliders, and frame-rate constraints. It requires the model not only to generate code but also to understand visual goals and performance boundaries. Evaluation goals can be divided into visual fidelity, atmosphere depth, interaction completeness, and smoothness at approximately 60 FPS.
In terms of visuals and atmosphere, the time slider should control sun position, light intensity, sky color, and fog tint. Fog effects should use lightweight particles to create a sense of volume, and the intensity slider should range from clear to dense fog without overexposure. The water surface needs a custom shader to implement waves and specular reflection, with distance fog allowing distant water surfaces to blend naturally. Acceptance should not rely on a single screenshot; sampling should be performed at multiple time points and fog intensities to check color transitions, horizon blending, and particle density.
Performance acceptance should fix the hardware, resolution, and sampling duration, and record average frame time, high-percentile frame time, memory usage, and maximum particle count. If the model’s implementation relies on too many draw calls or complex shaders, the visuals may look acceptable while the application cannot run stably. In this case, the model should be required to provide a performance budget and degradation strategy, such as reducing particle count, simplifying the water mesh, or controlling fog effects in tiers.
5. Build a Reproducible Pipeline: Make Scores Explainable and Regressable
The evaluation pipeline should run in containers with fixed dependencies, prompts, initial code, and acceptance scripts. Object storage should save generated artifacts and screenshots, logging services should record failure reasons, and monitoring alerts should track regression fluctuations. After each model or prompt change, rerun the same task set and compare functional completion, behavioral evaluation, and complex-task metrics. Failure samples should be classified into requirement misunderstanding, interface mismatch, missing boundary conditions, performance failure, or format violation. Only when the failure distribution is stable do the scores have reference value for selection.
In summary, the evaluation of medium-scale open-source code models should shift from “single leaderboard scores” to “dual-axis evaluation + complex-task stress testing.” A functional completion rate near 70% and a behavioral evaluation score near 60% can be regarded as the lower bound of usability for assisted coding. Whether a model can enter production still depends on a combination of private task sets, human review, and engineering guardrails.
