Lightweight Model Evaluation for Complex Tasks: Process Fidelity and Runtime Verification
This article presents a practical evaluation framework for lightweight language models working on complex web, geometry-construction, and animation-export tasks. It focuses on verifying not only the visual result, but also whether the model follows the required procedure, respects task constraints, and produces code that runs and exports a usable file. оц
1. Scope
In this article, “lightweight models” refers to large language models with relatively small parameter counts and lower inference costs. The article does not report benchmark scores or compare specific products. Instead, it focuses on evaluation methods for determining whether a model has genuinely completed complex web, geometric construction, and animation-export tasks as required.
The framework applies to front-end page generation, mathematical visualization, and multi-step code modification. Pure factual question-answering tasks are not covered because established benchmark datasets already exist for them.
2. Why Complex Web Tasks Are Useful for Model Evaluation
Tasks such as building forms, card-based pages, or color-switching interfaces already have many mature templates. A model can often produce a reasonable result by reusing familiar patterns, making these tasks less effective for distinguishing capability levels.
Constrained geometric animation tasks are different. They require the model to translate natural-language instructions into concrete steps and implement those steps progressively in code. Each step can be checked independently, and the location of a failure is relatively easy to identify.
The following reproducible test case can be used directly for internal evaluation.
Prompt:
Given a circle O with radius 1 and a point A on the circle, construct the figure using only compass-and-straightedge operations: draw a circle centered at A with radius OA, intersect it with circle O to obtain two intersection points, and select one of them as a new vertex. Then, using each obtained vertex as the center and OA as the radius, draw circles successively to obtain the remaining vertices. Display each step separately in the animation, and show the final inscribed regular hexagon. The page must provide Start, Pause, and Reset buttons and support MP4 export.
Passing criteria:
- The distance from each of the six vertices to the center O has an error of less than 0.000001, and the distance between adjacent vertices equals the radius.
- The auxiliary circles appear in sequence during the animation, following the order specified in the prompt.
- The page produces no console errors, and the exported MP4 file has a duration greater than zero and plays normally.
Failure condition:
If the code directly uses trigonometric functions to calculate the six vertices and then connects them, the result must be classified as “visual-only implementation,” regardless of whether the geometry looks correct.
3. Four Evaluation Dimensions
Each dimension should have explicit checks and passing criteria.
| Evaluation dimension | Specific checks | Passing criteria | Example of failure | Recorded fields |
|---|---|---|---|---|
| Objective correctness | Number of vertices, edge lengths, symmetry, and positions | All geometric conditions are satisfied | Five vertices, or unequal edge lengths | Vertex coordinates and error values |
| Procedural consistency | Construction order, omitted steps, and method substitution | Animation steps correspond one-to-one with the prompt | Skipping the arc-intersection step | Step names and appearance timestamps |
| Constraint completeness | Tool restrictions, visible trajectory range, and export functionality | Every constraint has a corresponding implementation location in the code | Retaining the entire trajectory instead of the required segment | Constraint ID and implementation location |
| Runtime validity | Loading, playback, export, and console errors | No errors; the output file is non-empty and playable | A syntax error, or a zero-byte export file | Error logs, file size, and duration |
A task is considered complete only when it passes all four dimensions. If any dimension fails, the reason should be recorded. Passing the other dimensions cannot compensate for a failure in one dimension.
4. Potential Advantages of Lightweight Models
The following potential advantages are worth investigating, although there is currently no unified data supporting them. They should therefore be treated as hypotheses to be tested.
Instruction decomposition
Can the model break down a request such as “add all omitted steps” into stages including initialization, auxiliary construction lines, intersection calculation, target-figure generation, and animation display?
One validation method is to ask the model to provide a plan first, then compare each planned step with a manually prepared decomposition.
Cross-domain integration
The task combines geometric computation, JavaScript, two-dimensional drawing, and file processing. A practical evaluation method is to check the geometry module and export module separately, then determine whether both remain correct when integrated.
Structured code organization
Does the model separate the data, computation, rendering, and interaction layers?
This can be evaluated by counting how functions are divided and checking how many locations must be modified when changing colors or animation steps.
A control group should use the same prompt and execution environment, then be compared with a larger model or a human implementation.
