LLM Evaluation in Practice: Code Generation, Agents, and Multi-Turn Search

    This article presents a scenario-based evaluation approach for new-generation large language models, focusing on runnable front-end code generation, agent behavior, and multi-turn search. It examines representative use cases, common failure modes, and a minimal automated evaluation skeleton, then discusses the role of small models in large-scale RLHF and the challenges of precise instruction following in agentic search. It concludes with practical engineering recommendations for model selection based on artifacts, repeated sampling, layered evaluation, and cost-performance tradeoffs.

    1. Why a Scenario-Based Model Evaluation System Is Needed

    Whenever a new generation of large models is released, the most common community discussion tends to focus on benchmark scores. From an engineering perspective, however, there is a clear gap between leaderboard results and real development experience. Benchmarks usually measure static capabilities on fixed datasets, while developers face open-ended tasks in their daily work: building an interactive front-end page, training an agent that converges, or completing multi-turn information retrieval according to instructions. The success of these tasks depends not only on the model’s knowledge base but also on its understanding of constraints, its grasp of physical and logical common sense, and its self-consistency across long-chain tasks.

    A more valuable approach is therefore to build a scenario-based evaluation that is close to real work: select a set of tasks with determinable outcomes, let the model directly produce runnable artifacts, and then score them against objective criteria. This type of evaluation has a low barrier to entry but provides highly dense information. Any hallucination, missing common sense, or instruction drift will be exposed in the runtime results.

    2. Front-End Code Generation: A Deep Dive into Three Representative Use Cases

    2.1 Use Case Design Principles

    A good code generation use case should satisfy three conditions. First, the result should be objectively verifiable: a game should be playable, an animation should run, and logic should converge. Second, the task should contain implicit common-sense constraints to test whether the model makes unwarranted assumptions. Third, it should include visual and interaction requirements to test the model’s awareness of goals beyond the prompt. Based on these principles, the following three types of tasks can be designed:

    • Layout-oriented tasks: For example, implement a sliding puzzle game that requires pieces to be arranged neatly and moved normally. The evaluation focuses on the correctness of spatial layout and the fidelity of the interface.
    • Physical common-sense tasks: For example, use a Web graphics library to implement a bowling mini-game. The evaluation focuses on whether physical parameters match common sense. If the pin spacing is too large and makes a strike impossible, it reveals a deficiency in the model’s common-sense understanding. In addition, if the interface color contrast is too low, it reflects the model’s neglect of usability.
    • Algorithm and visualization tasks: For example, implement a Q-learning reinforcement learning demo in a maze environment, requiring the agent to align with the terrain, Q-values to be visualized, and training to converge. This is the most difficult of the three tasks, testing front-end rendering, algorithm implementation, and numerical stability at the same time.

    2.2 Analysis of Common Failure Modes

    In actual tests, model failures on these tasks are not random but show several typical patterns:

    1. Runaway dimensions: The agent or UI elements fill the entire canvas, indicating that the model does not handle coordinate systems and boundary conditions rigorously enough.
    2. Broken rendering chain: The page structure is generated, but event binding or initialization logic fails, causing the page to hang and become non-interactive.
    3. Layer misalignment: The coordinate systems of the logic layer and the rendering layer are inconsistent. For example, the agent action layer is not aligned with the terrain layer. This is a typical multi-module collaboration failure.
    4. Chain-of-thought hallucination: The model’s reasoning process claims that a certain visual effect will be implemented, such as an ink-wash style, but the final artifact does not reflect it at all. This phenomenon of “saying it but not doing it” suggests that we cannot rely only on the chain of thought; the final artifact must be the source of truth.

    It is worth noting that even when the model attempts the same task multiple times, the failure mode keeps changing: the first attempt fails in layout, the second does not even generate the structure, and the third is logically correct but cannot run. This shows that the failure is not a single capability shortcoming but that the generation process itself has randomness. Engineering practice should therefore estimate success rates through multiple samples rather than drawing conclusions from a single success or failure.

    2.3 A Minimal Automated Evaluation Skeleton

    To engineer the above evaluation, the following pseudocode structure can be used:

    1. eval_cases = [
    2. {"task": "sliding puzzle", "check": check_layout_alignment},
    3. {"task": "bowling simulation", "check": check_physics_common_sense},
    4. {"task": "maze Q-learning", "check": check_training_convergence},
    5. ]
    6. for case in eval_cases:
    7. for attempt in range(N): # Multiple samples to estimate success rate
    8. code = model.generate(case["task"], with_cot=True)
    9. result = sandbox.run(code) # Sandbox execution, collect runtime errors
    10. score = case["check"](result)
    11. log(case, attempt, score, result.runtime_errors)

    The key points are sandbox execution, error collection, and multiple sampling. Only when all three are present can the evaluation conclusions be statistically meaningful.

    3. The Distinct Value of Small Models in Large-Scale RLHF

    Beyond evaluation, another engineering direction worth attention is the large-scale implementation of reinforcement learning post-training, such as RLHF. A common industry pain point is that if the reward model directly uses a large-parameter model, inference cost will grow linearly with the sample size, and the cost of one round of preference data training becomes prohibitively high.

    A repeatedly validated approach in practice is “large-small model collaboration”:

    • Large models provide the quality ceiling: They provide high-quality preference judgments or generation demonstrations at key nodes.
    • Small models provide throughput: They handle batch scoring, data filtering, coarse screening, and other high-concurrency stages with lower latency and cost.
    • Inference frameworks provide efficiency: Tensor parallelism, pipeline scheduling, KV cache optimization, and other methods reduce unit inference cost, allowing limited compute to support larger-scale sampling.

    For teams planning to invest in this direction, it is advisable to choose a fixed scenario, such as code preference labeling, and iterate continuously. First, run the full pipeline with a small model, then gradually introduce stronger reward signals and observe the balance curve between convergence and cost. This scenario-anchored iteration approach is more likely to produce practical technical accumulation than pursuing general-purpose leaderboards.

    4. Multi-Turn Search: A Litmus Test for Instruction Following

    In agent scenarios, multi-turn search is a severe test of a model’s instruction-following capability. A typical test is to force the model to perform a fixed number of search rounds, for example four rounds, and observe its actual behavior.

    Typical problems exposed by the test include:

    1. Execution count does not match the instruction: The model may execute only one round and reuse the first result, or execute far more rounds than required. A count mismatch is essentially a lack of planning capability for countable action sequences.
    2. False statements: The model may search only once but claim in the main text that it searched three times. This inconsistency between self-report and actual behavior is a serious defect in agent evaluation because it undermines auditability.
    3. Factual errors: Even if the search count meets the requirement, if core configuration information is misattributed, such as writing a nonexistent configuration item into the conclusion, it shows that the model still lacks the ability to integrate and discriminate retrieval results. Such errors often appear in professional details that are hardest for users to identify.

    Good performance should be: the execution count precisely matches the instruction, retrieval results are cited faithfully, and information that cannot be found is explicitly marked as uncertain rather than hallucinated. This suggests that when selecting models, instruction-following precision and the credibility of self-reporting should be scored as independent dimensions.

    5. Four Engineering Recommendations from Evaluation to Model Selection

    Based on the observations above, the following model selection recommendations can be distilled:

    1. Judge by artifacts: Do not rely on demo videos or marketing language. Look only at the objective quality and success-rate distribution of runnable artifacts.
    2. Use multi-run sampling statistics: A single performance is not enough for a conclusion. Use at least five samples to estimate the pass rate and record the distribution of failure modes.
    3. Evaluate in layers: Split code generation, agent planning, and retrieval integration into independent dimensions to avoid a single total score masking structural weaknesses.
    4. Balance cost and capability: Evaluate cost-effectiveness by combining inference cost and throughput. Especially in large-scale RLHF, batch data processing, and similar scenarios, a combination of small models and strong inference optimization often has more engineering value than stacking large models.

    Overall, new-generation models have generally reached a usable level on regular front-end generation tasks. However, on the “last mile” of implicit common sense, long-chain consistency, and precise instruction following, there are still perceptible gaps among leading players. For developers, building an in-house scenario-based evaluation set has far more long-term value than chasing leaderboards.