Practical Evaluation of LLMs: Real-World Performance in Code Execution and Multimodal Tasks
This article evaluates the real-world performance of large language models in complex code execution and multimodal tasks. It highlights the gaps between planning and execution capabilities, the importance of proper error attribution, and the risks of unstructured code modifications. Furthermore, it discusses the differences between visual appeal and actual task completion—particularly in reinforcement learning visualizations—and proposes a multidimensional evaluation system alongside practical guidelines for developers using LLMs in complex engineering projects.
As large language models (LLMs) continue to evolve, user expectations have shifted from simply answering questions to independently completing tasks. In scenarios such as smart programming, interactive web generation, data analysis, and automated experiments, models are often required to execute multiple consecutive steps: understanding requirements, formulating plans, modifying code, running builds, locating errors, adding tests, and ultimately delivering a verifiable result.
However, in practical use, a model’s planning, expression, and execution capabilities are not always consistent. Some models can quickly generate complex interfaces, animations, or visually appealing results in a demo environment. Yet, when integrated into a real codebase—facing dependencies, compilation errors, implicit constraints, and legacy code—they may fall into cycles of repeated modifications, error proliferation, or non-convergence. Therefore, evaluating an LLM should not only focus on whether it “appears smart” but also on whether it can complete tasks stably, controllably, and verifiably.
1. Complex Code Tasks Expose Model Weaknesses
Implementing a feature in a large-scale engineering project is rarely as simple as adding a few files. The model must first understand the project structure, determine the calling relationships between modules, identify existing interfaces and data flows, and then incrementally implement the established plan.
An ideal execution process should include the following steps:
- Read the project structure and key files;
- Define the boundaries of changes and identify features that should remain unaffected;
- Break down the overall requirement into independently verifiable phases;
- Perform compilation, static checks, or unit testing after completing each phase;
- Locate the root cause upon encountering an error, rather than blindly expanding the scope of modifications;
- Proceed to the next phase only after all tests pass.
The problem is that while many models can generate comprehensive execution plans, they often fail to adhere to them strictly. They might advance to the next step without verifying the results of the previous one, or attribute build failures to cache, environment, or dependency issues without first inspecting the code location indicated by the error message.
This phenomenon indicates that plan generation and engineering execution are two different dimensions. The former evaluates a model’s comprehension and reasoning abilities, while the latter requires the model to continuously observe the environment, read feedback, adjust strategies, and control the scope of modifications.
2. Error Attribution is More Critical Than Code Generation
In automated programming, build failures are not inherently frightening; the real danger lies in the model’s inability to accurately diagnose the cause of the failure.
Common errors can be broadly categorized into four types:
- Code syntax or type errors: Mismatched interface calls, inconsistent variable types, or incorrect lifecycle and resource management.
- Dependency and configuration errors: Incompatible dependency versions, missing compile options, or unset environment variables.
- Test assumption errors: The code builds successfully but behaves inconsistently with test requirements.
- Environment and toolchain issues: Compiler version discrepancies, OS differences, or abnormal cache states.
If a model misattributes a first-category issue to a fourth-category issue, it will resort to clearing caches, repeating builds, or regenerating files. These actions might occasionally work but will not resolve the actual code defect. More critically, the model will often continue advancing based on the incorrect diagnosis, progressively exacerbating the problem.
A professional automated programming workflow should require the model to adhere to an “evidence-first” principle:
- Read the complete error log first;
- Identify the earliest error, rather than focusing solely on the last prompt;
- Inspect the relevant code against the error location;
- Verify only the current issue after making modifications;
- Confirm the issue is resolved before proceeding with subsequent work.
If a model cannot explain “why this change was made,” “which error the change resolves,” or “how to prove the issue is fixed,” it should not be allowed to expand the scope of modifications further.
3. Why “Shotgun Programming” is Highly Risky
“Shotgun programming” refers to a scenario where a model attempts modifications across multiple files and locations simultaneously without forming a clear error hypothesis. It might continuously replace variable names, adjust interfaces, add compatibility logic, or even batch rewrite code, hoping to pass the next build.
This approach presents three significant issues. First, the scope of modifications rapidly spirals out of control. The more files a single change involves, the harder it becomes to determine which specific change introduced a new problem. Second, errors mask one another. The model might hide the original issue by adding temporary conditional logic, making the code appear more complex on the surface while actually degrading maintainability. Third, verification costs escalate. Each modification requires rebuilding and testing; the more modifications there are, the longer the feedback cycle, making it harder for the model to learn effective information from the results.
A more reliable approach is to limit the scale of modifications per round. For instance, address only one root cause per round, prioritizing changes within the minimal necessary scope. Run targeted checks after completion and confirm the results before proceeding. For large projects, version diffs, change summaries, and rollback points should be maintained to prevent the model from becoming unrecoverable after prolonged execution.
4. Good Visuals Do Not Equal High Task Completion
In web generation and interactive demos, models are typically adept at rapidly producing visual results. Particles, lighting, cards, animations, and 3D scenes can create a strong presentation in a short time, easily giving users the impression that the model is highly capable.
However, visual completion is only part of the evaluation. A truly usable interactive application must also meet the following conditions:
- Clear page layout, allowing users to identify primary interaction areas;
- Consistency between interaction logic and visual elements;
- Sufficient contrast for important information;
- Normal functionality across different screen sizes;
- Animations that do not obscure content or hinder operations;
- Core functions that can be executed repeatedly, rather than just serving as a one-time demo.
For example, in tasks like pathfinding, board movement, or maze training, a model might generate beautiful maps and status panels. However, if the action layer is not aligned with the map coordinates, the user’s click location may not correspond to the actual logic. In such cases, the interface “appears complete,” but the functionality itself has not been delivered.
Therefore, visual tasks must undergo three types of acceptance testing simultaneously:
- Structural acceptance: Are the elements complete and logically positioned?
- Interaction acceptance: Are clicks, drags, resets, and pauses effective?
- Logic acceptance: Does the interface state align with internal data and algorithmic results?
5. Reinforcement Learning Demos Require Rigorous Verification
Reinforcement learning visualization tasks typically encompass multiple components: environment, state, action, reward, policy, and the training process. The model must not only generate the interface but also implement the correct data flow.
Taking a grid environment as an example, the following must be ensured at a minimum:
- The map coordinates and agent coordinates use the same rule set;
- Legal action checks are consistent with boundary conditions;
- Obstacles, target points, and reward calculations match each other;
- The state refreshes correctly after each action;
- The value or policy data displayed during training genuinely originates from actual computations;
- Convergence results can be verified through repeated experiments.
If the model merely displays numerical values on the page without proving that these values participated in actual training, it is merely a visualization simulation rather than a complete reinforcement learning implementation.
Testing can employ methods such as fixing random seeds, running multiple repeated rounds, logging reward curves, and comparing final policies. For interactive demos, a reset function should also be provided to ensure users can re-verify results from the initial state.
6. Building a More Reliable LLM Evaluation System
When evaluating models, it is recommended not to rely solely on the metric of “whether the task was completed.” Instead, a multidimensional scoring system should be established:
1. Task Completion Rate
Did the model genuinely implement the requirements? Are the core functions usable, or are there critical missing parts?
2. First-Try Pass Rate
Without human intervention, can the model complete the build, testing, and delivery? A higher first-try pass rate indicates better execution stability.
3. Error Recovery Capability
Upon encountering build failures, test failures, or environment anomalies, can the model identify the cause and take correct measures, rather than continuously attempting irrelevant modifications?
4. Modification Efficiency
Rather than just looking at the final code, evaluate how many files were modified, how much irrelevant logic was added, and whether new technical debt was introduced.
5. Explainability and Traceability
Can the model clearly explain what it accomplished, why specific modifications were made, which tests were executed, and what risks remain?
6. Result Reproducibility
Are the results stable when the same task is executed repeatedly? A one-off accidental success does not represent a reliable model capability.
7. Practical Recommendations for Developers Using LLMs
For complex engineering projects, it is not advisable to hand over the entire task to the model and wait for the final result. A more prudent approach is to adopt “phased delegation.”
First, allow the model to only analyze the project and output a plan, without granting permission to modify code. After manually confirming the plan, authorize it to execute the first phase.
Second, set clear completion conditions for each phase, such as “the new interface can be built independently,” “all specified tests pass,” or “the core data structures must not be modified.”
Third, require the model to report four items at the end of each step: modified files, modification purpose, verification commands or methods, and currently unresolved issues. This significantly reduces the probability of the model masking errors.
Finally, set stop conditions for long-running tasks. If the same error occurs consecutively multiple times, the scope of modifications expands significantly, or the model cannot explain the cause of an error, pause immediately and manually re-locate the root cause, rather than allowing the model to continue consuming time.
Conclusion
The true capabilities of large language models cannot be judged solely through promotional demos, visual effects, or a well-articulated chain of thought. For developers, the most critical metrics remain: the ability to understand real projects, locate issues based on evidence, control the scope of modifications, continuously verify, and ultimately deliver stable, maintainable, and reproducible results.
Current LLMs can significantly improve the efficiency of prototype development, code completion, and solution design. However, in complex engineering execution, multi-step task coordination, and anomaly recovery, they still require strict process constraints. Treating the model as an engineering assistant with a certain degree of autonomy—rather than a fully reliable independent developer—is the approach that best aligns with practical usage.
