What Can Lightweight LLMs Do? Complex Task Evaluations and Capability Boundaries

In recent years, the parameter scale of Large Language Models (LLMs) has continued to climb. However, in enterprise applications and developers’ daily workflows, lightweight, low-latency models with strong instruction-following capabilities are playing an increasingly critical role. To objectively evaluate the real-world performance of some advanced lightweight LLMs in complex business scenarios, we designed a set of extreme tests covering complex frontend code generation and pure logical reasoning. This test is brand-agnostic, focusing on the capability boundaries of models across two dimensions: “code engineering implementation” and “tool-free pure reasoning,” providing technical selection references for developers.

Test Environment Specifications

To improve reproducibility, the core test parameters are summarized as follows:

Parameter	Description
Model Version	Mainstream open-source lightweight models released in Q1 2025 (specific versions found on their respective release pages)
Temperature	0.3, balancing determinism with a degree of creativity
Context Window	128K tokens
Runtime Environment	Chrome 120+ / Firefox 121+, WebGL 2.0 compatible
Testing Tools	Pure natural language interaction; external tools like code interpreters and calculators are disabled

Note: The above are example parameters. Actual tests should be adjusted according to the official documentation of the specific model used.

1. Complex Interactive Game Code Generation

Generating HTML5 applications with complete business logic via natural language is a touchstone for testing a model’s code generation capabilities. We constructed two high-complexity tasks to observe the model’s mastery of action logic, physical collision, AI behavior trees, and 3D rendering.

1.1 2D Action Fighting Game: Logic vs. Rendering

The testing task required the model to generate a stickman fighting game purely from prompts. The game needed to include blocking, normal attacks, double jumps, dodging, and five elemental skills, complete with robust enemy AI and environments.

During testing, some lightweight models demonstrated solid underlying logic construction capabilities. For instance, in action mechanics, the generated code accurately implemented character walking, jumping, and double jump mechanisms, with correct state machine transition logic. At the enemy AI level, the generated logic lacked inert “standstill” states and could actively pursue and counterattack.

However, shortcomings existed in visual presentation and hitbox accuracy: Canvas rendering for skill projectiles (non-DOM structure) failed to display correctly, and the collision box calculations for normal attacks were inaccurate. In contrast, another type of technical approach performed excellently in visual rendering with exquisite skill effects, but suffered fatal errors in basic physical jumping logic, preventing the character from leaving the ground.

This phenomenon indicates that models with different architectures have distinct “preferences” in code generation. Some models excel at building rigorous underlying control logic and state machines but lack detailed control over frontend styling and pixel-level rendering. Others, due to training data skewed toward visual presentation, are prone to critical flaws when handling complex physics engine logic. Developers must manually bridge these gaps based on their business focus (logic-heavy vs. visual-heavy).

1.2 3D Sandbox Game: Single-File Engineering and Library Integration

The second task required the model to implement a 3D sandbox game in a single HTML file using the three.js r160 rendering engine, featuring voxel terrain generation, tree distribution, and biomes.

Faced with this task, the tested lightweight LLM demonstrated robust single-file engineering integration capabilities. The generated code successfully initialized the WebGL 2.0 context, rendered basic 3D objects, and correctly imported the three.js r160 third-party library. However, severe logical misalignment occurred in perspective control (camera matrix operations), character movement, and world generation coordinate mapping. If the browser does not support WebGL 2.0 or if the three.js version is incompatible, WebGL context initialization will fail, requiring downgrade prompts or a fallback to WebGL 1.0 compatible code.

Other models used for comparison often caused code crashes when handling such massive single-file generation due to context length limitations or logic confusion. This reflects that high-quality lightweight models possess better syntactic correctness and foundational engineering framework capabilities in long-context code generation, though manual intervention is still necessary for complex mathematical operations (like 3D spatial coordinate transformation).

2. How Do Lightweight Models Perform in Pure Reasoning Without Toolchains?

An LLM’s pure logical reasoning ability—when external tools like code interpreters and calculators are disabled—directly determines its reliability in data analysis and complex problem-solving scenarios. We designed three mathematical problems featuring high deceptive traps and computational depth.

2.1 The Reverse Thinking Trap of Negative Squares

Test Question: “What is the smallest integer whose square is between 15 and 30?” (Correct answer: -5)

This question tests the model’s reverse understanding of the “square” concept. Since the square root of 15 to 30 ranges from 3.87 to 5.47, the model must identify that the integers in this range are 4 and 5, and then break the “natural number” mindset to realize that the square of -5 is also 25, and -5 is less than 4. In testing, the lightweight LLM successfully broke through the cognitive fixation and provided the correct answer. In contrast, some large-parameter models without deep thinking modes incorrectly answered 4. This proves that high-quality lightweight models have achieved a high standard in instruction alignment and logical generalization.

2.2 Deep Deduction of Permutations and Inclusion-Exclusion Principle

Test Question: “Arrange the six numbers 2, 0, 1, 9, 20, 19 in a row in any order to form an 8-digit number (the first digit cannot be 0). How many different 8-digit numbers can be produced?” (Correct answer: 498)

This is an extremely complex permutation problem involving number overlaps (20 and 0; 19 and 1 and 9), a non-zero leading digit constraint, and multidimensional inclusion-exclusion principles. Deduction key points: First, treat the six numbers as six “blocks,” yielding 6! = 720 total permutations. Subtract cases where 0 is the leading digit (place the 0 block first, permute the remaining 5 blocks: 5! = 120). Next, handle duplicate counting—because the 0 in “20” and the standalone “0” block, as well as the 1 and 9 in “19” and the standalone 1 and 9 blocks, can produce identical 8-digit numbers when concatenated, the inclusion-exclusion principle must be applied to deduct overlapping scenarios one by one. Through layered calculations, the final result is 498 distinct 8-digit numbers. This deduction process can be verified in the “Permutations of Multisets and Inclusion-Exclusion” chapters of combinatorics textbooks.

In repeated tests, the lightweight model consistently provided the correct answer and a complete reasoning chain. It should be noted that test frequency and specific model versions influence results; this serves as a capability demonstration rather than a rigorous benchmark evaluation.

2.3 Precise Calculation of Spatial Geometry and Water Level Rise

Test Question: “A solid iron cube with edge length 30 cm has a smaller cube with edge length 10 cm removed from each of its 8 corners. It is then placed into a container with a base area of 2500 cm², originally holding water to a depth of 20 cm. What is the water level after the cube is placed inside?” (Correct answer: 27)

This problem tests spatial imagination and piecewise function calculation. After corner removal, the base area of the iron block is 30×30 − 8×(10×10) = 100 cm². The initial water volume is 2500×20 = 50000 cm³. After placing the block, water first fills around it: when the water level is ≤ 10 cm, the effective base area is 2500 − 100 = 2400 cm²; when the water level is > 10 cm, the cross-section of the block becomes a full 30×30 = 900 cm², making the effective base area 2500 − 900 = 1600 cm². Assuming the final water level is h (h > 10), the volume equation is: 2400×10 + 1600×(h − 10) = 50000. Solving this gives h = 27 cm. During testing, the model clearly laid out the initial water volume, the immersed volume of the iron block, and the dynamic equation for the water level rise, demonstrating extremely strong step-by-step decomposition and multi-stage logical progression capabilities.

3. Core Findings and Selection Recommendations

Conclusions first—this test reveals the following key points:

• Lightweight models perform stably in underlying logic construction, state machines, and foundational engineering framework setup, making them suitable for rapid prototyping.
• Manual secondary debugging is still required for complex physical collisions, 3D coordinate transformations, and high-precision UI rendering.
• With high-quality instruction fine-tuning, they can match or even surpass some large-parameter models in reverse thinking and advanced discrete math problems, while offering significant latency advantages.
• Model-generated code must undergo manual security audits, reasoning results require cross-validation, and data privacy compliance must be strictly observed.

Deployment Architecture Recommendations: Enterprises can leverage cloud-native infrastructure to ensure model service stability. For instance, adopt containerized deployment and API gateways to achieve high-concurrency load balancing. Utilize object storage and serverless computing to cost-effectively host code preview and compilation environments. Implement logging and monitoring to track failure rates and response latency to continuously optimize prompts. The above architectural ideas are general recommendations and examples are for reference only; actual implementation should align with your specific tech stack and compliance requirements.

In conclusion, LLMs are not omnipotent “one-click generators,” but powerful “co-developers.” Understanding the strengths and weaknesses of different lightweight models, establishing standardized AI development workflows, and implementing robust risk management are the essential paths for enterprises to achieve technical cost reduction and efficiency gains.