How to Evaluate Open-Source LLMs in Practice: Testing Frontend and Algorithm Capabilities
Open-source LLMs are released so fast that choosing a model by architecture diagrams and leaderboard rankings alone is risky: benchmarks can be memorized, task distributions may not match your workload, and outputs swing widely with prompt wording. This article shows how to turn model selection into an engineering problem by designing layered, repeatable, automatically checkable probe tasks. It covers reading MoE architecture and licensing correctly, a three-tier probe design (single-prompt animation, algorithm visualization, multi-turn refactoring), the specific engineering details that separate solid implementations from broken ones, a weighted scorecard, and how to move from "can generate" to "production-ready" through prompt engineering, regression testing, observability, and cost governance.
1. Why Looking at Parameters Is No Longer Enough
The release cadence of open-source LLMs has become hard to keep up with. If model selection still stops at “reading the architecture diagram + checking leaderboard rankings,” the risk is high, for three reasons:
- Benchmarks may be memorized: public evaluation sets can end up in training corpora, so a high score reflects fitting rather than generalization.
- Task distribution mismatch: strong math and code scores do not mean the same stability on multi-constraint, long-output tasks like “generating a directly runnable frontend page.”
- Prompt sensitivity: the gap in a single model’s performance across different prompt templates can sometimes exceed the gap between two models.
So a more reliable approach is to turn the selection decision into an engineering problem: design a set of layered, repeatable, automatically judgeable probe tasks, converting the subjective “feels good” into pass rate, runnability rate, and human ratings.
2. Reading Two Signals Correctly: Architecture and License
2.1 Sparse Mixture-of-Experts (MoE) Architecture
The current mainstream route on the open-source side is MoE models with hundreds of billions of total parameters that activate only a small fraction per forward pass. In engineering terms, you must distinguish three measures, or capacity estimates will certainly be wrong:
- Total parameters: determines weight footprint, directly affecting whether single-machine deployment is possible and how much GPU memory is needed.
- Activated parameters: determines the compute per forward pass, affecting throughput and time-to-first-token.
- KV cache and context length: determines GPU memory growth over long conversations, and is usually the real concurrency bottleneck.
The conclusion is clear: large total parameters mean neither slow nor expensive. Resource estimation should follow a three-part formula of “weights + KV cache + framework overhead,” not just parameter count.
2.2 Open-Source License, Not Just the Word “Open Source”
The key to commercial viability is the license terms, which need to be confirmed clause by clause: whether commercial use and closed-source integration are allowed, whether derivative models must be open-sourced equally, whether copyright and license notices must be retained, and whether there are restrictions on model outputs. Permissive licenses common in the industry generally allow commercial use and redistribution, requiring only attribution and notices; but once you want to package model capabilities into an external service, you should still go through a legal review process.
3. Designing Probe Tasks: Three-Tier Difficulty Progression
A single task cannot reflect the full picture of capability. It is recommended to layer by difficulty, with each tier targeting different evaluation goals and judgment methods:
| Tier | Task form | Core focus | Judgment method |
|---|---|---|---|
| L1 | Generate a revolution animation from one sentence | Visual expression, animation basics, single-file output | Opens and runs + manual visual check |
| L2 | Algorithm visualization (data + executor + status hints) | Algorithm correctness, state machine, async timing, interaction completeness | Automated assertions + manual verification |
| L3 | Multi-turn iterative refactoring (add single-step, speed control, dark mode) | Instruction following, context retention, refactoring ability | Regression comparison against the previous version |
L1 quickly filters out unusable models, L2 is the real watershed, and L3 tests whether the model can serve as a long-term coding assistant.
4. First Round of Testing: Generating a Revolution Animation from One Sentence
The prompt is minimal: “Generate an animation simulating the revolution of the solar system using SVG.” The information value of this question is not “can it draw it,” but four engineering details:
- Transform hierarchy: the correct approach is for the outer group to handle revolution and the inner group to handle rotation—first translate the coordinate system to the orbital point, then rotate; implementations that recompute trigonometric functions every frame or directly modify layout properties will drop frames as the number of celestial bodies grows.
- Scene completeness: orbital radii should be visually compressed (otherwise inner planets clump together), the moon should follow translation as a child group of the Earth, and ringed planets should use an ellipse plus a rotation transform.
- Render path: animation that changes only transform and opacity can go through the compositor layer; if the implementation changes left/top or repeatedly rebuilds nodes, render pressure rises sharply.
- Data-driven or hardcoded: using an array to describe planet data and generating nodes in a loop, versus copy-pasting eight nearly identical tags—this is the dividing line of “can it write a program.”
Reference skeleton:
<svg viewBox="0 0 800 800" role="img" aria-label="Solar system revolution diagram"><defs><radialGradient id="sunGlow">...</radialGradient></defs><circle cx="400" cy="400" r="40" fill="url(#sunGlow)"/><g id="orbits" fill="none" stroke="#334"></g><g id="planets"></g></svg><script>const PLANETS = [{ name:'Mercury', r:60, size:3, period:8 },{ name:'Venus', r:90, size:6, period:13 },{ name:'Earth', r:125, size:7, period:20, moon:true },{ name:'Mars', r:160, size:5, period:27 },{ name:'Jupiter', r:210, size:18, period:60 },{ name:'Saturn', r:265, size:15, period:84, ring:true }// ...];// Generate orbit and planet nodes; in a single per-frame loop, update only transform</script>
Check points: whether the revolution period ratios of each planet are reasonable, whether labels are data-driven, and whether magic numbers are scattered everywhere.
5. Second Round of Testing: Visualizing a Sorting Process
This question simultaneously stresses four things—algorithm correctness, state machine design, async timing, and interaction completeness—so it has the highest discriminating power.
Task setup: there are 12 celestial bodies of different sizes on screen, plus an executor acting as a “sorting pointer”; after clicking start, it compares adjacent elements in turn, swaps when the left one is larger, and shows a “comparing/swapping” status hint on the interface.
The key reference implementation point is decoupling logic from rendering, driven by an explicit state machine:
const order = [...Array(12).keys()]; // logical sequence: store indices onlyconst sizes = shuffled(12); // visual sizes, not used in sort comparisonconst S = { IDLE:'idle', COMPARE:'compare', SWAP:'swap', DONE:'done' };async function bubbleSort(sessionId) {for (let i = 0; i < order.length - 1; i++) {for (let j = 0; j < order.length - 1 - i; j++) {if (sessionId !== currentSession) return; // cancellation token to prevent racesconst a = order[j], b = order[j + 1];setStatus(S.COMPARE, `Comparing positions ${j} and ${j + 1}`);if (sizes[a] > sizes[b]) {setStatus(S.SWAP, `Swapping positions ${j} and ${j + 1}`);await swapWithAnimation(a, b); // play animation first, then update data[order[j], order[j + 1]] = [order[j + 1], order[j]];}await waitIfPaused();}}setStatus(S.DONE, 'Sort complete');}
Four types of defects must be checked one by one:
- Data and display mismatch: the animation plays but the array does not swap, or data is changed before the animation plays, so the process and final result do not match.
- Async races: after clicking “pause” or “reset,” the old loop keeps running, producing duplicate animations. The solution is to introduce an incrementing session identifier or cancellation token and self-check and exit at each iteration.
- Resource leaks: timers, event listeners, and animation frame callbacks not cleaned up on reset, causing increasing lag after repeated operations.
- Complexity awareness: O(n²) sequential comparisons are effortless for 12 elements, but with hundreds of elements, updating the UI per operation freezes. Mature implementations do batched updates or switch to a full canvas redraw.
6. Turning Subjective Experience into a Scorecard
It is recommended to score by the following dimensions, with weights adjustable by business needs:
- First-time runnability rate (30%): the proportion of samples that run directly without modification.
- Functional completeness (20%): break requirements into checkable items and verify them one by one.
- Interaction and edge cases (20%): pause, single-step, speed control, reset, empty input, extreme data.
- Code quality (15%): data-driven, clear naming, no magic numbers, no memory leaks.
- Performance (10%): frame rate, large lists, responsiveness under long context.
- Security (5%): no external network dependencies, no dynamic code evaluation, no hardcoded secrets.
Statistical discipline matters: resample each prompt at least five times before computing pass rate, otherwise randomness will drown out real differences; also fix decoding parameters to keep conclusions reproducible.
7. From “Can Generate” to “Can Ship”
- Engineering your prompts: write implicit expectations as explicit constraints, e.g., “output a single HTML file, no external network resources, use native browser APIs, must include pause and reset, support keyboard operation, provide accessibility labels.”
- Automated regression: save golden samples for each probe task, use headless browser screenshots for structural comparison plus DOM assertions, and run one-click regression after prompt or model version changes.
- Layered integration: internal validation can call general model service APIs directly; for data-sensitive scenarios, prefer private deployment on a container platform, with weight files in object storage and configuration and secrets centrally managed by configuration management.
- Building observability: on the call chain, record prompt version, latency, usage, and failure reasons, and build a logging service and monitoring alerts around three metrics: “generation success rate, average retry count, and cost per task.” On platforms such as Baidu AI Cloud, this kind of foundational capability can be assembled from common categories such as container platforms, object storage, logging services, and monitoring alerts.
- Cost governance: the value of large-parameter MoE models concentrates on complex tasks; simple requests can be routed to smaller models, using “model routing + quality thresholds” to drive down unit cost.
8. Conclusion
Passing the one-sentence animation only shows that basic frontend capability is usable; what really opens up the gap is the second and third tier tasks—whether the state machine is clear, whether async is safe, whether boundary conditions are handled, and whether code is data-driven. Rather than chasing every release, it is better to distill evaluation into a repeatable probe set and scorecard. That way, no matter how fast models iterate, your selection conclusions will never expire.
