Evaluating a Multimodal Model Beta Release: Benchmarks and Code Generation in Practice
This guide explains how to evaluate and choose between internal beta releases of a general-purpose multimodal model, focusing on two capabilities: multimodal visual understanding and code generation. It lays out a four-dimension, three-tier evaluation framework, then walks through three concrete tests—visual recognition within compliance boundaries, text-to-3D interactive animation, and 2D diagram to structured drawing code. It also covers building a reproducible evaluation pipeline, double-blind review with inter-rater consistency metrics (Cohen's Kappa and Krippendorff's alpha), cost control through prompt caching and asynchronous batching, common pitfalls to avoid, and a recommended deliverables checklist.
1. Introduction: How to Choose a Beta Release and What to Watch For
A vendor’s internal beta usually ships before the official release. Whether a team can quickly map out its capability boundaries directly shapes its selection timeline and its trial-and-error cost.
Judging by public leaderboards alone has clear limitations: most rankings are multi-dimensional weighted averages where single-point gains get diluted, and differences in question distribution can cause mismatches. A common real-world pattern: a new version is noticeably stronger than its predecessor on image recognition and multi-file coding, yet its overall ranking does not move in step. Leaderboards answer “roughly how good is the model overall,” but what we need to answer is “is it right for my business”—and that can only come from a self-built evaluation set. The selection checklist reduces to three points: match dimensions to your business, distinguish “getting it right” from “fabricating,” and account for cost.
This article takes “an internal beta intermediate version of a general-purpose large model” as its subject and focuses on two capability areas:
- Multimodal visual understanding: recognizing people and scenes in images, and turning diagrams into structured code;
- Code generation: producing single-file, runnable 3D interactive animations that reconstruct complex 2D diagrams.
2. Evaluation Framework: Four Dimensions and Three Difficulty Tiers
A reliable beta evaluation rests on four design principles: tasks that can be tiered, metrics that can be quantified, a process that can be reproduced, and conclusions that can be compared (old and new versions run the same set of questions under the same scoring).
The four dimensions are: visual perception and recognition, image-text alignment and structural reconstruction, code generation and runnability, and complex tasks and 3D reasoning.
| Difficulty | Task examples | Focus | Expected performance |
|---|---|---|---|
| L1 Basic | Image content recognition; simple webpage/utility generation | Basic recognition and coding | Most models can complete; limited differentiation |
| L2 Intermediate | Restoring a UI screenshot into frontend code; turning a diagram into structured drawing code | Instruction following, detail fidelity, syntactic correctness | Strong and average models begin to diverge |
| L3 Challenge | Multi-stage 3D interactive animation; reconstructing a 2D diagram into a complex 3D structure | Long-horizon planning, 3D spatial reasoning, runtime stability | Most models have clear weaknesses; best reveals version differences |
2.1 Scoring Weight Example
Weights should shift with your business. Here is a starting point: visual perception and recognition 20%, image-text alignment and structural reconstruction 25%, code generation and runnability 30%, complex tasks and 3D reasoning 25%. If your business only involves frontend reconstruction, raise code runnability above 40% so the total score actually carries decision-making weight.
3. Test 1: Visual Recognition and Compliance Boundaries
3.1 Compliance Boundaries Come First
Recognizing people directly touches on personal information and portrait rights. Draw the boundaries before you start:
- Use only public images that are authorized, and archive the authorization source in the test set;
- Do not use them to track or match the identity of ordinary individuals, and do not run bulk matching against a face database;
- Redact personal names, organizations, and similar information in reports;
- Comply with portrait rights, privacy regulations, and platform safety policies.
Misjudgments fall into two types: missed recognition (failing to identify, lower impact) and wrong identification with fabrication (supplying a seemingly professional but mismatched identity or résumé, far more harmful).
3.2 Task and Scoring
L1 task: upload a photo of a speaker at a public conference and ask the model to identify the scene content and the person, provide key background information, and explain the basis for its judgment.
The model under test returned a specific name, an affiliation with two organizations, and a scene description covering the podium, the background screen, and the audience seats. From a screenshot alone, the guesswork is hard to rule out, so credibility rests on two points: whether it names concrete, verifiable entities, and whether the information is common content in public corpora.
Additional controlled experiment: swap in new photos with different poses and outfits; use lightly cropped or compressed images; and add an instruction such as “state clearly when uncertain.” If the results stay correct on the perturbed samples and the model clearly says “cannot determine” for unfamiliar faces, it means vision and knowledge are reliably linked; if it still returns high-confidence wrong answers, language priors are more likely at work.
| Output type | Description | Score |
|---|---|---|
| Correct recognition with accurate information | Identity and background are both verifiable | Full marks |
| Clearly expresses uncertainty | e.g., “cannot confirm identity” | Partial credit; encourages honest answers |
| Wrong identity or fabricated details | Confident but wrong | Zero, and counted in the “hallucination” tally |
4. Test 2: From Text Description to a 3D Interactive Animation
4.1 Task Setup
L2/L3: Generate a single-file, fullscreen 3D web page that loads a rendering library via CDN. The scene is a rocket launch under daylight: idle (rocket standing on the launch pad) → ignition (high-temperature flame, accumulating vibration) → smoke burst → clamp release (the holding mechanism swings open and detaches from the rocket body) → vibration and liftoff → camera follows the climb and plunges into the clouds.
Five key checkpoints: whether the state machine is clear, whether the physics intuition holds up, whether the camera follow is smooth, whether single-file size and particle count are balanced against frame rate, and whether code naming and structure are easy to modify later.
4.2 Structured Prompt
Role: Senior frontend graphics engineerTask: Use a CDN-loaded 3D rendering library to generate a single-file,fullscreen, runnable rocket launch animation.Scene requirements1. Daylight: blue sky background, directional light with soft shadows,launch pad and tower clearly visible;2. Rocket assembled from primitives: body, fairing, fins, engine nozzle.Timeline (total ~20 seconds)0-4s Idle: rocket stands still, only slight ambient motion4-6s Ignition: nozzle flame builds up gradually, body begins to shake6-8s Smoke burst: white smoke and ground dust spread rapidly outward8-9s Clamp release: hold-down arms on both sides swing open and retract9-20s Liftoff: rocket accelerates upward, camera follows smoothly,finally passes through the cloudsTechnical requirements1. Drive the animation with requestAnimationFrame and compute displacementfrom elapsed time so speed stays consistent across frame rates;2. Interpolate the camera with an easing function to avoid sudden jumps;3. Reuse particles via object pooling to control performance cost.Output requirements1. Output a single complete HTML file that runs directly in a browser;2. Add brief comments to key functions;3. At the end of the file, output a self-check list as comments;4. Provide runnability evidence: browser screenshots, console error logs,or a screen recording.
A structured prompt typically brings four improvements: more complete stages, explicit coverage of easy-to-miss details such as clamp release, smoother camera follow, and lighting closer to real daytime outdoors.
4.3 Runnability Acceptance
A text self-check list is not enough to prove the code works. Acceptance must add three kinds of evidence:
- Screenshots: one frame at each key moment (ignition, smoke, liftoff), compared item by item against the requirements;
- Console errors: open DevTools Console; it passes only if there are no uncaught exceptions;
- Run records: elapsed time, peak memory or frame rate, and any load failures.
Correct on paper does not mean correct on screen. A submission without screenshot and error evidence must not receive full marks on the code dimension.
5. Test 3: From a 2D Diagram to Structured Drawing Code
L2 task: given a 2D diagram that hints at 3D appearance, reconstruct it with structured drawing syntax (coordinates, paths, line styles, fills, draw order). There are three difficulties: inferring 3D structure and occlusion relationships from a 2D image; translating them precisely into coordinates and instructions; and arranging draw order sensibly—back layers first, front layers last.
| Dimension | Checkpoint |
|---|---|
| Element completeness | Are the main geometries and annotation lines all present? |
| Geometric accuracy | Are proportions, angles, and relative positions close to the original? |
| Attribute correctness | Do solid/dashed lines, arrows, fills, and transparency correspond? |
| Occlusion and layering | Are front/back relationships correct and back layers properly covered? |
| Compilability | Does it compile in one pass, with no syntax errors? |
Illustrative result: the new version outperforms the previous visual version on most items, with common problems concentrated in fine line details and layering/occlusion—indicating that “understanding 3D structure” is still the bottleneck. Comparing side by side with the original and checking off item by item locates the gap better than a vague overall score.
5.1 Advanced: From Diagram to 3D Structure Reconstruction
Given a three-view drawing or a diagram with complex occlusion, ask the model to first output a structural analysis report (shape, relative dimensions, and connection method of each part), then model it with a rendering library and support rotate/zoom viewing. In testing, the overall silhouette and main parts reconstruct reasonably well, but curved surfaces, seam transitions, and real structure show clear gaps, and some parts have deviations in proportion and position.
A more sensible approach is a human-in-the-loop correction workflow:
- The model outputs a structured understanding report, forming a “discussable” intermediate artifact;
- The engineer marks the errors and corrects proportion and dimension constraints;
- Feed the corrected constraints back into the prompt and regenerate the code;
- Re-verify item by item against the 5.1 table and record the differences across both rounds.
This is more efficient than chasing “end-to-end perfect reconstruction” in one shot, and closer to a real production workflow.
6. Engineering: Automated Evaluation Pipelines and Metric Recording
6.1 Call Example and Parameter Compatibility
import jsonimport osimport urllib.requestBASE_URL = os.environ.get("LLM_BASE_URL", "https://example.com/v1")API_KEY = os.environ["LLM_API_KEY"]MODEL = os.environ.get("LLM_MODEL", "model-id")HEADERS = {"Content-Type": "application/json", "Authorization": f"Bearer {API_KEY}"}def call_model(messages, effort="medium", temperature=0.7, timeout=300):payload = {"model": MODEL,"messages": messages,"temperature": temperature,"max_tokens": 8192,}# The following are extension fields for some services, not a universal# standard; probe capability before sending, and fall back on failureif supports_field("reasoning_effort"):payload["reasoning_effort"] = effortreq = urllib.request.Request(f"{BASE_URL}/chat/completions",data=json.dumps(payload).encode("utf-8"),headers=HEADERS,method="POST",)try:with urllib.request.urlopen(req, timeout=timeout) as resp:data = json.load(resp)except Exception as exc: # record timeouts and rate limits separately; do not mix into quality scoresreturn None, {"error": str(exc)}usage = data.get("usage", {})details = usage.get("completion_tokens_details") or {}meta = {"prompt_tokens": usage.get("prompt_tokens", 0),"completion_tokens": usage.get("completion_tokens", 0),# completion_tokens_details is also an extension field; record 0 when unavailable"reasoning_tokens": details.get("reasoning_tokens", 0),}return data["choices"][0]["message"]["content"], meta
Compatibility notes: reasoning_effort and completion_tokens_details are extension fields of some services; the standard interface does not guarantee support, and sending them directly may return a 400 or raise an error. The safe approach is to run one probe request first to confirm whether the fields take effect; if unsupported, substitute equivalent parameters per the service documentation, or drop the field and record the reasoning tier in local metadata. When reading nested fields, always use .get() with an empty-dict fallback to avoid a KeyError interrupting the whole batch.
6.2 Double-Blind Review and Consistency Calculation
Scoring generated results is subjective, so use double-blind review: hide the model identifier, randomize the numbering, and have reviewers check off the acceptance list item by item.
CHECKLIST = {"stage": ["idle", "ignition", "smoke", "clamp release", "liftoff", "camera follow"],"detail": ["daylight", "sky and ground", "clamp retraction", "cloud passage"],}def checklist_score(review):return {dim: sum(1 for ok in flags if ok) / len(flags)for dim, flags in review.items()}def disagreement_rate(a, b):"""Inter-rater disagreement rate: mismatched items / total items"""total = sum(len(v) for v in a.values())diff = sum(1 for dim in a for x, y in zip(a[dim], b[dim]) if x != y)return diff / total if total else 0.0
For two reviewers, use the disagreement rate directly; for more than two, compute a consistency coefficient:
- Cohen’s Kappa:
κ = (p_o - p_e) / (1 - p_e), wherep_ois the observed agreement rate andp_eis the expected agreement rate by chance; - Krippendorff’s alpha: suitable for multiple reviewers, missing values, or ordinal scoring, ranging from 0 to 1.
Rule of thumb: if κ or alpha falls below 0.6, the acceptance items themselves are poorly worded (for example, subjective phrasing like “reasonably good fidelity”). Revise the standard and rewrite the items first, then re-score. Fixing the standard before scoring is far more reliable than adjusting weights after the fact.
6.3 Metadata Recording
{"case_id": "l3_rocket_v2","model_id": "internal-beta-20260826","model_date": "2026-08-26","reasoning_effort": "high","sampling": {"temperature": 0.7, "max_tokens": 8192},"usage": {"prompt_tokens": 7820, "completion_tokens": 2140,"reasoning_tokens": 860, "cache_hit_tokens": 4000},"latency": {"first_token_ms": 1240, "total_ms": 48600},"review": {"reviewer_a": {"stage": [1,1,1,0,1,1]}, "kappa": 0.78},"evidence": ["shot_ignite.png", "console_clean.txt"]}
Beta versions often have a fixed validity window, so recording the version date avoids losing traceability later. The same model can differ markedly across high, medium, and low reasoning effort, so always record by tier.
7. Cost Control: Keeping Evaluation Sustainable
Using a typical usage-based billing model as an example: cache-hit input has the lowest unit price, then cache-miss input, and output is the most expensive; some services also price peak and off-peak hours differently.
7.1 Cost Illustration (Consistent Basis)
The values in the table below only demonstrate the calculation method; actual prices follow each service’s official billing documentation.
| Billing item | Unit price (CNY per million tokens) | Example usage (thousand tokens) | Cost per call (CNY) |
|---|---|---|---|
| Cache-hit input | 0.02 | 1 | 0.00002 |
| Cache-miss input | 1.00 | 7 | 0.00700 |
| Output | 4.00 | 2 | 0.00800 |
| Total | — | 10 | 0.01502 |
Computed consistently as “8k input + 2k output per test”:
- Before optimization (no caching): input 0.008 + output 0.008 = 0.016 per call, about 16 for 1,000 calls.
- After optimization (system prompt and requirement docs make up about half the input and are cacheable): 4k hit × 0.02/M = 0.00008, 4k miss × 1.00/M = 0.004, output 0.002 × 4 = 0.008, totaling 0.01208 per call, about 12.08 for 1,000 calls.
- Stacking the 50% off-peak discount: about 0.00604 per call, about 6.04 for 1,000 calls—roughly a 62% reduction versus the pre-optimization 0.016.
Keep the basis consistent: the 0.01502 figure corresponds to the “1k hit + 7k miss + 2k output” 10k-token scenario, which is a different usage profile from the 8k/2k three-way comparison above—do not mix them.
Prompt caching usually matches on the request prefix as the unit, so only the fully identical prefix can hit, and the cache has a validity period. Putting stable system prompts first and variable content last is therefore key to a higher hit rate.
7.2 Five Engineering Optimizations
- Fixed prefix, variable suffix: keep version notes, interface conventions, and output format at the start of the prompt so the cache can be reused;
- Move batch tasks to asynchronous: if the service offers an offline batch API, run the evaluation queue during off-peak hours;
- Assign reasoning effort by difficulty: use medium-low effort to quickly filter basic items, and reserve high effort for challenge items and the final comparison;
- Set output limits per task: avoid inflating costs with overly long outputs;
- Track failure retries separately from quality scores: exclude timeouts and rate limits from quality scoring so you do not burn budget re-running just to “finish.”
8. Common Pitfalls in Evaluating Beta Versions
- Leaderboard-only thinking: leaderboards are a reference frame, not a conclusion—rely on your own task set;
- Deciding from a single run: generation is stochastic, so for key tasks sample multiple times and take the mean or the majority result;
- Missing parameter records: without version date and reasoning effort, conclusions cannot be reproduced;
- Reading code without running it: you must actually run it in a browser and compare against screenshots;
- Ignoring uncertainty and safety: counting only “how much it got right” while ignoring “how much it fabricated” overstates reliability;
- Avoiding compliance: when human images are involved, authorization and redaction belong in the evaluation design stage, not added just before launch.
9. Summary and Deliverables
The core of beta-version evaluation is turning “it feels stronger” into “on which dimension, by how much, and is it worth using.” Three conclusions:
- Uneven capability is the norm: people recognition, 3D animation, and diagram-to-drawing show noticeable progress, while complex 3D structure reconstruction remains clearly insufficient. Match dimensions to your business rather than chasing an all-around optimum.
- Methodology matters more than a single hands-on impression: a tiered task set, structured prompts, double-blind review with consistency checks, and metadata recording together form a reusable system.
- Balance cost and effectiveness: cache reuse, asynchronous batching, and reasoning-effort tiering can substantially cut evaluation overhead.
A four-step path forward: build a three-tier evaluation set from representative tasks → design unified prompts and acceptance criteria for the old and new versions → run them automatically with complete records → aggregate via double-blind review into a decision report.
The deliverables list should include five items: a three-tier difficulty task set, a prompt template library, a structured scoring sheet (with weights and consistency thresholds), a cost report (with a consistent token basis), and a decision report (with screenshots and console evidence).
If your evaluation pipeline needs to schedule large numbers of asynchronous tasks, archive evaluation artifacts, or continuously monitor model service quality and latency after launch, you can rely on general-purpose cloud infrastructure such as object storage, message queues, container platforms, logging services, and monitoring/alerting. Baidu AI Cloud offers general capabilities in these categories, on top of which you can build a stable evaluation and deployment workflow.
