A Systematic Method for Evaluating Large Language Models in Engineering
Enterprises evaluating large language models are easily swayed by a single demo, a social media screenshot, or a vendor's self-assessment. The more reliable engineering approach is to treat a model as a replaceable inference component and measure it against tasks, constraints, and the cost of failure rather than against slogans. The sections below follow the questions that actually surface during selection: is capability sufficient, is cost predictable, is deployment controllable, and are licensing and data terms clear.
Define the Decision Problem First
Fix four things before comparing candidates. Without them, comparisons have no shared frame of reference.
- Task boundary. Code completion, document generation, retrieval QA, structured extraction, or multi-step agentic code editing. These differ by an order of magnitude in what they demand for context length, tool calling, and format stability.
- Quality floor. Which errors are unacceptable — fabricated APIs, a broken build, leaked sample data — and which can be accepted after human review.
- Operating constraints. Whether data must stay inside a defined boundary, whether external inference services are allowed, peak concurrency, acceptable time to first token and total latency.
- Cost basis. Per million tokens, per request, per reserved compute, or the total cost of one successfully delivered outcome. Looking only at list prices misses retries, long contexts, and human rework.
Write these up as a one-page constraint document and stop adding requirements during the evaluation. If a constraint changes, re-run the evaluation instead of verbally adjusting an old score.
Capability: Use a Task Set, Not a Feeling
Claims like “close to a flagship generation” cannot be reproduced. The workable alternative is a small golden set you build yourself: sample from real tickets, code reviews, and support conversations, redact sensitive data, then fix the inputs, expected key points, and scoring rules. The set does not need to be large, but it must cover failure modes: long-context forgetting, format drift, over-refusal, wrong tool arguments, and missed edits across multiple files.
Split the scoring rather than collapsing it into one number:
- Task completion: whether the key acceptance points are met, checked against a checklist rather than an overall impression.
- Faithfulness: whether the output introduces facts or interfaces that are not in the input.
- Executability: whether the output passes existing validation such as unit tests, schema checks, or static analysis.
- Stability: whether variance is acceptable across repeated runs of the same input. Temperature and sampling parameters must be recorded.
- Efficiency: input and output tokens, turns, and wall-clock time spent to satisfy the same acceptance point.
An evaluation record can be simple; what matters is fixed, traceable fields:
case_id: refactor-042input_hash: <hash of the fixed input>constraints: no new dependencies; public interfaces must stay unchangedchecks:- unit_tests: pass/fail- schema: pass/fail- forbidden_claim: pass/failusage:prompt_tokens: ...completion_tokens: ...latency_ms: ...retries: ...notes: humans record the failure category only, scoring rules stay unchanged
When comparing two candidates, use the same golden set, the same validator, and the same retry limit. One may be faster or cheaper, but the rework cost after failures must be reported alongside. Many gaps that “look about the same” get amplified by format errors and multi-round patching.
Human blind evaluation is a useful supplement: place both outputs side by side, hide their source, and ask only which side could be merged directly. Blind evaluation resolves preference; automated validation resolves fact. Neither should substitute for the other.
Cost and Latency: Account Along the Success Path
Inference cost has at least four parts: input tokens, output tokens, failed retries, and human review. The longer the context, the larger the input share. Agentic loops resend the same repository instructions repeatedly, so the “cheap” list price can be eaten by turns.
Quote per task, not per token:
- Pick representative tasks and record the token and latency distribution of the success path — median and tail, not just the average.
- Cap retries. Beyond the cap, record the case as a failure, hand the sample to a human, and estimate person-hours.
- Compare options as “model cost of successful samples + person-hours of failed samples.”
- Look at throughput separately: for batch offline work, tokens/s and queue time; for interactive work, time to first token and full response.
Local or private clusters add reserved costs: amortization of accelerators or high-memory hosts, electricity, and idle rate. An interactive coding service left running around the clock costs money while idle; bursty usage fits on-demand inference better. Do not extrapolate a monthly bill from a single demo — demos usually run short contexts, with no retries and no concurrency queueing.
Deployment Model: Cloud vs On-Premises Inference
The two models solve different problems.
Inference on public or private cloud fits cases where models update frequently, peaks are unpredictable, the team does not want to maintain weights and runtimes, and processing data inside a contracted region is acceptable. What you inspect: identity isolation, log retention, content filtering, and region and network access control — not how busy the demo site looks. If you deploy on a cloud inference or container platform such as Baidu AI Cloud, make credentials, audit logs, and network boundaries go-live gates rather than additions afterwards.
On-premises or in-datacenter inference fits cases where data cannot leave the boundary, offline operation is required, or the goal is to push marginal call cost toward electricity and hardware amortization. The price is owning weight size, runtime, quantization, and the concurrency model. Quantization changes numerical behavior; do not assume “smaller means equivalent.” The correct approach is to compare full precision, or the officially recommended precision, against the quantization candidate on the same golden set, and record whether the quality regression lands inside the acceptable range. Estimate memory as weights + KV cache + runtime overhead; as concurrency rises, the cache often fills before the weights do. Which machine class actually fits must be established by measurement, not by someone else’s screenshot.
Either way, plan for degradation: on inference timeouts, format failures, or exhausted quotas, the flow should fall back to rules, retrieval, or a human rather than retrying without limit.
Licensing, Weights, and Compliance
“Can run locally” is not the same as “can be used freely.” At minimum, check:
- Whether the weight license permits commercial use, redistribution, and redistribution of fine-tuned variants.
- Whether the rights and obligations over training data and outputs are stated in the contract or license.
- Retention periods and regions for logs, prompts, and user content.
- Safety policy: whether refusal behavior on dangerous requests is stable and whether refusals are auditable.
- Supply chain: whether weight file checksums, runtime versions, and image provenance are traceable.
The license text published by the model provider is authoritative. Summaries in evaluation articles or on social platforms are not legal conclusions. If a license is unclear, do not write “private deployment” into the architecture as a commitment.
Common Misjudgments
Single-page visual comparison. A landing page or mockup that looks right does not prove the code is correct, the citations are real, or the workflow is complete. Generative interfaces are especially likely to match in detail while failing on constraints.
Single-point anecdote. One user’s hardware, quantization, prompts, and task differ from your production path; the conclusion does not transfer.
Timeline narratives. An industry figure’s judgment about “catching up” is an opinion, not an evaluation protocol. Engineering decisions cite reproducible task results only.
Peak capability only. Tail latency, format stability, and safety refusals equally determine whether a model reaches production. A model that is occasionally excellent and frequently unparseable costs more to integrate.
Hidden-brand, advertorial-style conclusions. Framing one model as free access to a given level of intelligence skips licensing, quality, and operations. A selection document records measurements and constraints, not a position.
Implementation Order
- Freeze the task set, acceptance checklist, and failure categories — start small, then expand.
- Run cloud and on-premises candidates under identical decoding parameters, recording tokens, latency, retries, and validation results.
- Regress on the quantization or truncation strategy intended for production and confirm the quality regression is acceptable.
- Check licensing, logging, and data residency, and produce a one-page compliance note.
- Use canary traffic to observe the real distribution, not just the golden set; refresh the golden set periodically with samples from production failures.
- Define shutdown conditions: switch to a fallback model or rules when cost exceeds budget, the safety-fallback rate rises, or the validation pass rate drops below threshold.
Models iterate quickly, so the evaluation protocol should be more stable than any model name. Bind inputs, validators, cost, and licensing together, and a team can swap inference backends without chasing slogans — while knowing what got better, what got more expensive, and which risks were left in place.
