Evaluating New-Generation LLMs in Practice: Hallucination, Code Generation, and Character-Level Processing
A practical guide to evaluating new-generation large language models beyond leaderboard scores. It covers three volatile capability dimensions — context hallucination, code generation, and character-level processing — plus the engineering constraint of concurrency and cost. You will find how to separate the three types of hallucination, why long chains of easy steps discriminate between average and stable capability, how to report `pass@1`, `avg@k` and `worst-case` together, a reusable Python evaluation skeleton, a three-tier (syntax / unit-test / engineering) acceptance standard for generated code, sandbox execution patterns, precise first-divergence diagnostics for character-level tasks, and six concrete governance actions for concurrency limits, retry backoff, caching, model routing, output length and offline batching — ending with a pre-launch checklist.
1. After the Generation Leap: The Two Questions Developers Actually Have to Answer
Model vendors now ship on increasingly tight cycles. Minor releases land back to back, and refreshing the base generation has become practically a fixed routine. Landing in the global top tier on overall capability is good news, but on the engineering side developers are not looking at leaderboard ranks. They are looking at two very concrete questions:
- Which layer actually improved, and is the improvement stable? A single high benchmark score may mean “gets it right if you sample a few more times,” or it may mean “gets it right every time.” The engineering implications of these two situations are entirely different.
- What changed in terms of accessibility and cost? After a new-generation model ships, the constraints you typically run into include lower per-account concurrency, the newest model being available only on the top subscription tier, and longer average outputs stacking on top of a higher unit price. These constraints change your architecture decisions directly.
Breaking those two questions down yields the three “volatile capability dimensions” this article covers — context hallucination, code generation, and character-level processing — plus one engineering constraint that cannot be ignored: concurrency and cost.
2. Context Hallucination: Treat “A Stable Perfect Score” as the Only Passing Bar
2.1 Separate the Three Types of Hallucination First
In engineering practice, talking about “hallucination” in a blanket way scatters your optimization effort. At minimum, distinguish three layers:
- Factual hallucination: the output does not match external world knowledge (knowledge staleness, domain errors).
- Context hallucination (faithfulness): the output does not match the supplied material — content that was not in the material gets “filled in,” and content that was in it gets rewritten or dropped.
- Character/symbol-level hallucination: affected by subword tokenization boundaries, showing up as single-point offsets in operations such as character lookup, counting, and reversal.
Generation-over-generation improvements usually vary widely across these three. Context faithfulness is the easiest to improve through training-data mix and alignment strategy, which is why it is often the first place where a new generation pulls ahead.
2.2 How to Design an Eval Set That Actually Discriminates
One counterintuitive but very useful conclusion: tasks that are not hard but require many steps distinguish “average capability” from “stable capability” far better than a single hard step does.
A rough model explains why. Assume a per-step accuracy of p, and a task with n independent steps that must all be correct. The probability of a first-pass success is roughly:
pass@1 ≈ p^n
If p = 0.97 and n = 12, then p^n ≈ 0.69; if p = 0.99, then p^n ≈ 0.89. In other words, a two-point improvement per step gets amplified into a twenty-point difference in pass rate on long-chain tasks. These tasks should therefore carry meaningful weight in your eval set. Typical shapes include:
- Multi-source extraction: locating facts scattered across several sections of a long document and consolidating them; any single mis-citation fails the case.
- Long-text summarization: fixing a compression ratio and an element-coverage requirement, then validating against a structured checklist for missed key constraints.
- Multi-step compliance decisions: for example, applying layers of filtering rules in order (ranges, priorities, tie handling) to reach a conclusion, with no step allowed to go wrong in the middle.
2.3 Metrics: Do Not Look at pass@1 Alone
A single metric hides stability problems. Collect at least these three values:
| Metric | Definition | Engineering meaning |
|---|---|---|
pass@1 |
Share of single samples whose first output passes | Expected first-attempt success rate on the real path |
avg@k |
Mean pass rate across k samples | Separates “consistently correct” from “occasionally correct” |
worst-case |
Worst pass rate across the k samples (aggregated per case) | Determines whether manual review can be dropped |
The typical observation is this: one generation shows a clear rise in avg@k while worst-case barely moves from the previous generation — “the average got better, the unlucky day did not.” Genuinely top-tier models, on the other hand, get worst-case close to a perfect score on these tasks too. That difference directly determines whether you dare to put the model into an unattended automated pipeline.
2.4 A Reproducible Evaluation Skeleton
import asyncioimport statisticsclass LLMClient:"""Minimal abstraction over a concrete model service, so different models can be swapped in for head-to-head comparison."""async def complete(self, messages, temperature=0.0, max_tokens=4096):raise NotImplementedErrorasync def run_case(client, case, k=5, sem=None):async with sem:results = []for _ in range(k):out = await client.complete(case["messages"], temperature=0.6)# the grader must return a deterministic 0-1 score; never use another LLM for subjective scoringresults.append(case["grader"](out))return {"id": case["id"],"pass1": 1 if results[0] else 0,"avg": round(sum(results) / len(results), 3),"worst": min(results),}async def evaluate(client, cases, k=5, concurrency=4):# concurrency must be aligned with the target model's rate limits, so throttling does not distort the eval itselfsem = asyncio.Semaphore(concurrency)rows = await asyncio.gather(*(run_case(client, c, k=k, sem=sem) for c in cases))summary = {"n_cases": len(rows),"pass@1": round(statistics.mean(r["pass1"] for r in rows), 3),"avg@k": round(statistics.mean(r["avg"] for r in rows), 3),"worst-case": round(statistics.mean(r["worst"] for r in rows), 3),}return summary, rows
Two traps that are easy to fall into. First, using a large model as the judge, which injects irreproducible noise into the scores themselves. Second, letting evaluation requests and production requests share the same concurrency quota, which turns throttling into a confounding variable in the eval.
3. Code Generation: Replace “It Looks Like It Runs” with a Three-Tier Acceptance Standard
3.1 Tiered Acceptance
Splitting “coding ability” into three tiers is what makes it visible where a generational gain actually lands:
- L1 Syntax tier: the artifact compiles or parses — no truncation, no syntax errors, no obviously incomplete function bodies.
- L2 Unit-test tier: passes when executed against the given samples and boundary conditions.
- L3 Engineering tier: multiple files, cross-module work, dependency management, exception handling, readability, plus refactoring and migration across files.
The common pattern with new models is that L1 moves from “occasionally passes” to “passes across repeated samples,” while the L2 first-pass rate rises — most visibly on complex algorithmic problems, where it approaches top-tier levels. L3 improvement typically lags, because engineering-grade tasks demand more of long-context consistency and tool use. For that reason, claiming “engineering capability is there” on the basis of L2 alone does not hold.
3.2 Reasoning and Non-Reasoning Modes Must Be Tested Separately
Turning reasoning (chain-of-thought) mode on gives a clear gain on complex algorithmic problems and is often a net loss on simple ones — extra tokens spent for the same result. Non-reasoning-mode coding ability usually improves as well, but by a smaller margin than reasoning mode. The engineering recommendation is to route by task difficulty rather than switch it on globally:
def choose_mode(task):if task.estimated_complexity >= 0.6: # supplied by a task classifier or hand-written rulesreturn {"reasoning": True, "max_tokens": 8192}return {"reasoning": False, "max_tokens": 2048}
3.3 Sandbox Code Generated by the Model
Model-generated code is untrusted input and must never be executed directly in the host environment or a production container. Run it in an isolated process or a separate container, with timeouts, resource quotas, and a least-privilege network policy.
import astimport osimport subprocessimport tempfileMINIMAL_ENV = {"PATH": os.environ.get("PATH", "")} # do not inherit the host's full environmentdef syntax_ok(code: str) -> bool:try:ast.parse(code)return Trueexcept SyntaxError:return Falsedef run_and_grade(code: str, tests: str, timeout: int = 10):"""Run the candidate code and its acceptance tests in a separate process inside a temporary directory."""with tempfile.TemporaryDirectory() as d:with open(os.path.join(d, "candidate.py"), "w", encoding="utf-8") as f:f.write(code)with open(os.path.join(d, "test_candidate.py"), "w", encoding="utf-8") as f:f.write(tests)try:proc = subprocess.run(["python", "-m", "unittest", "test_candidate", "-v"],cwd=d, env=MINIMAL_ENV, capture_output=True,text=True, timeout=timeout,)return proc.returncode, proc.stdout + proc.stderrexcept subprocess.TimeoutExpired:return -1, "TIMEOUT"
The full pre-launch chain should be: static parsing → isolated unit-test execution → static complexity and dependency checks (whether an unnecessary high-complexity algorithm was introduced, whether exception branches are unhandled) → manual spot-checking of L3 samples.
4. Character-Level Processing: A Small Weakness with an Outsized Impact
4.1 The Root Cause Is Representation, Not Intelligence
Character-level tasks (counting, reversal, substring extraction by index, case conversion, order-preserving deduplication) are a widespread weak spot, and the reason is subword tokenization: the model sees a sequence of tokens rather than a sequence of characters, so character-level operations require a kind of “second addressing” and tend to produce single-point offsets on long strings.
The typical behavior is a stable perfect score on simple cases and an enormous spread on complex ones. Reading the reasoning trace shows that failures usually are not a wrong method — some step in the middle misread a single character, and every subsequent derivation is built on that false premise. These errors also have one nasty property: thinking longer does not fix them. It only extends the distance over which the error propagates.
4.2 Three Mitigations, in Priority Order
- Push deterministic operations out of the architecture: hand character-level work to regexes, string libraries, or dedicated functions, and let the model handle intent recognition and parameter orchestration. This is the only approach that drives
worst-caseto 100%. - Force an intermediate representation via prompting: require the model to first emit a numbered, tabular mapping (index — character — cumulative count) and then draw conclusions from that table, which substantially reduces mental arithmetic.
- Localize precisely in evaluation: do not use overall accuracy alone; report the position of the first divergence so that “misread one character” and “wrong method” are separated.
def first_diff(expected: str, actual: str) -> str:for i, (a, b) in enumerate(zip(expected, actual)):if a != b:return f"First divergence at character {i}: expected {a!r}, got {b!r}"if len(expected) != len(actual):return f"Length mismatch: expected {len(expected)}, got {len(actual)}"return "identical"
With this function you can quantify the share of character-level failures caused by single-point offsets. If that share is high, the model’s method is right and the problem can be solved by pushing work out to tools. If methodological errors dominate, switch models or tighten the prompt constraints.
5. Concurrency, Output Bloat, and Cost: The Real Constraints of Model Iteration
5.1 Three Constraints Commonly Seen Right After Launch
- Concurrency limits: ordinary accounts may be capped at a very low value (for example 1), which means synchronous serial calls inflate end-to-end latency into “number of requests × per-request latency.”
- Tiered availability: the newest model is open only to the highest subscription tier, so most people on a team still route to the previous generation by default. This readily produces inconsistent observations — “a colleague says it is strong, my tests say it is weak.”
- Output bloat plus a unit-price increase: average output length rises on the new model, and output tokens usually account for a far larger share of cost than input tokens, so a price increase compounds multiplicatively.
A rough cost structure:
cost per call ≈ in_tokens × price_in + out_tokens × price_outif out_tokens grows by 60%, total cost rises markedly even with price_in unchanged
5.2 Six Governance Actions You Can Ship
- Concurrency gate plus a local queue: use a semaphore to keep concurrency inside the quota, so requests queue instead of fighting each other into throttling.
- Exponential backoff with jittered retries: retry only on retryable errors (throttling, timeouts, 5xx), and always add jitter to avoid a synchronized retry storm.
import asyncioimport randomasync def with_backoff(call, max_attempts=5, base=1.0, cap=30.0):for attempt in range(max_attempts):try:return await call()except RateLimitedError:if attempt == max_attempts - 1:raisedelay = min(cap, base * (2 ** attempt)) * (0.5 + random.random() / 2)await asyncio.sleep(delay)
- Prefix caching and semantic caching: put stable system prompts, few-shot examples, and tool descriptions at the front of the request so the cache can be reused; cache results for high-frequency repeated questions.
- Tiered model routing: send simple classification, format conversion, and summarization to cheaper or previous-generation models, and reserve the new generation for hard reasoning and complex coding. This is the single highest-payoff action for cost control.
- Output-length governance: set a sensible
max_tokens, use structured output to suppress verbose padding, and avoid returning the full reasoning trace to the frontend. - Offline batching to flatten peaks: route non-real-time work (batch evaluation, document batch processing, code scanning) through a message queue for asynchronous consumption outside throttling windows, so it does not compete with online traffic for quota.
5.3 The Observability You Must Have
Collect call volume, input/output tokens, time to first token, end-to-end latency P95, error-code distribution, and per-task cost along the model dimension, and wire up alerts. Without this data you cannot answer “cost went up 50% — was it unit price, output length, or the routing configuration?” On Baidu AI Cloud, capabilities of this kind are typically composed in a generic cloud-native way: container platforms host the evaluation jobs and code sandboxes with elastic scaling, object storage archives datasets and evaluation reports, log and monitoring services provide attribution for tokens and error rates, and message queues absorb offline batch-processing peaks.
6. Pre-Launch Checklist
- Is your own golden set at least 100 cases, and does it cover the real distribution of your business rather than recycled public questions?
- Do you report all three of
pass@1,avg@k, andworst-case? - Do long-chain, low-difficulty tasks carry enough weight (20% or more is a reasonable target) to probe stability?
- Do context tasks force source citation and validate the citations?
- Have code tasks completed L1/L2/L3 acceptance, and is generated code executed in an isolated environment?
- Do character-level tasks report the first divergence position, and have you assessed whether pushing work out to tools is feasible?
- Have you implemented a concurrency gate, jittered backoff retries, tiered routing, and an output-length cap?
- Is there a model-dimension dashboard and alerting for cost and latency attribution?
- Does the upgrade go through a staged rollout with a one-switch rollback to the previous generation?
7. Conclusion
The generational gains in context faithfulness, first-pass code correctness, and simple character-level tasks are real. The signal most worth watching on the engineering side is not the mean moving up, but whether the worst case converges at the same time — that is what decides whether you can remove the manual-review layer and its cost. At the same time, concurrency limits, tiered availability, and output bloat are reminders that model selection and engineering governance have to be done together. Turn evaluation into a reproducible regression pipeline, turn concurrency and cost into controllable variables behind gates, and hand deterministic operations such as character-level work to tools instead of the model’s “mental arithmetic.” That is the path from a new-generation model to genuinely stable productivity.
