A Complete Guide to Evaluating LLM Coding Ability and Inference Performance

    Choosing a large language model is increasingly difficult: new models ship constantly, and the few benchmark scores shared at launch rarely reflect how a model behaves in your own workloads. This article presents a practical, reproducible evaluation framework built around three dimensions—overall capability, coding ability, and inference performance. It shows how to define evaluation goals, build a three-tier metric system spanning capability, performance, and cost, and design coding tasks that progress from algorithm-level problems to full applications (including an interactive 3D solar system example with scoring guidance). It also provides a token-by-token benchmark script for measuring time to first token, staged throughput, tail latency, and error rate, walks through five common result-interpretation pitfalls, and explains how to turn evaluation into an ongoing engineering practice. The goal is to help readers select models using their own business data rather than a single leaderboard.

    Introduction: Models Evolve Fast, and Evaluation Must Keep Up

    Over the past two years, large language models have improved at a noticeably faster pace. New models are released at regular intervals, and each release often brings changes to inference architecture and output speed. For developers and enterprise users, this is both an opportunity and a challenge:

    • Opportunity: With more models to choose from, it is easier to find differentiated capabilities that match specific scenarios.
    • Challenge: The handful of benchmark scores published at launch events rarely translate directly into real-world experience in your own workloads.

    A recent coding-model contest that drew attention in overseas developer communities is a good illustration. Several contenders were compared head-to-head on the same set of tasks. Their overall results were close, yet the rankings reversed on a few individual tasks. This demonstrates two things:

    1. A high overall rank does not mean a model is the best fit for every scenario.
    2. Relying on a single leaderboard easily leads to one-sided conclusions when selecting a model.

    Rather than watching leaderboards, it is better to master a reproducible evaluation method and make decisions with your own data. This article focuses on three main threads—overall capability, coding ability, and inference performance—and breaks down metric design, experiment setup, result interpretation, and engineering practices into a reusable evaluation framework.

    1. First, Clarify What the Evaluation Needs to Answer

    Evaluation is not about ranking models for their own sake; it is about answering engineering questions. Before you begin, clarify four things:

    Question What to decide Output
    What to evaluate The core capabilities your business relies on (e.g., code generation, long-document understanding, structured output) Capability dimension list
    How to evaluate Sample source, scoring method, and whether to introduce a judge model Evaluation plan
    To what standard The target line and disqualification line for each metric Admission criteria
    Who evaluates Participation from the business, algorithm, and QA teams Review mechanism

    On that foundation, build a three-tier metric system:

    • Capability tier: answers “how well does the model perform,” covering overall capability and coding ability;
    • Performance tier: answers “is the model fast and stable,” covering time to first token, throughput, tail latency, and error rate;
    • Cost tier: answers “can we afford it,” covering per-request cost, average token consumption per call, and the extra overhead introduced by retries.

    The three tiers constrain one another. A more capable model tends to consume more inference time and tokens; a model tuned for extremely low latency may make trade-offs on complex tasks. The purpose of evaluation is not to crown an all-around champion, but to find the best balance point for each type of business scenario.

    2. Overall Capability: Characterizing a Model’s “Baseline” with Multi-Dimensional Tasks

    Overall capability measures a model’s foundational level across general-purpose tasks. It does not map to any single feature, but it determines how well the model can fall back gracefully on unfamiliar tasks and edge cases. Evaluation can cover five categories of tasks:

    1. Language understanding and generation: long-document summarization, content rewriting, copy polishing;
    2. Knowledge Q&A and logical reasoning: multi-step reasoning, common-sense judgment, mathematical calculation;
    3. Instruction following and structured output: returning JSON in a specified format, extracting fields, obeying length and tone constraints;
    4. Multi-turn dialogue and context retention: remembering settings across turns without forgetting or going off-topic;
    5. Long-document processing: long-document Q&A and information summarization in retrieval-augmented generation (RAG) scenarios.

    A few points are worth noting in practice.

    First, combine public leaderboards with your own sample set. Public leaderboards are broad and comparable, which makes them good for quickly shortlisting candidate models. However, their sample distribution may not match your business, and as training data grows, the risk that benchmark items have already been “seen” during training also rises. Build your own sample set as the primary source and use public leaderboards as a supplement. Your own samples should come directly from real business requests (properly anonymized); a few dozen to a few hundred items are enough, and the key is to stay close to the real distribution.

    Second, match the scoring method to the task type. For questions with a definite answer, use exact matching or rule-based checks. For open-ended questions, use a “judge-model scoring plus human spot-check” approach, and define the scoring rubric in advance. For example, instruction-following tasks can be broken into decidable sub-items such as “is the format correct,” “are all required fields present,” and “does it stay within the word limit,” each scored separately.

    Third, treat overall capability as an “admission threshold,” not a ranking basis. In practice, a more sensible approach is to set a floor: models scoring below the threshold are eliminated outright, and only the surviving models are compared on coding ability and inference performance. After all, what users perceive is the quality of specific features, not a composite score.

    3. Coding Ability: From “Writing Code” to “Delivering Runnable Results”

    Coding ability is one of the dimensions where today’s models differ most, and it is closest to the real needs of development work. Evaluation can proceed in three escalating layers.

    3.1 Layer One: Algorithm and Function Level

    This covers classic algorithm problems, function completion, and code explanation, aiming to assess a model’s fundamental coding skills. These tasks have clear answers and are well suited to automated judging with unit tests: generate multiple candidate solutions per problem, then measure the pass rate and the average number of attempts.

    Example (pseudocode; the function name is illustrative only):

    1. def eval_algorithm(model, tasks):
    2. passed, total_attempts = 0, 0
    3. for task in tasks: # task includes the problem statement and reference test cases
    4. solved = False
    5. for _ in range(MAX_ATTEMPTS):
    6. code = model.generate(task.prompt)
    7. if run_unit_tests(code, task.tests):
    8. solved = True
    9. break
    10. passed += int(solved)
    11. total_attempts += MAX_ATTEMPTS
    12. return {"pass_rate": passed / len(tasks),
    13. "avg_attempts": total_attempts / len(tasks)}

    3.2 Layer Two: Engineering Level

    This includes fixing bugs, refactoring locally, and continuing code in an existing style. This layer is closer to day-to-day work, and the focus shifts from “does it write correctly” to “does it change things well”: whether the change is correct, whether it introduces regressions, and whether it respects the original design. It is best to use real open-source snippets of moderate size (complying with license requirements) or internal anonymized code, judged by comparing tests before and after the change.

    3.3 Layer Three: Real-Task Level

    Generating a runnable front-end application from a complete requirement is currently one of the more discriminating evaluation methods. For example, ask the model to “build an interactive 3D solar system demo that opens directly in a browser”:

    • 8 planets arranged by relative distance and orbital period;
    • The sun at the center, emitting light;
    • Support for drag-to-rotate viewing and mouse-wheel zoom.

    The difficulty of this kind of task lies not in syntax but in the combination of spatial relationship modeling, continuous rendering, and interaction control. A working implementation looks roughly like this (illustrative code; dependency names are replaced with placeholders, shown only to indicate the technical points that must be covered):

    1. <!DOCTYPE html>
    2. <html lang="en">
    3. <head>
    4. <meta charset="UTF-8">
    5. <title>Interactive Solar System Demo</title>
    6. <style>body { margin: 0; overflow: hidden; }</style>
    7. </head>
    8. <body>
    9. <script type="module">
    10. import * as THREE from '<three-d-rendering-library>';
    11. import { OrbitControls } from '<orbit-controls>';
    12. const scene = new THREE.Scene();
    13. const camera = new THREE.PerspectiveCamera(60, innerWidth / innerHeight, 0.1, 1000);
    14. camera.position.set(0, 40, 80);
    15. const renderer = new THREE.WebGLRenderer({ antialias: true });
    16. renderer.setSize(innerWidth, innerHeight);
    17. document.body.appendChild(renderer.domElement);
    18. addEventListener('resize', () => {
    19. camera.aspect = innerWidth / innerHeight;
    20. camera.updateProjectionMatrix();
    21. renderer.setSize(innerWidth, innerHeight);
    22. });
    23. const controls = new OrbitControls(camera, renderer.domElement);
    24. controls.enableDamping = true; // keeps inertia after dragging for more natural interaction
    25. scene.add(new THREE.AmbientLight(0xffffff, 0.35));
    26. const sun = new THREE.Mesh(
    27. new THREE.SphereGeometry(3, 32, 32),
    28. new THREE.MeshBasicMaterial({ color: 0xffcc33 })
    29. );
    30. scene.add(sun);
    31. scene.add(new THREE.PointLight(0xffffff, 2)); // light source at the origin, i.e., the sun's position
    32. // [radius, orbital period (seconds), color]
    33. const PLANETS = [
    34. [5, 8, 0xa9a9a9], [7, 16, 0xe6b87a], [9, 24, 0x3366cc],
    35. [11, 32, 0xcc5533], [15, 60, 0xd2a679], [19, 84, 0xddcc99],
    36. [23, 110, 0x88bbdd], [27, 164, 0x4466ff]
    37. ];
    38. const planets = PLANETS.map(([r, period, color]) => {
    39. const pivot = new THREE.Group(); // parent node for orbital revolution
    40. const mesh = new THREE.Mesh(
    41. new THREE.SphereGeometry(0.6, 24, 24),
    42. new THREE.MeshStandardMaterial({ color, roughness: 0.8 })
    43. );
    44. mesh.position.x = r;
    45. pivot.add(mesh);
    46. scene.add(pivot);
    47. return { pivot, speed: (2 * Math.PI) / period };
    48. });
    49. let last = performance.now();
    50. renderer.setAnimationLoop((now) => {
    51. const dt = Math.min((now - last) / 1000, 0.1); // cap per-frame step to avoid jumps when returning to the tab
    52. last = now;
    53. planets.forEach(p => p.pivot.rotation.y += p.speed * dt);
    54. controls.update();
    55. renderer.render(scene, camera);
    56. });
    57. </script>
    58. </body>
    59. </html>

    Three details in this sample code can serve as references for judging the quality of generated code:

    • It uses orbital “parent nodes” to carry the planets around the sun, avoiding manual per-frame coordinate calculation;
    • The animation advances based on real elapsed time rather than a fixed frame rate, keeping orbital speed consistent across screens with different refresh rates;
    • It caps the maximum per-frame step to avoid the scene “jumping” when the page is brought back into view.

    Similar evaluations can also be designed around topics like a “3D growing tree”: the tree grows progressively over time, branches split by probability, and clicking replays the growth. Such tasks assess a combined ability in recursive structures, animation state management, and interaction control.

    3.4 How to Score Visualization Coding Tasks

    These tasks require both objective checks and subjective judgment. A combined “automated validation plus blind human review” approach is recommended.

    Automated validation answers “does it run, and does it behave correctly,” for example:

    • Can the files load without errors, and does the browser console output any errors?
    • Are the key elements present, such as the number of planets, a central light source, and the wired-up orbit controller?
    • After running continuously for a few seconds, do the on-screen objects change position over time?

    Blind human review answers “is it usable and does it look good.” With the model source hidden, reviewers score across four dimensions—functional completeness, visual quality, interaction experience, and code quality—on a 1–5 scale, keeping screenshots as the basis for review.

    Three lessons are worth noting in practice:

    • It is normal for multiple models to trade wins on these tasks. The same model may lead on the “solar system” but lag on the “growing tree,” so the evaluation set should cover several topics to avoid a conclusion being amplified by a single problem.
    • Visual quality and code quality should be scored separately. A dazzling screen does not equal a reliable implementation. Code with chaotic variable names and magic numbers everywhere may run, but it is hard to maintain later.
    • The evaluation process must be reproducible. Archive the tasks, scoring rubric, rendering environment, and screenshots together so results can be re-tested after models are updated.

    4. Inference Performance: Breaking “Fast” into Measurable Metrics

    The “speed” users perceive is actually determined by several stages working together. Simply saying “fast response” is meaningless; at a minimum, it should be broken down into the following metrics:

    Metric Definition Experience it mainly affects
    Time to first token (TTFT) Time from sending the request to receiving the first visible token Whether users feel they get an “instant response”
    Answer-phase throughput Tokens output per unit time after the first token Whether streaming output feels smooth
    Thinking-phase throughput Output speed during the reasoning phase when thinking mode is enabled How long users wait before complex tasks start answering
    End-to-end generation latency Total time from sending the request to the last token being returned Completion speed of long answers and batch tasks
    Tail latency (P50/P95) The median / 95th-percentile value of all request latencies when sorted Whether long-tail requests drag down the overall experience
    Error rate Proportion of timeouts, rate limits, and server-side errors Whether the service is stable and available

    Two easily overlooked points deserve emphasis.

    First, median latency describes only half of the requests. Suppose the median response time is 3.1 seconds. The intuitive reading is “about half of requests return within roughly 3 seconds.” That looks fine, but the other half may be noticeably slower. So beyond the median, you should also track percentiles such as P95 and P99. If the P95 latency is several times the median, the service has clear long-tail jitter under load or with long inputs, which under high concurrency can surface as visible stuttering.

    Second, a “turbo mode” is not something to use all the time. Many inference services offer multiple operating modes, each with different emphases on latency, output length, or thinking depth. During evaluation, treat each mode as an independent configuration and measure it separately, then decide, based on business needs, which one to use by default and under what conditions to switch.

    4.1 Why You Must Test It Yourself

    Response speed is affected by many factors:

    • Server-side queuing and load scheduling;
    • Input length (the longer the context, the greater the preprocessing overhead before the first token, generally);
    • Output length;
    • Whether the thinking process is enabled;
    • Network latency between the client’s region and the service node.

    The measured results for the same model can vary widely across time periods and calling methods, and vendor-published figures usually reflect performance under specific conditions only. Therefore, performance evaluation must cover the most common input/output sizes in your target workload and be conducted during time windows closer to production.

    4.2 Evaluation Script: Collecting Latency and Throughput Token by Token

    The script below targets APIs with a streaming response format. It reads the response chunk by chunk and separately measures time to first token, thinking-phase and answer-phase throughput, and overall generation speed. The model name, endpoint, and other values are shown as placeholders; replace them with the parameters of your platform when using it.

    1. import time
    2. from concurrent.futures import ThreadPoolExecutor
    3. import numpy as np
    4. import requests
    5. API_URL = "https://<api-endpoint>" # streaming endpoint of your platform
    6. API_KEY = "<api-key>" # recommended to read from an env var, do not hardcode
    7. MODEL = "<model-id>" # use the identifier provided by your platform
    8. HEADERS = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}
    9. def measure_once(prompt: str, max_tokens: int = 512) -> dict:
    10. """Call the streaming endpoint once and collect time to first token plus staged throughput"""
    11. payload = {
    12. "model": MODEL,
    13. "messages": [{"role": "user", "content": prompt}],
    14. "stream": True,
    15. "max_tokens": max_tokens,
    16. }
    17. start = time.perf_counter() # timing start: before the request is sent
    18. first_visible = None # arrival time of the first visible token
    19. first_token = None # arrival time of the first token (including thinking content)
    20. think_chars = answer_chars = 0
    21. usage = {}
    22. with requests.post(API_URL, headers=HEADERS, json=payload, stream=True, timeout=120) as resp:
    23. resp.raise_for_status()
    24. for raw in resp.iter_lines(decode_unicode=True):
    25. if not raw or not raw.startswith("data:"):
    26. continue
    27. data = raw[len("data:"):].strip()
    28. if data == "[DONE]": # end marker
    29. break
    30. chunk = resp.json() if False else __import__("json").loads(data)
    31. now = time.perf_counter()
    32. if chunk.get("usage"): # some implementations return usage in the last chunk
    33. usage = chunk["usage"]
    34. choice = (chunk.get("choices") or [{}])[0]
    35. delta = choice.get("delta", {})
    36. # the field name for thinking varies by platform; follow the actual API docs
    37. thinking = delta.get("reasoning_content") or ""
    38. answer = delta.get("content") or ""
    39. if (thinking or answer) and first_token is None:
    40. first_token = now
    41. if answer and first_visible is None:
    42. first_visible = now # only count the first token the user can actually see
    43. think_chars += len(thinking)
    44. answer_chars += len(answer)
    45. end = time.perf_counter()
    46. if first_visible is None or answer_chars == 0:
    47. return {"ok": False, "raw_usage": usage}
    48. total_tokens = usage.get("completion_tokens")
    49. if total_tokens is None: # if usage is not returned, roughly estimate by chars (~1.5 chars/token for mixed text)
    50. total_tokens = round((think_chars + answer_chars) / 1.5)
    51. answer_tokens = usage.get("completion_tokens_details", {}).get("reasoning_tokens")
    52. if answer_tokens is None:
    53. answer_tokens = max(total_tokens - round(think_chars / 1.5), 1)
    54. t_first = (first_token - start) if first_token else None
    55. t_visible = first_visible - start
    56. answer_span = end - first_visible
    57. think_span = (first_visible - first_token) if (first_token and first_visible and think_chars > 0) else None
    58. return {
    59. "ok": True,
    60. "ttft_s": round(t_visible, 3), # TTFT (visible)
    61. "think_ttft_s": round(t_first, 3) if t_first else None,
    62. "total_s": round(end - start, 3),
    63. "total_tokens": total_tokens,
    64. "answer_tokens": answer_tokens,
    65. "total_tps": round(total_tokens / (end - start), 1),
    66. # answer throughput counts only the generation phase after the first token, better reflecting "reading speed"
    67. "answer_tps": round(answer_tokens / answer_span, 1),
    68. "think_tps": round((total_tokens - answer_tokens) / think_span, 1) if think_span else None,
    69. "raw_usage": usage,
    70. }
    71. def percentile(values, p):
    72. return round(float(np.percentile(values, p)), 3)
    73. def run_benchmark(prompts, concurrency=8):
    74. """Concurrent multi-threaded load test; outputs median/tail latency and throughput summary"""
    75. tasks = [(p, i) for i, p in enumerate(prompts)]
    76. with ThreadPoolExecutor(max_workers=concurrency) as pool:
    77. results = list(pool.map(lambda args: measure_once(args[0]), tasks))
    78. ok = [r for r in results if r["ok"]]
    79. failed = len(results) - len(ok)
    80. if not ok:
    81. raise RuntimeError(f"All requests failed: {failed}")
    82. ttfts = [r["ttft_s"] for r in ok]
    83. totals = [r["total_s"] for r in ok]
    84. tps_list = [r["answer_tps"] for r in ok]
    85. return {
    86. "samples": len(results),
    87. "success": len(ok),
    88. "failed": failed,
    89. "ttft_p50": percentile(ttfts, 50),
    90. "ttft_p95": percentile(ttfts, 95),
    91. "total_p50": percentile(totals, 50),
    92. "total_p95": percentile(totals, 95),
    93. "answer_tps_p50": percentile(tps_list, 50),
    94. }
    95. if __name__ == "__main__":
    96. # 1) single-request baseline: confirm the endpoint works and observe single-request latency and throughput
    97. print(measure_once("Explain load balancing in one sentence."))
    98. # 2) concurrency test: simulate latency and throughput under realistic traffic
    99. prompts = ["Write a quicksort function with a brief explanation."] * 40
    100. print(run_benchmark(prompts, concurrency=8))

    Several points in the script design are key:

    • Layered concurrency: measure a single-request baseline first, then run a concurrency test. The former reflects the model’s raw speed; the latter reflects how the service queues and schedules under load.
    • Staged timing: time the thinking process and the visible answer separately. Otherwise, once thinking mode is enabled, the time spent “thinking” is wrongly counted into the denominator of answer throughput, distorting the data.
    • Use the token usage returned by the API directly: prefer the token usage in the response over estimating by character count. Different models use different tokenizers, so character counts cannot be compared directly across models.
    • Pay attention to failed samples: count failed requests separately; do not mix them with successful ones when computing averages, or you will mask stability problems.

    5. How to Read Results: Five Common Pitfalls

    Pitfall one: treating the leaderboard rank as the final conclusion. Leaderboards use general-purpose samples, but your users care about specific scenarios. Build on leaderboard shortlisting, then re-test with your business samples and treat the re-test results as authoritative.

    Pitfall two: looking only at the average or the median. Averages are easily pulled up or down by a few extreme values, and the median describes only half of the requests. Production cares more about tail behavior like P95/P99, so report the mean, median, and percentiles together.

    Pitfall three: assuming high throughput means a good experience. If time to first token is long, users first feel “stuck,” and no amount of throughput can fully compensate. For interactive applications, time to first token is often more critical than generation speed.

    Pitfall four: ignoring the extra overhead from the thinking process. Once thinking mode is enabled, answer quality on complex tasks may improve, but both the number of output tokens and the elapsed time increase. Measure “enabled” and “disabled” configurations separately during evaluation, then decide the strategy based on task difficulty—for example, disable thinking for simple questions and enable it for complex ones.

    Pitfall five: neglecting cost as a dimension. Beyond speed, factor in token consumption:

    Cost per call ≈ (input tokens + output tokens) × unit price

    Retries, timeouts, and fallbacks all generate extra calls. An evaluation report should list quality and cost side by side, avoiding the “pick the fastest, ignore the bill” trap.

    6. Implementation Advice: Turning Evaluation into a Lasting Capability

    Evaluation is not a one-time task. Models keep updating, and business keeps changing. To integrate evaluation into the development workflow, focus on four things.

    1. Build a unified access layer to lower switching costs. Access different models through a unified LLM platform so that application code depends only on a standard interface. For example, a unified model-serving platform such as Baidu AI Cloud’s Qianfan provides an invocation style compatible with common interface conventions, so the application side only needs to change the model identifier to switch between models—making repeated testing and comparison easy. Unified access also lets you manage authentication, rate limiting, and usage centrally, which simplifies cost accounting.

    2. Route by scenario and match models to needs.

    • For high-concurrency, low-latency interactive scenarios, prioritize models with low time to first token and stable throughput;
    • For complex code generation and deep reasoning, slightly longer latency is acceptable, so prioritize more capable models.

    When conditions allow, let the two types of models work together—for example, first classify intent, then route to different models. However, a multi-model pipeline increases system complexity, so whether to adopt it should be weighed against the actual benefit.

    3. Set up continuous monitoring. Turn the evaluation script into a scheduled inspection job that continuously collects latency, throughput, error rate, and token consumption. Integrate common cloud capabilities such as logging, monitoring, and alerting, and trigger alerts promptly for spikes in P95 latency or rising error rates, so that a model update or service fluctuation does not silently degrade the user experience.

    4. Version your evaluation assets. The evaluation set, tasks, prompts, and scoring rubric should all be under version control. After each model update, re-test with the same assets so that results across rounds can be compared vertically, preventing distorted conclusions from changing tasks or standards.

    Conclusion

    The endpoint of model evaluation is not a leaderboard, but a set of data that answers business questions. Overall capability sets a model’s baseline, coding ability determines its ceiling in development scenarios, and inference performance directly shapes every user interaction. Break the three into measurable, reproducible metrics and validate them repeatedly with your own business data—only then can you see through the fog of leaderboards and marketing to find the model that truly fits. As models evolve faster than ever, this ability to “evaluate yourself, and keep evaluating” is itself part of a team’s technical competitiveness.