Evaluating LLM Coding Ability and Putting It into Engineering Practice
This guide explains how to evaluate large language models for real software engineering work, covering long-context understanding, code reasoning, multi-file changes, defect repair, and tool-driven iteration. It outlines the underlying techniques that affect context length and inference cost, then walks through access options, API calling patterns, typical coding scenarios, and how to build a programming agent around plan–act–test loops.
1. Introduction: Why coding ability still matters
Once large language models enter the software development workflow, the question is no longer whether they can write code, but whether they can deliver reliably in real projects. Function-level completion and simple script generation are relatively mature. Cross-file understanding, locating historical defects, and multi-step change-and-verify work remain hard—and those tasks largely determine engineering value.
Evaluating a coding-oriented model requires three kinds of capability:
- Input: Can the model ingest enough code, documentation, and logs in one pass to form a complete context?
- Reasoning: Can it follow control flow, data flow, and interface contracts, and judge whether a change would break existing logic?
- Engineering: Can it keep a task moving through tool calls, test runs, and multi-turn iteration, and converge when something fails?
The rest of this article covers those dimensions: technical principles, how to connect, typical scenarios, troubleshooting, and copy-ready code examples.
2. What programming work actually demands
2.1 Long-context understanding
Mid-size and large projects often include dozens of modules, tens of thousands of lines of code, plus API docs, configs, and tests. If the model can only handle a few thousand characters, developers must curate context by hand, and that curation quality dominates output quality.
In practice, long context shows up in two ways: locating the relevant snippets in a long input, and staying consistent across long-range dependencies. A parameter may be defined at the top of a file and used at the bottom; the model has to connect the two.
2.2 Code logic reasoning
Reasoning includes boundary conditions, exception paths, and state changes under concurrency. A useful evaluation prompt: given code with a race condition, can the model point to the problem and propose locking or atomic operations?
2.3 Multi-file and cross-module work
Real changes often touch interface definitions, implementations, callers, and tests together. Field names, types, and defaults must stay consistent across those files.
2.4 Defect localization and repair
Localization means inferring root cause from errors, stacks, logs, and reproduction steps. Repair means a small diff that does not introduce new issues. Watch whether the model “helpfully” edits unrelated code.
2.5 Tool use and autonomous iteration
Tool use means reading files, running tests, and looking up docs when needed. Autonomous iteration means revising the plan from test results until tests pass or a stop condition is hit. Those two capabilities decide whether the model stays a suggestion tool or becomes an execution tool.
3. Underlying techniques: long context and inference efficiency
Long context is limited by compute cost. In full attention, every token attends to every other token, so cost grows roughly with the square of sequence length. At million-token scale that is rarely practical. Common approaches include:
Sparse attention: Each position attends only to a subset of history (for example a local window plus a few global keys). Cost grows more slowly, but important information can be missed.
KV cache reuse: Cache computed key/value vectors at inference time. For multi-turn chat or repeated queries over the same repository, this cuts latency.
Chunking and retrieval: Split a very long input, retrieve relevant chunks, then feed those to the model. Cost is lower, but quality depends on retrieval, and dependencies can be lost at chunk boundaries.
Mixture of experts: Total parameter count is large, but each token activates only a subset. Capacity stays high while per-token compute is controlled.
None of these is universally best. Choose based on required context length, latency, and hardware. You do not need the low-level implementation, but you should understand the impact on latency, cost, and quality so you can size inputs reasonably.
4. Environment and access
4.1 Three common access modes
| Access mode | Typical use | Main concerns |
|---|---|---|
| Cloud API | Fast trials, small teams, elastic demand | Network latency, usage cost, data-residency and compliance |
| Private deployment | Strict data-security requirements | Hardware, operations, version upgrades |
| Local small models | Offline use, simple completion, privacy-sensitive tasks | Lower capability ceiling; keep the task scope tight |
4.2 Pre-integration checklist
- Data classification: Decide which code and docs may leave the network and which must stay internal.
- Capacity: Estimate daily call volume, average input/output length, and peak concurrency.
- Secrets: Inject keys from environment variables or a secret manager; never commit them or log them.
- Failure plan: If the service is down, queue work or fail over to a backup.
5. API calling practice
The following example uses only the Python standard library. It assumes an HTTP chat-style API; use the URL and model name from the relevant API documentation.
import jsonimport osimport timeimport urllib.errorimport urllib.requestAPI_URL = os.environ["LLM_API_URL"]API_KEY = os.environ["LLM_API_KEY"]RETRYABLE_CODES = {429, 500, 502, 503, 504}def chat(messages, model, temperature=0.2, timeout=60, max_retries=3):"""Send a chat request with exponential backoff for rate limits and transient failures."""payload = json.dumps({"model": model,"messages": messages,"temperature": temperature,}).encode("utf-8")req = urllib.request.Request(API_URL,data=payload,headers={"Content-Type": "application/json","Authorization": f"Bearer {API_KEY}",},method="POST",)for attempt in range(1, max_retries + 1):try:with urllib.request.urlopen(req, timeout=timeout) as resp:body = json.loads(resp.read().decode("utf-8"))return body["choices"][0]["message"]["content"]except urllib.error.HTTPError as exc:if exc.code not in RETRYABLE_CODES or attempt == max_retries:raiseexcept urllib.error.URLError:if attempt == max_retries:raisetime.sleep(2 ** attempt)
Notes:
- Temperature: Code generation and bugfix usually need more deterministic output; 0–0.3 is a better range.
- Timeout and retries: Retry only rate limits and temporary server errors. Client/auth errors should fail immediately.
- Validate output: Check syntax, run tests, or validate structure before applying anything.
If you need structured results, specify JSON in the prompt and validate in code:
import jsondef parse_fix_plan(text):"""Check that a repair plan matches the expected structure."""try:plan = json.loads(text)except json.JSONDecodeError:return Nonerequired = {"file", "reason", "patch"}if not isinstance(plan, dict) or not required.issubset(plan):return Nonereturn plan
On parse failure, send the error back and ask for a new response. Cap retries, then hand off to a human.
6. Typical scenarios
6.1 Code generation
A strong generation prompt usually has five parts: role and goal, relevant context, constraints, output format, and acceptance criteria. Example:
[Goal] Add a "export by time range" API to the order service.[Context] Existing order query functions and data model:<paste related code>[Constraints]- Reuse existing pagination and error-code conventions;- Do not change the response shape of existing APIs;- Maximum time range is 31 days; return a parameter error if exceeded.[Output]- List files to change and why;- Then the code diffs;- Then matching unit tests.[Acceptance] New tests pass; existing tests do not regress.
Clearer constraints make results easier to verify. Generation without acceptance criteria usually costs a lot of manual rework.
6.2 Refactoring
The point of a refactor is unchanged behavior. Prefer small steps: one kind of change at a time (rename, extract function, split a module), and run tests after each step. Asking the model to rewrite thousands of lines in one shot is hard to review and hard to bisect if something regresses.
6.3 Defect localization and repair
Provide the error, stack, related logs, reproduction steps, and recent changes. Ask the model for a root-cause hypothesis first, then a way to verify, then a fix. Separating “hypothesis” from “evidence” makes human review easier.
Limit the edit surface. State which files may change and require a justification for each edit.
6.4 Test generation
Models are useful for boundary values, exception paths, and concurrency cases. Check that assertions actually mean something; tests that pass as long as nothing throws are weak. A useful test should fail if the production code is broken.
6.5 Review assistance
Let the model do a first pass on naming, null risks, resource leaks, and sensitive data in logs. Humans should still own business logic, architecture, and long-term maintainability.
7. Building an engineering agent
An agent plans around a goal, calls tools, observes results, and adjusts. A programming agent typically loops as follows:
- Understand the task: inputs, outputs, and acceptance criteria.
- Plan: files to read and files to change.
- Read code and gather context.
- Propose edits and apply them in an isolated environment.
- Run tests and static checks.
- Revise from the results, or stop when tests pass or a stop condition is reached.
