LLM Coding Ability Tested: How Post-Training Produces Better Code
A hands-on look at modern LLM coding ability. Through a case study in which a desktop tool for batch audio extraction was built in about ten minutes, this article explains how post-training—without changing the base model—can significantly improve code quality. It covers the three main post-training techniques (supervised fine-tuning, preference optimization, and reinforcement learning from execution feedback), the real limits of post-training, and a five-dimension framework for judging whether generated code actually runs. It also walks through a reusable Python implementation, common pitfalls such as a frozen UI, command injection, and missing timeouts, and practical engineering practices for turning a one-off result into reliable team output.
1. Introduction: A Working Desktop Tool in Ten Minutes—What Does It Prove?
Recently, many users of interactive coding platforms have experienced the same thing: in the morning they hear that a next-generation model is about to open up, and by the afternoon, logging into the platform, they are told the new model is already live. Half curious and half skeptical, the author gave the assistant a request that was anything but simple: write a Python desktop application with a graphical interface to batch-extract audio from a set of video files.
The interaction went roughly like this:
- First have the assistant break down the requirements and propose an overall design, confirm the approach, then generate the code.
- Copy the code locally, run a syntax check, and then run it directly.
- Load in the test files and execute; the software processed them successfully in one pass, and the interface showed no obvious lag.
It took about ten minutes and consumed fewer than thirty interaction credits. The overall experience can be summed up in one sentence: the “engineering friction” between describing a need and getting a usable result has shrunk noticeably.
The question worth asking is not “why so fast” but “why did it work.” For developers, improvements in LLM coding ability come mainly from two directions:
- Switching the base model: high cost, and capability changes are unpredictable.
- Continued post-training on the existing base model: more targeted, with relatively controllable cost.
The model in this case took the post-training route. Public information shows that its coding ability improved markedly over the previous version, with overall performance in the first tier, yet without changing the base model. This raises two core questions:
- Why can post-training significantly improve coding ability?
- How can you tell whether a model’s code “looks right” or “really runs”?
This article answers these two questions in turn.
2. Getting Stronger Without Changing the Base Model: How Post-Training Reshapes Coding Ability
2.1 Base model and post-training: each covers a different stage
Put simply, the base model determines a model’s “knowledge boundary” and its “language and logic baseline”—like a well-read person. Post-training determines “how it organizes knowledge into an answer that meets the requirements”—like receiving targeted, systematic training.
In a coding context, the division of labor is:
- The base model provides the syntax system, common design patterns, and large amounts of code corpus.
- Post-training moves the model from “can write similar code” to “can write correct, runnable, and maintainable code for a specific requirement.”
Post-training mainly involves three types of techniques, which happen to complement each other on coding tasks.
2.2 Three main post-training techniques
1. Supervised fine-tuning: learning “what a good answer looks like”
Fine-tune the model on high-quality “requirement—code” sample pairs, which usually cover four task types:
- Generation from scratch;
- Fixing bugs;
- Explaining code;
- Refactoring and optimization.
Beyond function-level code, repository-level samples also teach the model to organize code across files and understand project structure—exactly the most common form of work in real development.
2. Preference optimization: learning “which answer is better”
Human annotators or stronger models rank multiple candidate implementations of the same requirement, and preference optimization methods then push the model’s output closer to the high-quality answers. Typical preferences include readability, completeness of boundary-condition handling, and soundness of exception handling.
3. Reinforcement learning from execution feedback: letting code “take its own exam”
This is the most direct and most distinctive means of improving coding ability. The code the model generates is placed in a sandbox environment to compile, run, and execute unit tests:
- If it passes, it receives a positive reward;
- If it fails, the error message is fed back to the model so it can correct the code.
Because the correctness of code can be judged automatically by a program, this kind of training can reliably obtain massive, objective feedback signals.
2.3 Three concrete manifestations of the improvement
(1) More accurate requirement understanding: from “translating code” to “understanding intent”
Past models commonly showed two types of problems:
- A slight change in the requirement led to an off-target answer;
- They implemented the function literally, missing default behavior.
After post-training, the model maps a natural-language requirement to the correct program structure: it identifies inputs and outputs, implicit constraints, and boundary conditions, rather than simply piling up lines of code.
(2) Higher code quality: it runs, and it runs well
A coding benchmark frequently cited in public evaluations is SWE-bench Verified, which uses real problems from open-source repositories to test a model’s ability to fix code.
Looking at how models’ scores on this benchmark have changed in recent years shows that relying on post-training alone can deliver substantial capability gains. In practice, this shows up as:
- Proactively filling in exception handling and resource release;
- Generated code that better matches language conventions and project standards;
- More thorough consideration of boundary conditions.
(3) Stronger iteration and self-correction ability
Faced with an error, the model can use the error message to locate the problem and propose a fix, rather than starting over. Models trained with reinforcement learning also tend to analyze the approach and break down the task before writing code.
2.4 Viewing post-training rationally: capabilities have limits, and risks must be guarded against
Post-training is not a cure-all, and two points need particular attention.
First, it cannot create knowledge out of nothing. The model will not know about libraries, frameworks, or new API interfaces that appeared only after the post-training stage. A new version released after the knowledge cutoff can easily lead to generating APIs that “look correct but don’t actually exist.” This is known as a “hallucinated API.”
Second, the reward signal may induce “exam-taking behavior.” If evaluation relies too heavily on automatically judgeable metrics—such as whether unit tests pass—the model may learn to “game the tests” while neglecting dimensions that are hard to measure automatically, like performance and readability.
For this reason, the capability that post-training brings is better understood as “the ability to turn existing knowledge into reliable output,” not as being “omniscient and omnipotent.”
3. How to Tell If It “Really Runs”: A Five-Dimension Evaluation Framework
Back to the opening case. Behind the feeling of “producing a complete piece of software in ten minutes,” what developers really care about is: how to judge, in the shortest time, whether a piece of generated code is trustworthy. I recommend evaluating across the following five dimensions.
| Dimension | Core question | Recommended verification |
|---|---|---|
| Correctness | Does the output match the expected requirements | Check item by item against the requirements; verify output with representative inputs |
| Runnability | Can it run in a clean environment on the first try | Syntax check, dependency check, sandbox execution |
| Completeness | Are exception, boundary, and concurrency scenarios handled | Construct abnormal inputs and boundary cases |
| Maintainability | Is the structure clear and easy to extend | Manual code review |
| Security | Are there risks such as injection or path traversal | Security scanning and review of key code |
The five dimensions can be scored quantitatively with the formula below. In practice, the weight of each dimension should be adjusted to the scenario:
Score = 0.3 × Correctness + 0.25 × Runnability + 0.2 × Completeness + 0.15 × Maintainability + 0.1 × Security
3.1 Runnability: the first gate
Runnability is the first gate. Check, in order:
- Syntax: can it pass a syntax check;
- Dependencies: are all imported libraries installed and version-compatible;
- Execution: can it actually run in a sandbox or isolated environment.
Running in a sandbox also carries a security meaning: generated code may contain dangerous operations such as deleting files or making network requests, and isolated execution avoids accidentally harming data on the development machine.
3.2 Completeness: what separates “works” from “usable”
A feature that runs is not the same as one you can ship. Four types of scenarios deserve particular attention:
- Empty input or a nonexistent file;
- Paths containing Chinese characters, spaces, or special characters;
- Whether the interface stays responsive when processing takes a long time;
- Whether the whole can continue when some tasks fail.
3.3 Maintainability: leaving room for the next iteration
Maintainability is often overlooked, yet it directly determines whether the code can become a long-term asset for the project. You can use code-standard checking tools to automatically scan metrics such as naming, function length, and cyclomatic complexity, and then combine that with manual review to judge whether the layering is reasonable.
3.4 Pitfalls: four common traps in evaluation
Trap 1: Only looking at public leaderboards. Leaderboard problems and prompts are fixed and differ greatly from real business needs. A model’s performance on a leaderboard cannot be equated directly with its performance in your scenario; be sure to validate with your own requirements.
Trap 2: Ignoring the effect of prompts. For the same requirement, different ways of organizing the prompt can yield very different results. During evaluation, keep the prompt as fixed as possible, to avoid mistaking prompt differences for model-capability differences.
Trap 3: A temperature setting that is too high. A high temperature makes output more random and increases the chance of getting different code for the same requirement across runs. When generating code, a lower value is usually taken to improve stability.
Trap 4: A polluted context. In a long conversation, erroneous code from earlier turns stays in the context and interferes with subsequent generation. Once a turn’s output shows a clear deviation, decisively start a new conversation.
4. Hands-On: Testing “Runnability” with a Desktop Tool
This section uses the opening case as a blueprint to fully reconstruct a reusable engineering practice. The code uses Python as an example and depends only on the standard library; video processing capability is provided by an external command-line program configured at deployment, and the specific tool and parameters can be swapped for the runtime environment.
4.1 Step 1: Say clearly what you want
The developer’s original request was “help me write a tool to batch-extract audio.” The sentence looks clear, but it actually lacks a lot of key information:
- Where does the input come from?
- What output format?
- Should the interface show feedback during processing?
- What happens when one file fails?
If you throw this sentence straight at the model, you’ll most likely get code that “runs but is hard to use.” A more professional approach is to define a clear requirements list before starting. For this case:
- Input: the user selects one or more video files;
- Output: the output directory is user-selectable, the audio format is selectable, and the bitrate is configurable;
- Interaction: show progress, the current file, and elapsed time in real time during processing;
- Fault tolerance: one file failing does not affect the rest of the tasks, and a failure list is output at the end;
- Experience: the interface stays responsive throughout processing, with support for stopping at any time.
On this basis, you can use a structured prompt template to express the requirements, constraints, and acceptance criteria all at once:
[Role] Senior desktop application development engineer[Task] Build a desktop tool that batch-extracts audio from video files[Functional requirements]1. Support selecting a single or multiple video files2. Configurable output directory, audio format, and bitrate3. Display total progress, current file, and elapsed processing time in real time4. A single file failure must not interrupt the overall flow; list failed files at the end[Non-functional requirements]1. The UI must stay responsive during processing and must not block the UI thread2. Support stopping at any time; no half-written output files after stopping3. Do not concatenate command strings yourself; external program calls must set a timeout[Implementation constraints] Use only the language standard library; no extra third-party dependencies[Output] Complete runnable code with runtime environment and usage instructions
The value of a structured prompt is that it turns implicit requirements like “no UI freeze” and “failures don’t interrupt” into explicit, verifiable clauses. Whether the model satisfies these clauses is precisely a good observation point for testing its code engineering maturity.
4.2 Step 2: Why “a UI that doesn’t freeze” is the number-one killer for beginners
Many developers have had this experience: once the program processes a large file, the window title bar shows “Not Responding,” dragging the window leaves ghosting, and the system may even deem it unresponsive.
The root cause is that most GUI frameworks use a single-threaded event loop: UI drawing, button clicks, timer callbacks, and so on all run serially on the same main thread.
When the main thread directly calls a long-running operation—such as waiting for an external program to process a video that takes tens of minutes—the event loop is “stuck” for a long time, and all UI events can only queue up and wait.
The reason “a UI that doesn’t freeze” has become a litmus test for the quality of AI-generated code is threefold:
- It is perceptible. No special tools are needed; dragging the window gives an immediate judgment.
- It tests understanding of the runtime model. Getting it right requires understanding the threading model and synchronization mechanisms, not just syntax.
- It is coupled with requirements like fault tolerance and cancellation. Once processing is moved to the background, cancellation and thread-safe progress updates must be solved together.
From an engineering standpoint, the correct approach is to decouple the UI layer from the processing layer, roughly divided into three layers:
- UI layer: responsible for display and interaction, communicating with worker threads only through a queue;
- Scheduling layer: responsible for task ordering, execution, cancellation, and status aggregation;
- Execution layer: encapsulates the specific audio/video processing logic, with no dependency on any UI components.
The three layers are connected through explicit data structures, which makes them easy to test individually and also easy to swap the processing engine later:
from dataclasses import dataclass, field@dataclassclass Task:src: str # Input file pathdst: str # Output file pathstatus: str = "pending" # pending / running / done / failedmessage: str = ""@dataclassclass Job:tasks: list = field(default_factory=list)cli_path: str = "" # External media processing program path (configured at deployment)bitrate: str = "192k"cancel_requested: bool = False
4.3 Step 3: Breaking down key implementation and pitfalls
Pitfall 1: Blocking the main thread
The processing loop must run in a worker thread. To do this, the read/write of the cancellation flag needs to be thread-safe: the flag is written by the UI thread and read by the worker thread, so it’s better to use a lock-protected boolean or an event object rather than a plain variable.
Pitfall 2: Operating on the UI directly across threads
Even after the background thread finishes processing, you cannot update UI controls directly in the background thread, or it will cause undefined behavior—at best garbled UI display, at worst a program crash.
The correct approach is for the worker thread to only post messages into a queue, and the UI thread to periodically pull them out and update. Different frameworks provide their own thread-safe invocation mechanisms, and you can also poll the queue on a timer:
import queueimport threadingclass App:def __init__(self, root, job):self.root = rootself.job = jobself.events = queue.Queue() # One-way message channel: worker -> UI threadself.worker = threading.Thread(target=self._process_loop, daemon=True)def start(self):self.worker.start()self.root.after(100, self._poll_events) # UI thread pulls messages every 100msdef _process_loop(self):for i, task in enumerate(self.job.tasks, start=1):if self.job.cancel_requested:self.events.put(("stopped", i - 1, len(self.job.tasks)))breaktask.status = "running"self.events.put(("progress", i, task.src))try:# Call the execution layer; all long-running work stays in the background threadself.job.process_one(task)task.status = "done"except Exception as exc: # One task failing doesn't stop the whole runtask.status = "failed"task.message = str(exc)self.events.put(("progress", i, task.src))def _poll_events(self):while True:try:kind, *payload = self.events.get_nowait()except queue.Empty:breakself._render(kind, payload) # Update the UI only in the UI threadself.root.after(100, self._poll_events)
Pitfall 3: No cancellation mechanism
When processing hundreds of files, if the user discovers they picked the wrong parameters, they can only force-quit the program. So before executing each task, the cancellation flag must be checked, and on stop the outputs of unfinished files must be cleaned up to avoid leaving half-written files.
Pitfall 4: External commands with no timeout
In batch processing, a corrupted file or a container in an unusual format can keep the external processing program waiting for input, causing the entire queue to stall. Setting a timeout ceiling for each task is the most basic and most easily overlooked safeguard.
Pitfall 5: Command injection and path-parsing errors
A lot of model-generated code tends to “concatenate strings”:
# Anti-pattern: user filenames may contain spaces or special charactersos.system(f"{cli} -i {src} -vn {dst}")
There are two problems here:
- If the filename contains spaces, the command gets split into multiple arguments, causing path errors;
- If the filename comes from an untrusted source, an attacker can use this to inject arbitrary commands.
The correct approach is as follows:
- Call the external program in the form of an “argument list,” letting the operating system parse the arguments directly, without going through the shell;
- Check whether the file exists before deleting any old file;
- Use a temporary filename for the output, rename it after success, and delete the temporary file on failure, thereby avoiding leaving a half-written result.
import osimport subprocessimport tempfiledef run_extract(cli_path, src, dst, bitrate="192k", timeout=1800):# Pass arguments as a list, bypassing the shell to avoid space-parsing issues and command injectionfd, tmp_dst = tempfile.mkstemp(suffix=".tmp.audio", dir=os.path.dirname(dst))os.close(fd)cmd = [cli_path, "-y", "-i", src, "-vn", "-b:a", bitrate, tmp_dst]try:result = subprocess.run(cmd,stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,text=True,errors="replace",timeout=timeout, # Auto-terminate the child process on timeout to prevent the queue from stallingcheck=False,)if result.returncode != 0:raise RuntimeError(f"External program returned {result.returncode}: {result.stderr[:200]}")os.replace(tmp_dst, dst) # Move to the target path only after successful processingexcept subprocess.TimeoutExpired:raise RuntimeError(f"Processing timed out (>{timeout}s)")finally:if os.path.exists(tmp_dst):os.remove(tmp_dst) # Clean up the temp file on failure or cancellation
4.4 Step 3 verification: does the code actually run
Once the code is generated, verify in the order below to rule out problems layer by layer:
- Syntax check: first run a compile check. For example, Python ships the command-line tool
python -m py_compile main.py, which can quickly confirm whether the file can be parsed correctly by the interpreter; - Dependency check: confirm the code only uses the standard library, with no hidden third-party dependencies;
- Sandbox execution: run it the first time in an isolated environment or container, and observe whether the external processing program is invoked correctly;
- Functional verification: run the main flow through a few representative small files, and check the output count and format;
- Exception verification: construct nonexistent or corrupted files, and filenames with spaces and Chinese characters, to confirm that a single failure won’t bring down the whole run;
- Interaction verification: test the UI responsiveness during processing with a larger file, then test whether the stop function takes effect correctly.
4.5 Step 4: problems exposed by hands-on testing
Even when the requirements are described clearly enough, the generated code doesn’t necessarily pass all checks on the first try. Common situations in hands-on testing include:
- Progress feedback is not detailed enough: total progress updates, but “which step the current file has reached” isn’t reflected;
- The UI is not self-adapting: on high-DPI screens, the proportion of fonts and controls is off;
- The code is too long: hundreds of lines generated in one shot, with high reading and verification cost.
These problems show precisely that “runs” is the starting point of model code, not the finish line. For the first situation, you can add a requirement via the prompt:
On top of the existing code, add the following capabilities while keeping all other logic unchanged:1. Parse the standard output of the external processing program, map the processing progress to a 0-100 percentage, and update the progress bar in real time2. Show elapsed time and estimated time remaining3. After completion, summarize and display the success count, failure count, and total elapsed time
Narrowing the scope of the change and clearly stating that the rest of the logic stays unchanged often yields more reliable results than regenerating the entire block of code.
5. Engineering Recommendations: Turning “One Experience” into “Steady Output”
A case can only prove “this time it worked.” To make AI-assisted programming continuously produce reliable results within a team, you also need a set of engineering mechanisms.
- Constraints first: build a library of prompt templates within the team, fixing down the tech stack, directory structure, coding standards, exception handling, and logging requirements, to reduce repeated communication in each conversation.
- Deliver in small steps: first have the model output a requirements breakdown and design plan, then generate code after confirmation. When requirements are complex, split by module to avoid generating thousands of lines in one shot.
- Sandbox verification: run the generated code in an isolated environment first. This exposes problems early and also avoids potential risks.
- Human oversight: automation can cover syntax, dependencies, and basic execution, but concurrency safety, resource release, and boundary conditions still need manual review.
- Knowledge accumulation: turn the problems found in each round of verification into a checklist—for example, “batch tasks must be cancellable” and “external commands must set a timeout”—and keep adding it to the prompt templates.
From a team-collaboration perspective, a fairly reliable flow is:
Requirements breakdown → Plan confirmation → Segmented generation → Static checks → Sandbox execution → Manual review → Integration testing
In this flow, the AI plays the role of “rapidly producing a draft,” while verification and decision-making are always done by people.
6. Summary and Outlook
First, the leap in coding ability does not depend solely on a bigger base model. As seen in the new version in the case, without changing the base model, post-training alone can significantly improve coding ability. This suggests that in the next stage of model R&D, the value of post-training may rival that of pre-training.
Second, the standard by which developers evaluate models should shift from “can it write” to “can it reliably run.” Runnable, maintainable, and safely deliverable are the true measure of engineering value.
Third, whether capability is unlocked ultimately depends on engineering method. No matter how strong the model, without a verification step it is only a “looks fast” draft generator. What developers really need to accumulate is the method of turning generation capability into steady output.
Looking ahead, programming intelligence is moving from “single-turn Q&A” toward a closed loop of “autonomous execution, observation, and correction”: the model continuously runs code in a controlled environment, reads errors, and iterates on fixes until the task is complete. In this direction, general cloud capabilities offered by Baidu AI Cloud—such as model fine-tuning and inference services, elastic containers and function compute, and log services with monitoring and alerting—can respectively take on the roles of “making the model better understand the business,” “letting code run in an isolated environment,” and “making every execution observable,” providing support for enterprises to build AI programming workflows that scale and are governed.
Back to the opening question: can AI-generated code be run directly? The answer depends on two things: whether the model has been well post-trained, and whether you have a reliable verification method. When both are in place, “building a usable piece of software in ten minutes” is no longer an occasional surprise, but a repeatable everyday routine.
