From Vibe Coding to Agent Engineering: The Evolution and Practice of Coding LLMs
Coding LLMs are moving from ad hoc prompt-and-patch workflows to agent engineering, where a model receives a goal, plans the work, uses tools, and validates results against acceptance criteria. This article examines the capabilities an effective coding agent needs, the mechanisms that support long-running tasks such as long context and sparse attention, multi-token prediction, and asynchronous reinforcement learning, and why model capability alone does not guarantee process reliability. It also provides an end-to-end evaluation framework, a reference agent loop with isolated execution and deterministic gates, a security baseline, and a phased adoption path for engineering teams.
1. Background: Why Coding Workflows Need to Change
Over the past few years, large language models have entered software development workflows in large numbers. The most common pattern is to describe a requirement to a model, receive a block of code, and then modify it step by step based on intuition. This ad hoc, prompt-driven style can be highly efficient for prototypes, but its problems quickly surface in projects that require long-term maintenance, multi-person collaboration, and high reliability. Generated results are difficult to verify, the scope of changes is hard to control, and errors are often discovered only during integration or deployment.
The industry has begun to use the term agent engineering to describe a different approach. Instead of only producing code snippets, the model receives a goal, decomposes the task, reads the code repository, calls tools such as compilers, tests, and retrieval systems, and revises its results based on feedback until acceptance criteria are met. The engineer’s role changes accordingly: from writing prompt after prompt to defining goals, specifying constraints, and accepting or rejecting results.
The core of this shift is to move from “the model can write code” to “the system can reliably complete development tasks.” The gap between the two is the engineering problem this article discusses.
2. Required Capabilities for Agentic Coding
An agent capable of completing real development tasks usually needs four capabilities working together.
1. Context understanding. Real projects often contain hundreds of thousands of lines of code, large numbers of configuration files, and historical documentation. The agent must locate files, interfaces, and call chains relevant to the task, and understand dependencies among modules. If it sees only a single file, a change can easily break other modules.
2. Task planning. For a requirement such as “add refund auditing to the order module,” the agent needs to decompose the work into data model changes, interface refactoring, test additions, documentation updates, and other steps, and define completion criteria for each step. Planning is the foundation for all subsequent execution. If the plan is unclear, execution will drift from the goal.
3. Tool use. The agent needs controlled interfaces to read files, run tests, execute static checks, and inspect build logs. Tool-call results are the only way for the model to obtain feedback from the environment, so tool output formats should be structured and parseable.
4. Self-verification. The agent must be able to determine whether its output meets the required standard. This verification should rely primarily on deterministic checks, such as unit tests, type checking, and code-standard checks, rather than on the model’s subjective evaluation of its own code.
3. Key Technical Mechanisms for Long Tasks
Coding tasks often require dozens of rounds of interaction, and each round accumulates substantial context. The following technical areas directly affect the cost, speed, and stability of long tasks.
3.1 Long Context and Sparse Attention
The computational cost of standard attention grows approximately quadratically with sequence length. The longer the context, the higher the inference cost and latency. The idea behind sparse attention is to let each position attend to only a subset of key positions, thereby reducing computation.
The trade-off is that if the selection rule chooses the wrong positions, critical information may be ignored. In engineering practice, three approaches can reduce this risk:
- Generate structured summaries of the code repository, including directory structure, module responsibilities, and key interface lists.
- Retrieve relevant files by task and place only highly relevant content in the context.
- Truncate extremely long logs and extract key lines while preserving the complete error stack.
The value of long-context capability lies in “fitting everything in.” The engineering challenge is ensuring that the right information is included.
3.2 Multi-Token Prediction
Conventional language models predict only the next token at each step. Multi-token prediction allows the model to predict multiple future positions at the same step. During training, this can provide richer learning signals; during inference, it can potentially improve generation speed when combined with methods such as speculative decoding.
For coding scenarios, generation speed directly affects the interactive experience. However, acceleration must not come at the cost of code correctness. Any inference acceleration approach should therefore be evaluated alongside test pass rates, not by throughput alone.
3.3 Asynchronous Reinforcement Learning Training
Training for agents typically uses reinforcement learning. A complete task trajectory may contain dozens of tool calls, and different tasks can vary widely in duration. With a synchronous approach, all samples must wait for the slowest one to finish, leaving substantial compute resources idle.
An asynchronous approach decouples sampling from parameter updates, so completed trajectories can be used for training immediately. Because the policy used during sampling differs from the current policy, training must introduce off-policy correction; otherwise, gradient estimates will be biased. The core of this design is balancing throughput and training stability.
3.4 Model Capability Is Not Process Assurance
These techniques improve a model’s ability to handle long tasks, but they cannot replace engineering processes. A model may complete tasks correctly in most cases while making mistakes on a small number of critical paths. Teams should therefore treat model output as candidate results to be verified, not as finished artifacts that can be merged directly.
4. Evaluation: From Single-Generation Success to End-to-End Success Rate
Evaluating agentic coding capability cannot be limited to whether a single generated block of code runs. More meaningful is the quality of end-to-end task completion. Teams should track the following metrics:
- Task pass rate: The proportion of tasks that pass all checks without human intervention.
- First-pass rate: The proportion of tasks that pass checks on the first submission, reflecting planning and generation quality.
- Average interaction rounds: The average number of tool calls required to complete a task, reflecting efficiency.
- Cost and latency per task: Including inference fees, compute resources, and waiting time.
- Regression defect rate: The proportion of changes that cause production issues after merging. This is the key final quality metric.
- Human modification volume: The number of code lines or review comments that reviewers need to change.
Public benchmarks can provide a reference point, but their task distribution differs from a team’s real business context. A more reliable approach is to sample from the team’s real tickets over the past year, stratify them by difficulty, use existing test cases as the basis for judgment, and build an internal evaluation set. The evaluation set should be updated regularly to avoid excessive overlap with training data or daily prompts, which can inflate metrics.
5. Engineering Practice: Building a Verifiable Agentic Coding Loop
The following pseudocode describes a basic coding-agent loop. Its core principles are that every change is made in an isolated environment, deterministic checks act as gates, failed attempts have a retry limit, and the final result is handed to a human for review.
function executeCodingTask(task):plan = generatePlan(task.requirements, repositorySummary)for each step in plan:change = model.generateChanges(step, relevantFiles)applyInIsolatedBranch(change)for retry in 1 to maxRetries:result = runChecks(unitTests, staticAnalysis, typeChecking)if result.allPassed:breakchange = model.fix(result.errorLogs, change)applyInIsolatedBranch(change)if not result.allPassed:return markForHumanReview(step, result)return submitForReview(diff, changeSummary)
This pseudocode contains four design points.
- Isolated execution: All modifications are made in an independent branch or temporary workspace, so the main codebase is unaffected.
- Deterministic gates: Whether to proceed to the next step is determined by test and check results, not by the model’s self-assessment.
- Retry limit: When a model repeatedly tries to fix the same problem, continuing the loop usually only increases cost. After the limit is exceeded, the task should be handed to a human.
- Human review: The final output consists of a diff plus a change summary. Reviewers should be able to understand the intent and impact scope of the changes quickly.
During implementation, teams should also record the input, output, and tool results of every model call to make it easier to analyze failures after the fact.
6. Risk Control and Security Boundaries
Agents can read and write code and execute commands, so their security risks are higher than those of ordinary question-answering scenarios. The following measures should be treated as baseline requirements.
- Least privilege: The agent receives only the repository permissions needed to complete the task and does not access production environment credentials.
- Sandboxed execution: Command execution takes place in a restricted environment, with network access limited and a command allowlist used.
- Secret management: Secrets are injected through environment variables or a dedicated secret service and are not written into code, prompts, or logs.
- Prompt injection defense: Code comments, dependency documentation, and external web content may contain misleading instructions. The agent should treat this content as data and must not treat it as operational instructions.
- Audit trail: Save prompts, tool calls, diffs, and review records to meet traceability requirements.
- License and copyright checks: Scan generated code and introduced dependencies for license issues to avoid compliance risks.
7. Team Adoption Path
Adoption of agentic coding should proceed in phases, gradually expanding task scope and autonomy.
Phase 1: Low-risk tasks. These include adding unit tests, generating API documentation, and standardizing log formats. Results are easy to verify and errors have limited impact, making these tasks suitable for building team trust in the process.
Phase 2: Bug fixes with clear test coverage. Bugs with clear targets and existing regression tests are good scenarios for testing the planning and verification loop. Teams should focus on the first-pass rate and regression defect rate.
Phase 3: Feature development and refactoring. These tasks involve cross-module design and require a stricter review process. It is advisable for senior engineers to write the design notes and acceptance criteria first, let the agent implement the work step by step, and have humans perform focused reviews of critical interfaces and data migrations.
Each phase should use the metrics described above as the basis for moving to the next phase. If the regression defect rate rises, teams should first tighten gates and review rather than directly expanding autonomy.
The adoption process also requires clear responsibilities. Developers are responsible for goal definition and final decisions; evaluation and security teams are responsible for defining check standards and permission policies; tool and platform teams are responsible for maintaining isolated environments and audit systems. Only with clear responsibilities can the process remain stable over the long term.
