From Pretraining to Agents: How Large Model Capabilities Evolve and What Engineering Practice Requires
Large language models are shifting from generating fluent text to completing tasks. This article outlines a practical capability stack from pretraining through alignment and self-correction, explains how to evaluate code and UI agents so results stay trustworthy, and describes the engineering controls teams should keep in place before agents operate in real workflows.
1. Background: From Answering to Doing
Over the past two years, the focus of large language models has moved from generating text to completing tasks. A model now needs to understand a request, break it into steps, call tools, operate an interface, and correct itself after errors. Evaluation has changed with that shift: fluency is no longer enough. Task completion rate, result reliability, and safety boundaries are the core measures of value.
This article addresses three questions. First, how can general intelligence capabilities be understood in layers? Second, how should code agents and UI agents be evaluated so the conclusions are trustworthy? Third, which engineering boundaries must hold once agents enter real business workflows?
2. A Capability Stack for Engineering Discussion
The industry often describes model capability as a progression of layers. The following split is a convenient reference for engineering discussion:
- Pretraining. The model learns statistical patterns of language and knowledge from large-scale text, code, and multimodal data. This layer sets the breadth of knowledge coverage and basic expressive ability.
- Alignment and reasoning. Instruction tuning, preference optimization, and related methods make outputs follow human intent and support multi-step reasoning and planning.
- Self-directed learning. The model checks and revises its own output from execution feedback—for example, regenerating after a failed test, or summarizing lessons from errors.
- Self-awareness of limits. The model can judge its own capability boundary with reasonable stability: when to refuse, and when to ask for confirmation.
- Consciousness. This layer concerns philosophical questions such as subjective experience. It has no verifiable engineering metrics today and should not be a product-development target.
The value of the stack is that it turns capability into checkable subgoals. The first three layers already have many quantifiable evaluation methods. The last two remain research topics. Engineering teams should concentrate resources on making the first three layers reliable.
3. Pretraining and Foundation Models: Data Quality Sets the Ceiling
The main engineering focus in pretraining is data. Deduplication, quality filtering, the mix of data sources, and time-based splits all affect downstream performance. For code capability, license compliance and code quality also matter. Low-quality or unclear-provenance code should not be the main training material.
Model selection should not rest on a single score. At a minimum, consider three kinds of evidence: the coverage of the training data, performance on a held-out test set, and measured results on your own business samples. Together, they indicate whether a model can handle a specific scenario.
4. Alignment and Reasoning: From a Correct Answer to a Correct Plan
Alignment aims to make the model understand intent and respect constraints. In agent settings, that also includes choosing the right tool, calling it with the correct parameter format, asking follow-up questions when information is missing, and requesting confirmation before high-risk actions.
Reasoning usually shows up in multi-step tasks. A reliable pattern is to have the model output a plan first, then execute it step by step, checking the result of each step. Separating planning from execution makes the process easier to audit and makes it easier to see which step failed.
5. Code Agents: Evaluation Design Determines Whether the Score Is Believable
Common evaluation tasks for code agents fall into three groups: fixing defects in real projects, generating and completing code across languages, and building and debugging in a terminal environment. Whether the result is trustworthy depends mainly on three points.
- Overlap between the test set and training data. If a problem already appeared in the training corpus, the score will be inflated. Prefer recently released or internally built held-out sets, and check for overlap.
- Judging by executable outcomes. Pass rate on unit tests is more stable and reproducible than subjective human scoring.
- Reporting results across multiple samples. A single-run success rate varies widely. Report pass@k, or the mean and standard deviation across multiple runs.
The following simplified evaluation sketch measures how often a patch passes on a held-out task set:
import subprocessfrom dataclasses import dataclass@dataclassclass Task:task_id: strrepo_dir: str # repository copy checked out at the baselinetest_cmd: list[str] # test command for this taskdef run_task(task: Task, patch: str) -> bool:"""Apply the patch and run tests. Return whether they passed."""apply = subprocess.run(["git", "-C", task.repo_dir, "apply", "-"],input=patch.encode(), capture_output=True, timeout=60,)if apply.returncode != 0:return Falseresult = subprocess.run(task.test_cmd, cwd=task.repo_dir,capture_output=True, timeout=600,)return result.returncode == 0def pass_rate(tasks, generate_patch, k=1) -> float:passed = 0for task in tasks:for _ in range(k):patch = generate_patch(task)if run_task(task, patch):passed += 1breakreturn passed / len(tasks)
This example assumes the repository is restored to the baseline before every attempt. Otherwise an earlier patch contaminates later results. A real evaluation should also set timeouts on test commands, block network access, and run inside a container so generated code cannot affect the host.
6. UI Agents: A Closed Loop of Perception, Decision, and Action
A typical UI-agent loop captures a screenshot or the interface structure, interprets the current state, chooses the next action, executes it, and observes the result again. Compared with code agents, the hard parts are changing interface state, unstable control localization, and page content that may be misleading—or even disguised as instructions.
The loop in pseudocode:
def agent_loop(goal, max_steps=30):history = []for _ in range(max_steps):observation = capture_screen() # current interfaceaction = plan_next_action(goal, observation, history)if action.type == "done":return Trueif action.is_sensitive: # payment, deletion, sending, and similar actionsif not request_user_confirmation(action):return Falseresult = execute_action(action)history.append((observation.summary, action, result))return False # step limit exceeded; treat as incomplete
Three design choices matter here. First, a step limit prevents an endless loop on the same screen. Second, sensitive actions require human confirmation, so irreversible operations stay with a person. Third, text on the screen is untrusted data: it may inform the agent, but it must not be executed as an instruction.
7. Four Engineering Guardrails
- Least privilege. The agent receives only the permissions the task needs, with read and write separated. Sensitive credentials are injected through environment variables or a secrets manager. They are not written into code and do not appear in logs.
- Isolated execution. Code and UI actions run in a sandbox that limits filesystem, network, and process access. On failure, the sandbox can be discarded as a whole so no residual state remains.
- Observability. Record the input, decision, action, and result of every step so the run can be replayed and audited. Logs should redact personal information.
- Cost and latency control. Multi-step tasks multiply model calls. Set a budget cap per task, use a smaller model for simple subtasks, and cache reusable intermediate results.
8. A Selection and Evaluation Checklist
- Define the task type: question answering, code generation, UI operation, or multimodal understanding. Evaluation methods differ substantially across these types.
- Run a held-out test on your own samples. Record success rate, average number of steps, and the distribution of failure causes. Failure causes guide improvement better than an overall score.
- Check safety behavior: whether the agent refuses requests that exceed its authority, and whether it recognizes malicious instructions injected into a page or document.
- Assess integration cost: API stability, rate-limit policy, and whether error-code semantics are clear.
- Set launch thresholds. For example, human confirmation coverage for high-risk actions should be 100%, and the success rate for low-risk tasks should meet the value agreed for the business.
Conclusion
Agent capability improves in stages. Teams should first make pretraining and alignment quality solid, make evaluation results trustworthy, and establish engineering safety boundaries—then gradually widen the scope of autonomous action. Each expansion should come with matching metrics and a rollback plan, so capability growth stays within a controllable range.
