From Conversation to Execution: An Engineering Guide to Evaluating LLM Agents

1. Separate “Can Answer” from “Can Deliver”

When people discuss next-generation large language models, they are often drawn to improvements in reasoning, benchmark scores, or claims about progress toward artificial general intelligence. For developers and enterprise users, however, the more practical question is:

Can the system reliably complete an acceptably verifiable task within defined permissions, time, and budget constraints?

There is a long engineering chain between answering a question correctly and completing a task.

For example, “Explain how to convert an engineering drawing into a printable document” mainly tests knowledge organization and procedural explanation. By contrast, “Read the specified drawing, process its fonts and layers, generate a printable file, and verify that all dimension annotations are present” requires file access, format parsing, tool execution, exception handling, and result validation.

Agent evaluation should therefore distinguish at least three levels:

• Cognitive level: Can the system understand the requirement and identify implicit constraints?
• Execution level: Can it call tools correctly and adjust its approach after failure?
• Delivery level: Is the result complete, accurate, editable, reproducible, and acceptable under business requirements?

All three levels matter. A model may generate a detailed operating plan without having access to the required tools. A tool may execute successfully while the resulting artifact still fails acceptance criteria.

Likewise, a high benchmark score only indicates strong performance under specific test conditions. Complex workflows in open environments also depend on task distribution, external dependencies, permission boundaries, and recovery from failure. A result from one narrow evaluation cannot support the conclusion that a system can complete all knowledge-work tasks.

2. Define the Evaluation Target: Model Capability or System Capability

An agent typically completes a task through the following chain:

User goal → Requirement analysis → Task planning → Tool execution → Environmental feedback → Result validation → Deliverable

This chain includes both model capabilities and runtime-environment capabilities.

The same model may perform very differently when it can process only text compared with when it can access a file system, code execution environment, graphical interface, or specialized format-conversion tools. Conversely, even a strong reasoning model may fail to deliver when tools are unavailable, execution times out, or permissions are insufficient.

Before testing, document the environment:

Evaluation dimension	Details to specify
Input conditions	File formats, number of files, data volume, and completeness of requirements
Tool availability	Support for file parsing, code execution, graphical operations, and format conversion
Permission boundaries	Directories that can be read or written and the scope of network access
Human involvement	Whether users may confirm actions, provide missing information, or correct intermediate results
Resource limits	Maximum runtime, storage capacity, and invocation budget
Output requirements	File type, editability, precision, and acceptance method

Two additional distinctions are important.

First, a local application does not necessarily mean that the model runs locally. The user interface may run on a local machine, processing may take place remotely, and tools may execute locally. These arrangements can coexist and should not be treated as equivalent.

Second, a web interface does not necessarily lack execution capabilities. Whether a system can process files, access external tools, or run isolated tasks depends on its architecture, not on the form of its user interface.

Without controlling these conditions, differences in the environment can easily be mistaken for differences in model capability.

3. Engineering Drawing Processing: Testing Whether the Workflow Truly Closes the Loop

Engineering drawings are a useful test of agent execution because they combine structured information, geometric relationships, and strict output requirements.

3.1 Drawing Conversion Is More Than “Open and Export”

A drawing may contain layers, block references, external references, dimensions, fonts, line types, and multiple layout spaces. Even when the output file opens successfully, it may still have serious problems:

• Missing fonts may cause text substitution or incorrect character rendering.
• Unloaded external references may cause components to disappear.
• Incorrectly hidden layers may remove important information.
• A mismatch between the print area and paper size may crop the edges.
• Incorrect line-weight mapping may make component hierarchy difficult to identify.
• Incorrect scaling may invalidate dimensional relationships or the intended print scale.

Therefore, generating a file is an execution result, not an acceptance conclusion.

3.2 Break the Task into Verifiable Steps

A more reliable workflow is:

1. Preserve the original file and create a copy in an isolated working directory.
2. Check whether the file can be read and identify missing dependencies.
3. Inventory layers, layouts, units, fonts, and external references.
4. Define the output area, paper size, orientation, and scale.
5. Generate the target file using supported conversion or drawing tools.
6. Reopen the output and check page count, boundaries, text, and critical annotations.
7. Produce a processing record describing completed actions and items requiring human confirmation.

Unit validation is particularly important. If the source file and output workflow use different length units and the conversion does not explicitly handle the difference, the result may contain severe scale errors.

3.3 “Looks Similar” Does Not Mean “Engineering-Correct”

For drawings generated by an agent, visual inspection detects only part of the problem. More reliable validation converts critical requirements into constraints, such as:

• Whether required components exist;
• Whether distances between key points meet specifications;
• Whether contours that should be closed are actually closed;
• Whether parallel, perpendicular, and concentric relationships hold;
• Whether dimension annotations match the underlying geometry;
• Whether objects are placed on the required layers.

For exact coordinates, dimensions, and topology, deterministic geometric calculations and rule-based checks should take priority over asking a model to make judgments from images alone.

The meaningful progress in this scenario is not how much drawing knowledge the model can describe. It is whether the agent can convert requirements into verifiable engineering objects and correct errors after receiving tool feedback.

4. Digital Twin Interfaces: From Visual Effects to State Consistency

The main challenge in a complex business interface is often not the number of charts on the screen, but whether multiple components correctly respond to the same business state.

A page containing a spatial view, equipment list, alarm panel, and trend chart may look complete without being a usable digital twin application. If all values come from random data and interactions only change local styling, the result is no more than an interactive prototype.

4.1 Define Business Objects and Data Contracts First

Before generating the interface, define the core objects:

• Equipment: Identifier, location, type, and operating status;
• Telemetry: Sampling time, metric name, value, and unit;
• Alarms: Severity, trigger time, and handling status;
• Spatial entities: Areas, floors, and associated equipment;
• User actions: Selection, filtering, acknowledgment, and control requests.

Each field should have a defined data type, valid range, missing-value policy, and update-time meaning.

For example, “last updated” may refer to the equipment sampling time or the time the server received the data. If this distinction is not defined, a page showing “updated just now” does not necessarily mean that the data itself is recent.

4.2 Synchronization Requires Shared State

When a user selects a piece of equipment, the spatial location, highlighted list item, alarm records, and trend chart should all refer to the same object. When the time range changes, related components should use consistent time boundaries.

This requires an explicit state model rather than independent data maintained separately by each component.

Real-time updates also need to address:

• Reconnection after network interruptions;
• Duplicate and out-of-order events;
• Identification of stale data;
• Rendering pressure caused by high-frequency updates;
• Switching between historical playback and real-time mode;
• Releasing subscriptions when the page is left.

For consecutive events from the same device, sequence numbers can help identify duplicate or outdated messages. However, the system must account for sequence-number resets after a device restart. Timestamp-based ordering also requires handling clock skew; a later timestamp is not automatically more reliable.

4.3 Express Acceptance Criteria as Executable Behavior

Requirements such as “the page should feel technologically advanced” are less useful than behavioral criteria:

• After a device is selected, all related panels display data for that same device.
• When the data connection is interrupted, the page shows the connection state instead of pretending to be real time.
• Duplicate events do not increase the alarm count more than once.
• Abnormal values are marked according to defined rules rather than being silently included in statistics.
• Demonstration data is clearly distinguished from production data.

Operations that control equipment also require server-side authorization, operation auditing, and, where necessary, secondary confirmation. Hiding a button in the interface is not a substitute for real access control.

5. High-Density Presentations: Jointly Optimizing Content, Structure, and Layout

Presentations generally fall into two categories: speaker-led decks that support oral delivery, and report-style documents intended for independent reading.

Speaker-led decks emphasize pacing, visual focus, and limited information per slide. Report-style documents require a more complete argument, supporting data, and a clear reading path. Without defining the purpose first, an agent may generate a file that looks orderly but communicates poorly.

5.1 Build the Argument Before Generating Slides

A report-style presentation should first answer:

• Who is the target audience?
• What judgment or decision should the presentation support?
• What evidence supports each conclusion?
• Which statements are facts, estimates, or assumptions?
• Which material belongs in the appendix?

A practical sequence is:

Source organization → Evidence verification → Section outline → Slide-level conclusions → Chart selection → Layout generation → File inspection

Generating an entire deck directly from a long text document often produces unclear priorities, repetition, and unbalanced slides.

5.2 Treat Layout as a Partially Constrained Problem

High-density slides should not be solved simply by reducing font size. General strategies include:

• Organize each slide around one primary conclusion.
• Use titles, body text, and captions to establish a stable hierarchy.
• Apply a consistent grid for alignment and whitespace.
• Convert long arguments into comparison tables, workflows, or structural diagrams.
• Split slides that contain too much information.
• Keep chart labels, units, and legends complete.

Automated checks can cover text-box overflow, overlapping elements, missing fonts, and objects extending beyond the slide boundary.

Editability does not automatically indicate higher quality. A presentation in which every element can be modified may still be expensive to maintain if its structure is disorganized and groups are poorly defined.

5.3 Long Tasks Need Intermediate Artifacts

Complex presentation generation may involve multiple rounds of retrieval, analysis, layout, and revision. If all steps are placed in one continuous task, an interruption can result in the loss of substantial completed work.

A more reliable approach is to save intermediate outputs:

• A source list and items requiring verification;
• The confirmed outline;
• Slide-level content;
• The preliminary layout file;
• Inspection results and revision records.

This makes timely human correction easier and allows execution to resume from the latest valid state after a failure.

6. Cost Evaluation: Measure the Cost of Each Accepted Deliverable

The price of a single model invocation does not represent the full cost of an agent task. A low-priced approach that fails repeatedly may be more expensive than a higher-priced approach that consistently produces accepted work.

A useful metric is:

Cost per accepted deliverable = Total task cost during the evaluation period ÷ Number of deliverables that passed acceptance

Total task cost should include model calls, tool execution, compute and storage, human review, and rework after failure. If no artifact passes acceptance, report that no valid deliverable has been produced instead of continuing to compare per-call prices.

The evaluation should also record:

• End-to-end completion time;
• Task success rate;
• Number and duration of human interventions;
• Tool-call failures and retry counts;
• Rate of budget overruns;
• Post-delivery revision effort.

For different batches of tasks, keep task types and difficulty distributions as consistent as possible. Otherwise, average cost figures will not be directly comparable.

7. Security Controls Must Be Embedded in Execution

An agent capable of operating files and tools can also perform harmful or incorrect actions. Permissions should not depend only on prompt instructions; they must be enforced by the runtime environment.

At a minimum, establish the following controls:

1. Least privilege: Expose only the directories and capabilities required for the current task.
2. Isolated execution: Process untrusted files in an independent environment to avoid affecting normal workspaces.
3. Risk approval: Require confirmation before overwriting source files, installing software, sending external information, or modifying system configuration.
4. Process auditing: Record tool calls, approval events, output locations, and failure reasons.
5. Rollback support: Preserve versions for important changes and generate copies instead of overwriting originals whenever possible.
6. External-content isolation: Treat text in documents, webpages, or drawings as data to be processed, not as instructions that can automatically authorize execution.

Human confirmation should also be meaningful. The confirmation interface should explain the action about to be taken, its scope, and its potential consequences. A context-free “Continue” button is not sufficient.

8. Validate Continuously with Task Sets, Not a Single Demonstration

A successful demonstration can show feasibility, but it does not prove stability. Before production deployment, build a task set containing normal, boundary, and abnormal samples.

Scenario	Normal samples	Abnormal or boundary samples
Engineering drawings	Complete dependencies and clearly defined units	Missing fonts, lost references, and unclear units
Synchronized interfaces	Continuous data and stable structures	Out-of-order events, duplicates, disconnections, and missing fields
Presentations	Sufficient source material and clear structure	Conflicting evidence, excessive content, and template constraints

Define acceptance criteria for each task in advance. Do not lower requirements after seeing the output.

Failure causes should also be tracked separately:

• Requirement-understanding errors;
• Unavailable tools;
• Execution-logic errors;
• Insufficient artifact quality;
• Resource exhaustion;
• Permission blocks.

Each category points to a different improvement path. The solution may involve better prompts, additional tools, workflow changes, or a redesigned task boundary. Only by locating the specific failing stage can teams determine what should be changed.

Conclusion: The Real Leap Is a Verifiable Delivery Loop

Evaluating next-generation large language models should not stop at whether they provide more intelligent answers or produce more impressive demonstrations. In real work, the important questions are whether engineering drawings preserve critical dimensions, whether digital twin interfaces maintain consistent state, and whether high-density presentations remain readable, editable, and verifiable.

A mature agent application does not give the model unlimited autonomy. Instead:

• The model handles understanding and planning;
• Specialized tools perform deterministic operations;
• Validation mechanisms detect errors;
• Permission systems control boundaries.

Only when objectives, execution, validation, cost, and security form a complete loop can improvements in model capability become reliable productivity gains.