Designing and Implementing the Execution Core of Large Language Models in the Agent Era

    As LLM applications evolve from single-turn Q&A to agentic systems, models now act as the execution core of complex workflows: planning tasks, calling tools, managing state, and recovering from failures. This article explains the five core responsibilities of such an execution core, a typical four-layer architecture, a minimal execution loop in Python, key safety and controllability constraints, an evaluation metrics framework, and a phased adoption roadmap for engineering teams building reliable agent systems.

    1. From Q&A Models to the Execution Core

    Early LLM applications focused on single-turn interactions such as question answering and copywriting. The model produced text, and humans carried out the final actions. As agent architectures have matured, models now take on more roles: understanding goals, decomposing steps, invoking external tools, observing execution results, and revising plans based on feedback.

    In this mode, the model is no longer just a content generator but the scheduling core of a complex system—an “execution core.” It connects user intent with underlying capabilities, deciding when to call which tool, in what order to complete tasks, and how to fall back when anomalies occur. Understanding this role shift is the starting point for designing reliable agent systems.

    2. Five Core Responsibilities of the Execution Core

    2.1 Task Understanding and Planning

    User goals are usually expressed as vague natural language, such as “organize last week’s online incidents and generate a retrospective report.” The execution core must convert this into a structured list of subtasks, specifying the input, output, and completion criteria for each step.

    Planning quality directly determines execution efficiency. A sound plan should meet three conditions: clear dependencies between steps; each step independently verifiable; and when information is missing, the ability to proactively ask clarifying questions instead of making unfounded assumptions.

    2.2 Tool Invocation and Code Execution

    An agent’s capability boundary is defined by the tools it can call. Common tools include file read/write, database queries, code execution, API requests, and search retrieval. The execution core must select the appropriate tool for each task and generate invocation requests in the parameter format defined by the tool.

    Code generation and terminal execution are the most typical capabilities in development scenarios. After generating code, the model must run it in an isolated environment, collect standard output, error messages, and exit codes, and then decide whether to modify the code and retry.

    2.3 State Management for Long Workflows

    Tasks spanning dozens or even hundreds of steps exceed the capacity of a single context window. The execution core needs to maintain external state, such as task progress tables, indexes of intermediate artifacts, and records of key decisions. At each reasoning step, only the state relevant to the current step is injected into the context to control cost and latency.

    State should be persisted so that tasks can resume after interruptions. In engineering practice, this is typically achieved with object storage, relational databases, or key-value stores.

    2.4 Multi-Agent Collaboration

    Complex tasks can be split among agents with different roles—for example, a requirements analyst, a code implementer, and a test reviewer. The execution core handles coordination: assigning subtasks, aggregating results, and resolving conflicts.

    The key to collaboration is a unified message format and a clear handoff protocol. If every agent defines its own output structure, the error rate in the aggregation stage rises significantly. It is advisable to define a fixed structured schema for inter-role interactions—for example, a set of fields covering status, artifacts, rationale, and pending items.

    2.5 Enterprise Knowledge Integration

    In enterprise scenarios, models must answer questions based on internal documents, ticket records, and policy specifications. The common approach is retrieval-augmented generation (RAG): relevant passages are first recalled via vector search or keyword search, then passed to the model together with the question. The execution core decides when to retrieve, which knowledge bases to query, and how to verify that cited sources are reliable.

    3. A Typical Layered Architecture

    A maintainable execution core is usually divided into four layers, each with a single responsibility, making them easy to replace and test independently.

    1. +------------------------------------------------------+
    2. | Interaction layer: receive user instructions, |
    3. | display progress and results |
    4. +------------------------------------------------------+
    5. | Orchestration layer: planning, scheduling, |
    6. | state machine, and retry policies |
    7. +------------------------------------------------------+
    8. | Capability layer: model inference, tool registry, |
    9. | and permission control |
    10. +------------------------------------------------------+
    11. | Infrastructure layer: sandbox, storage, queues, |
    12. | monitoring and alerting |
    13. +------------------------------------------------------+

    The orchestration layer is the core. It determines the task flow and should be decoupled from any specific model provider, invoking inference capabilities through a unified interface. This way, when models are upgraded or replaced, the orchestration logic does not need to be rewritten.

    4. Engineering Practice: A Minimal Execution Loop

    The following example shows the basic loop of the execution core: plan, execute, observe, and correct. The code is illustrative only—function and class names are placeholders and should be adapted to your actual environment.

    1. from dataclasses import dataclass, field
    2. @dataclass
    3. class StepResult:
    4. ok: bool
    5. output: str
    6. @dataclass
    7. class ExecutionState:
    8. goal: str
    9. history: list = field(default_factory=list)
    10. max_steps: int = 20
    11. def run_agent(state: ExecutionState, planner, tools: dict) -> str:
    12. for step_no in range(state.max_steps):
    13. # 1. Plan: decide the next action based on the goal and history
    14. action = planner.next_action(state.goal, state.history)
    15. if action.type == "finish":
    16. return action.payload
    17. # 2. Validate: the tool must be registered and arguments must pass validation
    18. tool = tools.get(action.tool_name)
    19. if tool is None:
    20. state.history.append(("error", f"Unregistered tool: {action.tool_name}"))
    21. continue
    22. if not tool.validate(action.arguments):
    23. state.history.append(("error", "Argument validation failed"))
    24. continue
    25. # 3. Execute: run in an isolated environment and capture the result
    26. result: StepResult = tool.run(action.arguments)
    27. # 4. Observe: write the result to history for the next planning round
    28. state.history.append(("result" if result.ok else "error", result.output))
    29. raise RuntimeError("Maximum step count reached without completion; task terminated")

    Three design points in this loop deserve attention. First, tool invocations must be validated before execution, so that invalid parameters generated by the model never reach the execution stage. Second, failure results are also written to history, allowing the model to correct its plan based on errors. Third, a maximum step count is set to prevent the agent from falling into ineffective loops that consume resources.

    5. Safety and Controllability Constraints

    Because the execution core has the ability to invoke tools, safety design is more critical than in a pure Q&A system.

    Least privilege. Grant each tool only the minimum permissions required to complete its task. For example, the code execution environment should have no external network access by default, and the file tool should only read and write designated directories.

    Isolated execution. Generated code and commands should run in containers or sandboxes, with resource quotas and runtime limits to avoid affecting the host system.

    Human confirmation for irreversible operations. Actions such as deleting data, sending messages, modifying permissions, or deploying to production should pause execution and wait for human confirmation.

    Defending against prompt injection. Web content, file contents, and external API data returned by tools are all untrusted input. They should be treated as data, not executed as instructions. This rule should be stated explicitly in the system prompt, and suspicious content should be flagged at the orchestration layer.

    End-to-end auditing. Record the planning result, tool invocation parameters, return values, and final decision for every step, enabling post-incident review and compliance audits.

    6. Evaluation Metrics

    The effectiveness of an agent system cannot be judged by whether a single response “looks reasonable.” A quantifiable evaluation framework is needed.

    • Task success rate: the proportion of goals correctly completed on a fixed test set.
    • Average step count: the average number of invocations needed to complete a task, reflecting planning efficiency.
    • Tool error rate: the proportion of tool calls that fail or fail parameter validation, reflecting the stability of the capability layer.
    • Cost per task: inference fees plus tool resource overhead, used to assess economic viability.
    • End-to-end latency: the time from receiving an instruction to producing a result, which affects user experience.
    • Regression test pass rate: how historical test cases perform after each change to the model or orchestration logic.

    It is recommended to maintain a test set covering both typical and edge-case scenarios, run it automatically after every iteration, and incorporate metric changes into the release gate.

    7. Adoption Roadmap

    Phase one: Choose a scenario with clear boundaries, such as internal code review assistance or incident retrospective generation. Connect only two or three read-only tools, and first validate the reliability of the plan-and-observe loop.

    Phase two: Introduce write operations and multi-step tasks while strengthening the permission system, sandbox isolation, and human confirmation mechanisms, and establish the evaluation test set.

    Phase three: Only then consider multi-agent collaboration and enterprise knowledge base integration. At this stage, prioritize solving state persistence and message format unification to prevent complexity from spiraling out of control too early.

    Each phase should have explicit exit criteria—for example, the task success rate reaching an agreed threshold and zero security incidents. Expanding scope only after meeting these criteria is far better for risk control than rolling out everything at once.