From Tool Calling to Code Execution: AI Agent Architecture Evolution and Efficiency Optimization

In the rapid development of artificial intelligence, major models are increasingly competing in coding capabilities and agent architecture design. Recently, leading AI research institutions have proposed a seemingly simple yet highly effective technical paradigm shift: upgrading the traditional “tool calling” model to a “code execution” model. This transition effectively resolves the “Token black hole” challenge in current LLM applications and points the way toward building more efficient agents.

1. The “Token Black Hole” Dilemma in Traditional Agent Tool Calling

Currently, the mainstream approach to building AI agents relies on open context protocols. These protocols aim to enable LLMs to connect to various external systems such as databases, APIs, and SaaS services. However, in traditional architectures, agents typically inject all available tool definitions directly into the model’s context window.

Every time the model executes a task, it must first “read” this long list of tool definitions, decide which tool to call, and then feed the tool’s output back into the context to determine the next step. This repetitive cycle creates a massive resource consumption trap.

Take a common business scenario as an example: Suppose we configure an agent connected to a document collaboration platform and a CRM system, each exposing dozens of tool interfaces. The schema definitions for these tools alone can consume hundreds of thousands of tokens.

If a user issues an instruction: “Download a 2-hour meeting transcript from the document platform and add it to the corresponding sales lead in the CRM system.” The traditional tool calling execution flow is as follows:

1. The model calls the document platform’s retrieval interface, returning a full text transcript of about 50,000 tokens.
2. These 50,000 tokens are loaded into the model’s context.
3. The model reads these 50,000 tokens again and uses them as parameters to write to the CRM system’s update interface, generating and transmitting another 50,000 tokens.

A logically simple business task not only consumes over 100,000 tokens but also incurs wait times for two long-text generations, leading to high latency and high costs.

2. The Breakthrough: Restructuring the Tool Calling Chain via Code Execution

To address these pain points, the industry has proposed a new approach: transforming external system tools into code APIs. In this architecture, the model no longer calls tools individually at the natural language level; instead, it writes code to call tools in a batched, logical manner.

Using the same meeting transcript transfer task, upon receiving the instruction, the model directly generates a piece of code with complete control flow:

// Import code libraries corresponding to external services
import * as docPlatform from './servers/doc-collaboration';
import * as crmSystem from './servers/crm-system';
// Write code to complete the task
const transcript = (await docPlatform.getDocument({ documentId: 'abc123' })).content;
await crmSystem.updateRecord({ 
    objectType: 'SalesMeeting', 
    recordId: '00Q5f000001abcXYZ', 
    data: { 
        Notes: transcript 
    } 
});

This code is sent to an independent execution environment (such as a sandbox container) to run. During this process, the retrieval and writing of the meeting transcript are fully enclosed within the code runtime. The model only needs to “write the correct code” without having to “transport” the data itself.

3. Architecture Comparison: Tool Calling vs. Code Execution

This shift essentially redefines the boundaries of the LLM’s work. We can compare these two architectures across three core dimensions:

1. Tool Exposure Method

• Traditional Tool Calling: Full definitions injected into the context. Whether the task uses them or not, all tool descriptions and parameter constraints must be presented in the prompt, leading to extremely crowded context and confusing the model’s selection process.
• Code Execution: On-demand reading from the file system or environmental context. The model only needs to know which modules are available; specific interface details can be dynamically imported and queried during coding, significantly reducing initial context pressure.

2. Intermediate Data Flow

• Traditional Tool Calling: All flows through the model’s context. Any intermediate results (like long text or large JSON) must be returned to the LLM for parsing, easily bursting token limits and causing sluggish responses.
• Code Execution: Passed only within the execution environment. Large data flows between variables without passing through the LLM, completely eliminating the data transportation cost of intermediate states.

3. Control Flow Expression

• Traditional Tool Calling: Relies on natural language prompts. For example, telling the model “please loop execution until complete.” This approach is highly unreliable, as models often get lost or stuck in infinite loops during multi-step reasoning.
• Code Execution: Employs complete programming structures. Leverages deterministic logic such as variables, loops, conditional branches, and exception handling, ensuring 100% reliable and precise task execution.

4. Engineering Practices: Building Efficient Code-Executing Agents

To implement this architecture, developers need to restructure existing agent development frameworks. Here are several key engineering practice recommendations:

1. Build a Secure, Isolated Code Execution Sandbox
Model-generated code is inherently uncertain and must be executed in a highly isolated environment. It is recommended to leverage container platforms or serverless computing capabilities provided by cloud service providers like Baidu AI Cloud to allocate independent sandboxes with resource quotas for each code execution. This not only prevents malicious code escapes but also handles high-concurrency requests through elastic scaling.

2. Standardize Tool Library Packaging
Package the APIs of various business systems into standardized code library modules. These modules should feature robust TypeScript/Python type hints so the LLM can accurately understand inputs and outputs. Additionally, authentication, authorization, and retry logic should be implemented internally within the module, remaining transparent to the model.

3. Introduce Object Storage as a Data Relay Station
For extremely large-scale data processing tasks, even the code execution environment should avoid retaining large files in memory for extended periods. Intermediate results can be written directly to object storage, with only file reference handles retained in the code. For instance, when processing a 100MB CSV file, the code only operates on the file path, and the final results are pushed to a message queue or logging service for subsequent flow.

4. Exception Handling and Self-Healing
In the code execution environment, model-generated code may throw exceptions due to network timeouts, incorrect parameter types, etc. The framework should capture these execution logs and error messages and feed them back to the LLM. Given the LLM’s strong programming comprehension capabilities, it can read the error stack like a human programmer, modify the code, and resubmit it for execution, forming a closed-loop self-healing capability.

5. Conclusion and Outlook

Shifting from directly calling tools to writing code to call tools may seem like just an extra layer of abstraction, but it is a crucial step for LLMs to leap from “chat assistants” to “junior programmers.” This architecture cleverly combines the semantic understanding and planning capabilities of LLMs with the determinism, efficiency, and low cost of traditional code execution environments.

For enterprise AI application developers, embracing this code-execution agent architecture not only drastically cuts token overhead but also significantly improves the stability and execution efficiency of complex business processes. As underlying cloud infrastructure further integrates AI computing power with containerized scheduling, future agents will increasingly resemble true digital employees, unleashing massive productivity across diverse digital scenarios in various industries.