From Tool Calling to Code Execution: Rethinking Context Efficiency in AI Agents

1. Where the Problem Starts: The Three Hidden Costs of Conversational Tool Calling

Most agent implementations today follow the same pattern, which can be summarized as “inject all definitions upfront + call tools turn by turn through conversation”:

1. Assemble the metadata of every available tool (name, purpose description, parameter structure) into the prompt at once.
2. The model reads through all the definitions and emits one tool-call intent.
3. The runtime executes the tool and writes the entire result back into the context.
4. The model reads the result and decides the next step.

This cycle repeats until the task is done. It has a low implementation barrier and is indeed a widely adopted general solution—but its cost structure has three structural problems.

Cost 1: Tool definitions are a fixed cost paid again on every turn. If you connect two classes of external systems at once—say, an online document collaboration platform and a CRM—with dozens of tools each, the tool definitions alone can occupy tens of thousands or even hundreds of thousands of tokens. The key point is that this cost is not paid once; it is paid again on every inference turn. The longer the task chain, the higher the multiplier on the waste. On top of that, the compressed context window dilutes the model’s effective attention, making it more likely to “forget” earlier constraints partway through a long task.

Cost 2: Intermediate data is forced to pass through the model. Take “read a set of meeting minutes and write them into a sales lead note” as an example. Reading the document returns a full transcript of several hours—assume about 50,000 tokens—and those 50,000 tokens enter the context. The model then has to “copy” the same content into the parameters of the write API, producing another output of the same scale. A transfer that is logically pure movement consumes over 100,000 tokens and forces two long-text generations. The problem: the model’s role here is just a courier. It performs no reasoning yet pays the highest unit price.

Cost 3: Control flow can only be expressed in natural language. Looping, conditional branching, batch retries, field-level filtering, exception fallbacks—each of these is a single line of code in a programming language, but in conversational tool calling they can only be described repeatedly in the prompt. The model has to re-“understand” the intent at every step, which makes stability and predictability hard to guarantee, and the logic cannot be reused or tested.

2. The Shift: Exposing the Tool Server as a Code API

The core shift can be compressed into one sentence: Don’t let the model call tools directly—let the model write code that calls tools.

Tool providers still implement their capabilities against an open integration protocol, but the way they are exposed at runtime changes: instead of injecting every schema into the prompt, capabilities are exported as a library that code can import (modules, type definitions, usage notes), together with a sandbox environment that can execute code. What the model sees is a set of files and a directory structure. It first retrieves, then selects, and finally assembles a script to run.

Illustrative code (pseudocode, structure only):

# An on-demand tool library instead of one-shot schema injection
from tools import docs, crm   # each module includes signatures, parameter types, and usage notes
items = docs.list_files(folder="meetings", modified_after="2026-08-01")
for item in items:
    transcript = docs.get_document(document_id=item.id).content
    summary = summarize(transcript)        # reason only over the necessary fragments
    crm.update_record(
        object_type="SalesLead",
        record_id=resolve_lead(item.owner),
        data={"Notes": summary[:2000]},    # trim before writing back
    )

The model’s output is no longer a stream of per-tool call messages, but an executable, testable, versionable program.

3. Breaking Down the Mechanism: Where Do the Efficiency Gains Actually Come From

Dimension	Conversational tool calling	Code-execution tool calling
How tools are exposed	All definitions permanently resident in context	Loaded on demand as modules
Intermediate data flow	Must detour through the model’s context	Passed directly within the execution environment
Control-flow expression	Natural-language descriptions, step-by-step probing	Full language constructs: variables, loops, branches, exceptions
Error handling	Re-explain and retry after failure	Retry, timeout, and rollback built in
Result granularity	Returned as one block	Can be sliced, aggregated, and filtered before entering context

1) Context moves from “always fully resident” to “loaded on demand.” The model only needs to see the directory structure and a concise index (a summary of a few dozen tokens per tool), while the real definitions are accessed by code at call time. As the tool set grows from dozens to hundreds, the context-occupancy growth curve shifts from roughly linear to near-logarithmic.

2) Intermediate data no longer enters the model’s view. The 50,000-token transcript from the example above is read into a variable inside the sandbox and trimmed there; only the fragments that genuinely require model judgment are sent back. The transfer cost drops from “two 50,000-token copies” to near zero, and the exposure of sensitive source text in the context narrows accordingly.

3) The number of reasoning rounds drops sharply. A single run completes several steps, so the model no longer has to round-trip between each one. A flow that originally needed N interactions can compress into two or three: retrieve definitions → generate the script → read the run results.

4) Capabilities begin to accumulate as engineering assets. Scripts can be stored in a repository, put under version control, and covered by unit tests. In effect, part of an agent’s capability is migrated from “experience living in prompts” to “maintainable code.”

4. Four Engineering Problems That Must Be Solved

1) Tool discovery and retrieval. Once tools grow numerous, neither “show the model everything” nor “show it nothing” is viable. A workable design is a three-layer structure: a directory-level index (overview of modules and capabilities) → a signature-level summary (function name, parameter types, a one-line description) → the full definition (with examples and error codes). Retrieval can combine keyword matching and semantic similarity, backed by a human-curated list of high-frequency tools as a stable baseline.

2) Execution-environment isolation and security. Letting model-generated code run directly means turning untrusted input into executable instructions, so it must be designed on the assumption that things “may go wrong and may be manipulated”: run in a dedicated container or lightweight virtual machine as a sandbox, with limits on CPU, memory, and execution time; block arbitrary outbound network access by default, allowing egress only to registered tool endpoints; inject credentials at runtime so that neither the script nor the model ever touches plaintext secrets, and issue short-lived tokens scoped to least privilege; put a secondary confirmation or human-approval gate on side-effecting operations such as writes, deletes, and external sends; and retain the script, an input summary, an output summary, and the elapsed time for each execution to meet audit requirements.

3) State, idempotency, and recoverability. A script may fail partway through, while tool calls often carry side effects. Write operations should be designed around idempotency keys, long flows should be split into re-entrant fragments, and an outer layer should uniformly wrap timeouts and retry policies—avoiding the “partial success” state that leads to inconsistent data.

4) Observability. Code execution moves part of the observability from conversation logs to program logs. You need to collect script standard output, exception stacks, and each tool call’s elapsed time and return code in a unified way, and feed them into a logging service with monitoring and alerting; at the same time, keep a snapshot of the context the model saw on that turn. Only when the two are aligned can you quickly tell, when quality regresses, whether the problem is that retrieval selected the wrong tool or that the script logic itself is flawed.

5. Applicable Boundaries and Tradeoffs

This is not a zero-cost replacement, and it should be viewed objectively:

• It depends on runtime capability: you need a stable code-execution environment and tool-code encapsulation, which is a substantial amount of rework.
• It raises the bar on model capability: the model needs fairly reliable multi-file code writing and self-debugging ability; weaker models tend to produce scripts that don’t run.
• The debugging chain is longer: failures can originate from three layers—retrieval, code generation, and the execution environment—which demands higher observability.
• It is not suited to very short interactions: for single-turn, single-tool Q&A, direct tool calling is actually simpler.

A more realistic path is therefore a hybrid one: simple tasks go through direct calling, complex multi-step tasks go through code execution; the same set of tool capabilities is exposed in both forms, and the orchestration layer picks one based on task complexity.

6. Conclusion

The value of this shift is not in “writing a few fewer lines of prompt.” It is in re-drawing the boundary of responsibility between the model and the runtime: the model handles judgment and planning, while the runtime handles data transfer and process control. Context is the scarcest resource, so any data movement that is unrelated to reasoning should be kept out of it as far as possible.

For any team building intelligent coding tools or enterprise-grade agents, three reusable design principles can be distilled here: turn tool capabilities into code, keep intermediate data inside the execution environment, and hand control flow over to the programming language. Combined with general cloud capabilities such as container platforms, object storage, logging services, and monitoring and alerting (for example, the equivalent general-purpose product categories offered by Baidu AI Cloud), this paradigm can be engineered at relatively low cost without binding to any specific framework or protocol implementation.