Designing and Building Multi-Tool LLM Agents

    Multi-tool LLM agents handle complex tasks by combining intent understanding, dependency-aware planning, tool execution, and iterative observation. This article explains how to define reliable tool contracts, coordinate web search, code execution, and multimodal generation, manage context and failures, enforce security controls, evaluate tool use, and design a production-ready runtime.

    Why Complex Tasks Require Multiple Tools

    Traditional conversational systems often expose only one model capability: text generation, or a prompt-based mechanism that temporarily activates retrieval, computation, or image generation. Real-world tasks are rarely that simple. A request such as “Compare how a metric has changed over the past three years and include a trend chart” involves information retrieval, numerical processing, visualization, and conclusion writing. If a system can follow only one capability path, it may produce outdated facts, inaccurate calculations, or an unsuitable output format.

    The goal of a multi-tool agent is to understand the user’s intent, break a complex instruction into executable steps, and call external capabilities such as web search, code interpreters, and multimodal generation when needed. This is not simply a collection of independent features. It uses a shared planning and observation mechanism to decide when to call a tool, what inputs to provide, how to interpret the response, and how to continue with the next step.

    Capability Layers: Understanding, Planning, Execution, and Write-Back

    A practical design usually separates the system into four layers instead of placing all logic in a single prompt.

    1. Intent understanding. Identify the task type, constraints such as time range, data definition, and output format, as well as the completion criteria. When a request is ambiguous, the agent should fill in critical missing slots before selecting a tool.
    2. Task planning. Break the objective into dependent subtasks. Mark which steps must run sequentially, which can run in parallel, and what constitutes success for each step.
    3. Tool execution. Select capabilities through consistent tool descriptions, validate parameters, execute the tool, and obtain structured results.
    4. Observation and write-back. Compress tool outputs into the following context and determine whether the objective has been met. If not, revise the plan instead of repeating the same failed call.

    Together, these layers form a short loop:

    Plan → Select a tool → Execute → Observe → Re-plan

    The number of iterations should be capped to prevent the agent from looping indefinitely on an ineffective path.

    Tools Should Be Contract-Based Capabilities, Not Prompt Tricks

    Using prompts alone to switch between “modes” produces unstable boundaries. The model may fail to call a tool, call the wrong tool, or estimate a value in natural language when it should perform an actual calculation. A more reliable engineering approach is to define a stable contract for every tool.

    Each contract should specify:

    • Name and purpose: A concise explanation of what the tool is suitable for and what it should not be used for.
    • Input schema: Fields, types, required values, and valid ranges.
    • Output schema: The success structure, error codes, and readable failure reasons.
    • Side effects and permissions: Whether the tool performs read-only retrieval, executable computation, or file generation. These permissions should be separated.
    • Expected cost and latency: Information that helps the planner choose among multiple valid paths instead of calling every available tool by default.

    A tool description can be simple. Machine validation matters more than polished wording:

    1. tool: web_search
    2. purpose: Retrieve time-sensitive information from public web pages; not intended for precise numerical computation
    3. input: { query: string, time_range: optional }
    4. output: { items: [{ title, snippet, source, time }], error: optional }
    1. tool: code_interpreter
    2. purpose: Execute data analysis and computation scripts in an isolated environment
    3. input: { task: string, tables: optional }
    4. output: { result, stdout, artifacts, error }
    1. tool: image_generate
    2. purpose: Generate visuals from text descriptions; not a substitute for fact checking
    3. input: { prompt, style, size }
    4. output: { image_ref, error }

    The planner should see these contracts and the current task state. It should not receive the complete history of tool logs in raw form.

    Boundaries of Three Common Tool Types

    Web-based capabilities address information that is not present in the model’s parameters or may already be outdated. They are suitable for policies, public datasets, product documentation, and other content that requires sources. The system should preserve the source and timestamp, and the final response should distinguish retrieved statements from the model’s own synthesis.

    Web search should not be the only calculation engine. Numbers found on web pages should be passed to an interpreter for verification rather than repeated directly in the answer.

    Code Interpreters

    Code interpreters are appropriate for deterministic tasks such as statistics, unit conversion, table cleaning, simple modeling, and result validation. Their strengths include reproducible processes and easier error localization.

    Their limitations must also be explicit. The execution environment should be isolated, with unrestricted access to internal networks and credentials prohibited. Input data should have size and type limits, and default policies should cover timeouts, memory, and dependencies. When presenting results to users, show the conclusion and key intermediate values first. Provide the complete script as a traceable artifact when needed.

    Multimodal Generation

    Multimodal generation is useful for visualizing confirmed conclusions, including trend illustrations, diagrams, and cover images. It does not verify data.

    The usual order should be:

    1. Retrieve or calculate the numbers and conclusions.
    2. Add those constraints to the generation instructions.
    3. Generate the visual.

    If the image and the text disagree, the structured result should take precedence and the image should be regenerated.

    Planning Strategy: Check Dependencies Before Choosing Tools

    A practical planning heuristic includes the following rules:

    • If facts are missing and may change over time, search first and then decide whether calculation is needed.
    • If a table or a clear formula is already available, use the interpreter directly instead of asking the model to perform mental arithmetic through generated text.
    • If a visual is required but the data has not been finalized, do not generate the visual first.
    • When sources conflict, present them side by side and perform a second search when necessary. Do not force them into a single number.
    • If the user only wants a conceptual explanation, tools may not be necessary. Avoid adding latency and fabricated-looking citations.

    Parallel execution is appropriate only for subtasks without data dependencies, such as searching for two independent topics at the same time. When a later step consumes numbers or files produced by an earlier step, execution must be sequential. Intermediate results should be stored in explicit fields rather than passed through ambiguous natural-language descriptions.

    Context Management, Failure Handling, and Observability

    Context is one of the most common failure points in multi-tool workflows. Search excerpts may be too long, interpreter logs may flood the context, and image descriptions may drift away from the original task. A useful approach is to write back only summary fields for each step, such as the number of matches, key values, error reasons, and artifact references.

    Original pages and complete standard output should be stored in object storage or a logging service and replayed only when needed, rather than inserted into every subsequent prompt.

    Failures should be classified instead of handled with a generic “try again” instruction:

    • Invalid parameters: Correct them according to the contract and retry only a small number of times.
    • No results or insufficient sources: Broaden the query or ask the user to clarify the scope.
    • Execution timeout or insufficient resources: Reduce the data volume or split the task.
    • Permission denied: Stop that path and explain the limitation. Do not attempt to bypass it.
    • Contradictory results: Mark the result as uncertain and preserve the evidence from each source.

    In production, record the plan version, tool name, duration, whether a retry occurred, and whether the final completion criteria were satisfied. These metrics are more meaningful than whether the response merely appears complete. Monitoring and alerts should focus on timeout rates, tool error rates, and abnormal loop counts.

    Security and Permissions

    Tools expand the agent’s ability to act, so security policies should be separated by tool type.

    • Search should normally be limited to approved public sources. Links and instructions included in retrieved content must not override system policies.
    • Interpreters should use least-privilege permissions and network isolation. Secrets and internal addresses must not be placed in executable context.
    • Generation capabilities must follow content and copyright requirements. Unauthorized material should not be treated as content that can be replicated.
    • User data should be retained for the minimum necessary period by default. Logs should be redacted, and access to generated artifacts should be controlled.

    An agent should be able to refuse a step that cannot be completed safely and offer an actionable alternative, such as narrowing the scope, switching to offline data, or explaining the method without executing it.

    How to Evaluate Tool Use

    Evaluation should not focus only on whether the final paragraph is fluent. It should also examine whether the process was appropriate.

    • Tool selection: The agent uses a tool when needed and avoids using one when it is unnecessary.
    • Parameter quality: Search queries, script tasks, and generation instructions contain the required constraints.
    • Facts and calculations: Key numbers can be traced to a source or interpreter output.
    • Process efficiency: Meaningless repeated calls count as failures.
    • Failure recovery: After an error, the agent changes its plan instead of repeating the same call.
    • Faithful expression: The final answer does not add conclusions that are absent from the tool results.

    A fixed task set can cover retrieval, calculation, visualization, and mixed workflows. It should also include counterexamples in which a pure question-answering task should not call any tool. During regression testing, compare planning traces rather than only comparing the final text.

    Production Product Forms

    For developers, multi-tool capabilities are better implemented as a composable runtime consisting of:

    • A conversational entry point
    • A planner
    • A tool gateway
    • An execution sandbox
    • Artifact storage
    • Audit logs

    Message queues can help absorb traffic spikes and support asynchronous long-running tasks. Container platforms can isolate interpreter workloads, while logging and monitoring systems support failure replay. These are general categories of cloud capabilities and do not need to be tied to one proprietary interface.

    For business users, the interface should expose progress and evidence. Users should be able to see whether the agent is currently searching, calculating, or generating, and expand the source associated with an important number. For complex tasks, allowing users to confirm the data definition or scope during execution is often more reliable than generating one long answer in a single pass.

    Conclusion

    The value of a multi-tool agent lies in turning intent understanding, step planning, external capability calls, and observation-based correction into a stable feedback loop. This enables the system to handle real tasks that combine retrieval, computation, and multimodal presentation.

    Building this capability as an engineering system depends less on slogans than on tool contracts, dependency ordering, context reduction, failure classification, permission isolation, and trace-based evaluation. With these constraints in place, an agent can remain interpretable and recoverable under complex instructions instead of only appearing intelligent occasionally.