On-Demand Tool Discovery: Optimizing Context for LLM Tool Calling

1. Why LLM Applications Need Tool Discovery

Tool calling in large language model applications typically relies on structured definitions. Each tool needs to describe its name, purpose, input parameters, parameter types, required fields, allowed values, outputs, and usage constraints. The model uses this information to decide which tool to call and generate valid arguments.

When an application has only a few tools, including every definition in the request context is simple and effective. For example, an application with just three tools—for checking the weather, reading a calendar, and sending notifications—can usually accommodate their full definitions without significant overhead.

As a system expands, however, its tool catalog may grow to dozens or hundreds of tools across multiple domains:

• User and access management
• Order, inventory, and shipment queries
• Document retrieval and content processing
• Data analysis and report generation
• Messaging and task scheduling
• File operations and format conversion
• Monitoring, auditing, and operations

Sending every definition with every request creates three immediate problems.

Growing Context Costs

Tool definitions consume input tokens. Even a short user request can carry substantial fixed overhead when the tool catalog is large. In high-volume applications, repeatedly sending this information can significantly increase total token consumption.

Tool definitions also compete for context space with user input, conversation history, business rules, retrieved information, and model output. Larger context windows do not mean every available token should be filled indiscriminately.

Higher Request Latency

Longer inputs generally require more processing. A larger tool catalog also gives the model more candidates to evaluate, potentially increasing both time to first response and total completion time.

This additional latency affects interactive user experiences and can accumulate across multiple tool-calling rounds in automated workflows.

More Tool-Selection Errors

Large catalogs often contain similar tools. Several query tools, for example, might have nearly identical names and parameter structures but different permission scopes.

The more irrelevant definitions the model sees, the greater the potential for incorrect tool selection, parameter confusion, or unnecessarily long call sequences.

The central question therefore shifts from “How do we expose every tool to the model?” to “How do we provide the most relevant, complete, and reliable definitions when the model needs them?”

2. How On-Demand Tool Discovery Works

Tool discovery exposes capabilities as needed rather than loading every definition upfront. It separates tool access into two layers:

1. Lightweight discovery: A compact tool index tells the model which capability categories exist and how to find relevant tools.
2. Full definition loading: When a task requires a specific tool, the system retrieves its complete definition.

This differs from the conventional preload-everything approach.

A traditional workflow looks like this:

User request
  ↓
Load all tool definitions
  ↓
Send the request and complete tool catalog to the model
  ↓
Model selects a tool and generates arguments
  ↓
Execute the tool

An on-demand discovery workflow looks like this:

User request
  ↓
Provide a lightweight tool index
  ↓
Model determines whether tool search is needed
  ↓
Retrieve candidate tools and their full definitions
  ↓
Model selects a tool and generates arguments
  ↓
Execute the tool

“Search” does not necessarily mean keyword search. It can combine classification, semantic matching, permission filtering, business routing, and multistage selection.

The objective is to show the model only the tool definitions relevant to its current task.

3. A Typical Tool Discovery Architecture

A complete tool discovery system generally consists of five components.

Tool Registry

The tool registry maintains core metadata, including:

• Unique tool identifier
• Tool name and short description
• Business domain
• Intended use cases
• Input parameter summary
• Risk level
• Required permissions
• Current version and status
• Whether automatic invocation is permitted

The registry does not need to contain all execution logic from the outset. It does, however, need stable identifiers and clear descriptions. Without consistent definitions, retrieval, auditing, and version management become difficult.

Tool Index

The tool index is a lightweight capability catalog for the model. It can include categories, short descriptions, and search guidance without exposing complete parameter schemas.

Tools can be organized by:

• Business domain
• Read versus write operations
• User task
• Risk level
• Data source
• Data freshness requirements

The index should be concise but informative. Too little detail makes it difficult to find the right capability; too much recreates the original problem of loading every definition into context.

Retrieval and Filtering Service

The retrieval service returns candidate tools based on the user request, conversation context, and current task state. It may combine:

• Keyword matching
• Semantic vector retrieval
• Business-domain classification
• Rule-based routing
• Permission filtering
• Tool availability checks
• Risk-policy filtering

Relevance must not be the only selection criterion. A tool should not be returned merely because it is semantically relevant if the user lacks permission, the tool is disabled, or the current task is not allowed to use it.

Full Definition Service

Once candidate tools have been selected, the system provides complete definitions, including parameter structures, field constraints, prerequisites, error-handling guidance, and necessary business context.

A definition must provide enough information to support execution. “Query order information” is insufficient. The model also needs to know the order identifier format, required fields, time-range representation, and expected response data.

Execution and Result-Delivery Layer

After the model generates arguments, the execution layer performs the actual call. Its responsibilities include:

• Argument validation
• Authentication
• Authorization
• Idempotency controls
• Timeout handling
• Retry policies
• Result trimming
• Sensitive-data masking
• Audit logging

Tool discovery determines which definitions reach the model and when. It does not replace execution-layer security. Model-generated arguments must always be treated as untrusted input and validated on the server.

4. The Tool Discovery Workflow

A complete discovery-and-execution cycle typically follows five steps.

Step 1: Analyze the User’s Task

The system receives the request and uses conversation history to identify the task objective.

For example, “Find anomalous transactions from last week and generate a summary” may require data querying, time-range calculation, and report generation.

At this stage, the system only needs to identify likely capability domains—not load every potentially relevant definition.

Step 2: Provide a Lightweight Capability Catalog

The model receives access to tool search along with a short catalog that explains the available search scope:

Available capability categories:
- Data queries: Retrieve business data and filter by time range or status
- Content processing: Extract, classify, summarize, and format text
- File operations: Read or generate files in specified formats
- Notifications and tasks: Send messages or create asynchronous tasks

These descriptions support discovery. They should not contain extensive parameter details.

Step 3: Retrieve Candidate Tools

The model or an intermediate layer initiates tool search for the current task.

The retrieval system can first filter by permissions and availability, then rank the remaining tools by semantic relevance and return a small candidate set.

Returning too many tools undermines the value of on-demand loading and makes selection harder. Complex tasks can use staged retrieval: identify the business domain first, then search for specific tools within it.

Step 4: Load Complete Definitions

The system loads full schemas for the final candidates. These should include official tool names, parameter structures, field constraints, and invocation instructions. Typical usage conditions and common errors can be included when necessary.

Definitions should remain as stable as possible. Frequent changes to field names, required parameters, or response structures can reduce invocation reliability and increase compatibility costs.

Step 5: Execute and Process the Result

The model generates arguments, and the execution layer validates them before calling the backend service. Results can then be returned to the model for further analysis or processed directly by application code.

Failures should produce structured errors that distinguish among:

• Invalid arguments
• Insufficient permissions
• Missing resources
• Rate limits
• Timeouts
• Internal errors

Clear error categories help the model decide whether to correct arguments, ask the user for more information, or stop the workflow.

5. Designing Effective Tool Definitions

The effectiveness of tool discovery depends heavily on definition quality.

Use Names That Identify the Action and Object

A tool name should make clear what action it performs and which object it acts on. Avoid obscure internal abbreviations.

Descriptions should distinguish similarly named tools—for example, summary queries versus detailed queries, or change previews versus final submissions.

Describe the Boundaries of Use

A description should explain both what a tool can do and what it cannot do.

For a query tool, relevant boundaries might include supported data scopes, whether partial matching is available, whether results are real-time, and whether additional authorization checks apply.

Make Parameter Constraints Explicit

Each parameter should specify, where applicable:

• Data type
• Whether it is required
• Allowed values
• Default behavior
• Format requirements
• Maximum length
• Dependencies on other parameters

If a field requires a standard timestamp format, state that requirement rather than expecting the model to infer it.

Distinguish Read and Write Operations

Queries, creation, updates, and deletion carry different levels of risk. Both the index and full definitions should identify operation types, particularly for tools with external side effects.

Actions such as sending messages, submitting orders, or changing configurations may require user confirmation before invocation or a separate approval step in the business workflow.

6. Performance Optimization and Cost Control

The main benefit of tool discovery is reducing irrelevant tool definitions in the input context. That benefit must be balanced against retrieval overhead.

Limit Search Rounds

Repeated tool searches can consume enough time and resources to offset context savings. Task classification, domain routing, and search-result caching can reduce redundant retrieval.

Cache Stable Definitions

Complete definitions can be cached when tool versions and permissions are stable, avoiding repeated reads from the registry.

Caches must be tied to version information and invalidated promptly when definitions change.

Expose Tools Hierarchically

A three-level structure can organize discovery:

• Level 1: Business domain
• Level 2: Capability category
• Level 3: Specific tool

Identifying a domain before loading specific definitions is often more reliable than searching a large, flat catalog.

Trim Tool Results

Tool outputs also consume context. On-demand definition loading cannot prevent waste if execution results contain large amounts of irrelevant data.

Support field selection, pagination, summarization, and result truncation to keep outputs manageable.

7. Security and Governance

Tool discovery broadens the range of system capabilities a model can encounter, making governance essential.

First, retrieval results should be filtered using the current user, session, tenant, and application identities. Discovering a tool must never imply permission to execute it.

Second, high-risk tools should be managed separately from ordinary read-only tools. Sensitive operations may require human approval, explicit confirmation, a designated network environment, or stronger authentication.

Third, maintain an end-to-end audit trail covering:

• The user request
• Retrieved tools
• The tool ultimately selected
• Submitted arguments
• Execution results
• Reasons for failures

These records support troubleshooting and investigation of incorrect calls or unauthorized access attempts.

Tool descriptions also require security review. They should not contain secrets, internal addresses, sensitive business rules, or unnecessary system details. Tool outputs must likewise be sanitized so that internal fields are not exposed directly to the model or user.

8. Evaluating Tool Discovery

Token savings alone are not enough to establish success. Evaluation should cover resources, quality, stability, and security.

Resource Metrics

Track average input and output tokens, retrieval count, the volume of tool definitions loaded, request latency, and cost per task.

Quality Metrics

Measure tool-selection accuracy, argument correctness, task completion rate, error recovery rate, and the proportion of invalid or unnecessary calls.

Stability Metrics

Evaluate success rates across task types, performance over long conversations, latency variability under concurrency, and compatibility after tool-version changes.

Security Metrics

Monitor unauthorized retrieval rates, unintended invocation of high-risk tools, sensitive-data exposure incidents, and audit-trail completeness.

Use full tool loading as the baseline and compare it with on-demand discovery on a task set spanning different levels of complexity. Token and latency improvements are meaningful only if accuracy remains broadly comparable.

9. A Practical Migration Plan

Systems that already inject every tool definition can migrate incrementally:

1. Inventory existing tools and remove duplicate, deprecated, and long-unused definitions.
2. Standardize tool names, parameter structures, and description formats.
3. Classify tools by business domain and risk level.
4. Build a lightweight index and permission-filtering logic.
5. Enable on-demand discovery first for low-risk, read-only tools.
6. Compare token usage, latency, and success rates before and after migration.
7. Add caching, version management, and auditing.
8. Gradually introduce higher-risk write operations.

Do not hide every tool at once. Maintain a fallback mechanism so that retrieval failures, insufficient candidates, or definition-version mismatches can return the system to a controlled version of the traditional tool-calling path.

Conclusion

On-demand tool discovery replaces the expectation that an LLM must understand every system capability upfront with a task-focused process of finding and loading capabilities as needed.

This approach can reduce irrelevant context, input costs, and latency while making tool governance, access control, and version management easier to maintain.

It is not simply a matter of adding a search endpoint. Effective implementation requires coordinated tool registration, indexing, permission filtering, definition management, execution validation, and evaluation. With consistent descriptions, reliable retrieval, and clear security boundaries, on-demand loading can become a foundational capability for large-scale LLM applications.