Native Multimodality and Long-Horizon Agents: Architecture Design and Engineering Practices
This article examines two major directions in recent large language model advances—native multimodality and long-horizon agents—and how they map to the two foundational bottlenecks of agent systems: information acquisition and expression, and process planning and control. It explains the technical substance, core challenges, layered architecture, and implementation essentials of both, covering context budgeting, idempotency and retry, milestone-based validation and fallback, security boundaries, and a four-layer evaluation framework. With runnable code and selection guidance, it helps developers choose technical directions and make sound engineering trade-offs.}
Native Multimodality and Long-Horizon Agents: Architecture Design and Engineering Practices
Recent capability upgrades in large models have concentrated on two directions: native multimodality and long-horizon agents. The former addresses whether a model can understand multiple types of information as naturally as a human; the latter addresses whether a model can carry a complex task through to completion like a human.
From an engineering perspective, these two directions correspond precisely to two foundational bottlenecks in agent systems: the acquisition and expression of information, and the planning and control of the process. This article follows these two threads to lay out the technical substance, core challenges, architecture design, and implementation essentials, with runnable code and selection guidance to help you make clear decisions about technical direction and engineering trade-offs in your projects.
1. Why “Native Multimodality” and “Long-Horizon Agents”
The evolution of large model capabilities broadly follows a chain of “perception—understanding—decision—action.” Early conversational models focused on single-turn text question answering; with reasoning enhancements, models began handling complex single-turn tasks such as code generation and data analysis. But real business needs go well beyond this:
- Business documents exist as mixed text-and-image layouts and scanned copies;
- User feedback mixes screenshots, voice, and text;
- Tasks require querying, cross-checking, and processing across multiple systems.
There is an obvious gap between capability supply and business demand. Native multimodality and long-horizon agents are two key paths to closing this gap. It is worth noting that “native multimodality” is primarily an architectural choice at the model-training level, while “long-horizon agents” are primarily an engineering form at the system-application level; the two are complementary: multimodality enhances the agent’s “senses,” while long-horizon capability shapes its “way of acting.”
2. Native Multimodality: From “Splicing” to “Unified Modeling”
2.1 Two Technical Routes
Multimodal models currently follow two main implementation approaches.
Route 1: Modality splicing (late fusion). With a language model at the core, separate vision and speech encoders extract features, which are then mapped into language model inputs through projection or adapter modules. This approach is engineering-mature and relatively easy to extend with new modalities, but modality interaction happens “after integration,” and fine-grained cross-modal associations are often limited.
Route 2: Native multimodality (unified modeling). Signals such as text, image, audio, and video are mapped into the same representation space during pretraining and jointly trained in a single model. Modality associations are modeled from the very start of training rather than aligned after the fact.
To see the difference between native multimodality and modality splicing at a glance:
| Dimension | Modality Splicing | Native Multimodality |
|---|---|---|
| Fusion stage | Modalities integrated into the language model after each is encoded | Unified representation and joint training during pretraining |
| Modality interaction | Interaction happens after integration; fine-grained association is weak | Unified modeling during training; fine-grained association is strong |
| Extending new modalities | Relatively easy; add an adapter module | Requires retraining or large-scale incremental training; higher cost |
| Training complexity | Single-modality training is simple, but cross-modal alignment needs extra data and processes | High training complexity; requires large-scale multimodal data and compute |
How to choose: Judge along three dimensions—whether fine-grained cross-modal interaction needs are high (e.g., precise reference between text and images), whether you have the data and compute for joint pretraining, and how frequently new modalities need to be added (frequent expansion favors splicing). Overall, most teams start with splicing and upgrade to native as needed, striking a balance between cost and capability.
2.2 The Engineering Value of “Native”
“Native” is not some mysterious switch; it shows up in a more natural cross-modal understanding experience:
- More complete input expression: Annotated screenshots, screen recordings, and spoken dictation can be understood jointly with text instructions in the same context, reducing the detail loss caused by “transcribing to text first and then handing it to the model.”
- More natural reference and citation: The model can directly reference “that screenshot you just sent” or “the third row of the table above,” reducing the burden of maintaining a separate reference identifier for each modality.
- Lower integration complexity: Multimodal information does not need to be split into multiple preprocessing pipelines, called separately, and then stitched together; a unified context simplifies the chain and state management.
- More agent-friendly: In long-horizon tasks, the model repeatedly switches between instructions, screenshots, and documents; a unified representation reduces the cost of switching.
From a business-scenario perspective, multimodal integration typically falls into three categories: information-entry (document recognition, form filling), interaction assistance (locating problems from screenshots, generating suggestions from interface state), and quality verification (comparing target requirements against actual result screenshots to judge whether a task is complete).
3. Long-Horizon Agents: Why “Walking a Few More Steps” Is So Hard
The gap between single-turn question answering and long-horizon tasks is not the number of steps but state management and error recovery:
- Single-turn question answering: One input, one output; state exists only in the context window;
- Fixed workflow: Steps and branches are pre-orchestrated, with fixed nodes and clear boundaries;
- Long-horizon agent: Task decomposition, step selection, tool calling, and result validation are all decided dynamically by the model, potentially spanning dozens of steps, multiple systems, and a long duration.
3.1 Four Core Challenges
First, error accumulation. Let the single-step correctness rate be p. Under the approximation that steps are independent of each other, the joint correctness rate of an n-step task is roughly p to the n-th power. For p = 0.95, 10 steps is about 60%, and 20 steps drops to about 36%. Therefore, long-horizon tasks cannot rely on “good single-step results” as a quality metric; error-correction mechanisms must be designed at the system level.
Second, context decay. The longer the trajectory, the more easily key information is diluted or overwritten by compressed summaries. Common phenomena include early constraints being forgotten, steps that repeat already-completed work, and intermediate conclusions becoming disconnected from later reasoning.
Third, state-process coupling. Tasks often run across sessions, across processes, or even across days. If state is kept only in memory or in the conversation window, a single interruption can cause the task to be lost and unable to resume.
Fourth, tool-call reliability. The number of tool calls in long-horizon tasks multiplies, amplifying problems such as parameter hallucination, duplicate calls, timeout retries, and wrong call ordering—and these are hard to locate from the final result.
3.2 Design Principles Derived from These
- Externalize state: Persist task state and execution progress to external storage rather than relying on the context window;
- Verifiable steps: Validate each critical step after completion; roll back or retry if it fails;
- Compressible context: Retain historical information in tiers by importance and control length through summarization;
- Observable process: Fully record the input, output, latency, and result of each step for easy localization and review.
4. Architecture Design: The Layered Structure of Long-Horizon Agents
A deployable long-horizon agent typically uses a six-layer structure: a perception layer (unified understanding and structuring of multimodal input), a memory layer (short-term trajectory, long-term knowledge, and task state), a planning layer (task decomposition, step ordering, and dependency management), an execution layer (tool calling, action execution, and result return), a validation layer (verification of critical results and retry decisions), and an evaluation layer (a quality loop of offline evaluation and online monitoring). The following sections focus on the memory, planning, execution, and validation modules.
4.1 Memory Tiers
- Working memory: The raw conversation and the results of the last several steps retained in the current context;
- Task memory: Structured storage of the goal, completed steps, intermediate artifacts, and pending items;
- Long-term knowledge: Domain knowledge, historical cases, and user preferences, retrieved and injected on demand.
The key point: do not stuff all information into the context; instead, put structured state into external storage and rebuild the minimal necessary context on demand each round.
4.2 Planning and Decomposition: Static or Rolling
- Static planning: Decompose all steps at once at the start and then execute sequentially. Simple to implement and direct to debug, but weak in adaptability. When to use: the flow is fixed, can be enumerated in advance, and compliance-audit requirements demand a deterministic path (e.g., an approval workflow).
- Rolling planning (replan): Define a coarse-grained plan first, then dynamically adjust it after each step based on the latest results, rolling back to a checkpoint if necessary. Higher implementation and debugging cost, but strong adaptability.
Long-horizon tasks generally benefit from rolling planning combined with milestone checks: validate once after each phase goal is completed, continue only if it passes, and otherwise enter a repair flow. Do not treat “always rolling planning” as a conclusion—a fixed approval workflow is often simpler with static planning.
4.3 Tool Calling: Parameter Validation and Idempotency
Tool calling is the main “source of action” for a long-horizon agent and the most error-prone area. Engineering must implement three measures: parameter validation, idempotency keys, and timeout retries. The following code is directly runnable (with dependency definitions, concurrency protection, and cache TTL filled in) and demonstrates parameter validation and idempotency handling:
import hashlibimport timeimport threadingfrom dataclasses import dataclass, field# Cache TTL (seconds): results reused across days must expire promptly to avoid resuming on stale dataTTL = 3600class ValidationError(Exception):def __init__(self, message: str):super().__init__(message)self.message = message@dataclassclass ToolResult:code: strok: booldata: dict = field(default_factory=dict)retryable: bool = Falsets: float = field(default_factory=time.time) # write time, used for TTL checksclass Session:"""Task session: serializable when persisted externally; uses in-memory concurrency protection here."""def __init__(self, task_id: str):self.task_id = task_idself.done = {} # idempotency key -> ToolResultself.lock = threading.Lock() # concurrency protectiondef call_tool(tool, args: dict, session: Session) -> ToolResult:# 1. Parameter validation: on failure, hand the structured error back to the model for self-correction rather than aborting with an exceptiontry:validated = tool.validate(args)except ValidationError as e:return ToolResult(code="INVALID_ARGS", ok=False,data={"error": e.message}, retryable=True)# 2. Idempotency key: incorporate the task identifier + a day shard to avoid false hits across tasks/daysday = time.strftime("%Y-%m-%d")raw = f"{session.task_id}|{tool.name}|{day}|{sorted(validated.items())}"key = hashlib.sha256(str(raw).encode()).hexdigest()[:16]# 3. get-or-compute (locked): reuse an unexpired successful result directly, avoiding repeated side effectswith session.lock:cached = session.done.get(key)if cached and not cached.retryable and time.time() - cached.ts < TTL:return cached# 4. Invoke the tool, timeout=30s (see the note in section 5.1 for how the value is chosen)result = tool.invoke(validated, timeout=30)result.ts = time.time()# Mark retryable only for retryable errorsresult.retryable = not result.ok and result.code in {"TIMEOUT", "RATE_LIMITED"}# 5. Cache only successful results: retryable failures are not written to the cache, so later retries are not short-circuited by this failureif result.ok:with session.lock:session.done[key] = resultreturn result
This fills in the ToolResult and ValidationError definitions missing from the original implementation; the idempotency key incorporates session.task_id and a day shard along with a TTL; session.done is protected by a lock, with get-or-compute concurrency handling; and retryable failures are not written to the cache, ensuring retries are not short-circuited.
4.4 Validation and Fallback: Milestone-Style Checkpoints
The core of the validation layer is “validate first, then proceed.” Common methods include rule validation (format, field completeness, value ranges), model self-validation (judging against the goal and completed results), and multimodal validation (using native multimodality to compare target requirements against actual result screenshots or files). A validation failure should return to the previous checkpoint to retry, or escalate to human intervention, rather than blindly continuing.
4.5 End-to-End Minimal Case: Read Order Screenshot → Query Logistics → Fill Form → Screenshot Verification
Using a “logistics form fill” task to string together the six-layer structure, demonstrating one validation fallback and one tool timeout retry:
- Perception layer: The user uploads an order screenshot; the model understands it uniformly and extracts
order number, recipient information, target status. - Memory layer: Write the task memory (goal = complete the logistics information form fill) and establish the task state.
- Planning layer: Generate a rolling plan
[query logistics, fill form, screenshot verification]. - Execution layer: Call
query_logistics(order_number). The first call returnsTIMEOUT—becauseretryable=True, it is not cached, and the system automatically retries; the second call succeeds, and the result is cached. - Validation layer: After calling
fill_form(logistics info), multimodal validation compares the “target required fields” against the “form-fill result screenshot” and finds the carrier field empty—the validation fails and falls back to this checkpoint, retryingfill_formto complete the field. - Evaluation layer: Record the trajectory (steps, latency, retry count, fallback count); after successful completion, enter online monitoring.
This chain fully covers parameter validation, idempotency, timeout retry, multimodal validation, and fallback, and it confirms that “the value of a long-horizon task lies in recovery ability in failure scenarios.”
5. Engineering Implementation: Context Budget and Safety Boundaries
5.1 Context Budget Management
The most common problem in long-horizon tasks is not insufficient model capability but context being squeezed out by historical information. It is recommended to allocate a fixed quota for each round’s context, assembled as “system prompt + task goal + structured summary + recent trajectory”:
SYSTEM_TOKEN = 2000 # role setup, tool descriptions, output specTASK_TOKEN = 1500 # task goal and current constraintsHISTORY_TOKEN = 5000 # quota for the recent execution trajectoryOUTPUT_TOKEN = 1500 # reserved generation spacedef build_context(system, task, history, summary, tokenizer):head = [system, task, summary]used = sum(len(tokenizer.encode(s)) for s in head) + OUTPUT_TOKEN# Backfill from the latest step backward; total trajectory must not exceed the HISTORY_TOKEN quotakept, history_used = [], 0for item in reversed(history):cost = len(tokenizer.encode(item))if history_used + cost > HISTORY_TOKEN:breakkept.append(item)history_used += costused += costkept.reverse()# Over-quota history goes to summarize, which structurally compresses it by "conclusions/completed/failure reasons/unfinished" and backfills itdropped = history[: len(history) - len(kept)]if dropped:summary = summarize(overflow=dropped, keep=summary)return head[:2] + [summary] + keptdef summarize(overflow: list, keep: str) -> str:# Placeholder: retain confirmed conclusions, completed actions, failure reasons, unfinished items; delete repetitive intermediate reasoning...
Here, trajectory retention is strictly constrained by the HISTORY_TOKEN quota (rather than indirectly controlled by a total cap), and over-budget history truly enters summarize for folding and backfilling rather than being directly discarded.
Regarding the timeout=30 value and the 2000/1500/5000/1500 values: they are all tunable hotspots that need empirical calibration for the model and scenario. The timeout value should reference the downstream tool’s P99 latency with margin; the token quotas need attention to the fact that Chinese and English have different token ratios (the same word count in Chinese typically consumes more tokens), and it is recommended to run percentile measurements on real samples with the target model’s tokenizer before settling on tiers. Summary generation should be goal-oriented rather than process-oriented: retain confirmed conclusions, completed actions, failure reasons, and unfinished items, and delete repetitive intermediate reasoning.
5.2 Safety Boundaries: Least Privilege and Sandboxed Execution
The action space of a long-horizon agent is naturally larger, so security management must be front-loaded:
- Least privilege: Grant only the tools and data scope necessary to complete the current task;
- Tiered approval: For high-risk actions involving funds, external publishing, or data deletion, require mandatory human confirmation;
- Sandbox isolation: Run code execution, file processing, and browser operations in isolated environments. Among these, browser and code sandboxes should use dedicated test accounts or a credential vault; hard-coding production accounts and passwords is prohibited, to avoid misusing a real identity during long-horizon tasks;
- Data deletion protection: For deletion-type actions, prefer soft delete/move to trash and set a retention period to reduce the cost of accidental deletion;
- Operation logging: Record the operator, parameters, results, and time for easy audit and traceability.
5.3 Asynchronization and State Externalization
Long-horizon tasks take a long time, so synchronous calls are often unsuitable for online services. A more robust approach is to convert them into asynchronous tasks: when a request arrives, first return a task identifier; the execution process is driven by a background task, with each step’s state change written to external storage and progress fed back through events or polling. This avoids occupying long-lived connections and also gives the task the ability to recover after interruption. At this point, mature cloud service capabilities (object storage, message queues, container platforms, logging and monitoring) can be used to carry state persistence, asynchronous decoupling, execution isolation, and full-chain observability.
6. Evaluation System: From “Single-Step Correctness” to “Task Completion”
Long-horizon agent evaluation cannot reuse single-turn metrics; a four-layer evaluation system is recommended:
- Component-level: Intent recognition, parameter extraction, and structural output format compliance rate;
- Trajectory-level: Step reasonableness, tool selection accuracy, and whether redundancy or loops occur;
- Task-level: Whether the final goal is achieved, including completion rate, human-intervention rate, and average latency and cost;
- Regression-level: After each model or prompt change, run the fixed evaluation set once to confirm no regression.
Building the evaluation set is equally critical: de-identified real samples (sample real tasks and de-identify them; names, phone numbers, ID numbers, and order numbers in screenshots must be covered, and de-identification failures must be automatically validated and blocked), difficulty grading (covering short/medium/long-horizon tasks, avoiding optimistic conclusions drawn only from simple examples), reference answers/golden trajectories (define the expected completion state and key checkpoints for each task), and adversarial samples (ambiguous instructions, missing information, tool exceptions, and other edge cases, to test robustness).
7. FAQ
- How to choose between native multimodality and splicing? Choose native when fine-grained cross-modal interaction needs are high and you have joint-training data and compute; start with splicing when new-modality integration is frequent and cost-sensitive.
- Why do long-horizon agents fail? The main causes are the four challenges of error accumulation, context decay, state-process coupling, and unreliable tool calling.
- How do you evaluate a long-horizon agent? Combine four layers—component-level, trajectory-level, task-level, and regression-level—along with golden trajectories and adversarial samples.
- How do you allocate the context budget? Assemble by quota as “system + task + summary + trajectory,” with trajectories constrained by an independent cap and over-budget content folded into the summary.
8. Common Misconceptions and Practice Advice
- Misconception 1: The longer the context, the stronger the long-horizon capability. Length is only a capacity ceiling; the key lies in memory tiering and budget management.
- Misconception 2: Using “prompt stacking” in place of architecture design. Prompts improve single-step output but cannot solve state management, error recovery, and observability.
- Misconception 3: Building full automation first and adding safety mechanisms later. Without approval and fallback for high-risk actions, the cost of errors is high; permissions and security boundaries should be built into the first version.
- Misconception 4: Demonstrating only with success cases. The value of long-horizon tasks lies in failure recovery, and evaluation must cover exceptions and edge cases.
In practice, follow the path of “small steps, gradual delegation”: first validate long-horizon capability in a low-risk, verifiable, closed scenario, then gradually expand to open scenarios while continuously improving evaluation and monitoring.
9. Conclusion
Native multimodality and long-horizon agents represent two key dimensions of large models moving from “answering questions” to “completing tasks”: the former broadens the breadth and depth of information input, while the latter determines the depth and reliability of task execution. For developers, what truly needs to be mastered is not a particular product form but a transferable methodology:
- Use layered memory to counter context decay;
- Use milestone validation to counter error accumulation;
- Use idempotency and retries to ensure tool-call reliability;
- Use four-layer evaluation to quantify long-horizon capability rather than relying on subjective feeling.
Combining these engineering elements with continuously evolving multimodal capabilities is what enables truly usable, maintainable, and scalable intelligent applications.
