Native Multimodal Models and Long-Horizon Agents: Architecture and Implementation Considerations
This article examines the architectural choices and engineering practices for building long-horizon agents on top of native multimodal models. It compares concatenation-based and native multimodal approaches, breaks down the core components of long-running agents—planning, execution, memory, reflection, and state persistence—and provides a reference execution loop, a layered evaluation framework, risk controls, and practical implementation recommendations.
1. Background: From Single-Turn Q&A to Continuous Execution
How large models are used is shifting from “answer a question” to “complete a task.” In the past, a typical scenario was text in and text out. In real business settings, inputs often include images, tables, screenshots, audio, and long documents. Tasks also span multiple steps and can take tens of minutes or even hours. This creates two engineering problems: how the model understands information from multiple modalities, and how the system maintains goal consistency and traceable state over long periods.
2. Native Multimodal Models: Unified Representations vs. Concatenation-Based Approaches
Multimodal approaches fall broadly into two categories. Concatenation-based approaches connect a vision encoder, a speech recognition module, and a language model through a projection layer. Each module can be trained relatively independently. The interfaces are clear, and the modification cost is low, but cross-modal reasoning depends on intermediate representations, and information may be lost during conversion. Native multimodal models instead let text, image, audio, and other tokens enter the same sequence during pretraining. The model learns cross-modal associations through a unified attention mechanism, and these associations are usually tighter.
The choice depends on three factors: the proportion of cross-modal reasoning in the task, the modality distribution of existing data, and the inference cost budget. If the business only needs to classify images, a concatenation-based approach is sufficient. If the task requires temporal reasoning between a video and its corresponding subtitles, a unified representation is usually more advantageous.
Input preprocessing is the first step in implementation. The following operations should be performed consistently:
- Resize images according to a fixed strategy, and record the original dimensions and aspect ratio.
- Split audio into segments with timestamps so that sources can be located.
- Parse the layout structure of documents first, then extract body text and tables separately.
Each segment should retain source metadata so that errors can later be traced back to the original input.
3. Core Components of Long-Horizon Agents
A long-horizon agent typically consists of five parts.
- Planner: Decomposes the overall goal into subtasks with dependencies. The planning result should be stored structurally, for example as a directed acyclic graph, so that failed steps can be retried and parallel branches can be identified.
- Executor: Calls tools to perform concrete actions, such as querying data, reading and writing files, and making API requests. Each call must record input parameters, output parameters, and latency as a basis for later troubleshooting.
- Memory module: Divided into short-term context and long-term memory. Short-term memory stores intermediate results of the current task, while long-term memory stores user preferences and historical experience. Long-term memory requires a retrieval mechanism; a common practice is vector retrieval combined with keyword filtering.
- Reflection and validation: After each step, check whether the result satisfies the constraints. On failure, the system can retry, switch tools, or hand the task off to a human.
- State persistence: Write task state to persistent storage so that after a process restart, the task can resume from the latest checkpoint.
These five modules are connected through explicit data structures. An agent without state persistence is prone to “restarting from scratch after an interruption” during long tasks, which affects both cost and user experience.
4. Reference Engineering Architecture
The following is a generic illustration of an agent execution loop. The example shows only the control flow. Model calls, tool implementations, and the storage layer are abstracted as interfaces and can be replaced according to the team’s technology stack.
from dataclasses import dataclass, fieldfrom typing import Callable, Any@dataclassclass Step:name: strtool: strargs: dictstatus: str = "pending" # pending / done / failedresult: Any = Noneretries: int = 0@dataclassclass TaskState:goal: strsteps: list[Step] = field(default_factory=list)memory: dict = field(default_factory=dict)MAX_RETRIES = 2def run_agent(state: TaskState, planner, tools: dict[str, Callable], store):if not state.steps:state.steps = planner(state.goal, state.memory)store.save(state)for step in state.steps:if step.status == "done":continuetry:step.result = tools[step.tool](**step.args)step.status = "done"state.memory[step.name] = step.resultexcept Exception as exc:step.retries += 1if step.retries > MAX_RETRIES:step.status = "failed"state.memory[step.name] = f"error: {exc}"store.save(state) # write a checkpoint after each stepreturn state
This code reflects three design principles. First, a checkpoint is written after each step is executed, so after an interruption, execution only needs to continue from unfinished steps. Second, failed retries have an upper limit, avoiding infinite loops that consume cost. Third, tools are called through a registry, so new capabilities can be added without modifying the control logic.
A production system also needs timeout control, concurrency limits, permission checks, and audit logs. Tools that perform write operations should require human confirmation by default.
5. Evaluation System: Do Not Look Only at the Final Answer
Long-horizon agents should be evaluated in layers.
- Task completion rate: Count the end-to-end success ratio on an annotated task set. The task set should cover both common scenarios and edge cases; too small a sample makes conclusions unstable.
- Step quality: Check whether tool selection and parameters are correct at each step. Many final failures originate from a parameter error in an intermediate step, and looking only at the result masks such problems.
- Cost and latency: Record the number of model calls, number of tool calls, total time, and resource consumption for each task. If looped calls occur, cost can quickly amplify.
- Safety metrics: Count unauthorized calls, sensitive data leakage, and write operations without confirmation. Such metrics should be set as hard thresholds and should not participate in weighted averages.
After each change to the model version or prompt, run regression tests on the same task set and keep version records to help locate the cause of performance degradation.
6. Risks and Controls
Multimodal inputs may contain personal information, which needs to be redacted during preprocessing. Logs should not store raw images or audio. During long-horizon tasks, the model may continue advancing on a wrong premise, so human checkpoints should be set before key decisions. Tool permissions should follow the principle of least privilege, exposing only the interfaces required by the task. Before model outputs enter downstream systems, they must undergo structural validation and must not be executed directly as trusted instructions.
7. Implementation Recommendations
- First validate the complete path for a single modality or a single tool on a small set of tasks, then expand to complex scenarios.
- Define clear completion conditions for each task, and avoid letting the model judge “completed” on its own.
- Design state, memory, and logs as queryable structured data to facilitate monitoring and review.
- Build a regression test set and continuously add online failure cases to it.
- Write cost limits into system configuration, and automatically degrade or pause tasks when limits are exceeded.
The combination of native multimodal models and long-horizon agents is moving large models from assisted Q&A toward an execution role capable of handling complete workflows. The focus for engineering teams is to organize planning, memory, tools, evaluation, and risk control into a reliable system; a single model’s metrics are only one input.
