Multimodal LLMs Enter the Agent Era: Vision, Computer Use, and Long-Horizon Tasks

    As general text capabilities converge across open and closed models, LLM competition is shifting toward multimodal perception, computer-use execution, and long-horizon agent reliability. This article examines the technical architecture and engineering challenges behind these three capability lines, from vision-language model design and screen-action loops to context management, error accumulation, and evaluation metrics such as task success rate and hallucination rate.

    Capability Leap: From Conversation to Seeing, Acting, and Persisting

    1. Background: LLM Competition Enters a Capability-Completion Phase

    Over the past year, the focus of competition among large language models has been quietly shifting. In the early period, the industry generally used parameter scale, training data volume, and benchmark scores as the yardsticks for model strength, and comparisons between models looked more like an intelligence contest. However, as open-source and closed-source models gradually converged in general text capabilities, the marginal returns from a pure parameter arms race began to decline, and the industry’s attention turned to whether models can actually complete complex real-world tasks.

    A new flagship LLM recently released by a Chinese vendor reflects this trend: its parameter scale increased from roughly 300B in the previous generation to 770B, and it introduced visual understanding as well as support for Computer Use for the first time. Notably, the vendor also acknowledged that its model still lags behind top international closed-source models. This restrained statement actually adds credibility to its self-positioning as being among the strongest open-source models, because the real gap is often not in the parameter table but in task success rates, hallucination rates, and other less flattering numbers.

    Using this as an entry point, this article systematically reviews the technical substance of three capability lines: multimodal understanding, agent execution, and long-horizon tasks. It also discusses key challenges in engineering deployment.

    2. Visual Understanding: Enabling Models to “See” the World

    2.1 Why Text-Only Models Are Not Enough

    The input boundary of a text-only LLM determines that it can only understand information that humans have already translated into text. However, a large amount of information in real tasks exists naturally as pixels: interface screenshots, charts, design drafts, and surveillance images. If a model cannot directly understand images, pre-processing pipelines such as OCR and object detection are needed to transcribe visual information into text. This not only introduces cascading errors but also loses a great deal of spatial layout and contextual information.

    2.2 Mainstream Vision-Language Model Architectures

    The industry generally recognizes two types of visual understanding approaches:

    • Dual-tower architecture: Images are processed by a vision encoder, usually based on a vision Transformer, to extract features, while text is encoded by a language model. The two are aligned through contrastive learning or cross-attention. This approach has high retrieval efficiency and is suitable for image-text matching tasks.
    • Unified architecture: Images are split into patches and mapped into vector sequences that are structurally consistent with text tokens, then fed directly into the language model backbone for end-to-end training. This approach has strong generation capabilities and is the mainstream route for current flagship multimodal models.

    A key engineering difficulty is that the number of image tokens is large. A high-resolution screenshot may produce thousands of tokens, significantly increasing inference cost and context usage. Common optimization methods include dynamic resolution splitting, visual token compression, and hierarchical feature fusion.

    1. # Illustration: unified serialization of multimodal input (pseudocode)
    2. image_patches = vision_encoder(image) # Visual feature extraction
    3. image_tokens = projection(image_patches) # Align to LLM embedding space
    4. input_seq = concat(system_tokens, image_tokens, text_tokens)
    5. output = llm(input_seq) # End-to-end inference

    3. Computer Use: Enabling Models to “Act”

    3.1 The Leap from “Saying” to “Doing”

    Computer-use capability is a key step in evolving LLMs from assistants into executors. The basic paradigm is as follows: the model receives a screen screenshot as visual input, combines it with task instructions, and outputs structured action commands, such as mouse click coordinates, keyboard input, and page scrolling. An executor carries these out in a real desktop environment and feeds the new screen state back to the model, forming an observe-decide-execute loop.

    1. # Illustration: control loop for a Computer Use agent (pseudocode)
    2. while not task_done:
    3. screenshot = capture_screen()
    4. action = model.predict(instruction, history, screenshot)
    5. # action is like {"type": "click", "x": 512, "y": 384}
    6. execute(action)
    7. sleep(interval) # Wait for the interface to respond

    3.2 Key Technical Challenges

    This direction may look intuitive, but it is highly challenging in engineering:

    • Interface element grounding: The model needs to precisely locate clickable controls in pixel space. A coordinate offset of a few pixels may cause it to click the wrong target. This requires the vision encoder to have high-resolution detail perception.
    • Operation semantic generalization: Interface layouts vary greatly across applications. The model must understand functional semantics such as “save button” and “dropdown menu” rather than memorizing coordinate templates.
    • Error recovery: Real environments contain uncontrollable factors such as pop-ups, loading delays, and state changes. The model needs to observe execution results, identify failures, and retry.
    • Safety boundaries: Letting a model operate a real computer involves permission control. It must run in a sandbox environment, and high-risk actions such as deleting files or sending messages must be subject to human confirmation mechanisms.

    4. Long-Horizon Agents: Extending Tasks from Minutes to Hours

    4.1 Why “Long” Is a Qualitative Difference

    Traditional conversational tasks can be completed within a few rounds of interaction, while real business tasks, such as preparing an industry research report or completing data migration across multiple systems, often require dozens or even hundreds of steps. When task duration extends from minutes to hours, a series of problems that do not exist in short tasks emerge:

    • Context management: Step sequences may exceed the model’s context window. Mechanisms such as memory compression, periodic summarization, and external storage are needed so the model can “remember what it has done.”
    • Error accumulation: Even if the single-step success rate reaches 95%, the overall success rate after fifty steps decays to less than 8%. Therefore, long-horizon tasks impose extremely strict requirements on single-step reliability. This is also why task success rate reflects a model’s true level better than demo videos.
    • Goal drift: During long-chain execution, the model can easily deviate from the initial goal. A planning module is needed to check progress and correct direction periodically.

    4.2 Mainstream Architecture Patterns

    Common long-horizon agent architectures in the industry contain several core components: a planner responsible for decomposing tasks into subgoals; an executor that calls tools or operates interfaces to complete individual actions; a memory module that maintains short-term working memory and a long-term experience store; and a reflection module that reviews failures or stage completions and adjusts strategy. This planning-execution-memory-reflection loop is the common skeleton of current agent frameworks.

    5. How to Evaluate These Models Objectively

    For teams that want to adopt such capabilities, it is advisable to build an evaluation system that goes beyond leaderboard scores:

    1. Task success rate: Measure end-to-end completion rate in a controlled environment rather than only single-step action accuracy.
    2. Hallucination rate: Measure the proportion of factual errors in visual question answering and execution.
    3. Step efficiency: Measure the average number of steps and time required to complete a task, reflecting planning quality.
    4. Cost metrics: Measure inference overhead caused by visual tokens and the cumulative invocation cost of long-horizon tasks.
    5. Safety and compliance: Measure the interception rate of high-risk operations and the completeness of audit logs.

    6. Conclusion

    A parameter scale of 770B is certainly eye-catching, but the more important signal is the simultaneous completion of three capability lines: visual understanding lets flagship models “see,” Computer Use lets them “act,” and long-horizon agents extend tasks from minutes to hours. For open-source models from China, the path from “comparable” to “dependable” depends precisely on these engineering details that require long-term refinement. For enterprises, model selection should rely less on parameter tables and more on real-task evaluation. Using less flattering numbers such as task success rate and hallucination rate as the basis for decisions is the way to find a truly dependable technology foundation.