Three Technical Paths in the LLM Paradigm Shift: Visual Encoding, Multi-Agent Collaboration, and Test-Time Scaling

1. Background: The Shifting Focus of Competition

The development of Large Language Models (LLMs) is no longer solely dependent on parameter scale. As foundational capabilities converge, what truly sets models apart is their structural efficiency when dealing with complex visual inputs, long-horizon task planning, and high-difficulty reasoning.

Currently, three trends in the industry deserve close attention:

1. Visual encoders are no longer just “preprocessing modules.” They are beginning to possess stronger sequence modeling and causal induction capabilities.
2. Single-model inference is being supplemented by multi-agent collaboration. Complex tasks are being decomposed into parallel subtasks.
3. Compute allocation during inference is changing. Models can now dynamically scale computation at test time, rather than relying solely on fixed parameters from the training phase.

These three trends correspond to three distinct paths: visual understanding, system engineering, and reasoning algorithms. Let’s explore each one in detail.

---

2. Architectural Reconstruction of Visual Encoders: From Raster Scanning to Selective Reading

2.1 Structural Limitations of Traditional Vision-Language Models

In common Vision-Language Models (VLMs), an image is typically divided into fixed-size patches, flattened into a one-dimensional sequence in a left-to-right, top-to-bottom order, and fed into the subsequent language model. While simple to implement, this approach presents significant issues when processing structured visual content like documents, tables, and charts.

First, spatial logic is disrupted. The reading order of complex layouts is rarely a simple line-by-line scan. Multi-column layouts, nested tables, and cross-region references require the model to establish non-linear associations between different spatial positions. Forced rasterization misleadingly implies that spatial adjacency equates to semantic adjacency.

Second, positional bias is introduced. A fixed scanning direction causes the model to implicitly learn false rules during training, such as “the top-left corner is more important” or “regions appearing first are more critical.” This bias might not severely affect natural images, but in document understanding, it directly leads to critical information being overlooked.

Third, computational efficiency is low. All image patches are converted into visual tokens and participate in subsequent computations. As resolution increases, the number of tokens grows quadratically, but a large portion of these tokens corresponds to backgrounds, whitespace, or repetitive structures, contributing very little to the final understanding.

2.2 New Design Approach: Giving Visual Encoders a Concept of “Reading Order”

A more rational approach is to stop forcing the model to read visual information in a fixed spatial order. Instead, let the model decide “where to look next.”

This concept draws inspiration from the causal attention mechanism in language models. A language model generates words sequentially over time, where each word can only attend to preceding words. If visual understanding is similarly treated as a “sequential decision-making” process, the model can decide which region to focus on next at each step, based on the content it has already read.

This design requires two conditions:

• A set of learnable query vectors must exist to act as a “reading pointer.”
• The attention mask must allow query vectors to access image patches step-by-step while preventing arbitrary mutual access between image patches.

Specifically, visual tokens and query vectors can be concatenated and input into the encoder. Suppose an image yields m visual tokens after visual tokenization, and n query vectors are initialized simultaneously. The attention mask must satisfy the following constraints:

Between image patches: Fully visible, allowing spatial information to mix thoroughly
Between image patches and queries: Patches cannot see queries; queries can see all patches
Between queries: Can only see themselves and preceding queries, maintaining causal order

The corresponding mask structure can be illustrated as follows:

┌─────────────────┬─────────────────┐
│   Full (Patches)│   Zero Matrix   │
├─────────────────┼─────────────────┤
│   Full (Queries)│  Lower Triangular│
└─────────────────┴─────────────────┘

The significance of this structure is that image patches can fully interact to encode spatial relationships, while query vectors read image information sequentially without leaking information backward to subsequent steps.

2.3 Practical Benefits

After adopting this architecture, the model’s performance in processing complex documents improves significantly. An observable metric is the score on document understanding benchmarks. Under the same volume of training data and model scale, introducing the “selective reading” mechanism yields superior parsing capabilities for multi-column layouts, table structures, and cross-page references compared to traditional rasterization schemes.

More importantly, because the number of query vectors can be much smaller than the number of visual tokens, the subsequent language model does not need to process all image tokens directly. This significantly reduces computational overhead in long-document parsing and batch processing scenarios.

---

3. Multi-Agent Collaboration: From Monolithic Reasoning to Parallel Task Orchestration

3.1 Bottlenecks of Monolithic Models

When processing complex tasks, a monolithic model typically generates a complete solution sequentially. For example, in a task involving information retrieval, format conversion, content validation, and result aggregation, the model must complete each step in turn. Sequential execution introduces several issues:

• A failure in a single step directly impacts subsequent steps.
• Intermediate results cannot be reused between tasks.
• In scenarios requiring multiple specialized capabilities, a single context window struggles to cover them all simultaneously.

3.2 Core Idea of Parallel Scheduling

Recent practices indicate that decomposing a complex task into multiple subtasks and assigning them to different agents for execution can significantly reduce latency and improve stability. The key is that the decomposition and scheduling strategy must be automated, without relying on manually predefined workflows.

A viable framework is Parallel Agent Reinforcement Learning. At its core, it trains a scheduling policy responsible for:

• Analyzing task objectives;
• Determining the required number of sub-agents;
• Assigning responsibilities and inputs to each sub-agent;
• Deciding the dependencies between sub-agents;
• Merging the outputs of all sub-agents into the final result.

Under this framework, sub-agents can run in parallel. For long-chain tasks, the latency is no longer the sum of all steps’ execution times, but rather approximates the execution time of the longest subtask path plus the merging overhead.

3.3 Engineering Considerations for Implementation

Parallelization is not a silver bullet. The granularity of decomposition must match task complexity. Decomposing too finely will cause communication and merging costs to outweigh execution benefits, while decomposing too coarsely will fail to leverage parallel advantages fully.

In practical engineering, a two-level scheduling approach is typically adopted: the upper level handles task decomposition and final merging, while the lower level executes specific subtasks. Sub-agents should ideally maintain stateless interactions, passing necessary information via shared contexts or message queues.

Taking a complex data processing task as an example, a typical workflow might look like this:

Master Scheduler
├── Sub-agent A: Responsible for input parsing and format recognition
├── Sub-agent B: Responsible for content extraction (can run parallel to A)
├── Sub-agent C: Responsible for rule validation (depends on B's output)
└── Sub-agent D: Responsible for result generation (depends on A, C)

Here, A and B can start simultaneously, C begins after B completes, and D executes last. The overall latency is determined by max(A, B) + C + D, rather than the sum of four sequential steps.

In experimental environments, this approach can reduce the end-to-end latency of certain complex tasks by approximately 70% to 80%. However, this is contingent on the task itself being decomposable and the scheduler’s decomposition decisions being sufficiently accurate.

---

4. Test-Time Compute Scaling: “Thinking While Computing” During Inference

4.1 Rebalancing Training and Inference Compute

Historically, improving model capabilities relied primarily on investing more compute during the training phase. However, training is costly and time-consuming, and as parameter scales increase, marginal returns diminish. Recently, an alternative path has gained prominence: Test-Time Compute Scaling.

Its core idea is that the model no longer outputs an answer in a single pass during inference. Instead, it performs multiple internal reasoning steps before delivering the result. The more reasoning steps taken, the higher the compute consumed per response, but the quality of the answer may improve accordingly.

4.2 From Fixed Steps to Experience Accumulation

Early test-time scaling typically employed a fixed-step strategy, such as “think for three steps before answering.” While easy to implement, this approach fails to distinguish between simple and difficult questions. Simple questions are forced through unnecessary multi-step reasoning, wasting compute, while difficult questions suffer from insufficient steps, degrading response quality.

A better approach is Experience Accumulation Scaling. During inference, the model maintains an “intermediate conclusion pool.” At each step, it can read the conclusions generated by all preceding steps and decide whether further reasoning is necessary. When the model determines that existing information is sufficient to answer, it outputs the final answer.

This approach offers three advantages:

• Adaptability: Questions of varying difficulty can trigger reasoning chains of different lengths.
• Accumulability: Subsequent steps can directly reuse valid conclusions from preceding reasoning, avoiding redundant computation.
• Diagnosability: Intermediate reasoning steps can be recorded and audited, making it easier to locate error sources.

4.3 Performance on Reasoning Benchmarks

On public scientific reasoning and multiple-choice benchmarks, models employing test-time scaling show significant improvements over fixed-step reasoning. Some results are now approaching the average performance of professionally trained humans on similar question sets.

It is important to note that the true value of test-time scaling lies not in “looking good on benchmarks,” but in how it changes the way model capabilities are scaled. Developers no longer need to upgrade to a larger model to improve reasoning quality; they can trade adjusted inference budgets for better results. This is particularly meaningful in cost-sensitive production environments.

---

5. Relationship and Common Direction of the Three Paths

These three paths do not replace one another. Instead, they answer the same question from different layers: How can models become smarter, faster, and more controllable without relying solely on parameter growth?

The reconstruction of visual encoder architecture addresses “how to understand complex inputs more efficiently.” Multi-agent collaboration addresses “how to organize the execution of complex tasks more rationally.” Test-time scaling addresses “how to dynamically allocate computational resources during inference.”

They collectively point to a clear conclusion: LLM competition has shifted from a contest of parameter scale to a comprehensive battle of structural design, scheduling strategies, and inference efficiency.

For developers, the focus should not be on the benchmark numbers of any single technology, but on whether these methods can be combined to solve real-world business problems. For instance, a document understanding system can simultaneously benefit from a better visual encoder, task decomposition scheduling, and an adaptive inference budget. Truly competitive solutions are often the result of multiple technical routes landing collaboratively within specific scenarios.