Visual Reasoning and Cost Practices for Medium-Sized Multimodal Models

1. Why Medium-Sized Multimodal Models Are Needed

When deploying generative AI applications, enterprises typically face a practical contradiction: highly capable models can handle complex reasoning and long-chain tasks, but they incur higher API costs, response latency, and resource consumption. Conversely, lightweight models are cheaper and faster but often fail to stably comprehend technical drawings, complex interfaces, and cross-modal contexts.

Medium-sized multimodal models fill this gap. Rather than pursuing top-tier performance across all general capabilities, they focus on enhancing several high-frequency enterprise tasks:

1. Comprehending screenshots, tables, flowcharts, and technical drawings.
2. Executing UI operations based on natural language instructions.
3. Extracting fields from unstructured documents and completing form entries.
4. Generating well-structured, natural-sounding technical content.
5. Performing multi-step reasoning while maintaining controllable costs and response speeds.

From an engineering perspective, a medium-sized model is not simply a scaled-down version of a top-tier model. A robust application architecture should optimize task decomposition, context length, visual input quality, action permissions, and failure recovery mechanisms around specific workflows. Only then can its price-to-performance ratio translate into tangible business value.

2. Core Capability 1: Visual Reasoning Beyond Image Recognition

Traditional image recognition answers “what is in the image,” whereas visual reasoning further determines “what is the relationship between these elements,” “what should be done next,” and “whether the current state complies with the rules.”

For example, an equipment layout diagram may simultaneously contain numbering, dimensions, connection relationships, and safety markers. The model must not only read the text but also understand graphic positions, routing directions, and the correspondence between local annotations and the overall structure. Such tasks generally involve the following layers:

1. Visual Content Parsing

The model first identifies text regions, tables, buttons, icons, lines, and hierarchical structures within the page. The input clarity, scaling ratio, and page cropping method will all impact the results.

If an excessively long page is directly compressed into a single low-resolution image, fine text and control boundaries are easily lost. A more reliable approach is to partition the page first and then process the title, content, table, and action areas separately.

2. Spatial Relationship Understanding

The model needs to determine the relative positions of objects, such as which header a specific field falls under, which nodes a connecting line links, and which parameter a warning icon corresponds to.

Spatial relationship understanding is the foundation for form filling, drawing interpretation, and UI operation. Relying solely on text recognition results without coordinate information often makes subsequent automated actions impossible.

3. Rule and Semantic Reasoning

After recognizing the content, the model must make judgments based on task rules. For example:

• Whether the total amount in a table matches the line items.
• Whether there are dimension conflicts on a drawing.
• Whether the current page has reached the target state.
• Whether a button is disabled because preconditions are unmet.
• Whether there are logical contradictions among multiple fields.

Therefore, when evaluating visual models, one should not only look at text recognition accuracy but also test relationship understanding, rule validation, and anomaly detection capabilities.

3. Core Capability 2: Closed-Loop Control for Computer Operations

Using a model to operate a computer interface is essentially a closed-loop system of “observe–plan–execute–verify,” rather than a one-time generation of a string of click coordinates.

A reliable operation flow can be abstracted as follows:

Input task objective
  ↓
Capture current page state
  ↓
Identify actionable elements
  ↓
Generate next action
  ↓
Execute restricted action
  ↓
Re-read page state
  ↓
Verify action result
  ├─ Success: Proceed to next step
  ├─ Recoverable failure: Adjust strategy and retry
  └─ High-risk or unrecoverable failure: Escalate to human

1. Observation Phase

The system should provide the model with the current screenshot, page structure, action history, and task objective. If the page changes frequently, a timestamp or state ID should be attached to prevent the model from making decisions based on stale views.

2. Planning Phase

The planning result should not merely be “click somewhere.” It should include the action purpose, target element, expected result, and risk level. For example:

Action type: Fill field
Target element: Contact input box
Input content: Approved contact name
Expected result: Field displays the full name
Risk level: Low
Failure handling: Relocate the input field, retry at most once

This structured planning facilitates logging, permission validation, and troubleshooting.

3. Execution Phase

The execution layer must be decoupled from the model layer. The model proposes action suggestions, while the actual operation component determines if the action is permitted. For actions involving data deletion, message sending, order submission, permission modification, or sensitive file uploads, explicit human confirmation checkpoints must be established.

4. Verification Phase

After an action is executed, the system must re-observe the page and confirm the result. Success cannot be assumed solely because a “click action was dispatched,” as the interface might fail to change due to network latency, popup blockers, insufficient permissions, or page refreshes.

Closed-loop verification significantly reduces the probability of consecutive misoperations and is the critical step in transitioning computer operation capabilities from demos to production environments.

4. Task Decomposition in Typical Enterprise Scenarios

1. Technical Document and Drawing Interpretation

For complex technical materials, the task can be decomposed into five steps: page classification, local recognition, field extraction, relationship analysis, and result aggregation.

First, distinguish between covers, tables of contents, parameter tables, structural diagrams, and instruction pages. Next, process high-value pages at high resolution. Finally, bind the model’s extracted results to their original page locations to facilitate human review.

Output results should ideally include source localization rather than returning an unverifiable conclusion. For instance, each field should record the page number, region coordinates, original text snippet, and confidence status. For critical data like dimensions, amounts, and dates, deterministic validation rules should be applied.

2. Form Filling and Data Entry

Form automation appears simple but is highly susceptible to ambiguous field meanings, format restrictions, and page state changes. The following workflow is recommended:

• Read all fields and mandatory rules first.
• Map business data to page fields.
• Format dates, IDs, and amounts appropriately.
• Read back entries item by item after filling.
• Generate a difference list before submission.
• High-risk forms must be submitted after human confirmation.

For batch entry tasks, the medium-sized model can handle field comprehension and anomaly handling, while fixed-format conversions, numerical calculations, and mandatory checks should be handled by deterministic programs. This utilizes the model’s semantic capabilities while preventing all steps from becoming unpredictable generation tasks.

3. Proposal and Technical Content Drafting

Medium-sized models can handle material summarization, chapter planning, draft generation, and language optimization, but factual content must originate from authorized internal knowledge sources.

A robust generation workflow is:

1. Retrieve materials relevant to the task.
2. Extract facts, constraints, and citation locations.
3. Generate a chapter structure.
4. Draft content chapter by chapter.
5. Check terminology, numbers, and logical consistency.
6. Submit to the responsible person for review and finalization.

If the model is directly asked to write a complete proposal without source materials, the output may be generic or factually incorrect. Enterprises should position the model as an information organization and expression component, not as an unverified source of facts.

4. Video Content Assisted Editing

Visual models can be used to identify scene changes, subtitle regions, visual anomalies, and content themes, as well as provide editing suggestions based on text requirements. However, actual video transcoding, cropping, and exporting should be performed by dedicated processing modules.

The model output can adopt a structure of time intervals plus editing intents:

Start time: 00:02:10
End time: 00:02:28
Processing suggestion: Keep the main explanation, remove the preceding pause
Subtitle check: One terminology inconsistency found
Human review: Required

This approach is easier to audit and roll back than having the model directly control the entire editing process.

5. Routing Between Different Model Scales

Enterprises generally should not use a single model to handle all requests. A more rational approach is to establish a layered routing mechanism:

• Lightweight models: Handle classification, simple extraction, format conversion, and low-risk Q&A.
• Medium-sized multimodal models: Handle visual comprehension, UI operations, document analysis, and moderately complex reasoning.
• High-capability models: Handle high-risk decisions, complex planning, difficult tasks, and failure fallbacks.
• Deterministic programs: Handle calculations, validations, permission controls, and data writing.

The router can select the processing path based on input type, task risk, historical failure rate, and latency requirements. The schematic logic is as follows:

If the task only involves fixed-format conversion:
    Use deterministic programs
Else if the task is simple and contains no images:
    Use lightweight models
Else if the task involves screenshots, tables, or UI operations:
    Use medium-sized multimodal models
Else if the task is highly complex or the medium model fails consecutively:
    Route to high-capability models or human review

The key here is not to achieve absolute correctness on the first try, but to provide escalation and de-escalation mechanisms. When a medium-sized model fails, it can pass the current state, executed actions, and failure reasons to the subsequent processing node, avoiding the need to start over.

6. Cost Considerations Beyond Single Call Pricing

Model costs are typically related to input volume, output volume, number of calls, and failure retries. Direct inference costs can be estimated using the following general formula:

Cost per call =
Input volume × Input unit price
+ Output volume × Output unit price
+ Visual processing fee
+ Other optional processing fees

However, the true cost in a production environment should also include:

Total cost =
Direct inference cost
+ Retry cost
+ Human review cost
+ Data processing and storage cost
+ Monitoring and auditing cost
+ Business losses from incorrect operations

For example, a model with a low single-call price might require multiple retries to complete a task correctly, making its total cost potentially higher than a model with a higher unit price but a more stable success rate. Therefore, model selection should focus on the “average cost per successfully completed task” rather than just comparing quotes per unit of input or output.

A more practical metric can be defined as:

Effective task cost =
Total processing cost during the test period
÷ Number of tasks that succeeded and passed quality inspection

For UI automation, the average number of actions should also be tracked. If a model can complete the same task in fewer steps, it usually implies fewer API calls, shorter wait times, and a lower probability of failure.

7. Building an Evaluation System for Real Workflows

Public benchmarks only reflect partial capabilities; enterprises still need to build their own task sets. It is recommended to evaluate medium-sized multimodal models across the following dimensions:

1. Task Success Rate

Determine whether the model fully achieved the objective, rather than just checking if intermediate outputs look reasonable. Form tasks should be judged on correct fields and successful submission; document tasks should be judged on accurate key fields and traceable sources.

2. Visual Localization Accuracy

Record whether target elements are correctly identified, whether coordinate deviations are acceptable, and whether the model remains stable during page scaling, theme changes, and popup appearances.

3. Reasoning and Expression Quality

Check whether conclusions are evidence-based, technical terms are accurate, paragraph structures are clear, and whether the model can explicitly distinguish between facts, inferences, and uncertain content.

4. Latency and Throughput

Measure the time to first result, total task duration, and throughput variations under concurrent loads. For computer operation tasks, end-to-end duration is more informative than single model response time.

5. Cost Metrics

In addition to the single-call fee, track the average cost per successful task, the proportion of retry costs, and the human intervention rate.

6. Security and Controllability

Test the model’s behavior against ambiguous instructions, unauthorized requests, malicious page text, and sensitive data. Specifically, verify that it does not mistakenly treat untrusted page content as system instructions.

The evaluation set should contain normal, boundary, and adversarial samples, and regularly incorporate new failure cases from the production environment. Only continuous regression testing can determine if a model upgrade genuinely improves business outcomes.

8. Security Design as a Prerequisite for Computer Operations

Visual operation systems interact with real pages and business data, making their risks higher than standard text Q&A. Production deployments must implement at least the following controls:

1. Principle of Least Privilege

Operation accounts should only be granted the permissions necessary for the current task. Capabilities like deletion, export, payment, and permission modification should not be enabled by default.

2. Action Whitelisting

The execution layer should only permit predefined action types and restrict accessible pages, fillable fields, and uploadable file scopes.

3. Sensitive Operation Confirmation

Actions involving funds, external communications, data deletion, contract submission, and permission adjustments must be explicitly confirmed by a human.

4. Comprehensive Auditing

Retain task inputs, page states, model plans, actual actions, execution results, and human confirmation records. Sensitive information in logs should be masked and access-controlled.

5. Prevention of Page Instruction Interference

Text within web pages, documents, or images may contain prompts designed to hijack the model into changing its objective. The system must clearly distinguish between trusted task instructions and page data, disallowing page content from directly escalating privileges or modifying security policies.

6. Failure Circuit Breaker

Upon consecutive localization failures, abnormal page states, or unverifiable operation results, the system should immediately halt automated execution rather than retrying indefinitely.

9. Recommended Engineering Architecture

A maintainable multimodal automation system typically consists of the following modules:

• Task Ingestion Layer: Receives user objectives, files, and permission details.
• Input Preprocessing Layer: Handles image cropping, page partitioning, text cleaning, and sensitive information identification.
• Model Routing Layer: Selects the appropriate capability level based on task complexity.
• Planning Layer: Decomposes objectives into executable steps.
• Execution Layer: Invokes controlled UI actions or business APIs.
• Validation Layer: Verifies results via rules, read-backs, and cross-checks.
• Security Layer: Enforces permission checks, action interception, and human confirmation.
• Observability Layer: Records latency, costs, success rates, and failure reasons.
• Data Layer: Utilizes object storage, logging services, and databases to retain necessary records.

Enterprises can also leverage general cloud computing resources, object storage, container platforms, and logging services provided by Baidu AI Cloud to build a scalable runtime environment. Note that model capabilities and infrastructure capabilities should remain decoupled to prevent business processes from becoming overly dependent on a single component.

10. Implementation Strategy: From Assistive to Autonomous

Medium-sized multimodal models are well-suited for high-frequency, repetitive workflows that require visual comprehension but carry controllable risks. Actual implementation should not aim for unattended automation from day one; instead, it should be phased:

Phase 1 adopts an assistive mode, where the model only outputs analysis results and action suggestions for human execution.
Phase 2 transitions to a semi-automatic mode, where low-risk actions are automated, and critical steps require human confirmation.
Phase 3 achieves limited automated execution in fully evaluated, fixed workflows.

Throughout this process, enterprises should continuously monitor four core metrics: task success rate, average completion time, effective task cost, and human intervention rate. If model costs decrease but error rates or human review costs rise significantly, the overall solution has not realized true value.

The competitive advantage of medium-sized multimodal models lies not in comprehensively replacing high-capability models, but in covering a large volume of real workflows at a more reasonable cost. Through task layering, closed-loop operations, dynamic routing, rigorous evaluation, and strict security controls, enterprises can transform visual reasoning and computer operation capabilities into stable, auditable, and scalable production capabilities.