Visual Interaction for AI Agents: A New Paradigm for Office and Data Analysis

From Prompt Writing to “Showing It”: A Shift in Interaction Paradigms

For the past two years, the core skill for making AI complete complex tasks has been prompt writing. Users had to turn vague requirements into structured text, repeatedly refine wording, and hope the model understood their intent. This approach has a natural limitation: many human needs are inherently spatial and visual. When a user wants to say “the horizontal axis in this chart is too crowded,” “move this block to the upper-right corner,” or “highlight this row in red,” describing it in words can be more cumbersome than directly marking it up.

A new generation of agent tools is trying to cross this gap. They introduce whiteboard-like interaction surfaces: users can circle areas, annotate, connect elements, and indicate priorities on a canvas. The agent then interprets the spatial semantics of these visual marks to infer what the user wants changed, added, or adjusted. This shifts interaction from pure text to multimodal fusion: text, graphics, and spatial layout.

This is not just a bundle of features. Text prompts are linear and discrete, while visual annotations naturally carry structural relationships and intent weighting. A circled region, an arrow, or a highlighted area silently signals where the focus is. Whether an agent can understand these signals is a key differentiator from traditional conversational AI.

Technical Core of Three Complementary Capabilities

Around this interaction paradigm, three complementary technical capabilities can be abstracted. Understanding their implementation logic is more valuable than memorizing any single product.

Visual Whiteboard: Multimodal Spatial Understanding

The core challenge of a visual whiteboard is “understanding what the user has marked.” The system needs several layers of parsing:

1. Visual element recognition: distinguish whether the user drew a text box, arrow, circled region, or handwritten annotation;
2. Spatial relationship inference: determine which elements belong to the same logical group, what an arrow points to, and what a circle encloses;
3. Intent mapping: translate “circle this paragraph + write ‘too verbose’ next to it” into an executable instruction such as “compress this paragraph.”

Architecturally, this usually relies on a multimodal large model to jointly understand canvas screenshots, plus an operation-intent parser that converts visual input into structured actions. An illustrative processing flow:

[Canvas screenshot] + [handwritten/text annotations]
        ↓
  Multimodal understanding model (joint image + text encoding)
        ↓
  Structured visual elements and spatial relationships
        ↓
  Intent parsing: {target region, operation type, constraint parameters}
        ↓
  Execution engine (edit documents / modify spreadsheets / recompute data)

Spreadsheet Generation and Editing: Structured Output

Allowing an agent to “directly create and edit spreadsheets” depends on structured generation under constraints. Large language models produce text sequences, but spreadsheets require strict two-dimensional structure, cell references, formula syntax, and formatting rules. Key engineering issues include:

• Formula correctness: generated formulas must be valid both syntactically and semantically, with accurate referenced ranges;
• Data consistency: after adding or modifying data, related aggregations and charts must update accordingly;
• Format fidelity: features such as conditional formatting, data validation, and frozen panes must not be broken.

In practice, models usually do not write an entire spreadsheet from scratch. Instead, they output an intermediate representation—such as a structured sequence of cell operations or a spreadsheet DSL—which the execution layer applies to the actual spreadsheet model. This improves controllability and makes rollback easier when errors occur.

Data Analysis and Visualization: From Natural Language to Insights

The third capability is analyzing data and generating visualization reports. Its typical pipeline converts a natural-language question into executable queries and charts:

Natural-language question ("How did quarter-over-quarter growth perform across regions in the past three quarters?")
        ↓
  Semantic parsing → structured query (e.g., SQL / aggregation expressions)
        ↓
  Execute on the data model to obtain a result set
        ↓
  Select an appropriate chart type (line / bar / heatmap)
        ↓
  Generate a visualization report with conclusions

In this pipeline, chart type selection and conclusion phrasing are often underestimated but especially important. An appropriate visualization is not chosen at random; it should match data dimensions (time, category, value) and analysis goals (trend, comparison, distribution). A mature system encodes domain experience—such as when to use a line chart versus a bar chart—into recommendation logic rather than leaving everything to model improvisation.

Does This Mean Traditional Office Software Will Be Obsolete?

Online jokes about “finally not needing to learn office software” are lively, but from a technical perspective, replacement is overstated. A more accurate judgment is “divergence of interaction methods,” not the overall disappearance of tools.

Two types of needs should be distinguished:

• Exploratory, temporary work: quickly analyzing a dataset, sketching a workflow, or producing a first draft. These tasks prioritize speed and tolerate errors; the agent’s “one sentence or one gesture gets a result” approach is attractive.
• Production and collaborative work: tasks requiring precise control over every cell, strict version management, complex multi-person collaboration, and stable macros or plugin ecosystems. These scenarios demand determinism and controllability, which are weaknesses of generative AI.

The real moat of spreadsheet software is not merely “doing calculations.” It is the industry standards, file-format compatibility, ecosystem plugins, and team collaboration workflows built around it. Agents can act as powerful copilots that handle tedious operations, but in the short term they are unlikely to fully replace this mature system. The rational positioning is: agents lower the usage barrier and let more people access data-analysis capabilities; professional tools continue to hold value in scenarios requiring extreme control.

Developer Perspective: Building Similar Agent Capabilities

To implement “visual interaction + document/data generation” in your own product, you can refer to the following layered architecture:

1. Multimodal input layer: receives visual signals such as screenshots, handwriting, and annotations, and encodes them together with text;
2. Intent understanding layer: parses visual and language input into structured action intents;
3. Tool execution layer: maps intents to concrete capabilities such as document editing, spreadsheet operations, and query execution. This layer should be deterministic and testable, rather than relying on model improvisation;
4. Validation and rollback layer: checks generated results for syntax/semantics, and preserves a recoverable operation history.

Building this type of system has several common infrastructure needs. General cloud capabilities can support them, such as object storage for saving canvas and document snapshots, message queues for decoupling asynchronous analysis tasks, container platforms for elastic scaling of inference and execution services, and logging and monitoring services for observability. Providers such as Baidu AI Cloud can offer general-purpose capability categories that serve as a reference technical foundation for such systems. The key is to make “multimodal understanding + deterministic execution” solid.

A practical recommendation: do not rely entirely on the model for correctness. For tasks with clear right/wrong answers, such as spreadsheet formulas and data queries, combining “model-generated candidates + rule validation + execution verification” is necessary to balance flexibility and reliability.

Unavoidable Real-World Challenges

Any new technical paradigm faces inherent limitations, and intelligent office capabilities are no exception:

• Reliability: model-generated formulas or queries may contain subtle errors; users need to maintain a review mindset rather than blindly trust outputs;
• Data security: analysis often involves sensitive internal enterprise data; protection and compliance across transmission, inference, and storage are hard constraints for deployment;
• Controllability boundaries: the more creative and fine-grained a task is, the more automation should be limited, returning control to the user;
• Hallucination and misleading output: when presenting data conclusions, models may “fabricate” plausible but incorrect interpretations, so report-type outputs especially require human review.

Conclusion: Boundaries and Directions of the New Paradigm

From “writing prompts” to “showing it,” AI agents are moving from understanding text to understanding intent and space. Visual whiteboards, spreadsheet generation, and data analysis jointly outline a direction that lowers the barrier to digital tools—allowing more people to express needs and obtain insights without specialized training.

For developers and decision-makers, the real value is not chasing the narrative of “replacing a certain office tool,” but identifying the boundaries of what agents do well and what they do not: assign repetitive and exploratory operations to agents, leave production tasks requiring precise control to mature tools, and architecturally insist on “model understanding + deterministic execution + result validation.” Understanding this technical logic will help you judge which trends represent real needs and which are merely marketing rhetoric.