From Prompts to Visual Interaction: A New Paradigm for AI Collaboration

1. An Easily Overlooked Interaction Revolution

Recent discussions around large models have mostly focused on their capability limits: whether they can write code, answer questions correctly, or scale to larger parameter sizes. However, what really determines whether a technology can enter everyday work is often not the model capability itself, but the way humans interact with it.

In traditional usage, the first step for users who want AI to complete a task is to write prompts. This may seem simple, but it is actually a hidden cognitive burden. Users must translate vague, spatial, or even unfinished requirements in their minds into linear, precise, and structured text instructions. Experienced users who are skilled at orchestrating complex requirements may manage this well, but for most ordinary professionals, questions such as “What kind of budget spreadsheet do I want?” or “How should I schedule the tasks in this project?” are difficult to express clearly in one sentence.

When AI begins to understand circles drawn on a whiteboard, dragged cards, or supplementary verbal explanations, the initiative of interaction quietly shifts from “humans adapting to machines” to “machines adapting to humans.” This is not merely a feature addition; it is a migration of the human-AI collaboration paradigm. Understanding the technical foundations behind this shift is more valuable than remembering a specific feature.

2. Why Visual Expression Can Be More Efficient Than Text Descriptions

Humans are naturally good at spatial and graphical thinking. In real meetings, teams often draw timelines on whiteboards, use sticky notes, and use arrows to show dependencies. These actions are intuitive, low-cost, and capable of carrying relational information that is difficult to express in text alone, such as which task comes first or which section needs more emphasis.

If AI can understand this familiar form of expression, a key change occurs: the cost of expression drops from “precise wording” to “rough gesturing.” Users can convey their intent even when requirements are still incomplete or in draft form. They can ask AI to build a framework first and then refine it step by step. This addresses the biggest pain point of traditional prompt engineering: it requires users to think clearly before taking action, which is often the hardest part of creative work.

Therefore, the value of such tools is not to replace human decision-making, but to reduce startup costs and shorten the distance from an idea to a first draft. Humans can focus on judgment and decision-making, while machines handle organizing, formatting, and structuring.

3. Multimodal Understanding: How AI Reads Human Gestures and Scribbles

To support the experience described above, a complete set of multimodal understanding capabilities is usually required. These typically include the following stages:

• Input parsing: Recognizing text, speech, hand-drawn shapes such as boxes, arrows, and circled areas on a whiteboard, as well as their positions.
• Spatial relationship inference: Mapping a circled area to a target to be modified, mapping an arrow to an order or dependency relationship, and mapping a group to a task module.
• Intent recognition: Combining context to infer whether the user wants to add, rearrange, refine, or delete content, even when the instruction is incomplete.
• Structured drafting: Converting the parsed result into a formal organizational form, such as a flowchart, timeline, Gantt chart, or modular task list.

The key technical challenge here is ambiguity handling. A user’s casual gesture or scribble often has unclear meaning. AI must balance between “proactive guessing” and “carefully asking for clarification.” Guessing too aggressively may lead to changes in the wrong direction, while being too conservative loses the benefit of requiring fewer prompts. A mature approach is usually to first produce a reversible draft and allow users to correct it by continuing to circle, drag, or edit, turning “explain everything at once” into “progressive co-creation.”

4. Structured Output: Turning Messy Ideas into Usable Results

The core of this capability is to convert unstructured input into structured output. For example, if a user places several scattered task notes on a whiteboard, AI needs to infer the order and dependencies among the tasks and generate a project timeline with dates, icons, and owners. When the user drags items to adjust the order, AI should also update the entire structure accordingly and maintain consistency.

From an engineering perspective, this process is similar to constraint satisfaction and template generation. The model must understand the user’s free expression while also following the internal structural rules of a specific output, such as a timeline, table, or flowchart. The intent behind this capability can be abstracted as follows:

Raw input (sketch / speech / sticky notes)
        │
        ▼
Intent parsing: identify objects, actions, relationships, and constraints
        │
        ▼
Structured intermediate representation (tasks, dependencies, dates, owners)
        │
        ▼
Render into target artifact (timeline / flowchart / table / presentation)

Furthermore, finalized whiteboard content should not be the endpoint. It should be exportable as documents, presentations, or synchronized into online collaboration tools that teams already use for further refinement. Transferability, editability, and traceability are key indicators of whether such tools can truly fit into real workflows, rather than locking content inside a single interface.

5. Natural-Language-Driven Spreadsheets and Data Analysis

If visual whiteboards are responsible for clarifying ideas, spreadsheet capabilities are responsible for calculating numbers and organizing tasks. The technical path can be divided into two main lines.

The first line is generating spreadsheet structures from natural language. Users only need to describe their requirements, such as: “Create a recruitment budget spreadsheet including compensation, recruiting channel costs, training expenses, and highlight overdue items.” The system can then automatically create worksheets, fields, formulas, dropdown options, and formatting. The essence of this capability is to translate language instructions into computable spreadsheet structures:

Requirement description
  → Extract entities (columns: budget item, amount, status)
  → Generate formulas (remaining balance = total budget - spent amount)
  → Apply formatting (conditional highlighting for overdue items)
  → Output editable spreadsheet

On this basis, precise cell-level modification is key to the collaboration experience. For example, when a user says “Add a budget balance column” or “Mark overdue tasks in red,” AI makes the corresponding changes and marks the modified areas for human review. This design of “small-step fast edits plus visible changes” allows users to retain control over the result and makes auditing and error correction easier. Output should also support export to common spreadsheet file formats so that it can connect with existing office workflows.

The second line is data analysis. More advanced capabilities can connect to enterprise data warehouses and business systems, allowing users to ask questions directly about real data. For example, “What are the customer acquisition cost trends by channel last quarter?” The system can generate a query, execute calculations, and produce a visualization report. A typical processing chain is as follows:

-- Example: translate a natural-language requirement into query logic (conceptual)
SELECT channel,
       SUM(cost) / SUM(new_users) AS cac,
       month
FROM   marketing_metrics
WHERE  quarter = 'Q_last'
GROUP  BY channel, month
ORDER  BY cac DESC;

From an engineering perspective, particular attention should be paid to permissions and data security when connecting to production data. The principle of least privilege should be followed, sensitive fields should be masked, query operations should be auditable and rollbackable, and safeguards should be in place to prevent an incorrect automatically generated query from affecting real business systems.

6. Engineering Foundations Supporting These Experiences

For the capabilities above to work reliably, a strong model alone is not enough. A complete engineering system is also required:

• Real-time collaboration: When multiple people edit the same whiteboard or spreadsheet at the same time, a conflict-merging mechanism, such as operational transformation or conflict-free replicated data types, is needed to ensure that each person’s changes are correctly synchronized and do not overwrite one another.
• Version management and traceability: Every AI modification should leave a record, allowing users to view history at any time and roll back to a previous version. This is critical for building trust.
• Data connectivity and governance: For data analysis, standardized data access methods, unified permission models, and query auditing capabilities are required.
• Export and interoperability: Outputs should be exportable in common formats and compatible with tools already used by teams, avoiding the creation of new data silos.

In cloud-based implementations, such capabilities are often built using general cloud product categories. For example, object storage can be used to store raw whiteboard and spreadsheet assets, message queues can drive event synchronization for multi-user collaboration, container platforms can host model inference and business logic, and log services and monitoring alerts can help ensure system stability. Cloud platforms such as Baidu AI Cloud provide these types of general capabilities, allowing upper-layer applications to focus on interaction and business innovation without rebuilding common infrastructure from scratch.

7. Practical Advice: How to Use These Tools Effectively

For ordinary users, the key to using these tools well is to change usage habits rather than pursue longer prompts:

1. Start rough, then refine: Do not expect to describe everything completely at once. First provide AI with a rough framework, then refine it step by step through circling, dragging, or editing.
2. Use visual expression: When relationships such as order, dependency, or grouping can be expressed graphically, drawing them is often more accurate than using text alone.
3. Keep human review: AI-generated formulas, queries, and schedules should be reviewed by humans, especially when they involve amounts, data, or external deliverables.
4. Pay attention to portability: Prefer output formats that can be exported, synchronized, and audited, so that AI-generated results can truly integrate into existing workflows.

Conclusion

The shift from “writing prompts” to “rough gesturing” appears to be a change in interaction methods, but at a deeper level it reflects a restructuring of the human-AI relationship: AI is gradually moving from being a tool that requires precise instructions to becoming a partner that can understand intent and co-create with users. Understanding the underlying logic of multimodal understanding, structured generation, and natural-language-driven data analysis is more important than chasing any specific feature. Tools will continue to iterate, but the ability to reduce expression costs and shorten the distance from idea to implementation is a long-term capability worth mastering.