Reading Intent from a Whiteboard: The Engineering Behind AI Workbench Sketch-and-Adjust Editing

1. From “typing a description” to “pointing and adjusting”: the interaction paradigm is shifting

Traditional human-machine collaboration is language-driven: users must state, in natural language precise enough to be actionable, both what to change and how to change it. For structural edits this is painfully inefficient. Describing the position of the third box and the fifth connector on its right in a diagram usually takes more effort than just drawing it.

The new generation of AI workbenches adds a second channel: spatially driven input. Users circle a target on the canvas, sketch a connector, and write a short note such as “split this into three steps.” The model then has to understand two dimensions at once — which object is being referred to, and what should happen to it.

This is not a simple multimodal recognition problem. It is a composite engineering problem of referring-expression resolution plus structured editing. Get it right and the experience feels like “it actually understands what I was gesturing at.” Get it wrong and the experience is “it confidently mangled a whole area.”

2. The common technical ground across whiteboards, spreadsheets, and data analysis

Whiteboards, spreadsheets, and data analysis tools look nothing alike, but from an instruction-parsing perspective they share one abstraction:

Surface	User’s spatial actions	What the target object really is
Whiteboard	Circle nodes, hand-draw arrows, attach comments	Nodes and edges in the canvas element tree, plus their properties
Spreadsheet	Select a range, comment on a cell, draw a divider	Cell coordinate ranges, formulas, pivot configuration
Data analysis	Select a region of a chart, annotate a definition	Query statements, aggregation dimensions, filters, visualization config

The key insight: what the user sees as a “pixel region” is usually stored by the system as “structured objects with IDs.” The whole point of intent parsing is to establish a reliable bidirectional mapping between those two spaces. Once that mapping is off by anything, every layer above it reasons from a wrong premise.

3. Five core problems that must be solved

1. Deixis resolution. A circle may fall inside a single element, span several elements, or enclose nothing but background. Candidate sets should be derived from geometric containment, hit testing, and area ratio; the annotation text semantics then narrows that set down to one target.

2. Timing and concurrency. In real interaction, a user draws the arrow, adds the text, nudges the position, and only then presses Enter. Those events are spread across several seconds and may interleave with other collaborators’ actions. The system must perform event merging rather than parsing each event individually; otherwise it generates a flood of duplicated or conflicting instructions.

3. Locality and the minimal-change principle. “Break this part down a bit more” must never trigger a full-page redraw. The execution layer needs an explicit scope and must explicitly protect everything outside it. This is the dividing line for whether you dare let the system edit automatically.

4. Traceability and rollback. Generative editing is inherently non-deterministic. Every execution should produce a diff and a snapshot, with one-click revert. In enterprise settings, automated editing without rollback capability is essentially unacceptable.

5. Semantic correctness, not surface similarity. In spreadsheets, a user circles a region but may mean “change the formatting in this range” or “aggregate this data by week.” The coordinates are identical; the semantics are not. Resolving this requires context — the current formula, the data structure, the chart type.

4. A layered reference architecture

(1) Interaction capture -> (2) Normalization -> (3) Intent parsing -> (4) Execution -> (5) Validation / replay
    (raw events)             (stable IDs + time)   (intermediate repr. IR)  (tool calls)    (diff + snapshot)

Layer 1 — Interaction capture. Collect events only; make no judgments: pointer down/move/up sequences, path points, selection bounding boxes, annotation text, timestamps, operator ID. Freehand strokes are best stored as normalized relative coordinate sequences so they can be reused across zoom levels.

Layer 2 — Normalization. Convert screen coordinates into canvas logical coordinates and resolve “the region the user circled” into a candidate object set with stable IDs. Stable IDs are critical: the moment array indices or ephemeral indices are used, any concurrent edit makes subsequent instructions point at the wrong object.

def normalize(raw_events, canvas_meta):
    events = merge_window(sort_by_ts(raw_events), gap_ms=800)
    out = []
    for ev in events:
        if ev.kind == "selection":
            box = to_canvas_coords(ev.bbox, canvas_meta.scale, canvas_meta.offset)
            out.append({
                "kind": "selection",
                "bbox": box,
                "candidates": hit_test(canvas_meta.object_index, box),
                "area_ratio": box_area(box) / canvas_meta.viewport_area,
            })
        elif ev.kind == "annotation":
            out.append({"kind": "annotation", "text": ev.text.strip(), "anchor": resolve_anchor(ev)})
    return out

Layer 3 — Intent parsing. This layer hands the normalized event sequence plus a description of the current object structure to the model and asks for an intermediate representation (IR) rather than UI elements or code directly. The IR is the central decoupling point of the whole design: it separates understanding from execution, which makes errors locatable and unit-testable.

{
  "target_ref": {
    "canvas": "board-01",
    "element_id": "node-17",
    "bbox": [120, 88, 480, 260]
  },
  "annotation_type": "freehand+text_comment",
  "verb": "refine_structure",
  "instruction": "Split this segment into three parallel subtasks, with dependency edges drawn as dashed lines",
  "scope": ["node-17.children", "edges[3..5].style"],
  "protected": ["node-*.id", "canvas.theme", "layout.auto"],
  "expected_delta": "additive"
}

The scope and protected fields deserve emphasis: the first declares what may change, the second declares what must never be touched. Having the model explicitly emit the protected list is more reliable than repeatedly admonishing it in the prompt.

Layer 4 — Execution. After validation, the IR is translated into concrete operations (create a node, change a property, write a formula, recompute an aggregation) and executed in a constrained environment. Where code or formulas are generated, a dry run should happen in a sandbox first: check whether referenced entities exist, whether a circular dependency would form, and whether the result set exceeds the expected magnitude.

Layer 5 — Validation and replay. Take a structural snapshot before and after execution, compute the diff, and highlight it for the user. Nothing is persisted until the user confirms; after confirmation it is written to version history and can be reverted step by step.

5. Context construction: do not feed pixels straight to the model

A common mistake is “hand the model a screenshot and let it find the elements.” For a medium-sized canvas (hundreds of elements) this is both expensive and unstable. A more pragmatic approach is structured description plus local cropping:

• Replace the full image with an object-tree summary: one line per element, containing ID, type, title, key properties, and connections.
• Attach detailed descriptions (full properties, neighboring context) only to the circled region.
• For spreadsheets, send the schema plus the formulas/values of the currently selected range, not the whole table.
• For data analysis, send the query structure, field semantics, and aggregation definitions, not the raw detail rows.

An illustrative prompt structure:

[Role] You are a canvas editing planner. Output only schema-conformant IR, never UI elements.
[Canvas summary] 23 nodes, 5 groups, auto layout currently enabled
[Selection context] 3 candidate elements, 1 hit at the center (node-17)
[User annotation] "Split into three parallel subtasks, dependencies as dashed lines"
[Constraints] Protect node IDs and theme config; do not modify elements outside the selection; do not add groups

6. Engineering considerations for production

Take small steps. Handle one scope per interaction. Better to ask “do you mean this box or the whole group?” than to change ten elements at once. The cost of repairing a local error is far below the cost of repairing a global one.

Tier confidence. High confidence: execute directly and show the diff. Medium confidence: offer two or three candidate plans for the user to pick from. Low confidence: convert into a clarifying question. This strategy substantially reduces “silently changed the wrong thing.”

Security and compliance. Freehand annotations frequently contain real business names, customer information, and internal metrics. The entry point should detect and redact sensitive data, logs should never retain the full annotation text (only a hash and structural summary of the IR), and code-generating execution environments need network and file-system isolation.

On the infrastructure side, generic capabilities are sufficient: object storage to hold canvas snapshots and version history, a message queue to buffer and replay editing events, a container platform to host sandbox execution environments, and logging plus monitoring/alerting to track intent-parsing success rate, average interaction turns, and rollback rate. All of these map to standard product categories on general-purpose cloud platforms such as Baidu AI Cloud — they are commodity components, not proprietary solutions.

7. Evaluation: how do you prove it actually understood?

When building your own evaluation set, cover four metric categories:

1. Target hit accuracy — the proportion of cases where the correct object inside the circled region is selected, including multi-candidate scenarios.
2. Instruction fidelity — the structural match between a human-annotated expected IR and the model’s output IR.
3. Minimal-change rate — how much the actual diff overlaps with the minimal necessary diff, i.e., whether it changed too much.
4. First-pass success rate and rollback rate — the share of edits completed without human correction, and the share reverted by users after execution.

The first two measure understanding; the last two measure usability. Optimizing only the first two tends to produce the “great benchmark, unusable in production” gap, because real experience is dominated by the erroneous-edit rate.

8. Common anti-patterns

• Using indices as references. Guaranteed to break under concurrent editing; stick to stable IDs.
• Parsing event by event. A single drag generates hundreds of events; parsing them all independently produces conflicting instructions. Time-window merging is mandatory.
• Skipping the protected list. Models tend to “tidy up the surroundings while I’m here,” which is exactly what users resent most.
• Executing without snapshots. Without a diff and a way back, users will not authorize automated editing.
• Treating clarification as failure. A modest follow-up question costs less than a confident wrong answer, so evaluation should not penalize reasonable clarification turns.

9. Closing thoughts

“Circling something on a canvas is enough to make the requirement clear” looks like an improvement in interaction design, but underneath it sits a full stack of intent-parsing and controlled-execution engineering. What determines success is not whether the model can recognize a hand-drawn arrow, but whether four things form a closed loop: a stable mapping from space to objects, a structured intermediate representation, minimal-scope execution, and a replayable safety net. Get those four solid, and users — on whiteboards, spreadsheets, or data analysis workbenches alike — will feel comfortable handing their gestures over to the system.