How Multimodal Models Turn Architecture Flowcharts into Development Plans
Multimodal models can read flowcharts, screenshots, and other visual engineering materials and convert them into structured implementation plans. This article explains why flowcharts matter before coding, how to prepare inputs and prompts, how to validate extracted steps, and how to map them to engineering tasks. It uses a mini-program login authentication flow to show how diagram-derived steps become token issuance, validation, and exception handling.
1. Why Multimodal Capabilities Are Entering Development Workflows
Plain-text input covers only part of engineering material. UI screenshots, business charts, scanned contracts, invoices, and flowcharts exist as images. When a model works with these materials, it needs to do three kinds of work: understand the content, convert it into executable steps, and output structured information or business conclusions.
In development scenarios, the first two kinds of work are most common. For example, a flowchart drawn during a design review contains roles, decision branches, and data flows. If a model can read this information accurately, the pre-coding analysis can be shortened significantly.
2. Why Flowcharts Are a Starting Point for Coding
Before coding, frontend and backend teams usually complete solution design together and produce a system architecture diagram or business flowchart. The arrow numbers, participants, and branch conditions in the diagram determine API boundaries, state transitions, and exception handling. Once the solution is fixed, coding can reduce rework caused by changing the design while writing code.
The difficulty at this step is that information in the diagram is often scattered and contains implicit conventions. For example, an arrow may represent a synchronous call or an asynchronous callback. If the model misses one arrow while reading the diagram, the subsequent plan will be missing a step. Flowchart parsing therefore needs an explicit verification mechanism.
3. Parsing Flowcharts with Multimodal Models
3.1 Input Preparation
Images need to be clear, text should be legible, and arrows should be numbered. Numbering lets the model output and manual verification point to the same steps. If the original diagram distinguishes roles only by color and has no textual explanation, add a legend before sending it to the model.
3.2 Prompt Structure
An effective prompt contains four parts: task goal, input material, output format, and constraints. The output format should be fixed so that downstream programs can parse it and humans can compare it.
Task: Read the attached flowchart, extract all participants, the complete steps ordered by arrow number, and the input, output, and decision conditions of each step.Output format:1. Participant list (name, responsibility)2. Step list (number, sender, receiver, action, data)3. Branches and exception paths4. Design summary (no more than 5 items)Constraints:- Step numbers must correspond one-to-one with arrow numbers in the diagram- Do not add steps that do not appear in the diagram; mark unrecognizable parts as "to be confirmed"
The constraint “mark unrecognizable parts as to be confirmed” is important. It makes the model express uncertainty explicitly instead of providing a seemingly complete guess.
3.3 Result Validation
After the model outputs results, three checks are needed:
- Whether the total number of steps matches the number of arrows in the diagram;
- Whether the sender and receiver of each step match the diagram;
- Whether every branch path has a destination and there are no dangling nodes.
Validation can be done manually or written as simple rule checks. For example, counting whether the number of arrows equals the number of steps is the lowest-cost first gate.
3.4 From Steps to an Implementation Plan
The step list can be mapped to engineering tasks. Each sender corresponds to a module or service, each cross-boundary arrow corresponds to an API or message, and each decision condition corresponds to a piece of validation logic. The plan should also include API definitions, state definitions, exception handling, and test cases.
4. Example: Mini-Program Login Authentication
Mini-program login usually involves three parties: the client, the business server, and a third-party identity service. A typical flow is as follows:
- The client calls the login capability provided by the platform to obtain a one-time temporary credential;
- The client sends the temporary credential to the business server;
- The server uses the credential to request a user identifier and session key from the identity service;
- The server generates a business-owned login token and records the session state;
- The server returns the token to the client, and the client stores it locally;
- Subsequent requests carry the token, and the server verifies the signature and expiration time on each request;
- When the token expires, the client goes through a refresh flow or logs in again;
- When the user logs out, the server revokes the token;
- Sensitive data is decrypted on the server and is not sent to the client;
- Exception branches cover invalid credentials, network timeouts, and duplicate requests.
4.1 Server-Side Token Issuance Example
function handleLogin(tempCode):if tempCode is empty:return error(INVALID_PARAM)identity = identityService.exchange(tempCode) // Exchange the temporary code for a user identifier and session key with the identity serviceif identity fails:return error(AUTH_FAILED)userId = userStore.findOrCreate(identity.userKey)token = signToken(userId, expiresIn = 2 hours)sessionStore.save(userId, token, identity.sessionKey) // Store the session key only on the serverreturn { token: token, expiresIn: 7200 }
This logic has three key points. First, the temporary credential can be used only once, and the server should reject repeated use. Second, the session key is not returned to the client. Third, the token lifetime should be short and paired with a refresh mechanism.
4.2 Token Validation Example
function authenticate(request):token = request.header("Authorization")claims = verifySignature(token)if claims is invalid or claims.expired:return error(UNAUTHORIZED)if sessionStore.isRevoked(claims.userId, token):return error(UNAUTHORIZED)return claims.userId
Validation failures return a unified unauthorized error and do not expose internal details to the client.
