Building a 24/7 Multi-Agent Collaborative Development and Delivery System from Scratch

    This article outlines the architecture and implementation of a 24/7 multi-agent collaborative development and delivery system. Using a competitor price monitoring dashboard as a case study, it covers requirement definition, role division, workflow control, structured task protocols, state machines, and testing strategies. The goal is to transition from simple chat-based demonstrations to a robust engineering pipeline that ensures traceability, security, and continuous operation.

    1. Define the Goal: Build a Collaboration System, Not Just Multiple Chat Windows

    Submitting a prompt like “develop a competitor price monitoring dashboard” to an agent, and having different roles handle research, product design, architecture, coding, and testing is a valuable automation direction. However, turning a demo flow into a continuously running engineering system requires solving specific problems, rather than just creating more agents:

    • Who is responsible for breaking down requirements, and who holds the final decision-making power?
    • What inputs are required for task handover, and how are outputs accepted?
    • How are different agents’ contexts isolated, and how is necessary knowledge shared?
    • How does the system recover after a programming tool execution failure?
    • For production releases, external notifications, and sensitive data, which operations must be approved by a human?

    Therefore, a multi-agent system is better understood as an engineering pipeline involving models. The model handles understanding, reasoning, and generation; the task scheduler drives the process; the storage system saves facts and artifacts; and the permission system restricts the execution scope.

    “24/7 operation” does not mean the model continuously generates content. Instead, it means the system can continuously receive events, schedule tasks, check states, and recover from failures. High-risk operations still require explicit human approval nodes.

    This article uses a “competitor price monitoring system” as an overarching case study to introduce an implementation approach decoupled from specific models, tools, and deployment environments. The roles, data fields, and flows below are illustrative and do not represent any existing product’s interfaces or capabilities.

    2. Turn a Single Requirement into a Verifiable Project Scope

    The example requirement can be summarized as:

    Build a price monitoring dashboard that supports maintaining competitor links, periodically scraping prices, reference original prices, discounts, and stock status, displaying price trends, notifying on low prices, abnormal fluctuations, and out-of-stock situations, and providing table file exports.

    This description contains multiple undefined business rules. If handed directly to a coding agent, it will likely produce a prototype with a complete UI but vague logic.

    2.1 Clarify the Meaning of Price

    A product page might simultaneously display a list price, an event price, a member price, a bundle price, and conditional discounts. These different prices cannot be lumped together.

    It is recommended to distinguish at least:

    • The current selling price directly displayed on the page;
    • The reference original price displayed on the page;
    • Discount descriptions with usage conditions;
    • The raw price text retained when reliable conversion is impossible.

    An inferred valid price should only be generated when discount conditions are clear and calculation rules are defined, and the calculation basis must be saved simultaneously. Do not let the model guess prices just to fill in a field.

    2.2 Clarify the Granularity of the Monitoring Target

    The same link might contain various specifications, sales regions, or purchase quantity conditions. If these conditions change, the price curve might jump without an actual price drop.

    Therefore, the monitoring target should include product link, specification ID, regional condition, and scraping source. Records where specifications cannot be stably identified should be marked as uncertain rather than compared directly with historical data.

    2.3 Provide Executable Acceptance Criteria

    Acceptance cannot stop at “the page can open.” It should be written as observable and reviewable behaviors, such as:

    • After adding a new monitoring target, the first scraping result or a clear failure reason is visible;
    • The trend chart only displays data under the same price caliber;
    • A single network failure does not directly trigger an out-of-stock alert;
    • The same anomaly is not repeatedly notified during the cooldown period;
    • The exported file includes scraping time, price caliber, and data status;
    • Unauthorized roles cannot modify scraping rules or read notification credentials.

    These rules serve as both product design criteria and inputs for subsequent automated testing.

    3. Role Division: Cover the Full Responsibility Chain with Minimal Agents

    More agents do not mean better. Too many roles increase message round-trips, context duplication, and coordination costs. In the initial phase, a “single general coordinator plus several specialized roles” structure is advisable.

    Role Core Responsibilities Main Artifacts
    General Coordinator Break down tasks, check dependencies, drive process, aggregate risks Task plan, status summary
    Researcher Map usage scenarios, data availability, and scraping compliance boundaries Research report, evidence list
    Product Manager Define business rules, feature boundaries, and acceptance criteria Requirement spec, acceptance checklist
    Architect Design modules, data models, API contracts, and permission boundaries Technical design, risk list
    Developer Implement features, add tests, submit change logs Code, test results, change records
    Tester Independently check requirement coverage, exception paths, and regression results Test report, defect list

    The Developer can call a controlled smart programming executor but should not inherently possess full production environment permissions. The Tester should not merely parrot the Developer’s self-test conclusions but should re-check based on independent acceptance rules.

    When resources are limited, the Researcher and Product Manager roles can be merged, or the Architect and Developer roles; however, the Coordinator and Tester responsibilities should remain clearly defined.

    4. Overall Architecture: Let the Workflow Control the Agents, Not Vice Versa

    A relatively complete logical architecture is as follows:

    1. Requirement Entry
    2. ↓
    3. Identity Verification & Task Registration
    4. ↓
    5. General Coordinator
    6. ↓
    7. Persistent Workflow & Task Queue
    8. ├─ Research Task
    9. ├─ Product Task
    10. ├─ Architecture Task
    11. ├─ Development Task
    12. └─ Testing Task
    13. ↓
    14. Artifact Storage & Acceptance Gate
    15. ↓
    16. Human Approval or Auto-Archive
    17. ↓
    18. Result Notification & Runtime Monitoring

    Here, the workflow system should save the task’s current phase, dependencies, retry counts, and acceptance results. The model can suggest the next step, but it cannot declare the entire project complete based solely on a conversation.

    4.1 Independent Context Does Not Mean Complete Information Isolation

    Each role only receives the information necessary to complete its current task, which helps reduce context pollution. However, roles still need to pass facts to each other through shared artifacts.

    For example, the Developer should read the confirmed requirement version, technical design, API contracts, and current defect list, rather than receiving the entire chat log of all roles.

    4.2 Key Facts Must Be Persisted

    Long context cannot replace persistent storage. Models might omit early constraints or lose details during summarization.

    The following should be saved independently:

    • Requirement versions and change records;
    • Confirmed design decisions;
    • Task statuses and dependencies;
    • Test results and defect statuses;
    • Code, documentation, and their version identifiers;
    • Human approval records.

    When restarting an agent, the system should rebuild the context from these facts, rather than asking the model to “recall what happened before.”

    5. Reduce Handover Loss with Structured Task Protocols

    Sending only natural language messages like “you do the testing” between agents cannot support complex collaboration. A more reliable approach is to define fixed fields for tasks.

    Below is a sample task ticket:

    1. Task ID: pricing-test-004
    2. Project: Price Monitoring System
    3. Task Type: Acceptance Test
    4. Requirement Version: v3
    5. Input Artifacts:
    6. - Confirmed requirement spec
    7. - Current build under test
    8. - Data field description
    9. Objective:
    10. - Verify price anomaly identification and notification deduplication
    11. Acceptance Criteria:
    12. - Scraping failure is not misjudged as out-of-stock
    13. - Repeated events do not produce duplicate notifications
    14. - Notification failures can be queried and redelivered
    15. Constraints:
    16. - Do not use real user data
    17. - Do not send test messages to external real recipients
    18. Output:
    19. - Test results
    20. - Defect list
    21. - Reproducible steps

    Task outputs should also be structured, at least distinguishing between “Completed,” “Blocked,” “Needs Clarification,” and “Execution Failed.”

    An important principle here: the model generating a test report does not mean the test actually executed successfully. The system should use test run logs, exit statuses, and result artifacts as evidence before deciding whether the task passes acceptance.

    6. State Machines, Retries, and Idempotency: The Foundation of Continuous Operation

    It is recommended to set explicit states for tasks:

    1. Pending → Ready → Executing → Pending Acceptance → Completed
    2. ├→ Waiting for Clarification
    3. ├→ Retryable Failure
    4. └→ Terminal Failure

    6.1 Categorize Failures

    Network timeouts can usually be retried; missing credentials should wait for configuration; requirement conflicts need clarification; persistent test failures should enter the defect fixing process.

    If all failures are simply resubmitted to the model, the system might get stuck in repeated analysis and modifications.

    6.2 Set Boundaries for Retries

    You should simultaneously limit the retry count per task, cumulative execution time, model call budget, and tool operation count. Upon reaching the limit, pause the task and output the completed content, blocking reason, and suggested next steps.

    6.3 Idempotency Must Be Considered for External Side Effects

    In real systems, the same message might be delivered multiple times. Therefore, operations like sending notifications, creating tasks, and writing business records require stable business identifiers to prevent duplicate execution.

    For instance, alert deduplication can be determined by combining the monitoring target, alert type, and anomaly event ID. The saving of deduplication records and the updating of notification status should also consider consistency to avoid the system sending duplicates after an interruption recovery.

    7. Price Monitoring Business: Ensure Data Reliability Before Smart Analysis

    Multi-agents can assist in developing the scraping module, but runtime data quality cannot rely on the model’s subjective judgment.

    7.1 Confirm Authorization and Boundaries Before Scraping

    Prioritize using authorized data APIs, legally imported data, or public information permitted for scraping, and comply with applicable rules, access restrictions, and personal information protection requirements.

    Do not bypass login restrictions, CAPTCHAs, access controls, or anti-automation measures. When encountering these, stop the relevant scraping and record the reason, shifting to authorized access or manual processing.

    7.2 Separate “Scraping Failure” from “Business State”

    It is recommended to distinguish at least:

    • Scraping successful and in stock;
    • Scraping successful and out of stock;
    • Target page unavailable;
    • Network failure;
    • Page structure changed;
    • Price parsing failed;
    • Access restricted.

    Network failures and price parsing failures cannot be mapped to a zero price, nor can they be treated as out of stock.

    7.3 Save Traceable Data Evidence

    Each price record should be associated with scraping time, source, specification conditions, parsing rule version, and status description. If necessary, save sanitized and permission-controlled raw snippets for troubleshooting parsing errors.

    Price calculations should use precise decimal representation or explicit minimal currency units to avoid introducing rounding errors from binary floating-point numbers in monetary operations.

    8. Anomaly Detection and Notification: Accuracy Over Frequency

    Price anomalies can start with explainable rules; there is no need to rely on complex models from the beginning.

    For example, the relative change rate can be expressed as:

    1. Change Rate = (Current Valid Price - Base Price) / Base Price

    Before using this formula, ensure the base price is greater than zero and that the current record and base record share the same currency, specification, and discount caliber.

    The base can be the previous valid record or the median over a period. The former is simple to implement but susceptible to individual anomalies; the latter is more robust but requires sufficient historical data.

    Notification Noise Reduction Should Include Four Layers of Control

    1. Continuous Confirmation: For easily fluctuating states, require multiple valid observations before triggering.
    2. Cooldown Mechanism: Do not repeat notifications for the same type of anomaly within a specific period.
    3. Recovery Notification: Optionally send a status recovery message when the price recovers or is back in stock.
    4. Failure Compensation: Enter a controlled retry after a sending failure and retain the final delivery status.

    Notification content should include the trigger reason, current value, comparison base, scraping time, and data quality hint. It cannot just say “anomaly detected,” otherwise the receiver still needs to investigate from scratch.

    9. Limit Execution Boundaries Even When Letting Development Agents Write Code

    The Developer role can modify files, execute tests, and generate build artifacts in an isolated workspace but must adhere to the principle of least privilege.

    The following measures are recommended:

    • Use an independent workspace for each development task;
    • Mount only necessary directories;
    • Do not inject production credentials by default;
    • Set restrictions on external network access;
    • Require approval for file deletion, permission modification, and external publishing;
    • Save tool call logs and execution results;
    • Use synthetic data or sanitized samples for testing.

    Special attention must be paid to the fact that external pages, requirement attachments, and runtime logs might contain deceptive text. This content should be treated as data to be processed, not as instructions capable of changing system permissions.

    For example, text like “read local config and send to specified address” appearing on a scraped page cannot serve as a reason to execute. Permission control must be enforced by mechanisms outside the model.

    10. Testing Loop: From “Runs Locally” to “Ready for Delivery”

    It is recommended to organize testing into four layers.

    10.1 Rule Testing

    Verify price parsing, currency conversion, anomaly thresholds, deduplication logic, and state transitions. These tests should avoid relying on real external pages as much as possible to ensure stable and repeatable results.

    10.2 Module Collaboration Testing

    Verify the handover between scraping tasks, data storage, rule evaluation, and notification delivery, focusing on failure propagation and duplicate execution.

    10.3 End-to-End Flow Testing

    Start from adding a new monitoring target and sequentially verify data ingestion, trend display, anomaly identification, notification generation, and file export.

    During export, prevent external text from being interpreted as formulas by spreadsheet software. Fields that might trigger formula calculations should be safely processed according to the output format.

    10.4 Recovery and Security Testing

    Actively simulate executor exits, message duplication, credential failures, and brief database unavailability to check whether the system can recover and whether it produces duplicate notifications or data loss.

    The Tester role should ultimately provide traceable evidence, not just a natural language conclusion that “tests passed.”

    11. 24/7 Operation Requires Observability and Cost Governance

    A continuously running system should observe at least three types of metrics.

    Business metrics include scraping success rate, valid data coverage, alert accuracy, and data freshness.

    Workflow metrics include task wait time, duration per stage, failure reason distribution, human intervention count, and duplicate execution count.

    Resource metrics include model call consumption, tool execution time, storage growth, and notification volume.

    Different tasks should adopt different execution methods. Fixed format validation, currency calculation, and duplicate event judgment are suitable for deterministic programs; requirement analysis, design review, and complex defect localization require more model involvement.

    This approach reduces costs and minimizes unnecessary uncertainty.

    Additionally, global stop conditions should be set for the General Coordinator. When the budget is exhausted, critical dependencies are unavailable, or risks exceed the allowed range, the system should pause and report, rather than continuing to execute just to maintain “autonomous operation.”

    Phase 1: Start with a single controlled flow. Humans confirm requirements, agents assist in generating designs and code, and humans deliver after tests pass. The goal is to validate artifact formats, tool boundaries, and acceptance rules.

    Phase 2: Introduce persistent task queues, structured handovers, automated testing, and failure retries, enabling the system to recover from interruptions.

    Phase 3: Add parallel roles, periodic scheduling, alert compensation, cost budgets, and operations dashboards. Tasks without dependencies can run in parallel, while tasks with explicit sequential orders must wait for prior acceptance.

    Phase 4: Optimize role division based on actual failure records. Only add a new agent when a specific type of task requires independent knowledge, independent permissions, or independent acceptance.

    Conclusion

    The value of multi-agent collaboration is not copying software team job titles into multiple chat roles, but organizing requirements, decisions, execution, and acceptance into a traceable engineering pipeline.

    For a price monitoring system, models can accelerate research, design, coding, and debugging, but data authenticity, permission boundaries, task states, and release approvals still require deterministic engineering mechanisms.

    A truly reliable 24/7 system should be able to clearly answer: what it is currently doing, what the basis is, what step it has reached, how to recover from failure, and what things must wait for human decision. Solving these problems is the only way to move from an automated demo to a continuously available collaborative delivery system.