Long-Running Coding Agent Test: Building a Video Generation SaaS in Three Hours

    This article documents a hands-on test of a long-running coding agent that was asked to build a UGC video generation SaaS for short-video e-commerce in about three hours. It covers how to set scope and acceptance criteria, use spec-driven development with stage gates, design the architecture, and handle asynchronous tasks, idempotency, a credit ledger, tenant isolation, content moderation, observability, and testing. Readers get a practical framework for judging whether generated code is ready for a commercial product.

    1. From Writing Pages to Delivering Systems

    Code generation tools have often been judged by simple tasks: completing a function, generating a page, or writing a sample app. A commercial system raises the bar. The code must run, and it must also be maintainable, auditable, scalable, and rollback-ready. It must also behave consistently under errors, concurrent access, and malicious input.

    This is why long-running coding agents deserve attention. Their real value lies in solving long chains of software engineering work, not isolated code generation:

    1. Requirement breakdown: Turning “build a video generation platform” into clearly bounded domains such as accounts, projects, assets, tasks, billing, moderation, and publishing.
    2. Cross-module consistency: Frontend state, backend APIs, database fields, the task state machine, and credit deduction rules must match each other.
    3. Long-context management: During an automated task that runs for hours, the agent must keep key constraints, design decisions, and open items. It must not lose the architectural conventions set earlier.
    4. Error handling loops: How should failed generations be retried? How are duplicate submissions made idempotent? Are credits refunded when a task is canceled? How is data isolated between tenants?
    5. Verification and delivery: The final deliverable must include more than code. It also needs migration scripts, tests, API documentation, a runbook, and an acceptance checklist.

    This article follows one experiment. The goal was an end-to-end orchestration lasting about three hours, building a UGC video generation SaaS for short-video e-commerce. Users upload product images or other materials, enter selling points and style preferences, and the system generates product videos asynchronously. The product includes basic accounts, credits, task tracking, and admin functions. Total runtime depends on queue load, compute availability, network conditions, and manual review. This article focuses on engineering workflow and architecture. The single run time is not a performance commitment for any product or model.

    2. Experiment Scope: A Demonstrable Commercial MVP, Not a Demo

    “Commercial SaaS” does not mean building every feature at once. It means delivering a correct business loop with room to grow, within a small scope. This experiment set the following boundaries.

    2.1 Core Roles and Workflow

    • Content operator: Signs up, logs in, creates projects, uploads assets, submits generation tasks, checks progress, and downloads or publishes results.
    • Team administrator: Manages members, reviews the credit ledger and task records, and handles failures and refunds.
    • Platform operator: Monitors queues, failure rates, costs, content safety blocks, and alerts.

    The core workflow is:

    1. The user uploads images and copy materials.
    2. The system creates a generation task and freezes the matching credits.
    3. The task enters an asynchronous queue, where a generation worker calls the video generation capability.
    4. The frontend shows stage progress through polling or a persistent connection.
    5. On success, the video asset is saved, credits are settled, and a preview is available.
    6. On failure or timeout, the reason is recorded, frozen credits are released, and a limited number of retries is allowed.
    7. Content safety and copyright checks apply throughout upload, generation, and before publishing.

    2.2 Required Non-Functional Properties

    A commercial MVP needs the following properties from the start. Adding them later costs much more.

    • Idempotency: Repeated clicks, network retries, and duplicate queue deliveries must not cause duplicate charges or duplicate generations.
    • Tenant isolation: Data and assets of different teams and users must not be accessible across boundaries.
    • Reliable async processing: Tasks that take minutes cannot depend on a single synchronous request. They need a task state machine and compensation mechanisms.
    • Auditable accounting: Credit changes are recorded only in an immutable ledger. Balances cannot be edited directly.
    • Security and compliance: Upload validation, content moderation, access control, audit logs, and deletion must be part of the design.
    • Observability: Task success rate, queue wait time, failure reasons, costs, and accounting discrepancies must be monitorable.
    • Maintainability: API contracts, data models, architecture decision records, and tests must be produced alongside the code.

    3. Spec-Driven Development: Define the Requirements, Then Let the Agent Write Code

    The biggest risk for a long-running agent is not that it cannot write code. The risk is that it moves fast on vague goals. Without a fixed spec, the model keeps changing pages, APIs, and database schemas, and the result is a growing pile of patches that never converges. The experiment used a spec-driven approach. The work was split into four phases, with research, architecture, development, and verification each guarded by gates.

    3.1 Stage Gates

    1. Requirements spec gate: Define target users, core use cases, out-of-scope items, and acceptance criteria.
    2. Domain model gate: Define entities, relationships, state machines, permission boundaries, and accounting rules.
    3. Architecture decision gate: Define frontend and backend layering, the task queue, storage, idempotency, error codes, and observability.
    4. API contract gate: Define requests, responses, error branches, pagination, authentication, and rate limits before implementation begins.
    5. Vertical slice gate: Build the minimum complete loop first: login, create project, upload, submit task, query status, and charge or refund credits.
    6. Verification gate: Unit tests, integration tests, end-to-end cases, authorization tests, duplicate submission tests, and failure compensation tests must pass.
    7. Delivery gate: Migration scripts, environment variable documentation, a runbook, alert definitions, and known limitations must all be complete.

    3.2 What the Spec Document Should Include

    • User stories: Who needs to do what, in which situation, and what result they get on success.
    • Acceptance criteria: Behavior that can be verified, such as “Submitting the same idempotency key ten times creates one task and charges once.”
    • Domain objects: Workspace, member, project, asset, generation task, task event, credit ledger, and moderation record.
    • State machine: Allowed transitions for each state, the triggers, and the compensation actions after failure.
    • API contract: Resource paths, authentication requirements, idempotency requirements, error codes, and rate-limiting rules.
    • Security boundaries: Which fields users can change, which fields only the server can write, and which operations require admin permissions.
    • Out-of-scope items: For the first version, no complex template marketplace, no multi-level distribution, and no custom model training, to prevent scope creep.

    4. Overall Architecture: Splitting Long Tasks Across a Reliable System

    The recommended structure is shown below.

    1. [Browser / Mobile]
    2. |
    3. [Access layer: authentication, rate limiting, auditing, replay protection]
    4. |
    5. [Business API layer: projects, assets, tasks, credits, administration]
    6. | \
    7. [Relational database] ---- [Cache and idempotency records]
    8. |
    9. [Task orchestration layer] --- [Message queue] --- [Video generation worker pool]
    10. | |
    11. [Object storage] --- [Content safety / transcoding / callbacks] --- [Task event table]
    12. |
    13. [Observability: structured logs, metrics, distributed tracing, alerts]

    The key point is not the number of components. It is clear responsibility for each one:

    • The business API handles only authentication, validation, transaction boundaries, and task orchestration. It does not handle long-running video generation directly.
    • Generation workers consume tasks only. They run retryable, idempotent jobs and write status back to the task event table.
    • The database holds the strongly consistent records for tasks, assets, and accounting.
    • The message queue handles traffic spikes and async decoupling. It must be paired with idempotent consumption and dead-letter handling.
    • Object storage holds raw materials and generated outputs. Access goes through controlled, temporary authorization.
    • Observability is not added after launch. Every task carries a tracing ID from its first step.

    5. Key Module Design

    5.1 Frontend: Designing Interactions Around Task States

    The frontend should not be just an upload button and a results page. Its information architecture should follow the task lifecycle:

    • Project list: Project status, recent tasks, credit balance, and failure notices.
    • Asset upload: Drag and drop, chunked or direct upload, progress display, format validation, and retry on failure.
    • Generation settings: Video aspect ratio, duration, style, selling point copy, and whether to add a watermark.
    • Task details: Queued, generating, under review, succeeded, failed, and canceled states.
    • Result review: Preview, download, regeneration, refund requests, and failure reasons.
    • Credit ledger: Frozen, deducted, and returned credits, along with monthly cost trends.

    The frontend should use an explicit state machine, not a set of boolean flags combined ad hoc:

    1. idle -> uploaded -> queued -> generating -> reviewing -> succeeded
    2. | |
    3. v v
    4. failed canceled
    5. failed/canceled -> refunding -> refunded

    Several interaction principles apply:

    1. Give immediate feedback after submission. Show “Task submitted and credits frozen” instead of a loading spinner that gives no information.
    2. Make progress explainable. Describe stages with task events. Do not use a fake fixed percentage.
    3. Use backoff for polling. Start with short intervals, increase them gradually, and add random jitter to avoid request storms.
    4. Make cancellation idempotent. Repeated cancel requests should not produce errors. The server determines the final state.
    5. Explain what to do next after failure. Distinguish between non-compliant materials, insufficient credits, generation failure, and moderation rejection.
    6. Enforce permissions up front. The frontend hides actions a user cannot perform, but the server must make the final permission decision.

    5.2 Backend API: Layering, Idempotency, and Error Codes

    The backend should be divided into access, application, domain, and infrastructure layers. All write endpoints follow three rules:

    • Authenticate and authorize first. Parse the user and workspace from the signed token, then verify ownership of the resource.
    • Require or generate idempotency keys. Operations such as creating tasks, charging credits, and canceling support idempotency keys.
    • Keep transaction boundaries explicit. Only strongly consistent database operations go inside a local transaction. External generation calls run asynchronously through the task queue.

    A typical resource API can be summarized as:

    1. POST /v1/projects
    2. POST /v1/assets/upload-intents
    3. POST /v1/assets/{assetId}/complete
    4. POST /v1/generations
    5. GET /v1/generations/{jobId}
    6. POST /v1/generations/{jobId}/cancel
    7. GET /v1/credits/ledger

    Keep these points in mind for API design:

    • Creating a task returns a task ID and its current status. It does not block until the video is finished.
    • Querying a task returns the status, stage, progress events, estimated completion time (which is not guaranteed), failure reason, and refund status.
    • List endpoints must support pagination, sorting, and filtering by status.
    • Every error returns a stable error code, a user-readable message, and a tracing ID.
    • External callbacks must verify their source, timestamp, and signature, and must handle replay attacks.

    5.3 Database: Domain Modeling and Tenant Isolation

    The core tables can be designed along these lines:

    1. user_account(
    2. id,
    3. phone_hash,
    4. status,
    5. created_at
    6. )
    7. workspace(
    8. id,
    9. name,
    10. owner_id,
    11. created_at
    12. )
    13. membership(
    14. workspace_id,
    15. user_id,
    16. role,
    17. status,
    18. unique(workspace_id, user_id)
    19. )
    20. asset(
    21. id,
    22. workspace_id,
    23. owner_id,
    24. object_key,
    25. media_type,
    26. size,
    27. checksum,
    28. status,
    29. created_at
    30. )
    31. generation_job(
    32. id,
    33. workspace_id,
    34. project_id,
    35. creator_id,
    36. state,
    37. input_snapshot,
    38. provider_task_id,
    39. attempt_count,
    40. reserved_credit,
    41. failure_code,
    42. created_at,
    43. updated_at
    44. )
    45. job_event(
    46. id,
    47. job_id,
    48. event_type,
    49. payload,
    50. created_at
    51. )
    52. credit_ledger(
    53. id,
    54. workspace_id,
    55. delta,
    56. reason,
    57. ref_type,
    58. ref_id,
    59. balance_after,
    60. created_at
    61. )
    62. idempotency_record(
    63. scope,
    64. idempotency_key,
    65. request_hash,
    66. response_snapshot,
    67. expires_at
    68. )

    Tenant isolation must be enforced at three layers:

    • Application layer: Every query carries the workspace ID, and the server verifies membership.
    • Data layer: Where row-level security policies are available, use them as a safety net. Key unique indexes include the workspace ID.
    • Storage layer: Object access uses short-lived authorization. Permanent public URLs are not exposed. Object keys contain no sensitive information.

    Task events and credit ledger entries should be treated as audit data. Avoid updating or deleting them. A balance can be computed directly from the ledger, or stored as a materialized balance. Either way, every balance change must have a matching ledger entry, and the two must be reconciled regularly.

    5.4 Async Tasks: State Machine, Retries, and Compensation

    Video generation takes minutes, so it must run asynchronously. The task state machine should include at least these states:

    • created: The task is created but not yet queued.
    • queued: The task has entered the queue.
    • running: A worker has picked up the task.
    • reviewing: Generation is complete, and content moderation or transcoding is in progress.
    • succeeded: The task succeeded and credits are settled.
    • failed: The task failed. Refund is pending or complete.
    • canceled: The user or the system canceled the task.
    • refunding: A refund is in progress.
    • refunded: The refund is complete.

    The core logic for submitting a task can be summarized as:

    1. begin transaction
    2. check membership and quota
    3. reserve credit by ledger
    4. insert generation_job(state=queued)
    5. insert outbox_event(job_queued)
    6. commit
    7. worker:
    8. consume job_queued
    9. check idempotency by job_id + attempt
    10. update state=running
    11. invoke generation capability
    12. on success -> write asset, state=reviewing, enqueue moderation
    13. on failure -> decide retry or failed
    14. on terminal success -> settle credit
    15. on terminal failure -> release credit

    Key design points:

    1. Idempotent consumption: When the same task message is delivered more than once, only one valid execution is allowed, or settlement happens only once.
    2. Limited retries: Retry only recoverable errors. Set a maximum number of attempts and use exponential backoff. Fail permanent errors immediately.
    3. Timeout recovery: If a worker crashes, timed-out tasks must return to a retryable state.
    4. Cancellation semantics: Queued tasks can be canceled directly. For running tasks, notify the external generation service to cancel, then refund according to the rules.
    5. Transactional outbox: After the database transaction commits, deliver events to the queue through a reliable outbox. This prevents the case where the database write succeeds but the message is lost.
    6. Progress events: Write an event to the event table at each stage change. The frontend uses these events to show explainable progress.
    7. Dead-letter handling: Tasks that still fail after several retries move to a dead-letter queue and trigger an alert for manual handling.

    5.5 Authentication and Credits: Ledger Thinking, Not Balance-Field Thinking

    The account system should include at least phone number verification code login, session management, membership and roles, rate limiting, and risk control. Verification codes should meet these requirements:

    • Short validity period, single use, and rate limits across multiple dimensions: phone number, device, and network source.
    • A count of failed attempts and a cooldown period.
    • Sending logs stored with personal data redacted, so abuse can be investigated.
    • After login, issue a short-lived access token and a rotatable refresh credential.
    • Support for signing out sessions, a device list, and audit logs for unusual logins.

    The credit system should use a freeze, settle, and refund ledger pattern:

    1. Task submission:
    2. check available credits
    3. write a freeze entry to the ledger (negative available quota)
    4. link the frozen amount to the task
    5. Task succeeds:
    6. write a consumption entry to the ledger
    7. release the freeze
    8. update the materialized balance
    9. Task fails:
    10. write a release or refund entry to the ledger
    11. ensure each task is refunded only once
    12. Reconciliation:
    13. compare sum(delta) with the materialized balance
    14. check per task that freeze, consumption, and refund entries close the loop

    Never run an update such as “balance = balance - price” without an audit record. Credits represent user entitlements. They must be idempotent, traceable, and reconcilable.

    5.6 Asset Storage, Content Safety, and Comp