Testing Long-Running AI Coding Tasks: Building a Video Generation SaaS in 3 Hours

    An evaluation of whether today's AI coding tools can handle multi-hour, multi-module projects rather than one-shot page generation. Using a merchant-facing short-video generation SaaS as the test case, this article breaks down the requirements that turn a working demo into a billable service — multi-tenant isolation, asynchronous scheduling, identity and quotas, cost control, and observability — and shows why those constraints force genuine architecture decisions. It lays out a Spec-Driven Development workflow (research → architecture → development, with documents and acceptance criteria at each gate), covers five core engineering test points with sample SQL and TypeScript, and closes with what actually consumed the three hours, a list of recurring failure modes, and a practical checklist for teams delegating long tasks to AI coding tools.

    Testing Long-Running AI Coding Tasks: Building a Video Generation SaaS in 3 Hours

    1. Why a Short-Video Generation SaaS Makes a Good Test Case

    Over the past year, generative models have become fairly reliable at “write one page in a single conversation.” But the complexity of commercial software has never lived inside a single function — it lives in the constraints between functions. The moment a video generation service goes from “looks like it runs” to “we’re willing to take money for it,” the problem set grows fast:

    • Multi-tenant isolation: data, assets, and finished renders belonging to different merchants must never be visible to one another.
    • Asynchronous scheduling: a single video render can take minutes or longer, so synchronous requests will inevitably time out.
    • Identity and quotas: registration and login, CAPTCHA-based abuse prevention, short-lived tokens, credit deduction.
    • Cost control: every generation burns model and rendering compute, so cost has to be held up front and settled afterward.
    • Observability: when tasks stall, failure rates spike, or per-item cost runs away, you need to be able to locate the cause.

    The reason for choosing this problem also comes from a plain observation: paying users don’t care whether you can wire up a workflow. They care about three things — what a finished video costs, how many can be produced in a day, and whether one click produces a result. The cost and throughput constraints in the product requirements are precisely what force metering, quotas, and an asynchronous queue into the architecture. Put differently, this problem inherently pushes the model into making a complete set of system architecture decisions, which makes it good material for testing long-horizon capability.

    2. What Makes Long-Running Tasks Hard: Three Kinds of Pressure

    The difference between single-shot code generation and hours of continuous work isn’t whether the model knows syntax. It’s three structural problems.

    Context coherence. A real project spans at least five or six modules: database migrations, the API layer, task consumers, frontend polling, billing logic. The model still has to remember cross-file constraints dozens of turns later — “credit deduction happens before the task is enqueued,” “finished video URLs are served through signed links.” Otherwise you get schema drift: a column gets added to the database while the code keeps using the old field.

    Decision consistency. Long tasks repeatedly hit forks: should the queue be database polling or a message broker? Sessions or tokens for auth? If the model re-decides at every turn, the artifacts contradict each other. So the fix isn’t making it smarter — it’s externalizing decisions into documents, so it reads the same source of truth every time.

    Error accumulation. At 95% correctness per step, overall reliability degrades noticeably after twenty steps. The only remedy is to cut the long chain into short chains that can be verified independently, each with an executable acceptance check, so a mistake gets rolled back immediately instead of being debugged all at once at the end.

    3. SDD: Turning the Prompt into a Process and an Evaluation Function

    This experiment used Spec-Driven Development (SDD). The core change is in how you interact with the tool: instead of “get me a feature,” you drive it through research → architecture → development, where each phase delivers a document first and only advances after you confirm it.

    The prompt skeleton looks roughly like this (illustrative):

    1. Role: senior system architect + full-stack engineer.
    2. Task: build a merchant-facing UGC short-video generation SaaS from scratch.
    3. Working method (strict order, no skipping steps):
    4. 1) Research: break down the target users' core needs and willingness to pay,
    5. list the feature scope and what is explicitly out of scope.
    6. Output: requirements list + technology selection comparison matrix
    7. (with rationale for the trade-offs).
    8. 2) Architecture: data model (including multi-tenant fields), module
    9. boundaries, task state machine, sequence diagrams, failure mode list
    10. (FMEA) and corresponding fallback strategies. Wait for my confirmation
    11. before moving to the next step.
    12. 3) Development: split into milestones that can be verified independently,
    13. as separate commits. After each step, provide the acceptance method:
    14. executable tests, SQL validation scripts, or API call examples.
    15. Hard constraints:
    16. - Every business table must carry a tenant identifier, with row-level
    17. security enabled by default.
    18. - Billing uses hold + settle + rollback on failure; the credit ledger is
    19. append-only and never updated in place.
    20. - Secrets are allowed on the server side only; the frontend always accesses
    21. them indirectly through server-side APIs.
    22. - Any change touching permissions, billing, or data deletion must explicitly
    23. list the risks and the rollback plan.

    The value of this prompt isn’t elegant wording. It’s that it turns acceptance criteria into a contract up front. The model isn’t guessing whether I’ll like the result — it’s satisfying a defined set of conditions.

    4. Five Genuine Systems Engineering Test Points

    4.1 Multi-tenant isolation and row-level security

    The cheapest approach is to add a tenant_id filter manually to every query in the application layer, but that depends on developers never forgetting — and long-running generation is exactly where they forget. A more robust approach pushes isolation down into the data layer: every table carries a tenant identifier, the current tenant is injected through a session-level variable, and database policies enforce the filter. Only a privileged server-side role (the service role) can cross tenants, and that role’s credentials must never appear in a browser or mobile client.

    1. -- Illustrative: the session variable is the only tenant context;
    2. -- the business role can only see its own rows.
    3. ALTER TABLE videos ENABLE ROW LEVEL SECURITY;
    4. CREATE POLICY tenant_isolation ON videos
    5. USING (tenant_id = current_setting('app.tenant_id')::uuid);
    6. CREATE POLICY tenant_isolation_insert ON videos
    7. FOR INSERT WITH CHECK (tenant_id = current_setting('app.tenant_id')::uuid);

    Three follow-ups are required. First, write authorization test cases: using tenant A’s identity to read tenant B’s resources must return 404, not 403 — a 403 leaks the existence of the resource. Second, audit every cross-tenant code path (admin console, scheduled jobs, exports); these are usually the leaks. Third, isolate storage: object storage bucket paths or prefixes must include the tenant dimension, and downloads should always go through short-lived signed URLs.

    4.2 Making minute-scale tasks asynchronous with a state machine

    Operations like video rendering and model inference take far longer than HTTP timeouts tolerate, so waiting synchronously will always fail. The standard pattern has three parts: the API only enqueues the job and returns a job identifier immediately; a consumer picks up the task, calls the generation service, and writes back the result; the client polls or holds a long-lived connection to query progress.

    The engineering details that need to be explicit:

    • An explicit state machine, with no boolean flags standing in for state: queued → running → succeeded / failed. A retryable failure goes back to queued with an incremented attempt count.
    • Idempotency: the enqueue endpoint deduplicates on “business unique key + tenant,” so double-clicks or frontend retries don’t create duplicate tasks.
    • Timeout reclamation: once a consumer picks up a task, it needs a lease or visibility timeout. Tasks not acknowledged in time return to the queue automatically, so a crashed process can’t leave a job stuck forever.
    • Backoff and dead letters: distinguish retryable errors (rate limiting, transient network failures) from non-retryable ones (non-compliant assets, invalid parameters). The latter terminate immediately with a message to the user rather than wasting compute.
    • Cost guardrails: per-tenant concurrency limits and daily quotas are checked before enqueueing, so one runaway loop can’t burn through the entire model budget.
    1. // Illustrative: idempotent enqueueing and state advancement.
    2. async function submitJob(tenantId: string, req: GenerateReq) {
    3. const key = `${tenantId}:${req.assetId}:${req.templateVersion}`;
    4. const existing = await jobs.findIdempotent(key);
    5. if (existing) return existing; // idempotent return, no double billing
    6. await ledger.hold(tenantId, COST_PER_VIDEO); // hold first, prevent overspend
    7. return jobs.enqueue({
    8. id: newId(), key, tenantId,
    9. state: 'queued', attempts: 0,
    10. payload: sanitize(req), // allowlist validation for params
    11. });
    12. }

    4.3 Identity, sessions, and abuse prevention

    CAPTCHA-based login looks simple, but shipping it commercially means filling in a fair number of gaps: rate limiting on sends plus graphical or behavioral challenges to stop abuse, one-time consumption of each code, separation of short-lived access tokens from long-lived refresh tokens, and token revocation with multi-device management. Keep access token lifetimes in the minutes range and make refresh tokens revocable records in storage — otherwise banning an account doesn’t actually take effect. Also, any resource that is “visible after login” should be re-verified in the data layer using the session context, not only in the API layer.

    4.4 Metering and billing: the hardest part, and the easiest to overlook

    This is the part of the system that most resembles money. Three points matter.

    The ledger is append-only. A credit balance isn’t a field you can freely UPDATE; it’s an immutable sequence of entries (top-ups, holds, settlements, refunds), and the balance is derived by aggregating them. Any reconciliation or customer dispute has a trail to point at.

    A closed hold–settle–rollback loop. Freeze the estimated cost before enqueueing, settle the actual cost when the task succeeds and refund the difference, and release the full amount on permanent failure. Each step must happen inside the same database transaction as the state update, and must tolerate repeated delivery without charging twice.

    Failure is data, too. Generation failure rates, average retry counts, and actual cost per video need to land in metrics, because they directly determine gross margin. For a business selling video generation to merchants, cost observability is margin observability.

    4.5 Storage and cloud-side supporting services

    Finished videos, source assets, and cover images belong in object storage, paired with content delivery and lifecycle policies that clean up expired intermediate artifacts. The queue can start as a database table with lease-based polling, then move to a message queue service once concurrency grows. Containerized deployment makes it easy to scale consumers based on queue length. Logs, metrics, and alerts need to tie a single request, a job identifier, and a billing action together. Building a comparable system on a cloud platform such as Baidu AI Cloud maps onto general product categories — object storage, message queue, container platform, log service, monitoring and alerting — and the architecture shape and selection logic are essentially the same: get the pipeline working with the simple option first, then replace components once load-test data supports it.

    5. Hands-On Observations: Where the Time Actually Went

    The whole experiment ran for about three hours end to end, but the time wasn’t consumed by “generating code.” It went to three things: waiting on model output, making trade-offs during the architecture phase, and verification plus rework after each milestone. This is worth emphasizing — the efficiency bottleneck of long-horizon tasks shifts from generation speed to decision and verification speed. When writing code is nearly free, human value concentrates on defining the problem, deciding the architecture, and reviewing security and billing: the irreversible decisions.

    The other clear impression was that service response slowed noticeably at peak times, and jobs spent longer waiting in the queue than actually generating. That in itself is a reminder: for a self-built system like this, quotas, queueing, and degradation strategies have to be designed before the features ship, not after.

    6. Pitfalls

    • Detail loss after context compaction. As turns accumulate, the model forgets early agreements, and field names and enum values drift apart. Countermeasure: put key conventions into spec documents and type definitions in the repository, so every change gets caught by static checks.
    • Repeated scaffolding rewrites. Without clear milestones, the model tends to tear everything down and start over. Countermeasure: require documents before code, and place confirmation gates between phases.
    • Migrations drifting from the data model. The database already has the new column; the code still uses the old field. Countermeasure: keep migrations and model changes in the same commit, and run migration validation in CI.
    • Security defaults quietly disabled. To get debugging working, row-level security gets temporarily turned off and privileged credentials get pushed into frontend environment variables — and then never restored. Countermeasure: make these switches hard startup validations that refuse to boot when misconfigured.
    • No checkpoints in long tasks. An interruption means starting from scratch. Countermeasure: commit at every milestone and keep the acceptance record, so resuming means reading the docs first and the code second.

    7. A Practical Checklist for Teams

    1. Specs before code. Requirements scope, explicit non-goals, and acceptance criteria — all three are mandatory.
    2. Acceptance must be executable. If it can be expressed as a test, a SQL validation script, or an API example, don’t leave it as prose.
    3. Default to deny. Tenant isolation enforced in the data layer, authorization checks done server-side, secrets never shipped to clients.
    4. Fix the human review points. Authentication, authorization, billing, data deletion, and migrations: no matter how confident the model is, these five categories get human review.
    5. Load-test instead of guessing. Concurrency limits, queue backlog, failure retry rates, and cost per item all need numbers before you tune parameters.

    8. Conclusion

    The point of this experiment isn’t what got built in three hours. It’s that it validated a different division of labor: describe the spec, the constraints, and the acceptance criteria clearly enough, and a long-horizon coding tool can take on most of the implementation, while the human moves back to architecture decisions and security sign-off. The real bar is shifting from “can you write code” to “can you articulate system boundaries, failure modes, and the cost model.” Whoever can articulate that clearly is the one who can hand long tasks to a machine and trust them to finish.