AI Coding Long-Horizon Test: Building a Video Generation SaaS MVP
This article reports a hands-on test of an AI coding agent on a long-horizon task: building a verifiable video generation SaaS MVP from scratch, covering image upload, job submission, status query, result retrieval, and credit billing. It explains why this is fundamentally a system architecture problem rather than a coding problem, and shows how spec-driven development—structured research, architecture, and staged development—keeps long tasks from drifting. The core modules are covered in detail: API contracts that avoid synchronous blocking, PostgreSQL row-level security for tenant isolation, an asynchronous task state machine with idempotency, and consistent auth and billing that prevents financial loss. It closes with lessons learned from long-horizon runs and an MVP acceptance checklist spanning cost, throughput, state transitions, refund compensation, tracing, and compliance.
Introduction: The Yardstick for AI Coding Is Being Redefined
For a long time, evaluating a coding assistant meant asking narrow questions: “Did it autocomplete that line correctly?” “Does the generated function look right?” But real software engineering is never a pile of scattered fragments. It is a complete chain that runs from requirements and architecture through implementation and verification.
The industry’s focus is now shifting from single-point code generation to three more fundamental dimensions: agent capability, long-horizon task capability, and systems engineering capability. A truly convincing test is not making an AI write a polished landing page—it is making it deliver a small system that runs, can be accepted, and can keep evolving.
So the goal of this experiment was not to “generate code that looks like a SaaS,” but to build a verifiable video generation SaaS MVP. To avoid dressing the experiment up as an unreproducible “miracle benchmark,” here is the test methodology up front.
Test Methodology: Making the Results Verifiable
- Tool under test: An AI coding agent with tool calling, terminal execution, code retrieval, and multi-file editing. The model version is whatever the platform provides at the time.
- Task input: Build a minimum viable version of a short-video / product-marketing video generation SaaS from scratch, covering image upload, job submission, status query, result retrieval, and credit billing.
- Execution time: Measured as resumable work; the core build took roughly 4–6 hours. If resource queuing or context rebuilds occur, it takes noticeably longer.
- Delivered modules: Backend API, database schema, asynchronous task queue, auth and billing, object storage integration, and a basic content moderation interface.
- Acceptance result: The main flow runs end to end—users upload assets, submit jobs, generate asynchronously, query status, retrieve results, and get credits deducted.
- Failures recorded: Early on there were query authorization leaks, refund failures when enqueueing failed, and duplicate job consumption causing repeated processing. These were fixed with row-level security, idempotency keys, compensating transactions, and state-machine checks.
One thing to stress: “production-ready” here means engineering-wise pilotable, verifiable, and extensible—not that production-grade compliance is done. Truly commercial operation still requires content moderation, licensing, payment compliance, privacy protection, and a risk-control system.
Why Choose a Video Generation SaaS as the Testbed?
Video generation SaaS was not chosen because it looks trendy, but because it simultaneously covers four kinds of engineering problems: cost, throughput, experience, and compliance.
Mainstream video generation models are improving fast, and cross-border and content-commerce demand for “batch, low-cost, scalable” marketing video assets is clearly growing. But here is a counterintuitive fact: the merchants who actually pay usually do not want to learn a complex workflow. They have been worn down by visual workflow editors and drag-and-drop automation platforms, and in the end they care about only three things:
- Cost: Is each video a few cents, a few dollars, or an unpredictable amount?
- Throughput: Can it reliably run hundreds or thousands of jobs a day?
- Experience: Can they upload an image, click generate, and get a finished video?
If you are searching for “how to build a video generation SaaS,” “what is spec-driven development,” or “common AI coding mistakes,” this article gives an actionable answer: Do not treat AI coding as one-sentence website generation—treat it as a system delivery with specs, acceptance criteria, and compensation mechanisms.
In the AI era, instead of teaching users “how to fish,” it is better to serve them the fish directly. Hiding complex model calls, parameter orchestration, and task scheduling behind the platform and leaving deterministic delivery to the user—that is exactly where the value of SaaS lies.
The Key Insight: This Is an Architecture Problem, Not a Coding Problem
If you treat the task as just “writing code,” you easily end up with a toy deliverable. A pilotable, commercially viable video generation SaaS concentrates its complexity in four coupled modules: backend services, data modeling, asynchronous task scheduling, and auth & billing.
This experiment fixed the MVP boundary clearly: only do four things—upload, generate, query, and bill. It does not include complex canvas editing, multi-user collaboration, fine-grained permission systems, complex reporting, or a multi-model orchestration marketplace. Those capabilities can evolve later, but they must not balloon in the first version.
| Module | Complexity | Common Failures | How AI Is Constrained | Acceptance Metric |
|---|---|---|---|---|
| Backend services | Medium-high | Sync blocking, endpoint authorization leaks, chaotic error codes | API-contract-first, constrained response structure | Main-flow success rate, error rate |
| Data design | High | Multi-tenant leaks, state inconsistency | RLS enabled, clear data dictionary | Cross-user queries must be 0 |
| Async tasks | High | Duplicate consumption, stalls, no refunds | State machine, idempotency, timeout retries | Job terminal states converge |
| Auth & billing | Very high | Duplicate charges, failed charges, financial loss | Transactions, conditional updates, idempotency keys | Financial-loss events must be 0 |
The conclusion: this is fundamentally a system architecture problem. I used to think no team smaller than five people should touch this kind of project; this experiment set out to verify whether a strong enough coding agent can shoulder this architecture independently under clear specs and acceptance constraints.
Methodology: Giving AI an “Architect’s Brain” with Spec-Driven Development
Handing over a single line—“build me a video generation website”—usually only yields a toy. A more reliable approach is spec-driven development: first make the AI externalize its thinking into a structured spec, then advance along a fixed pipeline.
A three-stage flow is recommended: research → architecture → development.
- Research stage: Produce a feature breakdown of similar products, target user personas, core business flows, and the MVP boundary—making explicit what to do and what not to do.
- Architecture stage: Produce technology choices, module divisions, the database ER model, API contracts, the task state machine, and error codes.
- Development stage: Implement module by module under the architecture document as a constraint; after each block, self-test and self-verify.
The spec document should at least contain these fields:
- Goals: What this delivery must accomplish.
- Non-goals: Which capabilities are explicitly out of scope.
- API contracts: Paths, requests, responses, and error codes.
- Data dictionary: Tables, fields, indexes, and constraints.
- State machine: How jobs transition, and when refunds happen.
- Acceptance criteria: How to prove a module works.
The value of this process lies in constraints and anchors. The biggest enemy of long-horizon tasks is goal drift: forgetting the early design as you write further. An explicit spec document becomes a single source of truth spanning hours, letting the AI look back at constraints at every step and stay coherent.
Implementation Details for the Four Core Modules
1. Backend Services and API Contracts: How to Avoid Synchronous Blocking?
The backend is modeled around resources, designed around the flow “asset upload → job submission → status query → result retrieval.” The core principle: synchronous endpoints only validate and accept; asynchronous jobs handle the heavy compute.
Example prerequisites:
- Python 3.11
- FastAPI and Uvicorn
- PostgreSQL 16
- SQLAlchemy 2.x async ORM with asyncpg
- A Redis queue, e.g., Arq, RQ, or Celery
- Object storage via signed URLs, so assets are not directly public
The example code highlights asset validation, idempotency keys, insufficient credits, enqueue failures, and compensating refunds.
from uuid import uuid4from fastapi import HTTPExceptionasync def submit_video_job(user_id: UUID, image_url: str, idempotency_key: str):if not idempotency_key:raise HTTPException(400, "Missing idempotency key")# Idempotency check: the same request from the same user must not create duplicate jobsexisting = await repo.find_job_by_idempotency(user_id, idempotency_key)if existing:return {"job_id": existing.id, "status": existing.status}# Asset validation: domain allowlist, MIME, size, temporary URL, virus scanawait validate_asset(image_url)job_id = uuid4()try:async with db.transaction():# Conditional update: affects 0 rows when the balance is insufficientupdated = await db.execute(update_users_set_credits(cost=10).where(users.id == user_id, users.credits >= 10).returning(users.id))if not updated.scalar():raise HTTPException(402, "Insufficient credits")await db.execute(insert_jobs(id=job_id,user_id=user_id,image_url=image_url,credits_cost=10,status="pending",idempotency_key=idempotency_key))except IntegrityError:# On concurrent idempotency-key conflict, return the existing jobexisting = await repo.find_job_by_idempotency(user_id, idempotency_key)return {"job_id": existing.id, "status": existing.status}try:await queue.enqueue("video_generate", {"job_id": str(job_id)})except QueueError:# Enqueue-failure compensation: mark failed and refund creditsawait compensation.refund_and_mark_failed(job_id, user_id, 10)raise HTTPException(503, "Failed to enqueue job; refund compensation initiated")return {"job_id": str(job_id), "status": "pending"}
A more robust production practice is to introduce the Outbox pattern: write a pending event in the same transaction that deducts credits and creates the job, then have a background relay deliver it to the queue. This avoids “the transaction committed but the process crashed, so the enqueue was missed.”
2. Data Model and Row-Level Security: How to Make the Database Guarantee Tenant Isolation?
Core entities can be split into users / jobs / assets, with the jobs table carrying the state machine and billing fields. Multi-tenant isolation must not rely only on an application-layer WHERE user_id = ?, because a single missing condition in one query causes an authorization leak.
Use PostgreSQL row-level security to make “every row belongs to some user_id” a hard database constraint.
CREATE TABLE jobs (id uuid PRIMARY KEY,user_id uuid NOT NULL,image_url text NOT NULL,status text NOT NULL DEFAULT 'pending',credits_cost int NOT NULL DEFAULT 0,idempotency_key text NOT NULL,created_at timestamp NOT NULL DEFAULT now());CREATE UNIQUE INDEX jobs_idempotency_uidxON jobs (user_id, idempotency_key);ALTER TABLE jobs ENABLE ROW LEVEL SECURITY;ALTER TABLE jobs FORCE ROW LEVEL SECURITY;CREATE OR REPLACE FUNCTION current_user_id()RETURNS uuidLANGUAGE sqlSTABLEAS $$SELECT nullif(current_setting('app.user_id', true), '')::uuid;$$;CREATE POLICY jobs_owner ON jobsFOR ALL TO authenticatedUSING (user_id = current_user_id())WITH CHECK (user_id = current_user_id());
Key points:
- Default deny: After enabling RLS, if no policy matches, ordinary users should not see data.
- Identity injection: The gateway or backend sets
app.user_idin the server-side session; clients must not be able to pass it arbitrarily. - Least privilege: The database connection role must not have high privileges that bypass RLS; the admin role should be managed separately.
- Integration testing: You must write cross-user tests, e.g., user A querying user B’s jobs must return empty or 403.
3. How to Design the Asynchronous Task State Machine?
Video generation is a minutes-long heavy task and must never block the request thread. Job status should transition strictly as:
pending → processing → succeeded / failed
| Current State | Event | Handling Action | Failure / Compensation Strategy |
|---|---|---|---|
| pending | Worker picks up | Change to processing, set timeout | Retry if not picked up before timeout |
| processing | Generation succeeds | Validate output, then change to succeeded | Missing output → move to failed |
| processing | Generation fails | Decide whether retryable | Retry if under limit; refund if over limit |
| failed | Terminal state | Stop retrying, release resources | If charged but no output, must refund |
| succeeded | Terminal state | Return the finished video URL | Do not charge again |
The key point is idempotency. Queue messages may be delivered more than once, and the handler must recognize “this job has already been processed” to avoid duplicate billing, duplicate generation, or duplicate refunds. A combination of job unique keys, database unique constraints, processing logs, and state locks can guarantee this.
4. Auth and Billing Consistency: How to Avoid Financial Loss?
For auth, stateless tokens are recommended: log in to issue a token, and the gateway validates the token and injects user identity. The login step can add basic risk controls such as captcha, device fingerprinting, and rate limiting.
Billing is the high-risk area for financial loss. These principles must be followed:
- Deduct credits and create the job in the same transaction whenever possible.
- Use conditional updates for credit deduction, so jobs are not created when the balance is insufficient.
- On job failure, run compensating refunds; the refund must also be idempotent.
- Write a ledger entry for every deduction and refund, for reconciliation.
- Never rely on the frontend alone to check the balance.
Meanwhile, object storage and privacy must also be part of the design:
- Uploads use signed URLs to avoid public buckets.
- Set expiry on asset links to avoid long-term exposure.
- Mask user identifiers, tokens, and signature parameters in logs.
- Define asset deletion and retention policies; support user deletion and compliance deletion.
- If faces, portraits, or brand assets are involved, require users to affirm copyright and authorization up front.
Pitfalls and Lessons from Long-Horizon Tasks
Once it actually runs, the problems are often not “can it write code,” but the consistency, resource constraints, and verification cadence of long-horizon tasks.
- Context coherence is the ceiling: The further a task goes, the more easily it forgets early constraints. The solution is not to rely on the model’s memory but to solidify key constraints into a spec document and have the model explicitly review them at the end of each stage.
- Resource contention slows the pace: During peak times, compute is tight and execution time can double. Long tasks must support resumable runs, failure recovery, and log tracing.
- Segmented validation beats one-shot delivery: Do not wait until the end to accept. Each time you finish a module, let it self-test and pass before moving on; this localizes errors to the most recent change and greatly reduces backtracking cost.
MVP Acceptance Checklist
| Acceptance Item | What to Check |
|---|---|
| Cost model | Per-video cost includes model calls, storage, queue, and retry costs |
| Concurrent throughput | Under target concurrency, the queue does not back up; success rate and latency are observable |
| Polling or callbacks | Frontend polling or a webhook can retrieve status; callbacks must be idempotent |
| Job state machine | All states transition, terminal states converge, no permanent pending |
| Refund compensation | Failed jobs can be refunded, refunds have ledger entries, duplicate refunds are blocked |
| Log tracing | Requests, jobs, queue, and model calls all carry a trace_id |
| Segmented validation | Each module has unit, integration, and authorization tests |
| Compliance boundaries | Asset moderation, copyright notices, deletion policy, and privacy statements can be implemented |
Conclusion: AI Coding Is Moving Toward “System-Level Delivery”
The most important takeaway from this experiment was not “how many lines of code it generated,” but that it verified a trend: when a coding agent has long-horizon task, tool-calling, and systems engineering capabilities, its role is upgrading from a code-completion tool to an engineering partner capable of architecture collaboration.
But there are two prerequisites for realizing this capability:
- Developers themselves must understand architecture. You need to know when to use RLS, when to wrap a transaction, when to enforce idempotency, and when to compensate with refunds—only then can you give the AI the correct spec.
- Use spec-driven development to constrain the process. Externalize the thinking, segment the acceptance, and make failures recoverable, so long tasks do not drift.
In other words, the AI is responsible for efficiently translating the architecture into code, while the judgment of system design stays firmly in the developer’s hands. That is what makes this round of AI coding evolution genuinely exciting.
