Building a Full-Stack Long-Running Agent for a Video SaaS: A Practical Evaluation
This article examines how long-running AI coding agents can help build a production-oriented UGC video generation SaaS. It covers specification-driven development, system architecture, asynchronous video jobs, state machines, multi-tenant isolation, authentication, credit accounting, retries, observability, security, cost controls, and acceptance criteria. The central conclusion is that the next stage of AI programming is not simply generating more code, but combining long-horizon execution with engineering controls, automated verification, and human review.
The Main Conclusion: The Next Stage of AI Programming Is Not “Writing More Code”
If AI programming is understood only as “generating a page from a single prompt,” it remains largely a demonstration. The real dividing line is whether an agent can work on a system-level objective over several hours: decomposing requirements, maintaining context, producing verifiable artifacts, diagnosing failures, rolling back changes, and continuing execution.
This article uses a more realistic business problem as its example: building a UGC video generation SaaS for short-form commerce content. Such a system needs, at minimum, asset uploads, video generation, job queuing, status queries, multi-tenant isolation, user authentication, credit billing, content moderation, retry handling, monitoring, alerting, and an operations console.
The challenge is not asking a model to generate a page. It is asking an agent to collaborate across the full lifecycle—from research and architecture to implementation and verification—under the constraints of Specification-Driven Development (SDD).
Long-running tasks do not mean sending one sentence to a model and waiting for a miracle. The longer the task, the more important engineering gates become. Without them, more code often means more hidden technical debt.
A commercial video SaaS must answer questions such as:
- How should backend services be divided, and which operations should be synchronous or asynchronous?
- How should a relational database represent tenants, users, assets, jobs, credit ledgers, and audit logs?
- Video generation can take minutes. How should queues, state transitions, timeouts, retries, and cancellation work?
- How can phone verification, token authentication, credit reservation, and settlement remain secure and idempotent?
- How can tenant data be isolated, and which queries must always include tenant conditions?
- How should content moderation, privacy, access control, and log retention be implemented?
- How can costs be controlled, failed jobs refunded, and operational issues investigated?
The following sections describe a complete implementation path using a general-purpose long-running agent and an abstract technical architecture. The examples use pseudocode and conceptual structures so they can be adapted to an existing technology stack.
1. Why Video Generation SaaS Is a Typical Long-Running System
The complexity of a video generation SaaS is not determined by the number of pages. It comes from the length of the business workflow, the number of possible states, cost sensitivity, and the need for reliable failure recovery.
A typical UGC video generation flow can be modeled as follows:
- The user uploads images, copy, or product information.
- The system validates parameters, performs content moderation, and preprocesses the assets.
- It creates a video generation job and reserves credits.
- The job enters an asynchronous queue. A worker calls a model service or video composition service.
- The generation process may require polling, transcoding, stitching, and quality checks.
- When successful, the system stores the result location, settles the credits, and notifies the user.
- When it fails, the system distinguishes retryable errors from permanent errors and refunds credits according to policy.
- Logs, metrics, and audit records are written throughout the workflow for operational diagnosis.
This workflow involves at least four categories of system capability:
- Business-domain modeling: tenants, users, assets, jobs, credits, orders, and audits;
- Asynchronous job processing: queues, leases, heartbeats, timeouts, retries, and dead-letter handling;
- Billing and ledgers: reservation, settlement, refunds, idempotency, and reconciliation;
- Security and governance: authentication, authorization, moderation, data isolation, and auditing.
If the system only generates a demo page with an “upload image and generate” button, many important problems remain hidden:
- Will duplicate submissions result in duplicate charges?
- Are credits returned when generation times out?
- Could one tenant read another tenant’s data?
- Can APIs return quickly when the queue is backed up?
These are the differences between a commercial system and a demonstration page.
2. Methodology: Using Specification-Driven Development to Control Long Tasks
Long-running agents commonly encounter three problems: context drift, invented interfaces, and insufficient verification. The solution is not to keep adding prompts. Instead, the project should be divided into reviewable specifications and stage gates.
2.1 Define Deliverables Before Writing Code
Before asking the agent to implement the system, require it to produce the following artifacts:
- A product requirements summary covering roles, core use cases, and non-goals;
- A research report covering common capabilities, risks, dependencies, and cost areas;
- An architecture document covering module boundaries, data flows, sequences, and failure models;
- A data model covering tables, constraints, indexes, and tenant-isolation rules;
- API contracts covering requests, responses, error codes, and idempotency semantics;
- A test plan covering unit, integration, end-to-end, and fault-injection tests;
- An operations runbook for queue backlogs, stuck jobs, and refund failures.
This changes the process from code generation to artifact-driven development. At every stage, the agent must produce something that can be reviewed instead of simply adding more code.
2.2 Use Stage Gates to Control Execution
An executable SDD workflow can look like this:
Stage 1: ResearchInput: business goals, user roles, constraintsOutput: use cases, risk list, non-functional requirements, abstract external dependenciesGate: requirements contain no unverifiable ambiguityStage 2: ArchitectureInput: use cases and constraintsOutput: module diagram, sequence diagrams, data model, state machine, error codesGate: every core use case maps to an API, table, or asynchronous jobStage 3: DevelopmentInput: architecture artifacts, API contracts, test planOutput: vertical-slice code, migration scripts, tests, change notesGate: the upload -> create job -> generate -> query -> bill flow is completeStage 4: VerificationInput: code, test plan, monitoring metricsOutput: test report, load-test results, failure-drill records, unresolved issuesGate: idempotency, retries, refunds, and tenant isolation all have test coverage
The instructions to the agent should explicitly require it to provide a plan before every modification and summarize the differences afterward. Specification documents should be treated as the single source of truth.
Long conversations are not reliable memory. A specification repository is more dependable. For complex projects, the agent can maintain documents such as spec.md, architecture.md, tasks.md, and test-plan.md, then generate a milestone summary containing changes, incomplete work, and risks.
2.3 Start With a Vertical Slice
Long-running systems are better developed by completing one end-to-end vertical slice before expanding horizontally:
User uploads an asset-> Create generation job-> Reserve credits-> Consume asynchronous queue-> Simulate success or failure-> Query job status-> Settle or refund credits-> Write audit log
Once this slice works, the team can add batch generation, templates, an operations console, and reporting. Otherwise, the project may end up with many pages but no functioning primary workflow.
3. Overall Architecture: Split the Long Workflow Into Recoverable Modules
A conceptual architecture can be represented as follows:
Client / Operations Console|vApplication Layer: authentication, tenant context, validation,job orchestration, credit APIs|+----> Relational Database: tenants, users, assets, jobs,| credit ledger, audit logs|+----> Object Storage: original assets, intermediate files,| final videos, moderation screenshots|+----> Message Queue: generation jobs, moderation jobs,| notifications, retry jobs|+----> Workers: preprocessing, model adapters, composition,| transcoding, quality checks|+----> Observability: structured logs, metrics, tracing,alerts, auditing
The application layer should not perform time-consuming video generation directly. It should validate input, create a job, and return a job identifier. Workers consume queued jobs and update their states.
Object storage should hold large files, while the database stores metadata and access policies. The message queue provides load smoothing and retry handling. The observability system connects users, tenants, jobs, and errors across the entire workflow.
3.1 The Core State Machine
A video job needs an explicit state machine rather than a single Boolean such as completed.
DRAFT -> QUEUED -> RUNNING -> SUCCEEDED|+-> FAILED_RETRYABLE -> WAIT_RETRY -> QUEUED|+-> FAILED_PERMANENT|+-> CANCELLED
Example pseudocode:
allowed = {DRAFT: [QUEUED, CANCELLED],QUEUED: [RUNNING, CANCELLED],RUNNING: [SUCCEEDED, FAILED_RETRYABLE, FAILED_PERMANENT, CANCELLED],FAILED_RETRYABLE: [WAIT_RETRY, FAILED_PERMANENT],WAIT_RETRY: [QUEUED, CANCELLED]}function transition(job, next, reason):assert next in allowed[job.status]if job.status == RUNNING and next == SUCCEEDED:assert job.result_url existsupdate_job_with_version(job.id, job.version, next, reason)append_audit_log(job.id, job.status, next, reason)
A state machine gives stuck jobs, duplicate consumption, manual intervention, refunds, and cancellation a defined path. Every state change should include a version number to prevent concurrent updates from overwriting one another. Each transition should also write an audit record so the system can answer who changed a job, when it changed, and why.
3.2 Queues and Workers
A queue message should contain at least:
- Job identifier;
- Tenant identifier;
- Retry count;
- Idempotency key;
- Version number.
After consuming a message, the worker should acquire a lease, execute the job, and periodically renew its heartbeat. If the worker crashes, the lease expires and the job can be delivered again.
function consume(message):job = load_job(message.job_id)if job.status in [SUCCEEDED, CANCELLED]:ack(message)returnlease = acquire_lease(job.id, worker_id, ttl=60s)if not lease:delay(message, 10s)returntry:mark_running(job.id, lease.version)result = run_pipeline(job)complete_job(job.id, result, lease.version)settle_credit(job.id)ack(message)catch TransientError as e:release_lease(lease)schedule_retry(job.id, backoff(e))ack(message)catch PermanentError as e:fail_job_permanently(job.id, e)refund_credit(job.id)ack(message)
Retries must distinguish error types. Network instability, downstream throttling, and temporary service unavailability may be retryable. Invalid parameters, policy violations, and corrupted assets generally are not.
Retries also need a maximum count and a backoff strategy. Jobs that exceed the limit should enter a dead-letter or manual-review queue.
4. Data Modeling and Multi-Tenant Isolation
A video SaaS typically serves multiple tenants. Each tenant has separate users, assets, jobs, and credits. Every business table should include tenant_id, and every query should include a tenant condition.
If the database supports row-level security, it can provide a second layer of protection. Otherwise, tenant filtering should be centralized in the data-access layer, and business code should not be allowed to issue unrestricted queries.
An abstract data model might look like this:
tenantid, name, status, created_atuserid, tenant_id, phone_hash, role, status, created_atunique(tenant_id, phone_hash)assetid, tenant_id, user_id, object_key, media_type, size, status, created_atvideo_jobid, tenant_id, user_id, input_asset_id, status,prompt_spec, idempotency_key, reserved_credit, settled_credit,result_object_key, error_code, retry_count,version, created_at, updated_atunique(tenant_id, idempotency_key)job_attemptid, job_id, worker_id, attempt_no, status, error_code,started_at, finished_atcredit_ledgerid, tenant_id, user_id, job_id, entry_type,amount, balance_after, created_ataudit_logid, tenant_id, actor_id, action, resource_type, resource_id,detail, created_at
4.1 Use a Credit Ledger Instead of Direct Balance Updates
The most dangerous credit implementation is:
Read balance -> check balance -> update balance
This is vulnerable to race conditions. A more reliable approach is a ledger model. Every top-up, reservation, settlement, refund, and manual adjustment creates a ledger entry. The balance can be derived from the ledger or maintained in a controlled balance table with ledger-based reconciliation.
function reserve_credit(tenant_id, user_id, job_id, amount, idempotency_key):if exists_ledger(idempotency_key):return existing_resultwith transaction:available = calculate_available(tenant_id, user_id)assert available >= amountinsert_ledger(tenant_id, user_id, job_id,type=RESERVE, amount=-amount,idempotency_key=idempotency_key)return successfunction settle_credit(job_id, actual_amount):reservation = find_reserve_by_job(job_id)if reservation.settled:returnwith transaction:unused = reservation.amount - actual_amountinsert_ledger(job_id, type=SETTLE, amount=0)if unused > 0:insert_ledger(job_id, type=REFUND, amount=unused)mark_reservation_settled(job_id)
Reservation and settlement provide several benefits:
- The system prevents overspending when a job is submitted;
- Successful jobs can be charged based on actual usage;
- Failed jobs can be refunded;
- Idempotency keys prevent duplicate financial operations.
All credit operations should be idempotent. Repeated requests must not create duplicate charges or refunds.
4.2 Apply Idempotency at Business Boundaries
The job-creation API must support idempotency. The client submits an idempotency key, and the server enforces a unique constraint on the combination of tenant and key. A repeated request returns the original job instead of creating another one.
function create_job(request, tenant_context):validate_asset_owner(request.asset_id, tenant_context.tenant_id)existing = find_job(tenant_context.tenant_id, request.idempotency_key)if existing:return existingwith transaction:job = insert_job(tenant_id=tenant_context.tenant_id,idempotency_key=request.idempotency_key,status=DRAFT)reserve_credit(tenant_context.tenant_id,tenant_context.user_id,job.id,request.estimated_credit,idempotency_key="reserve:" + job.id)transition(job, QUEUED, "submitted")enqueue_job(job.id)return job
Idempotency should not exist only at the API layer. Queue consumption, settlement, refunds, and notifications also need idempotent behavior. Otherwise, retries can amplify failures.
5. Authentication, Verification Codes, and Access Control
A consumer-facing SaaS may use phone verification, token authentication, role-based access, and tenant context. The principles are straightforward:
- Verification codes should expire quickly, be single-use, and be rate-limited;
- Access tokens should have short lifetimes and be refreshable;
- Every request should resolve the user and tenant identity;
- Authorization must be enforced server-side rather than relying on hidden frontend controls.
5.1 Verification-Code Flow
function send_code(phone, scene):rate_limit(phone, scene)code = random_digits(6)store_code(hash(phone + scene), hash(code), ttl=5min, max_attempts=5)send_sms(phone, code)audit("send_code", phone_hash=hash(phone), scene=scene)function verify_code(phone, scene, code):record = load_code(hash(phone + scene))assert record exists and not expiredassert record.attempts < 5increment_attempts(record)if hash(code) != record.code_hash:fail("INVALID_CODE")consume_code(record)issue_access_token_and_refresh_token(user)
Important details include:
- Store only a hash or controlled encrypted value for the code;
- Rate-limit both sending and verification;
- Invalidate the code immediately after successful use;
- Write login events to the audit log;
- Trigger risk controls when abnormal request frequency is detected.
5.2 Tenant Context and Roles
After authentication, the application should build a tenant context:
tenant_context = {tenant_id,user_id,role,permissions}
All data access should derive the tenant identifier from this context. An administrator may manage users and jobs within the current tenant but must not query other tenants.
If the operations console needs cross-tenant visibility, it should use a separate, controlled service identity and an audited workflow rather than reusing ordinary business APIs.
6. How a Long-Running Agent Maintains Context and Quality
The value of a long-running agent is not that it generates every file in one pass. Its value is maintaining consistency between plans and implementation across multiple milestones.
A practical approach combines a specification repository, a task list, and a verification loop.
6.1 Treat the Specification Repository as the Source of Truth
Recommended files include:
product.md: goals, users, scope, and non-goals;architecture.md: modules, sequences, state machines, and failure models;data-model.md: tables, constraints, indexes, and isolation rules;api-contract.md: APIs, error codes, and idempotency semantics;test-plan.md: test cases, fault drills, and acceptance criteria;tasks.md: task list, dependencies, and completion status;decisions.md: architecture decisions and the reasons behind them.
Before modifying code, the agent should read the relevant specifications and produce a change plan. After the change, it should update task status and record test results. This reduces context drift.
6.2 Require Verification in Every Iteration
A usable iteration loop looks like this:
Read specifications-> Select the smallest task-> Write or update tests-> Implement the code-> Run static checks and tests-> Generate a change summary-> Update the task list-> Record risks and unresolved questions
For long-running projects, testing is not a final stage. It is part of the completion criteria for every slice.
At minimum, test coverage should include:
- Parameter validation;
- Authorization and tenant isolation;
- Idempotency;
- Invalid state transitions;
- Queue retry behavior;
- Credit reservation, settlement, and refunds;
- Object-storage access control;
- Integration tests for critical APIs.
6.3 Prevent Invented APIs
Long-running agents may invent endpoints, fields, or external capabilities. Contract-first development helps prevent this:
- Write API contracts and data models before implementation;
- Use adapters and mocks for external dependencies;
- Validate request and response schemas;
- Review whether every new field is reflected in the contract;
- Review migration scripts and rehearse rollback procedures.
7. Observability, Recovery, and the Operations Runbook
A commercial system must answer not only “does the feature work?” but also “can the team locate and recover from failures?”
7.1 Logs, Metrics, and Tracing
Recommended log fields include:
request_id, tenant_id, user_id, job_id, trace_id,module, action, status, error_code, latency_ms
Core metrics include:
- Latency and error rate for job-creation APIs;
- Queue depth, waiting time, and consumption rate;
- Job duration distribution, success rate, retry rate, and dead-letter count;
- Average cost per video, credit consumption, and refund rate;
- Object-storage usage and lifecycle cleanup;
- Content-moderation queue length and rejection rate;
- Tenant-level quota usage and abnormal traffic.
Alerts should focus on user impact, such as:
- Jobs waiting beyond a threshold;
- A sustained increase in failure rate;
- Abnormal refund activity;
- Queue depth exceeding capacity.
Monitoring only machine resources is not enough.
7.2 Failure Recovery
Typical failures and responses include:
- Worker crash: the lease expires and the job is returned to the queue;
- Downstream throttling: use exponential backoff and lower concurrency;
- Job stuck in
RUNNING: a timeout scanner marks it failed or requeues it; - Duplicate consumption: idempotency and the state machine ensure only one successful outcome;
- Credit settlement failure: retry ledger operations and reconcile them through operations tooling;
- Object-storage failure: retry the write and verify the result before returning an inaccessible URL;
- Moderation rejection: terminate the job, refund credits, and preserve the required audit record.
7.3 Operations Capabilities
The operations console should support:
- Searching by tenant, user, and job status;
- Viewing job timelines and audit records;
- Safely retrying or cancelling stuck jobs;
- Reviewing credit transactions and handling refund exceptions;
- Viewing moderation records;
- Configuring tenant quotas and rate limits.
Operations actions must also be audited. Manual retries must respect the state machine, and operators should never directly modify database balances.
8. Security, Compliance, and Cost Control
Video generation involves user assets, generated content, and credit balances. Security and compliance therefore need to be considered from the beginning.
8.1 Content and Asset Security
- Validate asset type, size, and integrity during upload;
- Moderate content both before and after generation;
- Retain only the audit information that is necessary;
- Use short-lived signed URLs to access generated videos;
- Isolate object storage by tenant prefix and verify ownership server-side.
8.2 Data Isolation and Privacy
- Include tenant identifiers in all business tables;
- Inject tenant conditions consistently in the data-access layer;
- Encrypt or hash sensitive fields;
- Avoid logging complete phone numbers, tokens, or unrestricted asset URLs;
- Require approval and auditing for data export and deletion.
8.3 Cost Controls
Video generation is cost-sensitive. Controls should include:
- Preflight validation to reduce invalid jobs;
- Daily quotas and concurrency limits by tenant;
- Maximum retry counts;
- Resource-consumption records for every attempt;
- Cost reporting by job, tenant, and model type;
- Alerts for abnormal usage;
- Refund and compensation policies so users do not pay for failed jobs.
9. Acceptance Criteria: Distinguishing a Demo From a Commercial System
The following checklist can help determine whether the system is genuinely deliverable:
- Functional flow: asset upload, job creation, asynchronous generation, status queries, credit settlement, and result viewing all work end to end;
- Idempotency: duplicate submissions, duplicate consumption, and duplicate settlement do not create duplicate jobs or charges;
- State machine: illegal transitions are rejected and all changes are auditable;
- Tenant isolation: cross-tenant queries are blocked and authorization tests provide coverage;
- Asynchronous reliability: worker crashes, queue backlogs, and downstream throttling can be recovered from;
- Billing consistency: reservations, settlement, and refunds produce auditable ledger entries;
- Security and governance: verification-code rate limiting, token validation, moderation, and access controls are implemented;
- Observability: logs, metrics, traces, and alerts make failures diagnosable;
- Operations: the console supports search, retry, cancellation, refunds, and auditing;
- Cost control: quotas, retry limits, cost reporting, and abnormal-usage alerts are in place.
Only when these conditions are met can the result reasonably be called a video generation SaaS built from scratch. Otherwise, it remains a demonstration page.
10. Conclusion: Long-Running Agents Amplify Engineering Capability
The greatest value of using a long-running agent to build a full-stack SaaS is not merely saving a few hours of coding. It is placing requirements, architecture, code, tests, and operations documentation into one traceable workflow.
An agent can quickly produce research outlines, API contracts, initial data models, vertical-slice implementations, and test scaffolding. This can significantly shorten the path from an idea to a runnable prototype.
However, the engineering team remains responsible for the commercial system’s boundaries:
- Is the state machine rigorous?
- Can credits be reconciled?
- Are tenants isolated?
- Are failures recoverable?
- Are logs traceable?
- Is generated content governed appropriately?
- Are costs controlled?
AI can participate in generation, but it cannot replace accountability.
The more accurate description of the next stage of AI programming is therefore not “models write all the code for humans.” It is a combination of specification-driven development, long-running task execution, automated verification, and targeted human review.
Teams that can make this combination work are the ones most likely to realize the productivity gains of long-running AI agents.
