E-Commerce Short-Video Script Screening Pipeline: Filtering Batch AI-Generated Scripts Before You Produce

1. The Overlooked Cost Structure: Generation and Production Are Not in the Same League

When people discuss AI script writing, most of the attention goes to “which model writes better.” But what actually determines return on investment is the cost gap between two different kinds of work.

A 30-second short-video script is usually a few hundred characters of text: a shot list, dialogue lines, action cues, and the placement of the selling point. Generating pure text like this is cheap enough that its cost is measured in fractions of a cent. Once you move into video production, cost items stack up quickly: asset preparation or product-image processing, voiceover and lip-sync synthesis, digital human or live-action shooting, editing and transcoding, subtitles and sound effects, plus internal review and compliance checks. Run that whole chain and the cost of a single finished video jumps by one to two orders of magnitude.

Those two numbers being one to two orders of magnitude apart is the fulcrum of the entire pipeline design:

• A script is a cheap trial-and-error artifact. Writing 100 more of them can cost less than reworking a single finished video.
• A finished video is an expensive commitment. Producing everything in full simply hands the decision to randomness.

A schematic model makes this clearer. Let c_s be the unit cost of a script, c_v the unit cost of a finished video, and p the hit rate (the share of candidates that actually perform to standard). Producing all n candidates costs n × c_v. Filtering first, compressing n down to k for production, and assuming the screener retains most of the genuinely good candidates, costs n × c_s + k × c_v. Because c_v ≫ c_s, even a mediocre screener — say one that only lifts the hit rate from p to 2p — saves far more production budget at high volume than the screening itself consumes.

This leads to a conclusion that is easy to miss: spending budget on a stronger generation model raises the probability that a good script exists in the candidate pool, while spending it on more accurate screening raises the probability of picking a good script out of that pool. The second one acts directly on the most expensive line item in the cost structure.

2. Model Selection: Genre Distribution Determines Which Model Is “Best”

The most common observation in practice is that the highest-priced model does not necessarily produce the most usable output. That is hard to explain in terms of “capability levels,” but natural once you think in terms of distribution match.

Generation quality can be roughly decomposed into: how densely the training corpus covers a specific domain, which output preferences were reinforced during alignment, the decoding strategy (temperature, nucleus sampling parameters), and how well the model adapts to a given prompt structure. E-commerce short drama is a highly localized genre. It depends on the rhythm and particles of spoken Chinese, regionally flavored phrasing, the social consensus around family relationships, and the unwritten red lines of platform content moderation.

When a model’s coverage of everyday, colloquial Chinese is thin, the typical symptoms are:

• Lines read like written prose — like a press release or an essay, not like something a person would say.
• Conflict is advanced through shouting and thrown objects, with no psychological chain of motivation underneath.
• Plot templates come from other narrative traditions — a strong individual hero, a single-thread underdog rise — which are out of step with family ethics, in-law relationships, or court-intrigue material that centers on relationship networks and subtext.
• Memes and slang land a generation off, producing a dated feel.
• Character motivation does not hold up; turns are driven by coincidence rather than setup.

Conversely, in strong-conflict, high-visual-spectacle, single-protagonist narratives, the ranking of models may flip entirely.

The correct engineering conclusion is to avoid baking any single vendor into your architecture and instead treat the model as a replaceable implementation detail. Define one unified generation interface, hide the differences between providers behind it, and route by genre and task type.

from dataclasses import dataclass
from typing import Protocol
class ScriptGenerator(Protocol):
    name: str
    def generate(self, system: str, user: str, **kw) -> str: ...
@dataclass
class RouteRule:
    genre: str            # genre: family / suspense / beauty / snack ...
    provider: str         # backend implementation, swappable
    temperature: float = 0.9
    max_output_tokens: int = 1200
ROUTES = [
    RouteRule("family",    "provider_a", temperature=0.85),  # relationship-driven conflict
    RouteRule("suspense",  "provider_b", temperature=0.95),  # heavy-reversal content
    RouteRule("default",   "provider_a", temperature=0.90),
]
def pick_rule(genre: str) -> RouteRule:
    return next((r for r in ROUTES if r.genre == genre),
                next(r for r in ROUTES if r.genre == "default"))

Routing rules should not be chosen by intuition. They should be decided with data from a self-built evaluation set, which Section 5 covers.

3. The Structural Weakness: Strong at Generating, Weak at Judging

Suppose you ask a model to produce 100 scripts in one go. A stronger model may give you a wider distribution and a weaker one a narrower one, but both fail at the same question: neither can reliably tell you which script is the best.

Evaluation is genuinely harder than generation, for several concrete reasons:

1. The objective is ill-defined. “A good script” in business terms means higher conversion and completion after production — a lagging metric that can only be observed with real traffic. The model has no such label and has to guess from surface features.
2. Discrimination requires a reference frame. Absolute scoring has no anchors, so models naturally drift toward middle scores, compressing the distribution until discrimination disappears. Relative comparison (which is better, A or B) is far more stable than absolute rating.
3. Known systematic biases. Judging prompts commonly exhibit position bias (preferring options placed earlier or later), length bias (preferring longer answers that look more informative), and style self-preference (favoring text close to the model’s own generation style).
4. Mixing generation and judging contaminates both. Having the same model write and then evaluate in the same session reinforces its self-consistency preference.

So the naive approach — “generate, let the model score itself, take the top 5” — usually does not work well. Practical improvements include: drop the temperature close to zero when judging; replace absolute scores with pairwise comparisons; for each pair, evaluate once in each order and keep only verdicts that agree; sample multiple votes on the same pair; and require an explicit list of “deduction reasons” for each script rather than a score alone, using explainable reasoning to constrain scoring behavior.

4. The Target Pipeline: A Five-Layer Funnel

The recommended shape is not “generate → produce everything,” but a funnel that narrows layer by layer, always putting the cheapest judgment first.

Illustrative convergence ratios (adjust to your own data): 1000 generated → about 400 pass hard rules → about 120 enter scoring → about 60 after deduplication → 20 after human review → 5–8 produced in batches.

L1 — Hard-rule filtering: cut nearly half without spending on the model

This layer runs in deterministic code at zero marginal cost, and it has the highest return. The checks include:

• Duration and length match. A 30-second voiceover in Chinese has a roughly stable speaking-rate range; a script that clearly exceeds the character budget cannot keep the pace. Calibrate the threshold against the actual speaking rate of your own historical finished videos.
• Compliance red lines. Absolute and superlative claims, efficacy claims, price and promise statements, sensitive terms, and platform-prohibited content. A hit means eliminate or flag for human confirmation.
• Structural completeness. Whether it has a hook (first 3 seconds), a pain point or scenario setup, the core selling point, and a call to action; whether shot annotations or duration markers are missing.
• Production-cost constraints. Number of on-screen characters, number of scene changes, and whether special props or on-location shooting are required. This turns “filmability” into a computable metric.
• Structured-output validation. Require the model to return JSON and validate it against a schema; on parse failure, retry or discard.

FORBIDDEN = ["most effective", "cures it completely", "100% guaranteed", "the number-one brand"]  # illustrative; maintain per platform rules
def hard_filter(s: dict) -> tuple[bool, list[str]]:
    reasons = []
    body = s["script_text"]
    if not (s["char_count_min"] <= len(body) <= s["char_count_max"]):
        reasons.append("Length outside the target duration range")
    hit = [w for w in FORBIDDEN if w in body]
    if hit:
        reasons.append(f"Hit forbidden expression: {hit}")
    if s["scene_count"] > 3 or s["cast_count"] > 3:
        reasons.append("Too many scenes/cast members; production cost is high")
    for key in ("hook", "pain_point", "selling_point", "cta"):
        if not s.get(key):
            reasons.append(f"Missing structural element: {key}")
    return (len(reasons) == 0, reasons)

L2 — Structured scoring: decompose dimensions instead of one overall score

Asking “is this one good?” as a single question is nearly useless. What works is explicit dimensions, explicit weights, and an explicit requirement to explain the reasoning.

{
  "rubric": [
    {"key": "hook",        "weight": 0.25, "desc": "Does it form a clear hook in the first 3 seconds, enough to stop the target audience from scrolling?"},
    {"key": "audience",    "weight": 0.20, "desc": "Do the character setup, setting, and language fit the real-life context of the target audience?"},
    {"key": "selling",     "weight": 0.20, "desc": "Is the core selling point woven in naturally, or does it appear once and get buried by the plot?"},
    {"key": "conflict",    "weight": 0.15, "desc": "Does the conflict or reversal hold up, with setup rather than a forced twist?"},
    {"key": "delivery",    "weight": 0.10, "desc": "Are the lines colloquial and directly speakable, with phrasing and rhythm suited to 30 seconds?"},
    {"key": "compliance",  "weight": 0.05, "desc": "Any risk of exaggeration, promises, or values problems?"},
    {"key": "producible",  "weight": 0.05, "desc": "Given the scene and prop constraints, is it easy to shoot at low cost?"}
  ],
  "output": {"scale": [1, 5], "require_reason": true, "temperature": 0.0}
}

The prompt skeleton should contain: the full business context (product, audience, selling points, forbidden expressions), the complete text of one script, the scoring dimensions and weights, and an output-order constraint that requires per-dimension reasoning before the score. The judging prompt must be physically isolated from the generation prompt to prevent style contamination.

For insufficient discrimination, add a pairwise round: pair up candidates with close L2 scores (for example, the 10 adjacent-score pairs), evaluate each pair once in each order, build a win/loss matrix, and derive a final ranking with a simple aggregation method such as iterative strength updates or average win rate.

def win_rate(pairs):
    """pairs: [(a_id, b_id, winner_id)], with order-symmetry validation already applied"""
    wins, total = {}, {}
    for a, b, w in pairs:
        total[a] = total.get(a, 0) + 1
        total[b] = total.get(b, 0) + 1
        wins[w] = wins.get(w, 0) + 1
    return {k: wins.get(k, 0) / total[k] for k in total}

L3 — Deduplication and diversity sampling: make sure those 60 aren’t really the same one

LLM sampling distributions are highly concentrated, so batch generation routinely produces near-duplicates that differ only in wording: the same hook structure, the same conflict beats, only the line phrasing changed. Without deduplication, the 60 scripts handed to human review may represent only 8 distinct creative ideas.

The approach is to extract each script’s structural skeleton (hook type + sequence of conflict beats + selling-point placement + ending style), vectorize it, and cluster:

# Illustrative: vectorize the structural skeleton, cluster it,
# and keep only the top 1-2 by score per cluster
vecs = embed([skeleton_of(s) for s in scored])
labels = cluster(vecs, eps=0.25, min_samples=3)
buckets = {}
for s, lb in zip(scored, labels):
    buckets.setdefault(lb, []).append(s)
diverse = [max(v, key=lambda x: x.total_score)[:2] for v in buckets.values()]

At the same time, apply coverage quotas per dimension: hook types (suspense, contrast, question, scenario pain point), emotional arcs (grievance → reversal, curiosity → delight, anxiety → reassurance), and selling-point angles (ingredients, efficacy, price, scenario) should each be represented, so the final 20 are not all high-scoring variants of one formula.

L4 — Human review: look only at the head, and require structured feedback

The value of a human at this layer is not scoring one more time. It is doing three things models are bad at: judging whether the local colloquial feel is authentic, judging whether a compliance risk is acceptable, and judging whether the creative fits the current campaign cadence and product lifecycle stage.

Supporting requirements:

• Push only a small number of candidates from the top of the funnel to humans. The interface should show per-dimension scores alongside deduction reasons, so the reviewer only confirms or rejects.
• Human verdicts must be stored in structured form (pass/reject plus an enumerated rejection reason), not as a free-text note.
• Deliberately collect “human–model disagreement” samples; they are the most valuable calibration data source.

Production: validate cheaply first, then scale

A script that passes review does not need to enter the most expensive production mode immediately. A sensible approach is tiered validation: start with the lowest-cost format (static image plus text, voiceover over images, digital human) to test whether the hook works, and watch first-3-second retention. Only scripts that clear the retention bar get premium production. At the same time, run several candidates at small traffic as a canary, and let real conversion data decide what to scale.

5. A Gold-Standard Evaluation Set: Let the Screener Compound

Getting the pipeline running is only the starting point. Whether it keeps getting more accurate depends on having an evaluation set.

How to build it. Stratify by genre, select a batch of scripts that have already been produced historically, and use real online metrics as labels: 3-second completion rate, full-play rate, click-through rate, conversion rate, and production cost. The set does not need to be large — a few dozen per genre is enough to start — but it must cover both successes and failures.

What to measure. Not accuracy, but ranking quality: the rank correlation between the scorer’s ordering and the real business metric, plus top-K hit rate.

from scipy.stats import spearmanr
# Before/after comparison: does the score actually move in the same direction as business performance?
rho_before, _ = spearmanr(score_old, business_metric)
rho_after,  _ = spearmanr(score_new, business_metric)
# Derive weights from historical data instead of guessing 0.25/0.20/0.15
# X: per-dimension score matrix   y: normalized business metric
# coef, *_ = np.linalg.lstsq(X, y, rcond=None)

Three practices are non-negotiable. First, re-run the gold set after every model upgrade, prompt change, or weight adjustment, so you don’t “fix generation and break screening.” Second, periodically measure human–model agreement; declining agreement usually means business standards or genre style have drifted. Third, feed online failure cases back as new negative samples to keep expanding the gold set.

6. Cloud Engineering Implementation Notes

Batch generation is a classic asynchronous, bursty workload with external dependencies, and the architecture has to account for that:

• Asynchronous tasks. Push generation requests onto a message queue and have worker nodes consume them. A task table records status and an idempotency key, so timeout retries do not cause duplicate billing.
• Artifact archiving and retrieval. Store script text, intermediate artifacts, and scoring details in object storage, with structured metadata (genre, per-dimension scores, the backend that was routed, prompt version) in a database for traceability and re-evaluation.
• Elastic scaling and rate limiting. Scale out horizontally on a container platform for batch jobs, while enforcing concurrency caps and exponential-backoff retries against upstream generation services so rate limiting does not amplify the failure rate.
• Observability. Core metrics: cost per script, per-layer pass rate, score distribution, parse failure rate, and judging agreement. Set alerts for cost anomalies and sudden pass-rate shifts.
• Cost tiering. Run draft generation on low-cost models and allow multiple samples; run rewrites and judging on high-quality models with strict limits on call count; cache results for repeated requests with identical prompts.
• Data security. Unpublished product information from brands and campaign budgets are sensitive data and need access control, encryption in transit, and operation auditing.

7. Common Anti-Patterns

1. Producing everything. Using production cost to absorb the uncertainty of the generation stage is the most expensive mistake.
2. Single-model self-scoring. Stacked position bias, length bias, and style self-preference make the ranking essentially untrustworthy.
3. Only optimizing prompts, never building an evaluation set. What you cannot measure, you cannot improve — and style drift goes unnoticed.
4. Finalizing by personal preference and never feeding data back. Personal preference correlates with conversion rate far less than people expect.
5. Deferring compliance filtering. Discovering violations after the video is finished costs tens of times more to rework than at the text stage.
6. Treating “cost reduction” as “generate fewer.” Cutting candidates directly compresses creative diversity. The right direction is to widen the candidate pool and improve screening precision.

8. Phased Rollout

• Phase 1: Connect batch generation to hard-rule filtering, have humans spot-check everything that passes, and build the base data tables.
• Phase 2: Ship structured scoring and deduplication clustering; humans review only the top candidates, and human–model disagreement samples are recorded.
• Phase 3: Wire in online business metrics, complete weight calibration and genre-level model routing, so the screener enters a measurable state.
• Phase 4: Close the loop — batch generation, automatic screening, low-cost trial production, small-traffic validation, scale-up — and feed campaign results back into the gold set and the generation prompts.

Closing

Producing scripts for e-commerce short video is fundamentally a problem of cheap exploration and expensive confirmation. Progress on the generation side has already driven the cost of the candidate pool very low, so nearly all of the remaining gain comes from the evaluation side: whether you can cut unqualified candidates at close to zero cost, whether you can push the best candidates in front of a human with explainable, calibratable scores, and whether real online data keeps correcting the screening standard. Order the pipeline from cheapest to most expensive, let each layer do only the judgment it is best at, and that is the genuinely scalable path for this kind of content production.