Model as Product: Capability Cultivation and Irreversible Decisions in LLM Development

1. Shift the lens: models are cultivated capabilities, not delivered software

Traditional software development has an implicit assumption: requirements can be exhaustively specified, behaviors can be enumerated, and tests can cover most branches. A requirements document often says what happens after a user clicks a button, and acceptance criteria are explicit input-output pairs.

Large model development breaks that assumption. Before training starts, the team usually has only intuitions and hypotheses, such as “this parameter size combined with this data mixture should significantly improve coding tasks.” The real conclusion only emerges after training has reached a certain stage and a batch of evaluations has been run. Worse, model behavior is not a simple implementation bug. It is more like a shift in capability distribution: the model may outperform expectations in some subdomains and unexpectedly regress in others, and the latter often becomes visible only after deployment.

Therefore, treating a model like ordinary software with a “requirements → development → testing → launch” pipeline fails in two places: requirements cannot be exhaustively specified, and acceptance cannot cover the full behavior space. A more practical paradigm is to treat development as cultivation:

• Set capability goals: not a feature list, but capability distributions and reliability targets.
• Observe in stages: capability emergence during pretraining, capability boundaries in the middle phase, and instruction following and safety behavior after post-training.
• Intervene based on feedback: use real usage data to decide whether to adjust data mixtures or add constraints during reinforcement learning and alignment.
• Re-evaluate continuously: after each model version update, rerun the same evaluation suite to prevent fixing one scenario while regressing three others.

Under this paradigm, the core work on the product side changes from “pushing progress” to three tasks: defining capability goals, building an evaluation system, and identifying the points where intervention should be cautious.

2. Writing capability specs: from feature descriptions to distributions and criteria

A traditional requirements document answers “what should be built.” A capability specification answers: on which input distributions, under what criteria is success judged, and how much failure is tolerated?

A practical capability specification should contain at least four layers:

1. Scenario distribution: what target scenarios are included, where the long tail lies, and the ranges of input length, language, domain, and noise level.
2. Capability dimensions: correctness, instruction following, format stability, refusal boundaries, tool-calling success rate, latency, and cost constraints.
3. Success criteria: decision rules for each scenario. Use programmatic checks wherever possible, such as whether fields are valid, whether code passes tests, or whether a number matches the expected answer. For cases that cannot be programmatic, use human review with explicit scoring guidelines.
4. Failure tolerance: which errors are acceptable, such as wording that is not concise enough, and which are absolutely unacceptable, such as unauthorized operations, factual errors, or destruction of user data.

It is important to avoid goals like “make the model smarter.” Such statements cannot be validated, attributed, or scheduled. A better formulation is: “In multi-turn file-modification tasks, the tool-call parameter error rate is below a given threshold, and unauthorized writes are zero.” The goal becomes measurable, and intervention has a direction.

3. Feedback loops: using models to process model feedback safely

After a mature product is launched, feedback volume quickly exceeds the limit of manual reading. Reading feedback one by one can only produce low-information conclusions such as “users are not very satisfied.” The automatable part is to turn unstructured feedback into structured signals, and then turn those signals into actionable evaluation items.

A practical pipeline looks like this:

# Feedback -> themes -> evaluation set (schematic)
raw      = load_feedback(window="7d")          # Pull raw feedback from the recent window
clusters = embed_and_cluster(raw)              # Embed and cluster feedback to derive topic clusters
for c in clusters:
    if c.size < MIN_SUPPORT:                   # Archive low-frequency long-tail items separately; do not create evals yet
        park(c)
        continue
    if not is_actionable(c):                   # Product misunderstanding, missing documentation, etc.; route to product side
        route_to_pm(c)
        continue
    variants = expand_cases(c.samples, n=20)   # Synthesize variants from real samples to broaden coverage
    cases    = filter_leak(dedup(variants))    # De-duplicate, redact, and remove overlap with training data
    evalset.add(cases, tags=c.tags, owner=c.owner)

Several principles must be preserved in this pipeline, or the system will be distorted:

• Models should only perform filtering and clustering; criteria should be defined by humans. If the model both defines “what counts as a good answer” and evaluates its own answers, it reinforces existing preferences, and the evaluation set ends up endorsing the current state rather than improving it.
• Sample-check cluster quality. Manually inspect several items in each topic cluster to confirm semantic consistency; otherwise unrelated issues may be merged into one evaluation item.
• Prevent leakage. Samples synthesized from real feedback that heavily overlap with training data will inflate metrics. Synthetic samples should be checked against training data for similarity and isolated.
• Version the evaluation set. The evaluation set is an asset that should be iterated. It needs versions, change records, and corresponding metric history; otherwise metric fluctuations cannot be explained.

Two typical traps deserve special attention. The first is distribution drift: evaluation-set performance keeps rising while user experience declines, usually because the evaluation set was sampled months ago and no longer represents current usage. The second is score polishing: if the post-training stage repeatedly optimizes against the evaluation set, the model learns statistical regularities of that set rather than the target capability, and cross-scenario generalization deteriorates noticeably.

4. Agent memory governance: offline “reconsolidation” during idle periods

Once an agent takes on long-horizon tasks, it accumulates many intermediate conclusions, user preferences, and tool-return results. If this memory is not governed, three problems appear: context bloat squeezes out effective reasoning space, retrieval introduces noise, and outdated early conclusions are still treated as facts.

A common industry approach is layered memory management:

Layer	Content	Update frequency	Governance focus
Episodic memory	Task traces, tool-call logs	High-frequency writes	Compression, summarization, time-based archiving
Semantic memory	Conclusions, user preferences, domain facts	Medium frequency	De-duplication, contradiction detection, confidence maintenance
Procedural memory	Workflow templates, common operation steps	Low frequency	Versioning, deprecation marking

“Idle-time consolidation” essentially decouples memory writing from memory cleanup: during task execution, the agent only performs fast appends; when the agent is idle or lightly loaded, a background process traverses memory and performs secondary checks. Core actions include:

def consolidate(memory, now):
    # 1) Conflict detection: semantically similar entries with contradictory conclusions
    for a, b in candidate_conflicts(memory):
        if can_verify(a) or can_verify(b):
            resolve_by_evidence(a, b)     # Check authoritative data sources instead of letting memories overwrite each other
        else:
            demote(a, b)                  # Lower confidence, mark as pending confirmation, and clarify in later conversations
    # 2) Expiry cleanup: time-sensitive facts with TTL and temporary computation results
    prune_expired(memory, now)
    # 3) Compression: merge redundant entries into one conclusion with evidence references
    merge_redundant(memory)
    # 4) Audit: log every change to support rollback
    audit.write(memory.diff_since(last_run))

Conflict resolution needs a clear priority order. The rule should not be “newer overwrites older.” First, prefer verifiable evidence. Second, consider source credibility and recency. When neither can be verified, the correct action is to lower confidence for both entries and mark them as pending confirmation, rather than forcing a choice. This point is crucial: silent overwriting creates “confidently wrong memories,” which can be more dangerous than having no memory at all.

Two engineering constraints are also essential. First, auditability and rollback: every deletion or rewrite must be traceable to the specific entry and operation time. Second, protection of explicit instructions: constraints, prohibitions, and preferences that users have clearly stated should not be automatically compressed or pruned; they can only be explicitly updated.

5. One-way doors and two-way doors: spending design attention where it matters

When engineering implementation becomes much cheaper, a common consequence is decision inflation: every plan must be reviewed, every change must be scheduled, and the team spends much energy on low-cost, reversible local optimizations while under-arguing choices that truly affect long-term direction.

The judgment can be reduced to three questions:

1. Can the decision be rolled back within hours?
2. Does its impact stay within internal systems?
3. Have external dependencies already formed, such as user habits, interface semantics, contracts, or data retention?

If all three answers are “yes,” it is a two-way door. Move fast, try quickly, and adjust quickly. Do not write long documents for every reversible local change. If any answer is “no,” treat it as a one-way door and justify it upfront.

Typical one-way doors are concentrated in these areas:

• Model and training routes: in-house development, fine-tuning, or distillation; data mixtures; and compute investment structure.
• Data compliance boundaries: collection scope, retention periods, cross-border policies, and anonymization strategies.
• Brand and values settings: model persona, refusal boundaries, and expression style. Once external perception is formed, the cost of correction is very high.
• External interfaces and pricing semantics: once external dependencies form, later changes become breaking changes.

At the same time, do not treat “reversible” as an all-purpose shield. Many reversible decisions can accumulate into decision debt: after each local optimum stacks up, the overall system architecture drifts away from the target, and it becomes difficult to identify which step should be overturned. Reversibility reduces the cost of a single decision; it is not permission for unlimited decision volume.

6. Capability map for AI-native product managers

In traditional product management, much daily work is occupied by data retrieval, codebase investigation, and document search. When these tasks can be automated, the bottleneck shifts: the product manager must judge which conclusions are trustworthy and which must be personally verified.

A practical capability map can be divided into three tiers by trust level, with the source of judgment noted for each scenario:

• Trusted zone: mechanical and verifiable tasks, such as data extraction and statistics, codebase structure and call-chain scanning, format conversion, batch document summarization, and first-draft review checklists. The characteristic is that results can be quickly validated by programs or authoritative sources.
• Collaboration zone: intermediate outputs that require human review, such as solution candidates, impact analysis, key takeaways from user interviews, and test case design. The output direction is roughly correct, but details need human input.
• Prohibited zone: unverifiable or high-risk judgments, such as compliance conclusions, legal and contract interpretation, decisions involving personal information, and final attribution of real user sentiment. Here the issue is not “accuracy is not high,” but that errors cannot be discovered through subsequent checks.

This map needs to be redrawn periodically. Model capabilities change with versions. A task that was in the collaboration zone six months ago may now enter the trusted zone, or vice versa. Redrawing is not expensive: sample several items per task category, run them, have humans judge the adoption rate, and update the tiering by scenario.

On “multi-role debate,” where a model is asked to offer opinions on the same plan from different standpoints, such as implementer, risk reviewer, user advocate, and operations perspective, this can indeed expose blind spots of a single perspective and is especially suitable for stress-testing before a plan is finalized. But its boundary must be clear: it simulates the form of discussion, not real constraints. It will not fail in the middle of the night because the architecture is unsound, and it will not bear failure costs. Therefore, it is suitable as a divergence and self-check tool, not as the sole basis for architecture decisions.

Several common misuses are worth listing: treating confident expression as correctness; treating “generation” as “verification” (a model finishing code does not mean the code is correct; tests must be run); and treating average performance as worst-case performance (user experience is determined by tail failures, not by averages).

7. Suggested rollout cadence

Do not try to do everything at once. Proceed in dependency order:

• Phase 1 (about 1 month): define capability specifications and success criteria for core scenarios, build the evaluation-set skeleton and runtime pipeline, and connect telemetry and feedback channels. Without deliverables in this step, later optimization cannot be validated.
• Phase 2 (1–2 months): launch the pipeline for automatic feedback clustering and synthetic evaluation; bring memory-governance strategies under configuration management, with TTL, compression rules, and conflict-resolution priorities all versioned.
• Phase 3 (continuous): establish a one-way-door checklist and bind it to the review process; refresh the capability map quarterly; maintain the evaluation set as a long-term asset with versions and metric history.

To measure whether this system works, the metrics should not be “how many AI capabilities were used,” but closer to outcomes: the rate of human intervention per unit of task volume, the correlation between evaluation-set performance and online performance, the pre-review coverage rate of irreversible decisions, and changes in long-horizon task success rates before and after memory governance.

Conclusion

The divergence between large-model development methodology and traditional software methodology comes from one root cause: “capability” cannot be exhaustively defined; it can only be approached through observation, evaluation, and intervention. Several working principles follow consistently: replace vague goals with decision criteria; replace manual filtering with automation while keeping humans responsible for defining standards; replace real-time memory bloat with offline consolidation; and replace uniform processes with reversibility.

The value of a product manager is not to push more changes, but to identify choices that are hard to reverse after they are made, and to invest real research in those choices.