AI Generation Is No Longer Scarce: Why Engineering Is Shifting to Verification and Judgment

1. Starting From a Case of Going All In and Failing

Imagine a developer who buys a compute quota for an intelligent coding tool but finds that the quota sits idle. To use it, he pours the entire quota into a repository and asks the machine to do three things around the clock: first, precisely formulate a famous mathematical conjecture; second, search for and download related papers; third, read the literature, enumerate combinations of ideas, and attempt to prove or disprove the conjecture.

In recent years, many projects have aimed to let AI automatically solve century-old problems: open-source repositories for human-AI collaboration, self-built systems that produce “complete derivations,” and proofs posted to public forums for verification. Their outcomes are strikingly similar: most generated proofs fail under scrutiny and are rarely accepted by the serious mathematical community.

The real question is not whether such a project fails, but why it will very likely fail. The reason is usually not insufficient compute, incomplete literature review, or insufficiently ambitious ideas. It lies in the underlying structure of how truth is validated. Following this thread to the end reshapes how we view AI and software engineering.

2. The Real Strength of AI: Constrained Cross-Domain Association

The standout strength of large models is their ability to connect concepts and domains. Association and analogy are embedded in the training objective, giving them coverage beyond that of any single generalist individual. If innovation depends on connecting distant fields, AI is indeed a suitable candidate.

As a generator, however, its outputs are not “unbounded and unrestricted.” They are clearly constrained by four factors: the coverage of training data, access to tools and retrieval permissions, whether the evaluation system is in place, and the cost of a single call or long-chain reasoning. What becomes much cheaper is unit cost, not all cost. For problems beyond the distribution, generation quality degrades significantly.

Even so, the “all-in” project still failed. The reason is the same: on most hard problems, generation is not the primary bottleneck.

3. Why Generation Is Not the Main Bottleneck: Interpolation, Extrapolation, and Their Limits

The famous conjecture resisted for more than 160 years not because too few people read it. On the contrary, the world’s top mathematicians have already tried countless combinations of mainstream approaches. The most tempting “unexpected cross-domain bridges”—connecting the zero distribution of certain functions with random matrix theory, treating zeros as eigenvalues of a self-adjoint operator, or using noncommutative geometry as an approximation—are mostly directions that humans discovered and mined intensively for decades.

If a problem only needed existing building blocks to be reassembled once, it would most likely have been assembled already. Its continued resistance shows that the required idea does not lie in the high-frequency region of any existing corpus.

This does not mean the simple dichotomy that “models can only interpolate and cannot extrapolate.” Models have some ability to generalize and compose structures. Whether they can truly extrapolate depends on the distance between the training distribution and the problem, whether tools and retrieval are available, and whether the verification mechanism can push errors back. A more accurate statement is this: improvements in generation solve the problem of making the known faster, cheaper, and more abundant; they do not automatically solve whether a candidate is correct.

4. The Essence of Judgment: Structural Asymmetry Between Generation and Verification

The core issue is the structural asymmetry between generation and verification. Problems can be roughly divided into four categories:

• Generation is hard, verification is easy: for example, formalized proofs. Writing a correct proof is extremely hard, but checking step by step whether each derivation holds is mechanical, low-cost, and automatable. Machines can cheaply filter out obviously incorrect candidates in batches, although the exact hit rate depends on the proof format and checking rules.
• Generation is easy, verification is hard: for example, “write code that runs.” Today this is nearly free, but whether the code is truly correct under boundaries, concurrency, exceptions, security, and performance requirements is still expensive and often requires human judgment.
• Generation is hard, verification is also hard: for example, “whether an idea bears weight, whether an analogy holds, or whether a direction is worth pursuing.” There is neither a cheap generator nor a cheap verifier; only domain intuition and taste can help.
• Both are easy: such problems no longer constitute engineering challenges.

The AI revolution changed the difficulty curve of generation, but barely changed the difficulty curve of verification. The contradiction is therefore fully exposed: when you can produce thousands of “apparently valid” candidates at extremely low cost, your judgment capability does not grow proportionally.

When a good engineer connects two ideas, he can vaguely feel whether the “connection” bears weight—whether it is a structural component that truly transfers force, or merely decorative. This judgment ability is precisely what AI lacks most and what humans are hardest to replace at.

5. On the Engineering Floor: The Center of Responsibility Moves From Writing to Judgment

When generation is nearly free, the center of a team’s time and responsibility moves from “writing” to “judging”:

• Code review, testing, canary release, and observability rise from auxiliary activities to the main battlefield.
• The primary risk shifts from “cannot write” to “cannot see what is wrong.”
• The more that is generated, the more unverified candidates there are; defect density, cognitive load, and long-term maintenance cost all increase.

In terms of responsibility allocation, the following minimum responsibility matrix is recommended:

Stage	Who defines	Who executes/approves
Specs and invariants	Jointly signed off by the author and domain experts	Reviewers verify
Judgment automation (tests, static checks)	Module owner maintains	CI enforces
Canary release	Change author submits the plan	On-call owner approves rollout
Production alerts	SRE/on-call team defines thresholds	Responders handle and conduct postmortems
Evaluation sets	Evaluation set owner	Reviewed quarterly for additions and removals

Without this table, “judgment” is only a slogan: when an incident happens, no one owns it, and no one owns the specs.

6. Six Actions for Engineering Judgment Capability

Judgment cannot rely only on human eyes and intuition. It must be designed into a sustainable system.

1. Make specifications executable: tests and assertions are judges

The cheapest form of judgment is to write “correctness” into a form that machines can check. Property testing is especially suitable for AI-generated code: you do not need to enumerate all inputs; you only need to declare invariants and let the machine sample inputs at scale and search for counterexamples.

# Minimal runnable example: encode “sorting is correct” as machine-checkable assertions
from collections import Counter
from hypothesis import given, strategies as st
def sort(xs):                      # Placeholder implementation: code under test
    return sorted(xs)
def is_sorted(xs):                 # Placeholder implementation: invariant check
    return all(a <= b for a, b in zip(xs, xs[1:]))
def multiset(xs):                  # Placeholder implementation: multiset comparison
    return Counter(xs)
@given(st.lists(st.integers()))
def test_sort_invariants(xs):
    once = sort(xs)
    assert is_sorted(once)         # Invariant 1: result is sorted
    assert sort(once) == once      # Invariant 2: idempotent
    assert multiset(once) == multiset(xs)  # Invariant 3: no items added or removed

Two points are important: sort, is_sorted, and multiset in the example above are minimal placeholder implementations; in a real project, replace them with your own code and checking logic. Also, property tests can only find counterexamples; they cannot prove correctness. They approximate through sampling, and coverage and generation strategies determine the confidence boundary. For small finite domains, direct exhaustive enumeration is stronger than sampling.

2. Use mechanically checkable constraints as front filters

Automate the parts that machines can judge cheaply, and leave human judgment for what machines cannot judge: type systems catch many “interface misuse” errors at compile time; static analysis, lint, dependency scanning, and license scanning intercept style issues, null pointers, injection, and supply-chain risks; formal methods provide mechanically checkable constraints for safety-critical core logic. The principle is: errors that can be caught earlier and more cheaply should never be left to expensive methods later.

3. Build layered verification: cheap first, expensive later

Compile/type checks → static analysis → unit/property tests → integration tests
   → security scans → human review → canary release → production monitoring and alerts

Each layer should only judge the type of problem it is best at judging, allowing expensive expert review to focus on semantic and value questions.

4. Build offline evaluation sets and quantifiable judgment metrics

For scenarios where verification is also hard, such as content generation and model applications, “good” and “bad” should be turned into regression datasets and metrics so that judgment is repeatable, comparable, and quantifiable. At minimum, track the following metrics:

• False acceptance rate (incorrect candidates judged as correct, i.e., leakage) and false rejection rate (correct candidates judged as incorrect, i.e., false rejection).
• Average review time and review rounds per change.
• Defect escape rate: the proportion of total defects that escape to production or the next stage.
• Front-loading rate: the proportion of problems caught at cheaper stages.
• Evaluation set refresh frequency and negative example ratio.
• Alert precision: the proportion of alerts that truly require action.

The evaluation set itself is an asset, and it also degrades: only adding but never removing, lacking negative examples, and repeatedly tuning with the same samples can make the team overfit to the evaluation set—scores rise, while real judgment capability declines. Periodically introduce new samples and use holdout sets for cross-validation.

5. Human in the loop: preserve “load-bearing” judgment

Clearly define the boundaries where humans must make the final call: architecture tradeoffs, security boundaries, whether requirements match real intent, and whether a feature is worth building. These are precisely the hardest parts of judgment and the least outsourceable.

6. Let verification infrastructure go first

Judgment does not rely on human eyes; it relies on a sustainable system foundation. This usually requires coordinated common cloud capabilities: logs retain traceable behavioral evidence; monitoring and alerts detect anomalies as soon as possible; tracing reconstructs causality across services; CI/CD automates the above judgments and enforces gates; version and artifact management ensure that judgment conclusions are reproducible. The more complete the foundation, the faster, broader, and more trustworthy judgment becomes.

7. Common Mistakes

• Treating tests as decoration: writing them but not running them, allowing failed merges, checking coverage without checking assertion strength.
• Testing only the happy path: not writing boundary, concurrency, exception, timeout, or degradation branches.
• Weak assertions: asserting only “no exception,” without asserting results and invariants.
• Treating review as the only judgment layer: pushing all cheap mechanical checks into a human funnel and causing review backlogs.
• Generation outpaces judgment: batch submissions and large one-off changes cause judgment costs to spiral out of control.
• Evaluation sets only grow but never get fixed: no negative examples are added, expired samples are not removed, and metrics become inflated.
• Ignoring the supply chain: licenses and sources of AI-generated dependencies and code snippets are not scanned.

8. Conclusion: Choose the Strength of Judgment by Scenario

AI pushes generation toward superhuman levels, which is equivalent to making supply nearly sufficient. What becomes scarce, and what truly determines success, is judgment: selecting the one candidate that is truly valid and valuable from a large pool of candidates. Generation solves “possible”; judgment decides “credible.”

The strength of judgment does not need to be uniform: low-risk prototypes can use lightweight judgment, such as running the main path, basic assertions, and human confirmation. Safety-critical systems—payments, healthcare, automotive, identity and access, and infrastructure—must apply strong judgment: mechanical constraints, layered validation, canary rollout, two-person review, and human-in-the-loop are all indispensable. Configuring judgment cost according to risk is itself an engineering decision.

Therefore, the “all-in” plan is likely to fail not because AI is not strong enough, but because it placed all its bets on generation, which has been greatly improved, while failing to build the unresolved judgment layer. How to build a judgment system, how to define metrics, and how to avoid common mistakes—that is where the real engineering opportunity lies.