Why Were Large Language Models Underestimated? The Debate Over AGI Through the Lens of Technological Evolution

1. An Unprecedented AI Race

Some technology observers have described today’s artificial intelligence competition as “one of the fiercest competitions in the history of capitalism,” even comparing it to a modern Manhattan Project. The analogy is exaggerated, but it highlights an important reality: large-model development has moved from a niche academic pursuit to a strategic competition involving governments and major corporations.

What is this competition really about? The answer can be summarized in three areas:

1. Compute resources: Training a frontier-scale model can require tens of thousands of accelerator cards operating together for months. Compute has become one of the hardest barriers to entry.
2. Top-tier talent: Researchers who understand deep learning theory and can also manage massive-scale engineering projects are extremely scarce.
3. Data and engineering experience: Curating high-quality corpora, stabilizing training, and reducing inference costs all require long-term accumulation of expertise.

To understand this competition, we must first address a counterintuitive question: why did leading research institutions with critical technical advantages initially fall behind in this wave?

2. Why Large Language Models Were Underestimated: Three Technical Explanations

1. Inertia in Paradigm Selection

A strong belief prevailed in deep learning for many years: machines must learn like humans, through interaction with their environments and accumulated experience. This view helped drive major advances in reinforcement learning and deep reinforcement learning. From game-playing systems to robotic control, these applications demonstrated the effectiveness of experience-driven learning.

As a result, many experienced researchers initially questioned language models trained primarily to predict the next token. They regarded such systems as statistical pattern matchers without genuine understanding or reasoning ability.

This judgment was not entirely irrational. However, it underestimated the combined effect of two critical variables.

2. The Nonlinear Returns of Scaling Laws

Through systematic experimentation, the industry discovered that when model size, training data, and compute are increased in coordinated proportions, model loss declines according to predictable power-law patterns.

An even more important finding was that some capabilities do not grow linearly with scale. Instead, they can appear to emerge suddenly after a model crosses a certain threshold. Examples include multi-step reasoning, in-context learning, and cross-task generalization.

The evaluation logic can be illustrated with the following simplified example:

# Core evaluation logic for scaling experiments (illustrative)
for scale in [1e9, 1e10, 1e11]:        # Increase model scale step by step
    loss = train_and_evaluate(scale)   # Verify power-law loss reduction
    emergent = benchmark(reasoning_tasks)  # Detect leaps in reasoning ability

Research institutions that rely too heavily on small-scale experiments may incorrectly conclude that increasing model size will not produce qualitative changes. In reality, capabilities that cannot be measured in smaller models may represent the central value of large models.

3. Alignment Techniques Completed the Missing Piece

A raw pretrained model only learns to continue text. It does not naturally follow human instructions. The major turning point came with the maturation of reinforcement learning from human feedback (RLHF).

The general process is to first use instruction tuning to teach the model how to respond in a conversational format. Human preference data is then used to train a reward model, after which reinforcement learning optimizes the model’s outputs.

This process fundamentally improved usability. Models evolved from technical demonstrations into products that ordinary users could interact with.

The industry consequently reached an important consensus: model evaluation is shifting from academic benchmark scores toward real-world user experience. Users do not necessarily demand a perfectly error-free model. They mainly want a system that is useful and can improve through iteration. This change in evaluation criteria was itself one of the factors that had been underestimated.

3. The Dilemma for Incumbents: Brand Risk and Release Timing

Why, then, did organizations with strong technical resources not always launch mass-market products first? The answer lies in a structural dilemma:

• Different tolerance for errors: Generative models can hallucinate, producing incorrect information in fluent and confident language. For a startup, this may be a defect that can be corrected through rapid iteration. For an information service used by billions of people and built around accuracy, releasing a system that may confidently “make things up” creates significant brand risk.
• Heavier governance processes: Large organizations must balance safety evaluations, compliance reviews, and reputation management. Their decision-making chains are longer, which naturally encourages them to continue refining a product before release.
• Existing business moats can become liabilities: If a new product threatens a company’s core revenue sources, internal incentives to promote it may weaken.

This case offers an important lesson for technology leaders: during a paradigm shift, poor judgment often results not from a lack of information, but from the cognitive inertia created by previous success.

One way to respond is to maintain independent teams dedicated to aggressive exploration. These teams should be able to test high-risk directions without being completely constrained by standard decision-making processes.

4. After AGI: From Intelligence to a Post-Scarcity Society

Following the arrival of artificial general intelligence (AGI), discussions about the future of society increasingly focus on the concept of post-scarcity.

The basic logic is that if AI can replace large amounts of both cognitive and physical labor, the marginal cost of producing goods and services could approach zero. The way society creates wealth would then change fundamentally.

This possibility raises several issues that deserve serious attention:

1. Distribution mechanisms: If labor is no longer the primary source of individual income, how should redistribution systems be designed? Discussions already include proposals such as universal basic income.
2. The reconstruction of meaning: Much of people’s sense of value and identity comes from work. The AGI era will require society to reconsider what it means to be human.
3. Technology governance: The more powerful a system becomes, the more important it is to establish safety evaluations, red-team testing, and traceability mechanisms before deployment. This is why AI safety should not simply be dismissed as “doomsday thinking.” It should be treated as part of engineering practice, much like software testing is part of software development.

The industry remains divided over the level of risk. Some believe those risks are being exaggerated, while others argue that alignment research must advance ahead of capability research.

A rational position is to avoid both extremes: innovation should not be paralyzed by fear, but governance should not be abandoned because of optimism.

5. Lessons for Developers and Businesses

At a practical level, this debate offers at least four actionable lessons:

• Focus on capability curves rather than isolated performance scores: When selecting models and planning systems, evaluate how capabilities are evolving over time and preserve enough architectural flexibility to accommodate sudden improvements.
• Evaluate providers by alignment quality: When assessing a large model, examine its instruction following, hallucination mitigation, safety guardrails, and the overall maturity of its alignment engineering.
• Build a data flywheel: High-quality human feedback is a core asset for continuously improving model usability. Organizations should establish feedback collection and annotation processes as early as possible.
• Embrace AI-native workflows: For the generation of users growing up alongside large models, AI-assisted programming, writing, and analysis will become default ways of working. Redesigning products and organizational processes around these habits will be critical to capturing the benefits of the AI-native generation.

The story of large language models shows that correctly identifying the direction of technological change can matter more than the amount of resources invested. Ultimately, the ability to revise strategic assumptions depends on whether an organization preserves its sensitivity to new paradigms—and the humility to recognize when its existing assumptions may be wrong.