Three Years of Working with AI: What Has Changed, What Hasn’t, and What Developers Can Learn

The First Conversation: Testing an AI with a Number-Base Puzzle

I still remember my first conversation with a large language model. Instead of opening with “Hello, what can you do?”, I gave it a question that required reasoning across several concepts:

In base π, what is the area of a circle with radius 10?

The question may sound like a trick, but it was designed to probe several abilities at once:

1. Transferring the concept of a number base: Does the model understand that a base changes how a value is represented, not the value itself?
2. Reasoning about notation: Can it recognize that “10” in base π represents π?
3. Applying and combining formulas: Does the circle-area formula, (A = \pi r^2), still apply under this unconventional setup?
4. Checking its own work: Is its first answer plausible, and can it correct its reasoning when challenged?

Like many people’s early experiences with generative AI, that conversation was not flawless. The model initially gave an intuitive but incorrect answer. Only after I pointed out that “10” in base π represents π did it arrive at the right result: the area is π³, written as 1000 in base π.

The reasoning is straightforward:

• In base π, the numeral “10” represents (1 \times \pi^1 + 0 \times \pi^0 = \pi). So the radius, written as “10,” has the decimal value π.
• Apply the circle-area formula: (A = \pi \times r^2 = \pi \times \pi^2 = \pi^3).
• In base π, π³ is written as (1 \times \pi^3 + 0 \times \pi^2 + 0 \times \pi + 0), or 1000.

A short Python snippet verifies the result:

import math
# In base π, "10" represents the value π
radius = math.pi  # Radius "10" in base π → π in decimal
# Circle area: A = π * r^2
area = math.pi * radius ** 2  # The result is π^3
# Convert the base-π numeral "1000" back to decimal
restored = 1 * math.pi**3 + 0 * math.pi**2 + 0 * math.pi + 0
print(f"Area in decimal       = {area:.6f}")
print(f"1000 in base π, decimal = {restored:.6f}")
print(f"Do the values match?    : {math.isclose(area, restored)}")
# Output: Do the values match? : True

That exchange shaped how I use generative AI: I treat it like a knowledgeable, quick-thinking colleague who can still make confident mistakes. Its answers need verification, and important reasoning steps need to be checked.

What Hasn’t Changed: A First-Principles View of AI

The past three years have taken generative AI from research labs to hundreds of millions of users. Public opinion has swung between claims that AI will transform everything and predictions that the bubble is about to burst. Amid that volatility, several of my basic views have remained stable because they are grounded in how the technology works, not in the latest demo.

1. AI Is a Statistical System, Not a Truth Machine

Large language models learn conditional probability distributions over what text is likely to come next, based on large collections of text. This gives them broad knowledge coverage and the ability to recombine patterns. But it also means their output is plausible, high-probability text—not necessarily verified fact.

The model’s initial mistake on the number-base question can be understood as overgeneralizing a familiar pattern: “radius 10 → area 100π.” Recognizing this helps clarify a practical rule: the level of trust should match the risk of the task. The more consequential the use case, the stronger the verification process should be.

2. AI Can Surpass Humans at Generating Novel Combinations

Questions such as “Can AI come up with an unexpected idea?” and “Can its creativity exceed ours?” were being discussed even before generative AI became mainstream. I have generally thought the answer could be yes.

From a cognitive-science perspective, human inspiration also involves recombining things already stored in memory, though emotions, experience, and values influence how those combinations are selected. AI systems can explore conceptual space at a scale and speed that humans cannot, so it is not surprising that they can produce novel combinations at a high rate.

The real dividing line is not whether a system can generate new ideas. It is whether it can evaluate their value and take responsibility for them.

3. Functional “Understanding” Is an Achievable Capability

Can AI understand a parable? If understanding means a subjective experience like human consciousness, there is no straightforward way to settle the question in the near term. But if it means using a concept appropriately in contexts people recognize, then the answer is yes.

AI systems can use the metaphorical structure of a parable appropriately in writing, question answering, and rewriting tasks. In engineering terms, that makes “understanding” a capability that can be delivered and evaluated. Developers should focus on measurable functional outcomes rather than letting unresolved philosophical debates prevent practical work.

4. Development and Safeguards Should Proceed Together

The potential benefits of AI for medicine, education, and research are real, and so are its risks. The rational choice is not simply “develop” or “pause.” It is to continue advancing capabilities while building evaluation, auditing, and governance systems alongside them—a pattern also seen in the development of general-purpose technologies such as electricity and nuclear energy.

What Has Changed: Evolving Capabilities and Engineering Practices

Stable beliefs about AI do not mean ignoring how much the field has changed. Several developments have had a significant effect on technical practice.

Reliability has improved. Models from three years ago were fragile at complex reasoning, maintaining consistency across long contexts, and following instructions. Today’s models can more reliably handle multi-turn tool use, structured output, and long-document analysis. Hallucinations have not disappeared, but systems have moved from being difficult to use reliably to being usable with appropriate safeguards.

AI interactions have moved beyond chat windows and into workflows. Generative AI is no longer limited to conversational interfaces. Retrieval-augmented generation (RAG) lets models answer using private enterprise knowledge. Agent frameworks let models plan tasks, call tools, and execute multiple steps. In software development, coding assistants have become part of many developers’ everyday workflows.

Models iterate faster, making evaluation essential. Rapid releases mean that a capability profile based on the previous model version can quickly become outdated. For teams integrating AI into production systems, maintaining an internal regression evaluation set—not relying only on vendor demos—is basic engineering practice for protecting service quality.

What Usage Data Shows About How People Use AI

Public industry research offers a useful snapshot of how people use conversational AI. In a sample analysis of anonymized conversations from leading conversational products:

• About 78% of conversations focused on practical guidance, writing, and information search—for example, troubleshooting a washing machine or finding the winners of a particular year’s Nobel Prizes.
• About 4.2% involved programming.
• About 1.9% involved relationships and personal reflection.
• About 0.4% involved games and role-play.

The change in message volume is also striking. In June 2024, one leading product reportedly handled about 450 million messages per day: approximately 213 million work-related and 238 million non-work-related. By June 2025, the reported total had reached about 2.6 billion messages per day, including approximately 716 million work-related and 1.911 billion non-work-related messages—nearly five times the volume in one year.

These figures point to three takeaways:

First, AI is primarily a practical tool, not an emotional companion. Media coverage often focuses on AI companionship, but most people open a chat interface to solve concrete, everyday problems. Product design should therefore prioritize factual accuracy, coverage of common topics, and ease of use.

Second, a small share of a very large user base is still a substantial number of people. Emotional-support conversations make up less than 2% of the total, but across hundreds of millions of users and billions of daily messages, that share represents demand on the scale of tens of millions. Long-tail use cases may not warrant being a product’s central selling point, but they deserve serious attention.

Third, non-work use is growing much faster than work use. Rapid growth among consumers suggests that AI is evolving from a workplace productivity tool into a general-purpose entry point for everyday information. Its long-term effect on how people find information could be comparable to the impact search engines had on information access.

Practical Recommendations for Developers

Based on these observations, here are several recommendations for developers and engineering teams.

1. Build an Evaluation Set of “Reasoning Traps”

Questions like the circle-area puzzle in base π are useful not because of the answer itself, but because they reliably expose a model’s tendency to fall back on familiar patterns. Build evaluation sets around your own domain, including cases that test:

• Unit-conversion traps
• Time and sequence reasoning
• Negation and conditional branches
• Numerical consistency

Run regression tests after every model upgrade. Use results, rather than intuition, to decide whether to upgrade.

2. Add Engineering Safeguards Around Model Outputs

• Retrieval-augmented generation (RAG): Ground answers in a traceable enterprise knowledge base to reduce hallucinations.
• Tool calling: Delegate deterministic tasks such as calculations and lookups to code or other tools; let the model focus on interpreting and organizing the results.
• Structured output and validation: Check critical fields for correct formats and acceptable ranges, and use human review when necessary.
• Source attribution: Ask the model to identify its sources so people can verify its claims.

3. Start with Use Cases That Are Frequent, Valuable, and Verifiable

The usage data suggests that practical guidance, writing, and information processing serve the largest user groups. When adopting AI in an organization, start with workflows where answers are easy to check and benefits are easy to measure—for example, customer-support knowledge queries, first drafts of documents, or code completion. These are often more likely to deliver value quickly than ambitious, demo-oriented efforts to build fully autonomous agents.

4. Use Cloud Platforms’ General-Purpose AI Capabilities to Get Started

Major cloud providers now offer combinations of general-purpose capabilities such as large language model APIs, vector search, knowledge-base question answering, model evaluation, and monitoring. For example, developers can use Baidu AI Cloud’s large model platform and knowledge-base components to build RAG-based question-answering applications, rather than building the underlying infrastructure from scratch. This can leave more time for business data and workflow design.

Whichever platform you choose, data de-identification, access controls, and output auditing for private data remain essential.

5. Plan for Costs Before the Demo Becomes a Production Bill

At an industry scale of billions of messages per day, inference cost is a core economic consideration. Common approaches include routing simple requests to smaller models and complex requests to larger ones, caching answers to frequent repeated questions, and retrieving relevant passages before generating responses to long documents.

Cost management should be part of the architecture from the start, not an afterthought once the system is live.

Conclusion: Treat AI as a Tool and Change as the Constant

Looking back three years later, I still consider that question about the area of a circle in base π a fitting introduction to AI. It revealed both the technology’s impressive knowledge and reasoning potential and its limitations. It also foreshadowed the right way to use AI: trust it, but verify; expect a lot, but don’t treat it as infallible.

Models will keep improving, and the industry’s focus will keep shifting. But the view that AI is a powerful statistical system—one that needs engineering safeguards and is still worth developing—will remain relevant for the foreseeable future.

For developers, building your own evaluation systems, verification workflows, and domain knowledge assets is more valuable than chasing every product launch. Those are the foundations that hold steady as the technology changes.