New-Generation LLMs: From Answering Questions to Getting Work Done

The Short Version: A Capable Executor, Not an AGI Singularity

There has been no shortage of discussion around the new generation of large language models. Put official announcements, independent evaluations, community feedback, and real-world experience side by side, and the picture is far from uniform. Some look at benchmark scores and conclude that AGI has arrived; others find that the most obvious change after hands-on use isn’t “better at chatting” but “better at getting things done.”

For developers, the most important aspect of this upgrade is not parameter count or leaderboard position, but the model’s improved performance in task execution, long-running workflows, coding agents, and tool calling. It is genuinely stronger than before—but stronger at “doing work for you,” not at suddenly exhibiting general reasoning and autonomous learning. In other words, it behaves like a highly efficient executor rather than an agent that independently understands the world, sets its own goals, and makes complex decisions.

The Core Shift: From Talking Well to Getting Things Done

For a long time, the core interaction pattern of LLM products was question and answer. The user asks, the model responds—however polished the response, the actual hands-on work stayed with the user. Models at that stage could generate code, propose solutions, and explain concepts, but struggled to independently close the loop on a task spanning multiple tools, interfaces, and steps.

The new generation shows a clear capability migration: instead of just telling you what to do, it tries to do it itself.

Typical scenarios include:

• Opening software and completing configuration step by step
• Filling out forms, organizing customer records, merging documents
• Scheduling calendars automatically and resolving conflicts
• Scraping web pages and extracting structured data
• Analyzing data and generating reports and charts
• Writing code and building simple websites
• Taking screenshots to check whether pages render correctly
• Running multi-step automated workflows

The key change behind these abilities is the move from text generation to task planning and execution. The model has to understand a goal, break it into steps, call tools, handle feedback, and adjust when things go off track. For writing code, building reports, organizing documents, and creating presentations, this upgrade matters far more than simply answering more fluently.

Impressive Benchmarks—Read the Test Conditions Carefully

Every new model release leads with evaluation numbers, and this one is no exception. Some published results are indeed striking:

• Over 97% accuracy on high-difficulty math reasoning benchmarks
• Near-perfect scores on a general intelligence benchmark
• A 100% completion rate on a vulnerability exploitation benchmark

Taken at face value, these numbers suggest a qualitative leap. But test conditions often determine how high the score can go.

One important detail: the same model scores very differently under standard testing tools versus custom-built testing tools. On the general intelligence benchmark, for example, it scored roughly 62.7% with standard tools but approached a perfect score once custom tools were used. The gap comes down to factors such as:

1. Whether the model is allowed to retain hidden reasoning state
2. Whether stronger context management is provided
3. Whether prompting strategies are optimized for the specific task
4. Whether call-count and time limits are relaxed

This means the model itself is genuinely capable, but the high scores also depend on “special-purpose equipment.” The benchmark maintainers themselves cautioned that a perfect score on one benchmark does not mean AGI has arrived—a pragmatic reminder worth revisiting by anyone tracking LLM progress.

For engineering teams, the more useful takeaway is what this gap reveals: a model’s real-world capability is determined not just by its parameters, but jointly by inference-time compute, context management, tool interfaces, and task decomposition strategy. In other words, whether the model can actually deliver benchmark-level performance depends heavily on how you integrate it.

The Independent Evaluation View: Strong in Spots, Not All-Round

Compared with official launches, independent evaluations tend to be closer to real-world usage, because they typically use fixed task sets, a uniform environment, and reproducible procedures—better suited to side-by-side comparison.

Across common evaluation dimensions, the new model shows a clearly lopsided profile.

4.1 Outstanding Coding Agent Performance

On coding-agent benchmarks, the model lands in the top tier, roughly on par with the strongest AI coding agents currently available. More importantly, it completes comparable tasks at less than half the running cost of the alternatives it was measured against. This points to strong engineering usability across the full chain of code generation, debugging, testing, and deployment.

For development teams, this means it is worth seriously considering integrating the model into continuous integration, code review, and automated testing—rather than treating it as just a code-completion assistant inside a chat window.

4.2 Mediocre General Intelligence Scores

On comprehensive intelligence tests, however, the model scored about 61—only marginally above the previous generation and even behind flagship models from other vendors released around the same time. This indicates limited progress on traditional tasks such as broad knowledge Q&A, logical reasoning, and language understanding.

Put differently: if you expected “better at everything,” you may be disappointed. If you need “major efficiency gains on specific long-running tasks,” this upgrade may hit exactly the right spot.

4.3 Other Notable Details

Independent evaluations also surfaced a few easily overlooked but valuable observations:

• Output token counts dropped about 10% versus the previous generation, meaning more concise expression
• Complex knowledge work improved noticeably, but some occupational tasks actually regressed
• After the API price increase, the actual cost per completed task rose by roughly 75%

Piecing these data points together supports a fairly reliable conclusion: the new model is a specialist in complex, long-running tasks—not an all-rounder that wins in every scenario.

The Biggest Complaint: Costs Went Up, Not Down

Cost is unavoidable for enterprises and developers, and this model’s API pricing changes caught many early users off guard.

On unit price alone: roughly $10 per million input tokens and $50 per million output tokens—about 2.5 times the previous generation. That is not cheap to begin with.

What deserves even more attention, though, is how actual bills changed. Intuitively, if the model produces more concise output and uses fewer tokens, total cost should fall. The opposite happened, for three main reasons:

1. The model performs more rounds of internal reasoning during task execution
2. It calls external tools frequently, and every call adds extra context
3. Long-running tasks require maintaining more complete history, occupying more of the context window

The result: per-output tokens are indeed saved, but completing a task takes more reasoning steps and tool calls, so the bill grows. The savings never materialize.

The advice for engineering teams: when evaluating integration costs, don’t just look at unit prices—estimate the average cost of completing a single business task. How many API calls a task triggers from start to finish, how many input and output tokens it consumes, and whether it incurs extra tool-call overhead—these are the variables that actually determine cost.

How Developers Should Read This Upgrade

Putting the above together, the capability profile of the new model looks roughly like this:

Strengths:

• Planning and executing long-running tasks
• Code generation and coding-agent scenarios
• Tool calling and multi-step automation
• Structured information extraction and data processing
• Complex knowledge work and document generation

Weaknesses:

• Limited gains in broad knowledge Q&A and general reasoning
• Possible regression on some occupational tasks
• Higher call prices; actual per-task costs need re-evaluation
• Benchmark results highly dependent on test conditions—interpret with caution

Based on this profile, developers and technical decision-makers can think about adoption on three levels.

6.1 Scenario Selection: Favor Long, Repetitive, Rule-Clear Tasks

Scenarios worth prioritizing:

• Issue triage in automated testing and continuous integration
• Data cleaning, report generation, document structuring
• Customer ticket classification, information extraction, and workflow routing
• Automated orchestration of internal toolchains
• Code review assistance and defect analysis

Scenarios to avoid investing in too early:

• High-risk financial decision-making
• Q&A systems requiring continuously updated external knowledge
• Compliance scenarios with extremely high explainability requirements
• Online interactions that are extremely latency-sensitive

6.2 Architecture: Treat the Model as an Execution Agent, Not a Q&A Endpoint

To get real value from this class of model in production, integration needs to move from a simple Q&A endpoint to an agent architecture. A fairly generic design looks like this:

User goal
   │
   ▼
Task parsing layer: understand the goal, generate a task description
   │
   ▼
Planning layer: break down subtasks, determine execution order
   │
   ▼
Execution layer: call tools, access the environment, produce actions
   │
   ▼
Feedback layer: check results, detect deviations
   │
   ▼
Adjustment layer: revise the plan, re-execute, or report failure

Under this architecture, the model’s capability boundaries, tool permissions, cost ceilings, and retry-on-failure policies all need to be constrained at the system level. The stronger the model, the more important the guardrails.

6.3 Cost Control: Manage Uncertainty with Monitoring and Budgets

Because task execution can trigger multiple calls, consider adding these controls at integration time:

• A cap on calls per task
• A cap on tokens consumed per task
• A tool-call whitelist
• A manual-review switch for abnormal branches
• Per-task-type tracking of average cost and success rate

These mechanisms are not meant to limit what the model can do—they keep it operating within predictable, recoverable, auditable bounds. In production, controllability matters far more than occasional standout moments.

Conclusion: Hold Off on the AGI Hype—Start Putting It to Work

The biggest change this new generation brings is not a leap in benchmark numbers but a shift in capability shape. Models are moving from “answering” to “doing”—from conversational interaction to task execution. For developers and enterprises, this is a milestone worth evaluating seriously.

At the same time, keep a clear head:

• High scores come from specific compute conditions and test setups, not a qualitative leap in general ability
• The model improves markedly on some tasks while standing still or regressing on others
• Real-world costs may rise rather than fall, and need to be re-estimated per complete task
• Adoption value depends on scenario selection, architecture design, and cost control—not on the model alone

So the pragmatic read is: it really can deliver, but its strength lies mainly in execution. Rather than treating it as an AGI singularity, treat it as a new tool that can take a lot of work off your hands—that is probably the better fit.