From Test-Taking to Task-Doing: Key Gaps in General-Purpose Capabilities of Large Language Models

In evaluations of large language models, a common pattern appears. Models often perform well on public benchmarks, standard question banks, and closed-form Q&A. They can produce answers close to the reference on clearly defined questions. Once they face real general-purpose tasks, such as engineering changes with unclear requirements, cross-system troubleshooting, or long business workflows, their performance often fluctuates noticeably. The core issue is not only whether a model has enough knowledge. It is whether the model can keep converging on the result the user actually wants in open, changing, and interaction-heavy environments.

We can call the former “test-taking capability” and the latter “task-doing capability.” Test-taking has clear boundaries, fixed endpoints, and verifiable answers. Task-doing resembles a shifting system that keeps interacting with its environment. Goals change during the process, constraints emerge during execution, and acceptance criteria often need to be aligned along the way. This article starts from that difference and discusses the most important capability gaps for general-purpose task agents: intent understanding, sensible defaults, interaction pacing, and long-horizon autonomy. It also offers engineering recommendations.

1. The Essential Difference Between Closed Problems and Open Tasks

Closed problems have complete inputs, clear rules, and determinable answers. A model only needs to find an optimal or near-optimal solution within a limited solution space. Metrics such as accuracy, pass rate, and single-point tests suit this kind of task.

Open tasks differ. Users often give only a vague wish, such as “make this module more maintainable,” “help me find the intermittent timeout in production,” or “automatically complete this batch of data governance work.” At least three kinds of uncertainty are involved:

1. Goal uncertainty: The user states a means, but the real goal may be stability, delivery speed, or lower maintenance cost.
2. Environment uncertainty: Dependency versions, permissions, data quality, and the state of upstream and downstream systems can all change the execution path.
3. Acceptance uncertainty: “Done” rarely has a single standard, and it may require several rounds of alignment.

If a model applies test-taking logic to open tasks, it tends to look for one seemingly standard answer. It then relies on extensive unit tests, smoke tests, and canary checks to correct misunderstandings. These checks matter. However, when intent understanding is weak up front, verification becomes a way to spend engineering effort covering a semantic gap.

2. Solving Problems Is Not the Same as Doing Work: Over-Caution and Intent Drift

Test-taking models show two common problems on open tasks: excessive caution and weak intent understanding.

Excessive caution appears when a model faces a vague request and refuses to move forward, asks for confirmation repeatedly, offers conservative and vague plans, or treats every boundary condition as high risk. This looks safe, but it lowers the task completion rate. In real work, users usually prefer that the system propose a reasonable starting point for discussion, then refine it during execution.

Weak intent understanding appears when a model grasps the literal request but misses the real constraints in the context. For example, when a user asks to “reduce duplicate code,” a model might mechanically extract shared functions without considering module boundaries, release cadence, or the team’s maintenance cost. The code looks cleaner but becomes harder to maintain.

Therefore, the key to general-task capability is not knowledge at a single point. It is the ability to map a user’s vague wording onto a reasonable solution space. From a few stated wishes, the surrounding context, and historical constraints, the model needs to converge quickly on a sufficiently reasonable default plan. It should not wait for the user to write every detail into a specification.

3. Intent Understanding and Sensible Defaults: From Wishes to Executable Plans

Intent understanding is more than guessing what the user wants. It is a probabilistic constraint process. The model uses the user’s prior information, task context, domain knowledge, and system state to form a posterior plan, one that is likely to fall within what the user would accept.

Engineering this process breaks down into four steps:

1. Extract explicit goals: Identify what the user has stated and which hard constraints must be met.
2. Infer implicit goals: Based on the scenario, infer soft constraints such as stability, cost, maintainability, and delivery speed.
3. Generate candidate plans: Produce a small set of comparable paths instead of a single answer.
4. Choose a sensible default: When information is incomplete, select a default that is low risk, reversible, and easy to explain.

The process can be expressed in pseudocode:

def plan_task(user_goal, context):
    explicit = extract_constraints(user_goal)
    implicit = infer_priorities(context, explicit)
    candidates = generate_candidates(explicit, implicit)
    ranked = rank_by_risk_and_value(candidates)
    if ambiguity_is_high(ranked) and impact_is_large(ranked):
        return ask_user_for_key_choice(ranked)
    return choose_sane_default(ranked)

A sensible default here is not a guess. It is a constrained default. It prefers reversible actions, keeps existing behavior where possible, commits in small steps, and surfaces choices at critical branching points. Even if the model misreads the intent, the user can correct it at low cost.

4. Interaction Pacing: When to Ask and When to Proceed

General-purpose systems most often fall into one of two extremes. One is asking users constantly, which shifts decision cost onto people. The other is running ahead without checks and delivering a result that drifts from the goal. A better approach is to manage interaction pacing deliberately.

You can decide whether to ask by weighing impact against uncertainty:

• High impact, high uncertainty: Always ask. Examples include deleting data, changing public interfaces, and adjusting billing logic.
• High impact, low uncertainty: Follow established standards and keep an auditable record.
• Low impact, high uncertainty: Apply a sensible default, move quickly, and keep a rollback point.
• Low impact, low uncertainty: Execute directly to minimize interruptions.

This strategy is essentially a capability for “managing upward.” The system understands goal priorities, identifies critical branching points, asks for confirmation at the right moments, and advances autonomously on non-critical paths. For models, this requires dedicated training or explicit strategy orchestration. For enterprise systems, it means building permissions, risk levels, rollback mechanisms, and audit logs into task orchestration.

5. Long-Horizon Tasks and Autonomy: From Single-Step Correctness to Controllable Processes

Real general-purpose tasks rarely involve a single generation. They are long processes of multi-step planning, execution, observation, and correction. A model may perform well on individual steps and have broad world knowledge and development experience, yet still show goal drift, lost state, accumulating errors, and premature termination on long tasks.

Improving long-horizon autonomy requires three layers of design:

1. Task state machine: Split the task into observable stages, and record the current goal, completed steps, pending confirmations, and rollback points.
2. Execution feedback loop: Read environment feedback at every step instead of relying only on the model’s own report of completion.
3. Evaluation and stop-loss mechanisms: Set quality, budget, and risk thresholds. When a threshold is reached, pause, ask, or roll back.

At the infrastructure level, teams can use Baidu AI Cloud’s general-purpose cloud infrastructure, such as container platforms, message queues, and logging and monitoring services, to support task orchestration, state persistence, and observability. The core remains intent understanding and decision strategy. The cloud infrastructure keeps the process runnable, traceable, and recoverable.

6. Engineering Recommendations: Bringing Intent Understanding into Evaluation

If a team builds a general-purpose task agent, it should not evaluate the model on benchmark accuracy alone. It should add the following dimensions:

• Intent convergence rate: The share of first plans under vague requirements that users accept, either as is or after minor changes.
• Unnecessary confirmation rate: The share of questions that were not needed.
• Default adoption rate: The share of sensible defaults that users keep.
• Long-horizon task completion rate: The share of multi-step tasks that meet acceptance criteria within budget.
• Rollback and stop-loss effectiveness: Whether errors can be recovered at low cost.

The system design should also follow three principles: make critical branching points explicit, make non-critical paths default, and make every high-risk operation reversible. Only then can a model move from test-taking toward task-doing.

Conclusion

The bottleneck for general-purpose task capability often lies not in isolated knowledge but in intent understanding, sensible defaults, interaction pacing, and long-horizon autonomy in open environments. Test questions have standard answers. Real work requires continuously aligning with goals, managing uncertainty, and controlling process risk. Treating intent understanding as a dedicated capability to train and evaluate, and designing sensible defaults and interaction strategies as system capabilities, is the key path from test-taking models to general-purpose task agents.