Layered Practices for Large Model Agents in Complex Engineering Scenarios

1. From “Answering” to “Delivering”: Layered Requirements for Complex Tasks

In the past, the value of large models in software engineering, knowledge question answering, and general reasoning was mainly reflected in single-turn or short-chain tasks: the model understands a question, generates an answer, helps diagnose an error, completes a code snippet, or explains a technical concept. As engineering scenarios become more complex, task boundaries begin to exceed the scope of “one answer.”

Examples include operating system module development, formalization of mathematical theorems, theoretical physics calculations, electronic hardware design, robot control logic verification, and historical vulnerability remediation. These tasks often require coordination across stages, tools, and domains of knowledge.

Such tasks usually share several common characteristics:

1. Long cycle: The task is not completed in one pass. It requires continuous planning, execution, rollback, correction, and validation.
2. High context density: The system must simultaneously understand code, specifications, tests, logs, mathematical definitions, physical constraints, hardware parameters, or other domain-specific inputs.
3. Strong tool dependency: The model cannot only “generate an answer.” It must also invoke external tools such as compilers, test runners, simulators, static analyzers, version control systems, and formal verification tools.
4. High verifiability requirements: In production systems, security patches, and formal proofs, outputs must be validated by explicit evidence.
5. Human critical decision points: For goal definition, risk judgment, compliance approval, and final release, humans still need to take final responsibility.

Therefore, complex tasks are not suitable for relying only on a single large model. A more reasonable approach is to embed large models into an agent orchestration system, allowing different models, agents, tools, and verification mechanisms to work together.

2. Core Idea: Use High-Capability Models for Complex Tasks and Low-Cost Models for Routine Tasks

In complex tasks, a common mistake is to use the strongest model for every task. This may improve accuracy in some critical steps, but it significantly increases cost, latency, and resource consumption.

A more mature strategy is model-tier routing: assigning tasks to models of different capability levels based on task complexity, risk level, reversibility, and verification cost.

Tasks can be divided into several tiers:

Task tier	Typical scenarios	Model strategy
Basic question answering	Concept explanation, document retrieval, simple Q&A	Use low-cost, low-latency models
Routine generation	Comments, unit tests, local fixes, format conversion	Use balanced general-purpose models
Complex engineering	Module refactoring, cross-file changes, defect localization	Use higher-capability models with verification tools
High-risk tasks	Production changes, security fixes, formal proofs, low-level systems	Use high-reasoning models with mandatory human approval

The value of this layered approach includes:

• Controllable cost: Most routine tasks do not need to consume high-end model resources.
• Stable quality: Critical tasks are handled by stronger models, reducing the risk of error propagation.
• Governable workflow: Different models can be mapped to different permissions, budgets, and review policies.
• Team acceptance: Engineers do not have to accept a black-box mode where “all changes are automatically committed.” They can review outputs as needed.

A simplified model routing strategy can be represented as follows:

def select_model(task):
    if task.risk_level in ["production", "security", "formal_proof"]:
        return "high_reasoning_model"
    if task.complexity >= 8:
        return "strong_general_model"
    if task.complexity >= 4:
        return "balanced_model"
    return "low_cost_model"

In this logic, risk_level and complexity should not rely only on the model’s own judgment. They should come from task templates, code impact scope, test coverage, historical defect rates, and human annotation.

3. Agent Layering: Orchestrator, Specialists, Verification, and Human Nodes

In complex engineering tasks, a single agent is rarely able to complete planning, execution, validation, and risk control at the same time. A more reasonable approach is to build a multi-layer agent collaboration system.

3.1 Orchestrator Agent

The orchestrator agent is responsible for task decomposition, goal management, and execution scheduling. It does not complete all work directly. Instead, it decides “who does what, when to do it, and what counts as completion.”

Its main responsibilities include:

• Breaking down high-level goals into subtasks.
• Assigning resources, models, and tools to subtasks.
• Tracking task status and identifying blockers.
• Summarizing intermediate results into stage-level conclusions.
• Triggering validation, rollback, or human approval.

For example, for the task “develop a bootable minimal system,” the orchestrator agent does not generate all code at once. Instead, it first forms a task tree:

Bootable minimal system
├── Requirements and interface definition
├── Boot process design
├── Memory and device abstraction
├── Basic file system capability
├── Build and image generation
├── Boot testing
└── Troubleshooting and repair

3.2 Specialist Agents

Specialist agents focus on specific domains, such as code implementation, mathematical proof, physical modeling, hardware constraint solving, or robot control strategy generation. They usually have clearer tool sets and knowledge boundaries.

Examples include:

• A code specialist agent writes implementations, adds tests, and fixes compilation issues.
• A proof specialist agent decomposes a theorem into lemmas and calls formal verification tools to check steps.
• A simulation specialist agent generates experiment plans, runs simulations, and analyzes curves and errors.
• A security specialist agent identifies dangerous patterns, generates patches, and checks permissions and input validation.

Outputs from specialist agents should be as structured as possible rather than only natural-language conclusions. For example, a code change should include diffs, test commands, impact scope, and rollback plans. A proof task should include propositions, assumptions, lemmas, verification steps, and failure reasons.

3.3 Verification Agents

Verification agents are an indispensable part of complex tasks. Their role is not to “generate another answer,” but to independently check results.

Common verification methods include:

• Compilation verification: Confirm that code can be built.
• Unit testing: Confirm that local logic behaves as expected.
• Integration testing: Confirm that interfaces between modules are correct.
• Static scanning: Detect null pointer issues, out-of-bounds access, resource leaks, injection risks, and other defects.
• Formal verification: Check whether mathematical properties, protocol state machines, or control logic satisfy invariants.
• Simulation verification: Validate physical, hardware, or robot control behavior.
• Regression testing: Confirm that fixes do not introduce new problems.

The key principle for verification agents is: do not trust a single source. Content generated by a model must be confirmed again through tools, tests, or human-defined rules.

3.4 Human Review Nodes

Even in highly automated agent systems, humans must remain responsible for key review points. Human confirmation is especially needed in the following cases:

• Production environment changes.
• Sensitive logic involving permissions, secrets, authentication, or billing.
• Business rules that cannot be fully covered by automated tests.
• Legal, compliance, or intellectual property risks.
• Key assumptions in complex proofs.
• High-risk security patches.

Human review does not mean that humans must handle every detail. It means confirming the objective, boundaries, risks, and acceptance criteria.

4. Workflow Design for Complex Tasks

A practical workflow for complex tasks usually includes the following stages.

4.1 Structured Objectives

The input should not be a vague request such as “please develop a system” or “please prove this theorem.” It should be converted into a structured objective. For example:

task:
  name: basic_boot_system
  goal: produce_a_bootable_minimal_system
  constraints:
    - must_build_successfully
    - must_pass_boot_test
    - must_include_logging
    - must_be_reviewable
  deliverables:
    - source_code
    - build_artifacts
    - test_report
    - risk_list

Structured objectives make later decomposition, assignment, and acceptance easier.

4.2 Task Graph Generation

The orchestrator agent converts the objective into a task graph. A task graph is usually a directed acyclic graph that clearly defines dependencies between tasks.

Requirements analysis -> Interface design -> Module implementation -> Build -> Test -> Fix -> Acceptance

Each node should include:

• Input materials.
• Output artifacts.
• Completion criteria.
• Required tools.
• Model tier.
• Risk level.
• Timeout and retry strategy.

4.3 Agent Execution

During execution, agents should follow the principle of “small steps and frequent validation.” For tasks involving code, proofs, and control logic, it is not recommended to generate the full result in one pass. Instead, the work should be split into multiple verifiable pieces.

For example, in a code modification task, the following rules can be applied:

Each change must not exceed one logical module.
The build must be run after the change.
Related tests must be run after the change.
An impact statement for the change must be output.
The next step must not begin if validation fails.

This mechanism effectively reduces error accumulation in long-chain tasks.

4.4 Evidence Chain Preservation

Complex tasks should not retain only the final conclusion. They should preserve the complete evidence chain. The evidence chain includes:

• Task objectives and acceptance criteria.
• Diffs for each change.
• Test commands and results.
• Tool outputs.
• Failure records and repair processes.
• Model versions and prompt versions.
• Human approval records.

The evidence chain is important for later auditing, retrospective analysis, and continuous optimization.

5. Reference Technical Architecture

To engineer these capabilities, a layered architecture is required.

5.1 Control Plane

The control plane is responsible for task orchestration and governance. It includes:

• Task registration and lifecycle management.
• Agent scheduling.
• Model routing.
• Permission control.
• Budget and quota management.
• Human approval workflows.
• Audit logs.

The control plane should avoid placing all complex logic into prompts. Instead, it should use explicit state machines, task queues, and rule engines.

5.2 Execution Plane

The execution plane is responsible for running agents and invoking tools. It usually includes:

• Sandboxed runtime environments.
• Containerized toolchains.
• Code repository access.
• Build and test environments.
• Simulation environments.
• File and artifact management.
• Timeout, retry, and isolation mechanisms.

All external tool calls should be controlled to prevent agents from having permissions beyond what is necessary.

5.3 Data Plane

The data plane is used to store task process data. Common capabilities include:

• Object storage: Store build artifacts, test reports, and log bundles.
• Message queue: Drive asynchronous tasks and event notifications.
• Logging service: Record execution traces and error information.
• Monitoring and alerting: Track task success rate, latency, and failure reasons.
• Metrics platform: Measure human intervention rate, rework rate, and cost consumption.

5.4 Model Plane

The model plane is responsible for unified access to different models and implements routing, fallback, and caching. It should include:

• Model gateway: Unified authentication, rate limiting, and metering.
• Routing policy: Select models according to task level.
• Caching mechanism: Cache repeated queries and stable results.
• Fallback mechanism: Switch to alternative strategies when high-cost models are unavailable.
• Evaluation mechanism: Continuously evaluate model performance on specific tasks.

6. Typical Application Scenarios

6.1 Complex System Development

In operating system or base software tasks, an agent team can start from an empty directory and gradually complete interface definitions, module implementations, build scripts, boot processes, and test validation. The key is not to have the model write all code at once, but to ensure that each module can be built, tested, and rolled back.

Recommended control points:

• Each module must have a clear interface contract.
• Every commit must pass the build.
• Boot failures must preserve logs.
• Critical paths must have a minimal test set.
• Failures should generate automatic diagnostics rather than only reporting “failed.”

6.2 Formalized Mathematical Proofs

The key to mathematical proof tasks is logical verifiability. Agents should not only provide natural-language proofs. They should output structured proof steps and have them checked by formal verification tools.

Recommended process:

Restate the theorem
→ Clarify definitions
→ Decompose into lemmas
→ Select proof strategy
→ Perform step-by-step derivation
→ Run formal verification
→ Backtrack on failure

For steps that cannot be completed, the system should clearly record the failure location, missing definitions, or conflicting assumptions. It should not continue generating content that appears reasonable but cannot be verified.

6.3 Theoretical Physics and Simulation Computation

Theoretical physics tasks often involve modeling, approximation, numerical computation, and result interpretation. Agents can assist with problem formalization, parameter derivation, simulation script generation, and error analysis, but scientific validation is still required.

Recommended additions include:

• Dimensional checks.
• Boundary condition checks.
• Numerical stability checks.
• Comparison with known cases.
• Multi-parameter sensitivity analysis.
• Reproducibility checks.

6.4 Electronic Hardware and Robot Control

Hardware and robot tasks usually have strong constraints and strong feedback characteristics. Agents can generate design documents, control strategies, state machines, and test cases, but they must be validated through simulation or physical testing.

Key measures include:

• Clearly defining hardware constraints.
• Defining safety boundaries.
• Generating reproducible simulations.
• Performing stability analysis on control strategies.
• Running fault injection tests before deployment.
• Setting hardware-level or software-level protections for abnormal actions.

6.5 Historical Vulnerability Remediation

Security remediation tasks are suitable for a closed loop of discovery, validation, repair, and regression. Agents can read historical code versions, attempt to rediscover known vulnerability patterns, and generate fix patches.

Recommended steps:

Read historical version
→ Identify suspicious patterns
→ Construct trigger conditions
→ Verify the vulnerability exists
→ Generate a patch
→ Run regression tests
→ Run security scanning
→ Obtain human approval

These tasks especially require verification agents to participate, preventing superficial fixes that rely only on pattern matching.

7. Quality Control Mechanisms

Complex tasks often fail not because the model cannot perform them at all, but because errors are not detected in time. Therefore, quality gates must be placed early.

7.1 Verifiable Output Principle

All agent outputs should be converted into checkable results wherever possible:

• Code outputs should be buildable and testable.
• Proof outputs should be checkable by formal verification tools.
• Configuration outputs should be checkable by syntax and semantic checks.
• Documentation outputs should be searchable and comparable.
• Control strategies should be checkable through simulation or state machines.

7.2 Small-Step Submission Principle

Complex tasks should avoid submitting a large change all at once. Small-step submissions make problem localization easier and improve human review.

Recommended limits include:

• The number of files changed in a single submission.
• The impact scope of a single change.
• The runtime of a single task.
• The model call budget for a single task.
• The number of retries after a failure.

7.3 Cross-Validation Principle

For high-risk conclusions, different agents or different tools should be used for cross-validation. For example:

• Separate generation agents from verification agents.
• Separate code implementation from test generation.
• Separate proof derivation from formal checking.
• Compare simulation results with analytical results.
• Combine automated scanning with human review.

7.4 Rollback and Isolation Principle

Any automated change should support fast rollback. Execution environments should be isolated to avoid affecting production systems. For critical changes, the default path should be branch, pre-release, or sandbox validation.

8. Cost and Efficiency Governance

Multi-agent systems can easily lead to a surge in resource consumption. Without governance, it is possible to see “many model calls, little effective output.”

Cost can be controlled from the following aspects:

1. Model tiering: Use high-reasoning models only for difficult tasks. Use low-cost models for routine tasks.
2. Task budgets: Set maximum call counts, maximum runtime, and maximum tool calls for each task.
3. Result caching: Cache repeated retrievals, repeated builds, and repeated test results.
4. Incremental execution: Recompute only changed parts instead of rerunning everything.
5. Fail-fast behavior: When validation failures reach a threshold, trigger human intervention instead of continuing meaningless retries.
6. Value-oriented automation: Prioritize tasks that are verifiable, quantifiable, and reversible.

Efficiency should not be measured only by generation speed. Final delivery quality also matters. Recommended metrics include:

• Task completion rate.
• Human intervention rate.
• First-pass rate.
• Rework rate.
• Test coverage.
• Defect escape rate.
• Cost per task.
• Mean time to repair.
• Approval pass rate.

9. Risks and Boundaries

Multi-agent collaboration can improve the handling of complex tasks, but it also introduces new risks.

9.1 Error Amplification Risk

If the orchestrator agent misunderstands the goal, multiple specialist agents may continue working in the wrong direction. The solution is to introduce stage reviews and human confirmation at critical nodes.

9.2 Long-Run Drift Risk

Agents running for a long time may gradually drift from the original objective. The solution is to periodically generate summaries, realign with the original goal, and set stage-level acceptance checks.

9.3 Security and Permission Risk

When agents access code, logs, secrets, or production systems, they should follow the principle of least privilege. Recommended practices include:

• Secret isolation.
• Temporary credentials.
• Operation auditing.
• Environment isolation.
• Prohibiting direct access to production databases.
• Requiring human confirmation for high-risk commands.

9.4 Intellectual Property and Compliance Risk

Model-generated content may involve code licenses, data sources, and third-party dependencies. Source notes, dependency lists, and review records should be preserved.

9.5 Unverifiable Output Risk

For content that cannot be automatically verified, the level of automation should be reduced. For example, purely aesthetic judgments, complex business policies, and legal interpretations are not suitable for full automated decision-making by agents.

10. Implementation Recommendations

For enterprises that want to introduce complex-task agent capabilities, a practical starting point is as follows:

1. Select high-value scenarios: Prioritize tasks with clear processes, verifiable results, and high repeated cost, such as test completion, defect fixing, document generation, security scanning, and local refactoring.
2. Build task templates: Standardize objectives, constraints, acceptance criteria, and tool lists to avoid relying on temporary prompts.
3. Set up sandbox environments: Ensure agents can build, test, and roll back in isolated environments.
4. Integrate a model gateway: Unify model routing, permissions, budgets, and auditing.
5. Introduce validation gates: Results that have not passed testing, scanning, or approval should not enter the next stage.
6. Expand scope gradually: Start with low-risk tasks and gradually extend to complex system development, formal verification, and high-security scenarios.
7. Measure and optimize continuously: Adjust model routing and agent responsibilities based on human intervention rate, rework rate, and cost changes.

Overall, the value of large models in complex tasks is not only to generate more content, but to organize “professional judgment, engineering execution, automatic validation, and human approval” into a controllable pipeline. Only when models, agents, tools, validation, and governance mechanisms work together can complex technical tasks move from experimental attempts to stable engineering capabilities.