LLM Code Generation and Multi-Agent Collaboration: A Practical Deployment Checklist

    This article explains how to evaluate LLM code generation for real engineering work and how to deploy local and multi-agent workflows safely. It covers acceptance criteria, sandboxed closed-loop execution, orchestration roles and messaging, cost-per-passed-task comparison, cloud controls, and common selection mistakes, with checklists that teams can apply directly.

    Large language models have evolved from text Q&A tools into execution units in the development workflow. When evaluating them, leaderboard scores only show how a model performs on standard benchmark tasks. What really determines whether a model can be deployed is whether it can deliver runnable code for specific engineering tasks, whether it can self-repair after failures, and how much each passed task costs. This article examines the key questions and provides a checklist that can be applied directly.

    1. How to Validate Code Generation Quality

    To evaluate code generation capability, first write the task clearly. Take “a small web app for monitoring earnings-season intelligence” as an example. The requirements need to include the technology stack, data scope, and acceptance conditions. For example: the frontend uses React and TypeScript, the backend uses FastAPI, and the data source is a fixed JSON sample file containing 20 earnings records.

    Acceptance conditions should be checkable item by item:

    • The page loads normally, and list filtering, sorting by revenue, and chart switching work.
    • All sample data is rendered correctly, with no hard-coded fake data disguised as API responses.
    • Unit tests pass, the code passes static checks, and there are no unhandled exception branches.
    • Extension endpoints reserved by the backend are documented and consistent with how the frontend calls them.

    When comparing different models on the same task, fix the prompt, context, and sample data, and record whether the first generation passes all acceptance items. Some models can generate basically usable code in one pass on such well-bounded tasks, but results vary with prompt wording, context length, and task scope and should not be treated as a stable capability.

    Generated code should first be written to an independent branch or temporary directory, without directly overwriting existing files. If tests fail, restore to the last version that passed tests.

    2. How Local Agents Can Execute Safely in a Closed Loop

    A local command-line Agent usually goes through four steps:

    1. Environment perception: read the project directory structure, dependency files (such as package.json and requirements.txt), and relevant historical code.
    2. Execution and testing: run test scripts after changes and collect output.
    3. Self-feedback: when tests fail, parse error logs, analyze causes, and plan the next modification.
    4. Human confirmation and merge: merge changes into the main branch only after tests pass and human review.

    Steps 1 through 3 can be automated, but they must run in a controlled environment. Deployment should meet the following requirements:

    • Least privilege: The Agent can only read and write the target project directory. It must not access key files under the user home directory (such as ~/.ssh), credential files referenced by environment variables, or production database connection configurations.
    • Sandbox isolation: Run the Agent in a container or a separate Git worktree. Network access is disabled by default; when external services are truly needed, allow only whitelisted addresses.
    • Test coverage requirement: Modules without tests must not be automatically modified by the Agent. If modification is necessary, the Agent or a developer should add test cases first.
    • Iteration cap: Set an upper limit on automatic repair attempts for a single task, for example 3 to 5. After the limit is exceeded, the Agent stops and hands the issue to a human.
    • Human confirmation points: Human confirmation is required before deleting files, changing dependency versions, running database migrations, or pushing code.
    • Rollback method: Record a snapshot after each round of changes. On failure, restore the worktree to the specified snapshot. Do not perform forced resets or deletions on the main branch.

    The Agent should not run directly in the production directory or on the main branch. This point is more important than which model is selected.

    3. How to Orchestrate Multiple Agents

    A single Agent is suitable for well-bounded tasks. For example, earnings data cleaning, metric extraction, and format conversion only require one Agent to process according to fixed steps. Multiple Agents are needed only when a task spans multiple domains or requires staged planning and verification.

    Three common roles are as follows:

    Role Input Output Acceptance criteria Failure handling
    Planner User requirements, project status Subtask list with dependencies Each subtask has clear input, output, and completion conditions Ask questions when requirements are ambiguous; do not expand scope without authorization
    Executor A single subtask, relevant files Code changes or data results All corresponding tests pass Retry according to rules; hand to a human after the limit is exceeded
    Verifier Diff, test report Pass or reject with reasons All checklist items are satisfied Return to the executor with specific issues when rejected

    The orchestration flow proceeds in the following order:

    1. The Planner breaks requirements into subtasks and marks dependencies.
    2. The Executor processes subtasks in dependency order. Independent subtasks can run in parallel.
    3. The Verifier reviews the changes and test results for each subtask.
    4. Merge after all items pass; if any item is rejected, return the task to the corresponding Executor.

    Message protocol: Each message should include task ID, sender, receiver, status (pending, running, completed, failed), input reference, and artifact reference. A reference can be a file path or content hash. Roles pass references rather than entire contexts, which can significantly reduce token consumption.

    Communication cost control: Limit the number of message rounds per task, and pass only summaries and necessary fields between roles. The Verifier only needs the diff and test report, not the entire codebase.

    Failure retry and idempotency checklist:

    • Each subtask has a unique ID, and repeated execution produces the same result. For example, when writing files, use overwrite writes, not append writes.
    • Retry at most 3 times with exponential backoff.
    • Operations with external side effects, such as sending email, placing orders, or deploying, are executed only once. Check the state record before execution and skip if already executed.
    • Classify failure reasons as retryable or non-retryable. Timeouts and rate limiting are retryable; insufficient permissions and unclear requirements are non-retryable and should be handed directly to a human.
    • Log every state change for later tracing.

    4. Comparing Costs on a Consistent Basis

    When comparing model costs, do not look only at unit price or leaderboard scores. For the same task, different numbers of model calls can lead to large differences in total cost.

    Record the following metrics on a fixed task set:

    • Task pass rate: number of tasks passing all acceptance items divided by total number of tasks.
    • Token consumption per task: sum of input and output tokens across all calls.
    • Number of retries and average rounds.
    • End-to-end latency.
    • Number of human interventions.
    • Cost per passed task: total cost divided by number of passed tasks.

    Below is a hypothetical calculation example. The numbers are only for illustrating the calculation method, not measured results. Suppose the task set contains 20 tasks:

    • Model A: 16 passed, total cost 80 yuan, cost per passed task 5 yuan.
    • Model B: 12 passed, total cost 72 yuan, cost per passed task 6 yuan.

    Model B has lower total cost but fewer passed tasks, so its cost per passed task is higher. Use cost per passed task as the primary comparison metric in model selection.

    Quotas and rate limits also affect Agent design. Deployment can check the following items:

    • Set a concurrency cap on the task queue so request volume does not exceed the provider’s requests-per-minute and tokens-per-minute quotas.
    • Use exponential backoff on rate-limit responses, not immediate retries.
    • Split long tasks into small steps that can resume from checkpoints, and record progress.
    • Set a token budget for each task; pause and notify a human when the budget is exceeded.
    • Reserve quota during peak periods to prevent batch tasks from exhausting the limit.

    5. Cloud Deployment and Data Security

    Multi-agent systems are often deployed on the cloud. The following describes the purpose of several general cloud capabilities. Specific products should be selected based on the team’s existing environment; this article does not constitute a product recommendation:

    • Container platform: isolate runtime environments for different Agents and scale elastically with task volume.
    • Object storage: store sample data and intermediate artifacts, with access policies.
    • Logging and monitoring: record call chains, token consumption, and abnormal retries, and set alerts.

    For security, follow these rules:

    • Each Agent uses an independent service account with only the permissions required to complete its tasks.
    • Secrets are stored in a key management service and not written to code, configuration files, or logs.
    • Data sent to the model should be desensitized first. Undisclosed earnings data and personal information of employees and customers should be removed or replaced.
    • Record audit logs, including which Agent performed what operation at what time.
    • Set alert rules covering sudden token consumption spikes, repeated retries of the same task, and unauthorized access attempts.

    6. Common Mistakes in Model Selection

    1. Looking only at leaderboard scores and not at pass rate and cost per passed task on your own tasks.
    2. Giving the Agent excessive permissions and allowing it to modify the main branch directly.
    3. Having no iteration cap or human confirmation points, so the Agent keeps consuming tokens after a task fails.

    Large language models can complete increasingly long coding tasks, but their output still needs to be constrained by tests, permission limits, and human review. In model selection, define the task, acceptance conditions, and cost basis first, then compare different models; the results will be more reliable.