Autonomous LLM Development After 30 Days: 3 Billion Tokens, 110K Lines of Code, and a Capability Assessment
This article presents a 30-day black-box experiment on autonomous LLM-driven software development. With zero human code review, two models consumed about 3 billion tokens and produced roughly 110,000 lines of code while building backend services and algorithm logic. The experiment evaluates five dimensions: design and code structure, coding correctness, bug localization and repair, autonomous end-to-end test closure, and token/quota durability. It finds that structured design output is already mature, but the models diverge sharply in diagnostic depth: root-cause-oriented models converge faster and avoid patch cascades, while surface-level models tend to create whack-a-mole fixes. Unattended closure is achievable only with structured logs, binary acceptance criteria, and retry/circuit-breaker controls. The article concludes with engineering recommendations for gradual adoption and cost evaluation based on tokens consumed per successfully delivered task.
1. Experiment Background and Goals
1.1 Why We Ran This Experiment
Over the past year, LLM-powered coding assistants have evolved from completion tools into agents capable of taking over a closed task loop. There has been extensive community discussion about whether AI can independently deliver a complete service, but most of it remains at the level of short sessions and small tasks, lacking systematic validation over long periods and large workloads.
The core question of this experiment was: Under a fully black-box setup with zero human code inspection, can current mainstream LLM agents autonomously complete the full loop of design → coding → debugging → end-to-end testing and reliably deliver a runnable service?
To answer this question, the experiment used highly extreme conditions:
- Duration: 30 consecutive days of uninterrupted operation;
- Scale: about 3 billion tokens consumed in total and roughly 110,000 lines of code produced;
- Human involvement: apart from configuring tasks and final acceptance, no one read or reviewed AI-generated code, and no one actively corrected the model’s process behavior;
- Evaluation criterion: delivery was considered successful if the final end-to-end test script passed; code quality and readability were not part of the assessment.
1.2 Experimental Subjects and Task Definition
The experiment selected two capability carriers for side-by-side comparison:
- Task scheduling bus: build an “AI supervisor” role responsible for scheduling local containerized workers to execute decomposed subtasks, simulating a minimal closed loop for multi-agent collaboration;
- Project scope: cover only backend services and algorithm logic, excluding the frontend UI; the core output was a runnable service process and supporting scripts.
The two compared models came from a coding subscription service offered by a leading large-model vendor and an open-source flagship model from another Chinese vendor. Both ran at the highest available quota during the experiment. One model’s weekly quota was fully exhausted within four weeks, which reflects the enormous token appetite of this “no human intervention, continuous trial and error” operating mode.
2. Evaluation Dimensions
To make “autonomous AI development” measurable, capability first had to be broken down into observable subdimensions. The experiment established a baseline across five dimensions:
| Dimension | Subdimension | Evaluation criterion |
|---|---|---|
| A | Solution design and code structure | Consistency of directories, files, and path naming with the initial plan |
| B | Coding correctness | Compile/start success rate and runtime crash frequency |
| C | Bug localization and repair | Whether a single conversation can provide the root cause and a minimal change; ability to detect deep design flaws |
| D | Autonomous end-to-end test closure | Whether the “log analysis → modification → retest” loop can run unattended |
| E | Token/quota durability | Whether it can sustain long-running, high-frequency trial and error |
The value of this breakdown is that it turns the vague question “Is AI good at writing code?” into five engineering metrics that can be measured and attributed independently.
3. Measured Results by Dimension
3.1 Design and Coding Phase: Good Structural Consistency, but Over-Engineering Is Common
The two models showed highly similar behavior patterns in this phase:
- Understanding capability was sufficient: both could correctly understand the structure and conventions of an existing codebase, output a design plan in Markdown, and explicitly list new and modified file paths;
- Structural consistency was high: the probability that the generated directory structure matched human expectations exceeded 80%, indicating that current models have internalized common engineering conventions quite well;
- Over-engineering was widespread: the models proactively introduced unnecessary abstraction layers, redundant configuration items, and defensive code. One interesting finding was that after removing plugins used for specification documentation (open-spec-style tools), over-engineering was significantly reduced. The “noise” from prompts and toolchains itself can induce architectural inflation in models;
- Low-level implementation errors were unavoidable: both models made detail-level mistakes involving boundary conditions, concurrency timing, dependency versions, and similar issues, requiring multiple rounds of local repair. The difference between them was not significant, so they can be placed in the same capability tier.
3.2 Bug Diagnosis and Repair: Where the Real Gap Appears
This was the most clearly differentiated part of the entire experiment:
| Metric | Model A (reasoning-oriented) | Model B (general-purpose) |
|---|---|---|
| Localization depth | Tended to provide a root cause at the design level in one pass | Mostly stayed at surface-level code errors |
| Interaction rounds | Converged in an average of 1.x rounds | Often required 3–5 rounds of guidance |
| Solution completeness | First proposed multiple options for confirmation | Tended to modify code directly and needed interception mid-process |
| Probability of incorrect patches | Low | Higher, and likely to trigger new issues in a chain |
The conclusion can be summarized as two repair styles: root-cause-oriented vs. surface-level:
- A root-cause-oriented model first asks “Why does this bug exist?” and often finds that a surface error is backed by a design problem in an interface contract or module boundary, so one modification can eliminate an entire class of errors;
- A surface-level model tends to “change wherever the error appears,” fixing one bug and introducing two new ones, forming a whack-a-mole chain of repairs. This is the most dangerous failure mode in unattended scenarios: with no one nearby to stop it, it amplifies errors like a snowball.
3.3 Autonomous End-to-End Test Closure: Achievable, but with Preconditions
The unattended loop of “log analysis → modification → retest” can run successfully, but the success rate depends heavily on three preconditions:
- Logs and error information must be structured. The model’s accuracy in parsing unstructured text logs was noticeably lower than with structured output containing error codes and stack traces. After enforcing a unified error format on the worker side, the closure success rate improved significantly;
- Test scripts must have clear acceptance boundaries. “Test passed” must be binary and programmatically determinable; otherwise, the model will produce self-persuasive “false closure” under ambiguous acceptance criteria;
- Retry limits and circuit-breaker mechanisms must be set. During the experiment, a model repeatedly attempted the same error more than ten times. A circuit-breaker logic must be added at the scheduling layer to “pause and escalate to a human after N consecutive failures.”
An illustrative scheduling pseudocode is as follows:
def autonomous_loop(task, max_retries=5):for attempt in range(max_retries):result = worker.run(task)if result.exit_code == 0 and e2e_test(task).passed:return SUCCESSdiagnosis = agent.analyze(result.structured_log)patch = agent.propose_minimal_fix(diagnosis)if patch.confidence < THRESHOLD:escalate_to_human(task, diagnosis) # circuit-breaker escalationreturn PAUSEDworker.apply(patch)return FAILED_AFTER_RETRIES
3.4 Token Durability: An Underestimated Hidden Cost
The consumption of 3 billion tokens revealed a commonly overlooked issue: the main cost of autonomous development is not “writing code” but “trial and error.” A single failed repair loop—reading logs, reading context, generating a patch, and rerunning tests—can easily consume hundreds of thousands of tokens, and failure is the norm. This means:
- Quota models based on weekly or monthly allowances become a hard bottleneck in continuous trial-and-error scenarios;
- Context-window utilization management—which files must remain resident and which can be loaded on demand—directly determines the cost per task.
4. Conclusions and Engineering Recommendations
Based on a month of measured results, the following conclusions can be drawn:
- Structured output is already mature: the consistency of solution design and directory organization is sufficient to support a semi-automated process in which “AI proposes a plan and humans review it quickly”;
- The gap lies in diagnostic depth, not coding speed: when selecting models, teams should focus on root-cause analysis capability and interaction discipline—discussing a solution before acting—rather than looking only at code generation speed;
- Unattended operation is a system problem, not just a model problem: structured logs, binary acceptance, and circuit-breaker escalation are three engineering foundations that improve autonomous closure success more than switching to a stronger model;
- Cost models must be included in evaluation: using “tokens consumed per successfully delivered task” as an efficiency metric reflects true cost-effectiveness better than code line count alone.
For teams that want to introduce intelligent coding assistants into production environments, a gradual path is recommended: first enable an “AI-autonomous + human acceptance” mode on highly isolated backend tasks, accumulate design experience with circuit breakers and acceptance criteria, and then gradually reduce the depth of human involvement—so that process guardrails are in place before model capability is pushed further.
