From “Can Generate” to “Can Pass”: What AI Coding Evaluation Really Means

As AI coding capabilities continue to evolve, code generation tools have expanded from simple snippet completion to broader R&D stages such as requirements analysis, API design, business implementation, test generation, defect fixing, and refactoring suggestions. Mobile application development is especially challenging because of its runtime environment, resource limits, permission model, lifecycle management, and user interaction complexity. This makes it a valuable benchmark for assessing AI coding capabilities.

In recent industry evaluations, an interesting phenomenon has emerged: some AI coding models have a high completion rate but a noticeably low pass rate. Completion rate generally measures whether a model can produce a complete answer or close the task loop. Pass rate is stricter: it measures whether the generated code or solution can pass compilation, unit testing, functional verification, code review, or final acceptance. A high completion rate with a low pass rate shows that a model can “write something,” but not necessarily “make it run, fix correctly, and deliver.”

This gap reveals multiple weaknesses in mobile app development. A model may understand surface requirements but ignore platform constraints; it may generate seemingly complete code but omit exception handling; it may provide functional implementation without considering performance, security, compatibility, or maintainability. Therefore, building more rigorous evaluation systems and engineering practices around mobile app development has become an important topic for AI coding adoption.

Why Completion Rate and Pass Rate Diverge

1. Completion Rate Focuses on “Is the Answer Complete?”

Completion rate reflects whether a model can complete the task output. For example, if a user asks for a network request wrapper, a state management module, or an exception-handling flow, can the model provide complete code, complete explanation, or complete steps? If the model can output a fairly complete solution, it may be considered “completed,” even if there are minor issues.

This metric matters for user experience. When developers use AI coding tools, they first want a workable starting point: something they can edit, reference, and discuss. If a model often interrupts, misses key steps, or fails to produce structured results, the experience suffers even if the final correctness is not low.

2. Pass Rate Focuses on “Can It Really Work?”

Pass rate is closer to real R&D delivery standards. Mobile app code is not an isolated snippet; it must run inside a full engineering system. It must compile, conform to platform rules, handle exception paths, work across device states, and meet performance, security, privacy, and testability requirements.

Pass rate usually includes multiple acceptance conditions:

• Whether the code is syntactically and compilably valid;
• Whether core functionality meets expectations;
• Whether exception paths are covered;
• Whether platform rules or privacy requirements are violated;
• Whether obvious performance problems exist;
• Whether the code is easy to test and maintain later;
• Whether it follows team engineering standards.

When completion rate is high but pass rate is low, the model is better at “providing an answer” than “providing a qualified engineering implementation.” This reflects weaknesses in context understanding, constraint recognition, exception handling, and engineering validation.

3. Divergence Is Amplified in Mobile Development

In general code questions or single-file algorithm tasks, a model may only need to focus on local logic. In mobile app development, code is affected by many external conditions: device state, network state, permission state, system resources, user behavior, version differences, and lifecycle changes. If any condition is handled poorly, the generated code may not be usable in practice.

Mobile app development therefore acts as a magnifying glass for AI coding capability. It tests not only whether a model can write code, but also whether it understands complex runtime environments, recognizes implicit constraints, and makes safe and robust engineering decisions under uncertainty.

Common Weaknesses in Mobile App Development

1. Lifecycle and State Management Are Often Ignored

Mobile apps usually have clear lifecycle stages. When an app moves from foreground to background, a page is destroyed and rebuilt, a process is recovered after being reclaimed by the system, or the device rotates or enters split-screen mode, the code may need to save and restore state.

If an AI coding model only generates code based on the local requirement provided by the user, it may ignore these lifecycle scenarios and produce output that looks complete but is unstable. For example, data may be lost after page rebuilding, temporary variables may not be saved, asynchronous tasks may not be canceled at the right time, or background tasks may not release resources correctly. These can cause abnormal app behavior.

Such issues often do not appear in the first compilation. They may gradually appear during real usage, long-running sessions, weak network conditions, or resource-constrained environments. Therefore, mobile app evaluation must consider lifecycle and state recovery, not just whether the code runs once.

2. Permission and Privacy Handling Is Often Oversimplified

Mobile apps often need sensitive capabilities such as the camera, location, notifications, storage, and microphone. These capabilities usually involve user authorization, permission denial, permission revocation, minimal use, and compliance statements. If an AI coding model only generates “functional implementation” and ignores permission flows, the result may be unusable or create compliance risks.

Common weak points include:

• No fallback path after the user denies a permission;
• No differentiation between permission behavior across versions;
• Sensitive information printed in logs;
• Keys, tokens, or user privacy data hard-coded in the client;
• Permission requests that do not follow the principle of least privilege;
• Missing privacy explanations or user notification logic.

Unlike backend services, many security requirements in mobile apps are not limited to API layers. They also appear in user interaction, device capabilities, and data lifecycles. If a model lacks awareness of these constraints, its output is difficult to use directly in production.

3. Network Failures and Offline Scenarios Are Often Missed

Mobile app runtime environments are highly uncertain. Network quality may change from fast and stable to weak, lossy, disconnected, or frequently switching states. If generated code only covers normal request flows and ignores timeouts, retries, caching, idempotency, recovery, and offline degradation, it will be fragile in real environments.

Common weaknesses include:

• No timeout control for network requests;
• No retry limits or backoff strategies;
• No distinction between retryable and non-retryable errors;
• No handling for empty responses, abnormal responses, or partial failures;
• Unclear cache strategies that lead to inconsistent data;
• No offline fallback;
• Concurrent requests not merged, canceled, or deduplicated.

Mobile apps require higher tolerance for network exceptions. A qualified generated solution should make the feature work under normal network conditions and keep the app controllable, recoverable, and informative under abnormal network conditions.

4. Asynchronous and Concurrent Logic Can Introduce Hidden Defects

Mobile app development often involves asynchronous tasks, thread switching, callback chains, and state updates. If an AI coding model only focuses on business logic and ignores execution context, it may introduce blocking, race conditions, duplicate execution, memory leaks, or unreleased resources.

Typical weak points include:

• Running time-consuming tasks on the main thread where blocking should be avoided;
• Updating UI after a page has been destroyed when an asynchronous result returns;
• Multiple asynchronous tasks modifying the same state without coordination;
• Deep callback nesting that makes error tracing difficult;
• Missing task cancellation mechanisms that waste resources;
• Unclear state update ordering that causes intermittent problems.

These issues are often intermittent and hard to reproduce, which raises the bar for testing and debugging. Evaluation should therefore verify not only whether a feature appears, but also whether concurrent paths, exception paths, and boundary conditions are robust.

5. Resource Constraints and Performance Awareness Are Often Insufficient

Mobile devices have limited resources. Memory, battery, storage, network, and rendering performance can all become constraints. If an AI coding model only focuses on functional correctness, it may generate code that overloads resources through repeated requests, redundant computation, excessive animation, or frequent creation of large objects.

Common problems include:

• Frequent creation of large objects causing memory pressure;
• Images or resources without size, format, or cache management;
• Repeated computation or invalid rendering during list scrolling;
• Background tasks using too many resources;
• Animations or refresh rates that hurt smoothness;
• No consideration for low-memory, low-battery, or weak-network environments.

Mobile app development requires not only “correct” code, but also “reasonable” code. Performance should not be a later patch; it should be considered during code generation.

6. Compatibility and Version Differences Are Often Underestimated

Mobile apps often need to adapt to different devices, screen sizes, system versions, and runtime environments. If an AI coding model lacks clear constraints, it may use new features outside the target range or ignore behavioral differences in older versions, causing the generated code to fail on some devices.

Typical issues include:

• Using newer capabilities without compatibility handling;
• Ignoring different screen sizes and interaction methods;
• Ignoring orientation, resolution, or density differences;
• Ignoring behavior changes across versions;
• Ignoring special cases such as foldable devices, split-screen, or landscape mode.

Mobile app compatibility is not simply “adding adaptations.” It requires defining target devices, minimum support conditions, and degradation strategies from the requirements stage.

7. Insufficient Testability Increases Maintenance Costs

If AI-generated code is difficult to test, maintenance costs rise significantly. Mobile app logic is often tightly coupled with UI, device capabilities, and system state. If the model does not create reasonable separation, the generated code may be hard to unit test and hard to simulate in abnormal scenarios.

Common problems include:

• Business logic tightly coupled with UI logic;
• External dependencies difficult to replace;
• Missing interface abstractions on critical paths;
• Error handling scattered across the code and difficult to validate centrally;
• Unclear input and output boundaries;
• Generated results that cannot form repeatable test cases.

Testability is not an extra requirement; it is an important foundation for mobile engineering. When evaluating model output, teams should check whether the code naturally supports testing, dependency injection, and simulation of network, permission, storage, and lifecycle exceptions.

How to Design an AI Coding Evaluation System for Mobile Apps

1. Move from Single-Piece Code Evaluation to Layered Evaluation

Mobile app evaluation should not only check whether a piece of code can run. It should use a layered evaluation system:

• Syntax and compilation layer: Whether the code follows basic syntax and compiles;
• Functional correctness layer: Whether core logic meets requirements;
• Exception-handling layer: Whether timeouts, null values, permission denials, network failures, and resource shortages are handled;
• System integration layer: Whether the code correctly works with storage, networking, permissions, lifecycle, and other system capabilities;
• Performance and experience layer: Whether there are obvious blocking, jank, or resource overuse issues;
• Security and compliance layer: Whether sensitive information is exposed, permissions are abused, or storage is insecure;
• Maintainability layer: Whether the structure is clear and easy to extend, test, and review.

Layered evaluation avoids the one-sided judgment of “can it run?” and helps teams identify a model’s real capabilities across engineering dimensions.

2. Build a Mobile-Specific Scenario Library

Mobile app evaluation needs to cover many real-world scenarios. A scenario library can include:

• Normal network, weak network, no network, timeout, retry, cache hit, cache miss;
• Permission granted, permission denied, permission revoked, permission changed;
• App in foreground, background, page rebuilt, process restored;
• Low memory, low battery, resource constraints;
• Different screens, interaction methods, and runtime environments;
• Concurrent requests, repeated taps, rapid switching, task cancellation;
• Sensitive data write, read, delete, migration, encrypted storage;
• Abnormal logs, crash recovery, degradation prompts.

The closer the scenario library is to real user behavior, the more valuable the evaluation results. Mobile app evaluation cannot only verify ideal paths; it must also verify exception paths and boundary paths.

3. Metrics Should Cover Quality, Cost, and Efficiency

Mobile app coding evaluation should not only pursue “answer quality.” It should also consider development efficiency and usage cost. Recommended metrics include:

• Completion rate: Whether the model completes the task output;
• Pass rate: Whether the generated result passes predefined acceptance;
• First-pass rate: Whether the first generated result can pass;
• Average number of revision rounds: How many adjustments are needed to reach a usable state;
• Defect density: Number of exposed issues per unit of code or task;
• Test coverage: Whether the generated code is easy to test;
• Maintainability score: Structure, naming, layering, coupling, etc.;
• Number of security and compliance issues: Sensitive information, permission, or storage risks;
• Average usage cost: Computation or invocation cost required to complete the task;
• Response latency: Time from request to result;
• Reasoning level annotation: Performance differences under different reasoning configurations.

Reasoning level is an important variable. Complex tasks may require deeper reasoning, while simple completion or template generation may not. Evaluation should clarify the quality gains and cost trade-offs under different reasoning configurations rather than assuming “stronger is always better.”

4. Introduce an Automated Verification Loop

After code is generated, it should enter an automated verification loop. A feasible flow is:

Generate code
  -> Static analysis
  -> Compilation check
  -> Unit tests
  -> Scenario tests
  -> Security scan
  -> Performance baseline
  -> Manual review
  -> Merge or return for revision

The value of this loop is to turn “whether the model seems correct” into “whether the code has passed engineering validation.” For mobile app development, automated verification can significantly reduce manual review costs and expose hidden defects faster.

5. Evaluation Results Should Drive Improvement, Not Just Ranking

Leaderboards and scores are only presentation formats. The more important goal is to find problems. If a model performs poorly in exception paths, permission handling, lifecycle management, or performance optimization, the team can adjust task templates, context supplementation, verification rules, and knowledge base content.

The final goal of evaluation is not to decide which model is “better,” but to identify which capabilities can be used in which scenarios, which outputs need more manual review, which tasks need stronger reasoning, and which scenarios need stricter gates.

Engineering Practices to Improve AI Coding Output Quality

1. Constrain Model Output with Task Cards

Mobile app tasks often contain implicit constraints. To help models generate more usable code, task cards can define the goal, constraints, and output format. For example:

Task goal: Implement a network request wrapper with caching and retry
Constraints:
  - Must handle timeouts, empty responses, permission denial, and weak network switching
  - Sensitive information must not be output in logs
  - Must cover five test cases: normal, exception, retry, cache hit, and cache miss
  - Generated result must be easy to unit test
Output:
  - API definition
  - Implementation code
  - Error-handling matrix
  - Test checklist
  - Known limitations and risk notes

Task cards reduce deviations caused by free-form model output and make results closer to engineering requirements. For constrained scenarios such as mobile app development, task cards can significantly improve output controllability.

2. Provide Sufficient and Accurate Context

The quality of generated results largely depends on whether the context is sufficient. For mobile app development, teams should provide as much of the following as possible:

• Target runtime environment and version range;
• Development language and framework constraints;
• Project architecture layering;
• Existing APIs and dependencies;
• Coding standards;
• Common counterexamples;
• Historical defect patterns;
• Security and compliance requirements;
• Testing standards;
• Performance baselines.

If the context is insufficient, the model may infer based on general experience and generate content that seems reasonable but does not fit the actual project. For enterprise R&D, context governance itself is an important foundation for AI coding adoption.

3. Generate Complex Tasks in Stages

For complex mobile app tasks, asking the model to generate the entire implementation at once often makes quality difficult to guarantee. A more reliable approach is staged generation:

1. First generate requirements analysis and implementation plan;
2. Then generate API definitions and key structures;
3. Then generate core logic;
4. Then generate exception handling;
5. Then generate test samples;
6. Finally generate review checklist and optimization suggestions.

Staged generation makes each step checkable, verifiable, and rollback-able. Compared with one-shot output, this approach is better suited to mobile app scenarios with high coupling and strong constraints.

4. Strengthen Exception Paths and Boundary Conditions

Risk in mobile app code often lies not in normal paths but in exception paths. Engineering practice should require the model to explicitly explain:

• How to handle failures;
• How to degrade when the user denies authorization;
• How to prompt on network timeout;
• How to display empty data;
• How to discard asynchronous results after a page is destroyed;
• How to merge or cancel concurrent requests;
• How to protect sensitive data;
• How to release resources when resources are insufficient.

An error-handling matrix can be a fixed output item. For example:

Scenario	Trigger condition	Handling method	User perception	Testing suggestion
Network timeout	Request exceeds threshold	Retry or prompt	Loading failure prompt	Simulate weak network
Permission denied	User denies authorization	Degrade or guide	Limited-feature notice	Simulate denial
Empty data	Empty list returned	Show empty state	Empty-state prompt	Construct empty data
Asynchronous invalidation	Page already destroyed	Discard result	No perception	Rapid switching validation

Structured output helps reduce omissions and makes later testing and review easier.

5. Establish Static Checks and Rule Gates

Generated code should pass unified quality gates before entering the project. Gates can include:

• Code format check;
• Static syntax analysis;
• Security rule scan;
• Sensitive information detection;
• Dependency compliance check;
• API specification check;
• Logging specification check;
• Exception-handling rules;
• Test coverage requirements.

For mobile app development, gates can be further refined. For example, whether unnecessary permissions are requested, whether user privacy is output in logs, whether background switching is handled, whether there is a risk of blocking the main thread, and whether unreleased resources exist.

6. Use Feedback Loops to Continuously Improve Model Usage

AI coding is not a one-time action but a continuous iteration process. Teams should collect feedback such as:

• Which tasks have high completion rates but low pass rates;
• Which prompts are more likely to generate usable results;
• Which context supplements significantly improve pass rates;
• Which defect types recur frequently;
• Which scenarios require human intervention;
• Which test rules can be accumulated into automated gates.

This feedback can be used to optimize task templates, context libraries, review rules, and knowledge bases, thereby improving future generation quality.

Enterprise Adoption: Embed Evaluation into the Development Process

1. Bring AI Coding into the Engineering Quality Management System

When enterprises use AI coding models, they should not treat them as isolated tools. They should be integrated into engineering quality management systems. Generated code should go through the same or stricter review processes as human-written code, including:

• Code review;
• Automated tests;
• Security scanning;
• Performance evaluation;
• Compliance checks;
• Version management;
• Change auditing.

This helps avoid new quality risks introduced by generation efficiency. The value of AI coding is not to replace engineering standards, but to gain efficiency within those standards.

2. Establish Data Security and Privacy Protection Mechanisms

Mobile app development often involves user data, device information, and business-sensitive data. When using AI coding capabilities, teams should establish data protection mechanisms:

• Desensitize input context;
• Prohibit uploading unnecessary sensitive data;
• Perform security review on generated content;
• Constrain log output;
• Detect hard-coded sensitive information such as keys, tokens, and certificates;
• Perform compliance checks on permission requests and privacy statements.

Only by moving security requirements into task generation and acceptance can privacy and compliance risks after mobile app release be reduced.

3. Accumulate Team Knowledge and Counter-Example Libraries

The effectiveness of AI coding models is closely related to context quality. Enterprises can accumulate knowledge assets such as:

• Internal architecture standards;
• Common API definitions;
• Coding standards;
• Common defect patterns;
• Historical production issues;
• Counterexample code;
• Testing templates;
• Security and compliance requirements;
• Performance optimization experience.

These knowledge bases help models better understand enterprise engineering constraints and reduce deviations between generic output and actual project needs.

4. Complete the Evaluation Loop with Cloud-Based General Engineering Capabilities

Enterprises can connect mobile app code generation, evaluation, and verification processes with cloud-based general engineering capabilities to form a more complete R&D loop. For example, object storage can save evaluation samples, task cards, logs, and reports; message queues can decouple generation tasks, testing tasks, and review tasks; container platforms can provide isolated and repeatable test execution environments; logging services can record invocation processes, failure reasons, and quality metrics; monitoring and alerting can detect abnormal fluctuations, quality degradation, or cost anomalies.

This engineering approach does not depend on a single tool. It embeds AI coding capabilities into the development pipeline so that generation, verification, monitoring, and improvement form a continuous loop.

5. Promote Human-AI Collaboration Rather Than Full Automation

Mobile app development involves substantial business understanding, user judgment, and architecture decisions. AI coding models are suitable for tasks such as:

• Generating draft code;
• Completing repetitive logic;
• Generating test samples;
• Providing refactoring suggestions;
• Explaining exception reasons;
• Generating documentation;
• Assisting code review.

However, critical business logic, permission compliance, data security, and architecture decisions should still be confirmed by engineers. The best mode for AI coding is not full automation but human-AI collaboration: models improve efficiency, engineers make judgments and provide fallback, and automated testing ensures quality.

Conclusion: From “Can Generate” to “Can Run, Maintain, and Deliver”

AI coding evaluation in mobile app development reveals a clear gap between completion rate and pass rate. A high completion rate shows that models have strong generation capabilities. A low pass rate shows that real mobile app development requirements go far beyond code generation. Lifecycle, permissions, privacy, network exceptions, asynchronous concurrency, resource limits, compatibility, and testability together form a high barrier for mobile app development.

For development teams, the meaning of evaluation is not to compare rankings simply, but to identify capability boundaries, discover weak points, and build verification loops. What mobile app development really needs is not a seemingly complete piece of code, but an engineering implementation that can be compiled, tested, reviewed, maintained, and delivered.

Therefore, for AI coding to truly improve R&D productivity, it must be combined with task constraints, context engineering, automated verification, security gates, and cost governance. Only when model output moves from “can generate” to “can run, maintain, and deliver” can AI coding continuously create value in mobile app development.