Mobile Development AI Coding Evaluation and Engineering Selection Practices

1. From “Code Generation” to “Development Process Integration”: Why AI Coding Evaluation Matters

In mobile application development, AI coding assistance has evolved from simple code completion to broader scenarios such as requirement understanding, interface generation, bug fixing, unit test generation, code explanation, refactoring suggestions, and build script generation.

For engineering teams, the key concern is not whether a model can produce a plausible code snippet in a single demo. Instead, the focus is on whether it can reliably generate verifiable, maintainable, deployable, and low-risk code changes in real engineering environments.

Therefore, building an AI coding evaluation baseline for mobile development has become an important prerequisite for technology selection and engineering efficiency improvement. Mobile development is characterized by strong lifecycle requirements, complex compatibility issues, permission sensitivity, numerous build dependencies, and obvious performance constraints. Generic code generation capabilities alone are often insufficient for covering key difficulties in real development.

Teams need an evaluation system that reflects the actual distribution of development tasks so they can determine whether AI coding assistance is genuinely suitable for business requirements.

From an engineering practice perspective, the value of evaluation is mainly reflected in three aspects:

1. Reducing technology selection risk
   Different models vary significantly in code correctness, complex task understanding, tool invocation, long-context handling, and cost control. Choosing models only based on experience may lead to rework.

2. Improving engineering efficiency
   Through quantitative evaluation, AI assistance can be embedded into the most suitable development stages, such as test generation, code review, documentation completion, and defect localization.

3. Ensuring code quality and security
   Mobile development involves user permissions, privacy compliance, API stability, and runtime performance. The evaluation system must include quality gates and security checks.

AI coding capability evaluation is therefore not a one-time leaderboard comparison. It should be a continuously evolving engineering mechanism that is reproducible, regression-capable, and observable.

2. Mobile Development Scenario Specifics: Why Generic Coding Evaluation Cannot Be Directly Applied

Generic code evaluation often focuses on algorithm correctness, code conciseness, or the ability to implement a single feature. However, mobile application development faces more complex constraints.

2.1 Strong Lifecycle and State Management Requirements

Mobile applications usually run on resource-limited devices. Page switching, background execution, network recovery, memory reclamation, and other state changes occur frequently.

AI-generated code must not only implement functionality but also correctly handle lifecycle, thread safety, state restoration, and exception branches. Otherwise, even if unit tests pass, the code may still crash, freeze, or lose data on real devices.

2.2 Significant Compatibility and Device Differences

Different device models, screen sizes, system capabilities, permission policies, and runtime environments can cause the same code to behave differently across devices.

Evaluation must consider compatibility issues, such as low-version compatibility, resource adaptation, missing permissions, degradation strategies, and exception fallback.

2.3 High Requirements for Permissions and Privacy Compliance

Mobile development often involves sensitive capabilities such as device information, location, storage, network requests, and user authorization.

If AI-generated code lacks permission checks, least-privilege principles, or privacy notices, it may introduce compliance risks. Therefore, evaluation should not only assess whether a feature can be implemented, but also whether it is secure, standardized, and auditable.

2.4 Complex Build and Dependency Management

Mobile projects often rely on build scripts, resource packaging, dependency versions, signing configurations, and environment variables.

If AI coding tools cannot understand the project build structure, the generated results may fail to compile or may break the existing build process. Evaluation should cover tasks such as build repair, dependency upgrades, script generation, and error recovery.

2.5 Obvious Performance and Power Consumption Constraints

Mobile development requires not only functional correctness but also startup speed, memory usage, network request efficiency, rendering performance, and power consumption to meet business expectations.

If AI-generated code introduces unnecessary polling, repeated rendering, invalid network requests, or high-cost computations, it may negatively affect user experience. Therefore, evaluation should also focus on performance optimization suggestions and potential resource risks.

3. Core Metrics for an AI Coding Capability Baseline

A reliable mobile development AI coding evaluation baseline should not rely on a single success-rate metric. It should cover correctness, completeness, cost, stability, and security across multiple dimensions.

3.1 Task Pass Rate

Task pass rate reflects whether the AI-generated code meets expected functional requirements. Common evaluation criteria include:

• Whether unit tests pass;
• Whether API calls conform to contracts;
• Whether page behavior meets expectations;
• Whether the build succeeds;
• Whether the fixed defect no longer reproduces.

Task pass rate should be associated with task difficulty. Simple code completion and complex refactoring should not use the same weight. Otherwise, real capability differences may be hidden.

3.2 Task Completion Rate

Task completion rate focuses on whether the AI completes the full task loop rather than only a local code fragment. For example:

• Whether a complete file structure is generated;
• Whether necessary exception handling is added;
• Whether related callers are modified;
• Whether tests and comments are added;
• Whether static checks pass;
• Whether existing functionality is broken.

A high completion rate indicates that AI is closer to engineering-level delivery capability, rather than merely producing a few lines of code.

3.3 Reasoning Levels and Complexity Tiers

Different models or different configurations may perform very differently on complex tasks. It is recommended to label tasks with reasoning levels, such as:

• L1: Basic code completion, comment generation, and simple function implementation;
• L2: Module-level API implementation, unit test generation, and local refactoring;
• L3: Cross-file defect fixing, build error handling, and permission compliance modification;
• L4: Complex business logic refactoring, performance optimization, and architecture-level code migration.

Reasoning-level labeling helps teams more accurately determine at which complexity level a model becomes practically useful.

3.4 Call Cost and Total Cost

Average call cost is an important metric, but it should not be the only decision criterion. A low-cost model that repeatedly fails to generate usable results may actually increase manual review and rework costs.

Total cost should consider:

• Cost per call;
• Average number of retries;
• Manual fix time;
• Rework cost caused by failed tests;
• Code review cost;
• Potential defect cost after release.

A more reasonable evaluation formula is:

Total cost = call cost + manual correction cost + failed test cost + maintenance risk cost

3.5 Stability and Repeatability

Evaluation results should be reproducible as much as possible. For the same task, multiple runs are recommended to observe result stability. Stability metrics include:

• Consistency of multiple generations for the same task;
• Frequency of generating non-compilable code;
• Frequency of introducing unrelated changes;
• Obvious fluctuations across different contexts;
• Excessive sensitivity to prompt changes.

A model with insufficient stability may perform well in a single run but is difficult to integrate into long-term development workflows.

3.6 Security and Compliance

AI-generated code in mobile development must undergo security validation. Key checks include:

• Whether plaintext sensitive information is introduced;
• Whether permission checks are bypassed;
• Whether unsafe data storage methods are used;
• Whether high-risk network requests are generated;
• Whether dependency license requirements are violated;
• Whether untrusted external resource loading logic is included.

Security metrics should be part of the evaluation loop, not addressed only after release.

4. Building a Task Evaluation Set for Mobile Development

The key to building an evaluation set is not quantity, but coverage of high-frequency and high-risk tasks in real development. It is recommended to start with the following categories.

4.1 Feature Implementation Tasks

These include page initialization, list rendering, network requests, local caching, state management, button events, route navigation, and similar tasks. This category evaluates basic coding capability.

Example task:

Add four display states to a user detail page: loading, loading failed, empty state, and normal state. Ensure that page restoration does not trigger duplicate requests.

4.2 Defect Fixing Tasks

These include null pointer issues, state loss, concurrency conflicts, duplicate requests, unreleased resources, build errors, and inconsistent API fields. This category assesses the model’s ability to understand error context.

Example task:

After a page returns from the background, the list is repeatedly loaded. Identify possible causes and provide a minimal-change fix.

4.3 Refactoring and Optimization Tasks

These include splitting large files, extracting common components, simplifying conditional logic, reducing duplicate code, and improving readability. This category focuses on code quality rather than simple feature implementation.

4.4 Test Generation Tasks

These include unit tests, boundary tests, exception path tests, API contract tests, and page interaction tests. Test generation is one of the most practical scenarios for AI-assisted development and should be evaluated thoroughly.

4.5 Build and Environment Tasks

These include build failure analysis, dependency conflict localization, resource file path repair, environment variable configuration, and packaging script optimization.

4.6 Security and Compliance Tasks

These include permission request standardization, privacy notices, sensitive data masking, secure storage, log masking, and API authentication checks.

5. Evaluation Workflow Design: Making Results Comparable, Regression-Capable, and Explainable

To make evaluation results truly useful for engineering decisions, a standardized workflow is required. The following steps are recommended.

Step 1: Fix Repository Snapshots

Each task should be executed based on a fixed code version to avoid result differences caused by context changes. Repository snapshots, build artifacts, or minimal project samples can be used to ensure reproducibility.

Step 2: Build Unified Prompt Templates

Different models have different sensitivity to prompts. Therefore, unified prompt templates should be used, while allowing reasonable context to be passed in. The template should include:

• Task objective;
• Related file paths;
• Required code snippets;
• Constraint descriptions;
• Expected output format;
• Testing requirements;
• Security requirements.

Step 3: Automatically Generate Candidate Patches

After the AI model outputs a code patch, the patch should go through parsing, application, formatting, static checking, and compilation validation. It is recommended not to directly overwrite source files. Instead, generate patches for automatic validation.

Step 4: Automatically Execute Tests and Quality Gates

Common quality gates include:

Compilation succeeds
Static analysis passes
Unit tests pass
Key-path tests pass
No new high-risk warnings are introduced
No unrelated files are modified
Sensitive information scan passes

Step 5: Manually Review High-Risk Tasks

For high-risk tasks involving security, permissions, architecture, or performance, manual review should be retained. Automatic evaluation is used for initial screening, while manual review confirms release readiness.

Step 6: Record Metrics and Maintain Continuous Regression

Evaluation results should be consolidated into a metrics dashboard. At minimum, it should include:

• Task pass rate;
• Task completion rate;
• Average number of retries;
• Average call cost;
• Manual fix duration;
• Number of security warnings;
• Compilation failure rate;
• Test coverage changes;
• Rationality of change scope.

6. Balancing Cost and Effectiveness: Avoid Choosing Only the Most Expensive or Cheapest Option

In practical technology selection, a common mistake is to treat “strongest capability” or “lowest price” as the only criterion. A more reasonable approach is to match model capabilities according to task tiers.

6.1 Simple Tasks: Prioritize Cost and Response Efficiency

For tasks such as comment generation, variable naming, simple function completion, formatting, and code explanation, lower-cost and faster model configurations can be used to reduce resource consumption from large-scale calls.

6.2 Medium-Complexity Tasks: Focus on Completion Rate and Stability

For tasks such as unit test generation, API adaptation, local refactoring, and defect fixing, completion rate, pass rate, and stability should be prioritized. This helps avoid hidden costs caused by repeated failures.

6.3 High-Complexity Tasks: Emphasize Reasoning and Context Capability

For tasks such as cross-file fixing, build error analysis, performance optimization, and complex state management, models with stronger reasoning capability or higher reasoning-level configurations should be used, combined with manual review.

6.4 Establish a Task Routing Strategy

A task classifier or rule engine can route requests to different models or different reasoning levels. Example logic is as follows:

def choose_model(task_type, risk_level):
    if task_type in ["comment", "format", "rename"]:
        return "low_cost_model"
    if task_type in ["unit_test", "api_adapter", "local_fix"]:
        return "balanced_model"
    if risk_level == "high" or task_type in ["cross_file_fix", "performance_optimization"]:
        return "high_reasoning_model"
    return "balanced_model"

This routing mechanism helps control cost while maintaining quality.

7. Engineering Integration: Embedding AI Coding Assistance into the Development Pipeline

AI coding assistance can produce stable benefits only when embedded into the development workflow. Engineering integration should focus on the following areas.

7.1 Integrating with Repository Context

AI coding tools should be able to understand task-related files, module structures, API definitions, and call relationships. Techniques such as retrieval augmentation, context summarization, and symbol indexing can improve the consistency between generated code and the existing project.

7.2 Integrating with Testing Platforms

Generated code should automatically enter the testing process, including compilation, unit testing, static checking, compatibility testing, and key-path validation. AI-generated code without automated validation should not directly enter the review stage.

7.3 Integrating with Monitoring Systems

In environments such as Baidu AI Cloud or similar cloud platforms, teams can combine general capabilities such as object storage, logging, and monitoring alerts to build an observability system for AI calls and evaluation results. Key monitoring items include:

• Call success rate;
• Average response time;
• Cost per task;
• Distribution of failure reasons;
• Test pass rate;
• Number of security warnings;
• Manual fix rate.

7.4 Integrating with Permission Systems

AI coding assistance tools usually need access to code context, build logs, or test reports. Permission isolation, least privilege, sensitive information masking, and audit logs should be used to ensure the security of development assets.

7.5 Integrating with Knowledge Bases

Common mobile development standards, component usage patterns, historical defects, and architectural constraints can be accumulated into an internal knowledge base. Combining the knowledge base with AI capabilities can significantly improve the consistency of generated results.

8. Quality and Safety Fallback Mechanisms: Establishing a Closed Loop

AI-generated code cannot replace engineering gates. On the contrary, once AI is involved in development, quality and safety fallback mechanisms should be strengthened.

8.1 Front-Loaded Static Checks

Before code submission, format checks, complexity checks, duplicate code checks, null safety checks, and dependency checks should be executed to detect low-quality issues as early as possible.

8.2 Test Coverage Validation

For AI-generated test code, teams should also check whether coverage is effectively improved, rather than generating a large number of invalid assertions. It is recommended to comprehensively assess line coverage, branch coverage, and key-path coverage.

8.3 Change Scope Limitation

For AI-generated patches, the modification scope should be restricted. If a patch modifies unrelated files, deletes existing logic, or introduces large-scale dependencies, it should be automatically intercepted.

8.4 Sensitive Information Scanning

Code, logs, configuration files, comments, and test data should be scanned for sensitive information to prevent leakage of keys, tokens, user data, or internal addresses.

8.5 Manual Review for High-Risk Changes

Code involving high-risk modules such as permissions, payments, login, data storage, network communication, and performance optimization must retain a manual review mechanism.

9. Common Mistakes and Avoidance Recommendations

Mistake 1: Looking Only at Leaderboard Rankings

Leaderboard rankings can provide reference, but internal task distributions often differ from public benchmarks. Teams should build evaluation sets that match their own business scenarios.

Mistake 2: Looking Only at Average Cost

Low average cost does not necessarily mean low total cost. If failure rate, retry rate, and manual fix cost are high, the overall return may decrease.

Mistake 3: Ignoring Context Engineering

AI coding capability depends heavily on context quality. Without sufficient file structure, API definitions, historical constraints, and business rules, generated results may deviate from real requirements.

Mistake 4: Ignoring Version Changes

After model versions, reasoning levels, prompt strategies, or evaluation sets change, results may fluctuate. Therefore, continuous regression and version comparison are necessary.

Mistake 5: Treating AI-Generated Code as the Final Deliverable

AI-generated code should be treated as an engineering aid, not as the final deliverable. It must undergo compilation, testing, review, security scanning, and release validation.

10. Conclusion: Building a Mobile Development AI Coding Evaluation System Suitable for Your Team

The essence of evaluating AI coding assistance for mobile development is not to find an “all-capable model,” but to establish a selection mechanism that matches business tasks, engineering standards, quality gates, and cost control.

Teams should start from real tasks and build an evaluation system around metrics such as pass rate, completion rate, reasoning level, call cost, stability, security, and maintainability. Evaluation results should also be continuously integrated into the development pipeline.

When AI coding assistance is embedded into code generation, test generation, defect fixing, code review, and build optimization, its value becomes more than a one-time efficiency improvement. It becomes a long-term enhancement of the engineering system’s capability and code quality.