How to Evaluate Multimodal LLMs and Build a Low-Cost Coding Agent

    As automation scripts, code review, and internal development tools become more common, developers face a practical question: how should they choose a large model, and how can they keep invocation costs reasonable while maintaining output quality? This article proposes a repeatable engineering evaluation method for general multimodal large models, covering code understanding, task decomposition, long-context stability, executable outputs, and long-term cost. It then explains how to design a low-cost, sustainable coding agent with task layering, context management, patch-based output, automated validation, and budget controls.

    How to Evaluate Multimodal LLMs and Build a Low-Cost Coding Agent

    As automation scripts, code review, and internal development tools become increasingly common, many developers face a practical problem: how should they choose a large model, and how can they keep invocation costs within a reasonable range while ensuring effective results?

    Simply comparing model names or parameter sizes rarely leads to valuable conclusions. For a coding agent, the factors that truly affect the user experience usually include five aspects: code understanding, complex task decomposition, long-context stability, the executability of generated output, and the overall cost of long-term operation.

    Therefore, instead of debating whether a particular model is the “strongest,” it is more useful to establish a repeatable evaluation method. This article uses general multimodal large models as the subject and discusses how to complete testing from an actual engineering perspective, then build a low-cost, sustainable Coding Agent.

    1. Why the Model Invocation Approach Needs to Be Reassessed

    In the past, many teams focused mainly on answer quality when choosing a model, such as whether it could generate a piece of code that looked correct. In real projects, however, single-answer quality is only the first step. Whether a model is suitable for production also depends on the following dimensions.

    The first is stability. For the same task, output quality should not fluctuate excessively across different times or context lengths. If a model can occasionally complete a complex refactor but frequently misses edge cases, it is better suited as a personal assistant tool rather than being embedded directly into an automated workflow.

    The second is context utilization. Codebase analysis is often not simple question answering; it requires understanding directory structures, configuration files, core modules, test cases, and dependencies at the same time. Even if a model supports a long context, that does not mean it can effectively use all the information. What really matters is whether it can extract the parts relevant to the current task from a large amount of content.

    The third is cost controllability. During development, teams often need to repeatedly debug prompts, run tests, and correct outputs. If every experiment consumes a large budget, it becomes difficult for the team to validate the system thoroughly. Therefore, input and output costs, caching mechanisms, request frequency, and failure retry strategies all need to be included in the overall budget.

    2. How to Establish an Executable Evaluation Standard

    It is recommended to divide model evaluation into four types of tasks rather than using only a few simple question-answer samples.

    The first type is code reading tasks. Prepare a small project with a realistic business structure and ask the model to explain module responsibilities, call relationships, and potential risks. The evaluation focus is not whether the text is fluent, but whether the model can accurately identify key entry points, data flows, and exception-handling paths.

    The second type is local modification tasks. For example, ask the model to add interface validation, supplement logging, fix edge cases, or write tests for an existing function. Such tasks can directly verify whether the generated code is executable.

    The third type is cross-file refactoring tasks. Ask the model to modify multiple files according to requirements and explain the reasons for the changes, the impact scope, and the rollback approach. This step can test the model’s task decomposition ability and expose whether it misses implicit dependencies.

    The fourth type is multimodal tasks. Provide interface screenshots, flowcharts, log screenshots, or architecture diagrams, and ask the model to extract the information and provide processing suggestions. Note that visual understanding results must be verified manually or programmatically; image recognition results should not be treated directly as facts.

    Evaluation results should preferably be recorded quantitatively, including:

    • Task success rate;
    • First-generation executable rate;
    • Average number of revision rounds;
    • Error type distribution;
    • Input and output length per task;
    • Average response time;
    • Failure retry ratio;
    • Actual cost per valid task.

    Conclusions obtained this way are more reliable than “the answer felt good.”

    3. How Invocation Costs Should Be Calculated

    Model invocation costs can usually be abstracted into the following formula:

    1. Single cost =
    2. input token count × input unit price
    3. + output token count × output unit price
    4. + cache read token count × cache unit price
    5. + failed retry cost

    If one code analysis requires 500,000 input tokens and 20,000 output tokens, the variables can be expressed as:

    1. Cost = 0.5 × input unit price + 0.02 × output unit price

    The unit prices here must be based on the billing documentation of the platform actually used. They should not be estimated solely from promotional pages or screenshots from others. For long-running agents, retries, context compression, and cache hit rates should also be included in the calculation.

    For example, if a task fails on its first execution and is then retried twice, the actual cost may approach three times the theoretical cost. If the system does not limit the maximum number of retries, some abnormal requests may even form a loop of calls.

    Reducing costs does not mean simply choosing the model with the lowest unit price. A more effective approach is to optimize the task workflow:

    1. First use a lightweight model for intent recognition and file filtering;
    2. Invoke a stronger model only for complex reasoning or cross-file modifications;
    3. Cache project descriptions, interface specifications, and directory information that do not change;
    4. Replace repeated transmission of large historical conversations with summaries;
    5. Limit output length per request and ask the model to return only necessary content;
    6. Split a large task into multiple verifiable smaller tasks.

    A practical coding agent should not merely “forward questions to the model.” It should include modules for task management, code retrieval, model invocation, result validation, and human confirmation.

    A typical workflow is as follows:

    1. User requirement
    2. ↓
    3. Task parsing and risk assessment
    4. ↓
    5. Project file filtering
    6. ↓
    7. Retrieval of relevant code and documentation
    8. ↓
    9. Structured task context generation
    10. ↓
    11. Model analysis and code modification suggestions
    12. ↓
    13. Patch format output
    14. ↓
    15. Automated test execution
    16. ↓
    17. Failure feedback and limited retries
    18. ↓
    19. Human confirmation before writing to the codebase

    Among these steps, file filtering is critical. Do not pass the entire project directory with every request. Instead, first select the most relevant files based on task keywords, file dependencies, and historical modification records.

    For large projects, a simple code index can be built to store file paths, function names, class names, comments, and summaries. After receiving a request, retrieve candidate files first, then read the necessary code snippets. This reduces input length and lowers the chance of the model being distracted by irrelevant information.

    5. Why the Model Must Output Patches Instead of Directly Editing Files

    Although letting the model write directly to project files is simple to operate, it carries high risk. A safer approach is to require it to generate structured modification results, such as unified diff patches, a target file list, modification reasons, and test suggestions.

    An example format is as follows:

    1. {
    2. "files": [
    3. {
    4. "path": "src/example.py",
    5. "change": "add_validation",
    6. "reason": "Prevent exceptions caused by empty parameters",
    7. "patch": "..."
    8. }
    9. ],
    10. "tests": [
    11. "Verify empty parameters",
    12. "Verify normal parameters",
    13. "Verify abnormal input"
    14. ]
    15. }

    This approach has three benefits.

    First, the system can validate paths before applying modifications, preventing the model from accidentally changing unrelated files.

    Second, it can check whether the patch can be applied normally and run format checks, static analysis, and unit tests before writing.

    Third, all changes have a clear audit record, making rollback and issue diagnosis easier.

    6. Context Management Is More Important Than Simply Increasing Length

    Many people believe that the longer the context, the better, but this is not necessarily true. Too much irrelevant content can reduce the model’s attention to key information and may even cause it to ignore truly important constraints.

    A layered context is recommended:

    The first layer is system rules, including code style, security requirements, and output format.

    The second layer is project-level information, including directory structure, run methods, dependency constraints, and test commands.

    The third layer is task-level information, including user requirements, target files, error logs, and relevant code.

    The fourth layer is historical information, retaining only modification records directly related to the current task.

    For content that has already appeared repeatedly, use caching or summarization mechanisms as much as possible instead of transmitting it in full every time. For very long files, first ask the model to analyze the structure, then process them section by section by function or module.

    7. How to Handle Common Issues in Model-Generated Code

    Common issues in model-generated code include incorrect interface calls, inconsistent variable naming, missing exception handling, insufficient test coverage, and damage to existing logic.

    Multi-level validation can be established for this.

    The first layer is format validation, checking syntax, indentation, encoding, and basic structure.

    The second layer is static checking, identifying undefined variables, dangerous calls, type mismatches, and potential resource leaks.

    The third layer is automated testing, covering at least normal input, empty input, abnormal input, and boundary conditions.

    The fourth layer is behavior comparison, confirming that key functions have not changed unexpectedly before and after modification.

    The fifth layer is human review, especially for high-risk code such as database operations, permission logic, payment flows, user privacy, and production configuration.

    The goal of the agent is not to completely replace developers, but to automate repetitive work and clearly expose the parts that require judgment.

    8. Recommendations for Cost Control and Stable Operation

    If the agent is to run for a long time, the following mechanisms are recommended:

    • Set maximum input and output lengths for each request;
    • Set daily or monthly budget limits;
    • Set a limited number of retries for failed requests;
    • Deduplicate identical tasks;
    • Record the token count and latency of each invocation;
    • Require human confirmation for high-risk operations;
    • Store model invocation logs together with test results;
    • Regularly analyze frequent failure tasks and optimize prompts and workflows.

    In addition, do not focus only on unit price. A lower-priced solution that requires multiple retries may end up costing more than a higher-priced solution with a higher first-pass success rate. A more reasonable metric is “average cost per valid task,” not “price per million tokens.”

    9. How to Finally Judge Whether the Experience Is Good

    Whether a model is worth using for a coding agent cannot be judged only by promotional capability descriptions or a few conversation experiences. It is recommended to run it continuously for a period of time, covering tasks such as code reading, feature development, troubleshooting, test generation, and documentation organization.

    Conclusions can ultimately be drawn from the following questions:

    • Can it stably understand project context?
    • Is the first-time executable rate of generated code high enough?
    • Can it correctly correct errors based on logs after encountering them?
    • Can it follow the established output format?
    • Does it forget key constraints in long conversations?
    • Is the actual cost per valid task acceptable?
    • When anomalies occur, is the system easy to roll back and trace?

    If these questions receive relatively positive answers, then the model invocation approach truly has engineering value.

    Overall, the core of using large models at low cost is not finding an absolutely cheap API, but designing reasonable task layering, context management, caching strategies, and automated validation workflows. The model is responsible for understanding and generation, while the engineering system is responsible for constraints, validation, and fallback. Only by combining the two can you build a stable, maintainable coding agent with controllable costs.