Hardcore Coding and Mathematical Tasks: Layered LLM Selection and Implementation

    The core of Large Language Model (LLM) selection lies not in comparing the raw strength of individual models, but in dispatching tasks to different model tiers based on fault tolerance, context size, per-task cost, and verification overhead. This article outlines a systematic approach to LLM selection, covering evaluation dimensions, scenario matrices, output verification methods, reusable prompt templates, engineering implementation, and risk management for software development and mathematical reasoning tasks.

    The core of model selection is not comparing the raw strength of individual models, but dispatching tasks to different model tiers based on fault tolerance, context size, per-task cost, and verification cost. We will start by defining evaluation dimensions, followed by a scenario matrix, verification methods, prompt templates, and implementation steps.

    1. Define Evaluation Dimensions Before Model Tiers

    The industry often uses “heavy,” “regular,” and “lightweight” to describe models, but these terms lack unified standards. Different vendors have varying naming conventions, pricing, and capability boundaries. Making decisions solely based on labels is insufficient. Teams need to define each tier using quantifiable metrics.

    It is recommended to track at least the following seven dimensions:

    1. Task Pass Rate: The proportion of tasks where the model’s initial output passes all tests on a fixed benchmark set. This is the most direct quality metric.
    2. Effective Context Length: The amount of code a model can stably reference and maintain consistency across during cross-file tasks. This value must be empirically tested and cannot be directly adopted from the vendor’s nominal maximum window.
    3. Reasoning Budget: The allowed thinking time or token limit per request. A higher budget allows for more thorough complex reasoning, but increases time and cost.
    4. Tool Calling Capability: Whether the model can correctly invoke compilers, unit testing frameworks, code retrieval, and static analysis tools, and modify its output based on the returned results.
    5. Per-Task Cost: The model invocation fee for completing a task, plus the labor cost equivalent for manual review and rework. Looking only at the token unit price often leads to incorrect conclusions.
    6. Latency: Time to first token and complete output time. Interactive development is sensitive to the first token, while batch processing tasks prioritize throughput.
    7. Applicable Boundaries: Which programming languages, frameworks, and mathematical domains have been verified, and which scenarios have not.

    Teams can use these to define tiers. For example, a model might be classified as “heavy tier” if it “achieves a predefined pass rate threshold on the internal refactoring test set, with acceptable per-task costs.” Such definitions are only valid for the specific team but can be reused for re-evaluation when model versions are updated.

    Establish an Internal Benchmark Set

    The benchmark set is the basis for selection. It is recommended to extract 30 to 50 tasks from real repositories, covering three types of scenarios: long-range refactoring, protocol or interface migration, and daily development. Each task should have clear inputs, acceptance tests, and expected outputs.

    During evaluation, the following information should be recorded:

    • Model name, version number, release date
    • Temperature parameter, reasoning budget, system prompt version
    • Pass status for each task, number of manual fixes, token consumption, time taken

    The benchmark set must be re-run after any model version update. Version changes may alter output styles or even invalidate existing prompts.

    2. Selection Matrix for Typical Scenarios

    Different tasks have vastly different fault tolerance requirements. An error in a data cleaning script can be re-run, but an error in a payment interface protocol change could lead to financial losses online. When selecting a model, look at the cost of failure first, then at model capabilities.

    Scenario Fault Tolerance Context Size Cost & Latency Constraints Recommended Tier Necessary Verification Human Review Nodes
    Long-range cross-module refactoring Very Low Large, involves multiple modules Cost acceptable, latency relaxed Heavy Full regression testing, dependency checks Solution review, code review before merge
    Underlying protocol migration Very Low Medium to large, includes interface definitions and callers Same as above Heavy Compatibility testing, canary verification Protocol change review, canary rollout
    Mathematical derivation and formula verification Very Low Medium Latency relaxed Heavy Numerical verification, symbolic computation tool review Confirmation by domain experts
    Daily CRUD and routine bug fixes Medium Small to medium Requires fast response Regular Unit testing, code style checks Code review before merge
    Technical proposals and architecture docs Medium Medium Requires fast response Regular Review meetings, fact-checking Architecture review
    Batch data cleaning and test case generation Low, can be re-run Small Cost priority, throughput priority Lightweight Sampling checks, regex and rule validation Spot checking
    Prototyping and trial-and-error Low Small Cost priority, fast iteration Lightweight Functional smoke testing Confirmation before prototype launch

    The matrix is only a starting point. If a specific scenario has a high historical failure rate, you should elevate its tier or add human review nodes rather than blindly following the table.

    3. How to Determine if Model Output is Trustworthy

    Models often append explanations like “self-checked” or “edge cases considered.” These texts are merely part of the output and cannot replace verification. To determine if the output is trustworthy, check if it passes independent inspections.

    For coding tasks, it is recommended to require the model to output the following simultaneously:

    • Change List: List every modified file and function, along with the reason for modification.
    • Assumptions: List the assumptions the model made regarding inputs, environment, and dependency versions.
    • Uncovered Scenarios: Explicitly state which edge cases were not handled.
    • Test Cases: Provide inputs, expected outputs, and covered boundaries.
    • Rollback Steps: Explain how to revert to the pre-change state if issues arise.

    The value of these items lies in the fact that human reviewers or automated tools can check them line by line. If the test case itself is written incorrectly, the failure might be in the test rather than the code; therefore, test cases also require review.

    For mathematical derivation tasks, require the model to write out the derivation process step-by-step and verify key conclusions using independent methods. Common methods include numerical substitution, verification via symbolic computation tools, and testing boundaries with special values. The final conclusion provided by the model cannot be directly used as the sole basis for production decisions.

    4. Reusable Prompt Template

    The following template applies to refactoring and migration tasks. Teams can solidify it into an internal prompt library and adjust constraints based on the scenario.

    1. Task Objective: [One sentence describing the change to be made]
    2. Background and Constraints:
    3. - Language and framework version: [e.g., Java 17, Spring Boot 3.2]
    4. - Dependency versions: [List key dependencies and versions]
    5. - Compatibility requirements: [e.g., external interface fields cannot be deleted, only new optional fields are allowed]
    6. - Scope of modification not allowed: [e.g., database schema, authentication module]
    7. Input Materials:
    8. [Paste relevant code, interface definitions, or protocol files]
    9. Output Requirements:
    10. 1. First list assumptions and boundary conditions, mark uncertainties as "To be confirmed"
    11. 2. Output complete code without omitting key logic, and explain dependencies
    12. 3. Provide executable test cases, including inputs, expected outputs, and covered boundaries
    13. 4. List modules potentially affected by this change and compatibility risks
    14. 5. Clearly state which parts you cannot verify

    The “Scope of modification not allowed” in the template is crucial. Models tend to conveniently modify code they perceive as problematic, and these modifications may exceed the task scope. Clear boundaries reduce the risk of unintended changes.

    5. Execution Essentials: A Protocol Migration Example

    Suppose a microservice is migrating from a RESTful interface to gRPC. Merely having the model generate server and client stub code is insufficient. The following parameters need to be explicitly defined in the prompts and reviews.

    • Proto version and field numbers: Deleted field numbers must not be reused, and new fields must only use new numbers. Otherwise, old clients will parse incorrect data.
    • Timeout settings: The client timeout needs to be smaller than the upstream caller’s timeout to prevent cascading wait times along the call chain.
    • Retry strategy: Enable automatic retries only for idempotent interfaces. Retrying non-idempotent interfaces (e.g., payments) may cause duplicate executions.
    • Interceptor testing: Write separate test cases for timeout, retry, and authentication interceptors to verify their behavior on exception paths.
    • Canary ratio: First release a small percentage of traffic, observe error rates and latency, and then gradually scale up.
    • Rollback conditions: Predefine thresholds, such as the error rate exceeding a certain multiple of the baseline, or P99 latency exceeding the baseline. If any condition is met, roll back immediately.

    These parameters should be written into the task’s acceptance criteria. The code generated by the model can only enter the canary phase after passing these checks.

    6. Engineering Implementation: Routing, Logging, and Permissions

    Integrating model capabilities into the development process requires proper routing, isolation, and auditing at the infrastructure level. The following solution uses general cloud-native components, and the specific implementation can be adapted to the team’s existing tech stack.

    Model Routing: Deploy a model proxy gateway on a container platform. Requests carry task tags, such as task_type: hardcore_refactor or task_type: batch_script. The gateway forwards the request to the corresponding tier’s model endpoint based on the tag. Upper-layer applications do not need to care about the underlying model; switching models only requires modifying routing rules.

    Sandbox Execution: Code generated by the model should run in an isolated sandbox. The sandbox should not hold production environment credentials, cannot directly access production databases, and network egress must be restricted. This is a fundamental requirement of the principle of least privilege.

    Logging and Monitoring: Runtime logs should be desensitized before collection; do not record keys, tokens, user PII, and internal credentials. Monitoring rules should be based on test results, error rates, and latency, rather than the model’s self-reported success status. When an alert is triggered, the responsible person determines whether to roll back. Only pre-approved release processes are allowed to roll back automatically, and the rollback operation itself must be logged.

    Batch Processing Pipeline: For high-frequency script tasks, you can use object storage to receive input data, trigger function compute via events, and invoke lightweight models in temporary containers to generate and execute processing scripts. Processing results and intermediate logs are written to object storage. Bucket access permissions should follow the principle of least privilege, with retention periods set to automatically clean up expired data.

    7. Risk and Compliance Considerations

    When using models to generate code and documentation, teams need to be aware of the following risks.

    • License Risk: Generated code may contain snippets similar to open-source code in the training data. Use license scanning tools to check before release, and ensure that introduced third-party code complies with project license requirements.
    • Privacy Risk: Customer data, undisclosed keys, and internal credentials should not be placed in prompts. Teams should choose appropriate model deployment methods based on the company’s data classification policies.
    • Dependency Risk: Models may reference non-existent package names or recommend outdated versions. Dependencies must undergo actual installation checks via package managers and security scans.
    • Security Risk: Generated code needs static analysis and security review, especially parts involving authentication, encryption, and input validation.
    • Human Review: Changes involving funds, permissions, data deletion, and protocol compatibility must undergo manual approval before going live.
    • Mathematical Conclusions: Derivation results provided by models require independent verification and should not be used as the sole basis.

    8. Implementation Steps

    Teams can advance selection work in the following order:

    1. Extract tasks from real repositories to establish an internal benchmark set and define acceptance tests.
    2. Select two or three candidate models for each scenario, and empirically test the pass rate, cost, and latency on the benchmark set.
    3. Set go-live thresholds based on the scenario’s fault tolerance requirements, such as pass rate, number of manual fixes, and the upper limit for per-task cost.
    4. First use in a small-scale canary, record real-world failure cases, and add them to the benchmark set.
    5. Whenever a model version is updated, re-run the benchmark set to confirm that prompts and routing rules remain effective.

    By following this process, teams can use verifiable data to determine model allocation, rather than relying on vendor marketing or personal experience.