Reproducing Complex Formulas with LLMs: Practices for Programming Reliability

    The evaluation standard for complex technical tasks is shifting from single-point code generation to a verifiable engineering closed-loop. This article explores how to reliably reproduce complex formulas using large language models, covering task decomposition, explicit planning, key control points in the document-to-formula reproduction chain, missing function adaptation, stability strategies, evaluation methods, and cloud-based engineering implementation.

    The evaluation standard for complex technical tasks is shifting from “whether a piece of code can be generated” to “whether a verifiable engineering closed-loop can be completed.” Taking the reproduction of custom metrics in technical documents as an example, the ideal result is not merely understanding terminology and writing formulas; it also includes correctly extracting definitions, mapping to the target formula environment, handling missing functions, recording execution logs, and passing manual sampling and regression testing. This closed-loop requires much more from the model than single-point Q&A, involving planning, tool calling, code generation, error localization, and stability control.

    1. Capability Threshold for Complex Tasks: From Single-Point Generation to Closed-Loop Delivery

    Complex formula reproduction typically involves multiple stages: document parsing, terminology recognition, formula extraction, semantic alignment, target language rewriting, compatibility adaptation, and result verification. Any deviation in one stage will be amplified subsequently. For example, a “median moving average” in a document might depend on window length, price sequence, odd/even window handling rules, and boundary conditions. If the model merely restates the concept without converting these constraints into executable logic, the generated formula will be difficult to deploy.

    Therefore, evaluating a model should not solely focus on whether a single generation is correct, but on whether it can locate problems after failure, propose correction plans, and maintain output consistency. For developers, a more practical approach is to establish a “task decomposition checklist”: first require the model to output a task plan, then execute step by step; preserve intermediate artifacts and logs for each step; and finally verify using fixed samples, boundary samples, and regression samples. This helps distinguish between “accidentally correct” and “stably deliverable.”

    1. Explicit Planning and Interleaved Execution: Reducing Blindness in Complex Tasks

    When facing complex tasks, if the model directly generates the final answer, it tends to lack global planning, leading to variable omissions, step skipping, or tool calling failures. A more reliable approach is to adopt explicit planning and interleaved execution: first decompose sub-tasks, then alternate between reasoning and tool calling. For instance, first call document parsing capabilities to extract definitions, then check if the extraction is complete; if missing parameter descriptions are found, return to the document for supplementary retrieval; subsequently generate a formula draft and validate it with sample data; if validation fails, correct it based on the error type.

    This “reason-call-verify-correct” loop is closer to real-world engineering practice than one-shot generation. The key is not to make the model “think more,” but to ensure each step has a checkable intermediate state. In engineering, the model can be required to output structured plans, tool calling parameters, execution logs, and validation conclusions, with assertions set for critical steps. This way, even if the final result is imperfect, it is possible to quickly pinpoint whether it is a document extraction error, a semantic mapping error, or a target environment compatibility issue.

    1. From Document to Formula: Key Control Points in the Reproduction Chain

    A reusable reproduction chain can be broken down into five steps.

    First, definition extraction. Require the model to extract metric names, input variables, window parameters, calculation rules, boundary conditions, and exception handling from the document. The output should be as structured as possible, avoiding natural language summaries only.

    Second, semantic alignment. Map document terminology to variables and functions in the target formula environment. Special attention must be paid to terms with the same name but different meanings, default parameters, and unit differences. If the document description is ambiguous, the model should be required to list uncertainties rather than making assumptions.

    Third, formula rewriting. Convert mathematical expressions into syntax supported by the target environment. This stage requires checking the availability of loops, conditionals, sorting, and aggregation capabilities. If the target environment does not support certain higher-order functions, a compatible implementation should be generated.

    Fourth, sample validation. Prepare normal samples, boundary samples, and exception samples to compare the model’s output with manual calculations or reference implementations. Floating-point calculations, null values, and insufficient windows must be covered separately.

    Fifth, logging and replay. Save the input document version, extraction results, formula drafts, correction records, and validation results. Logging is not a mere formality but the foundation for subsequent debugging and regression testing.

    1. Missing Function Adaptation: A Median Moving Average Example

    In some formula description environments, built-in functions are incomplete; for example, a median function might not be directly provided. In such cases, one cannot assume the function exists but should generate a compatible implementation. The general pseudocode is as follows:

    1. function CustomMedianMovingAverage(PriceSequence P, Window N):
    2. // Handle invalid parameters
    3. if N <= 0: return empty sequence
    4. result = []
    5. for i from N-1 to length(P)-1:
    6. window = P[i-N+1 ... i]
    7. sortedWindow = sort(window)
    8. if N % 2 == 1:
    9. med = sortedWindow[(N-1)/2]
    10. else:
    11. med = (sortedWindow[N/2 - 1] + sortedWindow[N/2]) / 2
    12. result.append(med)
    13. return result

    If the target environment does not even support a sorting function, it is necessary to further degrade to an “element-by-element comparison statistics” implementation: for each candidate value within the window, count the number of elements strictly less than it and equal to it, then determine the median position based on the window length. In engineering, priority should be given to primitives already supported by the target environment, and the compatibility layer should be separated from the business formula to avoid future maintenance difficulties.

    1. Stability and Maintainability: Logging, Assertions, and Versioning

    The most common risk in complex tasks is not “complete inability,” but “mostly correct with local errors that are hard to detect.” Therefore, stability must be treated as a first-class goal. It is recommended to achieve at least three things: first, write all tool calls and intermediate results into logs to retain a replayable chain; second, set assertions for critical intermediate results, such as window length must be greater than zero, output sequence length must match the input, and null handling must be explicitly declared; third, version documents, formulas, and dependency environments to ensure regression results are comparable.

    At the organizational collaboration level, “model generation” should be treated as a draft rather than the final delivery. Manual review should focus on semantic ambiguity, boundary conditions, and target environment compatibility. Through a fixed sample set and an error classification table, one can continuously observe the model’s failure patterns across different task types and optimize prompts, tool constraints, and validation processes accordingly.

    1. Evaluation Methods: Real Tasks Beyond Benchmarks

    General benchmarks can reflect some capabilities but cannot replace real-task evaluation. For complex formula reproduction, it is recommended to establish a small but high-value internal evaluation set covering document extraction, formula rewriting, missing function adaptation, exception handling, and log integrity. Metrics can include first-pass rate, correction rounds, manual intervention duration, regression failure rate, and explainability score. It is especially important to record “whether it can self-localize after failure,” as this directly determines whether developers are willing to hand the task over to the model for further progression.

    1. Cloud-Based Engineering Implementation: Turning Capabilities into Pipelines

    When reproduction tasks move from individual experiments to team collaboration, engineering support is required. On Baidu AI Cloud, general cloud product categories can be combined to form a pipeline: use object storage to save original documents, extraction results, and versioned artifacts; use log services to record tool call chains and error contexts; use monitoring and alerting to observe task failure rates, duration, and manual intervention counts; and use container platforms to host reproducible execution environments. The general cloud product categories provided by Baidu AI Cloud can serve as the infrastructure for such processes, but the specific solution should still be designed around task verifiability and data security boundaries.

    1. Boundaries and Conclusion

    It is necessary to objectively view the capability boundaries of current models. They can significantly shorten preliminary exploration time in document understanding, code draft generation, and compatibility rewriting, but in highly professional, strongly constrained, and strongly verified tasks, semantic deviations, boundary omissions, and environment incompatibilities may still occur. A reasonable strategy is “models accelerate exploration, engineering ensures validation, humans are responsible for final judgment.” By breaking complex tasks into checkable steps, supplementing missing capabilities with compatible implementations, and turning every execution into a replayable log, one-time generation capabilities can be transformed into stable productivity.