Agent-Driven Quantitative Analysis and Automated Research Report Generation

    Handing quantitative analysis straight to a large language model often produces numbers that look plausible but can't be trusted, reproduced, or audited. This article explains why it's more robust to treat the model as a programmer rather than a calculator, then walks through a reproducible analysis pipeline built on code generation and sandboxed execution. It covers fixing percentile calibers with scripts, replacing blind trust with assertions, generating charts with plotting libraries instead of generative images, and designing fair multi-model evaluations. It also weighs cloud agent environments against self-hosted pipelines and closes with a reusable prompt template plus the engineering practices and risk boundaries needed to take this kind of automation into production.

    1. The Starting Point: Why “Just Let the Model Compute” Isn’t Reliable

    When you hand quantitative data analysis to a large language model, the most common failure isn’t that it “can’t write a conclusion”—it’s that the numbers themselves can’t be trusted. An LLM is fundamentally a probabilistic sequence predictor. It guesses the next token based on its training distribution and the surrounding context, and it doesn’t rigorously follow mathematical rules to perform a real division or sort. When asked “what historical percentile is the current inventory at,” the model often returns a “plausible-looking” approximation—and the further you go into the decimal places, the less trustworthy it becomes.

    This kind of bias is damaging in production in three ways:

    • Conclusion-level bias: a percentile off by a few points can flip a “low inventory, good time to add position” call into “neutral.”
    • Non-reproducibility: ask the same question twice and the numbers may differ, making reports impossible to trace or audit.
    • Unattributable accountability: once a number is wrong, you can’t tell whether the cause is the data source, the definition/caliber, or the model.

    The more robust engineering approach is to not cast the model as the “calculator,” but as the “programmer.” The model understands intent, writes code, orchestrates tools, and organizes the narrative; the actual numbers are computed by an interpreter running on real data. Python has no memory bias and doesn’t “guess” results—it faithfully executes the logic you write.

    2. Comparing Three Delivery Approaches: Text Reasoning, External Calculation, and Agent Execution

    Dimension Pure text answer Single code/calculation plugin Agent mode (code generation + sandboxed execution)
    Source of numbers Guessed from training distribution and context Model fills in parameters, plugin evaluates Script runs on real data
    Intermediate process Invisible Partially visible Fully auditable (code, logs, artifacts)
    Multi-step tasks Easily broken Usually single-step Automatically decomposed into todos and advanced step by step
    Failure recovery None Weak Can auto-rewrite code and retry on errors
    Output form Pure text Result values Integrated data tables, charts, and documents

    The key gain of agent mode isn’t “it can calculate”—it’s that it shifts uncertainty from the “number layer” to the “execution layer,” and errors at the execution layer are observable and fixable. Errors return stack traces, assertions fail loudly, and chart coordinates can be read back and verified. By contrast, errors in pure text output are silent.

    3. What a Reproducible Analysis Pipeline Should Look Like

    A mature agent workflow closely mirrors the execution paradigm of mainstream AI coding assistants: first auto-generate a task list (todo list), then implement each item as runnable code. A typical pipeline looks like this:

    1. Task decomposition: break “generate a quantitative daily report” into six todos—data loading, metric computation, distribution statistics, visualization, conclusion writing, and document export.
    2. Data ingestion: read local or remote table files, prioritize type and missing-value checks, and output a field list and sample size.
    3. Metric computation: compute derived metrics such as mean, percentile, and basis via scripts, persisting intermediate results to structured files.
    4. Visualization: call plotting libraries to generate figures according to the logic in the factor source code.
    5. Report assembly: merge text conclusions and images in order, then export a richly formatted document.
    6. Self-validation: assert on key numbers and spot-check charts, manually or automatically.

    3.1 Percentile Calculation: Fix the Caliber with Scripts

    Different libraries use different default interpolation methods, and an unfixed caliber is a common source of numerical disagreement. The safer approach is to implement it yourself and declare the caliber explicitly:

    1. import csv
    2. def percentile(values, q):
    3. """Linear interpolation percentile; q in [0, 1]"""
    4. data = sorted(v for v in values if v is not None)
    5. if not data:
    6. raise ValueError("Empty sample, cannot compute percentile")
    7. pos = (len(data) - 1) * q
    8. lo = int(pos)
    9. hi = min(lo + 1, len(data) - 1)
    10. return data[lo] + (data[hi] - data[lo]) * (pos - lo)
    11. def rank_percentile(sample, target):
    12. """Percentile of target within the sample (0~1)"""
    13. sample = sorted(sample)
    14. below = sum(1 for v in sample if v <= target)
    15. return below / len(sample)

    3.2 Replace “Believing” with Assertions

    1. cur = rank_percentile(inventory_hist, inventory_today)
    2. basis = rank_percentile(basis_hist, basis_today)
    3. assert 0.0 <= cur <= 1.0 and 0.0 <= basis <= 1.0, "Percentile out of range"
    4. assert abs(percentile(inventory_hist, 0.5) - median_check) < 1e-9, "Median caliber mismatch"
    5. print(f"Inventory percentile: {cur:.2%} | Basis percentile: {basis:.2%}")

    The point of this code is that any caliber error at any step immediately halts the pipeline, instead of carrying a wrong number all the way into the final report.

    4. Why Charts Must Come from Plotting Libraries, Not Generated Images

    There are two paths to producing figures in an agent environment, and they aren’t on the same professional level:

    • Code-based plotting (recommended): every point’s coordinates are computed from concrete data, so the output is reproducible, verifiable, and replayable. Axes carry units, legends carry mappings, color thresholds can be bound to business rules, and output can be raster or vector. A reviewer can trace the code back to the original data.
    • Generative images: essentially pixel-level probabilistic generation. The image “looks like” a line chart but has no real data behind it. It’s acceptable for report covers or illustrative graphics, but for professional charts in finance or industry it introduces unacceptable distortion—and no assertion can ever verify it.

    So the prompt should explicitly specify the plotting method. For example, in the output requirements: “Automatically call plotting libraries to generate figures, and merge the report body with the figures you generate into a single document.” Only when this constraint is written down will the agent reliably follow the “write code → generate chart → lay out → export” path, rather than casually pasting in a decorative image.

    Document assembly can stay simple:

    1. # Pseudocode: concatenate text conclusions and matching charts in order, then export the final document
    2. sections = [
    3. ("summary.txt", "summary.png"), # Overview and conclusions
    4. ("inventory.txt","inventory.png"), # Inventory percentile trend
    5. ("basis.txt", "basis.png"), # Basis percentile trend
    6. ]
    7. assemble_pdf(sections, out="quant_daily.pdf", toc=True)

    5. Multi-Model Comparison: Design the Evaluation Before Trusting Gut Feel

    Once different models all support agent mode, what’s really worth comparing isn’t “whose copy reads better” but execution stability on real tasks. A reliable evaluation must control at least these variables:

    • The same dataset and the same time window;
    • The same prompt template (identical role, input path, steps, constraints, and output format);
    • The same sandbox environment and the same dependency versions;
    • The same scoring rubric.

    Suggested scoring dimensions (0–5 each):

    Dimension What to observe
    Task planning Whether todos are correctly decomposed, whether the order is sound, whether key steps are missing
    Tool calling Code executability, self-repair ability after errors, whether there are repeated no-op calls
    Numerical correctness Deviation from independent script baselines, whether the caliber matches requirements
    Chart standards Presence of title/unit/source, correctness of threshold annotations, whether chart and text correspond
    Report completeness Whether the structure is complete, whether conclusions are data-backed, whether export succeeds
    Cost and latency Number of interaction rounds, execution time, quota consumption

    A typical finding in practice: under the same prompt and the same data, the “current inventory percentile” reported by different models scatters across roughly twenty-some to nearly thirty percentage points, while the “basis percentile” disperses even more widely, with some results differing from others by more than thirty percentage points. This spread alone proves two things:

    1. If you accept pure text answers, you have no way to judge which number is correct;
    2. As soon as you force the code-execution path and persist intermediate results, the numbers immediately converge to a single solution, and the only remaining model differences are in “expression and process organization.”

    This is also the core conclusion of the evaluation: the value of agent mode lies in eliminating numerical disagreement, not in introducing more “professional-looking” numbers.

    6. Tradeoffs of Cloud Agent Environments

    Beyond local deployment, there are also cloud coding environments that run in the browser. You don’t need to configure keys or runtimes locally; after registration you get free quota; and you can read and write files, run scripts, and call preset skill packages directly in a hosted session. The main advantage is extremely low onboarding cost, which suits quickly validating ideas and one-off analysis.

    The limitations are equally clear:

    • Quota constraints: once the free quota runs out, you can’t keep iterating, and long workflows tend to break midway;
    • Limited charting capability: some environments produce figures with generative images, and professional chart quality falls short;
    • Data boundaries: assess compliance requirements before uploading sensitive data—localized deployment or a private sandbox is more prudent;
    • Uncontrollable environment: dependency versions, available libraries, and network egress policies shift with the platform, which hurts long-term reproducibility.

    Engineering selection advice: use cloud environments for fast validation during exploration, then—once stable—solidify the work into a pipeline on your own container platform plus object storage. Lock dependency versions with images, and track the success or failure of each run with logs and monitoring alerts.

    7. Implementation Checklist and Risk Boundaries

    Prompt template (directly reusable)

    1. [Role] Senior quantitative researcher
    2. [Input] Data directory path, field descriptions, historical sample range
    3. [Steps] 1) Read and validate data 2) Compute metrics and percentiles 3) Call plotting libraries per factor logic to generate charts 4) Write conclusions 5) Merge and generate the document
    4. [Constraints] All numbers must be computed by scripts—no estimating from memory; charts must annotate units and data sources;
    5. calibers must be declared explicitly; key results require assertion checks
    6. [Output] Final document + generated code + intermediate result files

    Engineering recommendations

    • Deliver code alongside the report: reports must be traceable, and the code is the only audit evidence.
    • Persist intermediate results in structured form: store CSV/JSON archives for easy secondary verification and regression comparison.
    • Establish baselines and regression tests: maintain a manually verified golden sample, and run a deviation check each time the prompt or model changes.
    • Sandbox isolation: limit filesystem access scope and network egress, and default to no credentials when executing untrusted code.
    • Quota and concurrency governance: set maximum rounds, timeouts, and budget caps for long workflows to avoid infinite retries.
    • Manual review is non-negotiable: in high-value scenarios like investment and production scheduling, treat agent output as a draft and require a human to sign off on key conclusions.
    • Put data security first: define which data may enter cloud sessions and which must stay inside a private environment.

    8. Summary

    The real value of agent mode isn’t making the model “more like an analyst”—it’s turning analysis from “generating text” into “running programs.” When numbers are computed by scripts, charts are driven by data, and conclusions are backed by artifacts, the difference between models drops from “whose numbers are more accurate” to “whose process is more stable and whose expression is clearer”—and the latter can be continuously improved through evaluation and engineering constraints. Put reproducibility first and manual review last, and that is the prerequisite for this kind of automated workflow to enter production.