Agentic LLM Workflows for Quantitative Analysis and Automated Report Generation

    This article explores how AI agents transform LLMs from answer-generating chatbots into task-executing systems for quantitative research and automated report production. It explains why computation should be delegated to deterministic code while semantic tasks remain with the model, details a seven-stage pipeline for generating chart-enriched PDF reports, compares code-generated vs. generative graphics, and outlines an evaluation framework for multi-model benchmarking. Practical engineering considerations such as sandboxed execution, cost control, and validation checkpoints are also covered.

    From “Question Answering” to “Execution”: What Agent Mode Changes

    In the past, using LLMs for data analysis essentially meant “asking the model for an answer”: pasting a table into a chat window and asking it to compute quantiles, basis spreads, or period-over-period changes. This approach has a low ceiling — the model produces the most probable text, not the correct numeric value.

    Agent mode changes the execution structure: instead of directly generating conclusions, the model produces plans + tool invocations. It breaks down the task, builds a to-do list, and then sequentially calls scripts to read data, retrieve information, run calculations, render charts, and assemble documents. The model’s role shifts from “exam taker” to “programmer + project lead.”

    This shift yields three immediate benefits:

    • Reproducibility: Every step has a script and intermediate files; results can be re-run and audited.
    • Verifiability: Numeric outputs come from code and can be compared digit-by-digit against baseline implementations.
    • Deliverability: The output evolves from plain text into a document with figures and tables, ready for direct use.

    The Root of Reliability: Delegate Computation to Code, Expression to the Model

    Three Sources of Hallucinations

    When models make errors on structured numeric data, the cause is usually not “inability to compute” but three compounding mechanisms:

    1. Parametric memory bias: Numbers in training data are compressed into model weights, making recall inherently lossy.
    2. Probabilistic sampling: Decoding optimizes for fluency rather than exactness; digits beyond the first decimal are easily “guessed.”
    3. Attention decay over context: Information in the middle of long tables is recalled less reliably than at the edges.

    The conclusion is straightforward: anything computable with deterministic code should not be directly emitted by the model.

    A Reusable Implementation Principle

    Split tasks into two categories: the deterministic layer goes to code, the semantic layer stays with the model.

    1. # Example: quantile and statistical computation (illustrative pseudocode)
    2. import statistics
    3. series = [float(x) for x in rows["close"]]
    4. q = statistics.quantiles(series, n=4) # quartiles
    5. median = statistics.median(series) # median
    6. rank = sum(1 for x in series if x < current) / len(series)
    7. payload = {
    8. "current_quantile": round(rank * 100, 2),
    9. "basis_quantile": round(basis_rank * 100, 2),
    10. "median": round(median, 4),
    11. }

    Three design decisions are critical: define the calculation method before computing (whether quantiles use empirical, linear-interpolated, or rank-based methods must be specified in configuration); persist intermediate results to disk (JSON/CSV, referenced by both charts and documents downstream); and conclusion text is generated by templates or the model based on pre-computed values — the model should never re-state numbers from scratch.

    A Typical Workflow: How an Automated Research Report Is Produced

    A mature agent pipeline generally includes seven stages:

    1. Task planning: Generate a to-do list with inputs, outputs, constraints, and acceptance criteria.
    2. Data ingestion: Read datasets from local files or object storage, with type and missing-value validation.
    3. Information retrieval: Fetch supplementary context via web search or enterprise knowledge bases to explain drivers behind the data.
    4. Computation: Run scripts to produce metrics, quantiles, basis spreads, correlations, and other intermediate results, persisted to disk.
    5. Chart rendering: Call plotting libraries to generate figures from the same data source.
    6. Document assembly: Merge text, tables, and images into a template, exporting to PDF or other formats.
    7. Self-validation: Re-open the generated artifact to verify numbers, chart axes, legends, and body text for consistency.

    A single prompt constraint — for example, “automatically invoke charting tools for each section and merge the text with generated figures into a PDF report” — is usually enough for the agent to automate stages six and seven. This “coding-style agent” operates much like command-line AI coding assistants: plan first, then execute shell commands step-by-step to read and write files.

    Managed coding-agent environments have also emerged: browser-based platforms offering a ready-to-use runtime host without requiring API key integration. These typically include a free quota for new accounts, but note that quota exhaustion means no further edits to artifacts, and you should be aware of isolation boundaries between data and code within the sandbox.

    Why Professional Charts Must Be Code-Generated

    This is the fundamental divergence between agent mode and generative image models.

    Dimension Code-Generated Charts Generative Image Output
    Data source Every point computed from concrete values Pixels synthesized by a probability model
    Numerical accuracy Strictly consistent with input data No semantic guarantee; coordinates untrustworthy
    Reproducibility Same data + same script → same chart Different result each time
    Modifiability Change data or parameters and re-render Requires regeneration; precise correction difficult
    Auditability Traceable to the computation process No provenance possible

    For research reports, financial statements, and quantitative reviews, charts exist to communicate numbers credibly. A chart with fabricated axes, legends, or units has no professional value no matter how polished it looks. The chart pipeline must be “data → code → figure,” with the figure referencing the same intermediate results as the tables in the document.

    Multi-Model Evaluation: What Counts as a Valid Comparison

    Performance differences between agent modes are real, but “it feels stronger” is not a conclusion. Reproducible evaluation requires fixing four variables: same dataset, same prompt, same toolset, same acceptance criteria.

    Evaluate across four layers:

    • Task completion: Whether a complete document is produced with the required number of charts and sections.
    • Numerical correctness: Digit-by-digit comparison against a baseline implementation (e.g., standard-library quantile functions), noting any decimal drift.
    • Tool call quality: Success rate, retry count, tendency to loop, and whether the computation method was silently changed.
    • Cost and efficiency: End-to-end latency, token consumption, and overhead from failed reruns.

    Recommended fields for an evaluation tracking table:

    Field Description
    Model/version ID Anonymous identifier to reduce bias
    Numeric result matches baseline? Yes/No + magnitude of deviation
    Charts generated by code? Yes/No (a non-negotiable criterion)
    Tool call success rate / average retries Reflects engineering stability
    Document completeness Coverage of sections, figures, tables, appendices
    End-to-end latency / per-run cost For capacity and budget planning

    A recurring pattern in head-to-head comparisons: different models produce inconsistent results for the same inventory-quantile/basis-quantile task, and all may output approximate values in the lower decimal places. This confirms the conclusion from earlier: differences in quantile results are mostly not about “intelligence” differences between models, but about inconsistent calculation conventions combined with direct model answering. Evaluation must lock the calculation method first, force code execution, and only then compare models — otherwise you are measuring randomness rather than capability.

    Known Limitations and Risk Checklist

    • Numeric approximation: Model-direct number output is unreliable. Every number entering a conclusion must be computed by scripts and verified on read-back.
    • Chart semantic errors: Axis labels, units, time ranges, and legends may be rendered incorrectly. Programmatic assertions are needed (e.g., checking element counts match data point counts).
    • Unstable execution: Tool call timeouts, script errors, and infinite retries require maximum step counts and timeout limits.
    • Prompt injection: Web-retrieved content may contain malicious instructions. Treat external content as data, not instructions, and restrict it from reaching credentials.
    • Permissions and compliance: Data egress, crawling scope, and audit trails require clear policies.
    • Cost blowout: Multi-round tool calls amplify token consumption. Set budget thresholds for long-running tasks.

    Engineering for Production: Turning Agents into Operable Services

    Moving this workflow from demo to production requires standard cloud infrastructure:

    • Sandboxed execution: Untrusted scripts run in isolated containers with CPU, memory, and duration limits, with no access to metadata endpoints.
    • Object storage: Stores input datasets, all intermediate artifacts, and final documents, naturally providing versioning and audit trails.
    • Message queues: Decouple long-running tasks, supporting asynchronous submission, failure redelivery, and load smoothing.
    • Container platform: Hosts stateless charting and document-rendering workloads with elastic scaling.
    • Logging and monitoring: Records input summaries, duration, and exit codes for each tool call, with alerts on success rate and P95 latency thresholds.

    These capabilities can be assembled using standard product categories such as object storage, message queues, container platforms, log services, and monitoring alerting. For specific configurations on Baidu AI Cloud, consult the official documentation.

    Implementation Checklist

    1. Define metric calculation conventions in configuration files first; the model is not allowed to decide.
    2. All numbers, charts, and documents share the same set of intermediate result files.
    3. Charts are always code-generated; generative image output is forbidden from professional report pipelines.
    4. Add automated validation at the end of the pipeline: numeric comparison against baselines, chart element-to-datapoint consistency, and section completeness.
    5. Set maximum tool invocation steps, timeouts, and budget limits; failed runs can be retried and artifacts can be rolled back.
    6. Treat all externally retrieved content as untrusted data, stripping any embedded instructions.
    7. Build an anonymized evaluation set with fixed data, prompts, and tools; run regression comparisons periodically.

    The real value of agent mode is not making models “smarter at guessing numbers” — it is enabling models to invoke deterministic tools to compute numbers accurately, render charts truthfully, and deliver complete documents reliably. Understanding this boundary is more useful than memorizing any model version number.