Getting Started with Custom AI Agent Data: Three Data Types and Three Construction Paths

    Building a custom AI Agent often stalls at the data collection stage, particularly when it comes to capturing complete task trajectories. This article explores three essential types of training data—long-term agent trajectories, real-world engineering debugging samples, and scientific reasoning closed loops—and outlines three practical paths to construct them: synthetic data generation, business log extraction, and reinforcement learning environments. It also details the necessary Agent capabilities, evaluation metrics for long context retrieval, and a comprehensive quality checklist to ensure data reliability.

    1. Conclusion First

    Teams often get stuck on training data, especially data that covers the complete lifecycle of a task process. This article recommends starting with one of three types of data: long-term agent trajectories, real-world engineering debugging samples, and scientific reasoning closed loops. When starting out, prioritize low-risk business scenarios that can be directly obtained from internal systems, and then use synthetic data and reinforcement learning environments to expand coverage. This article does not recommend directly collecting high-risk data such as financial credit approvals or medical authorizations.

    2. Required Capabilities for an Agent Base

    An Agent model capable of handling business tasks must possess at least three core capabilities.

    1. Long-chain planning and backtracking error correction. The model must break down large goals into subtasks and adjust its path based on environmental feedback. When an API call fails or code throws an error, the model needs to understand the error message and retry with an alternative approach.
    2. Engineering debugging capability. The model should be able to read an entire codebase and perform multi-file linked modifications, dependency conflict resolution, and small-scale refactoring. Simply generating isolated code snippets is insufficient for real-world engineering tasks.
    3. Multimodal and long-context retrieval. The model must simultaneously process text, images, and tables, while accurately retrieving information within a long context.

    The table below maps capabilities, training data, and evaluation methods to help teams verify whether each capability is supported by data.

    Capability Primary Data Evaluation Method
    Long-chain planning & backtracking Long-term trajectories Task completion rate, average retry count
    Engineering debugging Real debugging samples Test pass rate after fixes
    Scientific reasoning Reasoning closed-loop records Conclusion reproducibility rate, contradiction correction rate
    Long context retrieval Synthetic long documents Accuracy on fixed retrieval tasks

    Metrics like “million-token context” require clear definitions. Simply providing the window length does not explain the model’s actual usable range. Teams should first clarify whether the calculation represents the total number of input and output tokens, or only the input portion. Then, use a set of fixed retrieval questions to measure accuracy. The evaluation report should include the following fields:

    • Number of questions: At least 200 questions, covering scenarios where the target information is located at the beginning, middle, and end of the context.
    • Input and output token counts: Recorded separately, with a clear note on the calculation method.
    • Accuracy: The number of correct answers divided by the total number of questions, reported in tiers based on context length.
    • Failure samples: Retain the original context position and the model’s answer for troubleshooting.

    These capabilities are difficult to achieve merely by increasing parameters or altering network architectures. The quality and complexity of the training data are usually the more direct limiting factors.

    3. Three Types of Scarce Training Data

    First, let’s define a few terms. A trajectory is the complete step-by-step record of an Agent completing a task. A sample is a trajectory or data entry usable for training or evaluation. A closed loop refers to the complete process from discovering a problem, analyzing the cause, to fixing and verifying the solution.

    3.1 Long-term Agent Trajectories

    Standard Q&A data only contains a single question and answer, which cannot cover the complete process of an Agent. A useful trajectory should include goal decomposition, subtask planning, tool calling, failure retries, and backtracking error correction. Each step records three types of information: Thought (the reasoning process), Action (the API called or code executed), and Observation (the result returned by the environment).

    Business trajectories in specific languages (such as Chinese) are particularly scarce. Examples include localized office automation flows, enterprise approval processes, and operations within specific regional office software.

    • Applicable scenarios: Low-risk processes like office automation and internal report processing. Annotation guidelines should define field formats, failure retry annotation methods, and desensitization rules, with sample quality checks on the annotated results.
    • Inapplicable scenarios: Processes like financial credit approvals or medical authorizations. Before collection, compliance reviews must be completed, personal information in samples must be desensitized, and data usage authorization must be obtained.

    3.2 Real Software Engineering Debugging Samples

    Open-source code datasets are usually cleaned, making the code clean and runnable. Real-world development, however, is full of noise, including error stack traces, reproduction steps, Diff files in version control, and multi-file linked modification records.

    For a model to learn debugging, it must be exposed to these noisy contexts. For example, when encountering a dependency conflict, the model needs to read the logs, find the version number in the configuration file, modify it, and re-run the tests. The complete closed loop—from finding the problem, locating the cause, to fixing and verifying—is the most helpful for engineering capabilities.

    • Applicable scenarios: Teams with internal code repositories and CI logs can extract samples from historical tickets and commit records. During extraction, filter out confidential information first, then perform a license scan.
    • Inapplicable scenarios: Code repositories containing customer source code or third-party code restricted by licenses. These codes must not enter the training set until usage permissions are confirmed.

    3.3 Scientific Reasoning and Logic Validation Closed Loops

    Scientific reasoning requires the model to provide a conclusion, display the derivation process, and proactively correct itself when logical contradictions are found. This type of data usually requires the participation of domain experts, including proposing hypotheses, designing simulation experiments, analyzing results, and correcting conclusions.

    The collection cost is relatively high. Experts need to review each entry one by one, with the time spent per entry depending on the problem’s complexity. Teams should first select a small-scale problem set, such as around ten physics or chemistry simulation questions, verify the process, and then scale up. Each record should include the hypothesis, experimental design, input parameters, results, conclusion, comparison before and after correction, and expert sign-off.

    • Applicable scenarios: Scientific research tasks with a clear simulation environment or calculable formulas.
    • Inapplicable scenarios: Unpublished research findings or confidential experimental data, unless authorization has been obtained.

    4. Three Construction Paths

    Path 1: Synthetic Data

    Generate trajectories using a strong model, then filter them using rules or tests.

    • Steps: Define task templates and tool lists; generate multiple trajectories covering both normal flows and failure branches; use executable tests to determine if the result is correct, keeping only passing samples; conduct manual spot checks to see if fabricated tool return values exist.
    • Required Permissions & Environment: API access to the generation model, and a reproducible sandbox tool environment.
    • Quality Threshold: Record the pass rate and manual spot-check error rate. Pause generation if the error rate exceeds the threshold set by the team.
    • Cost: Primarily model API call costs and manual spot-check labor.
    • Risk: Synthetic data can easily replicate the model’s existing biases or contain incorrect but seemingly plausible steps. It should be mixed with real samples before training, and a portion of real data should be reserved for evaluation.

    Path 2: Business Logs

    Extract trajectories from the operation logs of existing systems, such as ticketing systems, approval records, and office software operation logs.

    • Steps: Identify available log sources and apply for read access; chain events by task ID to reconstruct the goal, steps, and results; remove personal information, accounts, tokens, and internal addresses; have business personnel supplement the Thought field and annotate failure reasons.
    • Required Permissions & Environment: Log read access, an independent desensitization processing environment, and clear log retention periods.
    • Quality Threshold: Every trajectory must have complete start and end events; trajectories missing events are excluded from the training set.
    • Cost: Labor for desensitization and annotation accounts for the majority.
    • Risk: Logs may contain personal information, and log formats can change as systems upgrade. Extraction scripts should record version numbers and be re-validated when formats change.

    Path 3: Reinforcement Learning Environments

    Build a repeatable simulation environment where the Agent can trial-and-error, and provide rewards based on the results.

    • Steps: Write clear success conditions for each task type; build a sandbox ensuring consistent state after each reset; design a reward function and check for exploitable shortcuts; record all interactions as a source of trajectories for subsequent training.
    • Required Permissions & Environment: Container or virtual machine clusters, and a network environment isolated from production systems.
    • Quality Threshold: The result consistency rate for repeated runs of the same task meets the team’s set standard, and the reward function has been manually reviewed.
    • Cost: The engineering effort for environment development and maintenance is significant, typically making it the highest among the three paths.
    • Risk: The Agent may easily learn to exploit rule-bypassing shortcuts, such as directly modifying test results. The environment must prohibit access to the scoring code.

    5. Quality Checklist

    Check Item Pass Criteria
    Trajectory completeness Includes start, end, and all intermediate steps with no breakpoints
    Field completeness Each step has Thought, Action, and Observation
    Failure retry annotation Failure steps indicate the cause; retry steps correlate with the original steps
    Backtracking error correction Steps after the backtracking point align with the reason for backtracking
    De-identification Names, accounts, tokens, and intranet addresses in logs have been replaced