Core Datasets for LLM Training and Methods for Building General-Purpose Corpora

1. Why Data Effectively Defines the Model in LLM Training

In the large language model technology stack, differences in algorithm architecture are gradually narrowing. What increasingly separates model capabilities is the scale, quality, diversity, and mixing ratio of training data. A well-trained LLM’s knowledge, reasoning ability, language style, and even value tendencies can all be traced back to the corpus it has been trained on.

From an engineering perspective, an LLM training data system needs to answer four core questions:

1. Where data comes from — acquisition channels and compliance boundaries for raw corpora.
2. How data is processed — cleaning, deduplication, filtering, mixing, and annotation.
3. How data is validated — quality assessment, contamination detection, and performance attribution.
4. How data is iterated — using model evaluation results to improve the data strategy.

Understanding this system is a fundamental prerequisite for building a usable LLM. The following sections cover dataset types and general-purpose corpora.

2. Core Dataset Types for LLM Training

2.1 Pretraining Corpora

Pretraining datasets are the largest and most broadly scoped, typically measured in terabytes or even petabytes. The goal is to help models learn linguistic patterns, world knowledge, and basic reasoning ability from massive amounts of text. Typical components include:

• Web text: Content collected from the public web, covering news, encyclopedias, blogs, forums, technical documentation, and other formats. This type of corpus is large in scale but uneven in quality, requiring strict cleaning and filtering.
• Books and literature: Formal publications, academic papers, technical manuals, and other long-form text. This type of corpus tends to have standardized language and rigorous logic, making it important for improving deep reasoning and knowledge expression.
• Code corpora: Open source code repositories, code Q&A, and programming documentation, used to develop code understanding and generation capabilities. The share of code corpora has continued to rise in recent generations of LLMs.
• Multilingual corpora: In addition to Chinese and English, text in other languages is also required to ensure multilingual capability and cross-lingual transfer.

The core requirement for pretraining corpora is balancing broad coverage with high quality. Pursuing scale alone while ignoring quality causes models to learn noise and incorrect information. Over-filtering, on the other hand, reduces corpus diversity and narrows the model’s knowledge coverage.

2.2 Instruction Fine-Tuning Datasets

Instruction fine-tuning datasets are far smaller than pretraining corpora, usually ranging from hundreds of thousands to millions of samples, but their quality requirements are extremely high. The core format is the instruction-input-output triplet, and the goal is to teach models to understand human intent and complete tasks according to instructions.

Instruction data mainly comes from three sources:

• Human-written: Annotators manually construct instructions and standard answers based on task requirements. This produces the highest quality but also the highest cost.
• Model-generated with human filtering: Existing LLMs generate candidate instructions and answers, which are then evaluated and filtered by humans. This balances cost and quality.
• Real user interaction data: User questions and satisfactory answers collected from actual product usage, included after desensitization and review. This type is closest to real needs.

Key dimensions for instruction fine-tuning data include task type coverage (generation, extraction, reasoning, rewriting, code, and so on), diversity of instruction phrasing, consistency of answer formatting, and the presence of obvious errors or harmful content.

2.3 Human Preference Alignment Datasets

Alignment-stage data is used to train reward models or perform reinforcement learning from human feedback. The typical format is multiple answers to the same input plus human preference rankings. This type of data teaches models to distinguish between a correct answer and a better answer.

The key to building preference alignment data lies in annotation guideline design. Annotators need to rank candidate answers based on multiple dimensions such as helpfulness, truthfulness, and safety, and the ranking results must be sufficiently consistent. In practice, multi-round cross-annotation and consistency checking are commonly used to control annotation quality.

2.4 Domain Enhancement Datasets

When general-purpose LLMs are applied to specific industries, domain data is usually required to strengthen professional capabilities. Common domain enhancement data includes:

• Healthcare: Medical literature, clinical guidelines, drug instructions, medical Q&A, and similar content.
• Finance and legal: Annual reports, announcements, contracts, legal provisions, case analyses, and related material.
• Industrial manufacturing: Equipment manuals, process documentation, fault records, standards, and specifications.
• Education and research: Textbooks, exam questions, papers, lab reports, and so on.

Domain data can be introduced through continued pretraining or domain fine-tuning, depending on data scale and task requirements.

2.5 Multimodal Alignment Datasets

For multimodal LLMs that support image, audio, and video input, cross-modal alignment data such as image-text pairs and audio-text pairs is also required. This type of data requires strong semantic consistency between the text description and the corresponding modality content, making it significantly more difficult and expensive to build than text-only data.

3. Main Sources of General-Purpose LLM Corpora

3.1 Public Web Corpora

This is the largest source of corpus data. Text is collected from public web pages through crawlers and then processed through parsing, cleaning, and deduplication to form usable datasets. The advantages of public web corpora are broad coverage and relatively low acquisition cost. The challenges include copyright compliance, uneven content quality, and the need to handle large volumes of low-value repetitive content.

In practice, web corpora usually go through multiple filtering stages. The first stage is format filtering, which removes navigation bars, advertisements, scripts, and other non-content elements. The second stage is quality filtering, which uses language models or rule-based scoring to filter out low-quality pages. The third stage is deduplication, including both exact and near-deduplication.

3.2 Structured Knowledge Bases

Encyclopedic, academic, and industry structured knowledge bases are important sources of reliable factual knowledge for LLMs. This type of data is usually reviewed and organized by humans, and its quality is significantly higher than ordinary web text. Its value lies in providing high-credibility knowledge anchors that help models build accurate factual associations.

3.3 Open Code Repositories

Code corpora are critical for LLM programming capability. Sources include open source projects on public code hosting platforms, code snippets and answers from technical Q&A communities, and various programming tutorials and documentation. Code corpora require particular attention to license compliance as well as code completeness and executability.

3.4 E-Books and Academic Literature

Long-form text has unique value for developing deep comprehension and logical reasoning. E-books and academic papers are typically well structured and rigorously argued, effectively compensating for the lack of depth in short web text.

3.5 User Interaction and Dialogue Corpora

Subject to compliance requirements, desensitized real user interaction data has extremely high training value. This type of data reflects real user expression habits, question types, and need distributions, which significantly improves model practicality and scenario adaptability. However, data security and privacy protection rules must be strictly followed.

4. Data Processing Pipeline for General-Purpose Corpora

Regardless of the source, all corpora must go through a standardized processing pipeline before entering training. This pipeline typically includes the following key stages:

4.1 Collection and Parsing

Raw data is obtained from different channels and parsed into a usable format. For example, HTML pages are converted to plain text, PDF documents are extracted into readable text, and archives are decompressed before file traversal. The goal of this stage is to produce a uniformly structured raw text stream.

4.2 Cleaning and Normalization

The cleaning stage handles tasks such as removing HTML tags and special symbols, unifying character encoding, fixing garbled text, deleting empty lines and duplicate paragraphs, and normalizing full-width and half-width characters as well as letter casing. Cleaning quality directly affects all downstream stages.

4.3 Quality Filtering

Quality filtering typically combines rule-based and model-based methods. Rule-based methods include length threshold filtering to remove overly short or long text, language identification to remove non-target languages, and sensitive word filtering. Model-based methods use lightweight classifiers to score text quality and filter out low-quality content.

4.4 Deduplication

Deduplication has two levels: exact and near-deduplication. Exact deduplication is implemented through hash comparison and can directly remove completely identical text. Near-deduplication requires algorithms such as locality-sensitive hashing to identify documents with highly similar content. Deduplication not only reduces training waste but also lowers the risk of overfitting to specific content.

4.5 Privacy and Safety Processing

Personally identifiable information, contact details, financial data, and other sensitive content in the corpus must be identified and desensitized. Harmful content such as illegal material, violence, and explicit content must also be filtered out to ensure that training data meets safety and compliance requirements.

4.6 Mixing and Ratio Configuration

Data from different sources, languages, and domains must be mixed according to preset ratios. Ratio strategies are usually determined through small-scale ablation experiments, with the goal of finding the data combination that optimizes overall model capability.

5. Quality Evaluation and Data Iteration

Data quality evaluation cannot rely solely on manual sampling. A systematic indicator framework is required. Common dimensions include:

• Diversity: Whether the data covers enough domains, languages, styles, and task types.
• Correctness: Whether sample annotations or answers are accurate.
• Consistency: Whether processing standards for similar samples are uniform.
• Safety: Whether the data contains harmful, biased, or non-compliant content.
• Balance: Whether there is severe long-tail distribution or skew across categories.

After model training is complete, evaluation results should be used to trace data weaknesses. For example, if a model performs poorly on mathematical reasoning tasks, the quantity and quality of math-related corpus in the training data should be reviewed and supplemented accordingly. This evaluation-driven data iteration loop is a key mechanism for continuous LLM improvement.

6. Practical Recommendations and Summary

For teams planning to build an LLM training data system, the following points are worth noting:

First, build the data infrastructure first. This includes engineering capabilities such as data storage, version management, pipeline scheduling, and quality monitoring. The maturity of data infrastructure determines the efficiency of subsequent iterations.

Second, pay attention to data ratio experiments. There is no universally applicable ratio strategy. Systematic experiments are needed to find the optimal combination for your specific model goals.

Third, establish data quality red lines. Issues involving safety, compliance, and privacy should be subject to a zero-tolerance policy. The pursuit of scale should not lower these baseline requirements.

Fourth, treat data as a continuously iterated asset. LLM training data is not a one-time deliverable. It is a core asset that must be continuously updated and optimized as models evolve, user feedback changes, and application scenarios shift.

The upper limit of LLM capability is determined by data. This assessment remains valid at the current stage of technological development. Systematically understanding dataset classification, corpus sources, processing pipelines, and evaluation methods is essential knowledge for anyone working with LLMs. Only by building solid data engineering can model training achieve its intended results.