Continued Pre-Training and General Generation Capability Benchmark Plan for LLMs
This article presents a benchmark plan for comparing continued pre-training and general generation capabilities across three candidate model types: general-purpose open-source models, code-specialized models, and domain-enhanced models. It explains how training intensity, prompt format, domain knowledge coverage, code ability, instruction following, inference cost, statistical significance, prompt sensitivity, robustness, and license compliance affect model selection. The article also provides a layered evaluation framework, sample construction guidance, training budget considerations, and practical recommendations for running reproducible, cost-aware comparisons under a given data volume and training budget.
1. Breaking Down the Problem: Why These Three Candidate Model Types?
In domain continued pre-training, candidate models usually fall into three categories:
- General-purpose open-source models: pretrained and instruction-tuned on massive general corpora. They offer broad knowledge and strong reasoning and instruction-following capabilities, but support for domain-specific terminology and special symbols is limited.
- Code-specialized models: trained on large-scale code and “code–natural language” alignment data. They perform well on code semantics, English technical text, and comments, but have relatively limited Chinese corpora.
- Domain-enhanced models: built by continuing pre-training from a domain-specific open-source base, followed by instruction tuning and preference alignment. They are closest to the distribution of this task.
Two key variables affect the outcome:
- Continued pre-training intensity: the amount of domain training tokens, learning rate, corpus mixture, and number of unfrozen layers. Higher intensity may erase differences through domain training; lower intensity amplifies the impact of the initial model choice.
- Prompt form: code completion typically uses one- or two-line prefixes, while general generation involves long instructions, multi-turn dialogue, and structured output. The capabilities tested in these settings differ significantly.
Therefore, broad claims such as “one model class is better for continued pre-training” are of limited value. A more rigorous question is: under a given domain data volume and training budget, which initial model type yields the strongest, most stable, and most cost-effective result after training?
2. Technical Comparison: Capability Boundaries of the Three Model Types
- The domain knowledge ceiling depends on the base pretraining
Domain models only know what the continued pre-training corpus covers, while general models possess reasoning, multi-turn dialogue, and instruction-following capabilities. Domain labels may cover training data but may not cover special symbols, function signatures, long expressions, or machine instructions, leading to a situation where models “recognize tokens but do not understand structure.”
- Strengths and weaknesses of code models
Code models are better at completion, refactoring, and cross-file understanding. English documentation and technical comments are usually more abundant in their training corpora. However, their Chinese corpora are relatively limited, they overlap less with general generation tasks, and their coverage of domain terminology cannot be guaranteed.
- Adaptability and challenges of domain models
Domain models adapt fastest, but they are more prone to overfitting when data is limited or distribution shifts occur. Their instruction-following and open-ended generation capabilities require additional alignment. Selection should be based on return on investment rather than “who is stronger.”
- Inference load determines real-world cost
Sparse architecture models usually have activated parameters far smaller than total parameters and can use more aggressive quantization. However, under high concurrency, more experts may be activated per batch, increasing memory and bandwidth pressure. Dense models have predictable memory usage and simpler implementation; token throughput and unit price are mainly determined by the number of activated parameters.
3. Evaluation Plan: A Layered Metric System
The evaluation has three layers: the base layer (PPL, compression ratio) identifies distribution; the capability layer (generation, code, instruction following, Chinese) determines business fit; and the robustness layer (format, consistency, stress) ensures engineering usability.
| Dimension | Key metrics | Test content | Notes |
|---|---|---|---|
| Domain adaptation | Continued pre-training PPL, compression ratio | Calculate cross-entropy on 1/4/8/16 KiB samples; calibrate with zlib reference values | Whether difficulty is saturated by training data |
| Generation quality | Automatic metrics + human evaluation | ROUGE-L, BERTScore, key-point hit rate, 1–5 preference score | Double-blind human evaluation; golden-label agreement ≥ 0.8 |
| Code capability | Syntax/compilation pass rate, unit-test pass rate, domain API correctness | Completion, rewriting, function-level generation and explanation | Cross-file repository-level tasks should be evaluated separately |
| Instruction following | Format compliance rate, parameterized rewriting accuracy, win rate for general Q&A | JSON/table extraction, field-level validation, side-by-side comparison | A lightweight judge model can be used for initial screening, with human spot checks |
| Inference cost | P50/P95/P99 latency, tokens/s, GPU memory, concurrency limit, cost per task | Three-level concurrency stress test: business average, peak, peak ×2 | Include long-prompt scenarios; record token count per request |
Sample construction and budget recommendations:
- Coverage: Cover at least 8 subdomains; sources include historical support tickets, manually constructed samples, and real replays; deduplicate after anonymization; isolate training and evaluation sets at the document level.
- Scale: At least 300 evaluation items; perform at least 3 samples per generation target.
- Training budget: If fewer than 50B domain tokens are available, prioritize “domain model + shallow adaptation” rather than re-running large-scale pre-training on a general base.
# Pseudo-configuration: unified settings for the three candidate types to ensure comparabilitydata:domain_tokens: 20b # Available domain corpus sizedomain_code_ratio: 0.6 # Domain code-to-text ratiopacking: true # Pack samples to reduce padding wastetrain:method: lora # Full parameter updates only when budget is sufficientlr: 1e-5unfreeze_layers: bottom_half # Only general base models undergo continued pre-trainingkeep_fraction:general: 0.7code: 0.3domain: 0.9eval:ppl_samples_kib: [1, 4, 8, 16]max_new_tokens: 512temperature: 0.0 # Reproducibility firstseed: 42bench_concurrency: [1, 8, 32]
4. Experiment Implementation: Statistical Significance and Engineering Considerations
- Significance: For n samples and success rate p, the 95% confidence interval is approximately ±1.96·√(p(1−p)/n). For example, with n=300 and p=0.8, the interval is about ±4.5 percentage points. If the difference is smaller than this, it should not be treated as a meaningful advantage; increase sample size or use paired tests.
- Fair comparison: Use the same data mixture, training steps, and evaluation prompts for all three candidate types. Record random seeds, hardware models, software versions, and quantization precision.
- Prompt sensitivity and length effects: Format, number of examples, and prefix length can affect results. Use fixed templates and test at least three rewrites.
- Robustness and cost: Validate JSON validity after repeated sampling; measure cost and P95 latency per task and convert them to “hourly business throughput”; compare throughput under target concurrency after constraining GPU memory.
- License compliance: Check each model’s license and commercial-use terms; retain data provenance, cleaning rules, and deduplication records for training data; do not mix general base models with unauthorized data.
5. Conclusions and Practical Recommendations
- The scenario determines the initial choice: For completion, prioritize code models; for general generation, prioritize general-purpose models; when data is abundant, domain models start fastest.
- Unified evaluation and statistical significance are prerequisites: Use layered metrics, fixed prompts and decoding parameters, and report confidence intervals.
- Estimate high-concurrency deployment by activated parameters: Focus on throughput and GPU memory under peak concurrency rather than average token cost.
- Continued pre-training intensity determines whether a later model can catch up: Long-context and structured-output capabilities must be explicitly preserved during training.
- Recommended starting combination: Use a general base model as the foundation, set the code corpus ratio to 0.6, and keep domain models as a control group.
Further testing conditions: Re-evaluate when there is a major version update to any candidate model or when domain data grows by more than double. The report should include raw scores, confidence intervals, and cost tables, and should state clearly under which data scale and budget one model is better, avoiding conclusions that cannot be generalized.
