Why Model Accuracy Fluctuates with the Same Data and Hyperparameters: A Reproducibility Checklist

    Repeated training runs with the same dataset, network architecture, and hyperparameters can still produce validation accuracy that differs by more than ten or even twenty percentage points. This article explains the common engineering causes, including uncontrolled random seeds, inconsistent data splits, class mapping errors, incorrect training and evaluation modes, numerical nondeterminism, and evaluation logic bugs. It also provides a practical reproducibility checklist, guidance on comparing checkpoints and predictions, and recommendations for reporting results across multiple runs.

    In real machine learning projects, developers often encounter a confusing phenomenon: the dataset is unchanged, the network architecture is unchanged, and hyperparameters such as learning rate, batch size, and number of training epochs remain the same, yet two repeated training runs produce validation accuracy that differs by more than ten or even twenty percentage points.

    The first reaction is often that the model is unstable or that the training framework has a problem. From an engineering perspective, however, this type of fluctuation is usually not caused by a single factor. It is more often the result of multiple random components compounding with one another. If the training pipeline contains an unfixed random state, or if the validation pipeline has data leakage, inconsistent splits, or inconsistent preprocessing, then identical configurations do not necessarily represent identical experiments.

    1. Same Hyperparameters Do Not Mean the Same Training Process

    Hyperparameters describe the rules of the training process, such as learning rate, batch size, optimizer type, and number of epochs. The actual training trajectory that a model experiences also depends on initial parameters, sample order, data augmentation results, and low-level computation.

    A training run can be abstracted as:

    1. Final model = initial parameters
    2. + data split
    3. + sample order
    4. + data augmentation randomness
    5. + batch construction
    6. + gradient computation
    7. + parameter updates
    8. + numerical computation error

    If any one of these changes, the model may move toward a different local optimum.

    The loss function of a deep neural network is usually high-dimensional and non-convex. Training does not follow a single unique descent path. Instead, it continuously adjusts within a complex parameter space. Two models with slightly different initial states may enter completely different regions after hundreds of iterations. This difference is further amplified when the dataset is small, class distribution is imbalanced, or model capacity is large.

    Therefore, a so-called identical configuration must include at least the following:

    • The same data files and directory structure;
    • The same training, validation, and test splits;
    • The same class mapping;
    • The same preprocessing and data augmentation strategy;
    • The same random seed;
    • The same model initialization method;
    • The same batch order;
    • The same hardware and computation precision;
    • The same evaluation code.

    If only the learning rate and number of epochs are fixed while the variables above are not controlled, the experiment is not truly reproducible.

    2. Most Common Cause: Random Seeds Are Not Fully Fixed

    Model training usually contains at least three types of randomness.

    1. Parameter Initialization Randomness

    Network weights are generally not all initialized to fixed values. They are generated randomly according to some distribution. Different initial parameters affect:

    • The loss reduction speed in the first few epochs;
    • The learning order of features for different classes;
    • The direction of gradient updates;
    • The final convergence position.

    When the dataset is limited, initialization differences are especially noticeable. One initialization may learn stable features faster, while another may become biased toward a few classes too early.

    2. Data Order Randomness

    Training usually shuffles sample order. Even when the samples are exactly the same, different batch arrangements change the gradient estimate.

    Suppose a batch happens to contain more samples from similar classes. The model will reinforce certain features in the current iteration. The next batch may push parameters in another direction. After multiple epochs, small differences caused by early sample order may persist.

    3. Data Augmentation Randomness

    Operations such as random cropping, random flipping, random occlusion, and color perturbation generate different inputs each time a sample is read. Data augmentation helps improve generalization, but it also increases experimental variance.

    If augmentation strength is high, the training data the model sees each epoch is not actually identical. In this case, even if other conditions are consistent, two training runs may produce noticeable differences.

    From an engineering perspective, it is not enough to set a single random seed. You should also check:

    • The main process random state;
    • The numerical computation library random state;
    • The deep learning framework random state;
    • The random state of data loader workers;
    • The random state used by the data augmentation module;
    • The random state used by cross-validation or data splitting.

    In addition, a random seed can reduce random differences but cannot guarantee complete consistency across all hardware environments.

    3. Inconsistent Data Splits Can Be More Serious Than Model Randomness

    Many accuracy differences are not caused by the training process. They are caused by the two experiments actually using different data splits.

    Common issues include:

    1. Randomly splitting data again every time the program starts;
    2. Saving only split ratios but not the split result;
    3. File traversal order depending on the operating system or directory return order;
    4. Inconsistent sorting of class directories;
    5. Duplicates or overlap between training and validation sets;
    6. Different experiments reading different versions of data files;
    7. The same validation ratio but different specific samples.

    For example, one validation set may happen to contain many easy-to-recognize samples, so accuracy is naturally higher. Another validation set may contain more boundary samples and noisy samples, so accuracy may drop significantly. In this case, the model itself may not have changed to the same degree. What changed is the evaluation target.

    A more reliable approach is to save the sample identifiers for the training, validation, and test sets to separate files after the first split. All subsequent experiments should read this fixed list instead of performing random splitting again.

    At the start of training, output the following information:

    1. Number of training samples
    2. Number of validation samples
    3. Number of test samples
    4. Number of samples per class
    5. Sample identifier digest for each dataset
    6. Class name-to-index mapping

    This makes it possible to quickly determine from experiment logs whether two runs used exactly the same data.

    4. Class Mapping Errors Can Cause “Abnormal Accuracy”

    In classification tasks, class names usually need to be mapped to integer indices. If the class-to-index mapping depends on directory order, string sorting, or file read order, the mapping may change across environments.

    For example, during training the mapping may be:

    1. Class A -> 0
    2. Class B -> 1
    3. Class C -> 2

    During evaluation, however, it may become:

    1. Class B -> 0
    2. Class A -> 1
    3. Class C -> 2

    The model outputs may be numerically correct, but they are interpreted inconsistently with the labels, causing a significant drop in accuracy.

    Therefore, the class mapping must be saved explicitly and reused during both training and evaluation. Do not rely on implicit sorting, and do not generate mapping tables separately in different scripts.

    Also check whether labels start from zero and are continuously numbered, and whether there are duplicate class names, case differences, whitespace differences, or hidden characters. Many cases of “sudden accuracy drop” are actually caused by inconsistent label handling.

    5. Training and Evaluation Modes Must Be Switched Correctly

    Some network layers behave differently during training and inference, such as:

    • Dropout layers;
    • Layers related to running statistics;
    • Normalization layers that depend on batch statistics.

    Training mode should be enabled during training, and evaluation mode should be enabled during evaluation. If the validation stage is still in training mode, the model may continue to apply dropout or update statistics, causing the same model to produce different results across multiple validation runs.

    A standard flow should clearly separate the two stages:

    1. Training stage:
    2. Switch to training mode
    3. Compute loss
    4. Backpropagate
    5. Update parameters
    6. Validation stage:
    7. Switch to evaluation mode
    8. Disable gradient computation
    9. Iterate over the fixed validation set
    10. Aggregate predictions

    In addition, the validation set should not use random data augmentation. The training set can use random transformations, but the validation and test sets should use deterministic preprocessing. Otherwise, each evaluation sees different inputs, and accuracy will naturally fluctuate.

    6. Parallel Computation and Hardware Introduce Numerical Differences

    In parallel computing environments, the order of floating-point operations may change. Because floating-point numbers have finite precision, different addition orders, thread reductions, and parallel algorithms can all produce extremely small rounding errors.

    A single computation error is usually very small, but deep networks go through many iterations, and some optimization algorithms are sensitive to tiny differences. After long training, an initial small numerical deviation may evolve into a different parameter trajectory.

    The following scenarios can easily introduce non-determinism:

    • Multi-process data loading;
    • Parallel reduction operations;
    • Automatic algorithm selection;
    • Mixed-precision training;
    • Different hardware devices;
    • Different drivers or low-level math libraries;
    • Dynamically selected computation kernels.

    If the goal is to diagnose the problem, start with a deterministic experiment: use single-process data loading, disable random augmentation, use a fixed data order, disable mixed precision, and keep the hardware environment as consistent as possible. After the problem is located, gradually restore parallel and performance optimization settings.

    Note that full determinism often sacrifices some training speed. It is therefore better suited to debugging, regression testing, and issue reproduction than to all production training tasks.

    7. Large Validation Accuracy Gaps May Also Indicate Evaluation Errors

    Accuracy does not always reflect model performance, especially on imbalanced datasets.

    Suppose one class accounts for ninety percent of the validation set. A model that predicts only the majority class may still obtain apparently high accuracy. If different experiments change their ability to recognize minority classes, overall accuracy may fluctuate significantly, but this metric change does not necessarily mean the model has become comprehensively better or worse.

    It is recommended to monitor the following at the same time:

    • Per-class accuracy;
    • Precision;
    • Recall;
    • F1 score;
    • Confusion matrix;
    • Macro-average metrics;
    • Weighted-average metrics;
    • Prediction confidence distribution.

    If overall accuracy differs by twenty percentage points between two training runs, compare the confusion matrices first instead of immediately concluding that the model has failed. The confusion matrix can show which classes the problem is concentrated in, whether one class cannot be recognized at all, or whether all classes have declined uniformly.

    Also confirm that the denominator used for accuracy calculation is correct. Common errors include:

    • Counting only the last batch;
    • Ignoring samples in the final incomplete batch;
    • Averaging by batch instead of weighting by sample;
    • Misaligning labels and predictions;
    • Exiting early during validation;
    • Failing to aggregate multi-GPU results correctly.

    To systematically address accuracy fluctuation, investigate in the following order.

    Step 1: Fix the Data

    Save the data version, sample list, data splits, and class mapping. Ensure that every training run reads the same set of samples.

    Step 2: Fix Randomness

    Set all random sources consistently and record the random seeds. Data loader workers should also use traceable random states.

    Step 3: Disable Random Augmentation

    During troubleshooting, temporarily disable all random data augmentation. If results still fluctuate, the problem may come from initialization, data order, evaluation flow, or the computation environment.

    Step 4: Reduce Experiment Scale

    Train on a small set of fixed samples only. If the model cannot even fit a small dataset stably, check labels, loss function, gradient updates, and mode switching first.

    Step 5: Record Complete Logs

    Record training loss, validation loss, overall accuracy, per-class metrics, learning rate, and best model epoch for each epoch. Do not keep only the final accuracy.

    Step 6: Compare Parameters and Predictions

    After two training runs, compare:

    • Model parameter differences;
    • Prediction counts for each class;
    • Confusion matrices;
    • Lists of error samples;
    • Output confidence;
    • Best epoch;
    • Validation loss curves.

    This helps distinguish between “different training processes” and “an incorrect evaluation process.”

    9. Do Not Judge Only the Last Model; Track the Best Checkpoint

    Training for one hundred epochs does not mean the last epoch is necessarily the best. Validation performance may rise first and then decline due to overfitting. Two training runs may have similar best accuracy, but very different accuracy at the last epoch.

    A more reasonable strategy is to save the best checkpoint according to validation metrics and record the best epoch and corresponding metrics. For tasks with obvious random fluctuation, repeat experiments with multiple random seeds and report the mean and standard deviation instead of showing only one result.

    For example, experimental results should be expressed as:

    1. Accuracy: mean ± standard deviation
    2. Macro-average metric: mean ± standard deviation
    3. Best epoch: median or range

    This better reflects the model’s true capability than “one training run reached a certain accuracy.”

    10. Conclusion: Prove Experiment Consistency Before Discussing Model Quality

    Large accuracy differences with the same dataset and hyperparameters do not necessarily mean the model itself is unreliable. More common causes are uncontrolled random factors, different data splits, class mapping errors, mixed training and evaluation modes, or inconsistent metric calculation logic.

    When troubleshooting this type of issue, follow one basic principle:

    1. First verify input consistency
    2. Then verify training process consistency
    3. Finally verify evaluation consistency

    Only after the data, random state, computation environment, and evaluation flow have been confirmed can the remaining differences be attributed to model randomness or generalization capability.

    In engineering practice, a mature training pipeline should not provide only a single accuracy number. It should also provide complete experiment records, fixed data lists, reproducible random states, clear metric definitions, and statistical results from multiple runs. This is how a team moves from “occasionally getting a good result” to “reliably obtaining credible results.”