NRI and IDI in Cox Regression: Advanced Evaluation Methods for Survival Models

    When adding a new biomarker or clinical variable to a Cox model, a small AUC difference may be too coarse or too insensitive to support a reliable conclusion. This article explains how Net Reclassification Improvement (NRI) and Integrated Discrimination Improvement (IDI) complement AUC in survival model comparison. It covers the statistical rationale for categorical and continuous NRI and for IDI, explains why survival analyses must anchor evaluation to a fixed time point, and outlines a practical workflow for fitting base and extended Cox models, predicting event probabilities, calculating NRI and IDI, and estimating bootstrap confidence intervals. It also summarizes interpretation criteria and practical cautions, including clinical threshold selection, over-optimism, sample size, joint reporting with AUC, and the distinction between discrimination and calibration.

    Why AUC Is Often Insufficient for Model Comparison

    In survival analysis, the Cox proportional hazards model is the most commonly used multivariable analysis tool. After building a base model, researchers often introduce a new biomarker or clinical indicator to test whether the new variable improves predictive performance. At this point, the first instinct is usually to compare the area under the curve (AUC) of the two models.

    In practice, two problems stand out:

    1. Limited sensitivity. AUC measures the model’s overall discrimination for whether an outcome event occurs, and it depends heavily on sample size. When the effect of the added variable is small, the AUC difference between the two models is often only on the order of 0.005 to 0.01 and may fail to reach statistical significance. Researchers may therefore draw the conservative conclusion that the new variable has no value.
    2. Coarse information granularity. AUC is a highly summarized single number. It cannot answer questions such as “on which part of the population does the new model perform better?” It also obscures the specific process by which the model reassigns risk levels near a threshold.

    To address these problems, two complementary evaluation metrics have been proposed: Net Reclassification Improvement (NRI) and Integrated Discrimination Improvement (IDI). Both are more sensitive than AUC and can characterize the improvement of a new model over an old model more precisely.

    Statistical Principles of NRI

    Core Idea

    NRI focuses not on the model’s overall discrimination, but on how the new and old models reclassify individuals. Consider a binary prediction setting: according to a risk threshold, such as classifying predicted probability greater than 50% as high risk, each individual is assigned by both the old and new models to a predicted category of “event” or “non-event.”

    A 2×2 cross-table can describe the movement of categories from the old model to the new model:

    • Event group (individuals who truly experience the outcome event): Movement from “low risk” to “high risk” is a correct movement (upward reclassification); movement from high risk to low risk is an incorrect movement (downward reclassification).
    • Non-event group (individuals who do not experience the outcome event): Movement from “high risk” to “low risk” is a correct movement; movement from low risk to high risk is an incorrect movement.

    Calculation Formula

    NRI is defined as:

    • Event-group improvement = proportion of upward reclassification in the event group − proportion of downward reclassification in the event group
    • Non-event-group improvement = proportion of downward reclassification in the non-event group − proportion of upward reclassification in the non-event group
    • Total NRI = event-group improvement + non-event-group improvement

    If NRI is greater than 0, the new model assigns more “individuals who will truly experience the event” to high risk while assigning more “individuals who will truly not experience the event” to low risk. In other words, the overall direction of reclassification is correct, suggesting that the new model provides improvement.

    Categorical NRI and Continuous NRI

    Depending on whether risk thresholds are prespecified, NRI is divided into two types:

    • Categorical NRI: Requires one or more clinical thresholds, such as 10% and 20%. Its advantage is that it is close to the clinical decision-making setting; its disadvantage is that results are sensitive to threshold choice, and different thresholds may lead to different conclusions.
    • Continuous NRI: Does not perform any threshold discretization. It directly counts the proportion of all individuals whose predicted probabilities are shifted upward to a higher value versus downward to a lower value. It uses all continuous information and has higher statistical power; in recent years it has been applied more widely.

    Statistical Principles of IDI

    Unlike NRI, which is based on “classification counts,” IDI is based on differences in predicted probabilities. It measures the change in the ability of the two models to distinguish the event group from the non-event group.

    Its calculation logic is:

    • Old-model discrimination = mean predicted probability in the event group − mean predicted probability in the non-event group
    • New-model discrimination = calculated in the same way
    • IDI = new-model discrimination − old-model discrimination

    The meaning of IDI can be understood as the combined improvement from how much the new model increases the average predicted probability in the case group while decreasing the average predicted probability in the control group. An IDI greater than 0 suggests that the new model has stronger discrimination overall. Because IDI uses continuous probability information and does not depend on a threshold, it is generally also considered more sensitive than the AUC difference.

    Implementation Workflow in Cox Regression

    Survival analysis differs from ordinary logistic regression: the outcome is “whether an event occurs during a given follow-up period.” Therefore, the calculation of NRI/IDI must explicitly anchor to a time point. The complete implementation workflow is as follows.

    Step-by-Step Breakdown

    1. Determine the evaluation time point t: For example, “5-year event risk.” NRI and IDI are defined only at a specified time point, and results without a specified time point have no interpretable meaning.
    2. Fit the two Cox models separately: The base model (without the new variable) and the extended model (base variables + new variable).
    3. Calculate each individual’s predicted event probability at time point t: Based on the baseline survival function and the individual covariate linear predictor, convert the result into each subject’s cumulative event probability by time t.
    4. Define high risk and low risk (for categorical NRI): Discretize the predicted probabilities according to prespecified thresholds and construct the cross-table.
    5. Calculate the point estimates of NRI and IDI.
    6. Obtain 95% confidence intervals through bootstrap resampling: Because the sampling distribution of reclassification proportions is complex, the mainstream approach is repeated resampling, such as 1000 replications, to estimate confidence intervals.

    General Pseudocode

    The following neutral pseudocode describes the core logic and can be implemented in any statistical environment that supports survival analysis:

    1. # 1. Fit two Cox models
    2. fit_old <- coxph(Surv(time, status) ~ x1 + x2 + x3, data = dat)
    3. fit_new <- coxph(Surv(time, status) ~ x1 + x2 + x3 + x4, data = dat)
    4. # 2. Calculate predicted event probabilities at time point t
    5. p_old <- predict_survival_prob(fit_old, times = 1825) # 5 years
    6. p_new <- predict_survival_prob(fit_new, times = 1825)
    7. # 3. Calculate NRI and IDI (confidence intervals estimated internally via bootstrap)
    8. result <- compute_nri_idi(
    9. event = dat$status,
    10. p_old = p_old,
    11. p_new = p_new,
    12. cutoff = c(0.10, 0.20), # thresholds for categorical NRI; omit for continuous NRI
    13. t_horizon = 1825,
    14. n_boot = 1000
    15. )

    Interpreting Results

    Typical output includes the following components. The interpretation principles are:

    Indicator Criterion for Improvement
    Event-group NRI Confidence interval excludes 0 and is positive
    Non-event-group NRI Confidence interval excludes 0 and is positive
    Total NRI Confidence interval excludes 0 and is positive
    IDI Confidence interval excludes 0 and is positive

    Only when total NRI, its two components, and IDI all point in the same direction and their confidence intervals all exclude the null value can the conclusion that “the new model is better than the old model” be robustly supported.

    Practical Considerations

    1. A time point must be specified. NRI/IDI for survival data depend strongly on the evaluation time. Improvement at 5 years does not mean that improvement remains at 10 years. It is advisable to declare the time point in the study design in advance to avoid post hoc selection.
    2. Thresholds should have a clinical basis. Thresholds for categorical NRI should come from guidelines or existing literature, such as common cardiovascular risk stratification criteria, rather than being reverse-engineered from the data. Otherwise, there is a risk of “tuning the threshold to fit the results.”
    3. Beware of over-optimism. If the new variable was selected on the same dataset, directly calculating NRI will overestimate the degree of improvement. It is advisable to use separate training and validation sets, or to resample the entire workflow of “fit model—calculate predictions—evaluate NRI” within bootstrap.
    4. Sample size requirements. When the number of events is too small, estimates of reclassification proportions become highly unstable, and NRI confidence intervals become extremely wide. In general, it is advisable to have a sufficient number of events, for example at least several dozen events per group, and to interpret the results cautiously.
    5. NRI, IDI, and AUC should be reported together. NRI reflects improvement at the “classification decision” level, IDI reflects improvement at the “probability discrimination” level, and AUC reflects overall discrimination. The three are complementary, and relying on any one alone may lead to misjudgment.
    6. Discrimination and calibration are different concepts. NRI/IDI evaluate only discrimination, not whether the predicted probabilities are “accurate.” A model that has improved discrimination but systematically overestimates risk still requires comprehensive assessment with calibration curves.

    Conclusion

    When introducing new variables into Cox regression modeling, NRI and IDI provide evaluation dimensions that are more sensitive and more granular than the AUC difference. NRI characterizes improvement in decision-making at the threshold level through the direction and amount of reclassification, while IDI characterizes the overall improvement in discrimination through changes in the separation between predicted probabilities in the event and non-event groups. By mastering the complete workflow—anchor the time point, fit the models separately, calculate predicted probabilities, derive NRI/IDI point estimates, and estimate confidence intervals with bootstrap—and by combining clinically justified threshold settings with an adequate sample size design, researchers can provide a persuasive quantitative answer to whether a new variable is worth adding to the model.