Python Calculate Confidence Interval From Roc Curve

Introduction

Receiver Operating Characteristic (ROC) curves are essential tools in binary classification problems. They visualize the trade-off between true positive rate (sensitivity) and false positive rate (1-specificity) across different classification thresholds. However, a single ROC curve doesn't provide information about the variability or uncertainty in the estimates.

Confidence intervals for ROC curves help quantify this uncertainty. They provide a range of plausible values for the area under the curve (AUC) and other ROC metrics, giving researchers and practitioners a more complete picture of model performance.

Formula

The confidence interval for the AUC can be calculated using the following formula based on the standard normal distribution:

This formula provides a conservative estimate of the confidence interval for the AUC, accounting for the finite sample size and the correlation between the true positive and false positive rates.

Python Implementation

To calculate confidence intervals for ROC curves in Python, you can use the following code snippet:

import numpy as np
from sklearn.metrics import roc_auc_score, roc_curve

def calculate_roc_ci(y_true, y_scores, confidence=0.95):
    """
    Calculate confidence interval for ROC AUC

    Parameters:
    y_true (array): True binary labels
    y_scores (array): Predicted probabilities or scores
    confidence (float): Desired confidence level (default: 0.95)

    Returns:
    tuple: (lower_bound, upper_bound)
    """
    auc = roc_auc_score(y_true, y_scores)
    n1 = sum(y_true)
    n2 = len(y_true) - n1
    q1 = auc / (2 - auc)
    q2 = 2 * auc**2 / (1 + auc)
    se = np.sqrt(auc * (1 - auc) + (n1 - 1) * (q1 - auc**2) + (n2 - 1) * (q2 - auc**2)) / n
    z = np.abs(np.percentile(np.random.normal(0, 1, 10000), (1 - confidence) / 2 * 100))
    ci = (auc - z * se, auc + z * se)
    return ci

This function calculates the confidence interval for the AUC using the formula described above. It takes the true labels, predicted scores, and desired confidence level as inputs and returns the lower and upper bounds of the confidence interval.

Example

Let's consider a binary classification problem with 100 positive cases and 100 negative cases. The AUC for this model is 0.85. Using the formula above, we can calculate the 95% confidence interval for the AUC:

The 95% confidence interval for the AUC in this example is approximately (0.786, 0.914). This means we can be 95% confident that the true AUC lies within this range.

Interpretation

Interpreting confidence intervals for ROC curves involves understanding what the interval represents and how it relates to the model's performance:

• The confidence interval provides a range of plausible values for the AUC, accounting for sampling variability.
• A narrower confidence interval indicates more precise estimates, while a wider interval suggests greater uncertainty.
• If the confidence interval includes 0.5, it suggests the model's performance is not significantly better than random guessing.
• Comparing confidence intervals across different models can help determine which model's performance is more reliable.

In practical terms, if the confidence interval for your model's AUC is (0.80, 0.88), you can be 95% confident that the true AUC falls within this range. This information is valuable when comparing models or deciding whether to deploy a model in a real-world application.