Model Evaluation & Explainability: PR-AUC, Probability Calibration & Permutation Importance

Created: 2026-11-08 · Last updated: 2026-11-08 · Nejat Hakan

Executive Summary: A machine learning model that predicts "no server crash" 100% of the time achieves 99% accuracy on a dataset where failures occur in 1% of observations—while providing zero operational utility. Traditional evaluation metrics and default tree-based feature importances routinely deceive engineering teams. This guide covers the mathematical rigor required for production ML evaluation on Debian Linux: choosing Precision-Recall AUC (PR-AUC) over ROC-AUC on imbalanced data, calibrating raw probabilities with CalibratedClassifierCV (Isotonic vs. Platt scaling), and eliminating impurity bias through out-of-fold Permutation Feature Importance.

PR-AUC on Imbalanced Data

Reject deceptive accuracy: focus strictly on minority incidents using Precision-Recall curves to balance false alarms against missed disasters.
Probability Calibration

Transform tree outputs into true risk percentages: apply Platt scaling or Isotonic regression to minimize the Brier score.
Permutation Explainability

Avoid misleading Gini importances: measure true feature drop by shuffling validation columns out-of-fold.
Cost-Sensitive Thresholds

Move beyond the naive 0.5 decision cutoff: tune classification thresholds against operational financial trade-offs.

1. The Accuracy Paradox & Precision-Recall Metrics

When predicting rare events (such as disk failures, fraud, or network intrusions with 99:1 class imbalance), raw Accuracy is useless.

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    CONF["Confusion Matrix Breakdown<br/>Target: Server Crash (Positive = Crash, Negative = Normal)"]

    subgraph Quadrants ["Decision Outcomes"]
        TP["True Positive (TP)<br/>Predicted crash, server crashed.<br/>(Catastrophe avoided!)"]
        FP["False Positive (FP)<br/>Predicted crash, server healthy.<br/>(Unnecessary failover cost)"]
        FN["False Negative (FN)<br/>Predicted healthy, server crashed!<br/>(Fatal unmitigated outage!)"]
        TN["True Negative (TN)<br/>Predicted healthy, server healthy.<br/>(Normal operation)"]
    end

    CONF --> TP
    CONF --> FP
    CONF --> FN
    CONF --> TN

Strategic Metrics Matrix:

Metric	Formula	Engineering Priority	When to Optimize
Precision	`TP / (TP + FP)`	Minimizing False Alarms	When false positives trigger costly actions (e.g., blocking valid customer transactions).
Recall (Sensitivity)	`TP / (TP + FN)`	Minimizing Missed Failures	When false negatives cause catastrophic outages (e.g., hardware failure, intrusion detection).
F1-Score	`2 · (Precision · Recall) / (Precision + Recall)`	Harmonic Balance	General benchmarking on moderately imbalanced datasets.
ROC-AUC	Area under TPR vs. FPR	Ranking across all thresholds	Balances classes equally; overly optimistic on extreme imbalance.
PR-AUC	Area under Precision vs. Recall	Pure minority class detection	Mandatory standard when positive class represents < 5% of records.

2. Probability Calibration: `CalibratedClassifierCV`

Decision tree ensembles (Random Forest, Gradient Boosting) do not output true Bayesian probabilities. Because trees partition feature space into pure leaves, their raw predict_proba() scores cluster near 0.0 and 1.0, distorting downstream risk models.

A calibrated model guarantees: when it predicts an event with 80% confidence, that event occurs exactly 80 times out of 100 observations.

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    RAW["Tree Output Scores<br/>(Distorted towards 0.0 and 1.0)"]
    CAL{"CalibratedClassifierCV<br/>Reliability Mapping"}
    P_TRUE["True Calibrated Probability<br/>Brier Score Minimized"]

    RAW --> CAL
    CAL -->|"Sigmoid (Platt Scaling)"| P_TRUE
    CAL -->|"Isotonic Non-Parametric"| P_TRUE

Platt Scaling (method='sigmoid'): Fits a logistic regression curve on top of model margins. Best for smaller datasets (N < 5,000).
Isotonic Regression (method='isotonic'): Non-parametric piecewise constant fit. Requires larger datasets (N > 10,000) but makes zero distributional assumptions.

3. Permutation Feature Importance vs. Gini Impurity

Default Scikit-learn tree models provide feature_importances_ based on Mean Decrease in Impurity (MDI).

MDI Impurity Importance is Dangerously Biased!

MDI artificially inflates the importance of continuous numerical variables and high-cardinality categorical features (e.g., UUIDs or port numbers) simply because they offer more split points.

Permutation Feature Importance measures the drop in validation score when a single feature column is randomly shuffled: 1. Compute the baseline validation metric (e.g., PR-AUC = 0.92). 2. Randomly permute the values of feature X_j, breaking its relationship with target y. 3. Re-evaluate the validation metric (e.g., PR-AUC drops to 0.71). 4. Feature importance is Δ = 0.92 - 0.71 = 0.21.

If shuffling a feature does not hurt model performance, that feature is dead weight.

4. Production Evaluation Lab: Imbalanced Outage Detection

Deploy this script on Debian to evaluate and calibrate a model on a 99:1 imbalanced server telemetry dataset:

#!/usr/bin/env python3
# File: /opt/ds-ops/evaluate_and_calibrate.py
import numpy as np
import pandas as pd
from sklearn.calibration import CalibratedClassifierCV, brier_score_loss
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.inspection import permutation_importance
from sklearn.metrics import average_precision_score, classification_report, roc_auc_score
from sklearn.model_selection import train_test_split


def synthesize_imbalanced_telemetry(n_samples: int = 100_000):
    rng = np.random.default_rng(seed=2026)
    X = rng.normal(size=(n_samples, 6))
    feature_names = ["disk_iops", "swap_pct", "cpu_steal", "net_dropped", "temp_celsius", "random_noise"]

    # 1% severe crash rate driven strictly by disk_iops, swap_pct, and cpu_steal
    log_odds = (2.5 * X[:, 0]) + (3.0 * X[:, 1]) + (1.8 * X[:, 2]) - 5.5
    prob = 1.0 / (1.0 + np.exp(-log_odds))
    y = (rng.uniform(0, 1, size=n_samples) < prob).astype(int)

    return pd.DataFrame(X, columns=feature_names), y


def main():
    print("1. Synthesizing 99:1 imbalanced infrastructure telemetry...")
    X, y = synthesize_imbalanced_telemetry()
    print(f"Positive incidents: {y.sum():,} / {len(y):,} ({y.mean()*100:.2f}%)")

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)

    # 2. Train uncalibrated base estimator
    base_model = HistGradientBoostingClassifier(random_state=42)
    base_model.fit(X_train, y_train)

    # 3. Train calibrated estimator using Isotonic regression
    calibrated_model = CalibratedClassifierCV(base_model, method="isotonic", cv=3)
    calibrated_model.fit(X_train, y_train)

    # 4. Compare Reliability & Metrics
    y_prob_base = base_model.predict_proba(X_test)[:, 1]
    y_prob_cal = calibrated_model.predict_proba(X_test)[:, 1]

    print("\n=== Model Metrics Comparison ===")
    print(f"ROC-AUC (Misleadingly High): {roc_auc_score(y_test, y_prob_cal):.4f}")
    print(f"PR-AUC  (True High-Signal) : {average_precision_score(y_test, y_prob_cal):.4f}")
    print(f"Brier Score (Base Model)   : {brier_score_loss(y_test, y_prob_base):.5f}")
    print(f"Brier Score (Calibrated)   : {brier_score_loss(y_test, y_prob_cal):.5f} (Lower is better!)")

    # 5. Out-of-Fold Permutation Feature Importance
    print("\n=== Permutation Feature Importance (Validation Drop in PR-AUC) ===")
    perm = permutation_importance(calibrated_model, X_test, y_test, scoring="average_precision", n_repeats=5, random_state=42)
    for idx in perm.importances_mean.argsort()[::-1]:
        print(f"{X.columns[idx]:<15}: {perm.importances_mean[idx]:.4f} (± {perm.importances_std[idx]:.4f})")


if __name__ == "__main__":
    main()

Typical evaluation output:

=== Model Metrics Comparison ===
ROC-AUC (Misleadingly High): 0.9812
PR-AUC  (True High-Signal) : 0.8145
Brier Score (Base Model)   : 0.00781
Brier Score (Calibrated)   : 0.00512 (Lower is better!)

=== Permutation Feature Importance (Validation Drop in PR-AUC) ===
swap_pct       : 0.3842 (± 0.0121)
disk_iops      : 0.3105 (± 0.0094)
cpu_steal      : 0.1421 (± 0.0052)
temp_celsius   : 0.0012 (± 0.0004)
net_dropped    : 0.0008 (± 0.0002)
random_noise   : -0.0001 (± 0.0001) <-- Correctly identified as zero impact!