The algorithms that decide who gets a credit card, a mortgage, or a business loan are among the most consequential machine learning systems in the world, steering portfolios worth trillions of dollars in exposure. Yet according to a new study published in Discover Informatics, the way researchers evaluate these models is dangerously out of step with the way regulators judge them. A team led by Ravindra Reddy Mallireddy and Raza Hasan of Southampton Solent University, together with Salman Mahmood of Nazeer Hussain University, has built and demonstrated a multi-metric evaluation framework that subjects five popular machine learning models to the kind of scrutiny that Basel III and IFRS 9 actually demand, and the results upend the conventional leaderboard.
The core problem is a metric called AUC, the area under the ROC curve, which measures how well a model separates defaulters from non-defaulters. It dominates the academic credit scoring literature, but it captures only one dimension of model quality. A systematic review cited by the authors covering one thousand sources confirms that discrimination-focused metrics such as AUC, accuracy, and F1-score remain systematically disconnected from the multi-dimensional requirements of regulatory validation. The disconnect has real consequences: a classifier that always predicts non-default achieves 77.9 percent accuracy on the benchmark dataset used in the study while providing zero information about actual defaults. Fewer than 5 percent of published credit scoring studies include any form of stress testing or calibration assessment, meaning most models in the literature have never been examined the way a bank supervisor would examine them.
The regulatory stakes are defined by two post-crisis frameworks. Basel III strengthened the Internal Ratings-Based approach with explicit validation expectations, including minimum discrimination targets of AUC above 0.70 and a Kolmogorov-Smirnov statistic above 0.20, plus continuous Population Stability Index monitoring with action thresholds at 0.10 and 0.25. IFRS 9, effective since 2018, requires forward-looking expected credit loss provisioning through a three-stage impairment model that demands scenario-conditioned probability of default estimates. Together, these frameworks insist that a credit model not merely rank-order risk but also produce well-calibrated probabilities, remain stable as borrower populations shift, and survive economic adversity. The authors translated these principles into a six-metric validation suite combining AUC, Gini, the Kolmogorov-Smirnov statistic, the Hosmer-Lemeshow goodness-of-fit test, the Brier Score, and the Population Stability Index.
Five architectures faced the gauntlet: a logistic regression scorecard built on weight-of-evidence encoded features, a random forest, XGBoost, LightGBM, and a multi-layer perceptron. All were trained on the UCI Credit Card Default dataset, a Taiwanese credit card portfolio with 30,000 accounts and a 22.1 percent default rate, split stratified into training, validation, and test sets with all feature engineering performed only on training data to prevent leakage. The engineered features captured delinquency severity, credit utilization, repayment behavior, and temporal trends, with a composite DELAY_RISK variable resolving severe multicollinearity among payment delay predictors that individually carried extraordinarily high information values of 0.807 to 0.928.
The multi-metric suite immediately exposed failures that a single-metric evaluation would have missed. The multi-layer perceptron, despite its reputation for expressive power, fell below the minimum discrimination benchmark with an AUC of 0.6964, under the 0.70 threshold. XGBoost, one of the most celebrated gradient boosting libraries in machine learning, failed calibration goodness-of-fit with a Hosmer-Lemeshow p-value of 0.0146, well below the 0.05 acceptance level, meaning its predicted default probabilities did not match observed default frequencies. Only the humble weight-of-evidence logistic regression and the random forest passed every baseline threshold. The logistic champion achieved an AUC of 0.7392, a Gini coefficient of 0.4785, and a Population Stability Index of just 0.0005, placing it in the most stable tier with an enormous buffer before any supervisory investigation would be triggered.
The most striking findings emerged when the models were pushed beyond benign conditions. The team designed five ordered stress scenarios, from Minimal to Extreme, that translate macroeconomic distress into borrower-level behavioral shocks: rising credit utilization, lengthening repayment delays, and shrinking payment ratios. Under severe stress, the model rankings inverted dramatically. The random forest, which held the highest static AUC of 0.7649, suffered a 15.4 percent AUC collapse, falling to 0.6468 and below the regulatory threshold. LightGBM dropped 12.8 percent to 0.6473, also failing. The logistic regression champion, by contrast, lost only 5.3 percent and maintained an AUC of 0.7003, staying above the line. A monotonic five-level sensitivity sweep confirmed this pattern was not an artifact of a single parameter choice but a consistent property of the classifiers’ extrapolation behavior.
The authors offer a technical explanation rooted in model geometry. AUC is driven by the separation gap between predicted default probabilities for defaulters and non-defaulters, and stress compresses that gap for everyone. But the contraction is far worse for tree-based models, whose piecewise-constant decision boundaries are highly sensitive to distributional shifts. Under severe stress, the random forest’s separation gap shrank by 48 percent, while the linear model’s shrank by only 21 percent, because a linear decision boundary extrapolates smoothly beyond the training domain. Mean portfolio probability of default escalated 67.2 percent under severe stress, and the authors caution, citing prior evidence that parameter instability can inflate required capital by over 170 percent, that the observed collapse is likely a conservative lower bound.
The framework extends into portfolio-level tail risk through a Monte Carlo simulation of 2,000 correlated loss paths, drawing systematic shocks to utilization and payment delays from a bivariate normal distribution with a correlation of 0.60, reflecting how credit lines are drawn down while repayment capacity collapses during systemic events. The resulting conditional value-at-risk at the 99 percent confidence level came to 9.67 percent, roughly 16.5 percent above the central expected loss estimate, and sensitivity analysis varying the correlation parameter from 0.0 to 0.9 showed the tail-risk estimates were robust to that assumption. A stylized IFRS 9 staging proxy showed expected credit loss rising from 26.68 percent at baseline to 30.81 percent under severe stress, with 96.7 percent of the portfolio migrating into lifetime-loss stages under mild stress, illustrating how rapidly provisioning requirements expand when borrower risk shifts.
Calibration methodology itself proved consequential. Comparing isotonic regression against Platt scaling on the moderate validation sample, the authors found that the non-parametric isotonic approach overfit: for the champion model it yielded a failing Hosmer-Lemeshow p-value of 0.0040 out of sample, while the parametric logistic mapping passed at 0.0743. Brier Scores were nearly identical between methods, showing the difference lies in generalization rather than raw accuracy. The team also verified robustness across 50 random train-test splits, in which the random forest ranked first every time, and through hyperparameter tuning that left the champion selection unchanged. Their choice of logistic regression over the statistically superior random forest, a difference significant by DeLong test but equal to only about seven to eight additional correct pairwise rankings among 4,500 borrowers, rested on five criteria: the negligible AUC gap, inherent interpretability without fragile post-hoc explanations, decisive stress robustness, population stability 23.6 times better than the forest, and a clean sweep of backtesting checks.
The authors are careful about scope. The benchmark dataset comes from a single Taiwanese portfolio observed in 2005, lacks macroeconomic variables and longitudinal structure, and its 22.1 percent default rate far exceeds typical consumer portfolios, so the exercise is framed as a demonstration of the evaluation framework rather than a full regulatory validation. Still, the central lesson travels well beyond one dataset: model selection based solely on discrimination metrics can overlook critical robustness and governance considerations, and the flashiest algorithms on a static leaderboard may be the first to fail when the economy turns. As machine learning spreads deeper into financial infrastructure, the study argues, the true test of a credit model is not how well it ranks borrowers on a sunny day, but whether it keeps its judgment when the storm arrives.
Subject of Research: A multi-metric evaluation framework for machine learning credit scoring models aligned with Basel III and IFRS 9 regulatory validation requirements
Article Title: A multi-metric evaluation framework for machine learning credit scoring models aligned with Basel III and IFRS 9 regulatory validation requirements
Article References: Mallireddy, R. R., Hasan, R., & Mahmood, S. (2026). A multi-metric evaluation framework for machine learning credit scoring models aligned with Basel III and IFRS 9 regulatory validation requirements. Discover Informatics, 1(1), Article 18. https://doi.org/10.1007/s44564-026-00017-y
Image Credits: AI Generated
DOI: 10.1007/s44564-026-00017-y
Keywords: credit scoring, machine learning, Basel III, IFRS 9, stress testing, model validation, expected credit loss, Monte Carlo simulation, probability of default, calibration, population stability index, logistic regression
Cite Scienmag News
APA
MLA
Chicago
Copy citation
Download RIS
Tags: AI credit scoring model vulnerabilitiesBasel IIIBasel III and IFRS 9 compliance in AI modelscalibrationconsequences of inadequate model validation in lendingcredit scoringenhancing transparency and reliability in AIexpected credit lossIFRS 9impact of weak AI credit models on financial portfoliosimportance of comprehensive model performance metricslimitations of AUC and ROC curve in credit risk modelinglogistic regressionMachine learningmodel robustness in credit decision systemsmodel validationMonte Carlo simulationmulti-metric evaluation framework for machine learning modelspopulation stability indexprobability of defaultregulatory validation of credit scoring algorithmsstress testingstress testing in financial risk assessmentsystematic review of credit scoring evaluation methods
