Calibrated early-warning models with fairness auditing and selective prediction for course withdrawal risk: Evidence from OULAD

Suhan Wu, Jingyi Duan and Min Luo

PLOS ONE, 2026, vol. 21, issue 7, 1-20

Abstract: Early-warning systems (EWS) in learning analytics are increasingly used to identify learners at risk of course withdrawal, but their deployment-critical properties are often under-reported once predicted scores are converted into intervention policies. This study develops a deployment-oriented evaluation protocol for course-withdrawal risk using the Open University Learning Analytics Dataset (OULAD). An early-window feature set was constructed from the first four weeks of learner activity and evaluated under a group-wise train–test split by course presentation. Multiple classifiers were benchmarked, including logistic regression, histogram-based gradient boosting (HGB), random forest, support vector machine, AdaBoost, K-nearest neighbors, XGBoost, LightGBM, and CatBoost. A calibrated HGB model was then retained as the main probabilistic model for downstream analyses of probability reliability, threshold sensitivity, subgroup fairness with bootstrap uncertainty, selective prediction, and capacity-based Top-x% alerting. Several tree-based and boosting models achieved comparable held-out discrimination, while calibrated HGB remained competitive across classification, ranking, and probability-reliability metrics. Threshold choices substantially changed the precision–recall balance, indicating that operating points should be treated as policy choices rather than universal defaults. Fairness audits showed policy-dependent observed group-level differences, especially in alert rates for disability status, although several subgroup error-rate and positive predictive value (PPV) differences remained uncertain. Selective prediction reduced risk on accepted cases as coverage decreased, whereas Top-x% alerting fixed outreach volume and made workload–effectiveness trade-offs explicit. Robustness analyses supported the 28-day window as a practical early-warning compromise and showed that absolute PPV values varied across held-out course-presentation splits. The findings suggest that EWS should be evaluated as policy-linked decision systems, integrating model benchmarking, calibration, fairness uncertainty, and capacity-aware decision rules before deployment.

Date: 2026
References: Add references at CitEc
Citations:

Downloads: (external link)
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0352867 (text/html)
https://journals.plos.org/plosone/article/file?id= ... 52867&type=printable (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:plo:pone00:0352867

DOI: 10.1371/journal.pone.0352867

Access Statistics for this article

More articles in PLOS ONE from Public Library of Science
Bibliographic data for series maintained by plosone ().