Difficulty of Achieving High Precision with Low Base Rates for High-Stakes Intervention
Files
Date
2025
Authors
Baker, R.
Mills, C.
Choi, J.
Editors
Advisors
Journal Title
Journal ISSN
Volume Title
Type:
Conference paper
Citation
Fifteenth International Conference On Learning Analytics and Knowledge, Lak 2025, 2025, pp.790-796
Statement of Responsibility
Conference Name
15th International Conference on Learning Analytics and Knowledge : Dublin, Ireland)
Abstract
Automated detectors are routinely used in learning analytics for high-stakes, high-risk interventions. Such interventions depend on detectors with a low rate of false positives (i.e., predicting the construct is present when it is not present) in order to avoid giving an intervention where it is not needed, especially when such interventions can be costly or even harmful. This in turn suggests that such a detector needs to have high precision at the cut-off used by the detector for decision-making. However, high precision is difficult to achieve for the common case where the base rate of the target construct is low. In this paper, we demonstrate the difficulty of achieving high precision for low base rates, and demonstrate how other metrics (such as F1, Kappa, Specificity, and AUC ROC) are insufficient for this specific use case and situation, despite their merits and advantages for other use cases and situations.
School/Discipline
Dissertation Note
Provenance
Description
Access Status
Rights
Copyright 2025 The author(s) (https://creativecommons.org/licenses/by/4.0/)
Access Condition Notes: This is an open access article