Closing the implementation gap in pre-deployment medical AI study design

Abstract

The rapid development of clinical artificial intelligence, AI, technologies has outpaced the development of robust regulatory and clinical safety mechanisms. AI systems are cleared for use and deployed in practice relying on pre-clinical performance studies, without evidence of the impact this will have on patient and provider outcomes. This has led to concerns of an, implementation gap, where systems that appear to perform well on pre-clinical testing fail to produce the expected outcomes in practice. While there is an urgent need for direct clinical testing of AI systems and evaluation of the impact of these systems on patient and provider outcomes, it is implausible to expect the clinical evaluation will be performed at the scale necessary to mitigate potential AI harms of the many AI systems already in use and currently under development. In this body of work I look at factors which may contribute to the implementation gap, in particular the effects of low-quality training and testing data, flawed and incomplete study design methodologies, and an over-reliance on explainability methods to address safety. I suggest a series of improvements to how we design, evaluate, and utilise AI systems in clinical practice, with the goal of better estimating the potential harms of AI during the pre-clinical testing phase, and by doing so closing the implementation gap.