The rapid integration of Artificial Intelligence (AI) in medical devices coupled with a vacuum in risk guidance on how we evaluate these systems for use-related risk, raises the question: For medical devices with autonomous elements (including AI), do the current risk practices enable the evaluation use-related risk? If not, what do we need to consider.

This talk will present the work of a cross-disciplinary team who have been exploring this question over the last year. The effort began with a review of concepts and literature from the Massachusetts Institute of Technology (MIT) AI Risk Framework. From the articles identified, HFE themes and key topics were identified as they relate to medical devices design for various levels of autonomy.

We will bring these findings forward and discuss considerations for products that are designed to be Human-in-the-Loop (HITL), Human-on-the Loop (HOTL), or Human-out-of-the-Loop (HOOTL). 

Additionally, to explore this and present a framework for discussion, a review of cleared/approved products within the AI Medical Device Taxonomy2, is used to identify examples for case study. In these case studies, examples are built out using common tools of practice from HFE and Risk Management, specifically, examples will be presented from the following tools/methods:   

• Use scenario identification  

• 'Characteristics for Safety' questions (ref. ISO 14971) 

• Task Analysis (TA)/Perception-Cognition-Action (PCA)/ Use-Related Risk Assessment (URRA)  

Attendees will learn the following: 

• Tools to identify use-related risk considerations unique to AI-enabled medical devices across Human-in-the-Loop (HITL), Human-on-the-Loop (HOTL), and Human-out-of-the-Loop (HOOTL) configurations 

• Examples to apply risk assessment methodologies addressing human-AI interaction risks including over-reliance, loss of situational awareness, and functional allocation, utilizing ISO 14971 approaches and HFE methods