Who should decide? Testing a Human-First Alternative to Recommendation-Centric Medical AI

AI tools hold tremendous potential to transform healthcare. Yet while many models often perform well in controlled benchmarking evaluations, safe and reliable performance in real-world clinical settings remains difficult to achieve. A key reason is the design philosophy: tools generate predictions or recommendations to clinicians, who are then expected to interpret and then act on or override them. However, decades of cognitive science demonstrate that this 'human in the loop' approach is poorly matched to human cognitive strengths and vulnerabilities, including automation bias, anchoring, and resistance to revising initial decisions.

This project tests a human-first approach to healthcare AI design and deployment. Leveraging the HumanAI-team platform developed by CI Jawaid Shaikh, which enables experimental testing of different AI design principles, we aim to generate early empirical data to inform critical decisions as to whether the 'human in the loop' paradigms are sufficient as a risk control in medical AI regulation. We will use using simple and complex presentations of ST-elevation myocardial infarction (STEMI, or heart attack) as a clinically important acute care case study.

Researching team

This project is led by Dr Olivia Metcalf (Centre for Digital Transformation of Health), Dr Sonia Jawaid Shaikh (Faculty of Art), Dr Kit Huckvale, (Centre for Digital Transformation of Health), Prof Peter Steel, (Centre for Digital Transformation of Health; Royal Melbourne Hospital), Dr Teresa O’Brien (Centre for Digital Transformation of Health), Dr Angus McKerral, (Melbourne School of Psychological Sciences), Dr Paul Garrett (Melbourne School of Psychological Sciences).