Mitigating the Crumple Zone: Developing Ethical AI to Support Mental Health Discussion Forums


Researchers in the field of Natural Language Processing have started to recognise the ease with which data and the application of data for algorithmic purposes can be used and misused, potentially leading to harm. The field has begun to identify the need to take responsibility and mitigate harm and misuse through documentation for their AI output, such as through "Model Cards" (Mitchell et al., 2019) and "Data Statements" (Bender et al., 2018). We request funds to undertake the early stages ofco-design of an AI-powered search and moderation support for an online mental health forum. The forum is managed by SANE Australia, a mental health charity, who we will work with. We will use the findings from this case study to evaluate and enhance existing models of delivering systems in ethical AI. We propose that through thorough documentation of data provenance, intent of purpose and limitations of models, as well as a well-defined plan of deployment, we can mitigate the potential casualties of the "Crumple Zone"(Elish, 2019).

During our case study, we will assess every step of the AI development process from data creation to delivery to advocate for best practice in AI deployment. The study involves co-design with mental health professionals as well as people with lived experience of mental illness, followed by an evaluation of the technology design and of our design process. In this way data annotation will be designed in conjunction with professionals in the field as well as those it serves.

Model Cards aim to reduce potential harm by stating the limitations and intended use of a ready-built model and Data Statements aim to identify potential biases in the data by documenting the provenance of an annotated corpus. Through this proof-of-concept, we aim to learn if Model Cards and Data Statements are sufficient or lack information; and whether there should be an additional requirement to document the delivery of an AI model for real-world deployment.

Research Team

  • Dr Mel Mistica
    Dr Mel Mistica

    Research Data Specialist

    Melbourne Data Analytics Platform

    University of Melbourne

  • Dr Greg Wadley
    Dr Greg Wadley

    Senior Lecturer

    School of Computing and Information Systems

    University of Melbourne

  • Professor Nicola Reavley
    Professor Nicola Reavley

    Principal Research Fellow, Mental Health Literacy Program

    School of Population and Global Health

    University of Melbourne

SANE Logo