Major new grant to investigate credibility of social research claims

Image of research definition
A major international research program seeks to develop Artificial Intelligence to help evaluate the credibility of scientific evidence that we use to make decisions.

In response to the 'replication crisis' in a number of scientific fields, a major international research program seeks to develop Artificial Intelligence to help evaluate the credibility of scientific evidence we use to make decisions.

The University of Melbourne is currently the only Australian team selected by the US Government’s Defense Advanced Research Project Agency (DARPA) to work on the “Systematizing Confidence in Open Research and Evidence” or SCORE program.

With up to $US6.5 million in funding, the University’s team, Collaborative Assessment for Trustworthy Science or the repliCATS project, will assess the replicability of thousands of social and behavioural research claims.

This work will inform the AI component of SCORE. It will also help us understand how to recognise credible research.

The University’s team plans to crowdsource thousands of experts – working in psychology, sociology, criminology, economics, business, marketing, political science and education – to meet the SCORE challenge.

Associate Professor Fiona Fidler who is a reproducibility and Open Science expert will lead the University of Melbourne team. She said: “This is by far the most ambitious reproducibility project the social and behavioural sciences have seen.

“It will be a defining moment in how we understand the evidence base in the published literature of those fields.”

Describing her team’s approach, Associate Professor Fidler said experts would work in small groups, locally and virtually, to evaluate published research claims using a proven structured group deliberation approach known as the Investigate, Discuss, Estimate, Aggregate (IDEA) protocol.

“The IDEA protocol is designed to take advantage of diverse points of view, and minimise overconfidence, rather than force premature consensus. We are also interested in the information researchers rely on to make these predictions, and how they collectively reason about the various forms of uncertainty when making these judgements,” Associate Professor Fiona Fidler said.

Acting Deputy Vice-Chancellor (Research) Mark Hargreaves commended the research team on their funding success.

“We know that interdisciplinary research can be challenging, however the benefits of drawing upon expertise across our many disciplines are often research outcomes with considerable impact.  I look forward to seeing the insights provided by this exciting and interesting research project,” Professor Hargreaves said.

The repliCATS team will shortly begin collecting expressions of interest from individuals and groups interested in collaborating on this project.

The repliCATS project is an interdisciplinary research collaboration between the faculties of Science, Arts and the Melbourne School of Engineering.