The Open Science Atlas: Automating measurement of research transparency and rigour

High-quality scientific evidence is needed to confront global climate, health, social, and political crises.

The field of meta-research (research-on-research) has publicised deep problems with the reliability of research across the life and social sciences. In response, there are increasing expectations for research to meet higher standards. For example, more than 5,000 signatories of the Transparency and Openness Promotion (TOP) guidelines, including major publishers like Springer, Nature and Elsevier, have advocated for policies that expand the use of transparent and rigorous research practices, like data sharing and replication.

Further meta-research is needed to measure the impact of these reforms so that unintended consequences can be minimised and benefits can be maximised. However, currently meta-research is impeded by the laborious manual extraction and classification of information from research articles by human coders.

This project will combine MetaMelb’s expertise in meta-research with MDAP’s expertise in automation and natural language processing (NLP). Together, we will explore the potential of automated tools, like machine learning and large language models, to break the manual labour bottleneck and expand the capacity of meta-research to evaluate transparency and rigour across the scientific literature.

We will build an online platform – the Open Science Atlas – which combines crowdsourcing and automation tools to measure indicators of transparency and rigour at scale. This will generate ‘living maps’ of the research landscape at various levels of granularity, for example field-wide, sub-fields, specific journals, universities, funders, helping us gauge the effectiveness of reforms and enhance their benefit.

Automation has high potential to expand the capacity of meta-research, which will in turn raise research standards by maximising the benefit of scientific reform initiatives.

The project outcomes will also have a direct impact in policy settings. Several major organisations, including UNESCO, the Center for Open Science, the United Kingdom Reproducibility Network (UKRN), and the USA National Academies Strategic Council (NASC) have all signalled a need to monitor transparency and rigour at scale. The Open Science Atlas will fulfil this need.