University signs up to international agreement for best practice in research assessment

DORA encourages a meaningful and comprehensive assessment of research quality.

The University of Melbourne has become the first Australian university to sign up to the San Francisco Declaration on Research Assessment (DORA), a set of recommendations that encourage a meaningful and comprehensive assessment of research quality.

This contrasts with current practice, where research quality is sometimes mistakenly assessed with too much emphasis on numerical measure, such as the number of journal articles published, where they are published and their citations.

By adopting DORA, the University aims to drive research excellence by improving how decisions are made around research investments, including the the allocation of public resources, promotions and recruitment.

DORA recommendations also include using a broader definition of research impact to capture work that that leads to change in government policy, guides clinical practice, the development of new software, the value of interdisciplinary research or the production of datasets for other researchers.

The University of Melbourne joins other DORA signatories including the Australian Academy of Science, the Australian National Health and Medical Research Council (NHMRC), University College London (UCL), the University of Oxford, the University of Cambridge and the University of Edinburgh.

Professor Jim McCluskey, Deputy Vice-Chancellor (Research) at the University of Melbourne said that there was a pressing need to improve how research output is evaluated by funding agencies, academic institutions, and other parties.

“Numerical research metrics have gained popularity because they provide an easy-to-use measure, however they don’t adequately assess quality,” Professor McCluskey said.

“Methods such as ‘journal impact’ factors, which equate publishing work in particular journals with research quality, disadvantage certain types of research or researcher, and may particularly affect younger researchers who are establishing their careers.”

“By adopting DORA, we aim to improve how we measure research quality, which in turn will improve how we make decisions about funding, recruitment and promotion to continuously improve our research capabilities.”

The University will continue to use numerical research metrics as a tool to inform assessment, where appropriate, but not as the primary or sole determinant of outcomes such as promotion and funding opportunities.

Professor Justin Zobel, Pro Vice-Chancellor, Graduate & International Research, Chancellery (Research and Enterprise) will lead the University’s activities for establishing appropriate use of research metrics, with initial recommendations being circulated in coming weeks.

DORA was established by a group of editors and publishers of scholarly journals during the Annual Meeting of The American Society for Cell Biology (ASCB) in San Francisco, CA, on December 16, 2012.

The group were among many concerned with a growing trend towards reliance on quantitative over qualitative methods of research assessment. They were also concerned about the misrepresentation of research impact and contributions by different disciplines that can arise from a focus on journal impact factors.

For example, the top-ranked journals in mathematics have an impact factor of around 3, whereas the top-ranked journals in cell biology are far higher at around 30.

As a response they developed a set of recommendations, referred to as the San Francisco Declaration on Research Assessment (DORA). The Leiden Manifesto was subsequently developed as a set of ten principles to guide research evaluation.