'Citations Are Peer Review by the World'

Prof John Mingers

The research excellence framework - or REF - has become such an established part of the higher education landscape that it’s hard to imagine a world without it. Since 1985 it has provided the navigation points in a turbulent and changing landscape, marking progress and, like a modern day Lachesis, measuring the threads of individual research in readiness for Atropos’ shears.

As we gear up for REF2021 and wait to hear from HEFCE about what form it will take, it seems an appropriate time to take stock and question whether the game is worth the candle, and if the measurement of research should be undertaken differently.

The Danger of Management by Metrics

The Metric-Measured Man

There was depressing news from Kings earlier this month. The University plans to cut 120 academic posts in the Schools of Medicine and Biomedicine (from a total of 777)  in order to fund new building. Chillingly, the redundancies will be decided by two metrics: research income and teaching hours.

Using metrics is somewhat fraught at the best of times, but is positively toxic when jobs depend on it. The idea of having stated targets is something that all universities have toyed with and many have implemented. There are clear examples where these targets have had a real effect, focussing minds and ratcheting up activity. But at what cost? Anecdotally the morale in these metric-driven universities makes Airstrip One look positively Arcadian.

The Trouble with Citations

Gender and Citation: Nature breaks down the figures

For the past two years the Times Higher (THE) has published a league table of the top 100 universities under 50. This is a spin off of its more established World University Rankings. Whilst I accept that we live in a league table world, and have joked about this in the past on this blog ('Almost Everyone Can Score Well at Something'), I do have a niggling worry about this latest manifestation of the genre.

Lies, Damn Lies and Citations

Trying to assess research excellence using bibliometrics was considered, and then ditched, by HEFCE when it was developing the REF. However, citation-based metrics are still in the news. Using such a system, the EPSRC Citations Study 2009 has shown that there is 'no evidence' that blue skies research produces more highly cited papers than targeted funding. This is surprising, as the assumption is that 'blue skies' research is inherently more excellent, more world-leading and ground-breaking, than that which responds to specific calls.
Elsewhere the Science and Technology Policy Research unit at Sussex is attempting to pin down the humanities using citations, in a way that is currently used in the sciences. Although more robust than the systems currently used, it's had a sniffy reception by the sector. 'No one in the humanities agrees we need bibliometrics,' said Judi Loach of Cardiff. 'We don't think it is appropriate to our disciplines.'