Prof John Mingers |
The research excellence framework - or REF - has become such
an established part of the higher education landscape that it’s hard to imagine
a world without it. Since 1985 it has provided the navigation points in a
turbulent and changing landscape, marking progress and, like a modern day Lachesis, measuring the threads of individual
research in readiness for Atropos’ shears.
As we gear up for REF2021 and wait to hear from HEFCE about
what form it will take, it seems an appropriate time to take stock and question
whether the game is worth the candle, and if the measurement of research should
be undertaken differently.
This is a perennial question. It maps onto the tide times of
the REF itself. As the REF flows towards high tide, and a disproportionate
amount of academic and administrative time is taken up with judging outputs and
deciding on staff to be submitted, more voices are raised in protest. As it
ebbs and people return to their day jobs, the protest dies down.
Before REF2014 Derek Sayer of Lancaster University questioned
the whole framework of the REF in his book Rank Hypocrisies,
and even went so far as appealing against his inclusion in Lancaster’s
submission. Similarly Dorothy Bishop of Oxford looked at
alternatives, suggesting that the use of bibliometrics, which was rejected
before the last exercise, should be reexamined.
At the heart of their argument is that peer review is not
fit for purpose. The sheer scale of the exercise does not allow for an informed
assessment of outputs. ‘The REF,’ wrote Sayers, ‘is a system in which
overburdened assessors assign vaguely defined grades in fields that are
frequently not their own while (within many panels) ignoring all external
indicators of the academic influence of the publications they are appraising,
then shred all records of their deliberations.’
Whilst many might concur with this, most see no alternative.
And yet peer review is a relative infant in the world of academia. I’ve written before about the surprisingly
short history of this apparent gold standard. It’s current prevalence and
dominance is the result, essentially, of a confluence of the baby boomers
coming of age and the photocopier becoming readily available. As such, the term
‘peer review’ was only coined in 1969.
A couple of weeks ago the Kent Business School convened a
‘debate’ to reopen the wound. ‘The Future of Research Assessment’ heard from
both sceptics and believers, from those who were involved in the last REF and
those who questioned it, and looked again at the potential worth of Dorothy
Bishop’s bibliometric solution.
Prof John Mingers began by laying his cards on
the table. ‘Debate is the cornerstone of academic progress but there is not
enough of it when it comes to peer review and possible alternatives,’ he said.
Taking aim at the REF, he suggested that, whilst it was intended
to evaluate and improve research, it actually has the opposite effect. It
neither effectively evaluates research, and can have ‘disastrous’ effects on
the research culture. For him, the REF peer review was fundamentally subjective
and open to conscious and unconscious biases. The process by which panel
members were appointed was secretive and opaque, and the final membership came
to represent an ‘established pecking order’, and may not have the necessary
expertise to properly evaluate all the areas of submission.
Moreover, even accepting the good faith of the members in
assessing outputs objectively and fairly, the sheer workload (in the order of
600 papers) meant that ‘whatever the rhetoric, the evaluation of an individual
paper was sketchy.’
For Mingers, the exercise wasn’t fit for purpose, and didn’t
justify the huge expense and time involved. Instead, he suggested that a
nuanced analysis of citations offered a simpler, far cheaper and more immediate
solution. ‘After all, citations are ‘peer review by the world’.
He accepted the potential problems inherent in them - such
as disciplinary limitations - but suggested these could be allowed for through
normalisation. In return, they offer an objectivity that peer review can never
hope to achieve, however well intentioned.
Mingers compared the REF rankings to an analysis he had done
using the data underlying Google Scholar. The REF table produced an odd result, with the
Institute of Cancer Research topping the ranking, Cardiff coming seventh, and a
solid research institution such as Brunel sliding far below its natural
position.
Much of this is a result of the game-playing that the Stern
Review sought to remedy. However, whatever solution is finally agreed by HEFCE,
such a peer review based system will always drive behaviour in a negative way.
His alternative used Google Scholar institutional level
data, focussing on the top 50 academics by total citations. He examined a
variety of metrics - mean cites, median cites, and h index - before selecting
median five year cites as the measure which offered the most accurate
reflection of actual research excellence.
The resulting table is a less surprising one than that that
resulted from the REF. Oxford and Cambridge are at the top, followed closely by
Imperial and UCL, with Cardiff down to 34 and Brunel up to 33. Interestingly,
LSE - a social science and humanities institution that hosts disciplines which
don’t traditionally deal in citations, came 12th.
No comments:
Post a Comment