Schrodinger's REF

Schrodinger's REF: Portable and Non-Portable

Yesterday David Sweeney unveiled his initial plans for REF2021. ‘We’ve responded fully to the recommendations of the Stern Review, together with the comments and feedback we received in the consultation that followed,' he said.

‘Although they were diametrically opposed to each other, I’m delighted to say we’ve managed to create a hybrid framework for the next exercise which manages to accommodate all views.  

‘As a starting point we used the excellent 4* work of Erwin Schrödinger. To meet the conflicting views of all stakeholders, we have devised a system whereby everyone will be submitted, but at the same time there’s the option whereby no one will be. The system will be completely inclusive but absolutely exclusive. It will be a 100% submission and a 0% submission. I believe these two states of being can fully coexist.

Tracking the Impact Beast

Impact, that mythical beast

When impact first burst onto the scene seven years ago it was so radical an idea, so controversial a concept, that we used to run ‘impact surgeries’, to which academics could bring their research ideas and where we would suggest possible ways in which their work could have impact. We were no impact experts, but had the objectivity and distance to see beyond the immediate research to its potential effect on the wider world.

The Difficulty of Impact

Definitely not David Sweeney

The LSE'Future of Impact' conference yesterday had a valedictory feel. It was the last huzzah for ‘The Impact of Social Sciences’, a HEFCE-funded project which had sought to 'develop precise methods for measuring and evaluating the impact of research in the public sphere.' 

Whilst they have achieved a lot – their blog is a fantastic forum for discussing issues relating to impact, for example – I think even their fiercest admirers would be hard pressed to say they had succeeded in this.

That’s not necessary their fault. Rather, it's the inevitable consquence of grapling with the amorphous, shape-shifting beast that is impact. This was particularly apparent in the third session yesterday, ‘Next Steps in Assessing Impact,’ which saw the three speakers almost come to blows over what impact is, who wants it, why, and how fast. This was partly down to the ever-entertaining David Sweeney, the architect of the REF. Like an embarrassing uncle at a wedding, he can always be relied upon to speak his mind. Loudly.

He took issue with the previous speaker, Julia Lane, who had been talking about StarMetrics. I’ve spoken about this before on this blog. To me, it sounds like an eminently sensible system (although I don’t think she did it justice here); Sweeney, however, begged to differ. Not only did he think that academia should not snap to attention when governments ask for data, but he implied that the system was only partially successful at collecting the right information. 

Similarly, he turned on the third speaker, Cameron Neylon. Neylon had suggested that Twitter could be used to monitor and engage with users of academic research. He gave an example of South African research which had been retweeted by someone working in community health promotion. ‘That’s not impact’, said Sweeney, dismissively.

And therein lies the problem. Here were three eminent speakers working at the coal face of impact. And yet, between them, they couldn’t reach agreement on what constituted impact, or why we should be doing it. If they have problems defining impact, what hope is there for the rest of us?  

REF: the Plot Thickens

With the publication of the panel assessment criteria last week, we now have a clearer idea of what the REF is going to look like. And it's not a level playing field for all the panels. Research Professional did a good run down of how it differs between each, and here it is, in summary:

The Thorny Issue of Citations

Remember when the REF was just a glimmer in David Sweeney's eye? When all the talk was of bibliometrics and light touch review? Hah! Well, as we've known for some time, you can forget all that. The REF looks pretty much like it's the RAE with Impact, and is peer review-centric. In fact, citations are only going to be allowed in a minority of the 36 sub-panels, as follows:

• Panel A (Life Sciences): will allow citation data in all sub-panels;
• Panel B (Physical Sciences): will only allow citation data in sub-panels 7 (earth and environmental sciences), 8 (chemistry), 9 (physical sciences) and 11 (computer science)
• Panel C (Social Sciences): will only allow citation data in (some of) sub-panels 17 (geography, environmental studies and archaeology) and 18 (economics and econometrics). No panels will use journal impact factors.
• Panel D (Humanities): no citation data allowed.

Put Out by Outputs

As well as variation on citations, there's a wide variety on what's deemed acceptable as assessable outputs. 'RePro' (as no-one but me calls Research Professional) gives the example of the physical sciences vs life sciences. Life sciences are strictly 'old skool', and you can only include 'edgy' outputs like textbooks, databases or abstracts “exceptionally”, 'where they embody original research'. The physical sciences, on the other hand, are much more 'new wave', and you can submit patents, book chapters, computer algorithms and software as evidence of research output, alongside peer-reviewed publications.

The Impact of Impact

Finally, it's our old friend Impact. Generally there's a broadbrush consensus on what impact is all about, and the definition is wide enough to allow for a fairly catholic understanding of it. However, there is a minor spat in the offing when it comes to the impact of teaching. Within Panel B, research impact can include actions that have an effect on teaching or students where they extend significantly beyond your institution. However Panel C will not accept such heresy, and you can't include examples of this in your submission to them.

So, interesting times ahead. Thanks to - ahem - 'RePro' for their overview of the HEFCE docs. You can have your say on them by responding to HEFCE's consultation before 5 October.

Impact Hostility Might Be 'Melting Away,' but REF Still Delayed

Two interesting news items this week: first, David Sweeney, Demon Barber of HEFCE, has been quoted as saying that 'consensus is breaking out around the sector' about the worth of including Impact in the REF. Sweeney was speaking at an 'Impact in the Context of the REF' conference and stated that he believed that the Impact pilot exercise had gone 'really well.' This might be overstating the case: I've listened to some involved in it, and the message from them was luke warm at best, but generally thankful that they'd been involved to understand the issues involved.

The second piece of news is that Two Brains Willetts, in his speech on 9 July, said that he 'appreciates why scientists are wary' of impact, going on to announce a delay of a year to the REF. The delay would mean submissions in 2013, with the results in 2014, to inform funding from 2015.

Wariness or consensus? Or consensus around wariness? You decide.

Academics Welcome Impact Shock!

A group of around 50 academics have given a surprise endorsement to the REF Impact plans. At a HEFCE Consultation event on the REF proposals in London yesterday feedback from the 'breakout' groups suggested that academics now accept the need to justify the impact of their research to the government. David Sweeney, who opened the event, highlighted the massive increase in QR funding since 2002/03, up some 41%.
Whilst academic freedom was ‘a gift’, one that should be nurtured and cherished, and institutional autonomy was important, universities ‘shouldn’t be monasteries.’

He went on to explain that HEFCE was now fully behind peer review, a claim that couldn’t have been made a few years ago, and the battle they had had in establishing this should not be taken for granted.
HEFCE was aware of the ‘challenges to assessment’, but he believed that many of these would ‘meltaway’ when faced by ‘very bright people’ tackling them. He emphasised that the REF was not about quantifying impact, focussing on economic impact, assessing impact of every researcher or output,trying to predict future impact, discouraging curiosity-driven research, or trading off impact against excellence. 'High impact that results from poor research is not a feature of this exercise,’ he said.

Full notes on the event are available if you contact me.