Goldilocks Grants

The first, second and third reviewers disagree about the merits
of Goldilock's research methodology

Earlier this year Michael Lauer, the National Institutes of Health’s (NIH) Deputy Director for Extramural Research, wrote an interesting blog post examining the productivity of its funding. He looked for a correlation between the amount of funding a project had received and the number of citations it got. This he described as “citations per dollar”.

Such a stark metric had many rolling their eyes, and the comments that followed questioned the underlying supposition. “Numbers of citations rarely correlate with greatest discoveries”, wrote one commentator. “Citation numbers are strongly biased towards fast-moving areas of inquiry”, said another. A third noted that “the number of citations a paper receives is an extremely error-prone measure of scientific merit.”

Nevertheless, I think it’s worth exploring Lauer’s work further for two reasons. First, because it is entirely appropriate for funders to try to assess the most effective use of their limited resources. Second, because the conclusion he hints at runs counter to the current drive by the majority of funders to back larger and longer projects.

Fundermental Top 10 of 2014

Last year I compiled a list of the top ten posts on Fundermentals, based on page views. As we inch towards Christmas, and before the deluge of the REF, I reckon it's time to do the same for 2014.

The Trouble with Citations

Gender and Citation: Nature breaks down the figures

For the past two years the Times Higher (THE) has published a league table of the top 100 universities under 50. This is a spin off of its more established World University Rankings. Whilst I accept that we live in a league table world, and have joked about this in the past on this blog ('Almost Everyone Can Score Well at Something'), I do have a niggling worry about this latest manifestation of the genre.

REF: the Plot Thickens

With the publication of the panel assessment criteria last week, we now have a clearer idea of what the REF is going to look like. And it's not a level playing field for all the panels. Research Professional did a good run down of how it differs between each, and here it is, in summary:

The Thorny Issue of Citations

Remember when the REF was just a glimmer in David Sweeney's eye? When all the talk was of bibliometrics and light touch review? Hah! Well, as we've known for some time, you can forget all that. The REF looks pretty much like it's the RAE with Impact, and is peer review-centric. In fact, citations are only going to be allowed in a minority of the 36 sub-panels, as follows:

• Panel A (Life Sciences): will allow citation data in all sub-panels;
• Panel B (Physical Sciences): will only allow citation data in sub-panels 7 (earth and environmental sciences), 8 (chemistry), 9 (physical sciences) and 11 (computer science)
• Panel C (Social Sciences): will only allow citation data in (some of) sub-panels 17 (geography, environmental studies and archaeology) and 18 (economics and econometrics). No panels will use journal impact factors.
• Panel D (Humanities): no citation data allowed.

Put Out by Outputs

As well as variation on citations, there's a wide variety on what's deemed acceptable as assessable outputs. 'RePro' (as no-one but me calls Research Professional) gives the example of the physical sciences vs life sciences. Life sciences are strictly 'old skool', and you can only include 'edgy' outputs like textbooks, databases or abstracts “exceptionally”, 'where they embody original research'. The physical sciences, on the other hand, are much more 'new wave', and you can submit patents, book chapters, computer algorithms and software as evidence of research output, alongside peer-reviewed publications.

The Impact of Impact

Finally, it's our old friend Impact. Generally there's a broadbrush consensus on what impact is all about, and the definition is wide enough to allow for a fairly catholic understanding of it. However, there is a minor spat in the offing when it comes to the impact of teaching. Within Panel B, research impact can include actions that have an effect on teaching or students where they extend significantly beyond your institution. However Panel C will not accept such heresy, and you can't include examples of this in your submission to them.

So, interesting times ahead. Thanks to - ahem - 'RePro' for their overview of the HEFCE docs. You can have your say on them by responding to HEFCE's consultation before 5 October.

Lies, Damn Lies and Citations

Trying to assess research excellence using bibliometrics was considered, and then ditched, by HEFCE when it was developing the REF. However, citation-based metrics are still in the news. Using such a system, the EPSRC Citations Study 2009 has shown that there is 'no evidence' that blue skies research produces more highly cited papers than targeted funding. This is surprising, as the assumption is that 'blue skies' research is inherently more excellent, more world-leading and ground-breaking, than that which responds to specific calls.
Elsewhere the Science and Technology Policy Research unit at Sussex is attempting to pin down the humanities using citations, in a way that is currently used in the sciences. Although more robust than the systems currently used, it's had a sniffy reception by the sector. 'No one in the humanities agrees we need bibliometrics,' said Judi Loach of Cardiff. 'We don't think it is appropriate to our disciplines.'