Success Rates and Scratch Cards

Revealed: the design of the new Je-S form

At the end of last month I wrote a piece on my blog about the success rate of applications submitted to the Economic and Social Research Council in July. Of 144 applications submitted, only 14 awards were made, giving a success rate of 10 per cent.

This is a frightening figure. It suggests that we’re getting to lottery levels of success when applying to the research councils. In fact, the ESRC odds are half those you have on a Health Lottery scratch card, which suggests most universities have got their research funding strategies all wrong.

What made the ESRC data more frightening still was the number of high quality applications that were turned down. The ESRC uses a one-to-ten scale for grading its applications. The cut off line for funding in July fell towards the top of those applications that had scored eight.

Thus, according to the ESRC’s own definitions, it had had to turn away more than half of the excellent proposals which are of significant value, and are “highly likely to make a very important scientific contribution and/or will significantly enhance the development of the applicant's academic career”.

Clearly, this situation is untenable, and doesn’t work well for either the applicant or funder. The applicant is told that, despite the significance and value of the application, it can neither be funded nor resubmitted; meanwhile the funder has had to waste its resources on processing a large number of applications, which realistically have no chance of success.

We need a wholesale rethink of peer review, as I’ve discussed before in this column. However, the funders clearly don’t have the stomach or the political appetite for that yet. Instead, they will continue to use the broken system, but will throw the responsibility for moderating the flow of applications back on to the applicants and their institutions.

Last week the Natural Environment Research Council became the latest research council to introduce ‘demand management’ measures. It announced that, if any institution had a less than 20 per cent success rate in the previous six grant rounds, it would be limited to submitting one application in the next round.

This isn’t a solution: it merely pushes the problem further back down the development pipeline. My blog post last week ignited an interesting Twitter debate, storified here by Paul Benneworth of the University of Twente. The participants raised a number of issues with demand management. Manchester’s Kieron Flanagan cut to the chase by suggesting that, “If peer review [is] supposed to be the gold standard, why swap a large pool of reviewers for a tiny one, full of internal politics?”

This point was fleshed out by Benneworth on his blog. “Could universities, as private institutions with their own politics, really be trusted to make a distinction between holding back a superstar Prof’s very good proposal to allow a post-doc’s excellent one to go in?...Why ask researchers to spend their costly time to run their own institutionally based research councils?

“If the effect is as desired, driving a further upward shift in proposal quality, demand management is effectively adding to the lotterisation of the system: you buy a ticket with an excellent proposal and hope the stars shine on you.  And ultimately, if you believe universities can be trusted [to] identify the best research proposals (and that’s by no means uncontroversial), then why have research councils at all?”

Why indeed? Interestingly, the Terms of Reference for the Nurse Review suggest that it will be looking more at whether the disciplinary division between the research councils (RCUK) is fit for purpose, and whether they play nicely together. Nevertheless, it is implicit within those terms that the councils need to justify their work: they need to demonstrate not only their inherent worth and coherence, but also their ruthless, Thatcherite efficiency. They need to be irreplaceably important while getting outside ‘suppliers’ to do all the work for them.

It’s a tough balancing act to pull off. Get it right and they’ll be Blofeldian, sitting in their volcano HQ and getting their minions to run about and adjust their super (funding) laser. Get it wrong and they’ll be like the Wizard of Oz: grand pronouncements but little of substance behind the curtain. In the meantime research managers like me will have to continue to encourage people to apply for funding from RCUK, knowing that they might be better off buying a scratch card.

This article first appeared in Funding Insight on 10 February 2015 and is reproduced with kind permission of Research Professional. For more articles like this, visit www.researchprofessional.com