NERC Demand Management: a Clarification

NERC

Polaris House

Swindon
SN2 1EU

Dear Vice Chancellor

I am writing to you in connection with our recent announcement about measures we are introducing to manage the demand for NERC funding.

As you will have seen, we are planning to cut the maximum funding level for standard grants (from £1.2m to £800k, fEC), but we will also be restricting the number of applications from those institutions which have a success rate of less than 20%.

We have had a number of queries about the methodology we will be using to decide whether institutions fall within this bracket. When we explain the system, we are often met with looks of puzzlement and confusion, and are asked to explain it again. And again. Some have even gone so far as to suggest that it is an unnecessarily complicated and burdensome system.

NERC Introduces Demand Management

NERC has become the third Research Council - after EPSRC and ESRC - to explicitly state that they want universities to 'self regulate' their applications. This announcement was triggered by worries about success rates in some of their schemes falling to 16%.

Whilst NERC already has in place some measures to 'manage demand' - eg limiting the number of applications an investigator can submit per call and restricting resubmissions - this hasn't stopped the success rates from sliding in recent years. They're hoping to reverse this by encouraging institutions to strip out applications which NERC would define as 'uncompetitive' (defined by them as scoring 6/10 or below at panel).

So what are they going to do?

• firstly, ask institutions to nominate a point of contact for demand management;
• secondly, in the summer, provide data on past performance to them. This will be repeated annually from autumn 2013. The data will apply to Urgency, Large and Standard Grants, but not Fellowships or outlines. It will include: success rates for all schemes; distribution of grades for funded and unfunded proposals by scheme; final moderated grades for all proposals from institution/department; relative performance of institution/department.
• thirdly, from autumn 2012 NERC will (ahem) 'engage in a strategic dialogue' with institutions to provide information and advice in support of demand management, including setting targets for changes in submission behaviours. They can't meet with everyone in the first year, so those with the most applications, or with black marks in the NERC copy book, will be the first to get a visit from 'the management.'

So, at the moment, it looks to be relatively light touch: more ESRC than EPSRC. However, there will be the expectation that all research organisation will have their own internal quality control systems in place.

Who will be the next Research Council to fall in to line? Given the recent rumblings from Death Star House, my money's on the AHRC...

ERC Success Rates

Interesting stats from UKRO on the ERC Starting (StG) and Advance (AdG) Grants. There's now been three complete calls for each scheme, and UKRO's statistics show how the success rate for both StG and AdG has risen:

• Starting Grants: 3.3% (2007-8) to 14.9 (2009-10)
• Advanced Grants: 12.7% (2007-08) to 13.2% (2009-10)

Interestingly, the success rates for Grants hosted at UK HEIs is higher than the European average for StG (17.3%), but only marginally so for AdG (13.6%).

Now compare these figures with those published by the ESRC in their Demand Management Consultation Paper on the recent success rates for all the UK Research Councils:

• AHRC: 16%
• ESRC: 17% (small and standard grants combined)
• MRC: 19%
• BBSRC: 22%
• NERC: 24%
• EPSRC: 30% (following the introduction of its blacklisting procedure)
• STFC: 53%

So the ERC success rates aren't a million miles away from the bottom three Research Councils - and the UK Starting Grants are actually the same or better than the AHRC and ESRC. No wonder the ESRC is making moves to introduce demand management measures, and the AHRC is following fast in its footsteps.

Changes to the ESRC - Part 2

Last week I wrote about imminent changes to the ESRC funding schemes. Yesterday I went along to their regional event at Brighton, and got further clarification and detail about how they see these changes being implemented:

• Risky research. As mentioned before, they will be introducing a new mechanism into their grants scheme for risky research, with a 'breakpoint' mid way through at which the success or otherwise of the pilot project will be assessed. However, they made it clear that, in effect, this would mean the reintroduction of the small grants scheme, but with a very specific remit of encouraging risky, innovative, ambitious research.
• Advanced Sifting. As well as outline applications, the ESRC would introduce 'advanced sifting' of their Research Grants. In practice, this would mean that applications go through an initial peer review by 2-3 academics. If it's an application for a small amount, this might be all the assessment it gets: depending on the outcome, the application will get funded or rejected. If it's for a larger amount, this peer review will decide whether the application goes to full panel.
• Outline Applications. The ESRC will simplify the JeS form for outline applications, including the costing element. They will shortlist approximately three times as many applications as they can fund, so that the success rate for the second stage (full applications) will be around 33%.
• Right to Reply. A right to reply would be built into all their funding schemes.
• Demand Management. The deadline for the consultation process on the demand management options closed on 16 June. They will now consider the responses. It is hoped that they will not have to introduce any of the more draconian measures. They will allow a year to see how the 'interim measures' have worked - eg outline applications, no uninvited resubmissions, encouraging HEIs to implement 'quality assurance' procedures. They admitted that a year might not be enough for these to have a real effect, but it should be enough to see the 'direction of travel'. If they are happy with the 'direction', they will allow more time for them to have further effect.
• Statistics. To support the demand management measures, the ESRC will provide HEIs with stats on their comparative performance. These will be provided three times a year and, it is hoped, they will be more nuanced than just presenting simple success rates. For example, they should show the relative position of an institution's applications in the peer review panel's prioritisation list, so that HEIs can get a sense of the quality of their applications.
• Working with the Other Research Councils. The ESRC made clear that they will be working with their sister councils to implement a common form of demand management. Of course, this will have to allow for variance that arises from the culture and patterns of those working within a council's disciplines. Thus, what works for the EPSRC wouldn't necessarily work for the ESRC. However, as far as possible they hoped for a consistency across RCUK.

AHRC Follows ESRC to Manage Demand

Disturbing - but, let's be frank, unsurprising - news from the Faculty of Arts & Humanities on Death Star Avenue. They're planning to follow their sisters in the Faculty of Social & Economic Sciences and will introduce some form of demand management.

As I say, unsurprising. The clues have all been there. It was clear in their Delivery Plan that they wanted to 'mange demand' (Key Point 10, first bullet point). However, I had assumed that this would be the more touchy-feely end of demand management, and sometime in the future. Why? Because they talked (4.7.1) about the relatively low-level steps they'd taken already, such as rolling deadlines. Elsewhere (4.7.2), they said that they would 'systematically collect, analyse and disseminate to HEIs' success rates and trends, and, following this, will have stern, headmasterly 'strategic discussions with key HEIs...falling below the average, to develop self-management of demand and quality control of proposals.'

As far as I'm aware this hasn't happened yet. However, the word on the street is that they're rushing to follow the ESRC into a more Darth Vaderish understanding of management. As I'm sure you know, the ESRC is currently consulting on possible forms this will take, including quotas, sanctions, and paying to apply.

Whatever comes out of that process will probably end up as AHRC policy sooner rather than later. As the ESRC said in the Consultation Document: 'the Research Councils, where possible, will harmonise their demand management strategies.' I just didn't expect them to be so quick, as their pace is other areas can seem glacially slow. I mean, have you had an application reviewed recently?

Anyway, it's all the more reasons for us to crack on with the Internal Peer Review System which is close to its final draft form, and will hopefully be in place by the time Darth comes knocking.

ESRC Demand Management: This Time They're Serious

After the ESRC's messed up with its last consultation on 'demand management', which went off half cocked and was only sent out to one sleepy, post-Christmas academic and his dog, the Council is now trying to do it properly. It's issued a new consultation document, and is seeking responses by 16 June 2011.

To be fair to the ESRC, it has, in the interim, developed a more thorough and thoughtful document. Whereas the previous one was a relatively breezy eight pages, the current consultation document is a meaty 25 pages.

And this time they're serious.

They will be introducing an ‘initial programme of measures’, starting in June 2011, which will be a fairly light touch form of demand management. This includes:

• Universities/individuals having to demonstrate that they are ‘looking at ways to improve self-regulation’.
  
• Providing more ‘performance data’ to institutions;
• Invited-only resubmissions;
• Earlier ‘sifting’ of applications, and greater use of outline application processes.
  

They’ll review this after 12 months to see if it’s had any effect. Depending on this, they may look to introduce some further ‘demand management’. They suggest that this could be:

• Researcher sanctions: limiting the number of applications from individuals who consistently fail to meet an agreed quality threshold – along the lines of the EPSRC’s ‘blacklisting’.
• Institutional sanctions: limiting the number of applications from institutions which don’t meet an agreed quality threshold. This could take the form of limiting applications to 50% of the number submitted before, followed by a 12 month ban if quality hasn’t improved. Quality threshold may be at least 50% of applications Alpha rated or above each year.
• Institutional quotas: institutions would be put into one of four quota ‘bands’, based on previous performance, and could be ‘promoted’ or ‘relegated’. The four bands would be: top ten, next twenty, next thirty, and all the rest.
• Charging for applications: The ESRC is suggesting a maximum of £1k per application, refundable if the application passes a quality threshold.
  

More details of these proposals are available here.

It’s worth noting that it looks like demand management will come in for all Research Councils, that they’ll be some consistency between them, and that they will expect institutions to have their own peer review systems in place, as suggested on p8 of the document:

‘The Research Councils, where possible, will harmonise their demand management strategies. There is general agreement that HEIs should be encouraged to self regulate with a particular emphasis on structured peer review aimed at the submission of significantly fewer but better quality applications. This self regulation will be underpinned by the regular supply of performance data to institutions alongside better applicant guidance.’

As I said, the deadline for responses is 16 June 2011. You can respond using the form here. They will discuss proposals and announce the outcome in autumn 2011. Once they’ve reviewed the ‘initial programme of measures’, they’ll announce any new measure in October 2012.

BBSRC Hints at Demand Management

Prof Douglas Kell, CEO of the BBSRC, has sent out a letter detailing changes in light of its Delivery Plan and its move to the Shared Services Centre. Whilst not explicitly introducing demand management systems (as the ESRC has stated it will), it is moving to limit neuroscience applications and has hinted that there may be some form of 'triage' in the futre.

SSC: Delays, and Future Deadlines
There will be some delays to grant applications submitted on 11 Jan, as a result of the BBSRC's move to the SSC. In addition, the next two deadlines will be later: 25 May 2011, and 25 Oct 2011.

Changes to Committee Structure
The new quadripartite 'clover leaf' system of grant committees will remain, although Committee C will expand, from 25 May, to take over responsibility for genetics and development.

Changes to Grading of Proposals
From 11 Jan the BBSRC will move to grading proposals on a 1-6 scale, in line with the other Research Councils. It notes, ominously: 'Enabling referees to score grants means that, if necessary, it will be possible in future to operate a triage of grant proposals based on referee scores, in order to eliminate lower-scoring applications before the committee meeting and reduce the burden at meetings.' Whilst I understand that having a grading system would allow triage to happen, should we read anything in to the fact that the BBSRC will sharing the same grading system as that used by the other Councils? Does this hint at the Research Councils introducing a kind of 'Shared Triage Centre'?

In addition they will be grading the 'Pathways to Impact' section, and this will have an effect on the overall grading.

Limiting Allowance for Errors
It will also limit the allowance it makes for errors in applications that have been rushed to be submitted. Apparently there are too many of these. If errors are picked up the BBSRC won't allow applicants to amend and resubmit to the same deadline, but will have to wait until the next.

Use of JeS for Referees
It sounds like they're getting strict on referees, too. 'Individuals with a poor track record of responding to such requests [to referee proposals] are unlikely to be invited to join Committees and other bodies involved in developing and delivering BBSRC science,' it states. In addition, they will be corresponding with all potential referees through the JeS system. So make sure your profile is up to date.

'The Current System Discourages People from Undertaking Inexpensive Research'

A refreshing counterblast from Prof Dorothy Bishop, Professor of Developmental Neuropsychology and a Wellcome Principal Research Fellow at the Department of Experimental Psychology at Oxford.

She appreciates the need for the ESRC to stem the flow of applications, but thinks that they are looking for solutions in the wrong place. The ESRC needs 'to change the basic structure of university funding so that institutions and individuals are no longer assessed on amount of research funding, but rather on an output/input function, i.e. how much bang do you get for your buck,' she argues.

'Currently, a person who secures a £500K grant which leads to two publications in lower-ranking journals will be given more credit than one who generates five high-ranking publications over the same period with a £50K grant. Clearly, some sorts of research are much more expensive than others; the problem is that the current system discourages people from undertaking inexpensive research.'

And this is only going to get worse, with the ESRC's emphasis on longer and larger grants, and its (and the BA's) cutting of Small Grants. Academics who, in the past would have got value for money from relatively small amounts of money, are now going to try and artificially inflate their projects so that they flop over the £200k threshold.

Instead of backing any of the ESRC's proposals, Bishop is instead suggesting a different approach: allocate, review and reward those who demonstrate thrifty and cost effective use of funds.

Radical.

Wot No Big Society?

An interesting viewpoint from Sue Richards of the Institute for Government on the ESRC's plans for the next four years. She makes three points: that the ESRC missed a trick by not making their case in the context of Cameron's 'big society'; that it seems to be wanting to push for more concentration; and that the need for further investment in the ESRC's longitudinal studies is not necessarily justified.

All valid points. We're currently having discussions internally about the ESRC's proposals for demand management, and I agree with Richards on this. It does look like an effect of the plans will be to limit funding to the favoured few. Kent does reasonably well out of the ESRC, and might benefit from such a system. However, I think there's a danger here of doing a disservice to social sciences more broadly. Unlike many of the hard sciences, it does not rely on large infrastructure, and a lot of good work is done in small 'pockets of excellence' in institutions that are not research intensive. These could be adversely affected by the ESRC's move to concentration.

I also agree with Richards on the Big Society slip. Whether the ESRC agrees politically with Cameron's drive or not, it should have used it as a tool to bolster its position and lever a better settlement. It looks like an opportunity missed.

ESRC Consults on 'Demand Management'

Following the ESRC's response to the Government's budget allocation for 2011-15, its Chief Executive, Prof Paul Boyle, has stated explictly that it will be introducing some form of 'demand management'. This means that he intends to limit the number of applications individuals or institutions can submit.

According to Boyle, recent increases in the volume of applications have resulted in 'considerable redundant effort across the social science community in the preparation and peer review of applications. It is also creating administrative inefficiencies both at HEIs and the ESRC itself in the processing of such applications at a time when we are all being asked to reduce our administrative costs.'

Thus, reasons Boyle, 'the ESRC is particularly keen to reduce the number of applications which fail to reach minimum quality thresholds and concentrate everyone's effort on preparing and assessing those applications which are fundable.'

However, he wants to involve the academy in the decision making process. In a discussion paper (available via the link here), he lays out five options:

• Researcher Sanctions: This involves limiting the number of proposals from individual researchers who consistently fail to submit applications that reach an agreed quality threshold;
• Institutional Sanctions: This involves introducing sanctions for HEIs whose applications fail to meet a certain success rate and/or quality threshold;
• Institutional Quotas for ‘managed mode’ schemes. This involves the introduction of institutional quotas for certain schemes (e.g. early career researcher schemes, Large Grants/Centres, Professorial Fellowships).;
• Institutional quotas for all schemes: This involves responsive as well as managed mode schemes;
• Charging for applications. Levying an agreed fee for institutions submitting applications, with the option that this levy is redeemable if the application is successful.

As well as these, Boyle suggests that the ESRC could introduce other measures, such as tougher sifting, resbumissions by invitation only, and spreading good practice.

The consultation will ask the following broad questions:

• which, if any of options, should not be considered for further development and why?
• overall, which of the options offers the best opportunities to effectively manage demand whilst ensuring the flow of high quality research applications
• how might these options be further developed and refined?
• what other possible options should be considered that are not covered here?
• which of the other additional options might be taken forward as complementary to a main demand management strategy?
• how might any of these complementary options be further refined?

The deadline for responses is 18 Jan. Do get in touch with them with your thoughts. Responses should be sent to Jeremy Neathey.

RCUK Success Rates & Demand Management

Interesting piece in the Times Higher this week. The success rate for EPSRC has gone up, from 26% to 30%, whilst many of its sister Councils have seen rates decrease. No surprise there, but it will act as a spur to the others to introduce some form of demand management, as Adrian Alsop of the ESRC suggested in his talk to us in October. He said that the ESRC was considering a range of options, including barring individual serial low achievers,spreading best practice, introducing institutional quotas, or demanding financial deposits.
The case for formalising internal peer review is getting stronger and stronger, a fact recognised by the Directors of Research at their recent Network meeting. However, any system should be supportive, non-bureaucratic and relatively quick. Otherwise it will act as a disincentive, and may put people off from even considering applying. As Oxford's PVC noted in the Times Higher article, 'it is a bit demoralising when you see your proposal has only a 10 per cent chance of success, but it is worse if you exclude ideas from being aired and evaluated.'
Do read the article in full if you get the chance. In the meantime here are the Research Councils' current success rates:

STFC: Changes to its Grant Schemes

After its consultation earlier this year, the Science and Technology Funding Council (STFC) has decided to implement changes to its grants schemes. The existing system of standard and rolling grant mechanisms will be replaced by a single consolidated grant scheme containing core and non-core staff post funding. Core staff posts would be those key posts identified by the grants panel as being crucial for long term support of research activities.

What does that mean? Well, time will tell. But it looks like they're mixing and matching elements of Wellcome's funding of the big stars of the reserarch firmament, and EPSRC's 'demand management'.

The main features will be:

• One consolidated proposal per department (or equivalent) per subject area submitted every 3 years;
• Core staff posts could be funded up to 4 years, non core staff posts up to 3 years (but with flexibility to spend over 4 years using the existing rules for applying for a grant extension);
• An individual academic can only be supported for exploitation on a maximum of one consolidated grant.

To be honest, I haven't entirely got my head around this. However, STFC say that doing things this way will reduce the number of grant applications submitted over 3 years from ~600 to ~110. So you can see where they're coming from. But we'll have to wait for the dust to settle a little before decided on the effect this will have.

Details on the key features of a consolidated scheme and the other Panel recommendations can be found in the full report (PDF - link opens in a new window).

More Demand Management from EPSRC

The EPSRC has issued guidance on the number of applications it will accept to it's Postdoc Fellowships in the Mathematical Sciences.
This year it will only accept five per institution. In subsequent years, it will alter this allowance, depending on the quality of applications submitted previously, according to the following criteria

1. The cap will increase by one if all proposals submitted by that research organisation ranked above the quality cut- off as determined by the peer review panel.
2. The cap will remain constant if between 70% and 100% of proposals submitted by that research organisation ranked above quality cut-off.
3. The cap will decrease by one if between 50% and 70% of proposals submitted by that research organisation ranked above quality cut-off.
4. The cap will decrease by two if less than 50% of proposals submitted by that research organisation ranked above quality cut-off.
   

A minimum of two applications per research organisation will be guaranteed at any given year (independently of the above table). Also, no more than eight proposals will be accepted by any research organisation at any given time.
Every year, the Sift Panel will be asked to draw the quality cut-off, which will be used to determine variations on the number of proposals for the following year. EPSRC will email HoDs and Research Offices to tell them how many applications can be submitted.
At the moment this cap is being limited to the Mathematical Sciences, which is particularly oversubscribed. I called them to get a sense of whether it would be rolled out into other areas, and they were adamant that they would not. But watch this space...

EPSRC Demand Management and 'Office Rejection'

I've been seeking further clarification on the way the EPSRC's new 'demand management' - or blacklisting - system will work. In particular, I was wanting to know whether proposals that had not gone to the panel, so called 'office rejects', would be included in the three unfunded proposal allowance applicants get before blacklisting kicks in. The answer is, unfortunately, yes. However, EPSRC are reassuring on this; a proposal would only be rejected by the office if:

• it had been sent out to peer review, and the reviews were so unsupportive that there was no point in it going to panel. This may seem fair enough, but I know of at least one case where a proposal was rejected was rejected on these grounds, but the reviews were based on a misunderstanding, and were factually inaccurate;
• it doesn't fit with the call criteria, or with the remit of the EPSRC. Once again, this seems fair, although there is the question of interdisciplinary proposals. In these situations, however, EPSRC encourages applicants to contact it first and submit a 2 page 'remit query' so that it avoids being rejected on these grounds.

I asked if a proposal would be rejected by the office on a technicality - eg not having the right attachment, or a section of JeS being filled incorrectly, or the costings being incorrect. No, she said, this would not happen: if there was enough time before a deadline (or the deadline was open) the office would come back to you to rectify this. Also, JeS would be unlikely to allow you to submit an incorrectly completed form. If there wasn't time before the deadline and the application was incomplete in some way, the application would be returned, but it wouldn't count as a rejection, as it would not have gone through an assessment process.