Friday, 8 February 2013

ESRC Peer Review Process

As I said in my the previous post, I took part in a PGCHE mock panel today. I preparation I did some background reading on the ESRC process, and thought it would be interesting to set down the way that they assess applications.

  •   First Stage – ESRC receive the application

o   Roughly 10% of applications get rejected at this stage on technicalities, such as not having the right attachments, sections not being filled, format not being adhered to, etc

  •  Second Stage – External Reviewers

o   Each application gets sent to at least 3 academic reviewers, and a user reviewer (if relevant). These are identified using key words.
o   Reviewers use a scale of 1 (low) -6 (high) to rate the application.
o   Applications are assessed on:
§  Originality and potential contribution to knowledge;
§  Research design and methods;
§  Value for money;
§  Outputs, dissemination and impact;
§  Scheme specific criteria (not relevant to responsive mode).
o   Any application with an average score of less than 4/6 it will be rejected at this stage. This applies to about 30% of applications

  •  Third Stage – Introducers

o   The remaining applications are allocated to Grant Assessment Panel (GAP) members who will act as 'introducers' at the panel meeting. There are usually two introducers per application.
o   The ESRC tries to match the applications to the GAP member with the most relevant experience. However, there are only three GAPs, which cover a wide range of disciplines, so applications may well be introduced by someone with limited knowledge or understanding of the discipline.
§  The disciplines covered by the three GAPs are:

GAP A
Education 
Psychology 
Linguistics

GAP B
Sociology 
Social Work 
Social Policy 
Social Legal 
Area Studies 
Anthropology 
Statistics and Methods 
Politics and International Studies 
Science and Technology Studies
GAP C
Economics 
Management 
Demography 
Environmental Planning 
Geography 
History


§  If it is felt that there is no one with relevant experience on the GAP, they can either cross refer to another GAP, or even to another Research Council.
o   Each introducer gets around 7-10 proposals to assess each meeting, and 4-5 weeks to write their assessments. Their assessments will highlight key strengths and identify any weaknesses that need to be addressed.
o   Based on this assessment, they rate the application using a scale of 1 (low) – 10 (high).
o   The ESRC will analyse these scores and work out approximately what score the applications need to have got in order to go forward to the GAP meeting. Roughly 30% of applications get rejected at this stage, and realistically only those scoring 6 or above are likely to be funded.

  •  Fourth Stage – The Panel Meeting

o   Each application is introduced by the two GAP members. It is not always clear beforehand who will be the leading introducer and who will be the seconder.
o   The panel works through the applications in order of introducers’ scores: the ones with the highest score get discussed first, those with a lowest score get discussed last.
o   The Panel works with PDFs which save on paper but do make it difficult to refer back, check, and follow the discussion of applications easily.
o   ESRC officers are present, and do have input:
§  By saying roughly how many applications they can fund in that round;
§  By highlighting any problems with applications that have not been picked up before (eg they have been submitted to the Council before)
o   Whilst the panel will take a steer from the introducers, the panel discussions allow for proposals to be pulled up or down the rankings. Most of the discussion is around marginal or controversial proposals.
o   The Chair is key. S/he summarises the discussion and, if there is no consensus, has the final say.

No comments:

Post a Comment