ESRC Peer Review Process

As I said in my the previous post, I took part in a PGCHE mock panel today. I preparation I did some background reading on the ESRC process, and thought it would be interesting to set down the way that they assess applications.

•   First Stage – ESRC receive the application

o   Roughly 10% of applications get rejected at this stage on technicalities, such as not having the right attachments, sections not being filled, format not being adhered to, etc

•  Second Stage – External Reviewers

o   Each application gets sent to at least 3 academic reviewers, and a user reviewer (if relevant). These are identified using key words.

o   Reviewers use a scale of 1 (low) -6 (high) to rate the application.

o   Applications are assessed on:

§  Originality and potential contribution to knowledge;

§  Research design and methods;

§  Value for money;

§  Outputs, dissemination and impact;

§  Scheme specific criteria (not relevant to responsive mode).

o   Any application with an average score of less than 4/6 it will be rejected at this stage. This applies to about 30% of applications

•  Third Stage – Introducers

o   The remaining applications are allocated to Grant Assessment Panel (GAP) members who will act as 'introducers' at the panel meeting. There are usually two introducers per application.

o   The ESRC tries to match the applications to the GAP member with the most relevant experience. However, there are only three GAPs, which cover a wide range of disciplines, so applications may well be introduced by someone with limited knowledge or understanding of the discipline.

§  The disciplines covered by the three GAPs are:

GAP A Education  Psychology  Linguistics

GAP B Sociology  Social Work  Social Policy  Social Legal  Area Studies  Anthropology  Statistics and Methods  Politics and International Studies  Science and Technology Studies

GAP C Economics  Management  Demography  Environmental Planning  Geography  History

§  If it is felt that there is no one with relevant experience on the GAP, they can either cross refer to another GAP, or even to another Research Council.

o   Each introducer gets around 7-10 proposals to assess each meeting, and 4-5 weeks to write their assessments. Their assessments will highlight key strengths and identify any weaknesses that need to be addressed.

o   Based on this assessment, they rate the application using a scale of 1 (low) – 10 (high).

o   The ESRC will analyse these scores and work out approximately what score the applications need to have got in order to go forward to the GAP meeting. Roughly 30% of applications get rejected at this stage, and realistically only those scoring 6 or above are likely to be funded.

•  Fourth Stage – The Panel Meeting

o   Each application is introduced by the two GAP members. It is not always clear beforehand who will be the leading introducer and who will be the seconder.

o   The panel works through the applications in order of introducers’ scores: the ones with the highest score get discussed first, those with a lowest score get discussed last.

o   The Panel works with PDFs which save on paper but do make it difficult to refer back, check, and follow the discussion of applications easily.

o   ESRC officers are present, and do have input:

§  By saying roughly how many applications they can fund in that round;

§  By highlighting any problems with applications that have not been picked up before (eg they have been submitted to the Council before)

o   Whilst the panel will take a steer from the introducers, the panel discussions allow for proposals to be pulled up or down the rankings. Most of the discussion is around marginal or controversial proposals.

o   The Chair is key. S/he summarises the discussion and, if there is no consensus, has the final say.

From Me to You: ESRC Panellists Advice to Reviewers

The ESRC has recently held workshops - or 'masterclasses' - for members of its peer review college. These involved some of the Grants Assessment Panel (GAP) members talking about their experience, about what they have to bear in mind when assessing applications, and on the importance of the reviewers in the process.
As you can imagine, this provided some interesting insights. The GAP members were generally grateful to the reviewers, and recognised their reliance on the reviewers' knowledge to make their decisions. Some points to highlight:

Firstly, the process itself:

• Each application gets sent to at least 3 academic reviewers, and a user reviewer (if relevant)


• If the average score for these is above 4/6, they get sent to GAP members who will act as 'introducers' at the panel meeting. Introducers usually get 7-10 proposals each to assess each meeting, and 4-5 weeks to write their assessments.


• Each application will have 2 introducers assessing them. If the average introducers' score is above 4/10, they go to panel. Only those scoring 6 or above are likely to be funded. Thus, as I've said before, it's worth noting how important the introducers are.


• However, the panel discussions allow for proposals that fall below this to be pulled up the rankings.


• Most of the discussion is around marginal or controversial proposals.

What are the core assessment criteria for reviewers:

• Scientific quality and intellectual contribution;


• originality and potential contribution to knowledge;


• timelineness;


• robustness of research design and methods;


• value for money;


• outputs, dissemination and impact.

So what should reviewers bear in mind when assessing an application?

• Give yourself enough time to properly assess the application.


• Judge only what is written, not what you imagine the project to be. If the applicant hasn't made clear what they're going to do, that is their fault.


• Base your judgement on the research question the applicant asks. Is it interesting/important? Are the methods appropriate for answering the question?


• Evaluate, don't advocate. Not every proposal can be funded. Be frank, and indicate risk versus benefit.


• Justify your arguments, and provide constructive criticism. It's important that (a) the panellists understand why you have scored as you have, and that (b) your score matches your comments. And, of course, it's useful for the applicant if they are rejected.

And what should they not do?

• be personal or aggressive;


• be too brief or too verbiose;


• be ambiguous;


• make inappropriate, irrelevant or polemic remarks.


• forget that reviewing research proposals is different from reviewing papers: here, the research is speculative, so you have to evaluate the likely results;


• forget to draw attention to ways in which the proposal meets specific assessment criteria particularly well, and to point to any major logical flaws, contradictions or omissions;


• forget what the point of the review is, namely to weigh up the positive aspects of the proposal against the negative ones.

After the workshops the ESRC has updated its website with FAQs and a checklist (short and long). It's worth having a look at these, to get an idea of what the reviewers will be considering when they start to read your proposal.