The
ESRC has recently held workshops - or 'masterclasses' - for members of its
peer review college. These involved some of the
Grants Assessment Panel (GAP) members talking about their experience, about what they have to bear in mind when assessing applications, and on the importance of the reviewers in the process.
As you can imagine, this provided some interesting insights. The GAP members were generally grateful to the reviewers, and recognised their reliance on the reviewers' knowledge to make their decisions. Some points to highlight:
Firstly,
the process itself:
- Each application gets sent to at least 3 academic reviewers, and a user reviewer (if relevant)
- If the average score for these is above 4/6, they get sent to GAP members who will act as 'introducers' at the panel meeting. Introducers usually get 7-10 proposals each to assess each meeting, and 4-5 weeks to write their assessments.
- Each application will have 2 introducers assessing them. If the average introducers' score is above 4/10, they go to panel. Only those scoring 6 or above are likely to be funded. Thus, as I've said before, it's worth noting how important the introducers are.
- However, the panel discussions allow for proposals that fall below this to be pulled up the rankings.
- Most of the discussion is around marginal or controversial proposals.
What are the
core assessment criteria for reviewers:
- Scientific quality and intellectual contribution;
- originality and potential contribution to knowledge;
- timelineness;
- robustness of research design and methods;
- value for money;
- outputs, dissemination and impact.
So what
should reviewers bear in mind when assessing an application?
- Give yourself enough time to properly assess the application.
- Judge only what is written, not what you imagine the project to be. If the applicant hasn't made clear what they're going to do, that is their fault.
- Base your judgement on the research question the applicant asks. Is it interesting/important? Are the methods appropriate for answering the question?
- Evaluate, don't advocate. Not every proposal can be funded. Be frank, and indicate risk versus benefit.
- Justify your arguments, and provide constructive criticism. It's important that (a) the panellists understand why you have scored as you have, and that (b) your score matches your comments. And, of course, it's useful for the applicant if they are rejected.
And
what should they not do?
- be personal or aggressive;
- be too brief or too verbiose;
- be ambiguous;
- make inappropriate, irrelevant or polemic remarks.
- forget that reviewing research proposals is different from reviewing papers: here, the research is speculative, so you have to evaluate the likely results;
- forget to draw attention to ways in which the proposal meets specific assessment criteria particularly well, and to point to any major logical flaws, contradictions or omissions;
- forget what the point of the review is, namely to weigh up the positive aspects of the proposal against the negative ones.
After the workshops the ESRC has updated its website with
FAQs and a checklist (
short and
long). It's worth having a look at these, to get an idea of what the reviewers will be considering when they start to read your proposal.
As ever, EPSRC is blazing a trail in developing new hurdles for potential applicants. Hot on the heels of controversially 'managing' its remit and 'managing' demand via its blacklist, the Council has just announced that, as of 15th November 2011, all applicants will have to identify the national importance of their research.
