'Citations Are Peer Review by the World'

Prof John Mingers

The research excellence framework - or REF - has become such an established part of the higher education landscape that it’s hard to imagine a world without it. Since 1985 it has provided the navigation points in a turbulent and changing landscape, marking progress and, like a modern day Lachesis, measuring the threads of individual research in readiness for Atropos’ shears.

As we gear up for REF2021 and wait to hear from HEFCE about what form it will take, it seems an appropriate time to take stock and question whether the game is worth the candle, and if the measurement of research should be undertaken differently.

Responding to Reviewers' Comments

Prof Sally Sheldon

Yesterday we re-ran a session for the Early Career Researcher Network on 'Getting Published: targeting top journals and writing book proposals.' It was led by Prof Sally Sheldon (KLS) and Prof Mick Tuite (Biosciences). I've written up some notes from it before, but I wanted to add to these by highlighting an important aspect of the process: how to respond to reviewers' comments. 'We don’t take it seriously enough,’ suggested Sally, and yet it's crucial if you want to get published - or funded.

Peer Review and the Xerox Machine

Martha had finally found
a use for the pasta maker
they'd been given for
their wedding

Last year I wrote a column in response to a letter from a group of eminent academics bemoaning the current state of peer review. “Support for research that might lead to major new scientific discoveries is virtually forbidden nowadays,” the letter claimed, “and science is in serious danger of stagnating.”

I explored the problems of the system and what could be done to fix it. I looked forward but didn’t think to look back. I’d assumed, like many others, that the current system had always been so. I hadn’t realised that the first use of the term peer review was in 1969.

ESRC Success Rate Drops to 10% for July Open Call

'Swindon, we have a problem'

My colleague Brian Lingley drew my attention to a rather alarming set of data from the ESRC last week. The scores and funding outcomes published in November show that it received 144 responsive mode applications for July. Of those, only 14 were funded, giving a success rate of 10%.

Is this an anomaly? It is certainly less than the 25% ESRC average for 2013-14 given in the Times Higher round up published at the same time. More alarming, perhaps, was the number of high quality applications which didn't make it past the funding cut off point, which was in the 8-8.9 score range. In fact, there were more unsuccessful proposals in this range than there were successful ones.

Thus, according to ESRC's own definitions, it has had to turn away more than half of the 'excellent proposals which are of significant value, and are highly likely to make a very important scientific contribution and/or will significantly enhance the development of the applicant's academic career.'

Saving Peer Review

In June an eminent group of academics, including Nobel laureates, wrote the following to The Daily Telegraph: "Peer review [of funding applications] is now virtually unavoidable and its bureaucratic, protracted procedures are repeated for every change in direction or new phase of experimentation or for whatever an applicant might subsequently propose. Consequently, support for research that might lead to major new scientific discoveries is virtually forbidden nowadays, and science is in serious danger of stagnating."

Keith Bontrager, the legendary designer of bike components, once said about bicycles: "Strong, light, cheap: pick two." You can never have all three. A similar universal law seems to be at work with systems for selecting funding proposals and academic articles, but perhaps the elements here should be: robust, equitable, easy.

Thoughts from an ESRC Mock Panel

Last month we held an ESRC mock panel as part of the Grants Factory. This is a really useful exercise; it gives participants a flavour of the discussions and debates that take place in a real Grant Assessment Panel (GAP), but it also gives them an idea of the tough assessment their application will have to go through.

Whilst I've posted elsewhere on this blog about what makes a good application (eg here, here, and here), a couple of points were raised at the workshop that I thought that they were worth repeating:

• Firstly, panellists rarely read your proposal in the strict order in which its presented. The two panellists who led the workshop said that they normally read the JeS form first, to get a sense of what the project is about, then skip to the reviewers' comments and PI's response, before returning to the Case for Support. Your response, then, is crucial. This is true of all the Councils. A panellist for one of the other Research Councils said the following after returning from a panel meeting:

'The PI's responses were key and a substantial number of these were badly done (serving simply to refute or to point out disagreement between reviewers rather than rebut with argument, to clarify or to accept reviewers' suggestions).  Not all PIs made use of the whole space allowed.  Spending time reminding the panel of the positive things that reviewers had said was a waste of space if there were substantial issues to be addressed...I'm certain that I saw applications that would have received a higher final grade (and possibly funding) if the PI response had been better done.'

• Secondly, most panellists won't have a background in your area. The GAPs are quite broad (see their disciplinary configuration here), so you need to make sure of two things: first, that you explain your research in a way that an intelligent general reader can understand; and secondly, that your methodology is watertight. Why? Because although the panellists might not understand the specifics of your project, they will all understand (or think they understand) the underlying methodology. So that is where they're going to pick holes. In particular, you need to be strong on how you analyse the data. Try and preempt any problems they might see in your methodology, and head them off at the pass. 

The final mock panel of the year will focus on the EPSRC, and will take place on 4 June. Drop me a line if you want to take part.

Changes to NERC Peer Review

NERC is introducing changes to its peer review process to improve 'consistency, quality and transparency'. In an email to peer review college (PRC) members, the Council set out what will stay the same and what will change.

What will stay the same? 

The central role and importance of the PRC will remain. However, it wants to allow for more 'inclusivity and flexibility', whilst recognising the need for 'consistency, particularly in panel membership.'

What will change? 

• Timing of peer review periods, to try and avoid overlap with each other and major university holidays. They will also confirm membership and dates earlier in the process. 
• More consistent panel membership and more stable panel portfolios. There will be a new role of ‘core’ member, and they will have a primary responsibility for attending moderating panel meetings in a particular panel area, forming half of the panel membership. Other PRC members will be invited on an ad hoc basis. All current PRC members will be considered for 'core' membership, which will be fixed term.  
• More consistent review levels and the removal of the ‘sift’ process.  Reviewers will no longer be asked to complete reviews where they judge that they have low expertise. In addition, moderating panels will consider all proposals submitted to a call, so meetings will be longer (held over two days) and the role of panel member even more central to the funding and assessment process. 
• A pre-score process will be introduced to allow panels to prioritise proposals for discussion. Pre-scores for excellence will be required from panel members at least 1 week before meetings. 
• Better Performance monitoring and management. Performance standards will be introduced and more closely monitored on a PRC and individual level. There will also be on-line training for all members to ensure that members continue to be aware of what is expected of them and any changes to policy;

When will these changes happen? 

From January 2014. The PRC year will start from January rather than July from that point onwards. In the meantime there will be a call for new membership in June 2013.

Writing a Response to EPSRC Reviews

Making a virtue of criticism

Last week we had an impromptu Grants Factory workshop, prompted by the visit of the EPSRC's Jane Nicholson the week before. Focussing on applying to EPSRC, Profs Simon Thompson (Computing) and Sarah Spurgeon (EDA) spoke about their experience of sitting on EPSRC panels, before running a 'mock panel' exercise in the second half of the event. It was a really useful event, and I thought it would be good to jot down some notes from it, particularly around the vexed question of responding to reviewers' comments, which is crucial in applying to any of the Research Councils. So, if you have recently received your comments and are mulling over your response, some top tips:

• The panel aren’t re-reviewing the application. They are instead moderating the reviews and ranking the applications accordingly. Thus, the PI response is critical, as it’s your only chance to answer any concerns or criticisms. Whilst it’s not mandatory, you should always provide a response. As Prof Sarah Spurgeon said at the event, ‘writing a good response does make a competitive difference’.
•  Some of the reviewers’ comments can be hurtful. You’ve invested time in the application, and they can be dismissive or even wrong-headed. Don’t respond in haste. Take time to provide a measured, considered response. Don’t dismiss any comment that is ‘obviously’ wrong: you can make it clear that you disagree without using emotive language. For instance, suggest that you want to ‘clarify’ a point.
•  Go through the reviewers and pick out every comment that needs a response. List them, and answer them in order.
• Give evidence to rebut the criticisms. Once again, don’t be hasty and impassioned. This is the time to be clear and analytical. Give the panellists just the information they need, the information necessary to ‘empower’ them;  it’s not the time to be quoting complicated mathematical formulae – unless absolutely necessary.
• Some responses may not be given to the panel before the meeting, but tabled on the day. The panellists have to read them quickly, so make it easy for them: plain language, clear formatting, bullet points. Don’t be ‘clever’ with unusual fonts, minimal margins or complicated figures.
•  Don’t clutter the response by thanking the reviewers, or take too long highlighting the positive points. The panel will have already seen both the application and the reviews, and will already have a view.
• Don’t feel you have to write to the limit. To quote Prof Spurgeon again: ‘a short response is more powerful.’
•  Finally, if the reviewers have suggested a good idea, there’s no harm in welcoming it and agreeing with them – as long as it doesn’t contradict the main thrust of your proposal.

AHRC: A Final Check before Submitting

Check that you've got something written
next to your tickboxes...d'oh!

The AHRC's Morag Sullivan gave us a startling statistic at yesterday's AHRC event: between a quarter and a fifth of all applications submitted to the AHRC are rejected by the office for commonsensical and easily avoided oversights.

So, if you’re planning to apply to the AHRC shortly, make sure you’ve got the following ‘i’s dotted and ‘t’s crossed:

·         Does the proposal fit within AHRC’s remit?

·         Is the primary classification in the AHRC subject area?

·         Have appropriate key words been selected?

·         Does the proposal meet the scheme aims?

·         Are all required attachments present?

·         Are the page limits for each attachment right?

·         Are there surplus attachments?

·         Are costs listed under the correct cost heading?

·         Are they explained in the Justification of Resources?

·          Do costs match across different documents?

·         Is the Technical Summary complete?

·         Is the Early Career Researcher eligibility statement completed?

·         Do you have the necessary employment contract to apply?

·          Is the start date within correct timeframe?

·         Do the hours worked match the hours charged?

·          Is the RA of postdoctoral standing?

AHRC - View from the Committee Room

I took part in the AHRC’s inaugural ‘Developing Better Applications’ event yesterday. It was a great event, and a good opportunity to chat to others doing a similar job to me in a wide range of different institutions.

Prof Roberta Mock gave a really useful talk based on her experience as a peer review panellist for the AHRC. Amongst the points she raised were:

·         The AHRC is not a cabal. It is not us and them. They rely on academics reviewing other academics. We are all part of it, and necessary for its successful running.

·         Applicants should not run before they walk. Having a commensurate track record was crucial for getting an appropriate grant.

·         You should write with potential reviewers in mind, and imagine the ‘nightmare critic’. Preempt their criticism, but don’t be defensive. Reviewers smell fear.

·         Choose keywords wisely. These are used for choosing your reviewers, so do think about what specialism you want your reviewer to have.

·         Talk to colleagues, and share your application. Get some tough love. Better still, take it through internal peer review. It was always clear at panel which applications hadn’t.

·         Take time over preparing it. A good application takes at least two months (and at least 40 hours of intense writing) to draft. 

·         The standard has changed over recent years. What worked five years ago will not work now. Have to keep getting better to stay still.

·         Grammar, spelling and clear formatting do all make a difference.

·         Use the sections, and write what they ask for in the appropriate sections. Doing otherwise makes you appear arrogant.

·         Don’t over inflate claims for impact. The panel is not necessarily looking for the most impactful project , but just for reassurance that you’ve got an effective strategy in place.

·         Don’t hide or disregard ethical elements of your research. If you blank this, or claim not to have any, the panellists will look all the harder for them.

·         It’s all in the detail. Be specific about such issues as which journals you intend to publish in or which conferences you plan to attend. Give them a sense of how you arrived at your costs.

·         Have a realistic work plan that takes account of having a life beyond your research – i.e. factor in holidays, recruitment, potential illness, etc. You are not a robot, and neither is your RA.

·         Value for money is important. That’s not just a case of offering the lowest price. Rather, it’s asking for the money necessary to achieve the objectives and answer the research question. Moreover, it offers research that has both reach and significance.

·         Include information about monitoring of the project. This is often left out, but is really important. It needs to be built into the workplan. It demonstrates institutional buy in and shows that the stewardship of the award is taken seriously.  

·         A ‘super critical ’review is not the end of world, but a convincing right to reply is crucial. You need to be very gracious, but be aware that the panel sees everything. You don’t need to repeat praise from the reviewers, and don’t use one reviewer’s comments against another. The panel sees all the paperwork, and can see if any of the reviewers are out of line.

·         Yes, there is an element of luck. However, there is usually agreement about the first and second ranked applications. The grey area – and the luck – comes further down the list. So give yourself as much of a helping hand as possible. If there’s an early career researcher card, play it. If there’s a highlight notice you can latch on to, do it.

The questions that followed flushed out a final, interesting point: not all reviewers read applications in the same order. Roberta, for instance, flicks to the CV first. All the more reason to do as the Grants Factory suggests, and make sure that key messages are written through the application like words through Brighton rock. Wherever you bite into it, you can see what the research question is, why it’s important, why it offers value for money, and why you’re competent to handle the project.

The training event runs again in London on the 8 March. I'm not sure if it's booked up, but get in touch with the AHRC if you want to go along.

ESRC Peer Review Process

As I said in my the previous post, I took part in a PGCHE mock panel today. I preparation I did some background reading on the ESRC process, and thought it would be interesting to set down the way that they assess applications.

•   First Stage – ESRC receive the application

o   Roughly 10% of applications get rejected at this stage on technicalities, such as not having the right attachments, sections not being filled, format not being adhered to, etc

•  Second Stage – External Reviewers

o   Each application gets sent to at least 3 academic reviewers, and a user reviewer (if relevant). These are identified using key words.

o   Reviewers use a scale of 1 (low) -6 (high) to rate the application.

o   Applications are assessed on:

§  Originality and potential contribution to knowledge;

§  Research design and methods;

§  Value for money;

§  Outputs, dissemination and impact;

§  Scheme specific criteria (not relevant to responsive mode).

o   Any application with an average score of less than 4/6 it will be rejected at this stage. This applies to about 30% of applications

•  Third Stage – Introducers

o   The remaining applications are allocated to Grant Assessment Panel (GAP) members who will act as 'introducers' at the panel meeting. There are usually two introducers per application.

o   The ESRC tries to match the applications to the GAP member with the most relevant experience. However, there are only three GAPs, which cover a wide range of disciplines, so applications may well be introduced by someone with limited knowledge or understanding of the discipline.

§  The disciplines covered by the three GAPs are:

GAP A Education  Psychology  Linguistics

GAP B Sociology  Social Work  Social Policy  Social Legal  Area Studies  Anthropology  Statistics and Methods  Politics and International Studies  Science and Technology Studies

GAP C Economics  Management  Demography  Environmental Planning  Geography  History

§  If it is felt that there is no one with relevant experience on the GAP, they can either cross refer to another GAP, or even to another Research Council.

o   Each introducer gets around 7-10 proposals to assess each meeting, and 4-5 weeks to write their assessments. Their assessments will highlight key strengths and identify any weaknesses that need to be addressed.

o   Based on this assessment, they rate the application using a scale of 1 (low) – 10 (high).

o   The ESRC will analyse these scores and work out approximately what score the applications need to have got in order to go forward to the GAP meeting. Roughly 30% of applications get rejected at this stage, and realistically only those scoring 6 or above are likely to be funded.

•  Fourth Stage – The Panel Meeting

o   Each application is introduced by the two GAP members. It is not always clear beforehand who will be the leading introducer and who will be the seconder.

o   The panel works through the applications in order of introducers’ scores: the ones with the highest score get discussed first, those with a lowest score get discussed last.

o   The Panel works with PDFs which save on paper but do make it difficult to refer back, check, and follow the discussion of applications easily.

o   ESRC officers are present, and do have input:

§  By saying roughly how many applications they can fund in that round;

§  By highlighting any problems with applications that have not been picked up before (eg they have been submitted to the Council before)

o   Whilst the panel will take a steer from the introducers, the panel discussions allow for proposals to be pulled up or down the rankings. Most of the discussion is around marginal or controversial proposals.

o   The Chair is key. S/he summarises the discussion and, if there is no consensus, has the final say.

Notes from a Mock Panel

'Save me from my friends'. Never truer than in the bearpit
of academic peer review

I took part in the University's PGCHE research funding module today, which took participants through a 'mock panel'. This is always a useful exercise as it gives potential applicants a feel for the issues that peer review panellists are having to grapple with in real life.

There were about thirty participants, and we divided them into five panels: one of humanities academics (looking at AHRC applications), two of social scientists (looking at ESRC applications)  and two of scientists (looking at EPSRC applications).

In preparation I did some reading around the ESRC process, and I'll write these up in a post shortly, but in the meantime it's worth noting some of the key points that came out of the session:

• Firstly, no matter what your discipline, you can spot the weaknesses in the applications. This is interesting: I chaired one of the social science panels, and all those who took part were in disciplines that were very different from those of the applicants. Nevertheless they picked up on a lot of the weaknesses, and were pretty accurate in the ranking of the applications.
• Secondly, seniority matters. We pretend it doesn't, but if a PI has an impressive CV, we're more likely to let some vagueness in the application slide. This was a bit disheartening for the ECRs present. However, I pointed out that two of the best applications under consideration were from ECRs who had overcome this difficulty by either have a strong, robust and well thought through project design, or had more senior co-investigators on board to give it gravitas and reassurance. 
• Thirdly, time is short. You imagine the panellists have all the time in the world to consider your application, but time's snapping at their heals. Decisions need to be made, compromises struck, and the agenda moved on. You - as the applicant - have to help the introducers by giving the the information they need to support your application in a format and place that they can grab it quickly. Cut to the chase: what's your research question, why's it important, why's it timely, why are you the person to answer it, how are you going to do it, and how are you going to disseminate it.
• Fourthly, confidence shines through. If you believe in yourself and your research, it really helps. Don't be tentative, uncertain or - let's be frank - academic. You need to sell your proposal, and to do so you've got to believe in both its worth, but also in its achievability.

Peer Review: Roulette, Black Magic...or Fair & Transparent?

The next Grants Factory event will focus on ‘how the peer review panel works,’ and will take place on Wed 30 January at 12noon in Cornwallis NW SR6.

Prof Mick Tuite

Dr Simon Kirchin

To many the assessment of their well-crafted proposal by the funder is somewhere between a game of roulette and a black magic session. A sense of randomness, of unfairness, hangs over the system. But what really goes on? Are these concerns justified, or are the panellists doing the best they can with limited resources? This session will hear from two Kent academics who have had considerable experience of sitting on peer review panels. Prof Mick Tuite (Biosciences) has had experience of the BBSRC and the Wellcome Trust, amongst others, and Dr Simon Kirchin (SECL) has sat on panels for the AHRC. Both will talk about how these panels function, what they look for in proposals, and how decisions are made.

The event is free for staff at the University of Kent and lunch will be provided, but do let me know if you intend to come as places are limited.

ESRC Seeks New Panel Members

The ESRC’s peer review panels, which assess grant applications, are seeking new members in the following areas:

• Sociology, particularly sociology of health
• Socio-legal studies
• Science and technology studies
• Management and business studies, including accounting and finance
• Economics, particularly micro-economics

I would strongly encourage you to consider putting yourself forward, or suggest this to members of your School. The insights you will get as to the decision making process, as well as finding out about work going on at other universities, is invaluable. Deadline is 1 February 2012. More information is available here.

Hail the Kingmakers!

There has been an interesting development at the National Science Foundation (NSF) in America. Following on from their radical rethink on gathering metrics, the NSF has applied its imagination to the process of assessing grant applications.

According to Science Insider, the NSF will be doing away with external peer review for a new scheme that will fund unorthodox ideas. Applications to the Creative Research Awards for Transformative Interdisciplinary Ventures (CREATIV: an acronym that is almost European in its creakiness), which will provide grants of up to $1m for up to 5 years, will be judged by NSF program managers. Cutting out the academic review will dramatically slash the turn around time: applicants can expect a decision in 2 or 3 months, which is half the time it takes at the moment.

Is this a worrying step towards cutting out peer review completely? No, says the NSF. Richard Behnke, co-chair of the internal committee that designed CREATIV, said that 'for the great majority of proposals, we will continue the traditional merit-review process. The gold standard remains in place.'

Whilst it doesn't affect UK researchers directly (only US institutions can apply), I wonder whether UK funders will be looking at the experiment with interest. After all, the Research Councils are under pressure to find ways to save money, bureaucracy and time. This could be the answer to their prayers. But, to be honest, I don't think they'd dare. When I worked at the AHRC it was often very clear to us which applications were likely to rise to the top of the prioritisation lists. However, applicants need to be reassured that it is primarily on research quality that their proposals are being judged, and that reassurance can only come from peer review.

I'll be intrigued to know how the NSF scheme works. I wish them all the best, but I think that they'll be creating more problems than they'll be solving, and will be snowed under with appeals and complaints. The NSF officer-kingmakers will, I think, have only a short while in the sun.

The Mists Are Clearing...I See a Project of National Importance...

As ever, EPSRC is blazing a trail in developing new hurdles for potential applicants. Hot on the heels of controversially 'managing' its remit and 'managing' demand via its blacklist, the Council has just announced that, as of 15th November 2011, all applicants will have to identify the national importance of their research.

Yes, it's both Scientific Excellence AND national importance that will now be the primary assessment criteria. Impact, track record, resources and management will be secondary assessment criteria.

Reviewers and panel members have been issued with new assessment criteria/forms to reflect these changes. For applicants, the national importance section must now be included in the case for support.

Is that a collective groan I can hear from the sector? Having just got used to the concept of predicting their potential impact, they're now going to have to predict its importance.

I do like the amount of faith that EPSRC has in the visionary abilities of the scientists within its remit. I'm surprised, however, that they haven't put these abilities to better use and got them to predict the numbers for the National Lottery? It would be an excellent use of the collective brain power of EPSRC scientists if they could rustle up the Euromillions necessary to meet the 10% funding cut that resulted from the CSR flat settlement.

From Me to You: ESRC Panellists Advice to Reviewers

The ESRC has recently held workshops - or 'masterclasses' - for members of its peer review college. These involved some of the Grants Assessment Panel (GAP) members talking about their experience, about what they have to bear in mind when assessing applications, and on the importance of the reviewers in the process.
As you can imagine, this provided some interesting insights. The GAP members were generally grateful to the reviewers, and recognised their reliance on the reviewers' knowledge to make their decisions. Some points to highlight:

Firstly, the process itself:

• Each application gets sent to at least 3 academic reviewers, and a user reviewer (if relevant)


• If the average score for these is above 4/6, they get sent to GAP members who will act as 'introducers' at the panel meeting. Introducers usually get 7-10 proposals each to assess each meeting, and 4-5 weeks to write their assessments.


• Each application will have 2 introducers assessing them. If the average introducers' score is above 4/10, they go to panel. Only those scoring 6 or above are likely to be funded. Thus, as I've said before, it's worth noting how important the introducers are.


• However, the panel discussions allow for proposals that fall below this to be pulled up the rankings.


• Most of the discussion is around marginal or controversial proposals.

What are the core assessment criteria for reviewers:

• Scientific quality and intellectual contribution;


• originality and potential contribution to knowledge;


• timelineness;


• robustness of research design and methods;


• value for money;


• outputs, dissemination and impact.

So what should reviewers bear in mind when assessing an application?

• Give yourself enough time to properly assess the application.


• Judge only what is written, not what you imagine the project to be. If the applicant hasn't made clear what they're going to do, that is their fault.


• Base your judgement on the research question the applicant asks. Is it interesting/important? Are the methods appropriate for answering the question?


• Evaluate, don't advocate. Not every proposal can be funded. Be frank, and indicate risk versus benefit.


• Justify your arguments, and provide constructive criticism. It's important that (a) the panellists understand why you have scored as you have, and that (b) your score matches your comments. And, of course, it's useful for the applicant if they are rejected.

And what should they not do?

• be personal or aggressive;


• be too brief or too verbiose;


• be ambiguous;


• make inappropriate, irrelevant or polemic remarks.


• forget that reviewing research proposals is different from reviewing papers: here, the research is speculative, so you have to evaluate the likely results;


• forget to draw attention to ways in which the proposal meets specific assessment criteria particularly well, and to point to any major logical flaws, contradictions or omissions;


• forget what the point of the review is, namely to weigh up the positive aspects of the proposal against the negative ones.

After the workshops the ESRC has updated its website with FAQs and a checklist (short and long). It's worth having a look at these, to get an idea of what the reviewers will be considering when they start to read your proposal.

Call for Nominations to AHRC Peer Review College

The AHRC is seeking nominations for its Peer Review College. I would encourage all research staff in relevant areas to consider putting themselves forward. Assessing applications for a funder will help to raise your profile nationally, as well as being a useful way of getting an insight into how the funder works, keeping abreast of what work is being done in your discipline, and gaining an understanding of what it takes for an application to get funded. More details of the call for nominations is available here.

Applications are sought from academics at all stages of their career and, if chosen, you will serve a four year term. If you want to be nominated do get in touch with your CV. I will pass your details on to Prof John Baldock, the PVC Research, who will put forward nominations on behalf of the University.

Responding to Reviewers' Comments: Notes from Grants Factory Session

What turns panellists against an application? For Peter Bennett, who has had experience reviewing applications for NERC and other funders, it’s arrogance. ‘There’s no greater turn off in a grant application,’ he said at Wednesday’s Grants Factory event. Applicants should practice humility, and let the facts speak for themselves. Avoid bombast, pomposity and exaggerated claims. Successful proposals talk with assurance, clarity and confidence, and they respect the opinions of reviewers and panellists.

A good application ‘makes sense’, and doesn’t need to swagger. It will be built on a strong track record and a well-matched research environment, will acknowledge the key previous works, and will contain an unbreakable kernel of original, significant work.

Bad applications, on the other hand, are bloated beasts. Pedestrian and dull, they shout their ignorance, and betray their hasty construction with flawed methodology, speculative theory, and a lack of focus. The applicant’s limp track record drags along behind, and the whole just doesn’t hang together well.

When it comes to replying to the reviewers, then, it is the measured, thoughtful, clear attitude that will win through. Don’t flare up and respond in haste. Step back, take time, and plan your response. Extract the criticisms from the text, and work out how you will respond to them. All of them should be treated as valid, even if you feel that some are ridiculous. Remember, humility: thank the reviewers for their comments, and either:

• Address their concerns head on: if their feedback is valid, say that you have taken it on board, have made the necessary changes, and that the proposal is stronger as a result;
• Sweeten the pill: if their feedback is invalid, say that you consider it to be an interesting idea, but that the nature of the current project would not allow you to incorporate their suggestions, and that it might be possible to do so in a future project.

Respond to all the comments with humility, respect and honesty; it will get you much more of a hearing than if you rail and swear and curse.

Peter’s slides will be available on the Research Services website shortly.

Grants Factory: Responding to Reviewers' Comments

We’re currently putting together the Grants Factory programme for the forthcoming year. One of the first events will look at the thorny issue of Responding to Reviewers’ Comments. Most of the Research Councils (and some other funders) allow you to respond to the comments that their reviewers give your application. Often the comments seem uninformed, and it’s easy – and tempting ! – to respond quickly and angrily to these. However, there is a knack to dealing with them effectively, and if you can master this you can turn round the negative feedback and actually get your applications funded.

The workshop will take place at 2pm on 14 September 2011 and will be led by Dr Peter Bennett from the Durrell Institute of Conservation Ecology (DICE). He has had considerable experience as both an applicant, reviewer and panel member for NERC, as well as a number of other funders. His insight and advice will be relevant to academics in all three faculties, as there are common themes that apply to funders in all disciplines.

The event is free and open to all staff. Do let me know if you would like to come along.