Building an Internal Peer Review System

ARMA training: The Royal York Hotel

I took part in an ARMA event today. The focus was on how we, as research administrators and managers, could help to improve the quality of applications. I talked about the Grants Factory programme and Kent's internal peer review system. I shared the platform with Linsey Dickson from Heriot-Watt and Sue Coleman from Edinburgh, and it was interesting to hear what they did at their institutions, and how they compared and contrasted.

However, for me the most interesting part of the day was when the delegates talked amogst themselves about what an ideal peer review system would look like, and what challenges they would face in introducing one.  Common themes emerged:

• Timeliness is key. Applicants need feedback as early as possible. Of course, this isn't always possible: funder deadlines might be too tight, or applicants' collaborators may not give access to proposals early enough. Or (whisper it) the applicants themselves might just do things last minute.
• Feedback has to be useful. Well, durr. Perhaps I need to rephrase: reviewers have to be forced to give feedback which can be used. It's not enough to say, 'fine', or 'needs more work.' What applicants need is detailed feedback: what needs changing, and how?
• There needs to be more than one reviewer. This is something that Kent's system includes (our proposals have to be seen by a disciplinary reviewer and a funder reviewer), but some of the suggestions around the room included a user reviewer, or applicants having the opportunity to nominate their own, but having no guarantee that they'd be used. Additionally, reviewers should be compensated for their time, either through some form of (annual) fee, or perhaps factoring in their review work to the workload allocation model.
• There needs to be buy in. This is crucial: any new system has to have the backing of the PVC, the Heads of School and the Directors if Research. If it comes just from the centre, or is seen to be nothing more than a bureaucratic burden, then it's doomed. Moreover, or needs to be seen by the applicants as relevant and 'on their side'. Which brings me on to the final point:
• It should be Faculty/School based. This surprised me, and I don't think I agree with it. However, I understand the point being made: it needs to have ownership by the academics. If it's university wide there's a danger that this will be lost. This may be true at bigger universities, but I feel that at Kent the academic base is small enough for this not to be a problem. In addition, I tried to incorporate some of the school systems that already existed before our peer review system was launched. Nevertheless, we need to be alert to this as a potential issue.

And it's that kind of thing which makes involvement in these kinds of events worthwhile. Although I'm at the front pontificating and pretending I have all the answers, I've got as much to learn as anyone.