Goldilocks Grants

The first, second and third reviewers disagree about the merits
of Goldilock's research methodology

Earlier this year Michael Lauer, the National Institutes of Health’s (NIH) Deputy Director for Extramural Research, wrote an interesting blog post examining the productivity of its funding. He looked for a correlation between the amount of funding a project had received and the number of citations it got. This he described as “citations per dollar”.

Such a stark metric had many rolling their eyes, and the comments that followed questioned the underlying supposition. “Numbers of citations rarely correlate with greatest discoveries”, wrote one commentator. “Citation numbers are strongly biased towards fast-moving areas of inquiry”, said another. A third noted that “the number of citations a paper receives is an extremely error-prone measure of scientific merit.”

Nevertheless, I think it’s worth exploring Lauer’s work further for two reasons. First, because it is entirely appropriate for funders to try to assess the most effective use of their limited resources. Second, because the conclusion he hints at runs counter to the current drive by the majority of funders to back larger and longer projects.

Grands Projets: J'accuse

A previously funded large project.

The Times Higher last month published an article by Prof Bill Amos of Cambridge, who questioned the wisdom of funders moving increasingly to backing big projects, 'taking ever-larger bites from a modest and diminishing funding pot.' By doing so, Amos concluded, the funder 'starves the wider scientific community. Consequently, the next generation of scientists is already leaving, and many undergraduates I have spoken to feel that their chances of getting one of the handful of studentships on offer these days are too low to be worth the effort.'

Amos articulated what many of us have been feeling for some time. In recent years there has been an inexorable move by the Research Councils and the Wellcome Trust to concentrate funding on longer, larger grants. The logic is impeccable:  in order to answer the big questions that will make a big difference to society, we need big teams of big hitters doing big science. Or big social science. Or big humanities. Moreover, big projects don't require proportionately more administration, and by cutting back on all those pesky small grants, funders will be saving a lot of time and money in selecting and monitoring the things.

What has particularly rankled in this is a sense that the funders are shooting themselves in the foot. I always believed that the funders got many, many more bangs for their bucks by funding small grants. Principal investigators tended to makes sure that every pound, every penny counted. Better still, these small grants offered early career researchers a first step on the ladder of research funding. Sure, some funders do offer dedicated routes for ECRs, but they still tend to be be substantial, and the success rate as a result is negligible. Small grants with a high success rate allowed funders to light the blue touch paper and watch as the subsequent careers fizzed and whirred, glittering in a thousand different directions. To many, this initial light has been extinguished.

However, up until now this view was, for me, nothing more than a hunch. This changed last week when I stumbled across 'Big Science vs Little Science: How Scientific Impact Scales with Funding' via Twitter (thank you, Open Access).  Its authors, Fortin and Currie, looked at whether it was 'more effective to give large grants to a few elite researchers, or small grants to many researchers.'

Focusing on the Natural Sciences and Engineering Research Council of Canada and the National Science Foundation of America, Fortin and Currie start by asking what the goal of the funder is. Is it to maximise major discoveries (sometimes criticised as 'photo opportunity science'), or to maximise the 'summed impact' of the community. The authors looked at four measures of 'impact': total number of papers published, number of citations, number of citations for the most highly cited paper, and the number of very highly cited papers.

Interestingly, they found either no correlation or even an inverse correlation between size of grant and impact:  'impact is a decelerating function of grant size'. They continue: 'our results are inconsistent with the hypothesis that concentrating research funds on 'elite' researchers in the name of 'excellence' increases total impact of the scientific community. Quite the opposite: impact per dollar remains constant or decreases with grant size. Highly cited studies are no more likely to result from large grants than from spreading the same funds among multiple researchers.'

Furthermore, 'if maximising the total impact of the entire pool of grantees is the goal, then the 'few big' strategy would be less effective than the 'many small' strategy...funding more scientists increases the diversity of fields of research, and the range of opportunities available to students.'

The time has come to reconsider this passion for large 'ribbon cutting' projects. Whilst funding many small projects might mean that you can't blow your trumpet about unpicking the human genome, it will have a huge effect on a wide, diverse range of research and careers, which might actually have a more positive effect on the health of your discipline - and consequently your profile as a funder - in the long run. Funders, think again.

Encouraging EU-US Research Collaboration

The 2013 EARMA Conference kicked off with a full and frank assessment of the progress on the Bilat project. This had been funded under the FP7 Coordination pillar to try and encourage more involvement of American researchers in European projects. Who wouldn't want that?

Well, a whole raft of American universities, as it turns out. Although over 4,000 American academics and researchers have got involved in FP7 applications, of which around a quarter are successful, 40 per cent pull out during contract negotiations. The reason? The EC's fearsome bureaucracy.

When you think about it, this isn't a great surprise. The Americans have enough on their plates coping with the demands of the NIH, the NSF and other federal funders. To them the EC is small fry, and why should they bother untangling a whole new bureaucracy, and learning a whole new legal, contractual, financial and reporting system? To quote the Kids of Grange Hill, just say no.

And it was clear that the feeling was mutual. Agatha Keller of EU Grants Access spoke about her experience of having to learn the language of the NIH when the universities of Zurich were getting more than ten times as many projects from the EC.

However, all is not lost. Although a large number of American partners drop out during the negotiations stage, 88 per cent of them continue to maintain a relationship with the project. So the funding is not a deal breaker. The US collaborators really want to work with their EU colleagues.

What should be done to make this possible? Well the promised simplifications of Horizon 2020 will definitely help - as long as they materialise, of course. But Prof Manfred Horvat, who conducted a survey on behalf of the EC on its research cooperation with the USA, finished the session with some firm suggestions;

• Raise the visibility of the Framework Programme amongst American universities. This might be easier said than done; most American institutions are very 'mission oriented', and anything that falls beyond this is treated as an afterthought. 
• Get individual European countries to work more closely together, and present a more coherent, unified front. He gave a startling example, from his last visit to Washington, when he went to a number of European embassies, all of whom trumpeted their bilateral agreements with the US in nanotechnology, but none of whom were aware that any of their European colleagues were doing the same. At the very least all 28 EU states could set up shop together in the same building in Washington, so that it became a 'one stop shop' for American HEIs, as well as allowing the sharing of information between member states.
• Encourage 'bottom up' engagement in the US.
• Develop strategies for collaboration in the ERC, ETPs and JTIs.
• Stimulate mobility between US and the EU.

Whether this comes to pass remains to be seen. There has to be the political will to make it happen. With the simplification of H2020, as well as the desperate need to dig the world of the economic hole its in, the time might just be ripe.

Hail the Kingmakers!

There has been an interesting development at the National Science Foundation (NSF) in America. Following on from their radical rethink on gathering metrics, the NSF has applied its imagination to the process of assessing grant applications.

According to Science Insider, the NSF will be doing away with external peer review for a new scheme that will fund unorthodox ideas. Applications to the Creative Research Awards for Transformative Interdisciplinary Ventures (CREATIV: an acronym that is almost European in its creakiness), which will provide grants of up to $1m for up to 5 years, will be judged by NSF program managers. Cutting out the academic review will dramatically slash the turn around time: applicants can expect a decision in 2 or 3 months, which is half the time it takes at the moment.

Is this a worrying step towards cutting out peer review completely? No, says the NSF. Richard Behnke, co-chair of the internal committee that designed CREATIV, said that 'for the great majority of proposals, we will continue the traditional merit-review process. The gold standard remains in place.'

Whilst it doesn't affect UK researchers directly (only US institutions can apply), I wonder whether UK funders will be looking at the experiment with interest. After all, the Research Councils are under pressure to find ways to save money, bureaucracy and time. This could be the answer to their prayers. But, to be honest, I don't think they'd dare. When I worked at the AHRC it was often very clear to us which applications were likely to rise to the top of the prioritisation lists. However, applicants need to be reassured that it is primarily on research quality that their proposals are being judged, and that reassurance can only come from peer review.

I'll be intrigued to know how the NSF scheme works. I wish them all the best, but I think that they'll be creating more problems than they'll be solving, and will be snowed under with appeals and complaints. The NSF officer-kingmakers will, I think, have only a short while in the sun.

Automatic for the People

The second set of notes from yesterday's ESRC Seminar Series on Impact looks at the efforts made by the USA's National Science Foundation (NSF) to automate the collection of science metrics.

Julia Lane, Program Director for the Science of Science and Innovation Policy, gave an overview of background and development of the 'Star Metrics' system. As in the UK, the 17 federal funding agencies were asked to justify the investment the government had made in science.

Refreshingly, rather than offloading this burden on to individual researchers (as is currently happening with the RCUK ROP system), the NSF decided that:

• the information should be harvested automatically and electronically;
• the system should be voluntary.

I know. What were they thinking? What we need is mandatory forms, and lots of them! Do they know nothing about research funding management?

But no, they were thinking very logically. After all, in the twenty first century, when the internet allows us to order our groceries, book our holidays and buy our road fund tax, why can't it be used to automatically gather information on impact?

Thus, they created a system that does the following:

1. Follows the trail of grants through individual HEI financial system. This can tell them: who is funded (via the HR system), including PI, Co-I, RA and students, where the money is being spent (via the procurement system), and who they are subcontracting to or collaborating with (via the finance system);
2. Follows the trail of outputs, by linking with the patent office and publication databases;
3. Follows the individual via various CV systems, such as Vivo, Harvard Catalyst and Eurocris;
4. Analyses the areas funded by the federal agencies by scanning and machine reading the applications, and doing a key word analysis.

As a result Star Metrics can be used to identify:

• What expertise there is in a particular area, or at a particular site;
• Where there is a shortage of expertise;
• How much funding has been put into any discipline area;
• Areas of overlap between funders;
• What has been funded in any geographical location (such as a state);
• What has been funded in any institution, or for an individual;
• What local or national businesses have benefited from the funding;
• The outputs and outcomes from any funded project;
• The career development of anyone associated with the project.

And all, as Lane said, without the academic having to lift a pen. Now how refreshing is that?