Impact: the Wellcome View

I went along to the ESRC Seminar Series on Impact at UCL yesterday. It was a very interesting event, and wasn’t what I had feared: either an evangelical sermon by the funders, or a diatribe by academics. Instead, we were presented with four different takes on impact: the first, from the Wellcome Trust, the second from the American funder the National Science Foundation (NSF), the third from the Netherlands on research assessment, and fourth from the coal face by a practising scientist. I’ll cover two of these – the Wellcome Trust and NSF – in this and the next post.

The Wellcome Trust spends £650m per year, roughly equivalent to the MRC. Liz Allen, the Senior Evaluator Adviser at the Trust, outlined the challenges faced and tools used in understanding and quantifying the effect of this funding.

Wellcome, of course, does not have to justify itself to government; however, it does have a duty to report to the Charity Commission, which gives it tax breaks, and to understand what does and doesn’t work.

She highlighted a number of challenges in assessing impact in the biosciences:

• the time frame involved: for instance, it took Robert Edwards 18 years from first developing the technique for IVF to the first successful ‘test tube baby’
• the serendipity of science: for instance, Alec Jeffreys has been quoted as saying ‘our discovery of DNA fingerprinting was of course totally accidental...but at least we had the sense to realise what we had stumbled upon.’
• Attribution and contribution: there is often a long and diverse list of people and organisations involved in the evolution of a piece of research. For instance, John Todd had funding from Wellcome, MRC, JDRF and NIH. In addition, there is the ‘ripple effect’, the value of negative findings, and the ‘counter factual’ question: what would have happened if x didn’t discover y?

So pinning down impact is not easy. However, Allen, who was keen on quotes, quoted Charles Babbage: ‘errors using inadequate data are much less than those using no data at all.’ It’s better to do something with limited data than to do nothing at all. Wellcome’s attempt to ‘do something’ includes:

• Understanding impact in terms of ‘progress’ rather than ‘success’;
• With this in mind, the Trust looks for traditional indicators of progress, including: publications, people and training, products and interventions, software and databases, engagement with communities, policy, advocacy and influence, funding, and awards and prizes.
• In addition, it has explored ‘new’ indicators, such as article-level metrics (through software such as PLoS), post-publication peer review (such as F1000), clinical guidelines (NICE has recently digitised all citations used for clinical guidelines, and the Trust can check who has been involved), and the separate contributions of all members of a research team (via the ORCID initiative).
• The Trust is also making use of ‘softer’ indicators, such as case studies. However, it recognises the dangers inherent in this. Allen once again turned to another quote, this time from John Allen Clements: ‘when you ask the memory to reconstruct daily events from forty years ago, you’ve got to be appropriately cautious.’ The Trust needed to make the award winners know what it might be interested in right from the start, so that they can note and monitor it.
• Wellcome Trust Career Checker. This is a recent development, and aims to track the career choices and progression of cohorts of individuals after their funding has finished. What have they gone on to do?

Allen finished by highlighting the work that others had done in trying to quantify the effect that medical research had had. The Lasker Foundation had commissioned research that suggested that it had had a 20 fold return, so that for every $1 spent there was $2.17 in health benefits. A similar report in the UK showed that ‘for every £1 of public money invested in cardiovascular disease and mental health research, a stream of benefits is produced equivalent ot earning 39p and 37p respectively each year in perpetuity.’

Which is quite compelling. Sure, assessing the effect of biomedical research is a tough call, but Allen rounded off with a quote from Mary Lasker: ‘if you think research is expensive, try disease.’