The Difficulty of Impact

Definitely not David Sweeney

The LSE'Future of Impact' conference yesterday had a valedictory feel. It was the last huzzah for ‘The Impact of Social Sciences’, a HEFCE-funded project which had sought to 'develop precise methods for measuring and evaluating the impact of research in the public sphere.' 

Whilst they have achieved a lot – their blog is a fantastic forum for discussing issues relating to impact, for example – I think even their fiercest admirers would be hard pressed to say they had succeeded in this.

That’s not necessary their fault. Rather, it's the inevitable consquence of grapling with the amorphous, shape-shifting beast that is impact. This was particularly apparent in the third session yesterday, ‘Next Steps in Assessing Impact,’ which saw the three speakers almost come to blows over what impact is, who wants it, why, and how fast. This was partly down to the ever-entertaining David Sweeney, the architect of the REF. Like an embarrassing uncle at a wedding, he can always be relied upon to speak his mind. Loudly.

He took issue with the previous speaker, Julia Lane, who had been talking about StarMetrics. I’ve spoken about this before on this blog. To me, it sounds like an eminently sensible system (although I don’t think she did it justice here); Sweeney, however, begged to differ. Not only did he think that academia should not snap to attention when governments ask for data, but he implied that the system was only partially successful at collecting the right information. 

Similarly, he turned on the third speaker, Cameron Neylon. Neylon had suggested that Twitter could be used to monitor and engage with users of academic research. He gave an example of South African research which had been retweeted by someone working in community health promotion. ‘That’s not impact’, said Sweeney, dismissively.

And therein lies the problem. Here were three eminent speakers working at the coal face of impact. And yet, between them, they couldn’t reach agreement on what constituted impact, or why we should be doing it. If they have problems defining impact, what hope is there for the rest of us?  

Automatic for the People

The second set of notes from yesterday's ESRC Seminar Series on Impact looks at the efforts made by the USA's National Science Foundation (NSF) to automate the collection of science metrics.

Julia Lane, Program Director for the Science of Science and Innovation Policy, gave an overview of background and development of the 'Star Metrics' system. As in the UK, the 17 federal funding agencies were asked to justify the investment the government had made in science.

Refreshingly, rather than offloading this burden on to individual researchers (as is currently happening with the RCUK ROP system), the NSF decided that:

• the information should be harvested automatically and electronically;
• the system should be voluntary.

I know. What were they thinking? What we need is mandatory forms, and lots of them! Do they know nothing about research funding management?

But no, they were thinking very logically. After all, in the twenty first century, when the internet allows us to order our groceries, book our holidays and buy our road fund tax, why can't it be used to automatically gather information on impact?

Thus, they created a system that does the following:

1. Follows the trail of grants through individual HEI financial system. This can tell them: who is funded (via the HR system), including PI, Co-I, RA and students, where the money is being spent (via the procurement system), and who they are subcontracting to or collaborating with (via the finance system);
2. Follows the trail of outputs, by linking with the patent office and publication databases;
3. Follows the individual via various CV systems, such as Vivo, Harvard Catalyst and Eurocris;
4. Analyses the areas funded by the federal agencies by scanning and machine reading the applications, and doing a key word analysis.

As a result Star Metrics can be used to identify:

• What expertise there is in a particular area, or at a particular site;
• Where there is a shortage of expertise;
• How much funding has been put into any discipline area;
• Areas of overlap between funders;
• What has been funded in any geographical location (such as a state);
• What has been funded in any institution, or for an individual;
• What local or national businesses have benefited from the funding;
• The outputs and outcomes from any funded project;
• The career development of anyone associated with the project.

And all, as Lane said, without the academic having to lift a pen. Now how refreshing is that?