Automatic for the People

The second set of notes from yesterday's ESRC Seminar Series on Impact looks at the efforts made by the USA's National Science Foundation (NSF) to automate the collection of science metrics.

Julia Lane, Program Director for the Science of Science and Innovation Policy, gave an overview of background and development of the 'Star Metrics' system. As in the UK, the 17 federal funding agencies were asked to justify the investment the government had made in science.

Refreshingly, rather than offloading this burden on to individual researchers (as is currently happening with the RCUK ROP system), the NSF decided that:

• the information should be harvested automatically and electronically;
• the system should be voluntary.

I know. What were they thinking? What we need is mandatory forms, and lots of them! Do they know nothing about research funding management?

But no, they were thinking very logically. After all, in the twenty first century, when the internet allows us to order our groceries, book our holidays and buy our road fund tax, why can't it be used to automatically gather information on impact?

Thus, they created a system that does the following:

1. Follows the trail of grants through individual HEI financial system. This can tell them: who is funded (via the HR system), including PI, Co-I, RA and students, where the money is being spent (via the procurement system), and who they are subcontracting to or collaborating with (via the finance system);
2. Follows the trail of outputs, by linking with the patent office and publication databases;
3. Follows the individual via various CV systems, such as Vivo, Harvard Catalyst and Eurocris;
4. Analyses the areas funded by the federal agencies by scanning and machine reading the applications, and doing a key word analysis.

As a result Star Metrics can be used to identify:

• What expertise there is in a particular area, or at a particular site;
• Where there is a shortage of expertise;
• How much funding has been put into any discipline area;
• Areas of overlap between funders;
• What has been funded in any geographical location (such as a state);
• What has been funded in any institution, or for an individual;
• What local or national businesses have benefited from the funding;
• The outputs and outcomes from any funded project;
• The career development of anyone associated with the project.

And all, as Lane said, without the academic having to lift a pen. Now how refreshing is that?

RCUK's New System ROP-ey

Following my previous post, I've heard on the grapevine about some - ahem - 'discontent' with RCUK's Research Outcomes Project (ROP), which is due to go live on 14 November.

Whilst the RCUK is still whistling in the dark about its new system, both the Russell Group and 94 Group have written to RCUK asking them to think again about implementing it in its current form. One person I spoke to said that working with the system 'will make you cry'. Whilst they had issues with MRC's e-Val system, it was user friendly in comparison to the ROP. Apparently the reason that the ESRC-based system was chosen in preference to the MRC one was that it was the cheapest to develop.

The main issue with ROP is that it does not allow 'bulk uploads'. Thus, universities will not be able to centrally load details of all the outcomes from all their RCUK-funded projects. Each academic will have to enter details of the projects separately. Worse still, the new system does not currently have the capacity to link to an institution's own repository - such as the Kent Academic Repository (KAR) - so academics will end up having to enter the data twice.

The result will be that research offices will spend a lot of time chasing academics to fill in their details - individually - and that academics will spend twice as long as they need providing their data - individually. And, if push comes to shove, the academics might decide that they will only provide the obligatory ROP data rather than (say) the requested HEI data, which may foul up institutional preparations for the REF.

There's still a possibility that RCUK will find a way to allow bulk uploads, and they may even find a way of linking to individual institutional repositories. But it's unlikely to do so by 14 November.