Can and should science-for-policy systems be mapped and measured?

The Joint Research Centre is engaged in a long-term project focused on how to support evidence use in policy making across the EU. They invited me to spend some time thinking about the role of the science system, and how we might know what a ‘good’ science system looks like. This is partly about mapping science systems and trying to understand what makes them work; and also knowing how we can tell if they are functioning, by using indicators.

There are two main problems with setting up an indicator dashboard to assess science systems. First some relatively straightforward technical issues: we often measure what is measurable, which isn’t necessarily the same as what is useful to know; and the potential for perverse incentives, or even just over-attention to what is measurable, is always a risk.

The second challenge is more existential. Does the science system have a goal? Visualising and mapping science systems perhaps inevitably gives the impression that the system was designed, can be controlled and adjusted, and that it operates towards a shared objective. None of this is true of evidence/policy systems, which tend to have evolved organically. And members of this system have their own agendas and interests - not all researchers produce policy-relevant research, for instance. So, no – in itself, the system does not have a clear goal.

However, the JRC project is about supporting evidence use in policy. And the literature on evidence-informed policymaking does have a relatively shared understanding of what this means: the delivery of relevant, robust evidence to people with the capacity to absorb and fully understand it in order that it might inform their decision-making. This is not a simple goal - big questions about what evidence is produced, what evidence counts for policy, how evidence and expertise reaches decision-makers, and how they can and do respond to this evidence all need to be considered; and all these questions have political, socio-technical and philosophical aspects to them.

It is possible, though, to think about which types of organisations and individuals are involved in evidence production, mobilisation and use in decision-making; and to ask about their connections and activities. In fact, we can start to think about the total set of actors and connections within the entire system through which scientific knowledge is acquired, synthesised, translated, presented for use, and applied in the policymaking process. This is my working definition of the science-for-policy system. In the table below, I set out my preliminary thoughts about what these different groups might do in service of evidence-informed policymaking, and how we might know they’re doing it (well). How these signs of activity might translate into measurable indicators, I am not yet sure.

Although there are probably missing groups, activities, and indicators, mapping these out shows at least 3 interesting things:

1. Whether the groups and organisations involved agree on the goal of evidence-informed policymaking, or want to act in its service, all groups currently contribute towards this goal
2. Mapping out the connections in this way offers potential points of intervention which could help improve systems functioning
3. But interventions which seek to improve evidence use usually target one element or relationship, rather than at a systems level (which tends to add to a chaotic mass of activity rather than improving systems functioning)

More thinking on these activities, goals, and indicators is needed, particularly around what we want our science-for-policy systems to do, for whom, and how.

If you see gaps in the groups, activities, and indicators, or have thoughts on what we want our science-for-policy systems to do, please get in touch.