Policy evaluations: better together?

Introduction

This blog draws on the experience of three policy evaluations, undertaken by university research teams, of new NHS services for type 2 diabetes. These were large, complex policy evaluations of national initiatives.

Who is policy research for? What is its purpose?

Policy research is often undertaken with the intention of generating insights about a policy or service to inform decisions on whether to continue, adapt or change course.

Another purpose of policy evaluation is to offer broader insights, adding to the international body of evidence and influencing how other services are developed in the future.

Our evaluations were important to NHS England and other stakeholders such as service users and service providers, but there were also important lessons for international audiences on how to roll out these services at a national scale.

At the outset we expected to design a rigorous evaluation that was independent of the NHS. Over time we realised that loosening the ‘independence’ strings allowed us, as outsiders, to learn from the NHS’s immense experience with these diabetes programmes. Regular interaction provided us with opportunities to influence NHS services, leading to greater impact.

Three degrees of separation

We have found there was more than one way for researchers and policy makers to manage the tension between independence and interaction. Our relationships evolved over time, rather than being defined a priori (https://doi.org/10.1111/dme.15238). Across different phases of our research relationships varied, on a scale from quite distant to close partnership. We describe this as three degrees of separation.

• Maintaining distancene aspect of our research was to explore whether these new services improved patient outcomes, using statistical analysis of large primary care databases. Our evaluation teams maintained fairly strict independence from NHS England, adhering to pre-specified analyses plans. It was important to both us and NHS England that we produce findings that were perceived as impartial and robust.

2. Getting closer

Another aspect was to analyse the intervention content to see if it had fidelity to what NHS England had promised to deliver and fidelity to the evidence base on what works in changing health behaviours. This involved studying intervention manuals, interviewing service users and staff, and observing service delivery. This fidelity research started early in the evaluations and continued throughout, relying on NHS England and service providers for access to materials and participants. Our researchers provided rich descriptive analyses, delivered quickly, which NHS England could use to make real-time changes to the services. Researchers worked alongside NHS England in practical ways. For example, behaviour change experts were co-opted by NHS England into the procurement and re-procurement of the providers of the interventions across England. 

3. Close partnership

We worked in full partnership with the NHS England Diabetes Expert Reference Group and their analysts to see whether the effects of digital and face-to-face treatment delivery were equivalent. From the start this was a joint initiative: we agreed to combine our strengths to produce a better piece of work that we could each achieve alone. We agreed a memorandum of understanding for access to data and jointly planned the research. Unlike other work packages, this resulted in publications co-authored between researchers and NHS England.

Conclusion

There is more than one way for researchers and policy makers to work together successfully and we have suggested three models to consider. Each of these is suitable in some circumstances, but there is no one-size-fits-all. While our relationships developed organically along the way, we suggest that you (unlike us) might want to consider the options and make plans with your partner in advance. Whether or not you plan ahead, you will need time and some scope for flexibility when working with partners.

This blog draws on the experience of 3 evaluations: DIPLOMA evaluation of the NHS Diabetes Prevention Programme; Re:Mission evaluation of the NHS Type 2 Diabetes Path to Remission Programme; HED-LINE evaluation of NHS Healthy Living for people with type 2 diabetes. The themes for this blog were developed at a workshop in Manchester, May 2023, involving the research teams, policy makers from NHS England, members of the public involved in the three studies, Diabetes UK, and service providers.

These projects were funded by the National Institute for Health and Care Research (NIHR): Health and Social Care Delivery Research (HSDR) Programme (16/48/07 DIPLOMA; NIHR132075 Re:Mission) and Policy Research Programme (NIHR200933 HED-LINE). The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.