Reflecting on 'Making the Grade'

When we first wrote Making the Grade: Methodologies for Assessing and Evidencing Research Impact back in 2013, we could not have imagined the journey it would take. At the time, the UK was preparing for its first Research Excellence Framework (REF) exercise to include impact as part of the national research assessment, and conversations about research impact were still relatively new in many quarters. Our aim was modest: to capture the methodological challenges we had encountered in our work and to offer a way of thinking about purposes for assessment, through what became known as the “4 As” — advocacy, accountability, analysis, and allocation. Our argument was simple: purpose matters and the choice of assessment methodology (or methodologies), must be informed by this purpose.

Over ten years on, and as the chapter is now being republished, we are both surprised and grateful for the way the piece has continued to resonate within the research evaluation community.

A paper that kept travelling

Although published in what was then a small collection of essays by the University of Exeter, the chapter has been cited widely in policy and academic literatures. Colleagues have drawn on it to frame their own approaches, to argue for methodological pluralism, and to make sense of the trade-offs we highlighted. Others have even extended the model by adding new “A’s” – such as 'adjustment' and 'achievement' – a development we take as a sign of healthy debate and creativity in the field. We continue to find ourselves using it in teaching and workshops.

It is humbling to see the “4 As” still used as a shorthand for clarifying the objectives of research impact assessment exercises. What began as a practical framework to organise our own thinking has, in the hands of others, become a touchstone for structuring discussions about why impact matters and how it might be evidenced.

Why it endures

Looking back, a few things may explain why the chapter has had staying power:

• - Practical honesty. We did not try to provide a grand theory or a single solution but rather set out the challenges and trade-offs we had encountered.
• - Flexibility. By emphasising that method must follow purpose, the paper allowed others to adapt the ideas to different disciplines, funders, and national systems.
• - Openness. The “4 As” were never intended to be final — and the fact that others have added to them shows that the field is still evolving.
• - Alliteration. The power of this linguistic trick never fails to amuse us. It should not be underestimated!

Looking ahead

The republication of Making the Grade offers an opportunity to revisit these ideas in light of what has happened since 2013. New methodologies such as text mining have tested some of our assumptions, while large-scale databases such as the those from the UK’s REF impact case studies, Hong Kong’s research assessment exercise, and other repositories have given all of us a new evidence base for thinking about impact. We remain convinced that the central challenge is not just methodological, but conceptual: how we as a community define value, and who gets to decide what counts as impact.

As authors, we are humbled by the continued interest in this piece. Our hope is that the republication helps a new generation of researchers, funders, and evaluators reflect on how best to capture the many ways in which research makes a difference. We look forward to being part of those conversations.