Exploring societal impact: "You need to bring two worlds together."
As part of PÕ¾ÊÓƵ’s strategic partnership with The Association of Universities in the Netherlands (VSNU), we’re interviewing researchers from a whole range of disciplines about their experiences of creating societal impact through research.
In this interview, we speak to who combines her chair at Vrije Universiteit Amsterdam with the directorship of and an honorary professorship at UCL. She explains how she works with policymakers and educators in carrying out her research.
First of all, can you tell us a bit about your own research and where you think it links to societal impact?
My research looks at educational accountability systems and how these improve schools and education systems. So that could be school inspections, high-stakes testing, or any other type of monitoring of schools. And that also links into educational governance – looking in a wider sense at how schools are governed and how that improves education. Ultimately, my work is all about improving schools and so it has the potential to have a direct societal impact.
Can you tell us about some of the challenges you face in ensuring your work has an impact?
The biggest challenge is: how can you bring together two very different worlds? The world of research, which tries to understand the nuance and complexities of how something works and then the world of policymakers and inspectorates of education who are often looking for simple, succinct messages on what to do next.
That's the complexity in which you're operating if you're trying to have an impact. You need to try to bring two worlds together. I often get asked questions like, what is the best system? What is the best way to inspect schools? How should we hold schools accountable? Policymakers and inspectorates of education are often looking for the so-called ‘silver bullet’ – the imagined ‘best’ system.
But my own understanding, after so many years of research, is that there’s not really one best system. Systems operate within a context. And all these contextual variables affect how accountability systems, including inspections, operate and the type of effects that they have.
What has been your experience of working with policymakers and school inspectorates? How have you built relationships with them?
Inspectorates of education are often concerned about having their work researched because they operate in a highly politicised area. That makes this a challenging topic in which to have an impact because getting access to inspectorates of education is difficult. And then when you do have findings, being able to communicate about them openly requires a lot of diplomacy.
In the various projects I’ve been involved with, the crucial thing has been to communicate with inspectorates of education from the start of a project. Rather than waiting until you have findings and then sharing them.
It’s almost co-creation of the research. So, from the start, it’s vital to establish some contact within the organisations you’re wanting to work with. The aim should be to have frequent conversations about the research, the questions that you want to answer, the structure and setup of the research and how you're going to communicate about results.
And that latter point is particularly important. What we've done throughout our work is to make really good agreements on how we’re going to communicate research findings. So we’re responsible for what comes out of the research, but we do that in good conversation with inspectorates of education.
This is all about establishing trust and good relationships. And while that doesn't necessarily create an impact in and of itself, it's a precursor to it. In my field, any influence my research can have is in influencing policymakers and inspectorates. So if I communicate about my research in a way that antagonises them, then immediately I’ve lost my opportunity to have an impact.
You’ve mentioned the idea of co-creation of research and how that can help with building those relationships – can you tell us a bit more about how that can work in practice?
This isn’t always possible – depending on the aims of your research – but co-creation can be incredibly effective in helping to ensure your research has an impact.
You could include the people you want to communicate with on an advisory board, right from the start of the project, for example. Or even involve them in the research itself – recently we carried out a set of online research labs with inspectors of education, where we explored how the current school closures and Covid-19 pandemic is affecting their work. That then allowed us to create questions for further research together.
That's an involved approach to co-creation – where you establish a set of research questions as a collective and then actually do the research as well. That co-creation around research questions allows them to have their own interests researched, which not only means they’re more bought in – and perhaps more likely to act on the results – but it can also lead you to questions you might not otherwise have thought to include.
And it’s also important to consider this collaboration element when you start collecting and analysing data. I’ve found that sharing findings throughout the study, is the best way to have an impact.
Working in this way and building these relationships has then meant I’m often asked for input when new policies and frameworks are being developed. Of course, all I’m doing is providing research context – there are many other considerations that they take into account too – but building those relationships has given me a ‘seat at the table’.
We’ve talked about the importance of building relationships with policymakers and educators in your work. But are there other ways in which you’ve seen your work have an impact?
Absolutely. One example that springs to mind is teaching. When I worked in England, alongside my other work, I ran a summer school on evaluation and accountability. And we had signups from inspectorates of education across the world.
This meant I had the opportunity to train and teach school inspectors themselves. In that training, I was building on my own research to teach them about effective modes of inspection, organise conversations between them to apply that knowledge to their own work and later take that knowledge back to their own organisation. So that was definitely another way to have an impact.
What advice do you have for researchers when it comes to demonstrating societal impact in grant applications?
When writing applications, it's always very helpful to have a consortium with the main stakeholders. For example, a new proposal that we just submitted is looking at how to solve problems around teacher shortages. And the research consortium includes one large school board and a regional network of schools who are dealing with this particular issue. They're going to be the case studies for our research as well.
Ideally, you want to show that you have partners on a consortium, included at the research design stage and/or an impact strategy that engages key stakeholders throughout the research. I think this gives funders confidence that you’ve really thought through the impact side of your work.
It really comes back to the point I made previously about engaging your target audiences from the very beginning.
And do you have any other advice, especially for early career researchers, who want their research to have societal impact?
I think one of the key things for researchers early in their careers to do is actively seek out opportunities. For example, training, workshops, opportunities to communicate about their research.
Networks are so important as well and so this means early career researchers are, to a great extent, reliant on their supervisors and senior colleagues for support too as they would have the networks that are relevant to you. You need to be purposeful in organising conversations about societal impact with your supervisor or line manager, because they won’t always have that on their radar or agenda.
Finally, I’ve found it really helped me to work outside of academia for a while as well. It helped to build networks and experience of the ‘other side’. However, this is currently very difficult for researchers to do as it can have a negative impact on your academic career – for example interrupting publishing records and even whether or not you can apply for certain grants.
I believe both institutions and funders should, but often don’t, take into account the value of experience outside the world of research. Researchers who have experienced working outside of academia can bring new views, insights and networks to the table. So I hope more researchers will have the opportunity to do this in the future, without it negatively impacting their careers.
Melanie Ehren is a Professor in Educational Governance at the Vrije Universiteit Amsterdam, director of research institute LEARN! and honorary professor at University College, Institute of Education. Her academic work focuses on the effectiveness of accountability and evaluation systems and aims to contribute to a greater understanding of the interplay between accountability and the broader education system in tackling inequality and improving student outcomes. Some of her key projects are:
1. ESRC funded study on ‘Accountability, capacity and trust to improve learning outcomes in South Africa
2. EU Erasmus+/KA2 comparative study on polycentric inspections of networks of schools across four countries.
3. EU-funded comparative study on the impact of school inspections in six European countries.
4. Nuffield funded project on the Nature, Prevalence and Effectiveness of Strategies used to Prepare Pupils for Key Stage 2 Maths Tests.
