One of our core-funded programmes between 2015 and 2020, ‘Integrating research for policy and practice’ (CFS Programme 7), sought to improve our understanding of the impacts of Forest Research’s evidence and advice beyond the research community, especially within policy and practice, how this has been achieved, and how it can be enhanced, particularly through improved dialogue and collaboration.

Research impact evaluation framework

A framework was developed by Forest Research, with consultant Dr Laura Meagher, to improve how we understand, plan, evaluate and communicate the impacts of research, for example the contributions we make to environmental policy and practice. The framework built upon recent research into knowledge utilisation, plus our own experience of impact evaluation, and was piloted using a series of research impact case studies selected to represent the diversity of research projects led by Forest Research.

The framework asks us to consider three core evaluation questions:

Core evaluation questions

1. Impacts: Who or what changed, in what ways, and how do we know?
2. Causes of impact: Why/how did changes occur? Which factors or processes caused impact?
3. Lessons and actions: What lessons can be learned? Which actions should follow to generate impact?

Impacts

Ia. What changed? (Progress towards goals)

1. Instrumental: changes to plans, decisions, behaviours, practices, actions, policies
2. Conceptual: changes to knowledge, awareness, attitudes, emotions
3. Capacity-building: changes to skills and expertise
4. Enduring connectivity: changes to the number and quality of relationships and trust
5. Culture/attitudes towards knowledge exchange, and research impact itself

Ib. Who changed? (Influencers and influenced)

1. Policy-makers (including regulatory bodies; local, national and international)
2. Practitioners (public, private, NGO)
3. Communities (of place or interest, public)
4. Researchers (within and beyond the project and institution)
5. Other

Ic. How do we know? (Evidence and feedback)

• Which indicators and methods should be used, and questions asked, to demonstrate impacts, and progress towards generation of impacts?

Causes of impact

II. Why/how did changes occur?

1. Problem-framing: Level of importance; tractability of the problem; active negotiation of research questions; appropriateness of research design.
2. Research management: Research culture; integration between disciplines and teams; promotion of research services; planning; strategy.
3. Inputs: Funding; staff capacity and turnover; legacy of previous work; access to equipment and resources.
4. Outputs: Quality and usefulness of content; appropriate format.
5. Dissemination: Targeted and efficient delivery of outputs to users and other audiences.
6. Engagement: Level and quality of interaction with users and other stakeholders; co-production of knowledge; collaboration during design, dissemination and uptake of outputs.
7. Users: Influence of knowledge intermediaries, e.g. ‘champions’ and user groups; incentives and reinforcement to encourage uptake.
8. Context and contingencies: Societal, political, economic, biophysical, climate and geographical factors, risks and uncertainties.

Lessons and actions

III. Lessons learned for impact identification and generation?

1. What worked? What could (or should) have been done differently?
2. What could (or should) be done in the future?

The framework can be used in multiple ways:

• at any scale, from small projects led by a single researcher through to programmes implemented by any research organisation, where a number of cases representing the diversity of research activities can be selected for analysis.
• at any stage in the project or programme cycle: at the start to inform planning; mid-way to take stock and plan next steps; and at the end of a project to gain feedback, learn lessons, communicate impacts to external audiences and consider options for the future. (To use the framework at the start of a project, the question ‘what changed?’ might be reframed as ‘what changes are we aiming for, or what would success look like?’)
• by an individual researcher, or a research team or an end-user, but the quality of the findings, and lessons learned, would be enhanced if a wider range of stakeholders were involved.

When piloting the framework, we found that:

• the breadth of impact types stimulated researchers to think beyond ‘instrumental’ impact and identify other changes generated by their work.
• the generic list of ‘reasons for impact’ encouraged researchers to think about interrelationships that had led, or could lead, toward impacts, including less tangible reasons that might otherwise go unmentioned.
• asking for ‘lessons learned’ opened the way for researchers to step back and reflect critically on what had happened and what could be improved in the future.

Use of the framework can potentially transform the overall direction and detailed implementation of a research project, by providing a means to make the process of reflection more explicit, structured, thorough, balanced and credible, in a format that supports internal learning and external communication.

The framework also has the potential to inform changes to culture and procedures at an organisational level. It has been incorporated into Forest Research’s knowledge exchange and impact planning processes.

The framework has been adopted by other researchers and organisations beyond forestry in other applied fields and/or institutes. For example, it has since been used to assess the impact of agricultural research programmes by a government research agency in New Zealand.

Further reading

Journal article describing the framework: A framework to evaluate the impacts of research on policy and practice: A forestry pilot study – ScienceDirect

LSE Impact blog article: How to tell an impact story? The building blocks you need | Impact of Social Sciences (lse.ac.uk)