Understanding and evaluating the impact of integrated problem-oriented research programmes: Concepts and considerations

Brian M Belcher and Karl Hughes

Research Evaluation, 2021, vol. 30, issue 2, 154-168

Abstract: Researchers and research organizations are under increasing pressure to demonstrate that their work contributes to positive change and helps solve pressing societal challenges. There is a simultaneous trend towards more engaged transdisciplinary research that is complexity-aware and appreciates that change happens through systems transformation, not only through technological innovation. Appropriate evaluation approaches are needed to evidence research impact and generate learning for continual improvement. This is challenging in any research field, but especially for research that crosses disciplinary boundaries and intervenes in complex systems. Moreover, evaluation challenges at the project scale are compounded at the programme scale. The Forest, Trees and Agroforestry (FTA) research programme serves as an example of this evolution in research approach and the resulting evaluation challenges. FTA research is responding to the demand for greater impact with more engaged research following multiple pathways. However, research impact assessment in the CGIAR (Consultative Group on International Agricultural Research) was developed in a technology-centric context where counterfactual approaches of causal inference (experimental and quasi-experimental) predominate. Relying solely on such approaches is inappropriate for evaluating research contributions that target policy and institutional change and systems transformation. Instead, we propose a multifaceted, multi-scale, theory-based evaluation approach. This includes nested project- and programme-scale theories of change (ToCs); research quality assessment; theory-based outcome evaluations to empirically test ToCs and assess policy, institutional, and practice influence; experimental and quasi-experimental impact of FTA-informed ‘large n’ innovations; ex ante impact assessment to estimate potential impacts at scale; and logically and plausibly linking programme-level outcomes to secondary data on development and conservation status.

Keywords: research evaluation; impact assessment; evaluation tools; theory of change; transdisciplinary research; sustainability science (search for similar items in EconPapers)
Date: 2021
References: Add references at CitEc
Citations: View citations in EconPapers (2)

Downloads: (external link)
http://hdl.handle.net/10.1093/reseval/rvaa024 (application/pdf)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:oup:rseval:v:30:y:2021:i:2:p:154-168.

Access Statistics for this article

Research Evaluation is currently edited by Julia Melkers, Emanuela Reale and Thed van Leeuwen

More articles in Research Evaluation from Oxford University Press
Bibliographic data for series maintained by Oxford University Press ().