Evaluating Evaluations: Assessing the Quality of Aid Agency Evaluations in Global Health - Working Paper 461

Julia Goldberg Raifman, Felix Lam, Janeen Madan Keller, Alexander Radunsky and William Savedoff
Additional contact information
Julia Goldberg Raifman: Boston University
Felix Lam: Clinton Health Access Initiative
Janeen Madan Keller: Center for Global Development
Alexander Radunsky: Harvard T.H. Chan School of Public Health
William Savedoff: Center for Global Development

No 461, Working Papers from Center for Global Development

Abstract: Evaluations are key to learning and accountability yet the quality of those evaluations are critical to their usefulness. We assessed the methodological quality of global health program evaluations commissioned or conducted by five major funders and published between 2009 and 2014. From a universe of 299 large-scale global health program evaluations, we randomly selected 37 evaluations stratified by whether they were performance evaluations or impact evaluations and applied a systematic assessment approach with two reviewers scoring each evaluation. We found that most evaluations did not meet social science methodological standards in terms of using methods and data that would simultaneously assure relevance, validity, and reliability. Most evaluations (76 percent) asked questions relevant to the health program, but 43 percent of evaluations failed to collect relevant data. In addition, only about a fifth of the evaluations followed accepted social science methods for sampling. We also assessed whether evaluations took a systematic analytical approach and considered potential confounding variables. In this regard, only 16 percent of evaluations in our sample had high analytical validity and reliability. The study provides ten recommendations for improving the quality of evaluations, including a robust finding that early planning of evaluations is associated with better quality and noting the value of better sampling approaches in data collection and disclosure of potential conflicts of interest and data.

Keywords: evaluation; impact evaluation; health; foreign aid; learning; research quality (search for similar items in EconPapers)
JEL-codes: D04 F35 F53 H43 I18 (search for similar items in EconPapers)
Pages: 38 pages
Date: 2017-08-18
New Economics Papers: this item is included in nep-hea
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.cgdev.org/publication/evaluating-evalu ... ations-global-health

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cgd:wpaper:461

Access Statistics for this paper

More papers in Working Papers from Center for Global Development Contact information at EDIRC.
Bibliographic data for series maintained by Publications Manager ().