The performance of international organizations: a new measure and dataset based on computational text analysis of evaluation reports

Steffen Eckhard (), Vytautas Jankauskas (), Elena Leuschner (), Ian Burton (), Tilman Kerl () and Rita Sevastjanova ()
Additional contact information
Steffen Eckhard: Zeppelin University
Vytautas Jankauskas: Zeppelin University
Elena Leuschner: University of Gothenburg
Ian Burton: Zeppelin University
Tilman Kerl: Vienna University of Technology
Rita Sevastjanova: University of Konstanz

The Review of International Organizations, 2023, vol. 18, issue 4, No 6, 753-776

Abstract: Abstract International organizations (IOs) of the United Nations (UN) system publish around 750 evaluation reports per year, offering insights on their performance across project, program, institutional, and thematic activities. So far, it was not feasible to extract quantitative performance measures from these text-based reports. Using deep learning, this article presents a novel text-based performance metric: We classify individual sentences as containing a negative, positive, or neutral assessment of the evaluated IO activity and then compute the share of positive sentences per report. Content validation yields that the measure adequately reflects the underlying concept of performance; convergent validation finds high correlation with human-provided performance scores by the World Bank; and construct validation shows that our measure has theoretically expected results. Based on this, we present a novel dataset with performance measures for 1,082 evaluated activities implemented by nine UN system IOs and discuss avenues for further research.

Keywords: Performance; Evaluation; International organizations; Natural language processing; Machine learning (search for similar items in EconPapers)
Date: 2023
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
http://link.springer.com/10.1007/s11558-023-09489-1 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:revint:v:18:y:2023:i:4:d:10.1007_s11558-023-09489-1

Ordering information: This journal article can be ordered from
http://www.springer. ... iology/journal/11558

DOI: 10.1007/s11558-023-09489-1

Access Statistics for this article

The Review of International Organizations is currently edited by A. Dreher

More articles in The Review of International Organizations from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().