Effect of Emphasizing Methodological Limitations of Primary Studies in a Systematic Review Summary: Findings from Two Randomized Controlled Trials

Louis-Robert Beaulieu-Guay, Benoît Béchard, Mathieu Ouimet, François Claveau and Éric Montpetit

No kwe94_v2, SocArXiv from Center for Open Science

Abstract: Background: Various organizations expend resources to communicate the findings of systematic reviews to non-specialist audiences, such as policymakers and the general public. The Campbell Collaboration does so with its plain language summaries, which aim to communicate complex methodological information in an accessible manner. However, the effectiveness of such summaries in communicating the overall quality of the primary studies included in a systematic review remains understudied. Objectives: This study aims to assess how the design of a systematic review summary affects readers' perception of methodological quality of primary studies, their perception of the evidence's definitiveness and their attitudes towards the policy intervention discussed in the review. Data and Methods: Using a web-based experimental design, two studies were conducted to examine how various presentation formats, including the emphasis on methodological limitations and the use of images, influence readers’ perceptions of information presented in a systematic review summary. Results: Emphasizing methodological limitations significantly reduces readers' perception of the quality of the methods used in the studies included in a systematic review summary. Yet, highlighting methodological limitations does not influence people's perception of the definitiveness of the review findings or their attitude toward related policies. Furthermore, removing the images from the summary’s standard design appears to have minimal effect on how readers interpret information about methodological limitations. Conclusions: Modifications to the traditional summary design can amplify the awareness of laypeople and civil servants regarding methodological limitations in the scientific evidence. However, their way of processing information about these methodological limitations does not automatically propagate the change in attitude regarding methods to a change in attitudes regarding study results and policy options.

Date: 2026-03-28
New Economics Papers: this item is included in nep-exp
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/69c6f23f06e792b71773fe87/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:socarx:kwe94_v2

DOI: 10.31219/osf.io/kwe94_v2

Access Statistics for this paper

More papers in SocArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().