Fake–Real gaps in observed spread and detectability differ between health misinformation datasets built in different ways: a comparison of CoAID and FakeHealth

Ji Zhao and Jesús Valdiviezo

No c83dk_v1, SocArXiv from Center for Open Science

Abstract: Health misinformation datasets don't reach a common ground about what misinformation is. FakeHealth labels mainstream health news by expert-rated reporting quality, while CoAID compares fact-checked COVID-19 claims with material from health agencies and health information publishers. We compared model-standardized Fake/Real ratios of observed social media posts, including items with no observed posts, between the CoAID News and FakeHealth HealthStory subsets, and evaluated leakage-controlled text classifiers within, across and beyond the two datasets. Fake items drew fewer observed posts than Real items in both datasets, but the ratio was 0.253 in CoAID and 0.840 in FakeHealth (ratio-of-ratios 0.301; 95% interval 0.154–0.645). The CoAID gap lay entirely in whether an item was posted at all and was concentrated among Fake items with no identifiable publisher, which points to limited findability as well as limited spread. In two FakeNewsNet collections outside health, Fake items drew more observed posts in GossipCop (ratio 2.144), and PolitiFact's interval included parity. Text showed a parallel split: Fake and Real were highly separable within CoAID (macro F1 0.926) and modestly within FakeHealth (0.675), transfer was poor in both directions, and word–label associations were uncorrelated across datasets. Fake items were shorter in both datasets, and length alone ranked transferred items and independent COVID rumors better than the word-based models. The distance between Fake and Real items, in language and in observed engagement, differed between datasets that define and source misinformation differently; these are comparisons between dataset contexts, not estimates of a pandemic effect. Engagement studies and detection benchmarks should report how labels, sources and collection were defined, check for source–label entanglement, shared-post dependence and simple length baselines, and test beyond a single dataset.

Date: 2026-11-06
References: Add references at CitEc
Citations:

Downloads: (external link)
https://osf.io/download/6aee0c7632648d4effca3450/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:socarx:c83dk_v1

DOI: 10.31235/osf.io/c83dk_v1

Access Statistics for this paper

More papers in SocArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().