Recalibrating probabilistic forecasts of epidemics

Aaron Rumack, Ryan J Tibshirani and Roni Rosenfeld

PLOS Computational Biology, 2022, vol. 18, issue 12, 1-16

Abstract: Distributional forecasts are important for a wide variety of applications, including forecasting epidemics. Often, forecasts are miscalibrated, or unreliable in assigning uncertainty to future events. We present a recalibration method that can be applied to a black-box forecaster given retrospective forecasts and observations, as well as an extension to make this method more effective in recalibrating epidemic forecasts. This method is guaranteed to improve calibration and log score performance when trained and measured in-sample. We also prove that the increase in expected log score of a recalibrated forecaster is equal to the entropy of the PIT distribution. We apply this recalibration method to the 27 influenza forecasters in the FluSight Network and show that recalibration reliably improves forecast accuracy and calibration. This method, available on Github, is effective, robust, and easy to use as a post-processing tool to improve epidemic forecasts.Author summary: Epidemics of infectious disease cause millions of deaths worldwide each year, and reliable epidemic forecasts can allow public health officials to respond to mitigate the effects of epidemics. However, because epidemic forecasting is a difficult task, many epidemic forecasts are not calibrated. Calibration is a desired property of any forecast, and we provide a post-processing method that recalibrates forecasts. We demonstrate the effectiveness of this method in improving accuracy and calibration on a wide variety of influenza forecasters. We also show a quantitative relationship between calibration and a forecaster’s expected score. Our recalibration method is a tool that any forecaster can use, regardless of model choice, to improve forecast accuracy and reliability. This work provides a bridge between forecasting theory, which rarely deals with applications in domains that are new or have little data, and some recent applications of epidemic forecasting, where forecast calibration is rarely analyzed systematically.

Date: 2022
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1010771 (text/html)
https://journals.plos.org/ploscompbiol/article/fil ... 10771&type=printable (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:plo:pcbi00:1010771

DOI: 10.1371/journal.pcbi.1010771

Access Statistics for this article

More articles in PLOS Computational Biology from Public Library of Science
Bibliographic data for series maintained by ploscompbiol ().