Optimal stopping and worker selection in crowdsourcing: an adaptive sequential probability ratio test framework

Xiaoou Li, Yunxiao Chen, Xi Chen, Jingchen Liu and Zhiliang Ying

LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library

Abstract: In this study, we solve a class of multiple testing problems under a Bayesian sequential decision framework. Our work is motivated by binary labeling tasks in crowdsourcing, where a requestor needs to simultaneously choose a worker to provide a label and decide when to stop collecting labels, under a certain budget constraint. We begin by using a binary hypothesis testing problem to determine the true label of a single object, and provide an optimal solution by casting it under an adaptive sequential probability ratio test framework. Then, we characterize the structure of the optimal solution, that is, the optimal adaptive sequential design, which minimizes the Bayes risk using a log-likelihood ratio statistic. We also develop a dynamic programming algorithm to efficiently compute the optimal solution. For the multiple testing problem, we propose an empirical Bayes approach for estimating the class priors, and show that the average loss of our method converges to the minimal Bayes risk under the true model. Experiments on both simulated and real data show the robustness of our method, as well as its superiority over existing methods in terms of its labeling accuracy.

Keywords: Bayesian decision theory; crowdsourcing; empirical Bayes; sequential analysis; sequential probability ratio test (search for similar items in EconPapers)
JEL-codes: C1 J01 J50 R14 (search for similar items in EconPapers)
Pages: 28 pages
Date: 2021-01-01
New Economics Papers: this item is included in nep-ecm and nep-ore
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (3)

Published in Statistica Sinica, 1, January, 2021, 31(1), pp. 519 - 546. ISSN: 1017-0405

Downloads: (external link)
http://eprints.lse.ac.uk/100873/ Open access version. (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ehl:lserod:100873

Access Statistics for this paper

More papers in LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library LSE Library Portugal Street London, WC2A 2HD, U.K.. Contact information at EDIRC.
Bibliographic data for series maintained by LSERO Manager ().