Large Language Models Augment or Substitute Human Experts in Idea Screening

Cathy L Yang, Abhishek Borah, Brendon Rhodes and Pavel Kireyev
Additional contact information
Cathy L Yang: HEC Paris - Department of Information Systems and Operations Management
Abhishek Borah: INSEAD
Brendon Rhodes: INSEAD
Pavel Kireyev: London School of Economics

No 1597, HEC Research Papers Series from HEC Paris

Abstract: Firms that use crowdsourcing to gather advertising and product ideas often rely on internal experts to manually screen thousands of submissions, a costly and time-consuming process. Internal experts rate thousands of ideas to identify a small set of promising ones that are then submitted for additional review. We evaluate how large language models (LLMs), when combined with a machine learning model trained on historical expert ratings and final client selections, can improve the efficiency of this screening. Using data from a platform that engaged experts to evaluate 74,436 ideas across 153 contests for major advertisers, we show that evaluation effort can be reduced by 28.4% compared to the status quo. Of this reduction, 3.8% is directly attributable to the LLM output, while the remainder comes from better weighting expert scores to align with sponsor preferences. Notably, incorporating LLMs could make 5 out of 10 experts redundant, compared to 3 with machine learning alone. Importantly, the experts whose judgments are most replicable by the LLM are not necessarily the poorest performers. These findings offer a practical framework for integrating LLMs into idea screening pipelines and underscore their potential to streamline expert evaluation while maintaining alignment with client goals.

Keywords: Ideation; Idea Screening; Crowdsourcing; LLMs; Prompt Engineering; Machine Learning (search for similar items in EconPapers)
JEL-codes: C55 L23 M15 (search for similar items in EconPapers)
Pages: 55 pages
Date: 2025-10-21, Revised 2025-10-21
References: Add references at CitEc
Citations:

Downloads: (external link)
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5634331 Full text (text/html)
Our link check indicates that this URL is bad, the error code is: 403 Forbidden

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ebg:heccah:1597

DOI: 10.2139/ssrn.5634331

Access Statistics for this paper

More papers in HEC Research Papers Series from HEC Paris HEC Paris, 1 Rue de la Libération, 78350 Jouy-en-Josas, France. Contact information at EDIRC.
Bibliographic data for series maintained by Antoine Haldemann ( this e-mail address is bad, please contact ).