Man Versus Machine? Self-Reports Versus Algorithmic Measurement of Publications

Xuan Jiang, Wan-Ying Chang and Bruce Weinberg

No 28431, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: This paper uses newly available data from Web of Science on publications matched to researchers in Survey of Doctorate Recipients to compare scientific publications collected by surveys and algorithmic approaches. We aim to illustrate the different types of measurement errors in self-reported and machine-generated data by estimating how publication measures from the two approaches are related to career outcomes (e.g. salaries, placements, and faculty rankings). We find that the potential biases in the self-reports are smaller relative to the algorithmic data. Moreover, the errors in the two approaches are quite intuitive: the measurement errors of the algorithmic data are mainly due to the accuracy of matching, which primarily depends on the frequency of names and the data that was available to make matches; while the noise in self reports is expected to increase over the career as researchers’ publication records become more complex, harder to recall, and less immediately relevant for career progress. This paper provides methodological suggestion on evaluating the quality and advantages of two approaches to data construction. It also provides guidance on how to use the new linked data.

JEL-codes: C26 J24 J3 O31 (search for similar items in EconPapers)
Date: 2021-02
New Economics Papers: this item is included in nep-big, nep-lma and nep-sog
Note: LS PR
References: View references in EconPapers View complete reference list from CitEc
Citations:

Published as Xuan “Gabi” Jiang, Wan-Ying Chang, Bruce A. Weinberg. 2021. “Man Versus Machine? Self-Reports Versus Algorithmic Measurement of Publications.” PLoS One (September 29)

Downloads: (external link)
http://www.nber.org/papers/w28431.pdf (application/pdf)

Related works:
Journal Article: Man versus machine? Self-reports versus algorithmic measurement of publications (2021)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nbr:nberwo:28431

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w28431

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.. Contact information at EDIRC.
Bibliographic data for series maintained by ().