Linking Individuals Across Historical Sources: a Fully Automated Approach

Ran Abramitzky, Roy Mill and Santiago Perez

No 24324, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: Linking individuals across historical datasets relies on information such as name and age that is both non-unique and prone to enumeration and transcription errors. These errors make it impossible to find the correct match with certainty. In the first part of the paper, we suggest a fully automated probabilistic method for linking historical datasets that enables researchers to create samples at the frontier of minimizing type I (false positives) and type II (false negatives) errors. The first step guides researchers in the choice of which variables to use for linking. The second step uses the Expectation-Maximization (EM) algorithm, a standard tool in statistics, to compute the probability that each two records correspond to the same individual. The third step suggests how to use these estimated probabilities to choose which records to use in the analysis. In the second part of the paper, we apply the method to link historical population censuses in the US and Norway, and use these samples to estimate measures of intergenerational occupational mobility. The estimates using our method are remarkably similar to the ones using IPUMS’, which relies on hand linking to create a training sample. We created an R code and a Stata command that implement this method.

JEL-codes: C10 J01 J10 N00 (search for similar items in EconPapers)
Date: 2018-02
New Economics Papers: this item is included in nep-his
Note: AG DAE LS
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (8)

Published as Ran Abramitzky & Roy Mill & Santiago Pérez, 2020. "Linking individuals across historical sources: A fully automated approach*," Historical Methods: A Journal of Quantitative and Interdisciplinary History, vol 53(2), pages 94-111.

Downloads: (external link)
http://www.nber.org/papers/w24324.pdf (application/pdf)

Related works:
Journal Article: Linking individuals across historical sources: A fully automated approach* (2020)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nbr:nberwo:24324

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w24324

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.. Contact information at EDIRC.
Bibliographic data for series maintained by ().