Applications of machine learning in tabular document digitisation

Christian Dahl, Torben S. D. Johansen, Emil N. Sørensen, Christian E. Westermann and Simon Wittrock

Historical Methods: A Journal of Quantitative and Interdisciplinary History, 2023, vol. 56, issue 1, 34-48

Abstract: Data acquisition forms the primary step in all empirical research. The availability of data directly impacts the quality and extent of conclusions and insights. In particular, larger and more detailed datasets provide convincing answers even to complex research questions. The main problem is that large and detailed usually imply costly and difficult, especially when the data medium is paper and books. Human operators and manual transcription has been the traditional approach for collecting historical data. We instead advocate the use of modern machine learning techniques to automate the digitization and transcription process. We propose a customizable end-to-end transcription pipeline to perform layout classification, table segmentation, and transcribe handwritten text that is suitable for tabular data, as is common in, e.g., census lists and birth and death records. We showcase our pipeline through two applications: The first demonstrates that unsupervised layout classification applied to raw scans of nurse journals can be used to obtain valuable insights into an extended nurse home visiting program. The second application uses attention-based neural networks for handwritten text recognition to transcribe age and birth and death dates and includes a comparison to automated transcription using Transkribus in the regime of tabular data. We describe each step in our pipeline and provide implementation insights.

Date: 2023
References: Add references at CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1080/01615440.2023.2164879 (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:taf:vhimxx:v:56:y:2023:i:1:p:34-48

Ordering information: This journal article can be ordered from
http://www.tandfonline.com/pricing/journal/vhim20

DOI: 10.1080/01615440.2023.2164879

Access Statistics for this article

Historical Methods: A Journal of Quantitative and Interdisciplinary History is currently edited by J. David Hacker and Kenneth Sylvester

More articles in Historical Methods: A Journal of Quantitative and Interdisciplinary History from Taylor & Francis Journals
Bibliographic data for series maintained by Chris Longhurst ().