Integrating Manual and Automatic Annotation for the Creation of Discourse Network Data Sets

Sebastian Haunss, Jonas Kuhn, Sebastian Padó, Andre Blessing, Nico Blokker, Erenay Dayanik and Gabriella Lapesa
Additional contact information
Jonas Kuhn: Institute for Natural Language Processing, University of Stuttgart, Germany
Sebastian Padó: Institute for Natural Language Processing, University of Stuttgart, Germany
Andre Blessing: Institute for Natural Language Processing, University of Stuttgart, Germany
Nico Blokker: Research Center on Inequality and Social Policy, University of Bremen, Germany
Erenay Dayanik: Institute for Natural Language Processing, University of Stuttgart, Germany
Gabriella Lapesa: Institute for Natural Language Processing, University of Stuttgart, Germany

Politics and Governance, 2020, vol. 8, issue 2, 326-339

Abstract: This article investigates the integration of machine learning in the political claim annotation workflow with the goal to partially automate the annotation and analysis of large text corpora. It introduces the MARDY annotation environment and presents results from an experiment in which the annotation quality of annotators with and without machine learning based annotation support is compared. The design and setting aim to measure and evaluate: a) annotation speed; b) annotation quality; and c) applicability to the use case of discourse network generation. While the results indicate only slight increases in terms of annotation speed, the authors find a moderate boost in annotation quality. Additionally, with the help of manual annotation of the actors and filtering out of the false positives, the machine learning based annotation suggestions allow the authors to fully recover the core network of the discourse as extracted from the articles annotated during the experiment. This is due to the redundancy which is naturally present in the annotated texts. Thus, assuming a research focus not on the complete network but the network core, an AI-based annotation can provide reliable information about discourse networks with much less human intervention than compared to the traditional manual approach.

Keywords: annotation; automation; discourse networks; machine learning; migration discourse (search for similar items in EconPapers)
Date: 2020
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://www.cogitatiopress.com/politicsandgovernance/article/view/2591 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cog:poango:v8:y:2020:i:2:p:326-339

DOI: 10.17645/pag.v8i2.2591

Access Statistics for this article

Politics and Governance is currently edited by Carolina Correia

More articles in Politics and Governance from Cogitatio Press
Bibliographic data for series maintained by António Vieira () and IT Department ().