Annotating live messages on social media. Testing the efficiency of the AnnotHate – live data annotation portal

Gunther Jikeli (), Katharina Soemer () and Sameer Karali ()
Additional contact information
Gunther Jikeli: Indiana University Bloomington
Katharina Soemer: Indiana University Bloomington
Sameer Karali: Indiana University Bloomington

Journal of Computational Social Science, 2024, vol. 7, issue 1, No 22, 585 pages

Abstract: Abstract Labeling datasets to produce gold standard corpora for training machine learning algorithms a re increasingly important in social media research. The annotation process, including annotation tools, is of utmost importance to the quality of gold standard corpora. While measuring inter-annotator reliability has become standard practice, and research has been conducted on the annotators themselves and their possible influence on the annotation process, reflections on the annotation tools often remain neglected in descriptions of gold standard productions. Many social media posts are short and require more context to understand their meaning, which only the live environment can provide. However, most annotation tools work with offline data. We test a specially designed tool for live data annotation, including an experiment with 80 annotators. The tool is user-friendly for annotators, does not require any command line usage or installations, and reduces errors in the annotation process. It is time efficient in the annotation process, and efficient and transparent in collecting the data from the annotation.

Keywords: Annotation tool; Labeling; Gold Standard; Twitter; X; Social media; Hate speech (search for similar items in EconPapers)
Date: 2024
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s42001-024-00251-0 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:jcsosc:v:7:y:2024:i:1:d:10.1007_s42001-024-00251-0

Ordering information: This journal article can be ordered from
http://www.springer. ... iences/journal/42001

DOI: 10.1007/s42001-024-00251-0

Access Statistics for this article

Journal of Computational Social Science is currently edited by Takashi Kamihigashi

More articles in Journal of Computational Social Science from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().