AI and social science: Automatic classification tools for big data analysis in sociological research

Andrea Nucita, Assunta Penna, Antonia Cava, Giancarlo Iannizzotto and Massimo Mucciardi

PLOS ONE, 2026, vol. 21, issue 6, 1-14

Abstract: This study examines the use of Social Network Sites for public institutional communication through a sociological, data-driven lens, focusing on the challenges and potential of automated classification tools for data analysis. Although Large Language Models are increasingly used to process social media data, a key research gap remains: few studies systematically assess whether AI-based categorizations are as reliable as human coding, especially when categories are semantically ambiguous. The research addresses the following questions: How reliable are AI-generated classifications compared to those made by human experts? Is human–machine agreement comparable to the level of agreement observed among human coders? To experimentally test this approach, we conducted a case study on Facebook posts published by two Italian universities (March 2020–March 2023), classified into eight categories of public institutional communication. Three researchers independently annotated the dataset. Human annotations are used as a benchmark to assess agreement patterns and to compare them with classifications produced by AI-based systems. Results show substantial interpretive ambiguity across several categories, mirrored by variability among human coders. Nonetheless, automated models achieve agreement with human classifications that is broadly comparable to inter-coder agreement. Overall, the findings support integrating AI as an additional coder within hybrid workflows to enable scalable and transparent sociological analysis of complex social media data.

Date: 2026
References: Add references at CitEc
Citations:

Downloads: (external link)
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0351336 (text/html)
https://journals.plos.org/plosone/article/file?id= ... 51336&type=printable (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:plo:pone00:0351336

DOI: 10.1371/journal.pone.0351336

Access Statistics for this article

More articles in PLOS ONE from Public Library of Science
Bibliographic data for series maintained by plosone ().