Synonyms, Antonyms and Factual Knowledge in BERT Heads

Lorenzo Serina (), Luca Putelli (), Alfonso Emilio Gerevini and Ivan Serina
Additional contact information
Lorenzo Serina: Department of Information Engineering, Università Degli Studi di Brescia, Via Branze 38, 25100 Brescia, Italy
Luca Putelli: Department of Information Engineering, Università Degli Studi di Brescia, Via Branze 38, 25100 Brescia, Italy
Alfonso Emilio Gerevini: Department of Information Engineering, Università Degli Studi di Brescia, Via Branze 38, 25100 Brescia, Italy
Ivan Serina: Department of Information Engineering, Università Degli Studi di Brescia, Via Branze 38, 25100 Brescia, Italy

Future Internet, 2023, vol. 15, issue 7, 1-18

Abstract: In recent years, many studies have been devoted to discovering the inner workings of Transformer-based models, such as BERT, for instance, attempting to identify what information is contained within them. However, little is known about how these models store this information in their millions of parameters and which parts of the architecture are the most important. In this work, we propose an approach to identify self-attention mechanisms, called heads, that contain semantic and real-world factual knowledge in BERT. Our approach includes a metric computed from attention weights and exploits a standard clustering algorithm for extracting the most relevant connections between tokens in a head. In our experimental analysis, we focus on how heads can connect synonyms, antonyms and several types of factual knowledge regarding subjects such as geography and medicine.

Keywords: BERT; attention heads; synonyms; antonyms; real-world knowledge (search for similar items in EconPapers)
JEL-codes: O3 (search for similar items in EconPapers)
Date: 2023
References: View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://www.mdpi.com/1999-5903/15/7/230/pdf (application/pdf)
https://www.mdpi.com/1999-5903/15/7/230/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jftint:v:15:y:2023:i:7:p:230-:d:1182617

Access Statistics for this article

Future Internet is currently edited by Ms. Grace You

More articles in Future Internet from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().