Grammar In Language Models: Bert Study

Ksenia Chistyakova () and Tatiana Kazakova ()
Additional contact information
Ksenia Chistyakova: National Research University Higher School of Economics
Tatiana Kazakova: National Research University Higher School of Economics

HSE Working papers from National Research University Higher School of Economics

Abstract: The problem of language models’ interpretation is extensively inspected, but no universal answers have been found. Our study offers to combine widely accepted probing methods with a novel approach to a neural network under investigation. We propose to break grammatical forms on the pre-training step in order to get two "sibling" models, as it casts some light on how different linguistic features are encoded and distributed across the neural language architecture.

Keywords: probing; language models; transformers; BERT. (search for similar items in EconPapers)
JEL-codes: Z (search for similar items in EconPapers)
Pages: 20 pages
Date: 2023
New Economics Papers: this item is included in nep-big and nep-cmp
References: View complete reference list from CitEc
Citations:

Published in WP BRP Series: Linguistics / LNG, /November 2023, pages 1-20

Downloads: (external link)
https://wp.hse.ru/data/2023/11/28/2107751289/115LNG2023.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:hig:wpaper:115/lng/2023

Access Statistics for this paper

More papers in HSE Working papers from National Research University Higher School of Economics
Bibliographic data for series maintained by Shamil Abdulaev () and Shamil Abdulaev ().