HBert: A Long Text Processing Method Based on BERT and Hierarchical Attention Mechanisms

Xueqiang Lv, Zhaonan Liu, Ying Zhao, Ge Xu and Xindong You
Additional contact information
Xueqiang Lv: Beijing Information Science and Technology University, China
Zhaonan Liu: Beijing Information Science and Technology University, China
Ying Zhao: Beijing Information Science and Technology University, China
Ge Xu: Minjiang University, China
Xindong You: Beijing Information Science and Technology University, China

International Journal on Semantic Web and Information Systems (IJSWIS), 2023, vol. 19, issue 1, 1-14

Abstract: With the emergence of a large-scale pre-training model based on the transformer model, the effect of all-natural language processing tasks has been pushed to a new level. However, due to the high complexity of the transformer's self-attention mechanism, these models have poor processing ability for long text. Aiming at solving this problem, a long text processing method named HBert based on Bert and hierarchical attention neural network is proposed. Firstly, the long text is divided into multiple sentences whose vectors are obtained through the word encoder composed of Bert and the word attention layer. And the article vector is obtained through the sentence encoder that is composed of transformer and sentence attention. Then the article vector is used to complete the subsequent tasks. The experimental results show that the proposed HBert method achieves good results in text classification and QA tasks. The F1 value is 95.7% in longer text classification tasks and 75.2% in QA tasks, which are better than the state-of-the-art model longformer.

Date: 2023
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
http://services.igi-global.com/resolvedoi/resolve.aspx?doi=10.4018/IJSWIS.322769 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:igg:jswis0:v:19:y:2023:i:1:p:1-14

Access Statistics for this article

International Journal on Semantic Web and Information Systems (IJSWIS) is currently edited by Brij Gupta

More articles in International Journal on Semantic Web and Information Systems (IJSWIS) from IGI Global
Bibliographic data for series maintained by Journal Editor ().