On the Role of Summary Content Units in Text Summarization Evaluation

Marcel Nawrath, Agnieszka Nowak, Tristan Ratz, Danilo C Walenta, Juri Opitz, Leonardo FR Ribeiro, João Sedoc, Daniel Deutsch, Simon Mille, Yixin Liu, Lining Zhang, Sebastian Gehrmann, Saad Mahamood, Miruna Clinciu, Khyathi Chandu and Yufang Hou

Publications of Darmstadt Technical University, Institute for Business Studies (BWL) from Darmstadt Technical University, Department of Business Administration, Economics and Law, Institute for Business Studies (BWL)

Abstract: At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs). These SCUs are concise sentences that decompose a summary into small facts. Such SCUs can be used to judge the quality of a candidate summary, possibly partially automated via natural language inference (NLI) systems. Interestingly, with the aim to fully automate the Pyramid evaluation, Zhang and Bansal (2021) show that SCUs can be approximated by automatically generated semantic role triplets (STUs). However, several questions currently lack answers, in particular: i) Are there other ways of approximating SCUs that can offer advantages? ii) Under which conditions are SCUs (or their approximations) offering the most value? In this work, we examine two novel strategies to approximate SCUs: generating SCU approximations from AMR meaning representations (SMUs) and from large language models (SGUs), respectively. We find that while STUs and SMUs are competitive, the best approximation quality is achieved by SGUs. We also show through a simple sentence-decomposition baseline (SSUs) that SCUs (and their approximations) offer the most value when ranking short summaries, but may not help as much when ranking systems or longer summaries.

Date: 2024-04-02
Note: for complete metadata visit http://tubiblio.ulb.tu-darmstadt.de/144487/
References: Add references at CitEc
Citations:

Published in Computation and Language (2024-04-02)

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:dar:wpaper:144487

Access Statistics for this paper

More papers in Publications of Darmstadt Technical University, Institute for Business Studies (BWL) from Darmstadt Technical University, Department of Business Administration, Economics and Law, Institute for Business Studies (BWL) Contact information at EDIRC.
Bibliographic data for series maintained by Dekanatssekretariat ().