A Metamorphic Testing Approach for Assessing Question Answering Systems

Kaiyi Tu, Mingyue Jiang and Zuohua Ding
Additional contact information
Kaiyi Tu: School of Information Science and Technology, Zhejiang Sci-Tech University, Hangzhou 310018, China
Mingyue Jiang: School of Information Science and Technology, Zhejiang Sci-Tech University, Hangzhou 310018, China
Zuohua Ding: School of Information Science and Technology, Zhejiang Sci-Tech University, Hangzhou 310018, China

Mathematics, 2021, vol. 9, issue 7, 1-15

Abstract: Question Answering (QA) enables the machine to understand and answer questions posed in natural language, which has emerged as a powerful tool in various domains. However, QA is a challenging task and there is an increasing concern about its quality. In this paper, we propose to apply the technique of metamorphic testing (MT) to evaluate QA systems from the users’ perspectives, in order to help the users to better understand the capabilities of these systems and then to select appropriate QA systems for their specific needs. Two typical categories of QA systems, namely, the textual QA (TQA) and visual QA (VQA), are studied, and a total number of 17 metamorphic relations (MRs) are identified for them. These MRs respectively focus on some characteristics of different aspects of QA. We further apply MT to four QA systems (including two APIs from the AllenNLP platform, one API from the Transformers platform, and one API from CloudCV) by using all of the MRs. Our experimental results demonstrate the capabilities of the four subject QA systems from various aspects, revealing their strengths and weaknesses. These results further suggest that MT can be an effective method for assessing QA systems.

Keywords: textual question answering; visual question answering; metamorphic testing; metamorphic relations; quality assessment (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2021
References: View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://www.mdpi.com/2227-7390/9/7/726/pdf (application/pdf)
https://www.mdpi.com/2227-7390/9/7/726/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:9:y:2021:i:7:p:726-:d:525479

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().