A Weighted Composite Metric for Evaluating User Experience in Educational Chatbots: Balancing Usability, Engagement, and Effectiveness

Abeer Alabbas () and Khalid Alomar
Additional contact information
Abeer Alabbas: Department of Technical Support, Faculty of Technical College for Girls, Technical and Vocational Training Corporation, Najran 66253, Saudi Arabia
Khalid Alomar: Department of Information Systems, Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah 21589, Saudi Arabia

Future Internet, 2025, vol. 17, issue 2, 1-35

Abstract: Evaluating user experience (UX) is essential for optimizing educational chatbots to enhance learning outcomes and student productivity. This study introduces a novel weighted composite metric integrating interface usability assessment (via the Chatbot Usability Questionnaire, CUQ), engagement measurements (via the User Engagement Scale—Short Form, UES-SF), and objective performance indicators (through error rates and response times), addressing gaps in existing evaluation methods across interaction modes (text-based, menu-based, and hybrid) and question complexities. A 3 × 3 within-subject experimental design ( n = 30) was conducted, measuring these distinct UX dimensions through standardized instruments and performance metrics, supplemented by qualitative feedback. Principal Component Analysis (PCA) was used to derive weights for the composite UX metric based on empirical patterns in user interactions. Repeated-measures ANOVA revealed that the hybrid interaction mode outperformed the others, achieving significantly higher usability (F(2,58) = 89.32, p < 0.001) and engagement (F(2,58) = 8.67, p < 0.001), with fewer errors and faster response times under complex query conditions. These findings demonstrate the hybrid mode’s adaptability across question complexities. The proposed framework establishes a standardized method for evaluating educational chatbots, providing actionable insights for interface optimization and sustainable learning tools.

Keywords: educational chatbots; user experience; interaction modes; question complexity; usability; engagement; learning outcomes (search for similar items in EconPapers)
JEL-codes: O3 (search for similar items in EconPapers)
Date: 2025
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/1999-5903/17/2/64/pdf (application/pdf)
https://www.mdpi.com/1999-5903/17/2/64/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jftint:v:17:y:2025:i:2:p:64-:d:1583830

Access Statistics for this article

Future Internet is currently edited by Ms. Grace You

More articles in Future Internet from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().