Advancing Instrument Validation in Social Sciences: An AI-Powered Chatbot and Interactive Website based on a Research Instrument Validation Framework (RIVF)

Resti Tito Villarino and Maureen Lorence Villarino
Additional contact information
Resti Tito Villarino: Cebu Technological University

No rjyzg, SocArXiv from Center for Open Science

Abstract: Background: In social sciences, an appreciable degree of instrument validation is intended to maintain scientific rigor and research quality. Objectives: To develop and evaluate an AI chatbot and website for instrument validation, assess their impact on instrument validity improvement, and analyze user perceptions. Methods: Adopting a quantitative design, the study was anchored on the developed RIVF of Villarino (2024). Moreover, it was evaluated through users' perceptions (n=100) by administering an online survey, whereby the employment of pairing t-tests used contrasting instrument validity-pre-vs post-RIVF scores, and one-way ANOVA was used to determine if a relationship existed between users' perceptions and total improvement in validity. A G*Power analysis showed that there was enough statistical power for the analyses: for paired t-tests, it was 99.73% (n = 100, dz = 0.5, α = 0.05), and for one-way ANOVA, 80.95% (n = 100, f = 0.25, α=0.05, four groups). All data were analyzed using IBM SPSS version 26. Results: Post-RIVF use, all the validity domains showed significant improvements (p-value greater than 0.001), but the maximum significant gain was seen in construct validity [Mean difference=1.20±0.60, t(49)=14.14]. Subjects perceived the AI chatbot as more useful [4.30±0.70 vs. 3.80±0.80, p-value greater than 0.001]. Conclusion: This AI-powered ecosystem indicates a potential for increasing the validity of research instruments in RIVF, while an AI chatbot is efficient in incrementing the construct validity. These results would infer that the use of AI technologies along with traditional validation methods will improve the quality of research instruments in the social sciences. Keywords: artificial intelligence (AI), instrument validation, research methodology, social sciences

Date: 2024-07-29
New Economics Papers: this item is included in nep-ain and nep-pay
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/66b4feb656c30fcdf6519913/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:socarx:rjyzg

DOI: 10.31219/osf.io/rjyzg

Access Statistics for this paper

More papers in SocArXiv from Center for Open Science
Bibliographic data for series maintained by OSF (contact@cos.io).