LLM-Based Empathy Scoring for Psychological Counseling Dialogues Using Multi-Dimensional Client Feedback

Daniel Zhao and Michelle Chen

Global Journal of Science & Innovation, 2026, vol. 3, issue 1, 65-80

Abstract: Automatic assessment of counseling-dialogue quality requires models that connect long, role-structured interactions with the experience reported by clients after a session. This study presents a multi-dimensional empathy-scoring evaluation on KokoroChat, a Japanese psychological-counseling dialogue corpus released with 20 client-feedback dimensions on a 0-5 scale. The task is formulated as multi-output regression: given a complete counselor-client dialogue, predict the full feedback profile covering feeling heard, respect, insight, hope, conversational flow, acceptance, affirmation, questioning, summarization, clarification, goal identification, actionable suggestions, and encouragement. The evaluation uses a chronological split, with 2020-2022 dialogues for training, 2023 dialogues for validation, and 2024 dialogues for testing. Ten systems are compared: a global-mean baseline, structure-and-topic ridge regression, four hashed Japanese character n-gram ridge models, two compact self-attention encoders, a rubric-calibrated judge, and a linear ensemble. The rubric-calibrated judge obtains the lowest test MAE of 0.845 and RMSE of 1.056, with mean Pearson correlation of 0.449 and a bootstrap 95% confidence interval of 0.823-0.871 for overall MAE. Client-side text alone is numerically almost tied at MAE 0.846 and yields the highest mean correlation, indicating that client language contains strong evidence about perceived session quality. Topic-level analysis shows lower error for spouse-family and workplace sessions and higher error for small or heterogeneous groups. The results support detailed, reproducible evaluation of counseling quality through feedback profiles rather than a single empathy score.

Keywords: Japanese counseling dialogue; empathy scoring; KokoroChat; BERT; DeBERTa; LLM-as-judge; multi-output regression; client feedback; psychological counseling NLP (search for similar items in EconPapers)
Date: 2026
References: Add references at CitEc
Citations:

Downloads: (external link)
https://pinnaclepubs.com/index.php/GJSI/article/view/976/936 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:dba:gjsiaa:v:3:y:2026:i:1:p:65-80

Access Statistics for this article

More articles in Global Journal of Science & Innovation from Pinnacle Academic Press
Bibliographic data for series maintained by Joseph Clark ().