An Empirical Comparative Evaluation of BERT-Family and LLM-Family Models for Automatic CTCAE Severity Grading from Structured FDA FAERS Report Composites

Xizhu Liu

Journal of Sustainability, Policy, and Practice, 2026, vol. 2, issue 4, 129-141

Abstract: Severity grading of adverse events under the Common Terminology Criteria for Adverse Events (CTCAE) is a labor-intensive yet decision-critical task in clinical trial monitoring. While public adverse event repositories offer vast unstructured adverse event records, native CTCAE 1--5 labels are rarely available. This study presents an empirical comparative evaluation of two model families on automatic CTCAE grading: BERT-family encoders (BioBERT, ClinicalBERT, PubMedBERT) and large language models (GPT-3.5, GPT-4o, Me-LLaMA-13B). Using FDA FAERS quarterly extracts from 2018 to 2024 as the primary corpus, we constructed silver labels through a MedDRA-to-CTCAE heuristic mapping and curated a 2,000-report gold subset via dual pharmacist annotation reaching Cohen's κ = 0.78. Four supplementary corpora supported transfer evaluation. Under matched protocols, PubMedBERT-large attained the strongest in-domain performance (5-grade macro-F1 = 0.713, quadratic-weighted κ = 0.741), while LoRA-adapted Me-LLaMA-13B exhibited the most stable cross-domain behavior with transfer loss limited to 4.6 macro-F1 points. Zero-shot LLMs achieved competitive recall but lower precision and weaker calibration. The results suggest that supervised encoders remain economical for stable in-domain pipelines, whereas adapted medium-scale medical LLMs are preferable when distribution shift dominates. We release the silver-labeling rules, prompts, and evaluation harness to support reproducibility.

Keywords: CTCAE severity grading; pharmacovigilance NLP; FAERS; clinical language model evaluation (search for similar items in EconPapers)
Date: 2026
References: Add references at CitEc
Citations:

Downloads: (external link)
https://pinnaclepubs.com/index.php/JSPP/article/view/838/803 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:dba:jsppaa:v:2:y:2026:i:4:p:129-141

Access Statistics for this article

More articles in Journal of Sustainability, Policy, and Practice from Pinnacle Academic Press
Bibliographic data for series maintained by Joseph Clark ().