BACKGROUND: The VAC4EU SeValid project is developing a methodology to integrate large language models (LLMs) into VAC4EU routine validation pipeline to identify both false positives and false negatives. The methodology was piloted on myocarditis records identified from the hospital discharge registry (HDR) of a large hospital in Florence, Italy, using ICD-9-CM codes. A highly screening phenotype was created to detect records not coded as myocarditis, based on interviews with local clinical coders. To validate these records, a questionnaire based on the Brighton Collaboration definition of myocarditis was applied and converted into prompts for an open-source LLM (Mistral-Small 3.1).
OBJECTIVES: To compare sensitivity and positive predictive value (PPV) of human and LLM-based chart validation for records identified by a myocarditis screening phenotype.
METHODS: Records identified through the screening phenotype were extracted from the HDR with their corresponding electronic medical charts. A random sample of 150 records was independently reviewed by a trained human assessor (H) and by LLM. For each record and questionnaire item, LLM assessment was repeated three times, the majority option was chosen and consistency across runs evaluated. Records were classified as cases (C) if the Brighton Collaboration criteria for definite myocarditis (biopsy confirmation or biomarker plus imaging evidence) were met, and as non-cases (NC) otherwise. Concordance between H and LLM assessments was evaluated. For a sample of discordant records, a clinician completed the questionnaire to establish a reference standard (RS). Based on the RS, the estimated number of true positives for H (ETP_H) and LLM (ETP_LLM) was calculated. The total expected number of confirmed cases (TECC) equaled ETP_H + ETP_LLM. For both H and LLM, sensitivity and PPV were estimated as ETP divided by, respectively, TECC and C.
RESULTS: A total of 2446 records were retrieved from HDR. A random sample of 150 records were classified by H and LLM as: 7 (4.7%) were (C,C); 31 (20.6%) (C,NC); 8 (5.3%) (NC,C) and 104 (69.3%)(NC,NC). Internal consistency across LLM runs was 85% in C and 95% in NC. In samples of discordant records, RS confirmed myocarditis in 75% of both (C,NC) and (NC,C) records. TECC was estimated as 36, with ETP_H = 30 and ETP_LLM = 13. Sensitivity and PPV were 83% and 79% for H, and 36% and 86% for LLM
CONCLUSIONS: LLMs may support chart validation by identifying myocarditis cases missed by coding-based definitions. Human validation showed higher sensitivity than LLM, whereas LLM validation achieved a higher PPV. Internal consistency of LLM assessments may depend on the complexity and heterogeneity of clinical text. These findings support a complementary human–LLM approach within routine VAC4EU validation pipelines
Skip to main content
