Main Theme: General / Innovation
Introduction:
AI use in healthcare is increasing, with over 75% of patients willing to use generative AI for health information instead of traditional sources(1,2). Given that over 40% of adults lack sufficient literacy to understand general health information(3), it is crucial to understand both the accessibility and quality of information generated by AI. This study evaluates whether AI meets an acceptable standard for paediatric otolaryngology health information.
Methods:
ChatGPT, Microsoft Copilot, and Google Gemini (default models, no custom prompts) were asked to generate three patient information leaflets on paediatric ENT topics: Otitis media with effusion, Juvenile Nasal Angiofibroma, and post-tonsillectomy care. ENT UK e-Leaflets were used as comparators.
Responses were blindly assessed for readability using Flesch Reading Ease Score (FRES) and Flesch-Kincaid Grade Level. Quality was blindly rated with the EQIP instrument and Global Quality Score (GQS). ANOVA was performed to evaluate differences
Results:
AI responses were significantly less readable than traditional leaflets (FRES 53.6 vs 74.1; p=0.019, Reading age 8.0 vs 5.7 p=0.054) and significantly lower quality (EQIP 38.2% vs 67.8%; p=0.006). AI leaflets were also shorter (616 vs 1123 words). GQS favoured traditional leaflets (3.1 vs 4.7) though this was not statistically significant (p=0.065).
Conclusion:
ENT UK leaflets outperformed all tested AI models in both readability and quality. Clinicians and families should exercise caution when using AI-generated health information. With both AI models and patient use evolving at a rapid pace, future work to optimise AI produced healthcare information is needed.
References
1.Understanding AI uptake and sentiment among people and businesses in the UK – Office for National Statistics [Internet]. [cited 2026 Mar 8]. Available from: https://www.ons.gov.uk/businessindustryandtrade/itandinternetindustry/articles/understandingaiuptakeandsentimentamongpeopleandbusinessesintheuk/june2023
2.Shahsavar Y, Choudhury A. User Intentions to Use ChatGPT for Self-Diagnosis and Health-Related Purposes: Cross-sectional Survey Study. JMIR Hum Factors. 2023 May 17;10(1):e47564. doi:10.2196/47564 PubMed PMID: 37195756.
3.Rowlands G, Protheroe J, Winkley J, Richardson M, Seed PT, Rudd R. A mismatch between population health literacy and the complexity of health information: an observational study. British Journal of General Practice. 2015 Jun 1;65(635):e379–86. doi:10.3399/bjgp15X685285 PubMed PMID: 26009533.