Introduction
AI tools are increasingly accessed by primary care clinicians and parents for clinical guidance. Performance in urgent paediatric ENT referral decision-making remains largely untested, despite safety implications of mis-triage.
Methods
Ten standardised paediatric ENT primary care scenarios were developed across three urgency tiers: cannot-miss emergencies, urgent presentations, and over-triage-prone cases. Five paediatric ENT consultants independently established the consensus reference standard, with concordance to GIRFT guidance. Five AI models were tested: three consumer LLMs (GPT-5, Claude, Gemini) and two medical AI tools (ClinicalKey AI, Glass Health). Each scenario was posed three times per model on separate days, generating 150 anonymised outputs scored by two blinded raters using a pre-specified categorical checklist.
Results
Inter-rater agreement was almost perfect (Cohen's kappa=0.94). Claude achieved the highest accuracy (73.3%), followed by Glass Health (66.7%), GPT-5 and Gemini (both 56.7%). ClinicalKey AI failed structurally, unable to produce forced-choice triage (3.3%). All functional models reliably identified Tier 1 emergencies (epiglottitis, post-tonsillectomy bleed, orbital cellulitis: 100%). All models performed poorly on middle-urgency referrals: recurrent epistaxis (0% — universal under-triage to reassurance), B-symptom neck lump (20% — over-triage), and unilateral nasal symptoms (27%). Only Gemini produced unsafe major undercalls (6.7%). Test-retest consistency was 67–90%. Models did not capitulate to parental pressure.
Conclusion
AI reliably recognises paediatric ENT emergencies but under-uses routine ENT referral for symptoms warranting specialist assessment. Medical AI was not uniformly superior to consumer LLMs.