Accuracy; Artificial intelligence; ChatGPT; Head neck; Information; Intensive care; LLM; Otolaryngology; Surgery; Tracheotomy; Humans; Reproducibility of Results; Surveys and Questionnaires; Artificial Intelligence; Male; Female; Tracheotomy/methods; Patient Education as Topic/methods; Patient Education as Topic; Otorhinolaryngology
Abstract :
[en] [en] OBJECTIVE: To investigate the accuracy of information provided by ChatGPT-4o to patients about tracheotomy.
METHODS: Twenty common questions of patients about tracheotomy were presented to ChatGPT-4o twice (7-day intervals). The accuracy, clarity, relevance, completeness, referencing, and usefulness of responses were assessed by a board-certified otolaryngologist and a board-certified intensive care unit practitioner with the Quality Analysis of Medical Artificial Intelligence (QAMAI) tool. The interrater reliability and the stability of the ChatGPT-4o responses were evaluated with intraclass correlation coefficient (ICC) and Pearson correlation analysis.
RESULTS: The total scores of QAMAI were 22.85 ± 4.75 for the intensive care practitioner and 21.45 ± 3.95 for the otolaryngologist, which consists of moderate-to-high accuracy. The otolaryngologist and the ICU practitioner reported high ICC (0.807; 95%CI: 0.655-0.911). The highest QAMAI scores have been found for clarity and completeness of explanations. The QAMAI scores for the accuracy of the information and the referencing were the lowest. The information related to the post-laryngectomy tracheostomy remains incomplete or erroneous. ChatGPT-4o did not provide references for their responses. The stability analysis reported high stability in regenerated questions.
CONCLUSION: The accuracy of ChatGPT-4o is moderate-to-high in providing information related to the tracheotomy. However, patients using ChatGPT-4o need to be cautious about the information related to tracheotomy care, steps, and the differences between temporary and permanent tracheotomies.
Disciplines :
Otolaryngology
Author, co-author :
Khaldi, Amina; Intensive Care Unit, EpiCURA Hospital, Hornu, Belgium
Machayekhi, Shahram; Intensive Care Unit, EpiCURA Hospital, Hornu, Belgium
Salvagno, Michele; Intensive Care Unit, Erasme Hospital, Brussels, Belgium
Maniaci, Antonino; Faculty of Medicine and Surgery, University of Enna Kore, Enna, 94100, Italy
Vaira, Luigi A; Maxillofacial Surgery Operative Unit, Department of Medicine, Surgery and Pharmacy, University of Sassari, Viale San Pietro 43/B, Sassari, 07100, Italy
La Via, Luigi; Department of Anesthesia and Intensive Care, University Hospital Policlinico "G.Rodolico-San Marco", Catania, Italy
Taccone, Fabio S; Intensive Care Unit, Erasme Hospital, Brussels, Belgium
Lechien, Jérome ; Université de Mons - UMONS > Faculté de Psychologie et des Sciences de l'Education > Service de Métrologie et Sciences du langage ; Université de Mons - UMONS > Faculté de Médecine et de Pharmacie > Service de Chirurgie
Language :
English
Title :
Accuracy of ChatGPT responses on tracheotomy for patient education.
Publication date :
November 2024
Journal title :
European Archives of Oto-Rhino-Laryngology
ISSN :
0937-4477
eISSN :
1434-4726
Publisher :
Springer Science and Business Media Deutschland GmbH, Germany
J.R. Lechien A. Rameau Applications of ChatGPT in Otolaryngology-Head Neck surgery: a state of the Art Review Otolaryngol Head Neck Surg 2024 10.1002/ohn.807 39350512
Lechien JR (2024) Generative AI and Otolaryngology-Head & Neck Surgery. Otolaryngol Clin North Am.: S0030-6665(24)00064 – 1. https://doi.org/10.1016/j.otc.2024.04.006
M. Abou-Abdallah T. Dar Y. Mahmudzade J. Michaels R. Talwar C. Tornari The quality and readability of patient information provided by ChatGPT: can AI reliably explain common ENT operations? Eur Arch Otorhinolaryngol 2024 10.1007/s00405-024-08598-w 38530460
I. Tessler A. Wolfovitz E.E. Alon N.A. Gecel N. Livneh E. Zimlichman E. Klang ChatGPT’s adherence to otolaryngology clinical practice guidelines Eur Arch Otorhinolaryngol 2024 281 7 3829 3834 10.1007/s00405-024-08634-9 38647684
A. Maniaci M. Lazzeroni A. Cozzi F. Fraccaroli M. Gaffuri C. Chiesa-Estomba P. Capaccio Can chatbots enhance the management of pediatric sialadenitis in clinical practice? Eur Arch Otorhinolaryngol 2024 10.1007/s00405-024-08798-4 39394329 11393115
J.R. Bellinger J.S. De La Chapa M.W. Kwak G.A. Ramos D. Morrison B.W. Kesser BPPV Information on Google Versus AI (ChatGPT) Otolaryngol Head Neck Surg 2023 10.1002/ohn.506 37622581
J. Riestra-Ayora C. Vaduva J. Esteban-Sánchez M. Garrote-Garrote C. Fernández-Navarro C. Sánchez-Rodríguez E. Martin-Sanz ChatGPT as an information tool in rhinology. Can we trust each other today? Eur Arch Otorhinolaryngol 2024 281 6 3253 3259 10.1007/s00405-024-08581-5 38436756
D.J. Campbell L.E. Estephan E. Sina E.V. Mastrolonardo R. Alapati D.R. Amin E. Cottrill Evaluating ChatGPT responses on thyroid nodules for patient education Thyroid 2023 10.1089/thy.2023.0491 38010917 10024591
F.A. Mira V. Favier H. Dos Santos Sobreira Nunes J.V. de Castro F. Carsuzaa G. Meccariello C. Vicini A. De Vito J.R. Lechien C.C. Estomba A. Maniaci G. Iannella E.P. Rojas J.B. Cornejo G. Cammaroto Chat GPT for the management of obstructive sleep apnea: do we have a polar star? Eur Arch Otorhinolaryngol 2023 10.1007/s00405-023-08270-9 37980605
R.C.T. Cheong S. Unadkat V. Mcneillis A. Williamson J. Joseph P. Randhawa P. Andrews V. Paleri Artificial intelligence chatbots as sources of patient education material for obstructive sleep apnoea: ChatGPT versus Google Bard Eur Arch Otorhinolaryngol 2024 281 2 985 993 10.1007/s00405-023-08319-9 37917165
M. Ostrowska P. Kacała D. Onolememen K. Vaughan-Lane A. Sisily Joseph A. Ostrowski W. Pietruszewska J. Banaszewski M.J. Wróbel To trust or not to trust: evaluating the reliability and safety of AI responses to laryngeal cancer queries Eur Arch Otorhinolaryngol 2024 10.1007/s00405-024-08643-8 38652298
L.A. Vaira J.R. Lechien V. Abbate F. Allevi G. Audino G.A. Beltramini M. Bergonzani P. Boscolo-Rizzo G. Califano G. Cammaroto C.M. Chiesa-Estomba U. Committeri S. Crimi N.R. Curran F. di Bello A. di Stadio A. Frosolini G. Gabriele I.M. Gengler F. Lonardi F. Maglitto M. Mayo-Yáñez M. Petrocelli R. Pucci A.M. Saibene G. Saponaro A. Tel F. Trabalzini E.M.C. Trecca V. Vellone G. Salzano G. De Riu Validation of the Quality Analysis of Medical Artificial Intelligence (QAMAI) tool: a new tool to assess the quality of health information provided by AI platforms Eur Arch Otorhinolaryngol 2024 10.1007/s00405-024-08710-0 39394329 11392976
D. Charnock S. Shepperd G. Needham et al. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices J Epidemiol Community Health 1999 53 105 111 1:STN:280:DyaK1MzisVOqsQ%3D%3D 10.1136/jech.53.2.105 10396471 1756830
Brenner MJ, Pandian V, Milliren CE, Graham DA, Zaga C, Morris LL, Bedwell JR, Das P, Zhu H, Lee Y, Allen J, Peltz A, Chin K, Schiff BA, Randall DM, Swords C, French D, Ward E, Sweeney JM, Warrillow SJ, Arora A, Narula A, McGrath BA, Cameron TS, Roberson DW, Lee YJ, Grimm D, Divi V (2020) Global Tracheostomy Collaborative: data-driven improvements in patient safety through multidisciplinary teamwork, standardisation, education, and patient partnership. Br J Anaesth.; 125(1):e104-e118. https://doi.org/10.1016/j.bja.2020.04.054.
Nielsen JPS, von Buchwald C, Grønhøj C. Validity of the large language model ChatGPT (GPT4) as a patient information source in otolaryngology by a variety of doctors in a tertiary otorhinolaryngology department. Acta Otolaryngol. 2023; 143(9):779–782. https://doi.org/10.1080/00016489.2023.2254809.
Ayoub NF, Lee YJ, Grimm D, Divi V Head-to-Head Comparison of ChatGPT Versus Google Search for Medical Knowledge Acquisition. Otolaryngol Head Neck Surg. 2023. https://doi.org/10.1002/ohn.465.
S.A. Shen C.A. Perez-Heydrich D.X. Xie J.C. Nellis ChatGPT vs. web search for patient questions: what does ChatGPT do better? Eur Arch Otorhinolaryngol 2024 281 6 3219 3225 10.1007/s00405-024-08524-0 38416195 11410109
J. Langlie B. Kamrava L.J. Pasick C. Mei M.E. Hoffer Artificial intelligence and ChatGPT: an otolaryngology patient’s ally or foe? Am J Otolaryngol 2024 45 3 104220 10.1016/j.amjoto.2024.104220 38219629
L.A. Vaira J.R. Lechien V. Abbate F. Allevi G. Audino G.A. Beltramini M. Bergonzani A. Bolzoni U. Committeri S. Crimi G. Gabriele F. Lonardi F. Maglitto M. Petrocelli R. Pucci G. Saponaro A. Tel V. Vellone C.M. Chiesa-Estomba P. Boscolo-Rizzo G. Salzano G. De Riu Accuracy of ChatGPT-Generated information on Head and Neck and Oromaxillofacial surgery: a Multicenter Collaborative Analysis Otolaryngol Head Neck Surg 2024 170 6 1492 1503 10.1002/ohn.489 37595113
O. Kuşcu A.E. Pamuk N. Sütay Süslü S. Hosal Is ChatGPT accurate and reliable in answering questions regarding head and neck cancer? Front Oncol 2023 13 1256459 10.3389/fonc.2023.1256459 38107064 10722294
