Evaluating AI-Generated informed consent documents in oral surgery: A comparative study of ChatGPT-4, Bard gemini advanced, and human-written consents.
Vaira, Luigi Angelo; Lechien, Jérome; Maniaci, Antoninoet al.
2024 • In Journal of Cranio-Maxillo-Facial Surgery
AI in healthcare; Artificial intelligence; Consent accuracy; Document quality; Informed consent; Large language models; Maxillofacial surgery; Oral surgery; Patient education; Surgery; Otorhinolaryngology
Abstract :
[en] This study evaluates the quality and readability of informed consent documents generated by AI platforms ChatGPT-4 and Bard Gemini Advanced compared to those written by a first-year oral surgery resident for common oral surgery procedures. The evaluation, conducted by 18 experienced oral and maxillofacial surgeons, assessed consents for accuracy, completeness, readability, and overall quality. ChatGPT-4 consistently outperformed both Bard and human-written consents. ChatGPT-4 consents had a median accuracy score of 4 [IQR 4-4], compared to Bard's 3 [IQR 3-4] and human's 4 [IQR 3-4]. Completeness scores were higher for ChatGPT-4 (4 [IQR 4-5]) than Bard (3 [IQR 3-4]) and human (4 [IQR 3-4]). Readability was also superior for ChatGPT-4, with a median score of 4 [IQR 4-5] compared to Bard and human consents, both at 4 [IQR 4-4] and 4 [IQR 3-4], respectively. The Gunning Fog Index for ChatGPT-4 was 17.2 [IQR 16.5-18.2], better than Bard's 23.1 [IQR 20.5-24.7] and the human consents' 20 [IQR 19.2-20.9]. Overall, ChatGPT-4's consents received the highest quality ratings, underscoring AI's potential in enhancing patient communication and the informed consent process. The study suggests AI can reduce misinformation risks and improve patient understanding, but continuous evaluation, oversight, and patient feedback integration are crucial to ensure the effectiveness and appropriateness of AI-generated content in clinical practice.
Disciplines :
Otolaryngology
Author, co-author :
Vaira, Luigi Angelo ; Maxillofacial Surgery Operative Unit, Department of Medicine, Surgery and Pharmacy, University of Sassari, Sassari, Italy, PhD School of Biomedical Science, Biomedical Sciences Department, University of Sassari, Sassari, Italy. Electronic address: lavaira@uniss.it
Lechien, Jérome ; Université de Mons - UMONS > Faculté de Psychologie et des Sciences de l'Education > Service de Métrologie et Sciences du langage ; Université de Mons - UMONS > Faculté de Médecine et de Pharmacie > Service de Chirurgie
Maniaci, Antonino; Department of Medicine and Surgery, University of Enna Kore, Enna, Italy
Tanda, Giuseppe; Dental School, Department of Medicine, Surgery and Pharmacy, University of Sassari, Sassari, Italy
Abbate, Vincenzo; Head and Neck Section, Department of Neurosciences, Reproductive and Odontostomatological Science, Federico II University of Naples, Naples, Italy
Allevi, Fabiana ; Maxillofacial Surgery Department, ASSt Santi Paolo e Carlo, University of Milan, Milan, Italy
Arena, Antonio ; Head and Neck Section, Department of Neurosciences, Reproductive and Odontostomatological Science, Federico II University of Naples, Naples, Italy
Beltramini, Giada Anna; Department of Biomedical, Surgical and Dental Sciences, University of Milan, Milan, Italy, Maxillofacial and Dental Unit, Fondazione IRCCS Cà Granda Ospedale Maggiore Policlinico, Milan, Italy
Bergonzani, Michela ; Maxillo-Facial Surgery Division, Head and Neck Department, University Hospital of Parma, Parma, Italy
Bolzoni, Alessandro Remigio; Maxillofacial and Dental Unit, Fondazione IRCCS Cà Granda Ospedale Maggiore Policlinico, Milan, Italy
Crimi, Salvatore; Department of CHIRMED. Maxillofacial Surgery Section, University of Catania, Catania, Italy
Frosolini, Andrea ; Maxillofacial Surgery Unit, Department of Medical Biotechnologies, University of Siena, Siena, Italy
Gabriele, Guido; Maxillofacial Surgery Unit, Department of Medical Biotechnologies, University of Siena, Siena, Italy
Maglitto, Fabio; Maxillo-Facial Surgery Unit, University of Bari "Aldo Moro", Bari, Italy
Mayo-Yáñez, Miguel; Otorhinolaryngology, Head and Neck Surgery Department, Complexo Hospitalario Universitario A Coruña (CHUAC), A Coruña, Galicia, Spain
Orrù, Ludovica; Dental School, Department of Medicine, Surgery and Pharmacy, University of Sassari, Sassari, Italy
Petrocelli, Marzia; Maxillofacial Surgery Operative Unit, Bellaria and Maggiore Hospital, Bologna, Italy
Pucci, Resi ; Maxillofacial Surgery Unit, San Camillo-Forlanini Hospital, Rome, Italy
Saibene, Alberto Maria ; Otolaryngology Unit, Santi Paolo e Carlo Hospital, Department of Health Sciences, University of Milan, Milan, Italy
Troise, Stefania ; Head and Neck Section, Department of Neurosciences, Reproductive and Odontostomatological Science, Federico II University of Naples, Naples, Italy
Tel, Alessandro; Clinic of Maxillofacial Surgery, Department of Head & Neck Surgery and Neuroscience, University Hospital of Udine, Italy
Vellone, Valentino ; Maxillofacial Surgery Unit, "S. Maria" Hospital, Terni, Italy
Chiesa-Estomba, Carlos Miguel ; Department of Otorhinolaryngology-Head & Neck Surgery, Hospital Universitario Donostia, San Sebastian, Spain
Boscolo-Rizzo, Paolo; Department of Medical, Surgical and Health Sciences, Section of Otolaryngology, University of Trieste, Trieste, Italy
Salzano, Giovanni; Head and Neck Section, Department of Neurosciences, Reproductive and Odontostomatological Science, Federico II University of Naples, Naples, Italy, Maxillo-Facial Surgery Unit, University of Bari "Aldo Moro", Bari, Italy
De Riu, Giacomo ; Maxillofacial Surgery Operative Unit, Department of Medicine, Surgery and Pharmacy, University of Sassari, Sassari, Italy
Evaluating AI-Generated informed consent documents in oral surgery: A comparative study of ChatGPT-4, Bard gemini advanced, and human-written consents.
Abou-Abdallah, M., Dar, T., Mahmudzade, Y., Michaels, J., Talwar, R., Tornari, C., The quality and readability of patient information provided by ChatGPT: can AI reliably explain common ENT operations?. Eur. Arch. Oto-Rhino-Laryngol., 2024, 10.1007/s00405-024-08598-w.
Ayers, J.W., Poliak, A., Dredze, M., Leas, E.C., Zhu, Z., Kelley, J.B., et al. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Intern. Med. 183 (2023), 589–596.
Blackwood, R., Maio, R., Mrdjenovich, A., Vandenbosch, T., Gordon, P., Shipman, E., Hamilton, T., Analysis of the nature of IRB contingencies required for informed consent document approval. Account. Res. 22 (2015), 237–245.
Blease, C., Lilienfeld, S., Kelley, J., Evidence-based practice and psychological treatments: the imperatives of informed consent. Front. Psychol., 10(7), 2016, 1170.
Chiesa-Estomba, C.M., Lechien, J.R., Vaira, L.A., Brunet, A., Cammaroto, G., Mayo-Yanez, M., et al. Exploring the potential of Chat-GPT as a supportive tool for sialendoscopy clinical decision making and patient information support. Eur. Arch. Oto-Rhino-Laryngol. 281 (2024), 2081–2086.
Cocanour, C.S., Informed consent-It's more than a signature on a piece of paper. Am. J. Surg. 214 (2017), 993–997.
Decker, H., Trang, K., Ramirez, J., Colley, A., Pierce, L., Coleman, M., et al. Large Language model-based chatbot vs surgeon-generated informed consent cocumentation for common procedures. JAMA Netw. Open, 6, 2023, e2336997.
Di Battista, M., Kernitsky, J., Dibart, S., Artificial intelligence chatbots in patient communication: current possibilities. Int. J. Periodontics Restor. Dent., 2023, 10.11607/prd.6925.
Glaser, J., Nouri, S., Fernandez, A., Sudore, R.L., Schillinger, D., Klein-Fedyshin, M., Schenker, Y., Interventions to improve patient comprehension in informed consent for medical and surgical procedures: an updated systematic review. Med. Decis. Making 40 (2020), 119–143.
Guarino, J., Parvanova, I., Finkelstein, J., Characteristics of electronic informed consent platforms for consenting patients to research studies: a scoping review. Stud. Health Technol. Inf. 290 (2022), 777–781.
Hopkins, A.M., Logan, J.M., Kichenadasse, G., Sorich, M.J., Artificial intelligence chatbots will revolutionize how cancer patients access information: ChatGPT represents a paradigm-shift. JNCI Cancer Spectr., 7(2), 2023, pkad010.
Kasapovic, A., Ali, T., Babasiz, M., Bojko, J., Gathen, M., Kaczmarczyk, R., Roos, J., Does the information quality of chatGPT meet the requirements of orthopedics and trauma surgery?. Cureus, 16, 2024, e60318.
Kienzle, A., Niemann, M., Meller, S., Gwinner, C., ChatGPT may offer an adequate substitute for informed consent to patients prior to total knee arthroplasty-Yet caution is needed. J. Personalized Med., 14, 2024, 69.
Kinnersley, P., Phillips, K., Savage, K., Kelly, M.J., Farrell, E., Morgan, B., et al. Interventions to promote informed consent for patients undergoing surgical and other invasive healthcare procedures. Cochrane Database Syst. Rev., 6(7), 2013, CD009445.
Lamont, S., Stewart, C., Chiarella, M., Capacity and consent: knowledge and practice of legal and healthcare standards. Nurs. Ethics 26 (2019), 71–83.
Lechien, J.R., Naunheim, M.R., Maniaci, A., Radulesco, T., Saibene, A.M., Chiesa-Estomba, C.M., Vaira, L.A., Performance and consistency of ChatGPT-4 versus otolaryngologists: a clinical case series. Otolaryngol. Head Neck Surg. 170 (2024), 1519–1526.
Liu, J., Wang, C., Liu, S., Utility of ChatGPT in clinical practice. J. Med. Internet Res., 25, 2023, e48568.
Lorenzi, A., Pugliese, G., Maniaci, A., Lechien, J.R., Allevi, F., Boscolo-Rizzo, P., et al. Reliability of large language models for advanced head and neck malignancies management: a comparison between ChatGPT 4 and Gemini Advanced. Eur. Arch. Oto-Rhino-Laryngol., 2024, 10.1007/s00405-024-08746-2.
Nangia, D., Saini, A., Krishnan, A., Sharma, S., Kumar, V., Chawla, A., Logani, A., Quality and accuracy of patient-oriented Web-based information regarding tooth avulsion. Dent. Traumatol. 38 (2022), 299–308.
Pandiya, A., Readability and comprehensibility of informed consent forms for clinical trials. Perspect Clin Res 1 (2010), 98–100.
Patil, N.S., Huang, R., Mihalache, A., Kisilevsky, E., Kwok, J., Popovic, M.M., et al. The ability of artificial intelligence chatobots ChatGPT and Google Bard to accurately convey preoperative information for patients undergoing ophthalmic surgeries. Retina 44 (2024), 950–953.
Radulesco, T., Saibene, A.M., Michel, J., Vaira, L.A., Lechien, J.R., ChatGPT-4 performance in rhinology: a clinical case series. Int Forum Allergy Rhinol 14 (2024), 1123–1130.
Shiraishi, M., Tomioka, Y., Miyakuni, A., Moriwaki, Y., Yang, R., Oba, J., Okazaki, M., Generating informed consent documents related to blepharoplasty using ChatGPT. Ophthalmic Plast. Reconstr. Surg. 40 (2024), 316–320.
Szczesniewski, J.J., Ramos Alba, A., Rodríguez Castro, P.M., Lorenzo Gómez, M.F., Sainz González, J., Llanes González, L., Quality of information about urologic pathology in English and Spanish from ChatGPT, BARD, and Copilot. Actas Urol. Esp. 48 (2024), 398–403.
Vaira, L.A., Lechien, J.R., Abbate, V., Allevi, F., Audino, G., Beltramini, G.A., et al. Accuracy of ChatGPT-generated information on head and neck and oromaxillofacial surgery: a multicenter collaborative analysis. Otolaryngol. Head Neck Surg. 170 (2024), 1492–1503.
Vaira, L.A., Lechien, J.R., Abbate, V., Allevi, F., Audino, G., Beltramini, G.A., et al. Validation of the Quality Analysis of Medical Artificial Intelligence (QAMAI) tool: a new tool to assess the quality of health information provided by AI platforms. Eur. Arch. Oto-Rhino-Laryngol., 2024, 10.1007/s00405-024-08710-0.
Yildiz, M., Kozanhan, B., Tutar, M., Assessment of readability level of informed consent forms used in intensive care units. Med. Sci. 8 (2018), 277–281.
