A. Tran1, N.D. Rajeev1, J. Samaan2, Y.H. Yeo2, K. Samakar1, J.C. Lipham1, S. Shiraga1, C. Houghton1, L.R. Putnam1 1Keck School of Medicine, Division Of Upper GI And General Surgery, Los Angeles, CA, USA 2Cedars Sinai Medical Center, Karsh Division Of Gastroenterology And Hepatology, Los Angeles, CA, USA
Introduction:
Gastroesophageal reflux disease (GERD) is a common condition with multiple surgical treatment options. ChatGPT, an artificial-intelligence (AI)-powered large language model, has emerged as a powerful tool for obtaining wide-ranging information. This study aims to evaluate the quality of ChatGPT-generated responses to questions about GERD surgery.
Methods:
Frequently asked questions (FAQs) regarding GERD surgery were compiled from top-rated health institutions and Facebook support groups. Initial and simplified responses to these FAQs were generated by ChatGPT. Additionally, ChatGPT was prompted to reword institutional responses to improve readability. Responses and rewordings were graded independently by academic foregut surgeons for accuracy and comprehensiveness. Readability was determined using validated readability formulas.
Results:
Eighty-four FAQs and forty institutional responses were compiled. Initial responses were graded “Comprehensive” or “Correct but inadequate” 312/336 (93%) times. Relative to the initial responses, simplified responses were considered equally accurate 91% of the time, but equally comprehensive only 40% of the time. Institutional rewordings were deemed equally accurate and comprehensive 155/160 (97%) and 137/160 (86%) times, respectively.
Initial responses had a 12th grade to college-graduate reading level. Simplified responses were significantly more readable (8th to 11th grade reading level). Readability of institutional rewordings was significantly improved compared to that of the original institutional responses (7th to 10th grade vs 11th to college sophomore).
Conclusion:
ChatGPT produced high-quality responses to GERD surgery questions and was able to simplify responses to improve readability while maintaining accuracy. These results suggest that ChatGPT may be a useful patient resource regarding GERD surgery.