L.D. Bozeman1, A. Abdullah2, G.C. Hernandez-Marquez3, W.M. Oslock3, A. Amjad4, A. Abdullah4, L. Wood3, B.A. Jones3,5, R. Hollis3, M. Rubyan6, D.I. Chu3 1University Of Alabama at Birmingham, Heersink School Of Medicine, Birmingham, Alabama, USA 2University Of Pittsburg, Trauma And Transfusion Department, Pittsburgh, PA, USA 3University Of Alabama at Birmingham, Department Of Surgery, Birmingham, Alabama, USA 4Aga Khan University Medical College, Karachi, Sindh, Pakistan 5University Of Texas Southwestern Medical Center, Department Of Surgery, Dallas, TX, USA 6University Of Michigan, School Of Public Health, Ann Arbor, MI, USA
Introduction:
Effective perioperative education can reduce patient anxiety and improve surgical outcomes. Unfortunately, those with limited English proficiency (LEP) often experience inadequate and/or unavailable translation services. Artificial intelligence (AI) platforms may provide a scalable way to translate written education materials into other languages. Given this, we aimed to study the linguistic accuracy of publicly accessible and free-of-cost AI platforms in translating English patient education materials into Spanish, the second most spoken language in the US.
Methods:
English colorectal patient educational materials from a tertiary health center were collected for three categories: preoperative, postoperative, and ostomy-related care (n=63). The primary outcome assessed was readability, which was assessed using Flesch-Kincaid Grade Level (FKGL) and Simple Measure of Gobbledygook (SMOG) scores. The English materials then underwent Spanish translation by the four AI platforms, Chat GPT 3.5, Google Bard, Microsoft Bing, and Google Translate (n=252). Spanish materials were reviewed by a native Spanish-speaking healthcare professional to evaluate language accuracy. Readability of the AI-generated Spanish educational materials was assessed using the Spanish FKGL and SMOG scores. Results for translated materials were compared to English materials via paired t-tests.
Results:
From 63 original English educational materials, a total of 252 AI-generated Spanish educational materials were compared (63 for each AI platform). Most translated materials had at least one translation error: 89% for Google Translate, 85% ChatGPT and 86% Bard, though Bing only had errors in 49% of translated patient education materials. In terms of readability, existing English documents did not meet readability guidelines with an average grade level of 6.6 and 10.4 on the FKGL and SMOG respectively. Google Translate showed a significant difference in SMOG scores, with an average of 6.3 compared to the original English documents (p<0.05), though other translated materials were not significantly different. Variability in readability score values (FKGL) after translation from all the AI engines increased compared to that of the original materials. (Figure 1)
Conclusion:
AI platforms are able to translate English educational materials to Spanish, but significant issues exist in readability and linguistic accuracy. Further research is needed to address the scope of AI translation tools and accessibility of translation services.