S. Isaac1, S. Khairat1,3, R. Kitzmiller3, C.C. Greenberg2, M. Phillips2 1University Of North Carolina At Chapel Hill, Carolina Health Informatics Program, Chapel Hill, NC, USA 2University Of North Carolina At Chapel Hill, Department Of Surgery, Chapel Hill, NC, USA 3University Of North Carolina At Chapel Hill, School Of Nursing, Chapel Hill, NC, USA
Introduction:
While surgical education has seen significant advancements, the pace of innovation in this field has not been consistent. One such promising technology is using surgical video for education, performance evaluation, and improvement. However, the key to optimizing these systems lies in employing user-centered design and testing. This is crucial for their seamless integration into complex surgical workflows. Unfortunately, there is a lack of research on the usability, acceptability, and implementation of current surgical video review systems, including those using AI/ML techniques for surgical coaching and training. Usability, a term common in computer science, is the measurement of ease with which a technological innovation can be employed to complete a specified task. Our study aims to assess the usability of local video capture systems in the operating room, assist curriculum design, and assist system acquisition in bridging the gap in surgical educational innovation.
Methods:
To assess usability, a standardized questionnaire, the System Usability Scale (SUS)1, was distributed among a single surgical department at an academic medical center to ask surgeons about the usability of a local video capture system. The SUS is a 10-item questionnaire with five response options on a Likert-type scale. Practicing surgeons and residents completed the SUS, and the results were analyzed.
Results:
The odd-numbered statements of the SUS, 1,3,5,7 and 9, are positively oriented and associated with positive statements. Table 1 shows responses to the positively oriented statements from 16 surgeons who routinely use or have used the surgical video review system for skill assessment, improvement, training, and/or quality improvement. For each question, the five response options ranged from 1 to 5, with one being strongly disagree and five meaning strongly agree. The results were normalized, and the final SUS score was calculated. All the responses' mean scores were below 3, indicating disagreement with positively oriented statements regarding the surgical video review system currently in place.
Conclusion:
The local video capture system did not perform well in SUS testing at a single academic medical center. This underscores the need for significant improvements in user interface and workflow integration to enhance the adoption of video-based coaching in surgical education. These improvements should be made with the end-users in mind. This study will serve as a benchmark for comparing AI-based video capture systems used for surgical coaching and training.
References:
1. Lewis JR. The System Usability Scale: Past, Present, and Future. Int J Hum Comput Interact. 2018;34(7):577-590. doi:10.1080/10447318.2018.1455307