J.A. Kasmirski1, A.A. Harsono1, Z. Song1, A. Gillis1, J. Fazendin1, H. Chen1, B. Lindeman1 1University Of Alabama at Birmingham, Department Of Surgery, Birmingham, Alabama, USA
Introduction: Efforts to expose trainees to an environment that promotes well-being, incentivizes learning experiences, and fosters professionalism have been increasingly scrutinized in surgical education. We aim to assess the existence of highly-reliable surveys used to measure professionalism, the learning environment, and mistreatment in surgical education of both medical students and residents.
Methods: Studies that used surveys to measure professionalism, learning environment, and mistreatment in surgical residents and medical students were queried from Pubmed, Embase, Scopus, and Web of Science platforms. Data were collected from each study regarding: the number of participants, the country of origin, whether the survey was new or adapted, and the number of questions. The 10-item MERSQI instrument (score range 3.5-18) was utilized to evaluate study design, sampling, quality of data analysis, validity evidence, and outcomes. Three raters independently evaluated the surveys, with discussion to resolve disagreement. Intraclass correlation coefficient (ICC) was used to calculate inter-rater reliability. Median and interquartile range(IQR) scores were calculated for each MERSQI instrument criterion.
Results: Initial search yielded 2,385 articles. Only 40 were eligible for full-text review, with 25 suitable for data extraction. Among these 25 studies, none simultaneously applied more than one assessment instrument. MERSQI inter-rater reliability was 0.97 (95% CI 0.93-0.99), demonstrating excellent agreement between raters. Median MERSQI scores were similar among included surveys by category (n, median [IQR]): mistreatment surveys (n=7, 11.6 [9.6-14]; learning environment surveys (n=10, 12.2 [11-14]); and professionalism surveys (n=6, 12.1 [11.1-15]). A majority of (n=5/6, 83.4%) of professionalism studies were novel instruments, compared to 43% (n=3/7) mistreatment, and 30% (n=3/10) of learning environment studies. Single-institution studies comprised 80% of the articles included, and all were considered single-group cross-sectional studies. Nearly all (92%) utilized participant self-assessment. Only 36% of studies adequately mentioned or assessed validity evidence including content validity, relationship with other variables, and internal structure.
Conclusion: The majority of mistreatment and professionalism surveys in surgical education are created or adapted by single institutions and do not report sufficient evidence to support their validity. There is a need for surgical educators to define and subsequently promote utilization of well-established learning environment surveys to assess trainee perceptions, in order to best enhance the generalizability and comparability of study results.