N. Rossetti1, U. Panni1, T. Brocke1, A. Colbeck1, M. Houdos1, C. Donald1, P.E. Wise1 1Washington University in St. Louis, Surgery, St. Louis, MISSOURI, USA
Background:
Surgical residency programs receive high volumes of applications every year. To efficiently identify genuinely interested applicants, non-traditional strategies such as institution-specific secondary applications have been implemented, although their utility and effectiveness is unclear with the introduction of preference signaling and evolving Electronic Residency Application Service (ERAS) supplementary applications. We sought to assess the impact of an institution-specific secondary application in identifying interested and appropriate residency applicants to our university-based general surgery residency program.
Methods:
Our program's secondary application consists of three questions designed to gauge applicants' interest in working with a surgical mentor, relocating to St. Louis, and maintaining a work-life balance. Assistant program directors (APDs) and other faculty reviewing the applications were asked to complete a 3-point Likert-type question about the impact of our secondary applications on their holistic application review of 305 categorical applications, randomly selected from the 2023-24 match applicants. The surveys were completed during the application review process to avoid bias. The responses were analyzed across two reviewer types: APDs and faculty. To minimize bias, these surveys were conducted concurrently with the application reviews. Descriptive analysis was performed, and inter-rater reliability between APDs and faculty was assessed using the kappa statistic.
Results:
The review involved 5 APDs and 7 faculty members. Both APDs and faculty found secondary applications to negatively impact the review of the applicant in only a few cases (2.0% of applicants identified by the APDs, 2.6% of applicants identified by the faculty). Interestingly, APDs found the secondary applications to be positively impactful more often than faculty reviewers (190 (62.3%) applicants vs. 95 (31.1%) applicants). This discordance became more prominent in the comparative analysis with 114/190 (60.0%) applications where APDs noted the secondary applications as being positively impactful were considered unimpactful by the faculty. The kappa statistic for inter-rater reliability determined only slight agreement between the two sets of reviewers (0.109, CI: 0.025-0.193).
Conclusion:
Institution-specific secondary applications have limited effectiveness as a screening tool in the residency application review process, as evidenced by low rates of applicant exclusion and substantial disagreement between reviewers. These findings suggest the need for more reliable tools to assess applicants' genuine interest and fit for the residency program.