M. Narayan1, R. A. Edwards2 1Weill Cornell Medical College,The Division Of Trauma, Burns, Critical And Acute Care Surgery,New York, NY, USA 2MGH Institute Of Health Professions,Center For Interprofessional Studies And Innovation,Boston, MA, USA
Introduction:
There is concern that graduating chief residents lack skills in important non-technical areas. The non-technical skills aspects of the ACGME 6 core competencies have not been broadly evaluated. We examined the literature on assessment of one, Practice-Based Learning and Improvement (PBLI), to identify assessments being utilized by residency programs to meet this competency.
Methods:
A literature review was conducted using PubMed and Google Scholar to identify publications about the ACGME PBLI competency or any PBLI sub-competency. These were then sorted into 1 of 3 categories: Intervention Studies, Survey/Needs Assessment, or Reviews and identified specialty, year of study, journal type, type of assessment, and which sub-competencies were addressed primarily/secondarily. Intervention studies were assessed using the Kirkpatrick model to evaluate type of intervention effectiveness.
Results:
66 publications were identified: 34 Interventions, 15 Survey/Needs Assessments, and 17 Reviews, including commentaries. Internal Medicine (N=21, 31.3%) and Surgery (N=20, 29.9%) had the most. Publications occurred 1998-2017, with 69.7% in 2007-2014. Of the 16.7% in the Journal of Surgical Education (JSE), only 3 were intervention studies and 7 were reviews. 52.9% of Intervention studies were published in other Education journals compared to 8.8% in JSE. JSE published more review/commentary publications (41.2%) compared to other journals. A majority of intervention studies assessed the PBLI sub-competency on quality improvement (QI) as the primary objective. The sub-competencies associated with self-awareness including identifying strengths, deficiencies, and limits in one’s knowledge and expertise, setting learning and improvement goals, and identifying and performing appropriate learning activities were the most commonly assessed secondary objectives. Only 3 of 34 Intervention studies used qualitative methods. The most commonly used quantitative study designs were pre- and post-assessments (66.7%) followed by post-assessments only (18.2%). Only two studies (6.1%) used an OSCE as the sole method of assessment. 91.2% of intervention studies focused on Kirkpatrick’s Level 2 (knowledge/skills) while 64.5% included Level 1 (satisfaction). 6 studies (17.6%) focused on Level 3 (observed behaviors). If self-reported behavior is included, then 41.2% of studies included a behavior component. Only 2 studies focused on Level 4 (health outcomes).
Conclusion:
Current literature on the PBLI competency has primarily focused on QI. Assessment tools have been relatively limited in their scope, primarily using pre- and post-intervention tools. Higher Kirkpatrick levels have not been assessed much. Study designs can be improved through greater use of randomization, OSCEs, 360-degree evaluations and simulation. Much more work is needed to analyze how the PBLI sub-competencies can best be assessed to assure residents are meeting the prescribed ACGME competencies.