Scales, Intervals, and Dissonance: Do Interpretations of Harmonized ACGME Milestone Levels Vary by User Role and Specialty?

Background: The 2021 ACGME Milestones revision sought to harmonize sets of Milestones across specialties. It is not known whether different users understand the Milestone levels similarly, which is necessary for the harmonized Milestones to facilitate cross-specialty comparisons. Objective: We asked Milestone users of different roles and specialties to assess subjective differences in competence between levels on harmonized Milestones. Methods: Between May and August 2023, we conducted a unidimensional scaling study of resident, faculty, and program director ratings for 8 harmonized Pediatrics Milestones in a sample of pediatrics and internal medicine residency programs. We presented higher- and lower-level descriptions of each Milestone to participants (n=106, 101% of target enrollment of 105). They rated how much more competence the higher level represented on a 1-to-9 scale. We employed a psychological scaling technique to retrieve subjective intervals among Milestone levels and compared them by user roles and specialties. Results: Among the 100 pediatrics respondents included in the analysis, 4 Milestones, ICS1 (patient/family), PBLI2 (reflective practice), SBP1 (patient safety), and ICS2 (interprofessional/team), had 5 distinguishable levels with essentially equal intervals between adjacent levels; residents had somewhat less discrimination than faculty and program directors. For P3 (accountability) and P4 (well-being), there were only 4 distinguishable levels; for PBLI1 and SBP2, only 2. The internal medicine respondents had better discrimination among levels than pediatrics respondents for PBLI1 but worse for P4. Conclusions: For some Milestones, users did not meaningfully distinguish some levels. We also found differences in the subjective valuation of some harmonized Milestone levels between pediatricians and internists.