The National Resident Matching Program (NRMP) is evaluating the utility of a proposed Rank Order List (ROL) Lock functionality for programs and requests your input. The public comment period will be open through April 19, 2023, and may be accessed at: https://survey.alchemer.com/s3/7250540/Public-Comment-Survey-on-Program-Rank-Order-List-Lock-Functionality-2023.
We appreciate you taking time to provide your perspective so that discussion about the proposed functionality, including community feedback, can be held at the May 2023 meeting of the NRMP Board of Directors.
The call for public comment is also available as a PDF.
Background: The Limits of Virtual Interviewing
The COVID-19 pandemic disrupted residency and fellowship recruitment, forcing a pivot to a virtual interview format from traditional in-person interviews for the 2021 application, interview, and match cycle. This practice has continued through the current cycle, with several specialty groups and organizations providing guidance to programs about virtual versus in-person interviews. Some studies and surveys have highlighted both the benefits and challenges of virtual interviewing1-5. While the virtual interview format has resulted in reduced costs for applicants and programs alike and, in some specialties, has increased the diversity of interview candidates, the format has also created disparities in how programs and specialties conduct the interview season. Virtual interviews have also limited the ability of applicants to assess first-hand the program culture, organization, and the surrounding community and for programs to fully assess applicant “fit” and effectively highlight their program and community6. Additionally, some applicants are requesting in-person visits towards the end of interview season, prior to the Rank Order List (ROL) Certification Deadline, due to uncertainty as to whether programs use second looks as an evaluative component of the application process.
Call for Change: Staggered Rank Order Lists
To address perceived barriers and uncertainty, some specialty groups, the Coalition for Physician Accountability (CoPA), and the Organization of Program Directors Association (OPDA) have requested the NRMP consider implementation of a mandatory “staggered ROL” certification deadline for the Main Residency Match (MRM) and the Specialties Matching Services® (SMS®). As proposed, a staggered ROL certification deadline would require programs to certify their ROL before applicants (a two-week difference has been proposed), establishing a window during which applicants could visit programs of interest and engage in meaningful assessment of program culture without concern that programs may make changes to their ranking preferences afterward. Applicants without the financial means to engage in in-person visits, but who fear not visiting would be perceived as a lack of interest in the program, would not have to worry that lack of participation would impact their ranked status on a program’s list.
However, implementation of a mandatory staggered ROL certification deadline could have foreseeable and unintended consequences:
- Impact on medical education curricula: It is foreseeable in the Main Residency Match that large numbers of senior medical students would leave rotations simultaneously to travel and participate in the two-week visit period, disrupting the fourth-year curriculum. While medical schools may place limits on the number of visits and time away from the curriculum allowed for their students, this would introduce disparity between applicant types as independent applicants such as international medical graduates (IMGs) would not have similar restrictions.
- Impact on clinical rotations: Residents participating in the SMS would leave clinical rotations or other residency training assignments to travel and participate in the two-week visit period. With specialty matching being more distributed throughout the year, there is less impact at one time as can be expected with medical students, but there would still be schedule and coverage consideration for programs and residents.
- IMGs may experience inequitable opportunity due to an inability to participate in the visit period if not currently in the United States.
- A mandatory staggered deadline could force programs to prematurely complete their recruiting and ranking processes weeks in advance of the usual certification deadline, regardless of whether they intend to offer in-person visits.
- A mandatory early ROL certification deadline for programs could differentially impact applicants who have not completed USMLE (United States Medical Licensing Examination), COMLEX, ECFMG certification, or visa requirements by that deadline.
Proposed Alternative: Program Rank Order List Voluntary “Lock” Functionality
The NRMP has discussed the feasibility of a mandatory staggered program ROL lock with several organizations and specialty groups over this past two years and has substantial concern about foreseeable consequences that, once operationalized, are likely to lead to dissatisfaction across the community. However, in consideration of requests from the community, the NRMP is considering the benefits of “lock” functionality in the Registration, Ranking, and Results® (R3®) system such that programs could certify their ROL when they have completed their ranking decisions and then voluntarily lock the certified list when they are ready to demonstrate to applicants that no additional changes will be made. The voluntary “lock” could be implemented at any time after the ROL opens and before the ROL Certification Deadline.
Foreseeable benefits to a voluntary lock option include:
- Programs could announce their intention to certify and lock their ROL and their in-person visit dates to applicants in advance of the application and interview processes.
- Applicants could plan which programs they intend to visit as they complete virtual interviews and are free to decide whether to visit a program before submitting and certifying their own ROL.
- Specialties could establish guidance for residency and fellowship training programs aligning with specialty aims and needs.
- Once implemented, the program’s “lock” status would be visible to applicants and medical schools which may assist in their planning.
Guiding Principles of a Voluntary Rank Order List “Lock” Function:
- The ROL Certification Deadline would remain the same for both applicants and programs.
- The decision to lock a ROL would be made by each program on a voluntary basis and at a time determined by the program.
- The ROL would have to be certified before the list could be locked, and it must be certified before the ROL Certification Deadline.
- Once locked, the ROL could not be unlocked to make changes, even if the request is made before the certification deadline. This is an irreversible action.
Summary and Request for Public Comment
For the application and recruitment phases of the transition to residency to be effective, they must offer the most transparency and insight as possible for applicants and programs to make meaningful, educated decisions about ranking. The virtual platform remains a primary format for interviews and one that is heralded by both applicants and programs as an effective way in which to broaden consideration of training programs and candidates while also helping reduce costs of an already costly endeavor. Nonetheless, the limited ability to determine alignment of personal and professional goals and outcomes through virtual interviewing begs consideration of new features and processes that could leverage the format without also creating harm or additional burden on applicants.
To that end, the NRMP Board is soliciting public comment on the perceived benefits and risks with implementation of a voluntary program ROL “lock” concept. The public comment period will be open through April 19, 2023, and may be accessed at: https://survey.alchemer.com/s3/7250540/Public-Comment-Survey-on-Program-Rank-Order-List-Lock-Functionality-2023
On behalf of the NRMP Board of Directors and NRMP Leadership, I thank you for your time and input regarding this important discussion.
Donna L. Lamb, DHSc, MBA, BSN
President and Chief Executive Officer
1Robinson KA, Shin B, Gangadharan SP. A Comparison Between In-Person and Virtual Fellowship Interviews During the COVID-19 Pandemic. J Surg Educ. 2021 July-Aug;78(4), 1175-1181. https://doi.org/10.1016/j.jsurg.2020.11.006
2Tseng J. How Has COVID-19 Affected the Costs of the Surgical Fellowship Interview Process? J Surg Educ. 2020 Sep-Oct; 77(5):999-1004. https://doi.org/10.1016/j.jsurg.2020.05.018
3Vadi MG, Malkin MR, Lenart J, Stier GR, Gatling JW, Applegate II RL. Comparison of web-based and face-to-face interviews for application to an anesthesiology training program: A pilot study. Int J Med Educ. 2016 Apr 3;7:102-108. https://doi.org/10.5116/ijme.56e5.491a
4Hagedorn II J, Chen J, Weiss W, Fredrickson S, Faillace J. Interviewing in the Wake of COVID-19: How Orthopaedic Residencies, Fellowships, and Applicants Should Prepare for Virtual Interviews. J Am Acad Orthop Surg. 2021 Apr 1;29(7);271-277. doi: 10.5435/JAAOS-D-20-01148
5Strumpf Z, Miller C, Livingston D, Shaman Z, Matta M. Virtual interviews: challenges and opportunities for pulmonary disease and critical care medicine fellowship programs. ATS Sch. 2021 Sep 14;2(4):535-543. doi: 10.34197/ats-scholar.2021-0043PS
6National Resident Matching Program. Research Brief: Virtual Experience and Holistic Review in the Transition to Residency: An Examination of the 2021 and 2022 Mian Residency Matches. https://www.nrmp.org/wp-content/uploads/2022/07/2022-Virtual-Exper-Research-Brief_Final.pdf. Published July 2022. Accessed February 15, 2023.