September 21, 2017

In 2015, the New York Medical College (NYMC) School of Medicine became one of the very first schools to adopt Casper, using it to screen applicants that year. Fern Juster, M.D., the Associate Dean of Admissions at NYMC, used data from that year to research how Casper impacted the diversity of NYMC applicants. She presented her findings at the 2016 Canadian Conference on Medical Education (CCME) in Montreal.

In the post below, we present an overview of the highlights from Dr. Juster’s talk.

The current state of healthcare reform in the United States is focused on three things:

1. Improvements for individual and population health
2. Lowering costs
3. Developing a diverse and inclusive physicians workforce for an increasingly diverse population

But reform doesn’t just affect how physicians practice; it also affects who practices and where. That means that selection of the future physician workforce is key to successful healthcare reform.

Historically, medical school admissions placed major emphasis on grades and standardized test results, like GPA and MCAT scores. But those processes have evolved, and now medical schools have goals of selecting future physicians with both the academic and the interpersonal and intrapersonal competencies needed to work in the healthcare system of the future.

To meet those goals, the following tools are required:

• Holistic review
  An initiative that came out of the Association of American Medical Colleges (AAMC), it emphasizes a balanced approach to reviewing applicants. It considers three components — experiences, attributes, and metrics — to form an overall impression of whether they’ll be good students and good future physicians. It can help to increase diversity, but it’s time-consuming and inefficient.
• Multiple-mini interview (MMI)
  An interview-style assessment where applicants complete a circuit of several independent assessments with multiple interviewers, designed to gauge their soft skills. MMIs have been proven useful in predicting performance, but they require a significant dedication of time and resources, making them difficult to use with large applicant pools.
• Situational judgement test (SJT)
  SJTs, like Casper, are a type of psychological test that guide test-takers through realistic, hypothetical scenarios and ask them to either identify the most optimal course of action, or offer their own hypothetical approach. The AAMC has recommended the use of SJTs as a potential screening tool to measure the core non-academic competencies of applicants, such as empathy, collaboration, integrity, resiliency, and capacity for improvement. It’s feasible to run all eligible applicants through an SJT, giving it a distinct advantage over other tools.

GPA and MCAT scores are used in the screening of applicants, but for applicants who are considered Underrepresented Minorities Medicine (URM), these tools put them at a disadvantage. NYMC defines URM applicants as those of African American, Latino Hispanic, Native Hawaiian, and Native American descent.

Data proves it. Looking at U.S. national data from the MCAT exam administered between 2015 and 2016¹, men scored higher than women. When looking at race/ethnicity, Black or African American applicants scored the lowest out of all ethnic groups, and Hispanics scored the second to the lowest.

That’s why Dr. Juster and her team at NYMC, a medical school with a commitment to diversity and inclusion, was interested in examining what impact Casper, an SJT, could have on the school’s admissions.

Casper is a web-based situational judgement test that can be taken anywhere, so long as applicants have access to a computer. Casper has been proven to be reliable and useful in predicting important outcomes, but for this study, Dr. Juster was primarily interested in examining its impact on the diversity of incoming students.

In 2016, NYMC received approximately 13,000 medical school applicants (which represented about 20% of all medical school applicants in the U.S.). Dr. Juster looked at the number of URM applicants included in that total, and compared it with the number of URM applicants received by the school in 2015, before Casper was implemented.

From the results, Dr. Juster observed the following:

• Amongst various income groups, CASPer scores were equivalent
• Casper scores for URM applicants were lower, but the difference was much less pronounced than the differences observed in GPA and with MCAT
• Applicants with paid employment before the age of 18 (considered an element of disadvantaged status) had higher Casper scores than those who did not have paid employment before the age of 18
• Women scored higher on Casper than men, which is opposite to the gender differences observed with MCAT

In summary: As demographic differences in Casper test performance are smaller compared to GPA and MCAT, Casper, to a limited extent, may help encourage more URM applicants to be considered in the admissions process.

This NYMC study is the first of its kind to provide insights on how SJTs like Casper, used in conjunction with other evidence-based methods, can help encourage larger, more diverse applicant pools.

¹In the presentation, an older dataset from AAMC was presented from 2012 to 2014, but the pattern of results remain the same as the updated dataset.

Watch the Partner Case Study Video on New York Medical College below: