A case study from William Carey University College of Osteopathic Medicine

---

Key information

Program: William Carey University College of Osteopathic Medicine (WCUCOM)
Location: Hattiesburg, Mississippi

Challenges:

• Poor correlation between MCAT scores and OSCE performance, particularly in soft skills, and low level of diversity in the class composition

Solutions/Modules:

• Implementation of Casper into the admissions process to assess the soft skills of applicants

Benefits:

• High correlation between Casper scores and specific OSCE scores that indicate future professional success
• Increase in representation of minority students from 5% to 12%

---

In 2019, the administrators at William Carey University College of Osteopathic Medicine (WCUCOM) knew they had a problem.

Osteopathy is a holistic medical field. Doctors of Osteopathic Medicine are trained to see the body as a single, functioning unit and use manual techniques to heal.

Students applying to WCUCOM were primarily assessed by their MCAT scores.

The admissions team was looking for a more holistic way of evaluating applicants to better mirror the skills they would be learning.

The problem

In 2019, an analysis was done comparing WCUCOM students’ MCAT scores with the results of their board examinations. This analysis found a moderate correlation between the biological sciences portion of the MCAT and exam scores.

Unofficially, the team also looked at students who faculty had noted most often in terms of their lack of professionalism and other soft skills, and found that these students had scored highly on the MCAT.

While math and science are certainly important factors for anyone studying health sciences, soft skills including professionalism and communication often make the difference between a good doctor and a great one, because these skills create a caring, empathetic bedside manner.

The school also faced critically low racial diversity. WCUCOM is located in Mississippi where 36% of the population is Black, but on average, only 5% of the student body at WCUCOM was Black.

“We already knew there was bias in standardized testing, and this was misaligned with our focus on diversity and inclusion,” recalls Mallorie Davis, Director of Student Affairs at WCUCOM.

It was clear that in order to produce great doctors that represent the patient population, something needed to change.

Time for a Change

Mallorie Davis and Elizabeth Smith-Trigg, Associate Dean, Assessment and Curricular Affairs, were looking for a tool to help WCUCOM be more holistic and better align with principles that are at the core of osteopathic medicine.

They attended an AACOM conference where they heard about Casper, a situational judgment test (SJT) used by over 150,000 college and university applicants each year.

Casper assesses social intelligence and professionalism through both written and video formats, allowing test-takers to explain how they would handle a particular situation and why.

While Casper is a standardized assessment, its open-response format (as opposed to the multiple-choice format of other SJTs) and emphasis on letting applicants express individual experiences and rationale have resulted in smaller performance gaps between applicants from varying backgrounds, compared to knowledge-based tests, where there are bigger differences among applicants.

Building buy-in 

After learning about Casper, Smith-Trigg and Davis were confident they had found the solution they’d been looking for.

The Dean was on board right away, but they now faced the challenge of convincing their larger team about the value of situational judgment tests.

Building buy-in was a challenge, Davis recalls. “We really had to baby step our way into it. The team had a process that seemed to be working and didn’t understand why it needed to change.”

Davis and Smith-Trigg informed the stakeholders that Casper was being used in over 50 medical schools in the United States. But that wasn’t enough.

According to Davis, “They’re all research-minded and so you can’t just tell them something and expect them to believe it. They need to see it in action, and they need to see it work.”

To prove that integrating Casper was the right way to go for WCUCOM, Davis and Smith-Triggs decided that the best path ahead was to measure the impact of this new program and provide data-driven evidence to get the buy-in they needed from stakeholders.

Implementation

Casper was an optional part of their application process when it was first introduced at WCUCOM in 2019, and the data showed results right away. This caused a change in perspective from the overall team.

Smith-Trigg recalls that “after the first year when I was able to get some data, more of the faculty were on board with what we were doing. So then our admissions people began taking time to ensure students were completing the Casper test, which gave us even more data. Now we have more people come on board rather than shying away.”

Faculty participation played an equally important role. Davis explains that “the big thing for us was getting our people to do the interviews and sit on the committees and make the recommendation to understand what the score is and how it applies to the student.”

After two cycles, the policy changed at WCUCOM, and in 2020 Casper became a mandatory element of the applicant assessment process.

The team had found that not only was Casper an instrumental tool to introduce a holistic admission strategy to the WCUCOM team, but it could also predict success in soft skills, which became increasingly important because the COMLEX physical examination had been canceled due to the COVID-19 pandemic.

Results

Before using Casper, the WCUCOM team knew that high MCAT scores correlated with advanced skills in math and science, but not soft skills. This was based on analysis of how MCAT scores correlated with board examination results.

After implementing Casper as a measurement tool across the WCUCOM cohort, the team began to evaluate these test results alongside the Objective Structured Clinical Examination (OSCE) taken in OMS2 year and beyond.

Students who did well in their Casper testing also demonstrated all desired behaviors assessed on the OSCE, while higher MCAT scores were not a predictor of OSCE performance.

It soon became clear that situational judgment testing was a better indicator of advanced soft skills than the MCAT, including patient communication and relationship building.

In addition, before Casper, only 5% of WCUCOM students were Black. After building buy-in for Casper and making their admissions process more holistic to better align with principles of osteopathic medicine, that number jumped to 12%.

Casper has proven to be a transformative tool for WCUCOM, yielding remarkable results while also identifying which learners would benefit from proactive support, to help them succeed.

Davis and Smith-Trigg feel strongly that “the more information you have on an applicant, the better.”

When they look back on their experience, they both offer similar advice to admissions professionals introducing Casper to their schools. To them, staff participation and training is key.

“Have the administrators on board with the program and then spend as much time as possible teaching faculty and others, ensuring they understand what Casper is and how it will help. It’s not even about convincing them; it’s about helping them understand what is being done and why.”

To learn more about Casper and its potential impact on the admissions process at your college or university, visit: https://acuityinsights.com/about-us/contact-us/