February 15, 2024

This week on the Holistic Success Show, we welcomed Dr. Alex MacIntosh, Senior Research Manager at Acuity Insights, to discuss the findings of a recent study published by Acuity Insights and its research partners at Rutgers Robert Wood Johnson Medical School and Wake Forest University School of Medicine. The research investigates how language and SES (socioeconomic status) impact an applicant’s performance on the Casper test. 

What is the Casper test?

The Casper test is a unique open response situational judgment test (SJT) that measures how individuals understand, interpret, and respond to social and situational dilemmas. Unlike other tests with multiple-choice answers, Casper asks open-ended questions, delving into how someone approaches a situation and why they approach it that way. It encourages individuals to express their opinions and provide context, offering a deeper look into their communication and problem-solving skills in real-life situations.

Unlike standardized exams such as the MCAT and GRE, the Casper test shows fewer demographic disparities. Compared to knowledge-based standardized tests, Casper reduces the socioeconomic gap by 25 to 50 percent.

Study overview

In partnership with Rutgers and Wake Forest School of Medicine, Acuity conducted a study with data from over 10,000 applicants between two US medical schools. The focus was on the effect of language used at home, distinguishing between bilingual speakers and English monolinguals or those using English as a second language. Additionally, the researchers examined socioeconomic status, refining measures like household income and parent education into a more nuanced metric termed “degree of socioeconomic disadvantage.”

Surprisingly, after accounting for factors like age and gender, bilingual applicants outperformed English-only and English as a second language (ESL) applicants in Casper scores. Notably, English-only and ESL applicants showed similar performance on Casper. The study revealed a crucial insight – having an ESL background doesn’t inherently disadvantage applicants. Additionally, these findings underscore the importance for admissions programs to consider how an individual’s social and cultural factors interact, influencing their attributes, experiences, and evaluation throughout the admissions process.

Implications of the study’s findings on higher education

This study highlights the significance of recognizing and evaluating the impact of an individual’s social and cultural factors on their experiences. It emphasizes the need for multiple measures in the admissions process to assess various attributes comprehensively. Notably, the research reveals that bilingual and English-only applicants with similar socioeconomic status can have differing performance levels, indicating the value of digging deeper into how speaking more than one language could contribute to someone’s ability to consider various perspectives.

The study also debunks the idea that having an ESL background inherently disadvantages applicants. However, given that Casper is a time-bound written and oral assessment, lower English proficiency could still impact Casper’s performance, as this proficiency impacts ease of communication more broadly.

Real-time or near-real-time responses are necessary in addressing an issue, alongside long-form communication. Being a proficient speaker is valuable, but the admissions process should align with the communication demands of the profession. For instance, if the goal is to produce professionals serving a Spanish-speaking population, fluidity in English communication may be less critical.

The study found that admissions programs should consider various aspects of an applicant’s experience, thoughtfulness, and social intelligence, which might not be captured in a single assessment. It parallels how Casper correlates with MMIs and interviews, emphasizing the need for a multifaceted approach to understanding the whole person during the admissions process.

Keeping applicants in context

Understanding how individual experiences, such as languages spoken at home and socioeconomic status, shape a person’s life and educational journey is crucial. These factors have a lasting impact, influencing gaps in opportunities, experiences, and academic performance from an early age and persisting throughout the education system. Admissions committees need to recognize and appreciate these influences.

To truly grasp the individual, admissions need to adopt a mindset that goes beyond traditional metrics like GPA or standardized test scores. The findings of this study advocate for a holistic approach in admissions, urging the need for tools like Casper and thoughtful framing when using personal essays or interviews to capture a person’s unique qualities and experiences. 

In moving towards holistic admissions, varied assessment methods are necessary to capture the diversity of individuals and their experiences reliably. While an ideal scenario might involve extensive personal interactions, practical constraints necessitate using different pieces of information, such as Casper and interviews, to evaluate applicants holistically.

How have the results changed your perspective? 

The study initially focused on linguistic and lexical features in applicants’ writing, examining the frequency and order of verbs, nouns, and adjectives and how they vary based on language use and socioeconomic status. These findings were presented at the Association for Medical Education in Europe. However, the initial study led to the realization that individual variability among applicants surpasses trends observed across demographic groups.

The conversation shifts towards the importance of concentrating on individuals and their unique qualities, considering the ultimate goal of holistic admissions—to understand applicants as whole persons. The dialogue highlights the challenge of balancing macro population trends with a more personalized approach.

The discussion delves into the need for more granular and explicit measures, particularly regarding socioeconomic status (SES). The study also used a custom, sensitive SES measure, moving away from broad groupings and allowing for a more continuous representation of SES. This approach, based on the work of Dr. Henderson and Dr. Jarnette at UC Davis, aims to provide a better understanding of how social and economic opportunities impact educational attainment.

Takeaway

This study highlights that ESL status does not automatically lead to poorer Casper performance, which is a common misconception. However, lower English language proficiency can impact real-time communication in Casper, as it does with communication more broadly.

The focus now lies in understanding how their socioeconomic background and life experiences influence an individual’s use of language. These factors shape how an applicant communicates in different situations, which is why a holistic applicant review is important.  

Admission programs need to thoughtfully consider applicants’ academic preparedness as well as their social intelligence and professionalism within the context of their experiences and the opportunities they had to build these skills. 

Watch the full Holistic Success episode for more information.