Back to Blog
Have you ever wondered why Casper was developed as an open-response situational judgment test (SJT)? Or maybe you’re wondering what those words mean, exactly.
This week on the Holistic Success Show we welcomed Dr. Alex MacIntosh, Scientist and Senior Manager, Research at Acuity Insights, who spoke with our VP, Science & Innovation and Co-Founder, Dr. Kelly Dore about the world of open-response SJTs.
Read on to discover how Casper was intentionally designed to positively impact the world of higher education admissions – and the evidence behind the design.
What is a situational judgment test (SJT) and how do they work?
Situational Judgment Tests (SJTs) are a well-established assessment method used in both employee selection and higher education admissions. These tests present candidates with hypothetical social or workplace scenarios to gauge their reactions and behaviors. SJTs can be context-specific, assessing knowledge of expected behavior in a specialized workplace setting, or more general, evaluating the effective use of interpersonal skills in commonplace situations or settings. The format and design of an SJT play a crucial role in achieving specific assessment goals, whether context-specific or focused on interpersonal skills and attributes.
What are the two main formats of SJTs?
SJTs primarily come in fixed/closed response formats, offering predefined actions or behaviors for applicants to select, such as multiple choice options or ranking choices. However, there’s a growing trend in open response SJTs. In open response format, candidates aren’t constrained by preset choices; instead, they provide their responses in written or video form. This format not only assesses their ability to identify appropriate actions but also requires them to communicate their decision-making process, rationale, and how it aligns with their experiences.
What are the differences between open and closed response formats?
In the context of fixed/closed response SJTs the assessment primarily focuses on whether the applicant recognizes and selects the expected behavior.
On the other hand, open response SJTs reveal a more nuanced perspective. While some applicants may choose similar actions, the way they navigate, approach, and articulate their choices provides valuable insights. For instance, in a scenario involving confronting someone not pulling their weight, a fixed response might include “confront them directly.” Many candidates might opt for this action, but the specifics of how, when, and what they communicate during the confrontation become evident, shaping the evaluation of their response.
Why does Casper utilize an open response test format?
Casper by Acuity Insights uses an open response format because this format assesses an applicant’s social intelligence and professionalism in a nuanced and evidence-based way.
Applicants provide individualized and contextualized responses, revealing the nuances, or the “why”, of their decision-making process and how their response aligns with their own personal values and lived experiences.
Rating open response SJTs like Casper is more intricate, but yields a valid and reliable score, predicting success across diverse interpersonal situations. This approach highlights how applicants justify their approaches, offer creative solutions, and consider multiple perspectives. Casper helps programs to identify applicants who showcase holistic competence, beyond academics. In addition, open response SJTs are less susceptible to “gaming” or “faking” – allowing for a more accurate assessment.
How does Acuity use Subject Matter Experts in the development of Casper?
Acuity Insights takes a meticulous approach to scenario and question development for the Casper test, involving subject matter experts closely in the process. Our panel of experts represents diverse backgrounds and experiences, with approximately a quarter of them coming from health professions.
Additionally, we draw upon a wide range of individuals from the community, encompassing those with expertise in question and test development, as well as individuals passionate about promoting access and equity in education. These community members actively contribute to building the test, aligning their efforts with the communities we aim to serve. This collaborative approach ensures the Casper test’s quality and relevance to a broad spectrum of applicants.
How does Casper consider candidates’ cultural contexts and personal histories in its assessments?
The Casper test embraces the complexity of real-life scenarios by acknowledging that there is no single right answer. Our approach to test development is closely tied to the specific geography in which it is applied. For instance, when considering applicants in the United States and Canada, we craft scenarios and questions within the context of the prevailing social norms in these regions.
Take, for example, a scenario where an individual is reluctant to engage in public speaking within a group setting. In response, candidates may choose to advocate for pushing past one’s fears, emphasizing the importance of overcoming challenges in their cultural context. Conversely, they might prioritize autonomy and mental well-being, devising a supportive strategy for the individual. These distinct approaches, while leading to potentially different outcomes, underscore the richness of human responses.
The open response format allows candidates to incorporate their cultural context and personal history into their answers, providing evaluators with a comprehensive view of their decision-making processes. This nuanced approach ensures that applicants’ diverse perspectives and values are considered, contributing to a holistic assessment that aligns with our commitment to fair and equitable admissions processes.
How did Casper’s recent adaptation of video responses improve the test?
Casper’s recently integrated video responses within the assessment have demonstrated reduced demographic differences in test performance.
This new approach offers both text-based and video-based scenario presentation and response. Applicantsread or watch videos and type or record their responses, reflecting the multifaceted nature of real-life communication.
Recognizing that individuals from various social and cultural backgrounds may have preferences for different mediums, this mixed modality format accommodates diverse communication styles. Applicants find the video format positive, both in the assessment process and its fairness in evaluating personal characteristics.
Pilot studies exploring the relationship between Casper’s video response format and multiple mini-interview (MMI) performance reveal approximately a 0.1 increase in the Pearson correlation between the two outcomes. This video component brings Casper closer to MMI, enhancing fidelity and aligning with the goal of assessing candidates in a manner more akin to real-world scenarios.
How do you recommend programs adapt or use Casper in their admissions process?
Casper is most effective as a pre-screening tool when used early in the applicant selection process, particularly for assessing which candidates are likely to excel or struggle during interviews.
In situations where there is a large applicant pool and conducting in-depth holistic reviews for all candidates is impractical, Casper scores serve as a valuable means of categorizing applicants into different groups. It allows for prioritization, fast-tracking those who show promise and warrant closer consideration, and categorizing others for more detailed assessments based on their backgrounds and experiences.
Casper’s effectiveness lies in its ability to reveal a connection between top and bottom interview performance, aiding institutions in making informed decisions about which candidates to advance in the selection process.
Watch the full episode to learn more about the value of Casper and its open-response format!
