How Open-Response SJTs Reflect Nuances of Social Intelligence and Professionalism

December 17, 2024

The Research Team at Acuity Insights recently published a new paper in Frontiers in Medicine, marking a notable contribution to the field of open-response SJTs and construct validity research. 

The full paper, “Evaluating Factors that Impact Scoring an Open Response Situational Judgment Test: A Mixed Methods Approach”, is available here.

Acuity’s new study highlights the subtle and nuanced factors of social intelligence and professionalism that are reflected in responses to the Casper test and Casper scores. We hope this information will aid applicants as they prepare for their Casper test.

Details about the study

Open-response SJTs, like Casper, are often associated with stronger reliability and validity, when compared to close-ended SJTs (e.g., multiple choice or forced choice). The open-response format provides test takers the opportunity to incorporate their own lived experiences into their responses. 

While scoring guidelines are provided to assessors, the varied and complex responses resulting from the open-response format allows for greater nuance in interpretation. In addition to the scoring guidelines, assessors may take additional response attributes into account when assigning scores (Gingerich et al, 2011; Wood, 2014; Hasan & Jones, 2024). 

The Research Team at Acuity Insights sought to gain a deeper understanding of the implicit factors of social intelligence and professionalism that may be captured in open-response SJTs by asking: Which implicit factors, in addition to the scoring guidelines, play a role in assessors’ evaluation of test takers’ unique answers to open-response SJT scenarios?

What were the results of the study?

To provide a holistic view of the research question, we employed a mixed-methods approach. Using statistical analyses and qualitative interviews, we identified several subtle and nuanced factors of social intelligence and professionalism that are reflected in the scores. This study found that Casper scores provide insight into the extent to which an applicant can:

• Reflexively use examples to demonstrate key aspects of social intelligence and professionalism
• Understand and interpret social dilemmas in nuanced and complex ways
• Thoughtfully justify their approach to the presented situation
• Provide novel or creative ideas or solutions to the presented situations
• Apply critical thinking to situations that require a professional approach
• Demonstrate sound professional and ethical judgment
• Carefully consider multiple perspectives

What does this mean for applicants?

There are no right or wrong answers in the Casper test because its open-response format allows you to answer the question in any way that is most authentic to you. However, we do have some suggestions regarding response characteristics that may be helpful.

• Know the characteristics (empathy, communication, etc.) Casper probes for and consider how you could apply them.
• Use examples to demonstrate the characteristics Casper probes for.
• Justify your approach to the situation. The why is just as important as the what.
• Consider various perspectives when determining your approach to the situation.

Looking for more information?

• Explore our applicant website, where you can find more helpful information and a full-length practice test.
• Check out our Holistic Success Show episode on Casper scores
• Read our full paper in Frontiers in Medicine