May 28, 2021

Situational judgment tests (SJTs) like Casper are gaining popularity in educational and employment settings, as their high reliability, strong predictive validity, and relatively low cost of administration have made SJTs an attractive tool to assist in selection decisions. Their popularity has also given rise to a growing number of test preparation businesses that attempt to coach people on how to respond to SJTs more effectively.

Casper is no exception. A handful of test preparation companies offer Casper coaching courses in an attempt to improve applicant scores. If their coaching programs are actually successful in improving applicant scores, it would cause an unfair advantage towards those students who do not have access to these programs. As these preparation classes typically cost money, these coaching effects can be particularly detrimental for low-income applicants who are already disadvantaged in the medical selection process.

Research shows coaching effects are small for SJTs

The general literature suggests that coaching effects remain fairly small, with an increase in only 0.5 standard deviation after coaching and an increase in 0.32 standard deviation for students who retook the SJT in the following admission cycle. However, not all SJTs are equally susceptible to coaching effects. When SJTs are more complex and difficult, coaching does not seem to improve subsequent performance. This suggests that tests are only coachable when test-taking strategies are simple and straight-forward, but are ineffective when the strategies are more challenging.

Test format matters

A majority of SJTs use a closed-response format, where a series of pre-constructed answers are presented to respondents, but Casper uses a constructed or open-response format where respondents come up with their own answers to the hypothetical scenarios. Compared to the closed-response format, the constructed-response format tends to be more complex and challenging, requiring a deeper level of processing by applicants, which makes Casper a much harder test to coach than other SJTs. 

What our own data tell us

Preparation Method	Estimate	Std. Error	t	p
(Intercept)	-.34	.02	-20.87	<.001
Reviewed resources on TakeAltus (e.g., FAQs)	.18	.02	9.93	<.001
Completed the three-section practice test on TakeAltus	.12	.02	7.46	<.001
Completed the full-length practice test in Acuity Suite account	.17	.02	10.64	<.001
Participated in a third-party Casper test preparation course	-.10	.04	-2.65	.007
Studied potential Casper questions based on competencies	.13	.02	7.43	<.001
Rehearsed responses with technology	.17	.02	9.20	<.001
Rehearsed responses without technology	-.02	.02	-0.97	.333

We invited Casper test-takers to indicate whether they used any of the following preparation strategies:

• read the tips for applicants on the test website
• completed the free practice test
• participated in a commercial test preparation course
• studied potential questions based on assessment competencies
• rehearsed responses with technology
• rehearsed responses without technology

We then conducted a multiple regression analysis to compare the additive effect of each preparation method on SJT scores reported by our test takers. 

The results showed that of the six preparation strategies, only completing the free practice test on the test website, studying potential questions based on the assessment competencies, and rehearsing responses with technology provided significant additive benefit to test scores. The third-party test preparation method only accounted for 2% of the overall variance in test scores.

What does this mean?

Coaching effects are extremely small. These results highlight the importance of ensuring equitable access to practice tests, and relieve concerns over potential socially regressive impacts of commercial test preparation. 

While there doesn’t seem to be very much that students can do to significantly improve their Casper scores, they can implement a number of strategies to calm their test anxiety and familiarize themselves with the test. We provide a few sample questions and other prep tips to ensure a smoother experience for test-taking students. These students also have access to a full-length practice test after they create their Acuity Suite account and reserve their Casper test date.

Interested in adding Casper to your admissions process? Book a demo and find out more