Are situational judgment tests like Casper vulnerable to coaching effects?
May 28, 2021
Situational judgment tests (SJTs) like Casper are gaining popularity in educational and employment settings, as their high reliability, strong predictive validity, and relatively low cost of administration have made SJTs an attractive tool to assist in selection decisions. Their popularity has also given rise to a growing number of test preparation businesses that attempt to coach people on how to respond to SJTs more effectively.
Casper is no exception. A handful of test preparation companies offer Casper coaching courses in an attempt to improve applicant scores. If their coaching programs are actually successful in improving applicant scores, it would cause an unfair advantage towards those students who do not have access to these programs. As these preparation classes typically cost money, these coaching effects can be particularly detrimental for low-income applicants who are already disadvantaged in the medical selection process.
Research shows coaching effects are small for SJTs
The general literature suggests that coaching effects remain fairly small, with an increase in only 0.5 standard deviation after coaching and an increase in 0.32 standard deviation for students who retook the SJT in the following admission cycle. However, not all SJTs are equally susceptible to coaching effects. When SJTs are more complex and difficult, coaching does not seem to improve subsequent performance. This suggests that tests are only coachable when test-taking strategies are simple and straight-forward, but are ineffective when the strategies are more challenging.
Test format matters
A majority of SJTs use a closed-response format, where a series of pre-constructed answers are presented to respondents, but Casper uses a constructed or open-response format where respondents come up with their own answers to the hypothetical scenarios. Compared to the closed-response format, the constructed-response format tends to be more complex and challenging, requiring a deeper level of processing by applicants, which makes Casper a much harder test to coach than other SJTs.
What our own data tell us
Preparation Method | Estimate | Std. Error | t | p |
---|---|---|---|---|
(Intercept) | -.34 | .02 | -20.87 | <.001 |
Reviewed resources on TakeAltus (e.g., FAQs) | .18 | .02 | 9.93 | <.001 |
Completed the three-section practice test on TakeAltus | .12 | .02 | 7.46 | <.001 |
Completed the full-length practice test in Acuity Suite account | .17 | .02 | 10.64 | <.001 |
Participated in a third-party Casper test preparation course | -.10 | .04 | -2.65 | .007 |
Studied potential Casper questions based on competencies | .13 | .02 | 7.43 | <.001 |
Rehearsed responses with technology | .17 | .02 | 9.20 | <.001 |
Rehearsed responses without technology | -.02 | .02 | -0.97 | .333 |
Results from regression analysis of Casper scores on test preparation methods (N=16642)
Adjusted R2 = .04, F(7, 16634) = 106.5, p <.001
We invited Casper test-takers to indicate whether they used any of the following preparation strategies:
- read the tips for applicants on the test website
- completed the free practice test
- participated in a commercial test preparation course
- studied potential questions based on assessment competencies
- rehearsed responses with technology
- rehearsed responses without technology
We then conducted a multiple regression analysis to compare the additive effect of each preparation method on SJT scores reported by our test takers.
The results showed that of the six preparation strategies, only completing the free practice test on the test website, studying potential questions based on the assessment competencies, and rehearsing responses with technology provided significant additive benefit to test scores. The third-party test preparation method only accounted for 2% of the overall variance in test scores.
What does this mean?
Coaching effects are extremely small. These results highlight the importance of ensuring equitable access to practice tests, and relieve concerns over potential socially regressive impacts of commercial test preparation.
While there doesn’t seem to be very much that students can do to significantly improve their Casper scores, they can implement a number of strategies to calm their test anxiety and familiarize themselves with the test. We provide a few sample questions and other prep tips to ensure a smoother experience for test-taking students. These students also have access to a full-length practice test after they create their Acuity Suite account and reserve their Casper test date.
See how our partners have benefited from using Casper: