Comparing open and fixed-response situational judgment tests
August 1, 2023
As professionalism gains importance in admissions and in the curriculum of many service-oriented programs such as health sciences and teacher’s education, tools like situational judgment tests (SJTs) are being adopted to assess this important competency. There are many SJTs from which programs can choose, but it’s important to understand the differences between SJT formats and when each format is best used.
There are two main formats; open response and fixed response. In an open-response SJT, test takers are asked to explain how they’d respond to a dilemma in their own words. These responses can be typed, video recorded, or audio recorded. With a fixed-response SJT, test takers simply select an action that they’d take from a set list in response to a dilemma.
Does response format affect what the SJTs measure?
Yes, it does. For a fixed-response SJT, a group of people have already predetermined what the response options are, and the degree to which they are right or wrong, and therefore tap people’s knowledge of “standard responses” and societal norms. An open-response SJT allows for the possibility that there is not a single right or wrong answer. This means it measures ability instead of knowledge, since all true situational dilemmas have a variety of solutions.
What are the pros and cons of fixed-response SJTs?
| Pros | Cons |
| Can provide formative data to inform instruction and supports for in-program traineesGood for low-stakes situations, such as formative assessmentTests knowledge of “standard responses” and societal norms | More prone to gaming and inauthentic responses in higher-stakes settings, such as admissionsMore commonly used to remove bottom performers, as many test takers’ results group near the top of the rangeFixed responses may be prone to implicit bias and not relate to individuals’ diverse experiences and perspectives |
What are the pros and cons of open-response SJTs?
| Pros | Cons |
| Good for high-stakes situations, such as admissionsTest takers “construct” a response using higher-order skills (e.g., analysis, evaluation) Test takers can provide their own responses based on their lived experience and personal values, and have the opportunity to justify their responsesTests ability relative to the domain and constructsFormat provides raters with more information to evaluate test-takers’ responsesLess susceptible to gaming as there is no clear “right” or “wrong” answerMay be better at predicting excellence at intrinsic roles | Can’t provide subscores because open responses allow for freer movement between constructsOpen-ended format requires a different approach to rating to ensure fairness in scoresMay not be well suited for guiding learning development |
When should each test type be used?
Fixed-response SJTs have been shown to be most effective for formative assessment purposes, which are used to guide learning development. This is because formative assessments are essentially procedural knowledge tests, which are used to measure whether a student has an appropriate level of relevant declarative knowledge. Performance on this type of assessment could help programs identify students who need more support and guidance.
Open-response SJTs like Casper work well as summative assessments, which are typically used to evaluate learners at the end of an instructional unit (e.g., prior to entering an undergraduate or graduate medical program) by comparing it against a specific standard of benchmark. For instance, Casper scores reflect the individual’s performance relative to the rest of the applicants who took the same test. Since the open-ended format focuses on ability rather than knowledge, it can support high-stakes decisions like those made during the admissions process.
What makes Casper ideal for admissions?
Casper is uniquely designed to assess social intelligence and professionalism. Its open-ended format allows a person to demonstrate their ability since all true situational dilemmas have a variety of solutions.
Casper avoids gaming of responses. Every test instance presents applicants with unique scenario/question set combinations, making it impossible to prepare and memorize answers in advance. Moreover, Casper’s open-ended responses mean there is no “correct” answer and time limits to respond are short to make it impossible to covertly consult others prior to responding. The tests are also virtually proctored.
Casper is also administered remotely, making it accessible and an important tool in making your admissions more equitable. This provides a huge benefit to applicants who would be unable to travel to remote test locations (as required by other admissions tools).
Finally, Casper is based on scientific research that spans 15+ years. No other SJT has been used extensively in admissions, and Casper has systematically built strong and well-documented reliability and validity evidence.