Comparing open and fixed-response situational judgment tests

October 13, 2021

As professionalism gains importance in admissions and in the curriculum of many service-oriented programs such as health sciences and teacher’s education, tools like situational judgment tests (SJTs) are being adopted to assess this important competency. There are many SJTs from which programs can choose, but it’s important to understand the differences between SJT formats and when each format is best used.

There are two main formats; open response and fixed response. In an open-response SJT, test takers are asked to explain how they’d respond to a dilemma in their own words. These responses can be typed, video recorded, or audio recorded. With a fixed-response SJT, test takers simply select an action that they’d take from a set list in response to a dilemma.

Does response format affect what the SJTs measure?

Yes, it does. For a fixed-response SJT, a group of people have already predetermined what the response options are, and the degree to which they are right or wrong, and therefore tap people’s knowledge of “standard responses” and societal norms. An open-response SJT allows for the possibility that there is not a single right or wrong answer. This means it measures ability instead of knowledge, since all true situational dilemmas have a variety of solutions.

What are the pros and cons of fixed-response SJTs?

PROS	CONS
– Can provide formative data to inform instruction and supports for in-program trainees – Good for low-stakes situations, such as formative assessment – Tests knowledge of “standard responses” and societal norms	– More prone to gaming and inauthentic responses in higher-stakes settings, such as admissions – More commonly used to remove bottom performers, as many test takers’ results group near the top of the range – Fixed responses may not relate to individuals’ unique experiences and perspectives

What are the pros and cons of open-response SJTs?

PROS	CONS
– Good for high-stakes situations, such as admissions – Test takers “construct” a response using higher-order skills (e.g., analysis, evaluation) – Test takers can provide their own responses based on their lived experience and personal values, and have the opportunity to justify their responses – Tests ability relative to the domain and constructs – Format provides raters with more information to evaluate test-takers’ responses – Less susceptible to gaming as there is no clear “right” or “wrong” answer – May be better at predicting excellence at intrinsic roles	– Can’t provide sub-scores because open responses allow for freer movement between constructs – Open-ended format requires a different approach to rating to ensure fairness in scores – May not be well suited for guiding learning development

PROS

CONS

– Good for high-stakes situations, such as admissions

– Test takers “construct” a response using higher-order skills (e.g., analysis, evaluation)

– Test takers can provide their own responses based on their lived experience and personal values, and have the opportunity to justify their responses

– Tests ability relative to the domain and constructs

– Format provides raters with more information to evaluate test-takers’ responses

– Less susceptible to gaming as there is no clear “right” or “wrong” answer

– May be better at predicting excellence at intrinsic roles

– Can’t provide sub-scores because open responses allow for freer movement between constructs

– Open-ended format requires a different approach to rating to ensure fairness in scores

– May not be well suited for guiding learning development

When should each test type be used?

Fixed-response SJTs have been shown to be most effective for formative assessment purposes, which are used to guide learning development. This is because formative assessments are essentially procedural knowledge tests, which are used to measure whether a student has an appropriate level of relevant declarative knowledge. Performance on this type of assessment could help programs identify students who need more support and guidance.

Open-response SJTs like Casper work well as summative assessments, which are typically administered at a single point in time. This can happen at the end of an instructional unit (e.g., after graduating and prior to entering an undergraduate or graduate medical program), with the results being compared against a specific standard or benchmark. For instance, Casper scores reflect the individual’s performance relative to the rest of the applicants who took the same test. Since the open-ended format focuses on ability rather than knowledge, it can support high-stakes decisions like those made during the admissions process.

What makes Casper ideal for admissions?

Casper is uniquely designed to assess social intelligence and professionalism. Its open-ended format allows a person to demonstrate their ability since all true situational dilemmas have a variety of solutions.

Casper avoids gaming of responses. Every test instance presents applicants with unique scenario/question set combinations, making it impossible to prepare and memorize answers in advance. Moreover, Casper’s open-ended responses mean there is no “correct” answer, and time limits to respond are short to make it impossible to covertly consult others prior to responding. The tests are also virtually proctored.

Casper is also administered remotely, making it accessible and an important tool in making your admissions fairer. This provides a huge benefit to applicants who would be unable to travel to remote test locations (as required by other admissions tools).

Finally, Casper is based on scientific research that spans 15+ years. No other SJT has been used extensively in admissions, and Casper has systematically built strong and well-documented reliability and validity evidence.

Interested in learning more about our assessments for admissions? Book a call and one of our experts will be in touch!