How situational judgment tests can benefit your admission process

August 18, 2019

We want the best doctors in hospitals, the best teachers in schools, and the best engineers to build our bridges. Most resources have been devoted to ensuring that their respective institutions utilize best practices when making hiring decisions to make sure that people who are unfit for the job are being filtered out. But no single selection process is perfect, so having multiple filters is the best way to make sure that these people tick off all the boxes on a checklist before being placed in important positions.

We often forget that typically the first primary screening process that helps weed out unfit candidates starts from our education system. People must make sure that they possess the appropriate competencies to be admitted into programs that eventually give them the designation to practice their profession. That’s why educational institutions are continually trying to improve their selection methods – to ensure that they are selecting the best students, who in turn will become the top leaders in their respective fields.

Members of admissions committees are under enormous pressure to select the best students and much of the focus has been on ensuring that students have a strong cognitive abilities prior to entry. Most professional schools use a combination of GPA and performance on some standardized test (MCAT, LSAT, PCAT, GMAT, GRE) to guide their selection decisions. While these indicators are excellent in predicting academic performance in professional school, they are less useful in predicting actual performance on the job.

This is because job performance tends to be more multidimensional than academic achievement. Other factors beyond cognitive abilities become more prominent in affecting job performance, like personal competencies and interpersonal skills. For instance, although doctors need to have a good foundation of medical knowledge to provide high-quality service, 94 percent of the reasons why physicians are disciplined were actually due to lapses in professionalism, and not due to issues in medical knowledge or skills. It is estimated that this surmounts to over $50 billion dollar problem to our society! Therefore, academic institutions should try to incorporate additional tools beyond cognitive abilities in guiding their admission decisions.

Both businesses and educational institutions have developed and tested a number of different selection methods to assess the more “non-cognitive” aspects of candidates. However, with numerous studies showing the poor reliability and validity of widely used assessment tools like interviews, personal statements , and letters of recommendation, a huge gap needed to be filled.

After years of research and development, situational judgment tests (SJTs) or inventories (SJIs) have risen as one of the top contenders to fill this gap. SJTs are a type of psychological tests that first present test-takers with a series of realistic, hypothetical scenarios and respondents are then required to either a) identify the most optimal course of action in a closed-response format, or b) describe how they themselves would respond in the situation in an open-ended response format. SJTs were first implemented in the selection of US military personnel back in World War II and since then have extended to a variety of different employment settings and educational institutions. SJTs have been shown to be both reliable and useful in predicting for both academic and job performance, and their relative ease of administration and low-cost have made SJTs an attractive option among admission committees. It is important to note though that not all SJTs are equally reliable and valid, and subtle differences in how the SJT is administered can impact its psychometric properties (more on this later…).

Casper is an SJT that was constructed by a team of academic researchers and industry professionals to evaluate the personal and professional characteristics of incoming medical school applicants. Over time, the test has been extended to assist in the selection decisions of a broader range of professional programs, such as nursing, pharmacy, dentistry, optometry, veterinary, and, teacher education. Casper has demonstrated both high reliability and utility in predicting future performance outcomes such as national licensure scores among medical trainees and applicant ranking of surgery residents. The research team over at Acuity Insights and our academic collaborators from all around the world are continuously working on improving our assessment tool to demonstrate the potential power of Casper in identifying the candidates with the highest potential for success. After all, we should all be motivated to ensure that the right individuals are put in the right jobs.

By: Christopher Zou, Ph.D., Education Researcher

Updated in 2022 by Leanne Roberts, Content Writer

Photo by Fizkes on Adobe Stock