Back to Blog
The goal of standardized assessment (e.g., MCAT, GPA, SAT, ACT) is to provide reliable and valid measures of applicant performance that are comparable across locations, such as the entire country or an individual region (AERA, APA, NCME, 2014; Zwick, 2018).
These types of assessments can offset the localized biases and limitations of other measures such as
- Grade Point Average (GPA), which is subject to regionality, grade inflation (see Hurwitz and Lee, 2018; see website gradeinflation.com), or
- measures like personal statements and reference letters, which are resource intensive to administer, show low reliability and are generally weak predictors of future performance (White et al., 2012; Rajesh et al., 2019).
However, standardized assessments generally suffer in a few critical areas. They can lead to an overemphasis on teaching to the test and rote memorization over critical reasoning and communication abilities.
Score differences across demographic groups
Perhaps one of the most important criticisms of standardized tests is that there are consistent and substantial score differences between test-takers across demographic groups. Research conducted by Camara and Schmidt (1999) illustrates a fairly consistent pattern across several prominent standardized tests, with the largest gaps between White and African-American test-takers, followed by Hispanic test-takers. More than 20 years later, these differences still persist (e.g., Newman et al., 2022).
Opportunities to mitigate bias
Given the opportunity gap above, it is important to ensure that standardized tests are used appropriately in selection and admissions processes so the biases in these tests don’t close the doors for deserving candidates across all demographic groups.
We see many opportunities to address this challenge:
- Standardized assessment developers need to address the biases in their test design and support more free expression, reasoning, and communication abilities. We have been working for years on the design and implementation of the Casper Situational Judgment Test to continually find ways to mitigate systemic biases and promote equity in assessments. Casper employs an open-response format and leverages higher fidelity question-and-answer formats that are shown to reduce bias. Also, following evidence that video-response formats could further reduce bias, we invested in research and pilots, leading to the inclusion of an audio/video component in Casper.
- Education leaders need to support their admissions committees to understand and use standardized tests appropriately (Lucey & Saguil, 2020).
Improvements across recruitment and admissions
There are also opportunities to reduce bias within the applicant recruitment and evaluation process. This includes the recognition by admissions committees that standardized tests are a single data point and that stronger power and prediction come from using a holistic review process to understand the whole applicant and how their experience and opportunities led to their performance and presentation. This allows standardized tests of knowledge, which have the largest degree of bias, to be balanced with reliable and defensible assessments of other aspects of the applicant which demonstrate less bias.
At each step of admissions, there are appropriate improvements that could be made, including:
- At recruitment, programs can engage in localized outreach development with other schools. These initiatives can build a wider base of understanding and access, improve the transparency and visibility of the admissions process for less familiar applicants, and increase the application rates of the region overall.
- At the early stages of selection, where standardized assessments are designed to provide the most value, admissions officers can employ multiple methods such as using rubrics or formulas across assessments to avoid the bias of placing undue emphasis on exact scores (e.g. cut-scores), giving weight to assessments reflective of your program values, and understanding the relative impact of each measure concerning demographic biases and considering this in your weighting (Juster et al 2019). This can also include a standardized evaluation of an individual’s opportunities (e.g., Jarnet et al 2015 & 2018).
- During file review and interviews, several strategies can be used including consistent bias training for all evaluators, thoughtful and inclusive question and rubric development (e.g. MMI, multiple assessors from the school and the community), and a continuous analysis and modification process between admissions cycles, all to help ensure that your selection process is minimizing bias.
- Learning from other programs, about what works and what doesn’t is very important. However, make sure you are looking at the results through the lens of your program, your resources, your mission, and your goals. Application in another program may look different than in your program, and that is okay, but make sure you pay attention to the intended (and unintended) outcomes, both positive and negative.
- Finally, wherever possible, engage communities across your recruitment and selection process that align with your mission. Find opportunities to bring in partners, current students and learners, community members, or advocacy groups who may bring different and important perspectives to the table in the development and execution of your process.
In conclusion, standardized tests provide a view into your applicant’s knowledge and cognitive abilities, but they come with built-in bias which must be taken into account. Their use should be balanced with other measures to provide a holistic view of your applicants. Here are some additional articles you may find interesting:
