Pros and Cons of Standardized Assessments and How Best to Use Them in the Admissions Process

April 29, 2023

The goal of standardized assessment (e.g., MCAT, GPA, SAT, ACT) is to provide reliable and valid measures of applicant performance that are comparable across locations, such as the entire country or an individual region (AERA, APA, NCME, 2014; Zwick, 2018).

These types of assessments can offset limitations of other measures such as

• Grade Point Average (GPA), which is subject to regionality, grade inflation (see Hurwitz and Lee, 2018; see website gradeinflation.com), or
• measures like personal statements and reference letters, which are resource intensive to administer, show low reliability and are generally weak predictors of future performance (White et al., 2012; Rajesh et al., 2019).

However, standardized assessments generally suffer in a few critical areas. They can lead to an overemphasis on teaching to the test and rote memorization over critical reasoning and communication abilities.

Score differences across demographic groups

Perhaps one of the most important criticisms of standardized tests is that there are consistent and substantial score differences between test-takers across demographic groups. Research conducted by Camara and Schmidt (1999) illustrates a fairly consistent pattern of group differences across several prominent standardized tests. More than 20 years later, these differences still persist (e.g., Newman et al., 2022).

Opportunities to promote fairness

Given the opportunity gap above, it is important to ensure that standardized tests are used appropriately in selection and admissions processes so these tests don’t close the doors for deserving candidates across all demographic groups.

We see many opportunities to address this challenge:

• Standardized assessment developers need to address the fairness of their test design and support more free expression, reasoning, and communication abilities. We have been working for years on the design and implementation of the  Casper Situational Judgment Test to continually find ways to promote fairness in assessments. Casper employs an open-response format and leverages higher fidelity question-and-answer formats that are shown to be fairer. Also, following evidence that video-response formats could further enhance fairness, we invested in research and pilots, leading to the inclusion of a video-response section in Casper.
• Education leaders need to support their admissions committees to understand and use standardized tests appropriately (Lucey & Saguil, 2020).

Improvements across recruitment and admissions

There are also opportunities to enhance fairness within the applicant recruitment and evaluation process. This includes the recognition by admissions committees that standardized tests are a single data point and that stronger power and prediction come from using a holistic review process to understand the whole applicant and how their experience and opportunities led to their performance and presentation. This allows standardized tests of knowledge, which are most likely to disadvantage certain groups, to be balanced with reliable and defensible assessments of other aspects of the applicant.

At each step of admissions, there are appropriate improvements that could be made, including:

• At recruitment, programs can engage in localized outreach development with other schools. These initiatives can build a wider base of understanding and access, improve the transparency and visibility of the admissions process for less familiar applicants, and increase the application rates of the region overall.
• At the early stages of selection, where standardized assessments are designed to provide the most value, admissions officers can employ multiple methods such as using rubrics or formulas across assessments to avoid placing undue emphasis on exact scores (e.g. cut-scores), giving weight to assessments reflective of your program values. Understanding the relative impact of each measure and considering this in your weighting (Juster et al 2019) can improve outcomes. This can also include a standardized evaluation of an individual’s opportunities (e.g., Jarnet et al 2015 & 2018).
• During file review and interviews, several strategies can be used including consistent attitude training for all evaluators, thoughtful and rigorous question and rubric development (e.g. MMI, multiple assessors from the school and the community), and a continuous analysis and modification process between admissions cycles, all to help ensure the fairness of your selection process.
• Learning from other programs about what works and what doesn’t is very important. However, make sure you are looking at the results through the lens of your program, your resources, your mission, and your goals. Another program’s process may look different than yours, and that is okay, but make sure you pay attention to the intended (and unintended) outcomes, both positive and negative.
• Finally, wherever possible, engage communities across your recruitment and selection process that align with your mission. Find opportunities to bring in partners, current students, and community members who may bring different and important perspectives to the table in the development and execution of your process.

In conclusion, standardized tests provide a view into your applicant’s knowledge and cognitive abilities, but they come with limitations which must be taken into account. Their use should be balanced with other measures to provide a holistic view of your applicants.