How value alignment assesses “fit” more objectively

June 17, 2022

“Fit”, often referred to as “culture fit“, is the applicant’s alignment with a program’s culture in terms of their motivation, attitude, and values, not the individual decision-maker’s values. However, true “fit” should be much broader, including the consideration of an applicant’s values and needs, and a program’s mission, values, and offerings.

Why is this important? You could align with the program’s culture but could misalign with the other values and features of the program, implying you may not be the optimal applicant for that specific program. Here at Acuity, Duet supports mission-based holistic review of applicants and leverages forced-choice comparisons to produce a ranked list of priorities and values that captures all of these essential considerations. 

The birth of Duet

Acuity’s mission is to create a world served by exceptional professionals. We do this by designing and developing assessment tools that help programs select applicants based on non-cognitive characteristics. 

Through our research, we have identified programs and applicants could benefit from a value-alignment assessment to highlight the type of environments in which applicants have the potential to thrive and be successful. The purpose of this assessment is to understand applicants’ own values and preferences with respect to programs, which are not collected by other means. 

Duet is our standardized value-alignment assessment that helps programs consider “culture fit” alongside other important categories and characteristics in a more equitable way during the admissions process. Undergraduate Medical Education (UME) and Graduate Medical Education (GME) programs helped refine the categories and characteristics with the Acuity research team.

The categories within Duet

A series of surveys and qualitative analyses confirmed and refined characteristics within three categories: Mission and Culture, Teaching and Learning, and Program Features. The team used a modified version of the Delphi methodology, “a systematic process of forecasting using the collective opinion of panel members”, and a thorough literature review.

Each Duet category consists of seven characteristics. The Acuity research team in collaboration with several UME programs developed the UME characteristics. The team conducted a series of semi-structured interviews with subject matter experts from allopathic and osteopathic medicine programs to derive an initial pool of characteristics that differentiate programs from one another. The team developed GME program categories and characteristics in the same manner; however, the categories differ slightly in that they include Culture, Academics, and Clinical Experience.

How does Duet work

Duet is a paired comparison assessment consisting of forced-choice comparisons. To understand how important each characteristic is, the assessment taker (program stakeholders and applicants) is asked to indicate their preference for one characteristic over another, for all pairs of characteristics within each category. They also indicate their preference of each category over another, yielding a breadth of data programs can use to understand what each applicant desires in a program. Well established methods, incorporating aspects of Thurstone’s (1927) law of Comparative Judgment and the analytic hierarchy process (Saaty, 2008) are the foundations of the paired comparison format of the assessment. This leads to more accurate rankings by deriving an order based on individual comparisons of all possible characteristics. 

Programs create a single Program Profile by inviting various stakeholders such as, faculty, administrators, and students, to complete the assessment. Acuity recommends a minimum of 5 stakeholders, in order to have a reliable program profile. However, more stakeholders are better. As mentioned above, this profile has three categories (with subsequent characteristics) that reflect the mission, values, and priorities of the program type that the assessment targets. 

Once the Program Profile is created, the applicants complete the same assessment to determine which values they prefer in a program. Rest assured, although applicants spend a lot of time completing various assessments and other tasks for their applications, Duet only requires 15 minutes to complete. 

Use Cases

Duet has many use cases, including:

• Helping identify whose application to review first
• Discovering applicants that might be overlooked when using other metrics
• Finding applicants most aligned with program values to invite to interview
• Using the scores to determine who to offer available spots to following interviews

Scoring

To view the Program Profile, the program logs into Acuity Insights. A comparison of this profile with the applicant assessment ranking generates “Duet Alignment Scores.” Programs then receive the applicants’ scores in relation to the alignment with your Program Profile. They appear as “D – [category name],” “D – [Average Score]”, and “D – [Weighted Average].”

Figure 1. Duet UME scores on Acuity Insights

It is important to note, at this time, applicants do not receive their score. 

What do other programs think of Duet?

In a GME pilot, programs advised they liked having a tool to assess “fit” in an objective manner. Dr. Salcedo at the University of Texas Rio Grande Valley expected Duet to inform decisions on who to move up or down the admissions decision ladder. 

Overall, participating programs found Duet helped reveal unique qualities of applicants they wouldn’t have found using traditional tools. 

Summary

In a recent study surrounding virtual interviews during the 2020-21 match cycle, 73.9% of residency program directors (PDs) found “it challenging to gauge an applicant’s genuine interest…using a virtual platform.” Evidence showed 44.3% of PDs found the assessment of an applicant’s “fit” into their program was challenging. 

By using Duet, programs are able to assess “fit” more objectively. Using a robust methodology, such as Delphi, to identify and define values and having a standardized tool to assess value alignment enables programs to overcome challenges associated with evaluating applicants for fit.

[cta cta_id=”15094″]