Does ChatGPT Have Any Impact on Casper?

May 27, 2024

The rapid advancements in AI technology, particularly tools like ChatGPT, have raised questions about their potential impact on various assessment methods in higher education. With the growing reliance on these AI tools, concerns have emerged regarding their influence on the integrity and authenticity of admission tests. ChatGPT’s latest model, the GPT-4, has already been shown to pass incredibly challenging graduate-level business, law, and medical assessments.

Unlike traditional tests that rely on academic knowledge and are easier to game, Casper is a unique, open-response, situational judgment test that assesses social intelligence and professionalism and is safeguarded by multiple security measures that prevent cheating and response faking via AI tools.

In order to ensure the robustness of the Casper test, researchers from Saint Mary’s University, with the support of the Acuity Insights research team, independently investigated applicants’ potential misuse of ChatGPT in a two-phase study. Given the open-response nature of the Casper test, there were concerns about AI’s ability to artificially enhance response quality. However, the study reaffirmed the Casper test’s robustness and continued validity.

Phase 1: Unraveling the Myth

In Phase 1, spanning from June 2022 to May 2023, researchers analyzed a large dataset from 107,805 applicants, comprising ~1 million responses. With the rising use of ChatGPT, researchers wanted to investigate whether its wide adoption was having any impact on the validity of Casper. 

Phase 1 results revealed that the number of months since ChatGPT release was negatively associated with Casper performance. This means that we found a negative correlation between ChatGPT release and Casper scores. Despite shifts in response lengths, stringent security measures embedded in the Casper design mitigated the risk of AI-driven manipulation. 

Phase 2: Confirmation Through Experimentation

Our next step was to conduct an experimental study to explore the actual impact of ChatGPT on Casper scores and explore the efficacy of the protection mechanisms enforced to discourage ChatGPT misuse. We conducted a mock Casper test and randomly assigned 138 participants to three separate, simulated conditions:

• Group 1: Participants could not use any outside help, honest answers only.
• Group 2: Participants could use the internet, such as web searches, but could not use ChatGPT.
• Group 3: Participants could use ChatGPT only to help with their Casper responses.

Results indicated that participants in the ChatGPT group achieved merely 0.27 points (on a 9-point scale) higher than those who did not use ChatGPT, which is a statistically immaterial impact. Additionally, participants in the ChatGPT group reported that the restrictions embedded within the Casper test (e.g., time limit, disabled copy and paste functionality, etc.) made it extremely difficult to use ChatGPT within the testing window, further deterring them from using it.

Overall, evidence shows that ChatGPT’s availability has an inconsequential impact on Casper scores. The Casper design and functionalities seem to effectively limit any potential impact ChatGPT may have on test integrity and response authenticity.

When reviewing the impact, a few factors should be considered. 

• The Casper test is unique because it uses different scenarios for each test instance, making it nearly impossible to game. Applicants who have already taken the test cannot advise prospective applicants preparing for the test by providing the scenarios they were faced with, making it challenging to find cheating material online.
• Casper is a timed assessment that requires applicants to use their allotted time efficiently. Using AI during the allotted time could actually hinder an applicant, who could run out of time before completing their response. 
• Applicants who are participating in the Casper test cannot copy and paste answers they’ve typed or generated elsewhere, making it very difficult to use ChatGPT. Likewise, applicants cannot bring phones or other devices to use during their test.
• Casper is a proctored test. Applicants cannot easily cheat as proctors ensure that applicants aren’t using devices, speaking to others off-screen, or exhibiting any cheating behaviors. 

Implications and Future Directions

The culmination of Acuity’s research underscores the resilience of the Casper assessment in the face of technological advancements. As AI continues to evolve, Acuity Insights remains committed to upholding the integrity and fairness of admissions assessments through continuous research and development. Through continued vigilance and innovation, Acuity Insights empowers institutions to make informed decisions while providing applicants with a platform to showcase their diverse talents and capabilities.

About Casper

Casper is an open-response situational judgment test (SJT) that measures social intelligence and professionalism, and is backed by 20 years of data. Research consistently shows that demonstrating effective interpersonal skills contributes significantly to success during academic programs and in future careers.

Casper presents applicants with scenarios and questions that allow them to demonstrate these skills and the extent to which they can respond to challenging scenarios in a professional and socially intelligent manner, by using their diverse life experiences. Its open-response format, with both video and text, ensures it is resistant to gaming by humans and AI, unlike most other SJTs, which use a multiple-choice format.

Find out more about why an ethical AI framework is critical for all institutions.