September 25, 2020

When I was working in MedEd, I often wondered how enhancement and feature requests were prioritized at Altus. Having worked closely with IT at my school, I knew that my program’s needs might not be the same as others, and were prioritized accordingly. 

Coming onboard at Altus in product management suddenly expanded my view from the microcosm of my school to the wide and varied needs of our customers in healthcare education, from undergraduate to postgraduate medicine, dentistry, osteopathy, veterinary medicine, and beyond. 

I quickly learned that collaborating directly with MedEd institutions and engaging in product discovery with the right users at the right time was critical. The critical switch in my thinking came from understanding that our role is not to build features but rather to solve problems.

Even with my years in MedEd, and surrounded by people who have worked with schools for 15+ years, we can have an incomplete sense of the scope and relative priority of user problems. Even prioritizing those problems is a challenge, because they’re all critical to success.

We knew the biggest and most pressing challenge was reporting, followed closely by giving competence committees a mechanism by which they could log meeting decisions and outcomes. 

The key to understanding those problems, and delivering features that solve those problems, is product discovery.

Concurrent product discovery and delivery is a dual-track process (two processes occurring at the same time). The product discovery track is the process by which Altus determines what to build and establishes timelines and expectations for new features. The product delivery track is the software development, testing, and deployment process to get the features determined in product discovery onto the live Altus application.

The case for product discovery and CBME

The implementation of competency-based education (CBME) models across undergraduate and postgraduate medicine, as well as osteo, vet, nursing, and other healthcare disciplines, is evolving rapidly. 

In Canada, a move to CBME was mandated by the Royal College of Physicians and Surgeons of Canada. Over 70 programs are shifting to an EPA-based curriculum over many years. The programs that have launched are sorting through enormous amounts of data to prepare for resident reviews, or competence committee meetings.

Having spent two years implementing EPA-based curricula at a Canadian medical school, and attending competence committee meetings to try to gain a better understanding of their needs, I had a good sense of the problem. But I’m not a doctor or a competence committee member, so my understanding was incomplete. 

In order to enable efficient and valid entrustment decisions, and replace data dredging with respectful reporting, we had to talk to the people consuming the data to make those decisions. Most critically, we needed to connect with competence committee chairs and program directors. Program administrators were also a critical piece of this puzzle, as they often prepare data for competence committee meetings and attend them as well.

In our product discovery meetings with Canadian residency programs, we surfaced dozens of problems and opportunities, and prioritized them based on what we learned across programs. These are but a small sample of the enhancements we made to help competence committees make timely and valid entrustment decisions.

One critical problem we heard about over and over was that the assessment plans for each individual required competence committee members to painstakingly count each response to a contextual question, and log it against the assessment plan requirements.

The Altus EPA Achievement Report was able to solve this problem. But we wouldn’t have been able to calculate and display resident progress through each EPA and its varied and complex assessment plans without regular access to competency committee chairs and program directors.

By meeting regularly with the users of these reports, we were able to confirm the logic and calculations required for progress and then translate that into the product.

We also heard that programs wanted to get to the relevant data in fewer clicks. Due to technical constraints, adding more data to our already robust reports would have meant loss of speed and potentially crashing key reports, so we delivered the ability to open and run multiple reports concurrently and view them side by side. 

We learned the importance of evaluator names and their associated scores to provide context to entrustment decisions, so we expanded the information available in our reports, first by adding the score to the narrative feedback reports, then adding additional information when users click on data points in the trajectory report, as well as allowing users to click through to see the evaluation in its entirety if needed. 

Competence committees needed a central way to store meeting outcomes and entrustment decisions that would provide historical context for each additional meeting. The competency review feature allows them to both log meetings and promote learners through stages of training, and maintain that data in a central, easily accessible format.

Over six months, the concurrent product discovery and delivery processes helped us build several critical reports and features that reduced the amount of manual work by up to 75 percent. These cycles of discovery and delivery worked in tandem and helped ensure our work delivered high-value to the schools and programs we serve.   

Our collaboration with Canadian postgraduate residency programs is a hallmark of how we conduct product discovery in real time, addressing real problems, and delivering solutions that solve those problems.