AI in Healthcare: A Perspective

In a spring blog focused on AI, Andrew Rebhan, Director on Sg2’s Intelligence team, offered perspectives on this emerging trend. He sets the scene with this review of the recent past:

"Soon after the public release of ChatGPT in November 2022, researchers started to test this AI and other similar tools on a range of industry benchmarks. By February 2023, a study evaluated ChatGPT’s performance on the three subtests of the US Medical Licensing Exam (USMLE), showing the tool was able to perform at or near the passing threshold of 60% accuracy without any prior specialized training. In the following months, other models like Google’s Med-PaLM 2, specifically trained and fine-tuned for medical tasks, started to show even stronger performance on the USMLE and other measures."

His blog continues, "Despite these breakthroughs, it didn’t take long for a subset of health care stakeholders to dismiss AI’s performance on these exams, suggesting that assessments like the USMLE were now suddenly obsolete – a mere demonstration of rote memorization of facts that didn’t really reflect the complexities of clinical decision-making."

Then, when OpenAI’s GPT-4 was released by mid-March 2023, it provided another significant boost in LLM performance. According to Rebhan, "By November 2023 – just one year after the public release of ChatGPT – researchers from OpenAI and Microsoft demonstrated how GPT-4 could achieve impressive results on all nine of the benchmark datasets in the MultiMedQA suite.

The study showed how a generalist LLM like GPT-4 could outperform health care domain-specific models through focused prompting strategies, achieving a score of over 90% on the MedQA dataset. These industry performance evaluations are not slowing down, with several recent studies comparing AI and clinician performance in board residency exams and deeper dives in specialties like oncology and ophthalmology. OpenAI’s competitors are also keeping busy, with Google recently announcing its latest medical AI models that it claims outperforms GPT-4."

He warns, however, that one should not blindly accept the results of every new AI study. He notes, "A healthy dose of skepticism can help filter out the noise and push stakeholders to think more critically about AI, but it can also result in cyclical exercises of ‘moving the goalposts’ that consistently leads to inaction."

Here are some strategic steps to prepare for generative AI’s role in health care, according to Rebhan:

Get used to fast-paced change: LLMs like GPT-4 are pushing the boundaries of what we thought was achievable with AI, and it’s doing so on a timescale of months. GPT-5 is coming around the corner, and many organizations are simply not ready for it.

View AI through a lens of opportunity: If health care stakeholders seek to dismiss AI’s evolution at every turn, they run the risk of not leveraging this technology to better serve their interests or business goals.

Prepare to take the leap: The gap between early adopters and laggards on the technology adoption curve is going to expand faster than it has in the past. If your organization hasn’t made a serious effort to experiment with AI yet, it’s high time you do so – like yesterday. For more information, go here or connect at [email protected]. OSM