A study published in Nature Medicine evaluated the effectiveness of AI Consult, an artificial intelligence tool based on OpenAI’s GPT-4o, in 16 primary care clinics in Nairobi, Kenya. The randomized trial tracked nearly 10,000 patient encounters, with half of the clinical officers using the AI tool to review their electronic notes for potential gaps or errors in real-time. A panel of six Kenyan family physicians reviewed the records and concluded that the clinicians using the AI tool produced higher-quality diagnoses and treatment plans compared to the control group.
The study did not find a statistically significant improvement in patient health outcomes. While there was a 23% decrease in treatment failures, such as unresolved symptoms or death, Dr. Bilal Mateen, a study co-author and chief AI officer at PATH, stated the figure was not statistically significant due to the low baseline frequency of such events. Mateen estimated that a trial involving 139,000 people would be necessary to verify a meaningful difference in clinical outcomes within a primary care setting.
Medical professionals involved in the study reported that the tool, which cost approximately 4 cents per patient, served as a "second pair of eyes" to verify protocols and guidelines. Vyonne Njeri, a clinical officer in Nairobi, noted the tool helped identify an elevated heart rate in an infant that led to a specialist referral. However, some researchers expressed caution regarding the technology. Dr. Nicholas Okumu, an orthopedic surgeon at Kenyatta National Hospital not involved in the study, noted that AI systems could potentially provide incorrect advice, and emphasized the need for active oversight to prevent medical errors.