Jerusalem, 25 October, 2026 (TPS-IL) — Artificial intelligence models may recommend unnecessary medical tests even when patients are unlikely to benefit from them, with the likelihood varying sharply depending on which model is used, an Israeli study has found. The differences were greatest in cases where doctors themselves disagreed over the appropriate course of action.
The study, conducted by Clalit Health Services and Ben-Gurion University of the Negev, evaluated six artificial intelligence models to determine whether they could recognize when medical intervention was unwarranted. The researchers found that the models recommended unnecessary tests or treatments in 5.4% of cases where intervention was not warranted, while incorrectly advising against necessary care in 0.6% of cases where intervention was required.
The findings highlight a potential weakness as healthcare providers explore using artificial intelligence to help guide medical decisions: models may perform well overall but give sharply different recommendations when medical evidence or professional guidelines are inconclusive.
Unnecessary tests can uncover harmless abnormalities that lead to further examinations, referrals, and sometimes invasive procedures or surgery. Such follow-up care can cause anxiety, consume healthcare resources, and expose patients to avoidable risks. At the same time, excessive caution about testing can be harmful if it leads to necessary care being withheld.
“Patients feel that another test instills confidence, but an unnecessary MRI of the back sometimes reveals a herniated disc or erosion that also exists in healthy people without any pain, and from there the road to anxiety, additional tests, and even unnecessary treatments can be short,” said Dr. Dean Ariel, a family physician at Clalit Health Services and a medical artificial intelligence researcher who led the study.
“The language models learned medicine from our texts, including from our less good habits, and we wanted to check whether they also absorbed the ability to hold back when necessary,” Ariel added. “In most cases, they know how to stop, and they stumble mainly where we, the doctors, hesitate.”
The researchers developed medical scenarios based on recommendations from the Choosing Wisely initiative, an international campaign launched in 2012 to discourage medical tests and treatments that offer little or no benefit. They tested six language models developed by Google, Anthropic, and OpenAI and analyzed thousands of responses.
The scenarios contrasted situations in which the same test could be unnecessary or essential. A young man experiencing back pain after moving furniture, for example, generally does not need immediate imaging. By contrast, a woman with a history of breast cancer who develops new back pain that worsens at night may require imaging to investigate a potentially serious problem.
When Doctors Disagree
The models struggled most with two types of recommendations. They suggested unnecessary thyroid ultrasounds for small nodules in 42.8% of responses and routine electrocardiograms, which record the heart’s electrical activity, in 28.3% of responses involving low-risk patients.
These findings highlight the difficulty of determining when a test is justified. Medical organizations differ in their recommendations for evaluating some small thyroid nodules. Similarly, the need for an electrocardiogram depends on a patient’s symptoms and clinical circumstances, even when overall risk is low.
The differences between models were particularly striking in scenarios where medical professionals disagreed about the appropriate course of action. One model recommended an unnecessary test in fewer than 10% of responses, while another did so in 80% of responses to the same scenarios. The findings suggest that the choice of artificial intelligence model could substantially affect the advice patients receive in medically uncertain situations.
“We hear a language model’s answer as a single authoritative voice, as if there is an ‘artificial intelligence position,’” said Dr. Guy Lavan, a senior academic researcher in Ben-Gurion University’s Department of Industrial Engineering and Management.
Lavan also raised concerns that language models might change their recommendations when users insist on receiving a test, potentially undermining their ability to provide consistent medical guidance.
The researchers said artificial intelligence systems should be evaluated not only on their ability to identify necessary care but also on their ability to recognize when testing or treatment is unwarranted. However, because the study examined responses to constructed medical scenarios, it does not establish how the models would perform when advising actual patients.
The study was published in the peer-reviewed journal Annals of Internal Medicine.
