ISRAELI NEUROLOGISTS GAVE LEADING AI chatbots the cognitive exam used to assess U.S presidents fitness, a study intended as a wisecrack-but it found "real flaws" in technology increasingly used to guide clinical decision-making, study coauthor Dr. Roy Dayan told Newsweek.

Dayan, a senior neurologist at Jerusalem's Hadassah Medical Center, said he and his colleagues were inspired by countless studies about Al's outperformance of doctors. Over the past two years, research has shown that ChatGPT can ace the MCAT and the United States Medical Licensing Exam. Large language models, or LLMs, can produce more accurate diagnoses than physicians in certain specialties and even garner higher patient satisfaction scores when responding to digital inquiries.

International medical journals and major media organizations have deliberated whether artificial intelligence will eventually replace doctors. It's not an unreasonable question: 10 percent of consumers believe it should in the foreseeable future, according to a June 2024 survey from IT services and consulting firm CustomerTimes. If AI is going to take charge, it should be put through its paces, Dayan reasoned: "We thought it would be interesting to examine ChatGPT with our tools, the way we check patients if we suspect cognitive degeneration."

Does AI Have Dementia?

Dayan and his colleagues-Dr. Benjamin Uliel, senior neurologist and cognitive specialist at Hadassah Medical Center, and Gal Koplewitz, senior data scientist at Tel Aviv University and London-based QuantumBlack Analytics-administered the Montreal Cognitive Assessment, or MOCA, to five leading LLMs (ChatGPT-4, GPT-40, Claude, Gemini 1 and Gemini 1.5). This assesses cognitive impairment by giving patients a variety of simple tasks.

For example, copy this drawing of a cube. Give as many words as you can that begin with the letter "F." Subtract seven from 100 until you reach zero.