MAI-DxO vs. Doctors: Inside the Science Paper

Microsoft’s AI diagnostic system, MAI-DxO, reached the right diagnosis in 83.2% of complex New England Journal of Medicine cases in a simulated test, compared with 36.1% for generalist physicians using reference tools, according to a study published in Science on Oct. 22, 2026. The Microsoft AI diagnosis research comes with a Boston follow-up: Microsoft says it will test the system with patients at Beth Israel Deaconess Medical Center.

Image: Beth Israel Deaconess Medical Center in Boston. Photo: Ajay Suresh, via Wikimedia Commons (CC BY 4.0).

What the Microsoft AI diagnosis study tested

Most AI medical tests hand a model a complete case write-up and ask for one answer. A Microsoft team whose authors include first author Harsha Nori, Mustafa Suleyman and Eric Horvitz built something closer to how doctors actually work. Their Sequential Diagnosis Benchmark (SDBench) turns 304 New England Journal of Medicine clinicopathological conference cases, which are drawn from the Case Records of the Massachusetts General Hospital, and 325 emergency-department-to-admission cases from the MIMIC-IV-ED database into step-by-step encounters.

Each case starts with a short vignette. The diagnostician, either a physician or an AI, can ask questions, order tests or commit to a diagnosis. A “gatekeeper” agent releases only the information that question or test would actually produce, and every test adds to an estimated bill based on real hospital prices. A “judge” agent then grades the final answer against the reference diagnosis.

The system being tested, the Microsoft AI Diagnostic Orchestrator (MAI-DxO), works like a virtual panel of physicians. Several AI roles propose diagnoses, pick questions and tests, and challenge the leading theory, weighing each test’s likely value against its cost, according to Microsoft.

The results: accuracy and cost

The paper reports these headline numbers:

Complex NEJM cases: MAI-DxO, running on OpenAI’s GPT-5 with no budget limit, was correct 83.2% of the time at an average estimated test cost of $5,431. With tighter budgets it scored 80.9% at $2,241 and 79.6% at $1,303.
Physicians on the same cases: 22.3% accuracy without outside resources (closed book) and 36.1% with search and reference tools (open book), at average costs of $3,049 and $4,725. The best-performing individual physician reached 51.8%.
Emergency department cases: MAI-DxO scored 80.3% at an average cost of $4,979, versus 46.8% for physicians at $3,400.

The study also found a difference in style. Physicians asked an average of 5.6 questions before ordering their first test on the complex cases, while the AI system asked 25.5, doing far more history-taking before ordering anything.

The fine print: what the study does not show

The authors are explicit about the limits. The test was text-only and idealized, with no medical imaging, and it measured whether the final diagnosis was correct rather than what happened to patients. The physicians in the comparison were generalists, not specialists, so the results do not show that the AI outperforms specialist doctors or real-world clinical teams.

TechCrunch noted that cases this complex are usually handled by specialists or physician teams, who were not part of the comparison. Its review of the data also found that much of the improvement comes from more powerful underlying AI models. On its own, GPT-5 scored 78.9% on the complex cases, and on the emergency cases the orchestrator added little with the newest model, GPT-5.5: both versions scored about 85% at a similar cost, TechCrunch reported. The orchestrator helped most with weaker models; Google’s Gemini 2.5 Flash went from 46.4% alone to nearly 69% with it.

All of the paper’s authors are Microsoft employees, which readers should keep in mind. Microsoft says it is open-sourcing the SDBench benchmark under the MIT license so outside researchers can reproduce and challenge the results.

The Boston test: Beth Israel Deaconess

Microsoft says the next step is a prospective study with Beth Israel Deaconess Medical Center. Patients will receive standard care, including triage, specialist time, treatment and discharge decisions, and those who consent will also interact with the AI system in parallel, Dominic King, vice president of health at Microsoft AI, told TechCrunch. Microsoft and its partners will compare the system’s recommendations with the doctors’ independent assessments.

“This technology is ready for real-world evaluations, and you can imagine the complexity and diversity of issues that come from a Boston emergency department on a Friday evening,” King told TechCrunch. He said the study will cover emergency and urgent care settings.

Beth Israel is becoming a hub for this kind of testing. TechCrunch reported that Google published a clinical study with the hospital in The Lancet this month evaluating its AMIE conversational diagnostic AI in real urgent care visits, with physicians monitoring the conversations in real time.

Microsoft’s bigger push into health AI

Mustafa Suleyman, CEO of Microsoft AI, called the result a step toward “medical superintelligence,” TechCrunch reported. He told the outlet Microsoft is working with the Mayo Clinic on a foundation model for health care planned for release next year, and said, “I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.” TechCrunch noted that Microsoft already sells dictation and radiology tools to clinicians and sees the diagnostic orchestrator as an expansion of that business.

Boston-area research keeps showing up in health and AI news this month. See our coverage of a Harvard Belfer Center report on AI and terrorism risk and a study finding the BCG tuberculosis vaccine cut tau tangles in mice.

Sources

Like this story? Get the newsletter

The day's Boston news, weather and local business in one short email, with links to the full stories.

Latest

All latest news →

Microsoft AI Diagnosis Tool Outscores Generalist Doctors in Science Study