The Metric Behind Microsoft’s Claim About AI And Doctors

🔍 Read the full analysis: The Metric Behind Microsoft’s Claim About AI And Doctors on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on monitors, keyboards and dev gear

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO diagnostic AI scored 83.2% on 304 difficult medical case records, compared with 36.1% for generalist physicians using textbooks and the internet, in a peer-reviewed study published in Science on Oct. 22. The results are Microsoft’s own evaluation of its system in a text-based simulator, not evidence that the AI diagnoses real patients more accurately in hospitals.

The paper, titled Sequential diagnosis with language models, also reported results from 325 emergency department cases: MAI-DxO scored 80.3%, while physicians scored 46.8%. Microsoft described the complex-case result as more than twice the doctors’ score without reference materials; those physicians scored 22.3%. The comparisons were between the AI system and individual primary care doctors or hospital generalists.

In the benchmark, a case begins with a short summary. The AI or physician must request further information, such as a test result, from a simulated gatekeeper holding the full record. The system is assessed on its final diagnosis and the estimated cost of the tests it requests. MAI-DxO coordinates a group of virtual physician agents that suggest diagnoses, choose follow-up questions or tests, and evaluate one another’s reasoning.

The paper’s reported performance does not come solely from that orchestration method. Microsoft’s chart lists GPT-5 alone at 78.9% on the difficult cases, compared with MAI-DxO’s 83.2%. The source account says GPT-5’s estimated testing cost was $8,409 per case, while the orchestration layer added about four percentage points and reduced estimated costs. Those cost figures are modeled estimates, not bills from actual patient care.

What the Scores Mean for Care

The results are relevant because diagnosis often involves deciding which question to ask and which test to order, not simply selecting an answer after all information is provided. The benchmark attempts to measure that sequence, as well as the final diagnosis. It offers a way to test how a language-model system gathers information under modeled costs.

But the gap in scores should not be read as proof that MAI-DxO outperforms doctors in clinics. The study used historical, text-only cases, with no examination of a patient and no real-world imaging or care decisions. The comparator was individual generalists, while challenging cases in clinical practice may be handled by specialists or teams. The results therefore describe performance in a particular test design, not the likely effect on patient outcomes, safety, or hospital spending.

That distinction matters to patients, clinicians, hospitals, and insurers weighing medical AI. A high score on selected cases may justify further testing, but it does not by itself establish that a system can be safely integrated into care, identify when it is wrong, or improve outcomes compared with existing clinical teams.

How Microsoft Built the Benchmark

The study appeared in Science, volume 394, pages 423–428, on Oct. 22, 2026. Its 15 listed authors were affiliated with Microsoft, including Microsoft AI chief executive Mustafa Suleyman. The paper followed a June 2025 arXiv preprint, which had not been peer-reviewed and reported an earlier system and comparison. Microsoft’s blog post highlighted ratios between the system and physicians; the journal publication supplies the updated figures cited here.

According to the source account’s description of the benchmark and earlier preprint, a model judges answers against the published diagnosis. When a requested result is absent from a case record, the gatekeeper generates a synthetic result. Test costs are approximations based on billing codes and a 2023 U.S. hospital price list, with a fixed $300 physician-visit charge. The source notes that the final Science paper is behind a paywall, so some methodological details drawn from the preprint may differ from the published version.

The doctors were generalists rather than specialists. The earlier preprint described 21 physicians from the United States and United Kingdom, and said physicians were evaluated on a subset of held-out cases while the AI was assessed on all 304. The Science results add an open-book physician comparison using textbooks and internet access. Differences in case selection and evaluation mean the figures need to be read with the study design in view.

“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”

— Microsoft AI chief executive Mustafa Suleyman, speaking to TechCrunch

Limits Beyond the Simulator

The study does not establish how MAI-DxO performs with real patients, physical examinations, imaging, or routine clinical workflows. The source account says the case set contained no healthy patients and no harmless explanations, limiting what the test can show about false alarms. It is also unclear how the system would perform against specialist physicians or multidisciplinary teams, rather than generalists working alone.

The diagnostic scores rely on a model-based grading process, and the source account reports that physician re-grading of a sample showed good but imperfect agreement. Cost estimates depend on assumptions and price data rather than actual hospital bills. Peer review evaluates a paper’s methods and reporting; it does not mean independent groups have reproduced Microsoft’s results. The source material does not provide evidence of such external replication.

It remains unknown whether the reported advantage will persist in prospective care, whether it will improve patient outcomes, and how often the system might recommend unnecessary tests or miss a diagnosis. Those questions cannot be answered by the benchmark’s accuracy figures alone.

A Patient Study Is Planned

The next reported step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source account, consenting patients will interact with MAI-DxO alongside their normal care. That setting could provide evidence about how the system behaves during real clinical encounters, rather than on historical records alone.

Details about the study’s timing, size, outcome measures, and safeguards were not specified in the source material. Until results are reported, the Science paper should be understood as a benchmark evaluation. Further evidence will be needed to assess whether the system can support clinicians safely and whether its diagnostic performance translates into better care.

Key Questions

What did Microsoft’s MAI-DxO study report?

In a Science paper published Oct. 22, 2026, MAI-DxO scored 83.2% on 304 difficult NEJM case records. On 325 emergency department cases, it scored 80.3%. These were results from a simulated benchmark, not a trial of care delivered to patients.

Did the AI outperform all doctors?

The reported comparison was with individual generalist physicians, including primary care doctors and hospital generalists. The study did not compare MAI-DxO with specialist teams treating patients, so it does not show that the AI outperforms all doctors or clinical teams.

What does the 83.2% figure measure?

It is MAI-DxO’s diagnostic accuracy on 304 selected complex case records in the study benchmark. Participants had to request additional information from a simulated gatekeeper. The figure is not a measure of patient outcomes or performance across all medical cases.

Has MAI-DxO been shown to improve patient care?

Not in the evidence described here. The published evaluation used historical text cases. A prospective study involving consenting patients alongside normal care is planned at Beth Israel Deaconess Medical Center, but results were not available in the source material.

Source: ThorstenMeyerAI.com