Sequential Diagnosis with Language Models
- Harsha Nori ,
- M. Daswani ,
- Christopher Kelly ,
- Scott M. Lundberg ,
- Marco Túlio Ribeiro ,
- Marc Wilson ,
- Xiaoxuan Liu ,
- V. Sounderajah ,
- Jonathan M. Carlson ,
- Matthew P Lungren ,
- Bay Gross ,
- Peter Hames ,
- Mustafa Suleyman ,
- Dominic King ,
- Eric Horvitz
Nature | , Vol 394 (6822): pp. 423-428
We explored the capabilities of frontier language models in sequential diagnosis, a diagnostic paradigm that captures the iterative nature of clinical reasoning. Given a brief clinical presentation, clinicians selectively acquired information to reach a diagnosis, enabling assessment of diagnostic accuracy and workup cost. We developed an interactive benchmark and an artificial intelligence (AI) system that simulates a panel of physicians, continually revising the differential diagnosis and selecting cost-effective tests. Using this benchmark, the system achieved 83.2% accuracy on New England Journal of Medicine clinicopathological conference cases and 80.3% on MIMIC-IV-ED emergency department admissions, outperforming generalist physicians (36.1 and 46.8%, respectively). Although these findings were obtained in a controlled benchmark setting, they highlight the promise of language models for supporting more accurate, lower-cost diagnoses.