Sequential Diagnosis with Language Models

  • ,
  • M. Daswani ,
  • Christopher Kelly ,
  • Scott M. Lundberg ,
  • Marco Túlio Ribeiro ,
  • Marc Wilson ,
  • Xiaoxuan Liu ,
  • V. Sounderajah ,
  • ,
  • Matthew P Lungren ,
  • Bay Gross ,
  • Peter Hames ,
  • Mustafa Suleyman ,
  • Dominic King ,

Nature | , Vol 394 (6822): pp. 423-428

We explored the capabilities of frontier language models in sequential diagnosis, a diagnostic paradigm that captures the iterative nature of clinical reasoning. Given a brief clinical presentation, clinicians selectively acquired information to reach a diagnosis, enabling assessment of diagnostic accuracy and workup cost. We developed an interactive benchmark and an artificial intelligence (AI) system that simulates a panel of physicians, continually revising the differential diagnosis and selecting cost-effective tests. Using this benchmark, the system achieved 83.2% accuracy on New England Journal of Medicine clinicopathological conference cases and 80.3% on MIMIC-IV-ED emergency department admissions, outperforming generalist physicians (36.1 and 46.8%, respectively). Although these findings were obtained in a controlled benchmark setting, they highlight the promise of language models for supporting more accurate, lower-cost diagnoses.