Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

The reproducibility issues that haunt health-care AI

Health-care systems are rolling out artificial-intelligence tools for diagnosis and monitoring. But how reliable are the models?
  1. Emily Sohn

    1. Emily Sohn is a freelance journalist in Minneapolis, Minnesota.

    View author publications

    You can also search for this author in PubMed  Google Scholar

Each day, around 350 people in the United States die from lung cancer. Many of those deaths could be prevented by screening with low-dose computed tomography (CT) scans. But scanning millions of people would produce millions of images, and there aren’t enough radiologists to do the work. Even if there were, specialists regularly disagree about whether images show cancer or not. The 2017 Kaggle Data Science Bowl set out to test whether machine-learning algorithms could fill the gap.

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Nature 613, 402-403 (2023)

doi: https://doi.org/10.1038/d41586-023-00023-2

Updates & Corrections

  • Correction 12 January 2023: An earlier version of this feature erroneously stated that Sayash Kapoor discovered reproducibility failures and pitfalls in 329 studies across 17 fields. In fact, those studies had themselves reported reproducibility failures and pitfalls.

References

  1. Yu, K. H. et al. J. Med. Internet Res. 22, e16709 (2020).

    Article  PubMed  Google Scholar 

  2. Roberts, M. et al. Nature Mach. Intell. 3, 199–217 (2021).

    Article  Google Scholar 

  3. McDermott, M. B. A. et al. Sci. Transl. Med. 13, eabb1655 (2021).

    Article  PubMed  Google Scholar 

  4. Wong, A. et al. JAMA Intern. Med. 181, 1065–1070 (2021).

    Article  PubMed  Google Scholar 

  5. Mongan, J., Moy, L. & Kahn, C. E. Radiol. Artif. Intell. 2, e200029 (2020).

    Article  PubMed  Google Scholar 

Download references

Subjects

Latest on:

Nature Careers

Jobs

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing

Search

Advanced search

Quick links