Towards standardized quality control for AI systems in critical care
Goal
The overarching goal of this project is to develop quality assessment standards for machine learning approaches in critical care. As a part of a bigger initiative - redefining the metrology as a science applied to artificial intelligence and machine learning workflows, this project will develop machine learning solutions for selected problems in critical care with special emphasis on quality aspects such as robustness, uncertainty calibration, explainability and fairness. To overcome limitations imposed by privacy concerns, synthetic reference data will be generated from large real data sets as part of the process.
Overview
As machine learning approaches start to be applicable to solving ambitious problems in medicine, a need arises for controlling quality aspects as listed above using standardized evaluation protocols and benchmarks. Machine learning solutions are of particular interest in intensive care units (ICU), where data are continuously recorded with high temporal resolution and where timely decision making is key.
Funding
Partners
- Technical University of Berlin, Faculty IV
- Charité - Universitätsmedizin Berlin, Institute of Medical Informatics
Publications
Publication single view
Article
| Title: | Predicting lethal courses in critically ill COVID-19 patients using a machine learning model trained on patients with non-COVID-19 viral pneumonia |
|---|---|
| Author(s): | G. Lichtner, F. Balzer, S. Haufe, N. Giesa, F. Schiefenhövel, M. Schmieding, C. Jurth, W. Kopp, A. Akalin, S. J. Schaller, S. Weber-Carstens, C. Spies;F. von Dincklage |
| Journal: | Scientific Reports |
| Year: | 2021 |
| Volume: | 11 |
| Issue: | 1 |
| Pages: | 13205 |
| DOI: | 10.1038/s41598-021-92475-7 |
| ISSN: | 2045-2322 |
| Web URL: | http://www.nature.com/articles/s41598-021-92475-7 |
| Abstract: | Abstract In a pandemic with a novel disease, disease-specific prognosis models are available only with a delay. To bridge the critical early phase, models built for similar diseases might be applied. To test the accuracy of such a knowledge transfer, we investigated how precise lethal courses in critically ill COVID-19 patients can be predicted by a model trained on critically ill non-COVID-19 viral pneumonia patients. We trained gradient boosted decision tree models on 718 (245 deceased) non-COVID-19 viral pneumonia patients to predict individual ICU mortality and applied it to 1054 (369 deceased) COVID-19 patients. Our model showed a significantly better predictive performance (AUROC 0.86 [95% CI 0.86–0.87]) than the clinical scores APACHE2 (0.63 [95% CI 0.61–0.65]), SAPS2 (0.72 [95% CI 0.71–0.74]) and SOFA (0.76 [95% CI 0.75–0.77]), the COVID-19-specific mortality prediction models of Zhou (0.76 [95% CI 0.73–0.78]) and Wang (laboratory: 0.62 [95% CI 0.59–0.65]; clinical: 0.56 [95% CI 0.55–0.58]) and the 4C COVID-19 Mortality score (0.71 [95% CI 0.70–0.72]). We conclude that lethal courses in critically ill COVID-19 patients can be predicted by a machine learning model trained on non-COVID-19 patients. Our results suggest that in a pandemic with a novel disease, prognosis models built for similar diseases can be applied, even when the diseases differ in time courses and in rates of critical and lethal courses. |