Moving from in silico to in clinico evaluations of machine learning-based interventions in critical care

The patient at risk for critical illness seems a likely beneficiary of machine learning (ML)-based interventions. Their care decisions are high stakes, complex, replete with uncertainty, and informed by rapidly changing, high volume, and data inputs. Because ML methods are so suited to clinical decision support (CDS) for critical care, deterioration models have been among the earliest and most popular targets of ML model development efforts.

But do we know how ML models affect care decisions or patient outcomes? The majority of evidence for ML models in critical care is in silico or based on a model’s ability to predict an outcome in a retrospectively collected dataset. What the field largely lacks is robust evidence about a model’s impact in clinico or how a model might affect care processes and patient outcomes when deployed into clinical workflows.

Evidence about safety and equity are also needed but largely absent. Deterioration alerts are not entirely benign as they allocate resources toward some patients and away from others.

In this issue of Critical Care Medicine, Levin et al report the results of a pragmatic, nonrandomized, open-label, cluster trial (Realtime Streaming Clinical Use Engine for Medical Escalation [ReSCUE-ME]) of a ML-based early warning system to predict in-hospital deterioration among ward patients at a large academic hospital in New York City. In doing so, the authors make an important contribution to the literature by providing in clinico evidence for how this model affects both treatments and outcomes.

Gary E Weissman