A bold first toe into the uncharted waters of evaluating proprietary clinical prediction models

Would a clinician prescribe a new medication in the absence of any data about its efficacy or safety? Of course not. Regulatory authorities like the Food and Drug Administration (FDA) and good clinical judgment would prevent such a blunder. Then why would a health system deploy a clinical prediction model, designed to inform high-stakes decisions for patients at risk for critical illness, without any evidence of efficacy or safety?

Although the FDA’s regulatory strategy for clinical prediction models continues to mature and expand to include guidance around equity, transparency, and safety, significant gaps and uncertainties in oversight remain. For example, there are currently no federal regulatory standards for predictive clinical decision support (CDS) systems developed locally by hospitals. Those developed by private-sector companies for sale on the market may, in some cases, require FDA approval if they meet certain criteria. However, some of these criteria remain vague, and models released before these criteria were published have an uncertain fate.

The Epic Deterioration Index (EDI) is one such CDS system that may meet criteria for FDA regulation as a medical device and is reportedly in use in “hundreds of hospitals in the United States”. The EDI is a commercially available predictive CDS built by EPIC systems to identify patients at risk of clinical deterioration, was developed prior to the coronavirus disease (COVID-19) pandemic, and uses predictor variables such as patient age (but not race or sex), vital signs, nursing assessments, and laboratory values. However, the EDI is neither approved by the FDA nor had its performance, safety, or other important characteristics been reported in any peer-reviewed journal until now.

National Institutes of Health