Early warning systems: The neglected importance of timing

Automated early warning systems (EWSs) use data inputs to recognize clinical states requiring time-sensitive intervention and then generate notifications through different modalities to clinicians. EWSs serve as common tools for improving the recognition and treatment of important clinical states such as sepsis. However, despite the early enthusiasm, these warning systems have often yielded disappointing outcomes. In sepsis, for example, EWSs have shown mixed results in clinical trials, and concerns regarding the overuse of EWSs in diagnosing sepsis have grown. We argue that inattention to the importance of timing in EWS training and evaluation provides one reason that EWSs have underperformed. Thus, to improve care, a warning system must not only identify the clinical state accurately, but it must also do so in a sufficiently timely manner to implement the associated interventions, such as administration of antibiotics for sepsis. Although the literature has occasionally highlighted the importance of timing in electronic surveillance systems, no one has linked the temporal dependence of performance metrics and intervention feasibility to the failure of such warning systems and explained how to operationalize timing in their development.Using sepsis as an example, we explain why timing is important and propose new metrics and strategies for training and evaluating EWS models. EWSs are divided into two types: detection systems that recognize critical illnesses at a particular moment and prediction systems that estimate risk of deterioration over varying time frames.We focus primarily on detection systems, but our analysis is also important for prediction systems, which we will discuss in the last section