Measuring Timeliness of Detection

The evaluator first determines the start date, defined as the date the outbreak began. When the evaluator simulates outbreak data, the start date is known because it is a parameter of the simulation. When using real outbreak data, however, the start date may be difficult to establish. The question of when any particular influenza outbreak began would generate a healthy discussion among epidemiologists. For purposes of research, the date can be established by a voting procedure or some other method to achieve expert consensus (Buckeridge et al., 2005). The eval-uator also establishes an end date using similar methods.

The date of detection is the first date on or after the start date that the algorithm generated a true alarm, defined as an alarm that occurred after the start date and before the end date. Figure 20.3 illustrates a hypothetical outbreak for which an evaluator established a start date of February 10. The algorithm generated alarms, indicated by "X" marks, on January 26, r:r.\>> start Date^.

Jan 21 Jan 26 Jan 31 Fib 05 Feb 10 Fib 15

Jan 21 Jan 26 Jan 31 Fib 05 Feb 10 Fib 15

Data figure 20.3 Alarms generated by a detection algorithm running on surveillance data organized into a time series of daily counts. The date of detection is February 13, the date of the first alarm after the start date of the outbreak (February 10). The alarm on January 26 is a false alarm.

February 13, and February 15. In this example, the date of the first true alarm was February 13, which the evaluator would consider the date of detection.

Timeliness of detection is the difference between the date of the first true alarm and a reference date. In Figure 20.3, the timeliness is three days. If the algorithm does not detect the outbreak at all, evaluators typically set the timeliness of detection to a fixed large value that approximates when the outbreak would most certainly be detected by conventional means (e.g., the duration of the entire outbreak).

Note that for studies that compare the performance of two algorithms, the accuracy or choice of the reference date is not critical, although it is most useful for interpretation purposes if it corresponds to a date that has decision importance, such as the date that the outbreak was actually detected by a health department. However, it is important that the reference date is defined objectively so that it can be consistently calculated for each outbreak in the sample or used by other researchers in future studies whose results collectively would then be more amenable to meta analysis.

0 0

Post a comment