Algorithms for case detection, outbreak detection, and outbreak characterization are critical components in a biosurveillance system. Although these components ultimately must be tested under field conditions, this level of evaluation is expensive and difficult, especially for outbreak detection and characterization due to the rarity of real outbreaks. Field testing is appropriate for algorithms whose potential has been demonstrated by laboratory testing. The methods for measuring sensitivity, specificity, and timeliness discussed in this chapter are general methods that can be applied both to laboratory and field evaluations.
Evaluators employ well-developed methods for evaluating classifiers when they evaluate case detection algorithms. These existing methods measure sensitivity and specificity of case detection. They summarize results using ROC curve analysis.
Researchers have extended these methods to enable the evaluation of algorithms for outbreak detection and characterization. In addition to measuring sensitivity and specificity, these evaluations always measure timeliness. Since sensitivity, specificity, and timeliness are correlated, evaluators have extended ROC curve analysis to include the dimension of time.
Researchers have further extended these methods to allow analysis of the relationship between the size of outbreak and detectability. Since outbreak size is also correlated with sensitivity, specificity, and timeliness, they have further extended ROC curve analysis to a fourth dimension—outbreak size.We refer to studies of this type as detectability analyses because their goal is to understand the smallest outbreak that an algorithm can detect.
Diagnostic precision is an important determinant of the utility of a case or outbreak detection algorithm, although the surveillance data available to the algorithm are the primary determinant of diagnostic precision. Diagnostic precision, sensitivity, specificity, timeliness, and outbreak size are all correlated. Thus, we expect that future researchers will extend ROC analysis into this fifth dimension.
Was this article helpful?