Standard Evaluation Method for Evaluating a Classifier

A case detection algorithm is an example of a classifier. At the most general conceptual level, it classifies individuals into the categories "sick" or "not sick.'' In practice, these categories are typically more diagnostically precise ranging from, for example, "patient has E. coli 0157'' to "patient has fever.'' Similar to a diagnostic laboratory test, a case detection algorithm outputs a numeric value that is then compared to a threshold to determine whether to classify the individual as "sick'' or "not sick.''

To evaluate a case detection algorithm, the evaluator assembles two sets of individuals: a set of individuals with the disease or condition of interest (called cases) and a set of individuals that do not have the disease (called controls). The evaluator can find the cases from the records of a known outbreak, by a

Handbook of Biosurveillance ISBN 0-12-369378-0

Elsevier Inc. All rights reserved.

review of laboratory test results (e.g., microbiological cultures), or by asking experts to review medical or veterinarian records. The evaluator assembles the set of control patients by random selection.The control patients could be healthy individuals, or they could be individuals whose malady resembles the condition of interest, such as patients that have influenzalike illness, but do not have SARS. The choice depends on the intended use of the algorithm. If the algorithm will be used, for example, in an emergency department (ED) setting to automatically differentiate patients presenting with influenza-like symptoms into "has SARS'' and "does not have SARS,'' the evaluator would select controls from the set of all patients presenting with influenza-like symptoms determined not to have SARS. If the intended use is to differentiate SARS patients automatically from all other patients presenting to the ED, the evaluator would draw the control group from the set of all patients presenting to the ED known not to have SARS.

To illustrate the evaluation method concretely, let us suppose an evaluator has assembled a set of 100 patients with SARS and a control group of 100 individuals drawn from the set of patients coming to the ED who were subsequently proven not to have SARS. The evaluator's goal is to measure the accuracy of a case detection algorithm that classifies patients into SARS or not SARS based on patient data routinely collected in an ED. To measure the accuracy of the case detection algorithm, the evaluator would run the case detection algorithm on the patient data for each case and each control, recording whether the algorithm classifies each case and each control as "SARS" or "not SARS.'' The evaluator summarizes the results of this experiment in a 2 x 2 table, such as the one depicted in Table 20.1.

Table 20.1 shows the results of this hypothetical experiment. Of the 100 individuals with SARS, the case detection algorithm classified 90 correctly (true positives) and 10 incorrectly (false negatives). Of the 100 controls, the case detection algorithm classified five incorrectly as having SARS (false positives), and it classified 95 correctly as not having SARS (true negatives).

Was this article helpful?

0 0
Swine Influenza

Swine Influenza

SWINE INFLUENZA frightening you? CONCERNED about the health implications? Coughs and Sneezes Spread Diseases! Stop The Swine Flu from Spreading. Follow the advice to keep your family and friends safe from this virus and not become another victim. These simple cost free guidelines will help you to protect yourself from the swine flu.

Get My Free Ebook


Post a comment