Evaluation Methods For Nlp In Biosurveillance

The first step in evaluating an NLP application is to validate its ability to classify, extract, or encode features from text (feature detection). Most evaluations of NLP technology in the biomedical domain have focused on this phase of evaluation. Once we validate feature detection performance, we can evaluate the ability of the encoded features to diagnose individual cases of interest (case detection). Finally, we can perform summative evaluations addressing the ability to detect epidemics (epidemic detection). Figure 17.4 shows how the three levels of evaluation relate to one another, using the diagnostic system for SARS as an example.

figure 17.4 Relationship between the three levels of evaluation for biosurveillance. Evaluations of feature detection quantify how well variables and their values are automatically encoded from text. Evaluations of case detection quantify the ability to accurately diagnose a single patient from the variables encoded from text, which may or may not be combined with other variables. Evaluations in epidemic detection quantify whether the variable being monitored by detection algorithms can detect outbreaks.

figure 17.4 Relationship between the three levels of evaluation for biosurveillance. Evaluations of feature detection quantify how well variables and their values are automatically encoded from text. Evaluations of case detection quantify the ability to accurately diagnose a single patient from the variables encoded from text, which may or may not be combined with other variables. Evaluations in epidemic detection quantify whether the variable being monitored by detection algorithms can detect outbreaks.

0 0

Post a comment