## Finding the pValue for a Wsare Rule

When reporting the significance of the highest scoring rule, it is essential that we take into account the intensity of WSARE's search for anomalies. Even if surveillance data were generated randomly, entirely from a null distribution, we can expect that the best rule would be alarmingly high-scoring if we had searched over 1000 possible rules. In order to illustrate this point, suppose we follow the standard practice of rejecting the null hypothesis when the p-value is >a, where a= 0.05. In the case of a single hypothesis test, the probability of a false positive under the null hypothesis would be a, which equals 0.05. On the other hand, if we perform 1000 hypothesis tests, one for each possible rule under consideration, then the probability of a false positive could be as bad as 1 - (1 - 0.05)1000 ~ 1, which is much greater than 0.05 (Miller et al., 2005). Thus, if the algorithm returns a large score, we cannot accept it at face value without adding an adjustment for the multiple hypothesis tests we performed. This problem can be addressed using a Bonferroni correction (Bonferroni, 1936), but this approach would be unnecessarily conservative. Instead, we turn to a randomization test in which the date and each ED case features are assumed to be independent. In this test, the case features in the data set DB remain the same for each record but the date field is shuffled between records from the current day and records from five to eight weeks ago. The full method for the randomization test is shown below.

Let UCS = Uncompensated score: the score of the best scoring rule BR as defined above.

Let DB(j) = newly randomized data set

Let UCS(j) = Uncompensated p-value of BR(j) on DB(j) Let the compensated p-value of BR be

Cpv = # of Randomized Tests in which UCP(j) < UCP # Randomized Tests

CPv is an estimate of the chance that we would have seen an uncompensated score as large as UCS if in fact there was no relationship between date and case features. Note that we do not use the uncompensated score UCS after the randomization test. Instead, the compensated p-value CPv is used do determine the level of anomaly.