Using simulation to assess the sensitivity and specificity of a signal detection tool for multidimensional public health surveillance data

The objective of the work described in this paper is to develop a means for characterizing the validity of an empirical methodology for detecting signals potentially related to complicated adverse event (AE) coding terms in multidimensional public health surveillance data. The signal detection tool under evaluation is the multi-item gamma Poisson shrinkage (MGPS) estimation program. We were interested in its potential application to passive surveillance system monitoring. to screen for 'signals' of complicated adverse event coding terms (AE terms) in complex and noisy data. The research was to design and produce a flexible and user-friendly utility for probabilistically defining, complicated signals in a database. iterating large numbers of applications of the MGPS detection algorithm and establishing proportions of correct detection events. We sought to establish the specificity of the MGPS by developing a random background using a gradient that ranged from rigorous (but not very relevant) to relevant (but noisy). To establish the sensitivity, signals were defined based on recognized public health issues of interest (such as the introduction of a new vaccine into the population). Methods of representing a signal included a simple pair-wise association consisting of a new vaccine and one AE term. as well as a more realistic complex of multiple AE terms comprising a 'syndrome'. A web application has been developed to create and insert signals with user-defined probabilities in multiple iterations of simulated random background data. Three forms of simulated data based on the vaccine adverse event reporting system (VAERS) cumulative spontaneous database were defined to serve as background noise against which to contrast introduced vaccine adverse event signals: (1) completely random associations between vaccines and AE terms, (2) random associations of vaccine sets and AE term sets preserving naturally observed vaccine co-occurrences and AE term co-occurrences and (3) samples from the actual VAER data as reported. Rates of detection by the MGPS algorithm can be established for specific signal patterns at varying probabilistic intensities in a choice of random background data forms. Knowing these rates is important for determining the degree of response to an MGPS signal detection event in 'live' data. Published in 2005 by John Wiley Sons, Ltd.