Evaluation of new online automated embolic signal detection algorithm, including comparison with panel of international experts

Background and Purpose-The clinical application of Doppler detection of circulating cerebral emboli will depend on a reliable automated system of embolic signal detection; such a system is not currently available. Previous studies have shown that frequency filtering increases the ratio of embolic signal to background signal intensity and that the incorporation of such an approach into an offline automated detection system markedly improved performance. In this study, we evaluated an online version of the system. In a single-center study, we compared its performance with that of a human expert on data from 2 clinical situations, carotid stenosis and the period immediately after carotid endarterectomy. Because the human expert is currently the "gold standard" for embolic signal detection, we also compared the performance of the system with an international panel of human experts in a multicenter study. Methods-In the single-center evaluation, the performance of the software was tested against that of a human expert on 20 hours of data from 21 patients with carotid stenosis and 18 hours of data from 9 patients that was recorded after carotid endarterectomy. For the multicenter evaluation, a separate a-hour data set, recorded from 5 patients after carotid endarterectomy, was analyzed by 6 different human experts using the same equipment and by the software. Agreement was assessed by determining the probability of agreement. Results-In the 20 hours of carotid stenosis data, there were 140 embolic signals with an intensity of greater than or equal to 7 dB. With the software set at a confidence threshold of 60%, a sensitivity of 85.7% and a specificity of 89.9% for detection of embolic signals were obtained. At higher confidence thresholds, a specificity >95% could be obtained, but this was at the expense of a lower sensitivity. In the Is hours of post-carotid endarterectomy data, there were 411 embolic signals of greater than or equal to 7-dB intensity. When the same confidence threshold was used, a sensitivity of 95.4% and a specificity of 97.5% were obtained. In the multicenter evaluation, a total of 127 events were recorded as embolic signals by at least 1 center. The total number of embolic signals detected by the 6 different centers was 84, 93, 108, 92, 63, and 78. The software set at a confidence threshold of 60% detected 90 events as embolic signals. The mean probability of agreement, including all human experts and the software, was 0.83, and this was higher than that for 2 human experts and lower than that for 4 human experts. The mean values for the 6 human observers were averaged to give P=0.84, which was similar to that of the software. Conclusions-By using the frequency specificity of the intensity increase occurring with embolic signals, we have developed an automated detection system with a much improved sensitivity. Its performance was equal to that of some human experts and only slightly below the mean performance of a panel of human experts.