How good is intercenter agreement in the identification of embolic signals in carotid artery disease?

Background and Purpose There has been concern regarding the reproducibility of the detection of embolic signals, particularly in patients with carotid artery stenosis in whom the signals are of low intensity. No published studies have examined intercenter agreement in reporting specific embolic signals or the factors responsible for any lack of agreement. We examined reproducibility between two centers in which widely differing proportions of embolic signals have previously been reported in patients with carotid artery stenosis. Methods Recordings from the middle cerebral artery of eight patients with ipsilateral carotid artery stenosis in whom embolic signals had been detected during a previous study were independently examined by three experienced observers in one center and by one experienced observer in another center. We calculated agreement within and between centers by estimating the probability that one observer would identify a specific embolic signal if other observers had identified it (a probability of I indicates complete agreement). The influence of different characteristics of the embolic signal on the probability of its detection as an embolic signal was determined. Results A high level of agreement in the identification of specific embolic signals was found. This was similar between all observers (.90), between the three observers in one center (.89), and between observers in the two different centers (.94). The probability of detection was independently related to the relative intensity of the embolic signal (P<.0001). It was less (although significantly) independently related to the position of the embolic signal in the cardiac cycle (P=.02), with signals in systole being more reliably detected. There was no independent relationship between the probability of detection and either the duration of the embolic signal or the velocity at the maximum intensity increase. The use of threshold intensity as a criterion for embolic signal detection increased interobserver agreement but reduced the sensitivity in detecting signals. Conclusions The high level of interobserver agreement suggests that the technique is sufficiently reproducible for clinical use.