Visual analysis of serial T2-weighted MRI in multiple sclerosis:: intra- and interobserver reproducibility

We evaluated the effect of consensus formation and training on the agreement between observers in scoring the number of new and enlarging multiple sclerosis (MS) lesions on serial T2-weighted MRI studies. The baseline and month 9 MRI studies of 16 patients with a range of MRI activity were used (dual-echo conventional spin-echo sequence, TR 2000, TE 34 and 90 ms, 5 mm contiguous slices, inplane resolution 1 mm). First, the serial studies were visually analysed for the presence of new and enlarging lesions, on two occasions, by five experienced observers, without adopting any consensus strategy and in isolation. Next, the observers met to identify the common sources of inconsistencies in reporting between observers and formulate consensus rules. Finally, a further independent reading session was performed on the same MRI dataset, this time applying the consensus rules. Agreement between observers was assessed using kappa scores. Without the consensus rules, interobserver kappa scores for the first and second reading sessions for new lesions were only 0.51 and 0.39 respectively; agreement for enlarging lesions was even worse. The mean intraobserver kappa score for new lesions was higher at 0.72, reflecting the fact that the observers were consistently applying their individual assessment strategies. Application of the consensus rules did not lead to a significant improvement in inter observer kappas; the kappa scores adopting the guidelines were 0.46 and 0.21 for new and enlarging lesions respectively. Consensus guidelines thus did not improve the reproducibility of visual analysis of serial T2-weighted MRI, and the level of agreement between observers remained only moderate. Suboptimal repositioning is likely to be a major source of residual variability and this suggests a future role for image registration strategies; until then, a single observer, or pair of observers working in consensus, should be used in MS studies.