MASCIS evaluation of open field locomotor scores: Effects of experience and teamwork on reliability

The Multicenter Animal Spinal Cord Injury Study (MASCIS) adopted a modified 21-point open field locomotor scale developed by Basso, Beattie, and Bresnahan (BBB) at Ohio State University (OSU) to measure motor recovery in spinal-injured rats. BBB scores categorize combinations of rat hindlimb movements, trunk position and stability, stepping, coordination, paw placement, toe clearance, and tail position, representing sequential recovery stages that rats attain after spinal cord injury. A total of 22 observers from 8 participating centers assessed 18 hindlimbs of 9 rats at 2-6 weeks after graded spinal cord injury. The observers were segregated into 10 teams. The teams were grouped into 3 cohorts (A, B, and C), consisting of one experienced team from OSU and two non-OSU teams. The cohorts evaluated the rats in three concurrent and sequential sessions. After viewing a rat for 4 min, individual observers first assigned scores without discussion. Members of each team then discussed and assigned a team score. Experience (OSU vs. non-OSU) and teamwork (individual vs. team) had no significant effect on mean scores although the mean scores of one cohort differed significantly from the others (p = 0.0002, ANOVA). However, experience and teamwork significantly influenced reliability of scoring. OSU team scores had a mean standard deviation or discordance of 0.59 points, significantly less than 1.31 points for non-OSU team scores (p = 0.003, ANOVA) and 1.30 points for non-OSU individual scores (p = 0.001, ANOVA). Discordances were greater at the upper and lower ends of the scale, exceeding 2.0 in the lower (<5) and upper (>15) ends of the scale but were <1.0 for scores between 4 and 16. Comparisons of non-OSU and OSU team scores indicated a high reliability coefficient of 0.892 and a correlation index (r(2)) of 0.894. These results indicate that inexperienced observers can learn quickly to assign consistent BBB scores that approach those given by experienced teams, that the scores are most consistent between 4 and 16, and that experience improves consistency of team scores.