Validity and reliability of retrospective assessment of disease activity and flare in observational cohorts of lupus patients

Background: The use of validated retrospective tools to assess disease activity in an observational cohort of patients would allow researchers the flexibility to analyze unique exploratory questions. Valid, reliable tools exist for assessing disease activity and flare for Systemic Lupus Erythematosus (SLE) patients. However, these tools have been designed for use in structured settings. Many of these populations under study are subject to strict inclusion, exclusion criteria and disease management protocols. The ability to apply these tools to populations not subject to such control would allow researchers to explore questions about unique populations, new treatment applications or predictors of clinical outcomes. This study sought to establish the reliability and validity of retrospective medical record abstraction of SLE disease activity and flare instruments by two rheumatologists. Methods: From university rheumatology outpatient clinics, 22 patients were randomly selected to establish intra-rater reliability and 26 patients were selected to establish inter-rater reliability. Two rheumatologists using a Physician Global Assessment (PGA), the Systemic Lupus Erythematosus Disease Activity Index (SLEDAI) and the Safety of Estrogens in Lupus Erythematosus-National Assessment (SELENA) flare tool retrospectively abstracted patient charts. Agreement between the tools was evaluated to assess validity. Results: The mean patient age was 39 y; 96% were female, 54% Caucasian, 27% Hispanic, 19% Asian; median PGA was 1.4 ton a 0-3 scale) and median SLEDAI score was 4 (range 0-27). Intra-rater reliability for PGA, SLEDAI and the SELENA flare tool was 0.88, 0.87 and 0.52, respectively. Inter-rater reliability for PGA, SLEDAI, and SELENA flare was 0.79, 0.75, and 0.50, respectively. To assess validity, the tools were compared against each other to assess agreement. From the parent study sample (n=54 patients), the disease activity measures, PGA and SLEDAI demonstrated adequate agreement (r=0.60). However, the SELENA flare tool demonstrated poor agreement with either PGA-defined flare or SLEDAI-defined flare (weighted kappa, 0.29 and 0.40 respectively). PGA-defined and SLEDAI-defined flare also demonstrated poor agreement (weighted kappa, 0.35). Conclusions: These data suggest that investigators can reliably reproduce patient disease activity through retrospective chart abstraction using PGA and SLEDAI. Assessing flare is a more difficult concept. The validity of assessing flare at a specific patient-visit is poor. Retrospective assessment of patient flare risk over a specified time period is conceptually more valid and avoids difficulties assessing timing and duration of flare. As there have been no similar published prospective analyses of validity using the SELENA flare tool, it is not clear if this problem was unique to the method of retrospective chart abstraction, the nature of non-protocol study patient visits, the tool itself or a combination of all three aspects.