Reviewing measures of outcome: reliability of data extraction

Rationale, aims and objectives Although there is wide acceptance of the necessary domains for the evaluation of measures of health outcome, there has been little development or evaluation of methods of data extraction, such as checklists. This study aimed to assess the inter-rater reliability of data extraction from published articles using a new electronic database developed for structured reviews of health outcome measures. Methods Domains considered important in the evaluation of measures of health outcome provided the foundation for a checklist for data extraction using a new electronic database. The database utilized Microsoft Access 2000 software (Microsoft Corporation, 2000). Following training, three independent reviewers extracted data from seven selected articles using the electronic database, and assessed article and measure quality against pre-defined criteria. Data extraction was compared for consistency and item content. Inter-reviewer agreement for categorical data was assessed by multiple kappa correlation. Results Analysis demonstrated strong agreement between reviewers for all aspects of data extraction. However, lack of clarity in published articles affected the ability to clearly identify measures of outcome. Conclusions The high level of inter-reviewer agreement supports the use of multiple trained reviewers in data extraction for reviews of measures of outcome using the checklist and Access software described. The electronic database supports standardized data extraction from published articles, benefiting from the combination of data extraction and data entry in a single step. Adaptation of the database to support structured reviews of measures of outcome adopted in the evaluation of other health states is proposed.