Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers

被引：106

作者：

Rathbone J. ^{[1
]}

Hoffmann T. ^{[1
]}

Glasziou P. ^{[1
]}

机构：

[1] Bond University, Centre for Research in Evidence-Based Practice, Gold Coast

来源：

Systematic Reviews | / 4卷 / 1期

关键词：

False Negative Rate; Hemolytic Uremic Syndrome; Eculizumab; Recall Accuracy; Imbalanced Dataset;

D O I：

10.1186/s13643-015-0067-6

中图分类号：

学科分类号：

摘要：

Background: Citation screening is time consuming and inefficient. We sought to evaluate the performance of Abstrackr, a semi-automated online tool for predictive title and abstract screening. Methods: Four systematic reviews (aHUS, dietary fibre, ECHO, rituximab) were used to evaluate Abstrackr. Citations from electronic searches of biomedical databases were imported into Abstrackr, and titles and abstracts were screened and included or excluded according to the entry criteria. This process was continued until Abstrackr predicted and classified the remaining unscreened citations as relevant or irrelevant. These classification predictions were checked for accuracy against the original review decisions. Sensitivity analyses were performed to assess the effects of including case reports in the aHUS dataset whilst screening and the effects of using larger imbalanced datasets with the ECHO dataset. The performance of Abstrackr was calculated according to the number of relevant studies missed, the workload saving, the false negative rate, and the precision of the algorithm to correctly predict relevant studies for inclusion, i.e. further full text inspection. Results: Of the unscreened citations, Abstrackr's prediction algorithm correctly identified all relevant citations for the rituximab and dietary fibre reviews. However, one relevant citation in both the aHUS and ECHO reviews was incorrectly predicted as not relevant. The workload saving achieved with Abstrackr varied depending on the complexity and size of the reviews (9% rituximab, 40% dietary fibre, 67% aHUS, and 57% ECHO). The proportion of citations predicted as relevant, and therefore, warranting further full text inspection (i.e. the precision of the prediction) ranged from 16% (aHUS) to 45% (rituximab) and was affected by the complexity of the reviews. The false negative rate ranged from 2.4 to 21.7%. Sensitivity analysis performed on the aHUS dataset increased the precision from 16 to 25% and increased the workload saving by 10% but increased the number of relevant studies missed. Sensitivity analysis performed with the larger ECHO dataset increased the workload saving (80%) but reduced the precision (6.8%) and increased the number of missed citations. Conclusions: Semi-automated title and abstract screening with Abstrackr has the potential to save time and reduce research waste. © 2015 Rathbone et al.

引用

共 25 条

[1] Rathbone J., Kaltenthaler E., Richards A., Tappenden P., Bessey A., Cantrell A., A systematic review of eculizumab for atypical haemolytic uraemic syndrome (aHUS), BMJ Open., 3, pp. 1-11, (2013)
[2] Frunza O., Inkpen D., Matwin S., Building systematic reviews using automatic text classification techniques, (2010)
[3] Hoffmann T., Erueti C., Thorning S., Glasziou P., The scatter of research: cross sectional comparison of randomised trials and systematic reviews across specialties, BMJ., 344, (2012)
[4] Moher D., Liberati A., Tetzlaff J., Altman D.G., The PRISMA Group. 2009. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement, BMJ, 339, (2009)
[5] (2015)
[6] Higgins J.P.T., Altman D.G., Gotzsche P.C., Juni P., Moher D., Oxman A.D., Et al., The Cochrane Collaboration's tool for assessing risk of bias in randomised trials, BMJ., 343, (2011)
[7] Jefferson T., Jones M.A., Doshi P., Del Mar C.B., Hama R., Thompson M.J., Et al., Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children, Cochrane Database Syst Rev, 4, (2014)
[8] Kristiansen I., How up-to-date are Cochrane reviews?, Lancet., 371, (2008)
[9] Shojania K., Sampson M., Ansari M., Ji J., Doucette S., Moher D., How quickly do systematic reviews go out of date? A survival analysis, Ann Intern Med., 147, pp. 224-233, (2007)
[10] Leeper N., Bauer-Mehren A., Iyer S., Lependu P., Olson C., Shah N., Practice-based evidence: profiling the safety of cilostazol by text-mining of clinical notes, PLoS ONE., 8, (2013)

← 1 2 3 →