Application of recently developed computer algorithm for automatic classification of unstructured radiology reports: Validation study

PURPOSE: To validate the accuracy of Lexicon Mediated Entropy Reduction (LEXIMER), a new information theory-based computer algorithm developed by the authors for independent analysis and classification of unstructured radiology reports, based on the presence of clinically important findings (F, where T represents "true") and recommendations for subsequent action (R-T). MATERIALS AND METHODS: The study was approved by the Human Research Committee of the institutional review board. Consecutive de-identified radiology reports (n = 1059) comprising results of barium studies (n = 99), computed tomography (n = 107), mammography (n = 90), magnetic resonance imaging (n = 108), nuclear medicine (n = 99), positron emission tomography (n = 106), radiography (n = 212), ultrasonography (n = 131), and vascular procedures (n = 107) were independently analyzed by two radiologists and then with LEXIMER to categorize the reports into F-T and FTO (containing or not containing clinically important findings) categories and R-T and RTO (containing or not containing recommendations for subsequent action) categories. Accuracy, sensitivity, specificity, and positive and negative predictive values of LEXIMER for placing reports into F-T and FTO and R-T and RTO categories were assessed by using appropriate statistical tests. RESULTS: There was strong interobserver concordance between the two radiologists for placing radiology reports into F-T and R-T categories (kappa = 0.9, P <.01). For the LEXIMER program, accuracy, sensitivity, specificity, and positive and negative predictive values, respectively, were 97.5% (95% confidence interval [Cl]: 96.6%, 98.5%), 98.9% (95% Cl: 97.9%, 99.6%), 94.9% (95% Cl: 93.1%, 96.0%), 97.5% (95% Cl: 96.6%, 98.0%), and 97.7% (95% Cl: 95.8%, 98.8%) for placing radiology reports into FT and FTO categories and 99.6% (95% Cl: 99.2%, 99.9%), 98.2% (95% Cl: 95.0%, 99.6%), 99.9% (95% Cl: 99.4%, 99.99%), 99.4% (95% Cl: 96.3%, 99.9%), and 99.7% (95% Cl: 98.9%, 99.9%) for placing reports into R-T and RTO categories. CONCLUSION: LEXIMER is an accurate automated engine for evaluating the percentage positivity of clinically important findings and rates of recommendation for subsequent action in unstructured radiology reports.