The accuracy of expert-system diagnoses of mathematical problem solutions

Expert systems have the potential to help computer-based testing programs give qualitative feedback about examinee performance on constructed-response items. This study evaluated the accuracy of such feedback for algebra word problems. The responses of Graduate Record Examinations examinees were diagnostically analyzed by an expert system and by four human judges. Results showed that human judges agreed highly among themselves about whether errors were present in a solution, to a lesser degree when errors were categorized generally, and to only a limited degree on the detailed characterization of those faults. The expert system agreed very closely with the judges in characterizing responses as right or wrong but somewhat less so on classifying errors using either specific or general schemes. The accuracy of automatic qualitative judgments may be increased by using more general diagnostic categories and by integrating information from other sources, including performance on diverse item types.