Reliability and validity of a computer-based knowledge mapping system to measure content understanding

The motivating concern behind CRESST's knowledge mapping research has been the desire to assess students' understanding of relationships utilizing a format that departs from both standardized multiple-choice testing and discourse-dependent tasks (e.g. essays). Fusing computer technology and knowledge mapping environments creates new instructional and assessment opportunities, primarily by providing the capability for real-time scoring, feedback, reporting, networking, and Internet/Web access. Overall, 149 middle and high school students participated in two computer-based knowledge mapping studies: (1) students constructed group maps while collaborating over a network, and (2) students constructed individual maps, attempting to improve their maps while searching a Web space. Results showed that knowledge map scores increased significantly from fall to spring for students constructing maps individually while searching a web space containing relevant and irrelevant information about environmental science. This finding supports the position that students' abilities in searching and processing information on the Internet may have increased as a result of using computer technology over the course of the school year. On the other hand, virtually no differences were found for group knowledge maps from fall to spring, and therefore no clear relationships among content understanding and collaboration were found. Four teachers' knowledge maps served as 'expert criteria' to evaluate individual student's and group's knowledge maps. The most important reliability issue for both studies focused on inter-expert agreement using teachers' maps to score students' maps. In the studies presented here, inter-expert pairwise agreements for both studies were in the 80-90 percentage range. (C) 1999 Published by Elsevier Science Ltd. All rights reserved.