CASE tool evaluation: experiences from an empirical study

While research activity in software engineering often results in the development of software tools and solutions that are intended to demonstrate the feasibility of an idea or concept, any resulting conclusions about the degree of success attained are rarely substantiated through the use of supporting experimental evidence. As part of the development of a prototype computer assisted software engineering (CASE) tool intended to support opportunistic software design practices, we sought to evaluate the use of the tool by both experienced and inexperienced software engineers. This work involved performing a review of suitable techniques, and then designing and performing a set of experimental studies to obtain data which could be used to assess how well the CASE tool met its design goals. We provide an assessment of how effective the chosen evaluation process was, and conclude by identifying the need for an 'evaluation framework' to help with guiding such studies. (C) 2002 Elsevier Inc. All rights reserved.