ANALYSIS AND REFINEMENT OF SOFTWARE TEST DATA ADEQUACY PROPERTIES

Test data adequacy criteria are standards that can be applied to decide if enough testing has been performed. Previous research in software testing has suggested 11 fundamental properties which reasonable criteria should satisfy if the criteria make use of the structure of the program being tested. We show that there are several dependencies among the 11 properties as originally stated, making them questionable as a set of fundamental properties. We further show how the statements of the properties can be generalized so that the properties can be appropriately analyzed with respect to criteria that do not necessarily make use of the program's structure. We then perform this analysis and show the relationships among the properties with respect to different classes of criteria that utilize the program structure and the specification in different ways. We show that certain apparently intuitive assumptions about the manner in which test cases are chosen cause inconsistency among the properties for a wide range of criteria. These assumptions are also important in distinguishing a model of testing in which the specification is the only arbiter of correctness from one that acknowledges the possibility that the specification is defective. We show how the properties differ under these two models in order to maintain consistency. The dependencies are largely a result of five very weak "existential" properties. By modifying three of the properties we show how to eliminate these weaknesses. The result is a reduced set of seven properties, each of which is strong from a mathematical perspective. The new set of properties is conceptually similar to the original set.