A COVERAGE ANALYSIS TOOL FOR THE EFFECTIVENESS OF SOFTWARE TESTING

This paper describes the software testing and analysis tool, ''ATAC (Automatic Test Analysis for C)'', developed as a research instrument at Bellcore to measure the effectiveness of testing data. It is also a tool to facilitate the design and evaluation of test cases during software development. To demonstrate the capability and applicability of ATAC, we obtained 12 program versions of a critical industrial application developed in a recent university/industry N-Version Software project, and used ATAC to analyze and compare coverage of the testing on the program versions. Preliminary results from this investigation show that ATAC is a powerful testing tool to provide testing metrics and quality control guidance for the certification of high quality software components or systems. In using ATAC to derive high quality test data, we assume that a good test has a high data-flow coverage score. This hypothesis requires that we show that good data-flow testing implies good software, viz, software with higher reliability, One would hope, for example, that code tested to 85% c-uses coverage would have a lower field-failure rate than similar code tested to 20% c-uses coverage. The establishment of a correlation between good data-flow testing and a low (or zero) rate of field failures is the ultimate and critical test of the usefulness of data-flow coverage testing. We demonstrated by ATAC that the 12 program versions obtained from the U. of Iowa and Rockwell NVS project (a project that has been subjected to a stringent design, implementation, and testing procedure) had very high testing coverage scores of blocks, decisions, c-uses, and p-uses. Results from the field testing (in which only one fault was found) confirmed this belief. The ultimate question that we hope ATAC can help us ans,ver is a typical question for all software reliability engineers: ''When is a program considered acceptable?'' Software reliability analysts have proposed several models to answer this question. However, none of these models address the issues of program structure or testing coverages, which are important in understanding software quality.