Analyzing proficiency criteria of health technology systems: The case of drug testing

The management of certain types of health care technology requires that the integrity of the technological system ( which includes the technology and the capability of the users of the technology) be routinely assessed to ensure that the system is providing accurate information. Proficiency testing of these health care technology systems is important, not only to ensure the integrity of screening and diagnostic processes, but also for ethical and liability reasons. While the importance of proficiency testing is well recognized, and while considerable effort is often devoted to such testing, little attention has been given to assessing the validity of the criteria used to evaluate proficiency. Using systems analysis principles, this paper examines the criteria used in drug testing laboratory proficiency studies that were published in major medical and scientific journals from 1976 to 1996, Although we use drug testing as an exemplar, it should be recognized that the systems analysis methodology provides a rational and systematic approach that is equally applicable to proficiency studies of any process where several different types of tests are performed to make decisions about the presence or absence of a particular condition. In this paper, it is shown that all of the criteria used to evaluate the accuracy of drug testing are erroneous. Three general types of criterion errors are present: ignoring the sources of false results, using an improper unit of analysis, and improper treatment of incommensurable factors, As a result, these proficiency studies of drug testing technology systems are of dubious value for formation of health policy about drug testing, or for informing clinical decision makers about the probabilities that drug tests will correctly classify specimens.