Detection and determinants of bias in subjective measures

Many concepts in sociology are difficult or impossible to objectively measure. This limitation forces a reliance on subjective measures that typically contain both systematic and random measurement errors. Systematic errors, or "biases," are the focus of this paper. Campbell and Fiske's (1959) multitrait-multimethod (MTMM) research design is the best known social scientific procedure for uncovering systematic errors, but the data requirements for classical MTMM designs are too demanding for many areas of sociology in which secondary data are the norm. We show that the benefits of the MTMM design are available under more relaxed conditions. In addition, we illustrate how researchers can examine the determinants of systematic errors and gain insights into the potential for confounding or spurious effects caused by systematic errors. We demonstrate the usefulness of these methods using the subjective measures of liberal democracy used in several recent ASR papers and provide additional examples, including measures of the reputational quality of graduate programs and job evaluations for comparable-worth investigations. We conclude that sociologists can do far more to understand the systematic error present in their subjective variables.