Ecometrics: Toward a science of assessing ecological settings, with application to the systematic social observation of neighborhoods

This paper considers the quantitative assessment of ecological settings such as neighborhoods and schools. Available administrative data typically provide useful but limited information on such settings. We demonstrate how more complete information can be reliably obtained from surveys and observational studies. Survey-based assessments are constructed by aggregating over multiple item responses of multiple informants within each setting. Item and rater inconsistency produce uncertainty about the setting being assessed, with definite implications for research design. Observation-based assessments also have a multilevel error structure. The paper describes measures constructed from interviews, direct observations, and videotapes of Chicago neighborhoods and illustrates an "ecometric" analysis-a study of bias and random error in neighborhood assessments. Using the observation data as an illustrative example, we present a three-level hierarchical statistical model that identifies sources of error in aggregating across items within face-blocks and in aggregating across face-blocks to larger geographic units such as census tracts. Convergent and divergent validity are evaluated by studying associations between the observational measures and theoretically related measures obtained from the U.S. Census, and a citywide survey of neighborhood residents.