Predicting test-retest reliability from behavior coding

In attempting to mo ir questionnaire design from art to science, researchers use different evaluation techniques to help determine how well questions are working. Techniques such as behavior coding, respondent debriefing, interviewer debriefing, cognitive interviewing, and nonresponse analysis all provide information to help the questionnaire designer assess whether respondents understand questions as intended and whether they are able to provide adequate answers to them. However, these techniques do not actually measure question reliability. It is assumed that questions that pass the screen of the questionnaire evaluation techniques described above are also more likely to produce data that are reliable and valid. In this paper, we use behavior coding data to predict test-retest reliability. Respondent behavior codes significantly predict such reliability whereas interviewer codes-at least in this survey-do not. We also report the results of sensitivity testing to determine what percentage of adequate respondent answers best predicts test-retest reliability.