Detecting sequential patterns and determining their reliability with fallible observers

On the basis of recent work by W. Gardner (1995), implications of fallible observers for observational research are discussed. Analysis shows that for identically fallible observers, values for kappa are lower when codes are few and their simple probabilities variable than when codes are many and roughly equiprobable; thus no one value of kappa can be regarded as universally acceptable. Additional analysis shows that fallible measurement degrades indices of sequential pattern more when codes are few and their simple probabilities variable. Finally, a simulation study establishes likely values for intraclass correlation reliabilities for sequential indices generated by various circumstances and suggests principled ways to select both lengths of sequences and acceptable levels of kappa for observational studies.