Conclusions about frequently studied modified Angoff standard-setting topics

This article reviews the empirical literature on 9 topics about the modified Angoff standard-setting method that have been studied repeatedly in the literature, while taking into consideration the methodological warrant for the findings on the topics. It concludes that we can be reasonably confident about selecting the appropriate number of judges and about the extent to which judges' modified Angoff item estimates are ranked similarly to item difficulty. Item estimates probably deviate inconsistently from difficulty values too frequently, although this deficiency in the method might be remedied somewhat by the effects of judge activities between standard-setting rounds. More studies need to be done about the appropriate level of judge expertise and about the process of describing the performance level at which the cutscores are to be set. The warrants for the findings of much of the empirical modified Angoff literature are often insufficient for making firm conclusions, and many uncertainties about the method remain.