EXACT VERSUS ASYMPTOTIC MANTEL-HAENSZEL DIF STATISTICS - A COMPARISON OF PERFORMANCE UNDER SMALL-SAMPLE CONDITIONS

This study evaluated exact testing (Agresti, 1992) as a method for conducting Mantel-Haenszel DIF analyses (Holland & Thayer, 1988) with relatively small samples. Sample-size restrictions limit the standard asymptotic Mantel-Haenszel for many practical applications; however, new developments in computing technology have made exact testing procedures feasible. The highly discrete distributions that are likely to occur in small-sample DIF analyses could yield very different results for asymptotic versus exact methods. It is therefore important to determine under controlled conditions the extent to which the exact approach is effective in correctly identifying DIF. A series of computer simulations were conducted in which 3 levels of induced bias (IRT b-parameter differences between groups of .25, .50, and .75) and 4 sample sizes (reference group = 500, focal group = 25, 50, 100, and 200) were investigated. Power comparisons at .01 and .05 alpha levels were carried out between the exact testing procedure and the conventional Mantel-Haenszel.