Sweating the small stuff in educational psychology: how effect size and power reporting failed to change from 1969 to 1999, and what that means for the future of changing practices

Methodologists have written for years about the importance of attending to important details in quantitative research, yet there has been little research investigating methodological practice in the social sciences. This study assessed the extent to which innovations and practices are adopted by researchers voluntarily. In particular, I use the case of power analysis and effect size reporting as the primary example, but I also examine other reporting behaviours. Results show that while observed power and effect sizes in the educational psychology literature tend to be strong, researchers do not seem eager to adopt practices such as reporting effect sizes and power, and neither do they tend to report their testing assumptions or the quality of their measurement. There is room for much improvement in how we attend to the basics of quantitative research, and it does not appear that persuasion and professional communication are effective in changing practice.