Blurring the distinctions between p's and α's in psychological research

Confusion over the reporting and interpretation of results of commonly employed classical statistical tests is recorded in a sample of 1,645 papers from 12 psychology journals for the period 1990 through 2002. The confusion arises because researchers mistakenly believe that their interpretation is guided by a single unified theory of statistical inference. But this is not so: classical statistical testing is a nameless amalgamation of the rival and often contradictory approaches developed by Ronald Fisher, on the one hand, and Jerzy Neyman and Egon Pearson, on the other. In particular, there is extensive failure to acknowledge the incompatibility of Fisher's evidential p value with the Type I error rate, alpha, of Neyman-Pearson statistical orthodoxy. The distinction between evidence (p's) and errors (alpha's) is not trivial. Rather, it reveals the basic differences underlying Fisher's ideas on significance testing and inductive inference, and Neyman-Pearson views on hypothesis testing and inductive behavior. So complete is this misunderstanding over measures of evidence versus error that it is not viewed as even being a problem among the vast majority of researchers and other relevant parties. These include the APA Task Force on Statistical Inference, and those writing the guidelines concerning statistical testing mandated in APA Publication Manuals. The result is that, despite supplanting Fisher's significance-testing paradigm some fifty years or so ago, recognizable applications of Neyman-Pearson theory are few and far between in psychology's empirical literature. On the other hand, Fisher's influence is ubiquitous.