The Statistical Crisis in Science

There needs to be a balance between maintaining the strictest statistical controls and allowing researchers some flexibility to pursue analysis of unexpected trends observed in a study beyond the limits of pre-registered primary analysis. Given a particular data set, it can seem entirely appropriate to look at the data and construct reasonable rules for data exclusion, coding, and analysis that can lead to statistical significance. In such a case, researchers need to perform only one test, but that test is conditional on the data. If data are gathered with no preconceptions at all, statistical significance can obviously be obtained even from pure noise by the simple means of repeatedly performing comparisons, excluding data in different ways, examining different interactions, controlling for different predictors, and so forth.