A surge of p-values between 0.041 and 0.049 in recent decades (but negative results are increasing rapidly too)

被引:46
作者
de Winter, Joost C. F. [1 ]
Dodou, Dimitra [1 ]
机构
[1] Delft Univ Technol, Dept BioMech Engn, Delft, Netherlands
来源
PEERJ | 2015年 / 3卷
关键词
Significant differences; Bias; Physical sciences; Biological sciences; Social sciences; Science policy; SCIENCE-WISE FALSE; RANDOMIZED CONTROLLED-TRIALS; OVERESTIMATE EFFECT SIZES; PUBLICATION BIAS; DISCOVERY RATE; GENETIC ASSOCIATIONS; ERROR; TESTS; REPLICABILITY; CONSEQUENCES;
D O I
10.7717/peerj.733
中图分类号
O [数理科学和化学]; P [天文学、地球科学]; Q [生物科学]; N [自然科学总论];
学科分类号
07 ; 0710 ; 09 ;
摘要
It is known that statistically significant (positive) results are more likely to be published than non-significant (negative) results. However, it has been unclear whether any increasing prevalence of positive results is stronger in the "softer" disciplines (social sciences) than in the "harder" disciplines (physical sciences), and whether the prevalence of negative results is decreasing over time. Using Scopus, we searched the abstracts of papers published between 1990 and 2013, and measured longitudinal trends of multiple expressions of positive versus negative results, including p-values between 0.041 and 0.049 versus p-values between 0.051 and 0.059, textual reporting of "significant difference" versus "no significant difference," and the reporting of p < 0.05 versus p > 0.05. We found no support for a "hierarchy of sciences" with physical sciences at the top and social sciences at the bottom. However, we found large differences in reporting practices between disciplines, with p-values between 0.041 and 0.049 over 1990-2013 being 65.7 times more prevalent in the biological sciences than in the physical sciences. The p-values near the significance threshold of 0.05 on either side have both increased but with those p-values between 0.041 and 0.049 having increased to a greater extent (2013-to-1990 ratio of the percentage of papers = 10.3) than those between 0.051 and 0.059 (ratio = 3.6). Contradictorily, p < 0.05 has increased more slowly than p > 0.05 (ratios= 1.4 and 4.8, respectively), while the use of "significant difference" has shown only a modest increase compared to "no significant difference" (ratios = 1.5 and 1.1, respectively). We also compared reporting of significance in the United States, Asia, and Europe and found that the results are too inconsistent to draw conclusions on cross-cultural differences in significance reporting. We argue that the observed longitudinal trends are caused by negative factors, such as an increase of questionable research practices, but also by positive factors, such as an increase of quantitative research and structured reporting.
引用
收藏
页数:44
相关论文
共 102 条
[1]  
[Anonymous], 1964, Canadian Psychology, DOI DOI 10.1037/H0083036
[2]  
[Anonymous], 2013, ARXIV13101284
[3]  
[Anonymous], COMMUNICATION
[4]   Recommendations for Increasing Replicability in Psychology [J].
Asendorpf, Jens B. ;
Conner, Mark ;
De Fruyt, Filip ;
De Houwer, Jan ;
Denissen, Jaap J. A. ;
Fiedler, Klaus ;
Fiedler, Susann ;
Funder, David C. ;
Kliegl, Reinhold ;
Nosek, Brian A. ;
Perugini, Marco ;
Roberts, Brent W. ;
Schmitt, Manfred ;
vanAken, Marcel A. G. ;
Weber, Hannelore ;
Wicherts, Jelte M. .
EUROPEAN JOURNAL OF PERSONALITY, 2013, 27 (02) :108-119
[5]   A paradigm shift in the medical literature [J].
Atkin, PA .
BRITISH MEDICAL JOURNAL, 2002, 325 (7378) :1450-1451
[6]   Outlier Removal, Sum Scores, and the Inflation of the Type I Error Rate in Independent Samples t Tests: The Power of Alternatives and Recommendations [J].
Bakker, Marjan ;
Wicherts, Jelte M. .
PSYCHOLOGICAL METHODS, 2014, 19 (03) :409-427
[7]  
Basu S., 2014, PUBLICATION BIAS REC
[8]   Discussion: An estimate of the science-wise false discovery rate and applications to top medical journals by Jager and Leek [J].
Benjamini, Yoav ;
Hechtlinger, Yotam .
BIOSTATISTICS, 2014, 15 (01) :13-16
[9]  
Binfield P., 2009, RETHINKING ELECT PUB, P69
[10]   Reporting and Interpretation of Randomized Controlled Trials With Statistically Nonsignificant Results for Primary Outcomes [J].
Boutron, Isabelle ;
Dutton, Susan ;
Ravaud, Philippe ;
Altman, Douglas G. .
JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION, 2010, 303 (20) :2058-2064