The t-Test p Value and Its Relationship to the Effect Size and P(X > Y)

The p value from the two-sample t-test has been disparaged for its apparent inability to provide a measure for the magnitude of the mean effect. However, by knowing the p value and the sample size, a relationship is shown with Cohen's effect size and exact confidence limits on the effect size. These confidence limits can be used to create plots of the normal densities consonant with the data. Exact confidence limits can also be easily derived for P(X > Y) and the odds of X being greater than Y. These latter two statistics can help us express t-test results in terms of differences between individuals randomly chosen from the two populations, rather than in terms of differences in population means. The limitations of the commonly reported "p < 0.05" and "p < 0.0001" are discussed.