Thom Baguley points to the standardized or minimum LR (p381) to answer this question.
The minimum LR represents a worst case scenario for the null in that it compares the LR for against the MLE of the observed data, i.e. the most likely (strongest) possible hypothesis supported by the data, and is defined as
For samples drawn from a normal distribution with a known variance, this reduces to
At the limit of “significance” when = .05 and p=.05, this implies a z score of 1.96 in case of a normal distribution. The worst case scenario for H0 (the LR in favor of the null) is
or about 0.147. The LR in favor of is or about 6.83 at the maximum. The probability of being true is then
or about 13%, which is much higher than usually, and naively, assumed.
1 Reply to “Does the p-value overestimate the strength of evidence?”