Does the p-value overestimate the strength of evidence?

Thom Baguley points to the standardized or minimum LR (p381) to answer this question.

The minimum LR represents a worst case scenario for the null in that it compares the LR for H_0 against the MLE of the observed data, i.e. the most likely (strongest) possible hypothesis supported by the data, and is defined as

    \[LR_{min} = \frac{l(H_0|D)}{l(\hat\theta)} = \frac{l(\theta_0)}{l(\hat\theta)}\]

For samples drawn from a normal distribution with a known variance, this reduces to

    \[LR_{min} = e^{-z^2/2}\]

At the limit of “significance” when \alpha = .05 and p=.05, this implies a z score of 1.96 in case of a normal distribution. The worst case scenario for H0 (the LR in favor of the null) is


or about 0.147. The LR in favor of H_1 is 1/0.147 or about 6.83 at the maximum. The probability of H_0 being true is then


or about 13%, which is much higher than usually, and naively, assumed.

1 Reply to “Does the p-value overestimate the strength of evidence?”

Leave a Reply