Does the p-value overestimate the strength of evidence?

Thom Baguley points to the standardized or minimum LR (p381) to answer this question.

The minimum LR represents a worst case scenario for the null in that it compares the LR for $H_0$ against the MLE of the observed data, i.e. the most likely (strongest) possible hypothesis supported by the data, and is defined as

$LR_{min} = \frac{l(H_0|D)}{l(\hat\theta)} = \frac{l(\theta_0)}{l(\hat\theta)}$

For samples drawn from a normal distribution with a known variance, this reduces to

$LR_{min} = e^{-z^2/2}$

At the limit of “significance” when $\alpha$ = .05 and p=.05, this implies a z score of 1.96 in case of a normal distribution. The worst case scenario for H0 (the LR in favor of the null) is

$LR_{H_0}=e^{-1.96^2/2}$

or about 0.147. The LR in favor of $H_1$ is $1/0.147$ or about 6.83 at the maximum. The probability of $H_0$ being true is then

$P_{H_0}=\frac{1}{1+6.83}=0.128$

or about 13%, which is much higher than usually, and naively, assumed.

Does the p-value overestimate the strength of evidence?

1 Reply to “Does the p-value overestimate the strength of evidence?”

Leave a Reply Cancel reply