There’s more than one way to be wrong. It’s vital to understand whether you are wrong because you rejected the null hypothesis when it is actually true. This is called a type I error or a false positive: you can think of it as being an alarm without a fire. Or whether you are wrong because you failed to reject it when the alternative hypothesis is true. This is called a type II error or a false negative: a fire without an alarm.
Anyone who ever burns their toast knows how to avoid a type I error: just remove the batteries in the smoke alarm. Of course this increases the incidence of type II errors. The opposite is true: by making the alarm hypersensitive, you will get more false alarms.
These two types of errors are connected mathematically. A more rigorous threshold, say a significance level of 0.001 instead of 0.05, increases the likelihood of failing to reject the null hypothesis when it ought to be. If our burden of proof is too low we will sometimes reject the null hypothesis when it is true.
If 0.05 seems arbitrary, that’s because it is. You could choose 0.1 or 0.01, it depends on how much statistical heft you want to bring to your decision. Remember that a 0.01 significance level means that there is less than a 1 in a 100 chance of observing a result in the range if the null hypothesis were true.
It is important to note that a p-value only indicates the likelihood of obtaining the results by chance and does not provide evidence in favor of the alternative hypothesis. Additionally, a low p-value does not necessarily mean that the results are practically significant or important. The p-value should be used in conjunction with other measures, such as effect size and confidence intervals, to determine the practical significance of the results.
When using statistical analysis to support decision-making, it’s vital to pre-determine what your tolerance is to error. Space rocket engineers can’t afford to make mistakes. But most business decisions don’t need to be so precise. In fact, if you were correct four times out of 5 (a confidence level of 80%) you’d probably be pretty happy!
