Was the Result Important?

Significance vs. Importance

Statistical significance is a statistical assessment of whether observations reflect a pattern rather than just chance. When used in statistics, the word significant does not mean important or meaningful, as it does in everyday speech; with sufficient data, a statistically significant result may be very small in magnitude.

If a test of significance gives a $p$-value lower than or equal to the significance level, the null hypothesis is rejected at that level . Such results are informally referred to as 'statistically significant (at the $p=0.05$ level, etc.)'. For example, if someone argues that "there's only one chance in a thousand this could have happened by coincidence", a $0.001$ level of statistical significance is being stated. Once again, this does not mean that the findings are important.

$p$-Values

A graphical depiction of the meaning of $p$-values.

So what is importance? Importance is a measure of the effects of the event. For example, we could measure two different one-cup measuring cups enough times to find that their volumes are statistically different at a significance level of $0.001$. But is this difference important? Would this slight difference make a difference in the cookies you're trying to bake? No. The difference in this case is statistically significant at a certain level, but not important.

Researchers focusing solely on whether individual test results are significant or not may miss important response patterns which individually fall under the threshold set for tests of significance. Therefore along with tests of significance, it is preferable to examine effect-size statistics, which describe how large the effect is and the uncertainty around that estimate, so that the practical importance of the effect may be gauged by the reader.