What is statistical significance? Definition and meaning

Statistical significance, or a statistically significant result, are terms commonly used by statisticians which few of us properly understand. When a statistic is significant, it means that the person is fairly sure that it is reliable. Many people erroneously believe that it means that the finding is important – it does not mean that.

Statistical significance means that the chances of getting a particular result by pure chance are extremely low.

The Economist’s glossary of terms says that statistical significance refers to a result with a 95% chance of being right, and only a ¹/₂₀ change of occurring randomly.

When the result is ‘statistically significant’, it usually means that it is at least 95% reliable. One hundred percent reliable – in the world of statistics – does not exist. (Image: adapted from slideshare.net)

StatPac.com makes the following comment on its website regarding the meaning of statistical significance:

“Many researchers get very excited when they have discovered a “statistically significant” finding, without really understanding what it means.”

“When a statistic is significant, it simply means that you are very sure that the statistic is reliable. It doesn’t mean the finding is important or that it has any decision-making utility.”

While significance in medical science – as in clinical significance – relates to the practical importance of a treatment effect, the meaning is quite different in statistics. Statistical significance, quite simply, refers to how reliable the statistical data or results are.

Statistical significance and large studies

For a data set to be statistically significant, a large study needs to be carried out.

Imagine we want to compare the average IQ (intelligence quotient) of the inhabitants of cities A and B. Three thousand people are randomly selected in city A and given an IQ test, as are 3,000 people in city B, also randomly selected.

We find that the average IQ score for city A is 101.5 and for city B is 101. We can now say:

1. “There is no significant difference between the average IQ of the people of cities A and B.” In this sentence, the word ‘significant’ means ‘important’ or ‘major’.

2. “These results are statistically significant.” In this sentence, the word ‘significant’ means ‘reliable’ – the finding is unlikely to be the result of random coincidence (a study involving 6,000 people is generally reliable).

If you are a statistician, and you are talking to non-statisticians, you should carefully explain what statistical significance means in your profession.

According to Dummies.com: “When you perform a hypothesis test in statistics, a p-value helps you determine the significance of your results. Hypothesis tests are used to test the validity of a claim that is made about a population. This claim that’s on trial, in essence, is called the null hypothesis.”

Statistical significance and chance fluctuations

While it is important to be clear on what the technical meaning of statistical significance is in the world of statistics, it is also important to be clear on what the term means practically.

We should consider the following two important factors.

– Sampling Error: there is always the possibility that the differences observed when measuring a sample population that the finding are the result of chance fluctuations, happenstance, or random noise.

– Probability but Never Certainty: statistics is all about probability. One hundred percent certainty simply does not exist. For statisticians, their work is about managing risk. What degree of uncertainty are we willing to or can we live with – 5%, 10%, 20%, 30%?

This depends on the context. What does it cost to raise the probability of making the correct choice, and what are the consequences or potential consequences of making a bad choice?