Question

What is statistical significance?

Answer 1

A result is said to be statistically significant if the likelihood from pure coincidence is below a certain level.

Say you filp a coin. The zero-hypothesis says that it's a fair coin and it should show up heads in 50% of the flips.

You flip 10 times and come out with 6H 4T. Is that statistically significant? Not very likely.

But now you flip the same coin 1000 times and still get the same ratio between 600H and 400T. You would be sure the coin is unfair.

Somewhere in between lies the border of what is called significance. There are a lot of tests that can be performed. In the case of the coin a so-called Binomial test would be done, to calculate the probability of this happening.

Usually the so-called `p<0.05` criterion is used, which in plain English means that the chance of this happening by pure coincidence is below `5%` or below `1:20`.
But in other circumstances (e.g. side-effects of medicines being tested) much stricter criteria may be used.