I recently noticed an active discussion on the importance of statistical measures in a Business Intelligence group on XING. What a welcome opportunity to pose the question: Why do we have such faith in significance?

In newspapers we often read that industry orders have dropped significantly, that the stock market volume fell significantly, that a certain company wants to increase profits significantly. In addition, vaccinated people have a slightly – yet not significantly – lower risk of catching a certain illness than those who are not, and a group of 3,000 radiation victims had a significantly lower IQ than that of the comparison group.

Significant signalizes: that’s the way it is. Statistically significant is even stronger: without a doubt because it was statistically proven. Does significance deserve such an outstanding reputation? Let start at the very beginning and explain what significance is and what it isn’t. The following test will illustrate this point. Take it yourself and at the end you can compare your results with those of students and professors.

Test: What do you know about significance?
A 2002 study tested a means for increasing individual performance. 20 people from six universities were placed in a test and control group. After the experiment, both groups were compared based on the average number of points achieved in the assignment.

The null hypothesis (H0) stated: The new method is not effective because there is no difference between the average performance of people in both groups. The result of the t test (t=2.7, d.f.=18) is a p value of 0.01.

Please mark each of the statements below as “true” or “false”. “False” means that the statement does not follow logically from the above premises: