Question

What does it mean when it says reject or fail to reject a null hypothesis?

Answer 1

When you reject a null, the treatment had an effect; when you fail to reject the null, the treatment had no effect.

Explanation:

• The null hypothesis states that a treatment had no effect in an experiment. It can be rejected if the data is very unlikely to have occurred were the null hypothesis true.
• The alternative hypothesis states that the treatment did have an effect in an experiment.
• When the null hypothesis is rejected, that means the treatment did have an effect on the experiment. Therefore, the alternative is favored over the null.
• When the null hypothesis fails to be rejected, that means the treatment had no effect in the experiment (the definition of null hypothesis.) Therefore, the null is favored over the alternative.

Also two other things to note:

• A Type I Error (“false positive”) refers to the conclusion that a difference exists when in fact this difference does not exist.
• A Type II Error (“false negative”) refers to the conclusion that there is no difference when in fact there is a difference.
  

https://www.stat.berkeley.edu/~hhuang/STAT141/Lecture-FDR.pdf

http://www.radford.edu/~tpierce/610%20files/Data%20Analysis%20for%20Professional%20Psychologists/Independent%20samples%20t-test%2010-02-09.pdf