Dichotomous Thinking

Dichotomous thinking, or binary thinking, in statistics is the tendency to treat a p-value during null hypothesis significance testing as falling into one of only two discontinuous categories, above or below a fixed significance threshold such as 0.05, rather than as a continuous measure. Under this pattern a p-value of 0.0499 is interpreted the same way as a p-value of 0.0001, as rejecting the null hypothesis, while a p-value of 0.0501 is interpreted the same way as a p-value of 0.7, as accepting it, even though the first pair and the second pair of values are mathematically very close to each other. A commonly used measure of dichotomous thinking is called the cliff effect, and the pattern is considered a reasoning risk because p-values and other statistics naturally vary from study to study through random variation alone, making a firm decision about a hypothesis based on a single study's result unreliable; the same dichotomous pattern can also occur with other statistical tools, such as interval estimates, not only with p-values.