Type 1 Error

Statistics and hypothesis testing are not perfect. There's always a chance of drawing an incorrect conclusion such as a type 1 error (false positive) or type 2 error (false negative).

Statistical significance has traditionally been calculated with assumptions. Particularly, a decision is deemed valid when the sample size in question has been reached. This is known as a ‘fixed horizon’ calculation. In order to assess the validity of a hypothesis, it must be tested by looking at the data over time and not just on a single date.

Of course, it’s not the way things work in the A/B testing world. Statistics can be misused in an A/B test if there are no controls (a pre-determined sample size), results are not statistically significant, or sample sizes are too small.

Why You Should Look Out For Type 1 Errors

It is crucial to be wary of type 1 errors in your marketing strategy. If you make the wrong assumption, then you could end up costing your company a lot of money and having an impact on the overall success of your company.

The best way to prevent type 1 errors is to confirm your hypothesis and experiments for longer periods of time to get more data and increase your confidence threshold.

The #1 platform for delivering high-quality software releases, instantly.