Confirmation Bias is Killing Your App
Your app has been out in the world for about eight months now. Everything is going great! Except for a slight drop in revenue over the past month, but that’s just because it’s winter and everyone hates winter. You know “everyone” feels this way because you see people tweeting and posting about how much they hate winter all the time.
You also found that one blog the other day about how mobile consumer spending slows down every other year in February, so you’re all good. You could look at the user data or even run some A/B tests to see if revenue is down for other reasons, but that would just be a waste of time.
Right?
This type of thinking, formally known as Confirmation Bias, is going to kill your app.
Confirmation bias is the tendency to selectively search for and consider information that confirms one’s beliefs. Often, even if you think you’re being objective about data and results, your human brain is trying to find evidence to prove what you already thought to be true.
So what does all of this have to do with your app and A/B testing? Well, if you’re not conscious of confirmation bias — in yourself and others — you could be making one of the following mistakes.
1. You’re not testing at all
As in the example above, if you think you already know the answer to an issue or trend in what your users are doing, you might not bother to test at all. You may become so convinced at your understanding of the problem that you miss the opportunity to see the real truth that only tests and data will uncover.
The only way to change this behavior is to test. List your hypotheses for the cause of the issue, then create and run A/B tests to find the answer.
2. You’re only looking at data that will reinforce your beliefs
If you are A/B testing in your app, confirmation bias can still cause problems. For example, if you happen to think that users love the “Buy Now” button (that you designed), you might test changing the color of the “Back” button or changing the main image on the page. Running these tests and looking at the resulting data might show you that your “Back” button should be purple, but you’ve done nothing to prove or disprove that your users do, indeed, love your newly-designed “Buy Now” button.
One way to combat this effect of confirmation bias is to brainstorm tests with a group — rather than have one individual decide upon and design tests. Having your hypotheses and tests accountable to your peers can greatly help to reduce the possibility of your own bias clouding test results.
Confirmation bias is one of the many psychological traps that can lie in wait in your pursuit of the perfect app. The danger of being your own “yes-man” can keep you from seeing errors and problems that are staring you right in the face.
Knowing about our human tendency to do whatever it takes to find information that agrees with our own ideas is essential to steering clear of this trap. The next step is to put failsafes in place, like A/B testing and peer accountability, to decrease the likelihood of this bias doing real damage.
Even if real data from real tests reveal something negative, wouldn’t you rather know as early as possible? If you know, you can do something about it. As G.I. Joe used to say, “Knowing is half the battle.”
Taplytics is a fully integrated mobile A/B testing, push notification, and analytics platform providing the tools you need to optimize your mobile app.
