When we work with early stage start-ups, one of the most frequent questions we get asked is if it’s too early to start thinking about A/B testing. Before starting Taplytics we were app developers, so we know all about the confusion you must be feeling. You have probably heard countless times that A/B testing is key to driving success but in the back of your mind, you know you don’t yet have the users to be able to reach significance. We wanted to share a few key reasons you should absolutely start A/B testing now and how to interpret your data when you don’t reach significance.
Validate Your Assumptions
Trying to gain traction is one of the toughest parts when you’re just starting out and we’re sure you have had many discussions trying to figure out why you may not be hitting your numbers. A/B testing for a young app, or any app for that matter, allows you to be able to test completely different positioning and messaging of the app. When you’re designing and building your app, you make a lot of guesses along the way.
There are other methods to try to determine if a new feature will be a hit such as user interviews and mock prototypes. The struggle with these is that a user may think they will use a product one way but once it’s live in their hands, it might not be the case. The guesses you’re making are a good start, but having data to understand how your users are interacting with your app can help you understand what’s working and what isn’t.
What if you could try out a few completely different versions of your app at the same time? What if you could run completely different messaging about what the app can do, who it’s for and why they should care? What if you could try out a completely different set of features and see what version results in more users not only trying it but also sticking around?
All of this is possible with mobile A/B testing
You can use visual editing to modify your copy, your images, and even your user interface. We’ve had some users completely re-skin their app. If you need to create your experiment with new code, maybe to try out different interface structures, recommendation algorithms or turn on/off features of the app, code-based experiments are a great way to achieve this. Of course, you can also use your A/B testing results to measure the impact of the changes in your key metrics.
Now that you have these great reasons to A/B test you are probably still stuck on the fact that your experiments will never reach significance. So how can you make a decision from your data?
Look for Trends
Sometimes it’s ok to ignore the statistical significance of your experiments. If you have a new app with only a few hundred downloads, you’re never actually going to hit full statical significance in any reasonable timeframe. Even if you haven’t reached significance, making decisions based on trends in your data is much better than changing features based on your gut feel. When you first launch an A/B experiment, your data will be very chaotic but should eventually settle. Once it does settle, you can start to look for trends to help you make a decision. As shown below this is a prime example of a time you can use trends in your data to make decisions.
What you want to be looking for is a flattening of the conversion lines into trends. Once you’ve seen it stabilize as demonstrated above, you can start inferring a trend out of the data. As shown above, there’s a clear difference between the two variations and you should feel confident to which variation you should be moving forward with.
You want to be careful, however, of results that are too similar or still changing, making it less clear which variation is going to win as shown below.
Keep track of the key goals you’ve set for your experiment and pay attention to wide divergences in conversion rates. These can be considered trends in how your app is performing and can give you enough of an indication if the interface is working as expected for your users.
Test Early, Test Often
It’s never too early for mobile A/B testing, even at early stages of your app’s growth. It can give you a competitive advantage over other apps in your category.
Knowing what your users are doing and how they react to your experiments can be valuable when trying to figure out which features to keep and what to change. Having the flexibility to make changes on the fly, without any app review delays can be the difference between a successful launch or losing your marketing momentum due to a small bug. Even if you are not yet reaching significance, A/B testing has loads of benefits for your young app. There’s no excuse not to be A/B testing from day one!
Originally posted on the Taplytics blog. Taplytics is a fully integrated mobile A/B testing, push notification, and analytics platform providing the tools you need to optimize your mobile app.