Back to all posts Back

The Ultimate Guide to Mobile A/B Testing

Guide to ab testing - customer experience

47 Essential Resources for Mobile App Makers

Mobile A/B testing is the practice of running visual or code-based experiments by testing one variant against another. The more you experiment, the faster you learn and improve engagement, retention, and conversion.

While A/B testing has been around since the dawn of the web, running these tests for mobile apps was really hard for a long time. Running a simple test often took engineering resources and had to be pushed through app stores before they could be deployed live. Companies like Google and Facebook that had budgets to devote to building in-house experimentation platforms could run more tests and learn faster than the competition.

Today, there are tons of tools out there that help mobile app makers test faster and learn from data. But while running tests is easier, with nearly unlimited possibilities, it’s hard to know where to start—whether that’s testing x shades of blue for your calls-to-action or staging a feature rollout to a slice of your user base.

You can’t always predict what’s going to move the needle or what will work better for your users. Carefully testing, optimizing, and experimenting is how you can take calculated risks and move faster.

We’ve put together this guide to help mobile app makers separate the signal from the noise. We’ll keep this updated with new information—and please reach out with any resources you’d like included.

Learn mobile experimentation best practices to drive conversions, engagement, and retention in our free Mobile Experimentation Guide

Table of Contents


Hit the Ground Running with Mobile A/B Testing

Mobile AB Testing

Getting started with A/B testing can feel impossibly daunting. Should you run binomial or multivariate tests? How long do you need to run a test to reach statistical significance? When should you start running tests at all? We’ve collected the best advice from every corner of the web to share some A/B testing tips and best practices to help you get a running start.

Build a Culture Around Mobile Optimization

While it’s become trendy to pay lip service to principles like “test don’t guess” or “move fast and break things,” it’s much harder to effect real change within an organization around rapid experimentation and optimization—especially when it runs against the way things have been done for years. Building a culture of experimentation is hard, and it starts with the people on the front-lines.

  • How to Build an A/B Testing Culture From Scratch: Building an A/B testing mindset within your company is all about empowering team members to take risks, learn, and improve. It takes time and hard work, but you’ll get there if you follow these five steps.
  • How to Build an Experimentation Culture: Follow these 5 simple steps to get your team continuously building great products through iteration.
  • The Agony And Ecstasy of Building With Data: A/B testing helps companies experiment and optimize, but it supplements—rather than replaces—strong decision-making. Here’s a couple of common data and A/B testing pitfalls and how to think about them.
  • A/B Testing and the Benefits of an Experimentation Culture: On one level, A/B testing is about running tests that improve your product and increase revenue. On a deeper level, it’s about building a culture of calculated risk-taking and experimentation. Here’s some insight on the benefits of an experimentation culture and how to build one.

 Engage and Retain Users

Engage and Retain Users

Engagement and retention are the twin holy grails of mobile apps. A/B testing helps you understand your users better and build a product that your customers can’t get enough of. Here are some key resources that will help you run A/B tests.

  • Leaky Funnel? 5 Things You Have to A/B Test to Improve Retention: A lot of folks prioritize acquisition when it comes to mobile apps. But having millions of users pour in through the top of the funnel won’t help your business if they’re all gone six months later. These 5 A/B tests will help you optimize for retaining new users.
  • A/B Testing, Increase Message Engagement: The not-so-secret is a great product. The in-app messages, push notifications, emails you send users are all levers that you can pull to nudge your users and draw them back to your app again and again. Here are some guidelines for optimizing in-app and email communications.
  • Setting Up Conversion Funnels for A/B Testing Growth: Boosting key metrics like engagement and retention come from achieving a better understanding of your users and executing on that by reducing friction around a behavior pattern. Here’s a helpful way to think about mobile conversion funnels.
  • What to A/B Test for the Best User Onboarding Results: Your users will only stick around if they are able to find value in your product during their first experience. This is why it’s imperative to focus on improving your user onboarding and continuously A/B test and find ways to get your users to that value quickly.

 A/B Testing Ideas, Examples and Case Studies

AB Testing Examples

There’s no question that you have to build a process for experimentation around the unique characteristics of your user base and app. An e-commerce app might want to allow users to browse before sign-up—but that approach might not fly for a social media app, which is built around real user interaction.To build your own testing process, though, it’s often helpful to learn from what’s worked for others. We’ve gathered some A/B testing wisdom from the most cutting-edge companies around.

The Ultimate Guide to Mobile A/B Testing

In today’s world of mobile apps, it’s no longer necessary to invest millions in internal tools so you can run experiments. These are some of our favorite tools for mobile testing, experimentation, and analytics that will help you get started fast.

Resources and People to Follow

Experimenting and learning is a constant process. Learning and applying those learnings constantly is how you get better at what you do. Here are some top folks in the space that you can learn from.

Andrewchen.co
by Andrew Chen
@andrewchen

ben-evans.com
by Benedict Evans
@BenedictEvans

Black Box of Product Management
by Brandon Chu
@BrandonMChu

Coelevate
by Brian Balfour
@bbalfour

Casey Accidental
by Casey Winters
@onecaseman

lukew.com
by Luke Wroblewski
@lukew

Noteworthy
by Sam DeBrule
@SamDeBrule

Product Manager HQ
by Kevin Lee
@kevinleeme

Product Habits
by Hiten Shah
@hnshah

The Looking Glass
by Julie Zhuo
@joulee