The Ultimate Guide to Mobile A/B Testing
47 Essential Resources for Mobile App Makers
Mobile A/B testing is the practice of running visual or code-based experiments by testing one variant against another. The more you experiment, the faster you learn and improve engagement, retention, and conversion.
While A/B testing has been around since the dawn of the web, running these tests for mobile apps was really hard for a long time. Running a simple test often took engineering resources and had to be pushed through app stores before they could be deployed live. Companies like Google and Facebook that had budgets to devote to building in-house experimentation platforms could run more tests and learn faster than the competition.
Today, there are tons of tools out there that help mobile app makers test faster and learn from data. But while running tests is easier, with nearly unlimited possibilities, it’s hard to know where to start—whether that’s testing x shades of blue for your calls-to-action or staging a feature rollout to a slice of your user base.
You can’t always predict what’s going to move the needle or what will work better for your users. Carefully testing, optimizing, and experimenting is how you can take calculated risks and move faster.
We’ve put together this guide to help mobile app makers separate the signal from the noise. We’ll keep this updated with new information—and please reach out with any resources you’d like included.
Table of Contents
- Hit the Ground Running with Mobile A/B Testing
- Build a Culture of Mobile Optimization
- Engage and Retain Users via High-Tempo Experimentation
- Mobile A/B Testing Ideas, Examples and Case Studies
- Mobile A/B Testing Tools
- Resources and People to Follow
Hit the Ground Running with Mobile A/B Testing
Getting started with A/B testing can feel impossibly daunting. Should you run binomial or multivariate tests? How long do you need to run a test to reach statistical significance? When should you start running tests at all? We’ve collected the best advice from every corner of the web to share some A/B testing tips and best practices to help you get a running start.
- A/B Testing: an Easy to Understand Actionable Beginner’s Guide: For a full 101 on A/B testing, read up on the who what where when and why.
- 7 Reasons Why You Shouldn’t A/B Test In-House: Companies like Facebook, Google, and Netflix devote massive budgets to building out customized A/B testing and analytics platforms in-house. For most companies, that’s the exact wrong approach.
- It’s Never too Early to Start Mobile A/B Testing: People often wonder when they should start running A/B tests. Here’s the upside of testing early and often.
- How Rapid Experimentation Drives Mobile ROI: By testing, you figure out ways to cost-effectively promote and refine your app experience. The more frequently you experiment, the more you learn and the more you can improve your app’s user experience for greater returns.
- How to Use A/B Testing for Better Product Design: A/B testing helps you apply data to drive product design to make better decisions. This framework helps teach you how to look to your data to solve problems, rather than provide all the answers.
- Perils and Benefits of A/B Testing: The downside of paying close attention to your product data is using data as a proxy for making decisions. Here’s how you can avoid rabid A/B testing and instead run experiments that matter.
- A/B Testing for Startups [Video]: Here’s a dead-simple, four-step process for experimentation: Dig into the Data → Hypothesis → Experiment → Act
- ABBA Calculator: This resource explains the statistics behind running successful A/B tests. Includes a helpful tool for calculating statistical significance!
Build a Culture Around Mobile Optimization
While it’s become trendy to pay lip service to principles like “test don’t guess” or “move fast and break things,” it’s much harder to effect real change within an organization around rapid experimentation and optimization—especially when it runs against the way things have been done for years. Building a culture of experimentation is hard, and it starts with the people on the front-lines.
- How to Build an A/B Testing Culture From Scratch: Building an A/B testing mindset within your company is all about empowering team members to take risks, learn, and improve. It takes time and hard work, but you’ll get there if you follow these five steps.
- How to Build an Experimentation Culture: Follow these 5 simple steps to get your team continuously building great products through iteration.
- The Agony And Ecstasy of Building With Data: A/B testing helps companies experiment and optimize, but it supplements—rather than replaces—strong decision-making. Here’s a couple of common data and A/B testing pitfalls and how to think about them.
- A/B Testing and the Benefits of an Experimentation Culture: On one level, A/B testing is about running tests that improve your product and increase revenue. On a deeper level, it’s about building a culture of calculated risk-taking and experimentation. Here’s some insight on the benefits of an experimentation culture and how to build one.
Engage and Retain Users
Engagement and retention are the twin holy grails of mobile apps. A/B testing helps you understand your users better and build a product that your customers can’t get enough of. Here are some key resources that will help you run A/B tests.
- Leaky Funnel? 5 Things You Have to A/B Test to Improve Retention: A lot of folks prioritize acquisition when it comes to mobile apps. But having millions of users pour in through the top of the funnel won’t help your business if they’re all gone six months later. These 5 A/B tests will help you optimize for retaining new users.
- A/B Testing, Increase Message Engagement: The not-so-secret is a great product. The in-app messages, push notifications, emails you send users are all levers that you can pull to nudge your users and draw them back to your app again and again. Here are some guidelines for optimizing in-app and email communications.
- Setting Up Conversion Funnels for A/B Testing Growth: Boosting key metrics like engagement and retention come from achieving a better understanding of your users and executing on that by reducing friction around a behavior pattern. Here’s a helpful way to think about mobile conversion funnels.
- What to A/B Test for the Best User Onboarding Results: Your users will only stick around if they are able to find value in your product during their first experience. This is why it’s imperative to focus on improving your user onboarding and continuously A/B test and find ways to get your users to that value quickly.
A/B Testing Ideas, Examples and Case Studies
There’s no question that you have to build a process for experimentation around the unique characteristics of your user base and app. An e-commerce app might want to allow users to browse before sign-up—but that approach might not fly for a social media app, which is built around real user interaction.To build your own testing process, though, it’s often helpful to learn from what’s worked for others. We’ve gathered some A/B testing wisdom from the most cutting-edge companies around.
- How Booking.com A/B Tests Like Nobody’s Business: At Booking.com 75 different front-end product teams experiment and iterate to scale their app to millions of users. Here’s their battle-tested approach to A/B testing.
- The Tenets of A/B Testing from Duolingo’s Master Growth Hacker: Gina Gotthilf helped grow Duolingo from 3 million to 200 million users. She shares four of her all-time favorite A/B tests.
- How Lottery.com Drives Growth with Rapid Experimentation: Lottery.com’s growth team is committed to experimentation. By running A/B tests on their mobile app’s CTAs, they increased clicks by up to 60%.
- Selecting the best artwork for videos through A/B testing: Netflix is infamous for running a constant barrage of tests to build a better product for over 100M+ loyal subscribers. Here’s a blow-by-blow of their A/B testing approach for selecting the best video artwork on Netflix.
- How A/B Testing at LinkedIn, Wealthfront and eBay Made Me a Better Manager: Instacart VP of Product, Elliot Shmukler, has worked at some of the fastest growing companies across industries from social media to e-commerce. He shares A/B testing lessons learned fighting from the trenches.
- Casey Winters on Pinterest’s Retention Wins, and Why 90% of SEO Advice is Wrong: Pinterest’s premier growth hacker, Casey Winters shares his advice on finding your north star metrics, evolving a process for growth experimentation, and jetting beyond the local maxima.
- Airlock – Facebook’s A/B Testing Framework: A deep dive into Facebook’s framework for mobile A/B testing.
- How to Experiment like Facebook and Netflix By Adopting the “10,000 Experiment Rule”: Leading companies today break new ground by experimenting quickly and often, rather than trying to slowly gain expertise around a specific industry. Companies like Google, Amazon, Netflix, and Facebook, run thousands of experiments each year.
The Ultimate Guide to Mobile A/B Testing
In today’s world of mobile apps, it’s no longer necessary to invest millions in internal tools so you can run experiments. These are some of our favorite tools for mobile testing, experimentation, and analytics that will help you get started fast.
- Taplytics – platform for mobile A/B testing, analytics, and optimization
- Segment – single platform for collecting and storing user data
- Looker – data platform
- Amplitude – product analytics
- Google Big Query – analytics data warehouse
Resources and People to Follow
Experimenting and learning is a constant process. Learning and applying those learnings constantly is how you get better at what you do. Here are some top folks in the space that you can learn from.