How A/B tests results increase user experience and retention

You may be asking yourself: Do I even need push notification messages in my marketing strategy?

Mobile push notification campaigns can be highly effective and engage your customers in a personalized way. But if you don’t know how to execute them wisely, they can also be a huge customer turnoff.

Getting users to engage and re-engage with mobile apps is a challenge. One tactic to reach users who are online is push notifications. They can help you to reach your audience and quickly and easily communicate with them. Getting the attention of your customers comes with several benefits as you can communicate with them directly.

Push notification A/B testing helps you understand what resonates with your app users the most. For example, you can learn the best time of day and optimal content to create in your marketing messages—from copy to pictures or even specific emojis.

You can also evaluate how a user interacts with your message and how to tweak it to be more relevant to them. More insights result in better push campaigns and proof that they do work. Let’s walk through how to set up software A/B testing to improve your push notification strategy. We’ll also outline what variables you might want to test.

How to set up the best testing tools for your push notifications

It’s critical to understand how your push marketing campaign strategy plays into the entire experience with your product. That’s why optimizing your push campaigns through A/B testing is so important.

Determining when people are looking at or are engaged with your app because of a push informs the best time to get their attention. Do your users not open, receive, or interact with your app before work? Or do they only interact with it before work?

Asking one of these questions is the foundation for your A/B testing hypothesis. But you have to keep it simple.

The best way to approach A/B testing is to only measure one variable at a time.

Start by focusing on the most important thing you’d like to learn (e.g., timing, copy, or images).

For example, you could test the best time for sending out new messages about a Memorial Day Sale. Test sending it out the Thursday before—both at, say, 9 am and 1 pm to look at the impact

Be careful not to change the copy, images, or anything else. You have to isolate the timing change and learn what made those two pushes different.

Also, set a proper event goal that you can track and measure.

As I mentioned in an earlier post, you need to collect baseline data points through testing and experimentation. You should also limit the risk of how users will react to the tests.

Push campaign A/B testing best practices involve:

• Using a reasonable sample
• Monitoring KPIs, like opt-outs and churn
• Applying learnings and iterating for future push strategies

Your push notification management software needs to integrate with an A/B testing solution. It should also work with third-party analytics software to achieve optimal results.

Standard push notification A/B testing variables

Let’s examine some common variables you can test if you’re just getting started.

1. Content: This includes everything from copy to messaging, links, pictures, and even emojis.

Does a sales message work? Do people buy more with a sale or new arrivals? With an A/B test, you can measure each of those factors to find the content that resonates best with your audience.

2. Timing: Discover the best time of day to send push notifications—or days of the week, seasons, months, and more.

Do people open your app before bed, or are they checking their phone when they wake up? Is the best time to send your message at lunch, or right after lunch when they’re getting back into work? You can test all of these factors to find out when engagement rates with your app are the highest.

3. Location-based push notifications: Generally involves segmenting by country, province, state, and cities for Android and iOS users.

With geolocation-based pushes: You can test whether a 10-meter or 15-meter radius has a more significant impact when targeting customers with geofencing. Let’s say a  customer is near your store, or in a mall. You can see which variation was more effective and drove them into the store.

For location-based pushes or city-specific regional promotions: You might want to test different events that you can build off. For instance, Pride is happening in Toronto this summer. You can think of pushes notifications are relevant to that city’s experience

Or, with the NBA playoffs, how can brands test relevant events happening in cities at a specific time? And what are the best marketing messages to build around it—without oversaturating the market.

4. Segmenting by actions across channels: Helps you understand if a customer took a certain action in your app.

You could test if sending a push after an add-to-cart event will actually increase conversions. If, say, a retailer wants to encourage customers to complete a specific action within the app, you need to get customers really engaged.

Favoriting items is one of the key characteristics that predict high engagement. If a customer favorites an item and there’s a price drop, you can promote the sale through another push. It’s also an actioning hook to reward customers for favoriting more items in the future. This gives them an incentive to come back to your app to take action and make a purchase.

5. Testing different segments of your population: Focus on what products your customers like to buy, and which in-app messaging works the best

For example, if they order a lot of kids’ clothes, for different age groups, then segmenting these customers might help with special promotions. Or, if they frequently favorite age-specific items that they’ll buy later, you can test what messaging to send to increase conversions.

6. User lifecycle segments: Works for users who are new to an app. You can test different variables within three days of downloading, within one week, or a month.

If they’re lapsed users and haven’t been in the app for a while, you can test different messages for those who’ve been away for three to four months, a month, or three weeks. You can then look at how that messaging changed their behavior

Refer to our blog for examples of good app push notification campaign examples. Also, this post provides a good push notification testing strategy template you can use.

Getting internal buy-in for A/B testing push campaigns

Developing an internal push solution from scratch is a lot like building a car. You have to acquire the steering wheel, you have to attach it to the dash, and then you have to build all the other components.

It’s also an entirely different beast to develop an A/B testing solution on top of that software.

So, when you’re trying to sell the idea of buying versus building A/B testing software internally, think of all the engineering time you’d have to devote. Is it more valuable than dedicating their time to improving your own products? And do they have intimate knowledge about how to properly design and deploy tests, segment customer data, and evaluate the results?

Sometimes it’s hard to realize the complexity until you start building it. Unfortunately, it’s often too late because you’ve already committed the resources and time.

Here’s a helpful post on how to evaluate different push notification platforms and consider your buy versus build options.

Best practices: Select a variable, goal, test, and iterate

So there you have it. The best way to set up an A/B test to improve your push notification marketing strategy is to:

• Pick one variable to test and work it into your hypothesis
• Attach a specific goal, like conversions or favoriting to help you measure the impact
• Run your test with a reliable A/B testing solution
• Then learn and iterate on positive and negative (to learn) results

Getting started with A/B testing to create a good push notification message and campaign is easier with the right partner.