How A/B Tests Improve Push Notification Strategies
You may be asking yourself: Do I even need push notification messages in my marketing strategy?
Mobile push notification campaigns can be highly effective and engage your customers in a personalized way. But if you don’t know how to execute them wisely, they can also be a huge customer turnoff.
Push notification A/B testing helps you understand what resonates with your app users the most. For example, you can learn the best time of day and optimal content to create in your marketing messages—from copy to pictures or even specific emojis.
You can also evaluate how a user interacts with your message and how to tweak it to be more relevant to them. More insights result in better push campaigns and proof that they do work. Let’s walk through how to set up A/B testing to improve your push notification strategy. We’ll also outline what variables you might want to test.
How to set up the best A/B tests for your push notifications
It’s critical to understand how your push marketing campaign strategy plays into the entire experience with your product. That’s why optimizing your push campaigns through A/B testing is so important.
Determining when people are looking at or are engaged with your app because of a push informs the best time to get their attention. Do your users not open, receive, or interact with your app before work? Or do they only interact with it before work?
Asking one of these questions is the foundation for your A/B testing hypothesis. But you have to keep it simple.
The best way to approach A/B testing is to only measure one variable at a time.
Start by focusing on the most important thing you’d like to learn (e.g., timing, copy, or images).
For example, you could test the best time for sending out new messages about a Memorial Day Sale. Test sending it out the Thursday before—both at, say, 9 am and 1 pm to look at the impact
Be careful not to change the copy, images, or anything else. You have to isolate the timing change and learn what made those two pushes different.
Also, set a proper event goal that you can track and measure.
As I mentioned in an earlier post, you need to collect baseline data points through testing and experimentation. You should also limit the risk of how users will react to the tests.
Push campaign A/B testing best practices involve:
- Using a reasonable sample
- Monitoring KPIs, like opt-outs and churn
- Applying learnings and iterating for future push strategies
Your push notification management software needs to integrate with an A/B testing solution. It should also work with third-party analytics software to achieve optimal results.
Standard push notification A/B testing variables
Let’s examine some common variables you can test if you’re just getting started.
1. Content: Includes everything from copy to messaging, links, pictures and even emojis.
Does a sales message work? Do people buy more with a sale or new arrivals? With an A/B test, you can measure each of those factors
- Timing: Involves the best time of day to send push notifications—or days of the week, seasons, months and more.
Do people open your app before bed, or are they checking their phone when they wake up? Is the best time to send your message at lunch, or right after lunch when they’re getting back into work? You can test all of these factors if necessary.
- Location-based push notifications: Generally involves segmenting by country, province, state, and cities for Android and iOS users.
With geolocation-based pushes: You can test whether a 10-meter or 15-meter radius has a more significant impact when targeting customers with geofencing. For example, if a customer is near your store, or in a mall, was it more effective and will it drive them to that store
For location-based pushes or city-specific regional promotions: You might want to test different events that you can build off. For example, Pride is happening in Toronto this summer. What pushes are relevant to that city’s experience
Or, with the NBA playoffs, how can brands test relevant events happening in cities at a specific time? And what are the best marketing messages to build around it—without oversaturating the market.
- Segmenting by actions across channels: Helps you understand if a customer took a certain action in your app.
For example, you could test if sending a push after an add-to-cart event will actually increase conversions. If, say, a retailer wants to encourage customers to complete a specific action within the app, you need to get customers really engaged
Favoriting items is one of the key characteristics that predict high engagement. If a customer favorites an item and there’s a price drop, you can promote the sale through another push. It’s also an actioning hook to reward customers for favoriting more items in the future.
- Testing different segments of your population: Focus on what products your customers like to buy, and which in-app messaging works the best
For example, if they order a lot of kids clothes, for different age groups, then segmenting these customers might help with special promotions. Or, if they frequently favorite age-specific items that they’ll buy later, you can test what messaging to send to increase conversions.
- User lifecycle segments: Works for users who are new to an app. You can test different variables within three days of downloading, within one week, or a month.
If they’re lapsed users and haven’t been in the app for a while, you can test different messages for those who’ve been away for three to four months, a month, or three weeks. You can then look at how that messaging changed their behavior
Refer to our blog for examples of good app push notification campaign examples. Also, this post provides a good push notification testing strategy template you can use.
Getting internal buy-in for A/B testing push campaigns
Developing an internal push solution from scratch is a lot like building a car. You have to acquire the steering wheel, you have to attach it to the dash, and then you have to build all the other components.
It’s also an entirely different beast to develop an A/B testing solution on top of that software.
So, when you’re trying to sell the idea of buying versus building A/B testing software internally, think of all the engineering time you’d have to devote. Is it more valuable than dedicating their time to improving your own products? And do they have intimate knowledge about how to properly design and deploy tests, segment customer data, and evaluate the results?
Sometimes it’s hard to realize the complexity until you start building it. Unfortunately, it’s often too late because you’ve already committed the resources and time.
Here’s a helpful post on how to evaluate different push notification platforms and consider your buy versus build options.
Best practices: Select a variable, goal, test, and iterate
So there you have it. The best way to set up an A/B test to improve your push notification marketing strategy is to:
- Pick one variable to test and work it into your hypothesis
- Attach a specific goal, like conversions or favoriting to help you measure the impact
- Run your test with a reliable A/B testing solution
- Then learn and iterate on positive and negative (to learn) results
Getting started with A/B testing to create a good push notification message and campaign is easier with the right partner.