How Duolingo A/B tested their push notifications to retain users

Duolingo wanted to empower their users with the knowledge of learning new languages, but this required them to log in and engage with the app regularly in order to learn the language they signed up for. In order to get any value and learn a new language, users would need to build a habit of using the app on a regular basis in order to reap the benefits of their platform. So how would they engrain loyalty, engagement in order to build the habit of using the app? They leaned towards gamification.

Enter: Duolingo’s Streaks feature

They looked outside of the education industry, towards gamification as a way to make the process of learning “fun” to combat abysmal retention rates. With the initial rush of learning a few lessons, the novelty of the app would wear off.

The Experiment:

In order to set up the experiment, they would need to develop and set up the feature as an experiment itself, but also how users are notified to bring them back to the app creating a tight feedback loop for the users. Here they had to think really deeply about how push notifications would be a part of their gamification strategy outside of the app to pull users back in.

They experiment on the copy of the notifications in a meaningful way that wasn’t spammy.
“Now we actually have a reason to tell you to come back. It’s not just ‘Come back to Duolingo,’ which is super spammy and annoying. We can say, ‘You have a goal. Reach your goal today.’ Or ‘Time to do your daily lesson.’ That kind of prompt is good. You can come from a place of helping the user maintain her own goal.”

The result: This led to a 5% increase in Daily Active Users

Why we think this is a good A/B test:

Being mindful about your push notification strategy alongside your product development should be a net benefit and educates the user about being notified about their progress, reinforcing that this is something that they want and care about. Ensuring you nail the copy in a succinct fashion that captures the users attention and communicates the context with images and copy will help make for an app that your users will come back to time and time again.