Case Study: How Lottery.com Used Experimentation
Lottery.com empowers American lottery players to track their tickets and play on the go using the popular app. The company exhibits an exemplary experimentation culture, which is embodied by a growth team that practices high tempo testing.
In their market, it ultimately comes down to who enters first and who offers the best possible user experience. Additionally, companies must be able to work well with governments, public policy and law makers, while being proficient at building technology. Lottery.com’s ability to do both, in addition to their first mover advantage, gives them an upper hand in the game.
In order to preserve this advantage, Lottery.com works hard to understand their users’ wants and needs. This commitment is what inspired the growth team’s dedication to constant experimentation.
Get best practices for launching omnichannel experiments in our guide.
High Tempo Testing
High tempo testing is a framework used by the growth team at Lottery.com, under the lead of Travis Steffen. His team uses the frequency of new tests released as a KPI that measures their success. The working thesis is that the more tests that are run correctly, the more the company grows.
When implementing high tempo testing, Travis’ team started by releasing an average of two or three tests per week as a KPI to measure their effectiveness against. Not all of those tests were necessarily run in the digital product – the team has run rests on anything from messaging to marketing and hiring materials.
“If we’re doing our job right, we are lifting the KPIs of all departments. Because we are running so many experiments, we have to make sure that we can all function and launch tests and find new learnings without a reliance on Engineering.”
Below are examples of three experiments Lottery.com ran that exemplify their use of high tempo testing. These experiments have helped the team to make discoveries that grow the business and better engage their customers.
Experiment 1: Do people really think they have a chance of winning the lottery?
Lottery.com wanted to validate the assumption that the lottery preys on the uneducated. There’s a stigma among “Lottery disapprovers” that players don’t understand how low the probability of winning is. The growth team wanted to uncover if this was true to determine what kind of advertisement imagery would speak best to players.
First, the team did some qualitative measurement by conducting customer surveys, asking why they play and if they know the odds of winning. It turns out that people are much smarter than they’re given credit for.
The team learned that users enter the lottery with their eyes wide open and don’t play solely with the expectation of winning. They play because they have fun imagining and talking about what they would do if they won, an experience made more tangible by physically playing. Users may think “I can survive this tough week because I’ve got the winning Powerball ticket.”
Do they think they’re actually going to win? Not necessarily, but they have the satisfaction of participating, and a vision exercise that’s fun and costs only $2.
Appealing to the dreamers
With this new information in their back pocket, Lottery.com wanted to see what kind of advertisements would best speak to players with active imaginations.
Rather than enticing players with the jackpot value, the growth team experimented with ads asking people what they would do with the winnings, and used imagery that various age cohorts would associate with the good life. They found that those images didn’t just get clicks, but people actually started to have a conversation in the comments under the social ads, saying ‘this is exactly what i would do!’
By putting themselves in their customers’ shoes, the Lottery.com team learned what motivates them. Having that level of imagery in their advertising and asking the customer “what would you do?” was very powerful from a marketing perspective to spark engagement.
Experiment 2: Is the jackpot valuation correlated to the participant volume?
Lottery.com wanted to discover what jackpot level would increase the player volume. This was in an effort to help the operations team solve staffing inefficiencies and ensure that they were properly prepared for the jackpot from an operational standpoint.
The original idea was that the jackpot value drove participation, and that there was some “magic” number that customers thought was more significant. The team had previously tried to use predictive analytics, which unfortunately didn’t work. Ironically, they found the results to be extremely unpredictable because they didn’t see the spikes when they expected to.
The team knew there must be another event that was correlated with the spikes in participants.They noticed that the media was starting to cover the jackpots in news stories, which inspired a new hypothesis. Travis’ team decided to assess past jackpots with high volumes and find out when the first few articles began to disseminate around them.
Predicting the unpredictable
It was not as easy to apply these findings because the team was looking for a specific quantitative indicator of the volume spike to determine what operational changes needed to be made. As a solution, the team built a protocol to recognize when their news coverage was ramping up through volume on Google Alerts etc. and based on that, they were able to operationalize accordingly.
Experiment 3: What’s your favorite (CTA) color?
After the company’s rebrand from Autolotto to Lottery.com, most of the branded items in the app were red, including buttons. This goes against some generally accepted conversion best practices for CTAs. While this switch was important to the company’s rebranding efforts, the growth team wanted to see if re-changing the color of these branded CTAs would help them stand out by being more eye-catching and catering to emotional association and natural CTA instincts.
Using Taplytics, Lottery.com ran a multivariate test on one of the main CTA elements with several colors and shapes. They found that, overall, the branded red button was underperforming in comparison to other colors. Throughout this series of experiments, the lift in CTA taps ranged from 25-60%. Green was the highest performing color, so now most of the primary CTAs in the app are green.
“We were able to see what users were most responsive to, and what they most associate with a button based on experiences with other products versus their prior experience with our products. With Taplytics, we were able to streamline our experience with what users have been conditioned to respond to.”
Benefits of a high tempo testing environment
Each of these experiments targeted a different part of Lottery.com’s business and ability to grow. Companies who adopt an experimentation culture are more able to create both internal efficiencies as well as superior customer experiences. They are able to:
- Understand customer needs as they evolve and adjust accordingly
- Keep their entire team invested, motivated, and knowledgeable at all times
- Eliminate top-down thinking and decision making
- Keep up with a fast paced industry
“Our technology has to develop at a very fast pace to keep up with the rapid shift in organizational strategy, based on huge opportunities that come our way from national governments. Without tools that support this speed, it becomes difficult for us to offer the players any meaningful solutions.”
Drive CRO across channels with our A/B testing guide.
Thank you to Travis Steffen for sharing insights to his team’s success!
Travis Steffen is the Head of Growth at Lottery.com. As a growth engineer and data scientist, Steffen is of the mind that growth comes not from tactics or “hacking”, but from process and a sound, well-documented scientific workflow.