Use Analytics and A/B Tests to Supercharge Your Growth
Digital leaders from Taplytics, Shopify, Equinox, and Talkspace share insights on how to uncover what’s working and pinpoint where to double down to exceed customer expectations and keep pace with the competition.
Or, you can read the transcript below:
The session theme is accelerating growth through A/B testing and we’ll do just a quick round of introductions. This is going to be a pretty informal panel discussion, but I’m moderating, I’m Cobi Druxerman. I’m a co-founder and head of product at Taplytics and really excited to have this discussion.
I know we’re obviously very interested in A/B testing given our business but we have some great speakers for you today so I’ll let them introduce themselves.
Hi everyone, my name is Farhan. I’ve been in large companies and small companies. I spent a lot of time in mobile engineering, and so this is a topic near and dear to my heart. I’m currently VP Engineering at Shopify.
My name is Josh. I work at a company called Talkspace its online therapy and joined Talkspace in the fall when a startup that I’ve been working on called Lasting a couples counseling app was acquired by Talkspace. So I’m mainly bringing a perspective of small companies or startups to our A/B testing conversation.
Hi everyone. I’m Matt I work for Equinox on the digital side of their business, leading growth and retention marketing, overseeing the A/B testing program that we have there as well. Spent most of my career on the mobile marketing side, started to get more into the CPG and physical product side over the last couple of years and excited to be joining this panel.
One of the most interesting things about this panel is the varied experiences, different types of companies, size of company, and we definitely find at Taplytics that all companies, even the ones with like really strong experimentation culture and programs, think about it in various ways.
I really want to kick this off and start a discussion on the concept of testing and prioritization, something we’ve talked about a bunch. With an endless number of options of what to test how do you all think about how to determine test prioritization within your companies, within your teams?
It really comes down to a framework where anybody can suggest a test. We believe in a culture that anybody can have a good idea. Anybody can be a critical thinker about how to improve a sales funnel, how to improve a section of the app that we’re on, that will become more engaging, but then it comes down to prioritization from there and that comes down to the experts that are working on those individual functions. Coming with the data-informed POV on what the impact of that test could potentially bring them.
If you’re talking about a small change that maybe is going to drive more people to engage with a certain type of class content versus something that’s going to increase the overall amount of engagement on the app. If they both take about the same amount of time to bring to market. It’s really taking a look at that impact versus scope equation and getting to a net impact score to prioritize that list. It’s something we do on a weekly basis, and we try to at least get one to two tests out every single month that are combed through a list that’s several thousand tests long at this point.
At Shopify, we have an interesting approach because for most companies, the metrics we want to drive actually go in the opposite direction than what most companies want. For example, we want people to be in the Shopify admin for less time because we want them to get back to running their business.
We actually like to see high churn, because what happens is entrepreneurs tend to try multiple businesses before they get one. So we actually want them to come in and try something. It might fail, then try again and again. If you looked at the metrics and just tried to optimize for low churn you might not end up having the type of person who would try again and try again, because we want to make sure it’s easy to set up and easy to get going and easy to try again.
In some cases, we want to experiment with things that might on the surface look like they go in the opposite direction than what you would expect, but at the same time, that means that we want to be data-informed versus data-driven because if you get too into the weeds, you might lose the story.
I think that’s really interesting too, picking the right OKR to optimize towards. One might think more time on the website could be an interesting thing to optimize, but maybe you’re actually creating a more confusing user experience and you’re actually not driving towards the key OKR that you care about.
I think that OKR combing is super important there as well. So it’s interesting to hear how you guys kind of think a little bit opposite of how many direct-to-consumer businesses would be typically thinking there.
I think about prioritization partly from a workflow folk flow perspective, very practically.
How do you get a balanced portfolio of quick editing wins and larger bets that may or may not pay off in the way we expect. I also love to think about personalization from the lens of which test would teach us the most. If I’m having trouble prioritizing w e all do the impact effort graph, but I try to add another dimension when I feel stuck is which test, if we run would dramatically, change our view of a problem.
Understanding of who the user is or what they’re trying to solve here. So that’s the third angle I’ve been hearing.
Does anyone of this group like to optimize for quantity? I was actually hearing from one of our clients not too long ago that one of the things that they were trying to optimize for was actually a higher degree of quantity because they felt like if they were always trying to have a high impact and they weren’t actually seeing an equal number of tests that were having a negative impact, they felt like they weren’t doing enough.
We have this term, we just call it green pathing. We want to make sure it’s easy to do. We don’t quantify, we don’t look at ourselves and say there should always be two or 300 beta flags live at any point or A/B tests running or experiments running. We want to make sure that the things we want to have happen culturally are easy to do.
If you think testing and automated testing is good, then we make that easy to do. If we think experiments is a good thing to do, we make them easy to do. We make them think we make it easy to turn them on and off segment people. We try to approach it from a systems perspective and we don’t try to do that many comms around it. We don’t try to say you should do more testing instead, we just use the tooling to make it super easy and then people are like, wait a sec. This is pretty easy to do, I’m going to do this more.
I think on the idea of not necessarily dictating a certain amount of tests, going a little bit deeper on that, we just recently redesigned our whole website and we decided to launch the whole website as an A/B test. We had a lot of discussions on do we slowly shift the website over by testing each module? Testing each component and saying, is this actually driving incremental? But at the end of the day, we said this new website was built as a cohesive story.
Sometimes testing each individual thing might not give you the same picture as testing the whole thing at once. So every once in a while, it does make sense to take that big swing and say, let’s forego the 10 tests that we would run over the next two months to get us here and just run one big one.
See what learnings we can gather from there, and then start optimizing within that new experience as well.
I think that point actually brings me into the next thing that I wanted to talk about. You’re bringing up this concept of when you think of that redesign, it’s a shift in product.
I think one of the interesting points that we see and that I think everyone sees is that great products always grow mature and adapt to their customers or to the market in general. I think we can all say that over the past year, there’s been a lot of discussions about pandemics, about everything that’s happened.
I think one of the more interesting aspects is that it has forced a big shift and a lot of product lines and a lot of products. I know Equinox has been obviously highly impacted. Everyone here has had to adapt in some way. Within that how have you used experimentation strategies to guide that shift?
With our products, we have two key products, one being our Equinox plus app, which is a combination of all the best content you can get from Equinox, plus some third-party partners and a single digital platform. Then we have our true Peloton competitor on the bike side with the at home bike.
The people who are coming to our website primarily are people who are already primed from those two audiences that exist. And they’re used to those experiences being very physically oriented. The way that you sell a physical product and a physical experience is very different than a digital product or an at-home experience.
I think when the web experiences and the sales funnels were initially built, they were built very much in the mind of an extension of physical businesses. Learning how to shift that towards things that are more traditionally found in like a CPG or digital lens has been a journey really over the last year and a half.
The new website that we just launched today was just literally saying, okay, what are the best in class digital-first businesses doing and comparing that to what the best class physical businesses are doing and looking at where those gaps exist between the two, and then looking back at our product and saying, how can we apply those learnings here?
I could share that the tests that we just ran single A/B tests increased people reaching the middle of our funnel by over 30% on a single test, which was huge. But, there’s several month’s worth of work that went into getting us there.
Both of you are saying is that there’s still a matter of taste.
You’re talking about personas and the deep understanding of your customer, and then you made decisions based on that and I think that is the key to being successful here.
I think it was a Google website optimizer that just moves things around and we’re like, that’s not what it is. This is actually using intuition in combination with data to figure out what you think would be a better way for the people you want to reach to get into your platform.
At Shopify, we did a bunch of experiments. For example, we changed our free trial from 14 days to 90 days for the pandemic. We enabled some features earlier than usual, like buy online pickup in-store because you weren’t able to shop in the store anymore. We also enabled our retailers to switch from working in the store to doing local delivery, which is a radius around the store to drop off things.
So some of these things, we had intuition around. Some of them we will have to look at the metrics still and figure out are there tests we want to run to see now, if we can help these merchants get to their buyers in a way that is different from before, because the pandemic shut down an entire channel, which was retail.
I think you bring up an interesting point there too, in the sense that a lot of people, when they first think about dipping their toes into A/B testing, they think, okay, should I change the color of this button from blue to green. A/B testing could be used on really every aspect of your business.
You could be testing different pricing, different trial lengths, whole new website designs, whole new features. It doesn’t need to be just that visual editor that I think a lot of people typically think about A/B testing being confined to. So excited to hear that you guys kind of use that for other things beyond that scope as well.
We definitely have expanded our view of the way we use A/B testing over the last year. Whereas at one point we would release new features, we thought they were so good we just wanted to get them to everyone and we’ve moved away from that over time.
Having too many feature launches that we’re really proud of, but then three months later, we couldn’t tell if the initial bump was driven by seasonality if something had happened on the paid side. So an attitude of truly everything being a test which is simple in concept, but when you’re chasing down a hard deadline, you want to meet an OKR and it’s going to take an extra few days to set up the experiment correctly.It can be a trade-off.
The seasonality thing is terrifying and the main reason why I want to A/B test, especially for an at-home fitness business, Q4, Q1, everyone’s engaging. During the summer months, not as much. And when you’re trying to run that test, as people are transitioning, or if you’re going to launch something during the transition between cold weather and warm weather, you’re going to have a decreasing engagement trend line, regardless.
Similarly, if you’re going from hot weather to cold weather, it’s going to be an increasing trend. So the seasonality impact and the necessity of A/B testing to differentiate is so important there.
Have you seen shifting from that launch mode and just measuring the post and pre-impact versus A/B testing. What does that change for you guys look like as you’ve made that shift?
It’s increased our confidence, we’re getting results faster even though we’re delaying the results of some of them and the confidence to know whether or not something worked really helps when you’re trying to analyze the data.
One of the crazy trends that we are seeing or realizing that when you optimize your ads based on spend or efficiency when you improve conversion, those efficiencies get competed away by your advertising spend. You get slightly worse quality traffic because your conversion goes up.
So over time, you can make all these improvements in the funnel and not actually see your funnel metrics have any sustained improvement. Even though your top line is growing. And so in situations like that it just really helps to have a clear dataset of like, did it work? Did it not work?
Farhan with your metrics that you found with Shopify is kind of backward from normal. Have there been any really interesting insights over the past year where results or the KPIs that you’re looking at actually may have shifted from what you were expecting even internally to the company that you’re working at based on just changing market trends?
We saw a few interesting things. One was when retail shut down obviously, it was insane for retailers because no one had gone into stores and a lot of them didn’t have any setup to do anything outside of just selling in store. We saw something quite amazing, which was that I think 94% of retail revenue was replaced online.
And so people really were engaged with those brands. They wanted to engage their community members in local shopping and they still want it to continue they didn’t switch to something else. By allowing those merchants to just sell online and quickly, press a few buttons, cause they were already using Shopify they could still have a livelihood and have that connection. There are restaurants setting up gift carding. I think the win was that people really want to shop local versus just get some box from somewhere.
That was huge for us. We also saw a huge shift in shopping online. We have this event every year, Black Friday, Cyber Monday, where everybody shops online.
We went from that being an event to we saw that level of traffic every single day across our network. So it was nuts because all of a sudden it was just the normal situation for us. So that ended up being quite significant. One thing I want to just touch on is that you both talked about end goals. We really don’t focus on goals. We have this longterm, worldview that there should just be more entrepreneurs in the world. We do look at data of course, but we don’t try to do something towards some metric number. We really look at it as a long-term view. Is entrepreneurship increasing or decreasing is how we look at these experiments.
How do you think about measuring progress or whether you’re delivering on that vision?
We obviously look at the number of merchants that joined the platform, is GMV growing for those merchants? Are we able to talk to merchants all the time? We have a very deep culture around merchant empathy and talking to merchants directly. A lot of us are Shopify customers, I run the store for my kids’ school which sells sweatshirts and t-shirts so when things roll out we try them ourselves and I see them either working or not working.
There’s a lot of Shopify merchants who are employees. I’m sure Equinox folks use Equinox and Talkspace folks use Talkspace. It’s a great way to get feedback.
I think that’s a really interesting topic and something that has been really different for me in this role compared to other roles I’ve been in is that we’ve really gone deep into integrating our research team into our A/B testing process.
Being able to tap into a community of our current users and take them through a new web-flow design and pre-test things and make some adjustments before we get tests live has been really important. I think for us as a business to have that market testing we can try the products but we’re biased where we already have the products and we want to sell them. Really being able to get that consumer POV before the test is live has also been super helpful for us over the last few months.
I’ll move on to the next topic, which I think is an interesting one. We have seen at least from our company, a shift in the conversations that we’ve had with our customers and the product managers that we work with that really moves from this concept of just improving product A/B testing to more broadly thinking about the customer experience and customer journeys.
Thinking about A/B testing, not as a means unto itself, but as a broader piece of that journey and that experience. One, how do you guys think about this, but two, beyond your companies, have you seen any really interesting products or digital experiences lately and the trends that might be seeing the market in that area?
I gave one example earlier, we do have to look across multiple instantiations of someone trying to become an entrepreneur. Even this notion of pre-commerce where somebody has an idea, but they’re not sure how to bring that idea to the world. We had made an acquisition at a company called Oberlo, which helped doing dropshipping.
You can actually find cool products, you become a curator and that’s your journey to entrepreneurship versus creating something yourself. There are all these ways whether it’s YouTube and TikTok, so we partnered with a lot of these folks to say if you’re a creator of something and you want to share your story and share your brand in a different way, we want to make sure that that’s something you can do.
We know that the first try isn’t the try that usually works. We actually use a video game analogy more than anything, like you’re in the first screen of Mario and you learn how to jump and get the mushroom and hop over holes. Those things you learn then help you not get killed on levels 3, 4, 5.
The same thing is true with entrepreneurship. We want to teach you a bunch of things and you’re gonna try this market there. You’re going to try this product. You’re gonna ship this product to that location and you got to learn something and then you’re going to refine your world. So the customer journey for us is quite long.
There are many people who’ve been with us for years and years. We have to take a very long view versus just signing up in 90 days.
I think as someone who’s always been on the B2C side of the world, hearing the B2B in the B2B to C side is just such a different perspective.
I think one thing that’s been unique for us is we are really the only fitness company out there that is doing a digital meets physical story. You go to an Equinox club, the Equinox plus app is included in your membership and it’s a fully integrated experience where you can take your fitness experience from home to club. But at the same time, we’re also developing digital-only experiences for people who don’t live near an Equinox and want to be able to experience a group fitness class that you can get Equinox at home, or bring in other partners that Equinox has like Rumble, TV 12, for recovery, boxing, or cycling. Really moving into a post-pandemic world where people are expecting things that were previously physical only to be involved in a digital component as well.
I think it’s just going to continue to push what we’re doing and what the broader direct-to-consumer physical world has been doing for a long time, towards a more digitally oriented business.
I’m actually really curious from the B2B side for Cobi and Farhan. When you think about A/B testing, do you think about people in the consumer space, and what do you think those nuances are?
We try to have a worldview before we just try and experiment. If there’s an internal debate, someone could say, just test it.
It’s possible that you want to try things, but a lot of times you want to have some worldview behind the thinking to say, we think that this is an idea. We want to test it out and see what the best way to onboard somebody is. That’s where the testing comes in versus the testing of the framework. You don’t want to just run a daily thing unless we’re talking about conversion from Google Ads.
We are talking about long-running, multi-year does this thing work over a long period or not?
I’m sure it’s similar for you even more so pronounced for you than it is for us, but definitely with the progress of SaaS companies, when you think of B2B the buyer and the SaaS companies is very similar to a straight consumer, they have the same kind of problems that they’re trying to assess their search processes.
The way they think about how a product benefits are all the same. You tend to work with smaller numbers. So from that, you run experiments longer, you may experiment to be more limited in the areas in which you can see lift and see these changes within certain periods of time.
So you may have to be more selective on that side. But generally, when you think about the individual it tends to be similar. You may think about tests at the individual versus broader enterprise level, which I think is an interesting one. That is actually the time that we have, this was an amazing amount of really interesting insights.
Learn More with Taplytics
Start leveraging A/B testing to optimize your website and make data-driven decisions. Get started with a 14-day free trial of Taplytics.