Back to all posts Back

How Product Teams Implemented A/B Testing

a man writing on a whiteboard
Will Lam Profile
Will Lam
Product Marketing

If you want your A/B testing implementation to work, you have to introduce it in the right way—and the earlier the better. Get buy-in from all departments (especially the leadership), set a company-wide vision, select the right software, and stick to a methodology until every team understands how A/B testing makes their work easier.

What is A/B testing?

As the name suggests, A/B testing is when you test two variations of a campaign and compare which one performs better. This can be anything from a landing page, ad text, headline, or call-to-action.

The Importance of A/B Testing

Optimization

When time and effort go into a new website, the last thing you want is for your visitors to click away without viewing any pages. That’s where optimization via A/B testing comes in. In this new era, companies need to perform A/B testing to find the winning combination of elements on their pages. Tweaking headlines, fonts, and anything else will help you keep visitors on your site. This is a direct way to provide them with content that may lead to a sale.

Increased Conversions

A/B testing is the most effective way to produce more buyers. It’s easy to see what converts and what doesn’t when you put time into carefully crafting two versions of your campaign. The more time you spend on this, the more likely you are to convert more leads.

Learn About Your Customers

A/B testing not only helps you convert visitors into buyers, but it also helps you find out what your consumers want to buy. If you find a design that is converting well, don’t stop there! Keep testing, iterating, and communicating with your consumers to see what they want in the future.

When you find a design that is converting well for one type of product or service, test it out with a refined version of the same design to see if it will have a higher conversion rate for a more expensive product or service. With just a little tweaking and some patience, you might find that customers are willing to pay more for a more high-end product or service that has more perceived value.

Straightforward Results

An A/B test is a simple and easy-to-understand analysis. When you analyze data from an A/B test, you will be able to determine the “winner” and “loser” based on straightforward metrics (i.e. time spent on the page, conversions, etc.).

Reduced Risk

If you’re considering a major change to your site, it can be crucial to test first. A/B testing is a way to examine visitor and customer behavior without committing too much time, effort, and money on strategy decisions you might not want.

​​A/B testing helps you make better, more informed decisions by analyzing your resources for maximum effect and efficiency. This can help increase ROI based on short-term conversions, long-term customer loyalty, or any other metric that is important to you.

How to Implement an A/B Test

A/B testing implementation is simple: Draw out your funnel and pick a stage where you’d like to increase conversions. When in doubt, begin at the top. Brainstorm test ideas to improve conversions at that stage. Create an A/B test backlog. Prioritize those test ideas based on the effort-to-benefit ratio. Then, begin running tests.

 

A Five-Step Plan to Make Your A/B Testing Implementation Work:

 

1. Get Buy-In

A/B testing has the potential to benefit every single department in your company. Apps and sites that satisfy users and drive revenue are what makes the business grow. But A/B testing’s impact isn’t always clear, so you’ll probably need to get verbal buy-in from each department that they’re at least willing to give testing a try.

Gather case studies of how companies with similar products or in similar verticals hit goals you want to hit with A/B testing. Similarity is a critical factor. A best practice isn’t necessarily best for you unless you face a similar situation. If it’s a success story from an unrelated company, it can lead you astray, and this happens a lot. Plenty of companies have invested in retention efforts, for example, based on a famous statistic from Bain & Company. Few of them know that it’s based on a study with a sample size of one—a single credit card company that’s no longer in business.

Resources on A/B testing outcomes:

 

Sell the vision of A/B testing differently to each department, based on their interests. Marketing will want to know that it’ll increase top-of-funnel conversions. The product team will want to know that it’ll help you quantify product quality. The engineering team will want to know that it’ll lead to shorter development cycles and more exciting work. 

 

Learn how to sell your executives on A/B testing here

2. Establish Goals and Vision 

Companies don’t arrive at product perfection through random acts of testing. They get there by having a strategy replete with goals, vision, and an iterative testing plan to reduce risk. You can ensure that a culture of A/B testing takes hold in your organization by defining what you want the A/B testing program to accomplish before you purchase a software license. 

 

Our foundational A/B testing principles: 

  • Test to the funnel: Every A/B test should improve at least one measurable conversion in your marketing funnel (or buyer journey). Teams that don’t test to the funnel waste time and resources improving things that may not matter to users or the company. 
  • Start small: Break big ideas down into small tests to validate them iteratively. Instead of launching a recommendation feature, first see if users enjoy using a “top picks” button.
  • Share what you learn: Help your entire organization learn. Record knowledge in a test log so others within the organization commit fewer mistakes and make cumulatively better decisions.

Your vision for your A/B testing strategy should clearly describe the ideal state for your site or app. For instance, to become the number one gaming app in the iTunes app store, or to drive at least 10 percent of the entire company’s sales. That vision is a goalpost, and it doesn’t come from research—it comes from intuition and knowing your business. 

Companies new to A/B testing often make the mistake of thinking A/B testing alone can solve high-level design questions in their product. It can help, but you won’t generate good testing ideas if you don’t have a product vision.”

– Matt Aranoff, Co-founder, Lead developer & UX designer, Logical Animal

 

Optimize for a North Star metric

The most precise indicators are lagging ones, like revenue. Be careful that you’ve chosen the right one to follow. The history of failed enterprises is replete with stories of smart people working exceptionally hard on the wrong goals. When in doubt, measure revenue, or a KPI tightly related to revenue, like sign-ups or net order value.

 

How far should you plan in advance for A/B testing? 

Every company is different. Fast-moving mid-stage startups might be thinking 6 to 12 weeks in advance, whereas large enterprises think 6 to 12 months out. The ideal time horizon is not too short—you should know what you’re trying to achieve this month and the next–-but not so long that you can’t adapt. You need flexibility for your team to apply what they’re learning, otherwise they’ll spend all their time re-writing the plan, or worse—sticking to it after its outdated.

 

3. Purchase Software

Choose A/B testing software that supports all your must-have use cases and channels. For instance, will Amazon FireTV be important in the next year? Put it on the list. Will combining web visitor data with app user data for a full picture of user behavior be important? Add it. 

 

Important features of an A/B testing platform:

On top of that, you may want to add a data visualization platform, data repository, or additional testing tools.

4. Set Up and Install

A good A/B testing platform removes a lot of the work of installation. If the interface is intuitive, it requires less training and leaves less room for error. If the vendor’s professional services team is highly experienced, they’ll help handle the setup and tagging so it works right the first time around.

 

The A/B testing software implementation process:

  • Schedule a kickoff call with your team
  • Create a shared services project plan
  • Establish roles: Who participates? Who owns the backlog? Who schedules check ins?
  • Schedule a quarterly check-in with bi-weekly progress updates
  • Analyze existing funnels and test results
  • Implement the software and set up tracking
  • Get started

 

How do I practice my AB test?

Always start your A/B tests with phased rollouts: Run the test with only 10 percent of the sample population to make sure that everything is installed and firing correctly. Then, roll it out to larger and larger samples. Phased rollouts give you the opportunity to correct errors before they go live.

 

5. Stick to the Methodology 

Keep the spirit of your A/B testing methodology alive with recurring training sessions. For the first year of A/B testing, bring everyone back into a room with your most experienced testers a few times to ensure they’re sharp on the basics.

image of a flowchart in purple and blue hues

 

Celebrate Your Culture of Experimentation 

Your culture of experimentation will take hold when the positive results of A/B testing are clear to all. When your tests produce rules and knowledge that product, marketing, and engineering teams come to rely on, it becomes a vital part of work—and actually works.

 

Start A/B testing fast with Taplytics — Learn more

Back to all posts