Back to all posts Back

Before A/B Testing Have All The Facts

image of a computer screen
Jillian Wood
Senior Content Marketing Manager

Congratulations! Your business is ready to start A/B testing within your mobile app, website or other marketing channels. 🙌

Now comes the hard part: do you build your own solution internally or purchase A/B testing software instead?

Since some of the best and biggest companies build their own A/B testing tools in-house (e.g., Netflix, LinkedIn, and Facebook), it can be tempting to follow their lead.

However, just because you can do it, doesn’t always mean you should. (And, if you’re already using an in-house solution, maintaining the system may or may not be in your best interest either, depending on your goals.)

Assessing the time, resources, and tools required to execute on your A/B testing goals will help you make the right decision. In this blog, we’ll walk you through key considerations for deciding whether to build or buy an A/B testing platform, and share tips for evaluating different solution providers.

Build vs. Buy Comparison

Factors to Consider Pros of Buying Potential Cons of Buying
Time and Resources
  • Can start testing almost immediately once procured
  • Offers robust segmenting and random bucketing for quick launches (without relying on engineering for list builds)
  • New features and updates handled by external vendor, which frees up developer’s time and focus
  • Have to onboard and collaborate with an external team
  • Monthly or annual billing based on feature usage
  • May have to battle perceptions that external vendor can’t integrate or work with your tech stack
Costs
  • Avoid unexpected internal dev costs as tools, platforms or industry trends change
  • Offload long-term cost of maintaining tool and building new features
  • Requires an upfront investment
  • May need to renegotiate costs or deal with pricing changes
Efficacy and Potential Risks
  • Reduces risk of improper setup or inaccurate testing
  • Prevents unexpected issues around glitches, crashes, or slow delivery of experiments
  • Putting user’s data in the hands of another vendor; Must take time to assess vendor’s ability to handle user data properly
Support Requirements
  • Technical support offloaded to vendor, which frees up engineers’ time and ensures faster fixes
  • May offer strategic services to improve testing process and outcomes
  • May need internal systems expert to occasionally hop on calls with vendor’s support team for complex technical issues
  • Support teams may be focused on many clients and slower to respond than desired
Ease of Use
  • Product or marketing teams can help own and run tests instead of relying solely on developers
  • Code free visual editors empower less technical team members to launch test without engineering help
  • Build a library of code variables to start code-based testing fast
  • Reporting and tracking built-in
  • Must assess vendor’s ability to integrate with your tech stack
  • May want engineering to help initially set up and test Visual Editor
  • Reporting, tracking, and dashboards may require custom setup and integrations
  • Not always able to export raw testing data from your platform into other tools for more in-depth analysis

Things to Evaluate Before Making a Decision

1. Time and Resources


Pros of buying A/B testing software:

  • Buying enables you to run A/B tests much faster since you don’t have to rely on your engineering team’s time to build your solution’s infrastructure or set up each test. Purchasing an A/B testing solution also means your engineers can focus on your core business and make product changes faster based on your test results.
  • Many A/B testing tools have easy to set up SDKs that let you to start quickly experimenting (sometimes within hours) on many channels.
  • Having to ask developers to provide segmented lists for targeted A/B testing can slowdown testing time. External tools often include robust segmenting and random bucketing for quick launches.
  • Mobile A/B testing tools allow you to run experiments without having to upload a new version of your app to the app stores. This lets you experiment as much as you want without worrying about app store updates.
  • Developers don’t need to spend time staying on top of the latest A/B testing tools or trends. Vendor handles staying on top of new feature development or upgrading systems.

Potential cons of buying A/B testing software:

  • Having to onboard and potentially continually communicate with an outside vendor may be daunting or time-consuming. Weigh this against the benefits of having dedicated support from A/B testing experts vs. relying on internal resources who may or may not specialize in experimentation.
  • Will likely have monthly or yearly billing based on feature usage or monthly active users. This potential ongoing monetary cost must be considered against the ongoing costs for staff to build and maintain your own system, plus the opportunity cost of not having them focused on your core product’s features and functionality, which could be driving revenue or business value.
  • Must battle the perception development team may have that A/B testing software vendors won’t be able to integrate with their existing tech stack. Doing technical research and changing attitudes can make the purchasing process longer. (Note: most vendors will help answer technical questions and pitch the value of their platform to your team.)

2. Costs

Pros of buying:

  • Avoid unexpected internal development costs as tools, platforms or industry trends change. External A/B testing software providers are focused on staying on top of new channels or trends, and will often support them quickly as per market demands.
  • Long-term cost of maintaining and improving your testing infrastructure is offloaded to the vendor, who will provide updates and latest features for optimal testing.

Potential cons of buying:

  • Buying A/B testing software may require an initial upfront spend that can impact your budget and/or require approvals.
  • At the mercy of the vendor’s pricing model or pricing changes, which may need to be renegotiated each year. (This can be negotiated by signing multi-year contracts.)

3. Efficacy and Potential Risks

Pros of buying:

  • Knowing your testing infrastructure is set up correctly and that results are statistically valid since the platform was purposely built by A/B testing experts and has been validated by other customers.
  • Prevents unexpected issues like slower load times or crashes caused by something like a large influx of MAUs. External A/B testing software providers have tested platforms that are ready to handle a large volume of users and experiments—thereby safeguarding your testing process and your users’ experience.

Potential cons of buying:

4. Support Requirements

Pros of buying:

  • Most vendors provide continual technical support in their contracts. Leaning on an external team of developers and strategists who are experts in A/B testing allows you to move faster and worry less about issues or glitches. They can also help you with troubleshooting whenever you’re in a pinch, whereas your internal dev team may not be available in an emergency.
  • Customer Success or Services teams can offer you everything from consultation on your testing process, effective goal setting, proper setup and analyzing tests from start to finish—all of which can help you get up to speed and testing effectively fast.

Potential cons of buying:

  • Support teams for vendors may be focused on a number of clients—meaning they may not respond to every issue you have quickly. This must be weighed against the cost of having this kind of support available internally. Look for client testimonials around the support team’s responsiveness or inquire about priority support options if this is a concern with a provider.
  • May still need to ask an internal systems expert to talk with vendor now and then when troubleshooting

5. Ease of Use

Pros of buying:

  • Your product or marketing team can help own and run testing instead of relying solely on your developers for all tests. This empowers your entire team to participate and experiment more.
  • Vendor may offer a code-free visual editor that can be used by marketers and product managers to launch new visual tests without little engineering support.
  • Some providers offer code variable libraries so developers can launch code-based experiments and re-use them in future tests quickly.
  • Reporting and tracking are often built-in, making it easier for non-engineering team members to access data for analysis.

Potential cons of buying:

  • Product development teams may feel vendors cannot support the types of code-based tests they want to run. Ask vendors which kinds of tests and platforms they support. Most established vendors can integrate with nearly every programming language, but dig into technical specs early on to determine if anything will prevent implementation. (Vendors may offer custom integrations or setup if needed.)
  • May need technical teams to double check set-up of Visual Editor tool to ensure tests are rendering and being recorded properly (depending on the programming language used for your website or app)
  • Reporting, tracking, and dashboards may require custom integrations. Many A/B testing software vendors provide out-of-the-box and custom integrations (plus services) to help make reporting as valuable and customized as possible.
  • Some vendors may not offer the ability to export raw testing data from their platform into other tools for deeper analysis. Ask vendors if this is available or if access comes at an extra cost.

 

When Does It Make Sense to Build a Solution In-House?

Some businesses may prefer to build an A/B testing solution in-house when they have a complex product, or they want extremely customized testing options.

Booking.com developed their in-house testing solution 8 years ago. They did this because they had a complex product, ambitious testing goals (they have 1,000 tests running at any given time), and there were few A/B testing software solutions on the market. So, at the time, the considerable time and engineering investment made sense. Today, there are more A/B testing providers who can make quickly launching a robust experimentation program possible for even the most complex products.

If your A/B testing requirements are so unique that your experiments exclude you from being able to use an outside vendor, it usually means that your testing does not fall within any industry standard, and you should consider building in-house. However, the best A/B testing platforms have often already supported a tech stack similar to yours. Ask for a technical overview, or request custom solutions or integrations, so you can gauge early on if it’s worth exploring vendors vs. an in-house solution.

When To Abandon Your In-House Solution For An External Vendor

Companies who build their own in-house A/B testing solutions can run into issues their internal tool or have new needs arise that force them to consider adopting an outside vendor.

These issues may include:

  • Wanting to test faster—without worrying about version updates or app store submissions
  • Want to run new types of tests that require new functionality
  • Wanting less technical teams to be able to set up tests and analyze data to save engineering time
  • Hoping to avoid building new integrations with new platforms, tools or programming languages as industry trends or tech stack changes
  • Don’t want to continue with further engineering investment in A/B testing tool development as priorities have shifted

A/B testing software vendors can often solve these common issues, either by replacing or integrating with your current solution.

Top Features & Evaluation Factors For Vendors

While exploring whether to buy or build (or maintain your own) A/B testing tool, here are some key areas to examine with each software provider:

Top A/B Testing Software Features:

  1. Platforms and programming languages supported
  2. Easy to install SDK (or no-install testing options)
  3. Code-based testing
  4. Easy to use interface for non-technical team members (code-free testing, Visual Editor, Visual Optimizer, etc.)
  5. Server-side or client-side testing options
  6. Number of users or experiments that can be supported simultaneously
  7. Segmenting and bucketing capabilities
  8. Fast delivery of tests (read: avoiding the dreaded “flicker” effect users can see)
  9. Data security and user privacy options
  10. Flexible reporting and event tracking
  11. Breadth of integrations or custom integration solutions
  12. Data export options

Other A/B Testing Vendor Selection Factors:

  1. Cost (including onboarding/training, special services and/or custom integrations)
  2. Support and Services offered
  3. Experience with your industry or business type/size

Related Resources

Whether you decide to build or buy an A/B testing solution, the important thing is creating a system that allows you to start experimenting quickly and consistently. Here are a few resources to help your team when you’re first starting to A/B test:

Learn More with Taplytics

 

Back to all posts