How to Build an Experimentation Culture

Time and time again, you’ve heard about the importance of testing: the more you experiment, the faster you learn, and the faster you can drive growth.

What’s less obvious is how to put testing into practice. If your team isn’t aligned around experimentation, getting people to test often makes it seem like you’re asking them to do more work. The best you’re going to get is people running one-off tests that are too narrow to provide meaningful product insights.

To create a culture around testing and experimentation, you have to build from the core of your organization. Incorporate experimentation into your regular processes, and your team will build a strong habit of testing. The more procedural experimentation becomes, the more people will be ready and able to test.

In this article, we’ve highlighted the steps of building an experimentation culture, so you can continuously create great products as a team through iteration.

Learn how to A/B test like the best. Read our free Experimentation Guide.

What is a Culture of Experimentation?

In an era of innovation, it’s important to have a culture of experimentation. It is not enough that experimentation only be limited to departments and certain people. Experimentation needs to be embraced company-wide and across all levels of management. Managers and executives need to have a mindset where they’re always seeking out the opinions of their team members — and vice versa.

Experimentation is the key to innovation. A culture of experimentation makes innovation centrally located in any industry. To experiment, you must try out new ideas.

Performing many experiments means you will see some ideas flourish while plenty of others fail. A culture of experimentation is fundamentally about failing and then learning from mistakes. A company that isn’t experimenting and taking risks will never be able to come up with a truly innovative, groundbreaking idea.

Why an Experimentation Culture is Valuable

When you have a culture of experimentation, testing is something people automatically do. Experimentation is baked into your other processes, so you feel its benefits throughout your organization. Here are just a few of the benefits that an experimentation culture brings:

• You’re more likely to discover positive product changes. According to AppSumo, less than 25% of A/B tests produce positive results. Most experiments have no effect, or worse, have a negative impact. With these low odds, you have to conduct a lot of tests in order to see a greater amount of product results. With a culture of frequent experimentation, you have a greater chance of finding ways to add positive product value for your users.
• Hierarchy doesn’t dictate product decision-making. Relying on HiPPOs (the Highest Paid Person’s Opinion) to make product choices is dangerous — you risk missing out on great product ideas that come from lower-level employees. Building a culture of experimentation makes A/B testing the default decision-making process. Ideas aren’t prioritized by position — instead, the validity of everyone’s ideas is fairly measured through test results.
• Your team members feel motivated to come to work. According to a Forbes study, employees are more likely to be engaged in their jobs when they are more involved in projects. Making testing a regular team practice gives members an opportunity to make an impact and feel invested in their jobs. With an experimentation culture, team members can easily see their ideas tested and feel excited about their work.

Ready to enjoy these perks? It’s not as easy as turning to your team and saying, “Let’s test more.” Building an experimentation culture takes a few steps.

How to Build an Experimentation Culture

Creating an experimentation culture out of nothing takes time and effort — you have to build the right habits so that your team can reliably iterate and create the best versions of your products.

You have to introduce new practices gradually to get your team onboard with an experimentation culture. You slowly build a sustainable, repeatable process for generating growth ideas and implementing them as experiments. With these steps, your team has a foundation for constantly testing and improving your products.

Step 1: Start small

Instead of building an experiment around a big, high-level goal, such as revenue growth, start by focusing on something smaller and testable, like engagement on a product page. Setting actionable experiment goals allows your team to move forward with testing, rather than getting stuck in abstract objectives like “growth.”
To map out these tiny steps, move backwards from your major goal. Identify:

• Your high-level goal: The video company Upworthy, for example, found that their site visitors weren’t spending much time exploring their content, so they set a larger goal of increasing engagement on their site.
• What needs to happen to achieve this goal: Upworthy recognized that they needed to make it easier for users to move onto new pieces of content and share content to increase engagement.
• An A/B test you can implement to meet your goal: Upworthy experimented with different placements and designs of modules that showed users recommended and related content.

Their experimenting proved to be successful — they found that one test content module increased social sharing by 28% and also improved engagement.

Upworthy’s experimentation was successful because they started with a small, actionable step — testing content modules — to achieve their larger goal of greater user engagement.

Step 2: Practice good experiment hygiene

The backbone of a strong culture of experimentation is creating a standardized format for every test your team runs. Having an established testing protocol keeps your team’s test results accurate and meaningful, regardless of who’s controlling the experiment. We recommend keeping in mind the following testing guidelines as a group:

• Test one thing at a time. With more than one variable, your test results aren’t meaningful because you can’t pinpoint which factor is affecting your tests. Experimenting with a single variable, like module placement or imagery, allows you to isolate the cause of your test results.
• Know how to measure your results before the experiment begins. Your tests are useless if you’re unable to collect your experiment results. Usually, the tool you use to launch your test will have an Analytics feature to measure experiment outcomes. Our Taplytics software, for example, has a Dashboard feature to view A/B test results.
• Establish a time frame. Think about the requirements of your experiment, and set a time frame long enough to collect enough information to form insights. Once you’ve decided how long your experiment will run, set reminders for when you want to collect results.

Taking the time to think about these details before experiments begin will keep your team’s test running smoothly with little to no errors.

Step 3: Communicate results back to your team

Build momentum around experimentation by communicating your test results as a team. Sharing these outcomes provides your team with insights on how to iterate and improve future testing. Plus, communicating your wins will get people excited about further experimentation.

Find a tool your team can use to share experiment results internally. The marketing software Hubspot, for example, uses an internal wiki to communicate information about their tests. You could also publish experiment results with a shared document or spreadsheet, such as the example below.

[Source]

By sharing experiment results, your team’s testing becomes sustainable. Constantly learning from each others’ feedback, your colleagues have plenty of insights for building and improving new tests in the future.

Step 4: Test at a cadence

According to Richard Eckles, the growth lead at Dailymotion, it’s unrealistic to expect positive results from just a few A/B tests. “You’re not gonna change a call to action on a button and suddenly start seeing a massive spike in revenue,” says Eckles.

It takes more than a few tries to get positive A/B test results, so companies need to experiment frequently. Eckles recommends that companies run at least, “50 tests a month, not only one or two.”

The travel planning site Booking.com runs more tests than most companies.

Most companies run an A/B test once every week to three months, according to this study. Booking.com, on the other hand, has at least 1,000 experiments running at any time.

Running A/B tests in-house, Booking has the resources to break down their hypotheses into many atomic experiments. The company continuously iterates on their tests to create more tests. With so many experiments, Booking has a high chance of finding a version with a strong, successful conversion rate.

Booking’s strategy of heavy experimentation has proven successful. The company’s testing drives conversions at two to three times the industry average, according to research by Evercore Group L.L.C.

There isn’t a set number of times you should experiment — the figure varies depending on your company. It’s more important to commit to a weekly experiment schedule for a set period of time. Measure your progress by tracking:

• The number of experiments you run
• The number of winning tests

If the number of winning tests is lagging, increase your testing tempo in your schedule to ensure that you’ll have enough experiments to achieve more positive results.

Frequently experimenting like Booking doesn’t avoid failure. It does, however, increase the likelihood of successful test versions — giving you more opportunities to improve your product.

Step 5: Embrace failure

Failure is a part of testing — most experiments won’t have positive results. As mentioned earlier, it’s estimated that less than 25% of tests actually produce significant results.

Negative results can discourage your team from experimenting further — unless failure is normalized. Establish that everyone gets it wrong with testing, and your team will be motivated to continue experimenting.

Openly communicating about negative test results is a great way to embrace failure. The online marketplace Etsy, for example, asks employees to openly share their mistakes, testing-related or not, in public emails called PSAs. Here’s an example PSA from Etsy:

It sounds scary, but Etsy employees love it— more and more PSAs have been sent throughout the company since the practice started in 2010.

Don’t let failure stall experimentation — sometimes you can even find value in failure, so embrace your mistakes as a team so you can move on and continue testing.

Test More With Your Team’s Support

An experimentation culture doesn’t blossom overnight. Without a process of constant testing in place, your team will feel lost if you suddenly ask them to experiment more. Instead, you have to guide your team through testing with clear steps. Introduce a repeatable process for brainstorming ideas and implementing them as experiments, and your team will be ready to constantly test your products in the future.

We’ve highlighted a few steps that your team can use to build an experimentation culture. With our repeatable process, your team can perform well-planned, results-driven tests at a regular cadence. Constantly experimenting, you’ll always have insights to innovate and improve your products.

Get more A/B testing best practices in our free Experimentation Guide.

Header Image background: source