“Experimentation is a data-based approach to ensure we understand the impact of the changes we make to our service.”
Netflix dominates the streaming services market, accounting for 40% of viewing time in TV streaming households and over 100 million members. Its successful reign started when it followed users where they were actually looking to consume videos, leaving the DVD mail service to serve viewers on web, mobile and TV apps.
This success has continued because Netflix is a leader in creating experiences that evolve as users and their preferences do. During their 2017 third quarter earnings interview in October, Netflix’s Chief Product Officer Greg Peters said it perfectly:
“[I’m looking forward to continuing] to evolve the product experience that we have today to be effective at meeting the needs of the next 100 million, 200 million members so it can be as compelling as it is for the 100 million members we have today.”
Content consumption is an experience
Yes, we all know that content is king and that’s the main reason that viewers are flocking to Netflix — there’s no arguing that original content like Stranger Things and Riverdale are the talk of the internet. But if this great content was served in a crappy layout that was impossibly frustrating to navigate, Netflix would not be as popular as it is.
“There’s a tremendous opportunity, I think, to morph the product experience to be more and more effective at explaining to our members what’s great about one of the original shows that [the content] team is making”
— Greg Peters
What you should be getting out of this is that Netflix’s product experience is always changing to meet user needs and help viewers get the most out of the platform as possible.
And how do they do this? They are constantly experimenting!
Nirmal Govind, the director of studio production & streaming data science, wrote about Netflix’s experimentation processes and culture, saying that a key component of a great service is the quality of experience they provide for customers in various markets with differing expectations:
“How do we ensure that the Netflix streaming experience is as enjoyable in São Paulo, Mumbai, and Bangkok as it is in San Francisco, London, or Paris? The engineers and scientists at Netflix continuously innovate to ensure that we can provide the best QoE possible. To enable this, Netflix has a culture of experimentation and data-driven decision making that allows new ideas to be tested in production so we get data and feedback from our members.”
The data and feedback that is gained by running tests are invaluable and far more candid, honest and unbiased than anything you could get from a customer survey or focus group.
Netflix is the ultimate success story that advocates for experimentation, but there are failure stories that make even stronger cases.
Ah, Shomi. Canadian telecomm Rogers’ attempt to disrupt the market with yet another over the top streaming service. Rogers saw the platform as the only way for conventional TV to survive as a growing number of consumers cut the cord on cable. Shomi was introduced in 2014 and announced it’s fall out not too long after in 2016.
According to tech journalist Peter Nowak, one of the reasons that Shomi wasn’t able to keep up with bigger players like Netflix is that they did not have the same research and development capabilities:
“It’s doubly hard to compete when Netflix is spending upwards of $300 million (U.S.) each year on research and development. The streaming giant expends a great deal of resources developing algorithms that can measure, analyze and predict all manner of viewer patterns and desires, all of which keep it ahead of competitors.
Shomi, meanwhile, has had to resort to paying a handful of humans to make those decisions. It’s a folksy effort, but ultimately Shomi executives have learned that humans often lose out when matched against algorithms.”
Experiment often and everywhere
So there you have it. Experimentation and constant iteration is the key to creating experiences that keep up with changing user needs and trends.
Each device provides a unique experience for Netflix users. To make your testing efforts worthwhile and as comprehensive as possible, you need to test on ALL platforms. Most streaming service providers offer viewing experiences on multiple devices: web, mobile apps and TV apps, and testing on all of these will not only streamline the experience across screens, but also optimize the experience to each platform; what works for your mobile app will not necessarily work for your TV app.
“[we’re] making sure, again, that the product continues to be really, really effective at evolving to respond to the new needs of the consumers, the next 100 million, 200 million members that we’ll have. So we make sure we keep an eye to that shifting global set of use cases and requirements.” — Greg Peters
Do yourself a favour and follow suit. Test often, test everywhere and use the data from your experiments to inform the next iteration of your viewing experience.
Taplytics is a product experimentation solution committed to helping teams build digital products for their users and validate product decisions with real data from real users.
Stay tuned for a VERY exciting announcement from Taplytics on how we can help you take your experimentation strategy to the next level to craft evolving experiences wherever and however your users are consuming content.