PM Spotlight: How an Experimentation Mindset Helps Zappos Lead Product Development

Uncertain times. Unprecedented. Challenging. We’ve heard it all. We all know these are tough times, on both personal and professional levels, for all of us. But through the chaos, there are teams that are working to make a difference for their users and communities in positive ways.

One of whom is the product team at Zappos, led by Andrew Nguyen. Andrew is responsible for mobile, personalization, and advertising at the E-Commerce leader. As a product leader, he is also a source of wisdom on all things product management, experimentation, and personalization.

To learn more about how his team is adapting and his thoughts on product management today, we chatted with him earlier this week. Enjoy!

Let’s start with an obviously pressing topic for a lot of product teams right now: COVID-19. I’ve noticed Zappos introducing some really interesting initiatives to acknowledge unsung heroes and offer extra customer support for those in need. I’m curious how you approached the changing times and where these programs came from?

Our mission at Zappos is to live and deliver wow. Our motto is that we’re a customer service company that happens to sell shoes. Those initiatives mentioned embody the mission and motto well in these changing times. We’re also all about self-management and self-organization; those programs were born from that system and culture. It’s wonderful watching your work family do great things and find ways to support. Our approach remains fundamentally faithful to our commitment to customer service and experience.

I know you’ve also been working remotely as a product leader for some time. As many product teams adjust to the realities of remote work and collaboration, what advice would you share to those who are new to remote work?

As a general thought, I think it’s important for leaders to guide their teams through the changes they want to see. Things like how to conduct meetings, how to collaborate remotely, what types of communication need to be adjusted – these are all important foundations to get right.

Beyond general communication though, for product orgs, I believe that remote work has made writing even more important – particularly short-form and long-form writing.

For short-form writing (ie-Slack, messaging channels), we’ve historically been able to get away with a lot of short replies and discussions. Anything that was ambiguous could be worked out in person or through an informal chat. That’s changed. Now, with busy schedules, these short-form conversations need to be comprehensive enough to be used as references. Responses need to be clear and concise, but still thorough. Otherwise you’ll run into redundant back / forth inefficiencies.

For long-form writing, I think it’s vital to put extra effort into product artifacts – whether that’s a user story or product insight summary. Before, when using artifacts to make decisions, you may have been able to talk over important details. You may have been able to have spontaneous discussions.  Now, it seems that everyone is slammed with video calls, and it’s really hard to find time for those conversations.

To make meetings as productive as possible, I think long-form writing becomes the foundation to productive discussion. I recommend delivering long-form writing before any meeting. That way time together can be used to provide input and make decisions.

Overall, I’d say the maximizing for asynchronous over synchronous communication is important in this environment, especially for product teams.

Overall, I’d say the maximizing for asynchronous over synchronous communication is important in this environment, especially for product teams.

Switching gears, I know personalization is something you’re passionate about. For many, however, it seems like a distant goal that’s out of reach. How did you approach personalization and what tangible steps can teams make towards personalizing experiences? 

I think the first step is to have a personalization framework that works for you, then optimize what you’re doing over time.

From there, I see a few ways to personalize experiences for your customers. 

The end goal, the holy grail, is 1:1 personalization. This consists of collecting signals about your customers—from their case preferences to their demographics—and catering visit experiences accordingly. This often requires a lot of data management and machine learning.

If you’re not there yet, though, the question I’d ask is what do you know about the customer? What signals do you have, and what experience are you confident in delivering? With that, the second option is to explore contextualization, where you translate high level customer attributes into adaptive site changes. 

If you’re not there yet, though, the question I’d ask is what do you know about the customer? What signals do you have, and what experience are you confident in delivering?

Lastly, if you don’t have many customer attributes, you can also start with general segmentation, where you can find bigger customer buckets to optimize for. For example, if you have a high concentration of iOS or Chrome customers, may you want to change and optimize experiences for those specific groups. Another example is to target your top cities, maybe it’s San Francisco or New York. Start where you can as you fill the gap of missing customer attributes or data, personalize now and make it more granular over time.

Overall, we’re lucky to have an amazing machine learning team that helps us deliver 1:1, but those are some steps teams can explore in their personalization journey.

You’ve been managing and iterating on your experimentation program for some time now. I’m curious how your experimentation program has evolved and grown over the years? 

Previously, I’ve been on teams that take a “command and control” approach to experimentation, where someone needs to check off “boxes,” or fill a template before ideas are graded or even discussed. I find that this created bottlenecks and resulted in relatively low experimentation velocity. Through time, though, and through mistakes and lessons we’ve learned, we were able to slowly take off some of the guardrails and run more tests. Now, the majority of our team can run experiments without even telling me. They just need to execute and report on their experiments with a high level of integrity.

The lesson through that process is that there isn’t a “right” or “wrong” way of managing experimentation; you just need to find a balance for what works for your team culture.

There’s isn’t a “right” or “wrong” way of managing experimentation; you just need to find a balance of what works for your team culture.

Ultimately, I’m happy to say that while it was a big upfront investment to nurture this product discipline across our teams , the payoff for experimentation at scale has been exponentially higher than the smaller scale experimentation program of the past. We’re happy to fail 10,000 times to get 1,000 wins as long as we’re learning and getting better each time. And instead of having a “experiment checklist,” what I prefer now is a portfolio of large and small wins, autonomy amongst teams, iterations, and constant learning; those are the things that are driving long term success.

How do you see experimentation today?

As a product leader, my mission is to empower others and scale out decision making. When I think about experimentation today, I think of it as a means of making a whole organization more effective. It’s not just about product managers making effective decisions, but helping engineers become product engineers, and designers and UXers to become product designers. I want each team to be able to build things, iterate, and scale their own experimentation efforts. Every team can use experimentation as an advantage.

When I think about experimentation today, I think of it as a means of making a whole organization more effective.

So true! Thank you so much for the time and insights.