Building Netflix With Elizabeth Stone: Entertainment At Scale, Personalization, And Pushing Beyond Film and TV

0:00On a Friday night, the thing we show you might be different than what we show you on a Monday morning. And so it sounds so simple, different time of day, different device, you're on TV versus mobile. But those were less relevant when we had film and TV series than when we start to add some of these other content types. We have just as many pixels on the homepage. We have hopefully more moments and times a day and across more devices to try to be the right thing for a member in that moment. But the product should feel like that's effortless.

0:35Tomer Cohen:Netflix is one of those rare products that feel incredibly simple from the outside. You open the app, you scroll to find something you like, you press play, and you're ready to go. But to create something that simple, everything behind the product has to be world class. The technology, the operations, the content, the culture, and the craft itself. And now, that experience is evolving. Netflix is no longer just about movies and TV shows anymore. It's about live sports and games and podcasts and vertical video on mobile. And it's no longer just about AI implicitly powering accommodations in your feed.

1:11Tomer Cohen:It's about AI becoming more active and more explicit in your experience. Leading this work at Netflix is my guest today, Elizabeth Stone. She is the CTO and CPO of the company. She has one of the most unusual paths into product and technology. She has an economics PhD. She worked in Wall Street, healthcare, transportation, and now entertainment. We talked about how Netflix builds, how content and product work together, the tension behind expanding the product while keeping it simple and delightful. We also talked about the future of entertainment and what it means to become more personalized, more immersive, and more interactive for the user.

1:47Tomer Cohen:This evolution is what makes the conversation so much fun. With that, here's my conversation with Elizabeth Stone.

1:59Tomer Cohen:Elizabeth, it's awesome to have you on the show. Thank you so much for joining me. Thank you for having me. Great to be here. So we always start on the personal side. And like many guests on the show, you had a very unusual career. You did economics, PhD, Wall Street, healthcare, transportation, and now entertainment, the CTO and CPO of Netflix. When you look back, like, do you see a thread that connects the whole thing? I'm trying to find it. And then, well, I'm thinking when you started your PhD in economics, like your career turned out different than what you had in mind. So when I went to do a PhD in economics, it was because I wanted to teach.

2:36I learned very quickly that the thing that I would actually be focused on would be research versus pure teaching. and so I decided I would switch into an industry economic consulting which is sort of at the intersection between industry and academia. That ended up being a sweet spot for me which in some ways is not dissimilar to leading data science teams in healthcare and transportation and entertainment which is find a way to bring science to a business problem and that has ended up being common ever since I was in economics.

3:14Tomer Cohen:What did you have to unlearn? Because Netflix works very differently. But when you came in, you're like, okay, I'm used to working this way, but Netflix just builds differently. So one is the culture is unusual, and that gets reflected in the way things are built. So much less top down than many other companies. So ideas get generated more organically from the teams in many cases. There's a lot of decision-making and agency that individual contributors have on the team. And there's a sense that responsibility and decision-making sits very deep in the team. It's a tougher role for leaders in a way to find like, what's your role if it's not always determining exactly what we should do or exactly how we're going to do it.

4:02You have to be comfortable leaving space for the teams to have their own ideas, to lean into risk, to understand that bets are going to be made and some of them will fail, and to relinquish some of that control that you might have as a leader in other places. But do that because we get better outcomes. A company like Netflix or a technology company is conducting an orchestra of lots of different areas of expertise. You have engineers and product managers and designers and data scientists and so on, and how to get those teams to work together. I think companies like Netflix or other technology companies have more diversity and skills.

4:43And the thing you learn as a builder is how to get those things to be very complementary and powerful together. We're both entertainment company and technology company. So I had to learn about how How do you conduct and orchestrate a bunch of different product and technology roles in a company that also is very strongly oriented towards content and brand and things related to that that has to come together and be the whole is greater than the sum of the parts across technology and entertainment? Very few companies in the world have world-class content operations and world-class product operations and R &D operations at the same time.

5:23Tomer Cohen:And that's something that Netflix is very, actually, it goes all the way to the co-CEO relationship on the top. How does that work day to day? So at the overall Netflix level, there's a sense of like, what are we trying to do better at? Or how do we further improve for consumers across what Netflix is very good at already? But it was more of an optimization problem. One of the things I've seen at Netflix, which has been a ton of fun, is standing up some of the newer areas like live programming, like games, like now podcasts. us where then you're not in an optimization. You're in like we're trying to stand up something totally new that requires both a content perspective on it.

6:01So which live events should we program? What partnerships do we want to have? And how do we build the product and technology to do that well? And that goes back to the earlier days of Netflix where you have to have a weekly standup. You have to talk about like which deals are we thinking of signing? What would be the requirements for that? What's going to be easy or hard? And we've had to relearn how to work really, really closely together to make sure that these new efforts are more effective than they would probably be if we were working in parallel or not connected. And through these new efforts like live and games and podcasts, we've also gotten better at working together in the mature areas of the business.

6:40So what would be exciting that we could do? Like we redesigned our TV UI to think about how to better promote content and how to do a better job with recommendations for members. We're rethinking mobile now. Those things have been in much tighter partnership with the content team. I think because we learned a little bit about what we were missing when we were doing more local optimization versus if we partnered together. And that's where it sounds cliche, but it's true. Like the magic of Netflix, like what could only Netflix do would be to bring really creative content ideas together with the product to make it very successful.

7:15Tomer Cohen:If we just go back to the core streaming capability, is it that there's two roadmaps and you guys meet quarterly to align? You know what's coming, they know what you're building, and you're trying to make sure they're in concert with each other. Is it that they give requirements for product, product gives requirements back, and how is that coordination working? The role of definitely people on the product and technology team is to let the content organization know what is the direction of the product. What features do we have today? What features will we have in this quarter or the next year? But if we're just talking about film and TV content, that content is not dependent on new features.

8:00It's more what is the product doing to be even better for consumers? and the content team might have some requests of, could we find new ways to surface this type of content? Or can we think about merchandising differently on top of personalization? So there'll be ideas that we consider as part of our roadmap or learnings that we've had based on experiments that we've run that we work with content to decide which of these features are we more excited about rolling out. But they're not both roadmap-oriented teams. It's when a new content type comes, Like when our chief content officer says we signed a deal with NFL, then we say, well, then our product roadmap needs to be really prepared on a set of features that are critical in order to deliver live content.

8:46So that's where the roadmaps have become more aligned because of some of the newer business areas that had special feature requests. Otherwise, I would say that the content and product teams were in a constant state of like, let's always be improving and communicating with one another about what we're doing to improve. You know, there's still problems to be solved and things we want to do better, things that come with scale or things that come with change in consumer expectations. Or when we have a very large content catalog, we have to be excellent at personalization.

9:17Tomer Cohen:So you move from one product to multi-product. It's always a hard transition. It's never easy because, you know, your house was so organized before and everything had their places to be and the product was seamless and cohesive and people knew it really well. How did you think about that transition from one product to multiple products? It's one of the biggest challenges we're facing now to figure out how all of these new, I'll call them like content types, come together in one product. I hope not to create multiple products, to have one that has a broader set of types of entertainment or content, but that knows exactly what is the right thing to recommend for you in that moment.

10:01So the more it feels like we're able to broaden out what we're offering, but it doesn't feel more complicated or like it's multiple products to navigate for a consumer, the more successful we will be. we can't do that out of the gate necessarily for a couple reasons so first we have to introduce to members that we have something new and generating awareness of something there's a tension relative to personalizing because we're basically announcing we have games we have live we have podcasts it's learning who's interested it's learning about things like live is very different in a lot of the joy and value from live comes in anticipating it and then watching it in that moment when it's live.

10:47So when you go a layer deeper into things like what does personalization or the product need to be able to do to understand the difference between a timely piece of content and a not timely piece of content and how much anticipation should we build that doesn't cross the line into, this is annoying for a consumer. In addition to on a Friday night, the thing we show you might be different than what we show you on a Monday morning. And so it sounds so simple, different time of day, different device, you're on TV versus mobile. But those were less relevant when we had film and TV series than when we start to add some of these other content types.

11:26So we have just as many pixels on the homepage. We have hopefully more moments and times a day and across more devices to try to be the right thing for a member in that moment. But the product should feel like that's effortless for a consumer. So I think that's the core challenge that we're facing.

11:45Tomer Cohen:Anything you've tried that you're like, okay, that's not going to be the path? Or anything you're seeing from others that you're like, okay, this is a good principle? We underestimated how much the different content types need to be visually differentiated. So we're introducing live or games and we presume that's obvious to a member, but we need clearer visual cues. So even things like icons or different presentation on the page to signal this is something different than the rest of the content on the page was something that we've learned. We saw confusion around people are clicking on something and then realizing it's a game or seeing a row of things that are podcasts, but not realizing that it's something different.

12:27We also have learned some hard lessons about how much real estate to use for the new content types versus the more mature content types. And the trust buster for a member would be doing too much. We have to find that sweet spot of not over promoting. So whether that's in the navigation at the top of the page, we switched it from the side to the top so it would be prominent. so that it would allow members to explore the catalog more easily if they don't see what they want on the front page. And then we're experimenting with something now that is just below the billboard, which lets you hop into directly podcasts, games, different offerings we have.

13:12So that creates a visibility of the new offering without saying we should show title by title or game by game on the homepage exactly the specific things. Because if we get it wrong, then there's still a reliable pathway where people start to realize we have more to offer, but it's more subtle.

13:27Tomer Cohen:So in a way, there's the awareness discovery side. And then when you come in, how is there more of a comfortable or knowledgeable pathway? And then once I build habit with it, I know exactly where to go. I'm not trying to find my game instead of the experience space. That's right. I mean, things like we hear from members to like, if we move, continue watching, that can be a very frustrating experience. Like I'm just trying to find the title I was already watching. Make it easy for me to find that. And it's one of the things you can see in the data and the patterns of how members are exploring of when they're going to search right away, that they're either looking for a specific title that they've not yet watched, or they're looking to explore the catalog in a way that we're not offering on the homepage.

14:15There's other times continue watching, going straight to continue something. And there's other times which are more like explore. I want to see a whole bunch of things. And then you can see that across the homepage. Part of that is okay that we see members, you know, they're on one of those paths when they come in. But I also believe we should be able to anticipate that better. And so that's related to like, how does it make it feel like the Netflix product anticipates very well, which of those modes I will be in. And if we don't anticipate it well, how can we quickly get signal from you that you're in one of those modes?

14:50So it doesn't feel like we're trying to read tea leaves and making it more difficult. And that problem has gotten more challenging as we think about across devices or times of day or content types, because there's more to anticipate or there's more to figure out how can I get an explicit signal from you in that moment so that we can reset the recommendations we're making based on the context that's relevant for you.

15:11Tomer Cohen:Yeah. I'm curious when When you think about the true norths you're putting, or have you rethought, like, what are you optimizing for, or what are you holding as a true north metric? Same true north. This is actually a positive and a strength, I would say, at Netflix, that the true north has been consistent on how much value are we delivering to members. Which is time spent? More complicated than that, because I think time spent can always be immediately translated to value. So without revealing the details of the metrics that we track, things that we think are strong proxies for are we delivering great value to members?

15:49And we have the benefit of being able to ask members to through our consumer research of how much value are we delivering relative to the price? How do you think about what you're getting from Netflix versus competitors? So there's ways we can look at both the data of how people are behaving and then the feedback we get from our members. that has persisted not even in spite of the new content types but the new content types were motivation for driving more value so the reason to go into games or live or podcasts is because we think there are ways to drive more value for members it was in service of ultimate member value and that members feel like they're getting more from netflix the mindset that gives everyone is there's going to be trade-offs here.

16:33And as we measure what's the right moment to recommend this film versus this game, we're all in service of the thing that we think maximizes the value for that member. So as long as we're competent in the measurement of that and understand the trade-offs there, sometimes that's going to fall in the favor of recommending the movie, sometimes in favor of recommending the game, but we're doing the right thing for Netflix members overall?

16:57Tomer Cohen:Was duration also part of it? Like, for example, if I introduce live and I'm, I can see that being, okay, now with live and other aspects, we need to give more time to see the effect and, because I have to sign up for the live, you're taking prime real estate, the billboard. So that used to be a great way to kind of go and watch something, but now I'm signing up for something that might take away from an action I wanted to do, but really the value will come a bit later. So there's like more patience that you have to build into the system that you had before. Or not? Yeah. When you first said duration, I thought of a couple of things.

17:34So the first is different content types, like a game or a podcast or a live NFL game or a TV series actually have different duration for engagement with that content. And they're not apples to apples because a minute playing a game could be as exciting and valuable as 20 minutes watching this other thing. So we can't just index on time spent for that reason. Like there's different value attribution. The second thing is definitely in the early days of live games now for podcasts, you don't get success out of the gate. You get, do we see signs of life and signs of promise here and we have to be willing to take a headwind for a period of time because of the opportunity cost so like I could have for sure shown you a movie that I think you would have loved in the billboard instead I showed you the live title because we're trying to understand interest in it and see what viewership looks like and whether that's helpful for members but then I took the trade because we're basically making a bet as a company that if we want to be an amazing entertainment company, we've got to have more than film and series.

18:45And so we're going to have to be willing to take some of that trade-off and headwind in early days to learn. The toughest thing from the builder perspective and from the company strategy perspective is, are we learning enough fast enough to be confident that we're on the right path and these trade-offs will be worth it? So that creates an urgency in the teams of, we're not going to have forever to understand how this drives value or whether it drives value or whether we have the right strategy. So let's come out of the gate as strong as we possibly can, fail fast, learn quickly, because at some point we're going to need to say that trade-off was pretty big.

19:22Tomer Cohen:Now, when I think about my family's TV, there's two apps next to each other. It's Netflix and YouTube, and they're both heavily used. And they're both kind of unique in their own way, Netflix has this TV, laid-back, premium content. YouTube is UGC, a lot of vertical content. Put aside YouTube for a second, just entertainment in general. Do you think of Netflix as more, hey, there's concentric circles of entertainment and it's more progression of, you know, we started with on-demand, there's live, there's games, TV, there might be UGC in the future. Because there isn't really a reason why not. Or do you see that YouTube is just a very different island from what you're doing?

20:04I think the entertainment landscape has gotten a lot broader because there's a lot more types of entertainment now, but there's still the same number of hours in a day. and when Netflix started DVD to streaming over the internet but also the way of thinking about content at scale that we could distribute all around the world that's a great baseline to then say what are other entertainment types that we should integrate into what we're offering to consumers and that's where live and games and podcasts come from realizing that in order to entertain the world. We have to have more than just a narrow set of content types.

20:48But I don't know that I would take that to the limit to say we have to have everything in order to compete, as long as the sets of things like concentric circles that we're expanding into are very high value for consumers and make them excited to keep Netflix. I don't know that Netflix has to be the one-stop shop of everything in order to be really compelling as an entertainment offering. So I I think where we sit now is we know we want to have more than film and TV series, be best in class at film and TV, but expand beyond that. But I think it's still a valid question of like, do we have to have exactly the same things that every other entertainment competitor has?

21:28I don't know, because I don't think the end game is there's not tons of competition for entertainment time. It's that we have the right combination of things and we deliver them to our earlier conversation really seamlessly from a product perspective.

21:40Tomer Cohen:I want to shift a bit to AI. Because before this LLM era, a lot of it has been about personalization and I sometimes call it matchmaking, right? Is that still how you think about the core promise of AI for Netflix? Or now you're in this mindset of like personalization is just one aspect. There's so much we can do. So definitely for personalization, as you're saying. also in things like figuring out how best to localize content to many other languages. There were lots of AI tools that were used for things like visual effects, things on studio production. And then there's also a lot of AI that goes into how we optimize content delivery, to say it in the simplest way.

22:29So how do we actually shape and direct content delivery across our OpenConnect, our CDN, all around the world has a lot of AI to it as well. As optimizations, basically. As optimizations for how we encode content and how we actually shape and deliver content.

22:46Tomer Cohen:I'm curious for you, if you want to determine five to 10 years from now, what are you envisioning would be very different or potentially what would not change at all? And you're kind of looking at it as the vision for the industry. The words that I have as a headline in my head are more personalized, more immersive, more interactive. So that can take different formats. That could be to make it more concrete. Interactive can mean I'm able to change the outcome of what I'm watching. So today that looks like live voting for some of our content. How does that evolve over time where the consumer actually shapes the outcome of the title?

23:30Games is a way of thinking about immersive. Like I'm in the world. I'm experiencing that IP or that title and the characters myself. So that would be one thing that I think more entertainment beyond just games will go towards. and personalized is not just the matchmaking, but how content ends up really suited just for you. And so that could be because we have a greater breadth of content and we know how to match it in that moment to what you're looking for, or the content itself becomes more personalized. I think all of those things are going to be shaped by Netflix among others, but it feels like the direction of travel across a whole bunch of things, plus layering in the social, the community, the connectedness, which I hope is a very important part of it.

24:16Especially if I think about entertainment, it's a human endeavor. It's something that's part of the human identity.