How Google prototypes and ships at scale — Soufi Esmaeilzadeh (Google Search)

0:00Feeling stuck in the middle of your career? The Mid-Career GPS Podcast helps mid-level professionals navigate leadership, office politics, networking, interviews, and their next career move. Across more than 350 episodes, host John Narrell and his guests share a weekly dose of practical guidance to help you stop feeling stuck and start moving forward. Search for the Mid-Career GPS Podcast wherever you get your podcasts. Follow the show and listen today.

0:55career, or team. Find your next useful idea. Listen to the Business Takeaway podcast wherever you get your podcasts. Click the link in the show notes.

1:09People who have this builder startup mentality, this excitement of getting things done really quickly, of making it happen, of pushing the boundaries and moving at speed is something that I really seek in the DNA of the PMs that come and join the team.

1:31Randy Silver: Hey, welcome back to the product experience. I'm Randy Silver. And, you know, there's one of these things that you do a dozen times a day. It's usually the first thing I do when I log onto my computer in the morning is I go to the search bar, the address bar on my browser, I type something in, and I'm too lazy to actually do the dot com. So I get this search experience back and Yeah, I use Google. A lot of us do. It's one of these things that has become a copyright defensible word because it's not just search. It is Google things. But as the world is changing, as this experience is changing so much, we thought we'd get someone who has a really deep experience in this on today.

2:12Randy Silver: So we've got Sufi Esmalzadeh. She's the Product Director for AI Experience of Google Search. And welcome, Sufi. Thanks for joining us today. Thank you for having me. So I said what your title is. Do you mind giving everyone a little bit of an introduction? What does that actually mean, Product Director for AI Experiences on Google Search? And how did you get into this world in the first place? Yeah, of course. It is a mouth load, so let me describe it. I drive our AI experiences in search. What that really means is what people know is that box on the top of the search page called AI overviews and AI mode experience you may have heard of or you may have actually gone into when you ask a follow up question.

2:58really focused on how do we make those world-class consumer experiences for people whenever they use search. And I would argue I have one of the best jobs at the company because I have the privilege of thinking about how our billions of users who use search can really, with so much less effort, get to the heart of what they're looking for, ask what's on their mind without using keywords. And so every morning, I think top of mind for me is how do we continue to make an evolved search to really meet the moment and help people on their day to day. And how I got here by background, I studied engineering.

3:37I always love to build. And then I discovered this thing called product management, which was the perfect combination of, gosh, my engineering side that wants to make things, combined with really pushing that creative strategic side that I love, which is really thinking about what is the right thing to build and what really will resonate with consumers. And so I really fell in love with product management and have stuck to it since.

4:07Randy Silver: And how old have you been at Google? Oh, my God. I'm going to age myself. I'm coming on to actually fully did now 15 years at Google. Fantastic. OK, so you've seen this change dramatically. And, you know, for the longest time, search was one of those things that was it was a solved problem. It was a mature product, at least from the customer experience. I know engineering wise, there was always room for improvement, but we knew what search looked like as a mature thing. So what's changed? You know, the AI has come in. It's not 10 blue links at the top of the page. We're not arguing about advertising links anymore.

4:49Randy Silver: Now we're talking about the AI, the first return is the AI experience. How did this come about? What do you think about? How did it start? Yeah. So listen, one of the reasons, first of all, that it's kept me at Google for 15 years is because the job is constantly interesting and it's constantly interesting because we're constantly evolving. So in one way, I would say, listen, even search has been evolving over the trajectory of history. Like you said, it started as 10 blue links and then the page did continue to change and change and change. However, I think, Randy, you are pointing out that with this AI paradigm shift, it is opening up an entirely new class of opportunities that we didn't have before.

5:29Like things that you would dream up as a product manager are now actually possible to do. Things you wouldn't even thought of, your creativity is now getting and going. And so I think in this moment, it is one of the biggest evolutionary moments of Google search where our mission is sort of timeless, this idea of organizing information, of satisfying human curiosity. We're never going to be done. But now we really have the opportunity to push the boundaries on helping people just speak to us in a really natural way, helping people actually engage search, not as one off keywords, which has traditionally been how people have been using search, but as a multi turn conversation and something you can come back to and pick up where you've left off.

6:16And so it's just creating a whole new class of opportunities on how we can help people.

6:22Randy Silver: how did this come to you as you know you this position you're in it didn't start you know one day they someone just decided had to be there how did it come to you how did you end up in this position and how did the the ai experience of of google search of evolve over the last couple of years yeah so you know it's a combination of and i think this is how we show up but product managers every day now, like seeing what the underlying technology can do. We really, I remember in the early days, went to like first principles, where you start really thinking about your hypotheses of how can you actually change the way you help people?

7:03Why would that resonate? What do you think are the indicators that you're really helping them out? And I'm going back to first principles, because that was the time when we thought, gosh, you know, with the fact that you have these natural language models, can't you let people stop using keywords and just formulate in sentences? And let's try to get that product out there. And what would that product look like? We built that product and, you know, very quickly, and that's what's amazing about all this, you start seeing consumer behavior shifting. They're starting to ask longer questions. They're starting to say things like, not hotel in Los Angeles, but I'm going on a trip with my kid to celebrate his birthday and I'm looking for a hotel that is kid-friendly to him and has amenities, but is also relaxing for me and my partner.

7:50Nobody used to talk to us that way. So you saw that shift starting to happen. And I think building on top of that, when we then start people using that product, we started seeing people's expectations evolve even more. Well, wait a minute, why can't I then ask a follow-up? Why can't I predictably in search get this AI experience when we want it? And so that's when you're continuously building off the queue of also your customers to keep evolving and evolving where this is all going.

8:18Randy Silver: So when I do this in, say, Claude or ChatGPT or even in Gemini, there's a lag between the time I ask the question and the time it's thinking to generate the answer. But search, we're expecting the answer instantaneously. And one of the things I've always been really impressed with when I use Google search is how quickly that answer comes back. And it's not something you can pre-cache in the same way that you would with the most common search results. I'm guessing the queries are very specific. They're very different. They have to be personalized. How are you addressing something, a challenge at that scale?

8:54Well, and I'd love if you'd give me a moment to answer that in a couple of angles. Please. Because first of all, the first thing that I think is magical about search in this AI era is we recognize that people have all sorts of questions, simple questions, hard questions, and they shouldn't have to think. They should go into the search box. They should ask their question. So first of all, I think what makes search so magical in this is the capability to answer the simple questions like what's the weather today to the much more complicated that you raise. the second thing that i think is amazing in the search ai era is how do you do that effectively combining the best of the web and ai because that's the other expectation people have and the third is you know google does really bring ai at scale to billions of users it is one of our core differentiators and i think what you're hitting on is we have a phenomenal uh team of engineers with whom we work to ensure we are building products that can have the reach that search has.

9:57We are building products that can answer at the speed at which our user is expected. It's just a foundational expectation of search. And so that is some of the technical challenges we are constantly grappling with and optimizing for as we're building our products. Okay.

10:14Randy Silver: So let's stay in the search domain for a moment, but this is an interesting one, because you are the dominant player by far for search on the web. And we've seen people with the dominant market position any number of times, whether it's Kodak, whether it's Blockbuster, whoever it might be, have the problem of not disrupting themselves, not updating. And even in search, I mean, I'm old enough to remember Yahoo as a ranked directory and not actually be migrating to technology that was doing the page ranks and things like Google and other things like AltaVista and Ask Jeeves and everything else falling by the wayside over the years.

10:56Randy Silver: So how do you approach this? How do you avoid the problem? How does Google give you the power and what are you doing there? And what kind of challenges do you have to ensure that you've got the most valuable real estate in the company? and how are you using that to disrupt but not lose the dominance that you have and not lose the financial impact that search has? Yeah, I think the heart of what you're asking me is, to me, listen, search, how do you evolve it? You're saying it's a very ingrained product. How do you evolve this product? Both technically and then politically and strategically within the company.

11:36Randy Silver: Yeah. You know, I think that I was saying to you earlier, like part of the reason I've stayed at Google for so long, you asked me the duration, is because I fundamentally believe this company puts the user first. We constantly think about the consumer and what they want. And that is what we honor. And the reason I'm saying that is then in that is the effect of I don't view this as a disruption, but I view this as an evolutionary moment. And it's a great evolutionary moment for search. It's a moment at which we can really ask, how can we do better for our consumers? And I think when positioned that way, when these discussions were happening or saying, hey, listen, this AI paradigm shift is actually going to help us answer questions we couldn't before, get to the heart of what people are looking at we couldn't before.

12:23it really paved the way to then demonstrate that. And very quickly, we saw the fact that, yeah, consumers, it resonated with them. They are using the product more. They are shifting the way they are engaging with the product. And so for me, that's just an indication of, gosh, we're on the right track. And hence, from there, we then went to building AI mode and have this relentless focus around, like, next, we really want to build your personal information agent. Like that is the future that we're going to. And it's still search at the heart of it. What makes search search every day to me is the fact that we are here to organize the world's information.

13:01That mission is timeless. So as long as we're honoring really helping people get to the heart of what they're looking for and doing that better, that's the right product. And so I really don't see it as a disruption, but much more as a gosh, this is the next stage of a product that has been evolving. But for sure, we should acknowledge this phase, the degree and speed of innovation is much more rapid than what we've seen before.

13:25Randy Silver: I'm going to challenge you on the disruption and evolution words. I can totally understand from where you sit, it's absolutely an evolution. It's evolution of the experience. You want to delight your customers. You want to give them the answer in as short a time as possible and satisfy them as much as possible. And that makes all the sense in the world to me. But that information ultimately is coming from other places. There are content providers. And as someone who was a journalist in the past life who is making content now, the difference between SEO and GEO is massive. And the experience of you used to send people to me to get the information, you would give them a tease of it and they would come to me to get the final answer.

14:11Randy Silver: But now you're quoting me and providing the answer and not necessarily I'm I'm not necessarily seeing directly the benefit of that or I'm not getting the click through from it. So it's a two-sided marketplace. You're absolutely evolving for the consumer side, but for the provider side, it's got to feel like a disruption. So how do you deal with that? Or how do you in turn evolve the provider side? No, it's a great question. Listen, I think at the heart of why people come to search and we still see this today is they want to connect to the best of the web. They want to connect to unique voices.

14:47They want to connect to authoritative sources. And that's critical in our product development. When we talk about AI experiences in search, we are constantly talking about how to weave in the web ecosystem more effectively into the response and how to enhance better connections for people. So, for example, Randy, we just recently launched news in our AI overviews and AI mode products. We are constantly investing. I think you saw us talk about this, really bringing in web content via links in the response. And then on top of that, these models, because they're issuing more fan outs and asking more questions to organize information better, the result is that we can connect people to the actual publisher or unique voice that they want more effectively.

15:34Randy Silver: one of the other interesting things is and i recorded another episode last night with someone and he was talking about the fact that paradoxically sometimes a lot of the llms the more that you interact with them the worse the experience gets the more they know about you the worse it feels sometimes how do you keep the experience improving as you as you said with personal intelligence you know more it should get better it should feel more like a trusted a companion or a friend, but most of the time it doesn't. How do you deal with that? So I assume what he was referencing, I can't put myself in that person's shoes, perhaps over personalization.

16:12Is that what he was getting at, Randy, you think?

16:15Randy Silver: I think we went by it fairly quickly and it was something I didn't really push him on because it was tangential to the rest of the conversation. But I've definitely seen it where there's so much information and there's too much context or it ends up going down a very specific rabbit hole and losing the broader perspective. So, and things like that. Yeah. Okay. And I'll probably qualify that under the umbrella of over-personalization. That can happen, right? Like making associations that are not relevant. I'm assuming that's what they're referencing. Listen, I think that this is where we really get into a combination of like traditional quality hill climbing combined with actually having trusted testers use the product and tell us where the failures are.

16:59Like we're constantly actually receiving feedback, whether it's from public users giving us feedback through thumbs up, thumbs down, whether it's from actual like beta testers that are using the product and giving us feedback. And so that's how we continuously collect signals, especially something like personalization, where you need real users to use the product and tell you yes, No. And they'll do that. They'll do that through thumbs. They'll do that with a follow up statement like you got this completely wrong. You'll do that with actual user studies. So we use all of these various methodologies to constantly quality hill climb the product and ensure that that doesn't happen.

17:40Or ensure we're continuously progressing because it is hard to say perhaps it never happens, but that we are iteratively getting better and better at it.

17:49Randy Silver: Let's talk a little bit more about the approach that you have to beta testing. Because most of my career, I've worked with people who are either in B2B or more niche B2C. So what we're doing is solving a very specific problem. And we can test very specific slices or narrow goals. But this is the most wide open thing in the world. This is tell me anything and I want to organize the world's information. how do you approach testing and ensuring you have a good experience with something that that wide that general general you know what let's just say that why because i'm not going to get the word out there um yeah i i love this question because i think when i think about even my organization randy there's uh two tracks to talk about there's i'll talk about more of the traditional and then there's more of the very iterative frontier uh and both of those require different things.

18:46I think what I mean in the more traditional is you have a product out there, that there are billions of people using it. You continuously need to make that product better. And that's where there is an obsession over identifying places where the quality is not right, watching people use the product and every friction point that they're hitting, understanding that and just maintaining a constant bar of excellence and craft that we have to hit. And then there's the second bucket, which is you have an idea. And you're like, I don't know if this is a good idea, but I think there's a there there and it'll resonate.

19:22And I think in the world we are today, what I tell my team is you should go from idea to define it, to prototype, to demo with real data, to if there's a there there path to beta really quickly. Do not spend weeks and weeks debating and putting decks together. And we're in a world where you can, you should always anchor on hypotheses. Like, why do you think this matters? Why do you think this has real opportunity? That early thinking is worthwhile. But now you can spin up a prototype really quickly to even say, hey, this is what my idea is. This is how I think it. You could connect it to real data and scrappy demo, just the bare minimum of what you need very quickly gives you the sense of what's easier or hard about this.

20:08Does this feel good? Is the data completely wrong? and that either kills it and there's no there there or starts getting the momentum in the organization of wow I think there's something and then all the way very quickly to like a beta handful of testers that can that can feel it out and so I really think about making sure that that process is happening quickly I think one of the worst things to do is to spin and lose cycles in debate these are not one-way doors so that's really something on that second track I like to see happen really quickly. Of course, like I said, early thinking is important because you should really ask yourself, do I think this is a big opportunity?

20:46Do I think this matters? Will it really make a difference in people's lives?

20:50Randy Silver: Quick one for the product folks listening. AI can draft the PRD, prototype the idea, and help you build almost anything. But knowing what to build, that's the hard part. Jira Product Discovery helps product teams bring customer insights and ideas together, prioritize what's worth building as a team, and create living roadmaps that connect directly to delivery in Jira. So when more ideas are possible than ever, your team can focus on the ones that matter most. Try Jira product discovery for free today at atlassian.com slash MTP. I want to dig deeper into your team and your process, but let's use a concrete example, if you don't mind, is there something that's happened, say, in the last few months that was a decision that you and the team made that, based on user feedback, based on that experimentation, based on the beta, that was really surprising to you, that went against what your hypothesis was or what you expected?

21:49Yeah, so I think if I think about some of the experiences we've launched recently, you know, we talked about GenUI coming into the product and the advent of simulators and visualizations. That is a skunk group that first had this there, there, we should invest in here and we think there's an opportunity. But very quickly, I went back from a talented small group of folks saying, hey, these are the kinds of use cases that we think if we can create a simulation, it would really change the way that people would think about a product. And from there, you start seeing a little momentum and in a clear definition of what good is.

22:30And since getting that product out, it has truly resonated with users. But the time from ideation to actual launch was pretty quick. And you saw that it was just a clearer way for people to understand a concept because they can interact with it, because they could visualize it. That's something we never did in search before. And so that's an example.

22:52Randy Silver: What kind of things were you simulating? What kind of experiences were people trying to find? I mean, everything from like, at what angle should I do a basketball shot to explain to me the rate at which if I switch sort of my financing investment, it could affect my long-term savings to sort of examples there of like how you can toggle and play with elements. Similarly, there's another one that is near and dear to my heart, but like this idea of can we overlay audio on top of images? And we've now launched that. And it's like really resonating and explaining complex concepts to people. What is photosynthesis?

23:30It's a whole new way. It's as if you have a teacher in the classroom. And these are the kinds of ideas that I think are pushing the boundaries that as we got out there, we're getting really quick early feedback from consumers that it's responding to them.

23:42Randy Silver: Okay. So those, the way you explain them, they sound like no brainers, But were they when they first came up or were there any that came back that were really surprising and challenged the team's expectations? I think that all of these things in Notion, you look at it now, you're like, slam dunk, we should do it. But what's really the challenge of these things is how difficult is it to do and can you scale it out is often the question, right? Like it sounds really interesting because you do it at a high quality bar. What does even a high quality bar look like that resonates? That's where the early thinking really matters.

24:14Like simulations, I totally get you to look at that and you're like, yeah, who doesn't want to play with a simulator to do a basketball shot? Well, first of all, it's not obvious in search if they would want to do that. But second of all, I think the reason that these products landed is because the quality of the simulation, the definition of good was really high. The craft that we put out there. If we put out a subpar version of it, which is the early days as you're developing in order to get there, the product would have fallen flat. And so really understanding what that bar of excellence needs to be for a first MVP and getting those signals really quickly, that's what makes some of these things difficult.

24:49Randy Silver: When I talk to people at Google or Meta or OpenAI, places like that, the interesting part is for them, tokens are almost free, at least in the development side. But now you're balancing, you're running, making simulations in real time at scale. That's not free. That's going to generate a cost. So at what point are you balancing the cost of providing this experience versus anything else? You know, there are plenty of people who use customer service bots in the early days to generate code because they were jailbreaking the LLM in there. Where is the balance between something that is a free experience and good enough?

25:28Yeah, I think actually, Randy, all the time that is a question we should be asking ourselves, because the same I was telling you in this new AI world, people can build really quickly and prototype really quickly and demo really quickly. It's fantastic. The other flip side of that is you can have a ton of ideas and a ton of things you can do. And I think as leaders and in conjunction with our team, the question is, is it worth it? Is this thing worth it versus these five other opportunities that we have, whether it's the cost of people, attention, tokens? And so that's the stress of in a world with AI where you can build really quickly.

26:07The flip responsibility I think product managers have is build what really matters and that makes a difference. And I think associated to that, build it well, because it's really exciting to get something out really quickly. But it's that kind of craft and excellence that consumers see it. And when you don't do it right, they will tell you you did not do this right and they will stop using it. And that's the bar we have to continue to maintain and the culture we have to maintain in a world where everyone is moving really fast and has a ton of ideas and they're really excited. So that's how I really think about that tradeoff.

26:40Randy Silver: The pace of innovation is really relentless. The expectations are relentless, at least for prototyping. prototyping and then there's prototyping and then there's getting stuff into production. And we can talk about that challenge in a couple of minutes. But let's talk about the team that you've got, because I'm seeing lots of challenges about how teams are structured these days. And how do you approach that with your team? What does your team look like in terms of disposition? What is the nature of a product team? For a long time, it was two pizza teams. And now people are going down to Happy Meal teams in some cases of just one person.

27:13Randy Silver: And what does a team look like for you? What is good? How do you ensure that you're getting quality out of your people? Yeah, I am. For one thing, I think I have the benefit of really being able to think about and craft my organization from the ground up in this AI era. And the term that I like to use, pizza, french fries, is SWAT team structure, really. It's a lean, flat team. I look for pods where I have one senior lead who I look to be really like a strategic thinker that can ensure across the board we are making the right bets and thinking about things the right way with a very strong bench of ICs, small bench of ICs.

27:57And the reason I use the terminology SWAT team is I don't think, first of all, these days you need a lot to do a lot. You don't need a lot of people to do a lot. You need the right set of people. The second, the SWAT team analogy, because I want to keep the lines of communication very flat between myself and the rest of the organization. I do not want multiple layers. When you have multiple layers, you don't have context. When you don't have context, it's really hard to move fast and ship and ship with excellence. Like you need those lines of communications to be there. If anything, just to enable your team and help them be unblocked and make sure that they have the support that they need.

28:35The second thing I think about, and I think about this not within just my own team, more broadly, is really, and I talk about this a lot, building horizontally. And what I mean by that is, you know, in a world where you have these very capable models, you really want to build at scale. I often say you want to be inspired by vertical use cases. You really want to put yourself in the consumer's shoes and think about a journey that really matters to them. But you really want to try to challenge yourself and say, can I build this horizontally? And on a team basis, the effect of that is we are working more and more in these like virtual teams, bring in the right talent from different departments into it, but you're really breaking what may have been traditional organizational silos because you're building differently.

29:27Randy Silver: To enable good people to do good work, one of the things I keep seeing is organizations that had bad practices or any of their bad practices, even if they were largely good, they're getting hugely magnified by the pace and the tools. But the good practices are going even better, but the bad practices are getting in the way way more. How have you ensured that you're doing this at the right pace and getting the right quality? What have you done with the teams to ensure that as the pace increases and everything is changing, that some of the basics are really addressed? So a couple things, and I hope I am doing this every day, Randy.

30:12One is leading by example and setting the right tone. So that bar of being the user, thinking about the underlying technology, challenging ourselves, first of all, showing up that way. Because if I do it, the team does it. And that's critical. The second is, I think while we have a view of the destination, like we have an opinion of what search is going to become in the future that we're building and how we are unique because we are search and we have web content and we want to organize it. And there's a very clear line of set of where we want to go. I really do work with the team on what are we going to do, however, in the next three weeks, three months, six months, and having alignment and that clarity there on the speed of execution that we want to maintain.

30:58But the third that I spent a lot of time talking to my team about is you are the premier consumer AI team. And what that means is you represent the user and you represent excellent. You represent craft, you represent quality. If you build that way, if you really dig in, if you understand what's going on with the user, then we can ensure we're shipping high quality product, even if we are moving fast. And so those are kind the things that come in every day and the way we work and the way I work with the PM team and the way to then work with their counterparts across the board to ensure it. Now, the other things are traditional product things that you do not let go of.

31:37You have clear hypotheses on why you are doing what you're doing. If you do not have that, you are not building the right thing. You have clear goals with your counterparts that you're all bought into. You really understand what the destination is, what the metrics are that guide you, you spend time early on together defining what good is. Show me what that product is that you think is good and show me what not good is and is everyone speaking the same language. And those are just exercises that I think are good practice. It's good foundational product work that was always necessary that we should continue to bring even in this environment of moving fast.

32:17Randy Silver: Have you had to change your processes around the pace of change or has it always been very quick for you? So, you know, it's a good question. I think that without a doubt, the speed of execution has increased in recent times. You know, I think we are shipping much faster. We are innovating much quicker. Expectation of speed of development that I have for myself that I bring to the team is much higher. I think that there is a sense of urgency to meet this moment all the time. And what's interesting, and this is what's made me really happy, is when I look at how people are showing up, they're excited by the sense of urgency.

33:00People are excited that we can do so much that in the past either wasn't possible or would be such a huge lift to get done. But absolutely, we should all acknowledge there is this sense of urgency and responsibility that is more acute given the AI shift that's happening.

33:20Randy Silver: Has the speed been applauded consistently as in getting to prototype is one thing and then getting to general availability, getting it to everyone? That's another thing entirely. So where are you measuring this? Are you measuring it, getting it out into beta and getting it into the first stage and then potentially going through more traditional stage gates? Or what's the approach and has that changed? I measure it at every stage. really because I think when I say speed of execution it really is at the end of the day what I care about is is it helping people and then did I get the signals and then am I continuously making it better so that idea of constantly moving fast with a high bar and it's hard do it do it fast and do it well but most importantly do it well do not I do not want to put out products there that do not meet the bar of excellence, the bar of trust, the bar of quality, the bar of craft that people come to Expected Search.

34:18It is at every stage of development. You know, when I meet with my team, again, we're very clear on where we want to go, but also I really bring it home to where do you want to be in three months from now? It doesn't matter what stage of project you're working on. We have folks working on very novel kind of frontier ideas, and we have folks that are working on more established experiences that are out there. It's still that question of where do you want to be three months from now? And I think that's the right way to think about it. Because like I said, the technology is moving fast. The consumer expectations are evolving really quickly.

34:52And so we need to be showing up that same sense of urgency.

34:56Randy Silver: With everything moving so fast, one of the challenges I keep hearing about, and to be honest, seeing myself, is the challenge of context switching. You know, you've got so much going on. So how are you and your team avoiding problems related to that? How are you and the team avoiding burnout from just trying to keep on top of everything at this relentless pace? If I told you coffee, would you believe me? I'm in England. It's tea here. Many, many cups of coffee. Um, so this is where I think my job as a leader, one of the critical things is to give people sort of like very clean direction and minimize the noise.

35:40First of all, to your point in a world where there's a lot happening, you can very quickly see a lot of noise coming in. And that's one of the worst ways, in my opinion, to work. It's OK to work urgently. It's OK to develop quickly. chaos, spinning, debate, that is the thing that I try really hard to spot in the system and clear out for folks so that they can do their best work. I have not seen the effect of, my team does not go, this is moving too fast, I don't like it. They're like, this is actually energetic, I get to do great work. But I think the same is true that if your team is trying to do great work and they're spending their time sort of like debating with some other part of the organization about whether we should do something or not.

36:31Or they find themselves not having resources because XYZ is not convinced. Or they're saying, hey, is this other part of the system doing what I'm doing? That's the kind of energy. That's the kind of noise that I think as leaders, we need to eliminate for people so that they can bring their best work forward. And then the results for me so far, what I've seen is people are energized. They're excited. It's not that they don't want to work at this pace. So that's what I really look out for. And listen, at the end of the day, I talk about urgency and energy. We have to build sustainably. I think I mentioned I have two kids.

37:05They matter. I need to pay attention to them. So you really have to build sustainably, which is where you got to be really, like I said, focused on what's your destination? What are your priorities? What matters? What can wait? And then help your team think through that too.

37:19Randy Silver: Okay. I think we're running out of time. So I'm going to ask you a couple of questions about your team, if that's okay. And I'm just curious about as you're hiring now, as things have changed and the pace and the skill set and the disposition of team has changed, what do you look for when you're hiring and promoting people? What are the key things now? So first of all, for my organization in particular I talk about, listen, I'm really looking for consumer AI PM DNA. I will qualify this. Given the fact that we're driving the AI experience on search, the consumer side of the definition, like I need people who have demonstrated that they can really build user-facing products.

37:59They can be the user. They can put themselves in their shoes. They can really be creative about thinking what are the kinds of products that could resonate. So that's number one. The AI component of it is for me, I'm looking for people who are really keeping up to date with the underlying technology, where it's going, what opportunity it's opening, what other products are emerging out there, constantly maintaining an eye of the landscape because it helps them come and say, I actually think we can either A, build this new capability and here's why it would matter or B, build differently. The same problem space, but let's approach it differently because the technology is enabling us to do so.

38:39And then third, this zero to one DNA, where I can find it, I really gravitate towards it. People who have this builder startup mentality, this excitement of getting things done really quickly, of making it happen, of pushing the boundaries and moving at speed is something that I really seek in the DNA of the PMs that come and join the team.

39:03Randy Silver: A couple of months ago, the thing everyone was talking about was evals. And the last couple of weeks, it seems to be harnesses and things like that. So I'm curious, in terms of the specific skill sets, the real challenge that I've seen is taking a step back and realizing that the LLM, the agents that you're working with, they're a bit obsequious in a lot of the time. They'll tell you what you want to hear. They don't push back very well. So they will absolutely accelerate what you're doing, both the good and the bad. And it's a lot of rigor of your own practice to do it well and put the guardrails in place.

39:43Randy Silver: Are there any key skills that you work on with the team or that you're seeing or that you'd advise people? This is the thing that if you double down on this, it really pays off over the next few months. So I think that where, for example, we're spending a lot of time having conversations is where is it that these tools are helping us accelerate the way we work? However, with that, as a team, we often aware of the limitations of them. and what we have found is they're very useful in getting a lot off the ground whether it's like you want to do logs analysis and now you can actually do it in a much easier way or you want to build a prototype and you can get your hands out there prototype perhaps less of a better example of things speaking back to you but whether it's you want to sort of brainstorm an idea it is great at this stage for like getting things going but but we still recognize that we have to challenge and cross-check and verify.

40:36And that's okay. There's a lot of value in lifting that load and helping structure the thinking, for example. But also a lot of discussion on the team is where have you seen the limits? How are you then bringing that into your workflow to ensure that you stress test them? And I'm sure those limits are gonna change in the next few months. And so keeping that conversation in our team organization when we meet, when we have our group meetings and we're sharing ideas of like, well, how did you go about this? And how did you go about that? And that's where we can also have an honest conversation of what worked, what didn't work.

41:07Where is this really adding value in your productivity? Where is it less so? So that's a constant conversation that's being had. And like I said, when I look at how we built a year ago and how we're building, how we built six months ago and how we're building today, and I am positive, I joke about how my job feels like it's changing every six months. We are going to build very differently three to six months from now. that kind of knowledge sharing is integral in an organization right now because everyone is

41:35Randy Silver: coaching everybody and how are you thinking about longer term resource allocation in that it used to be that you would have teams organized around certain customer segments or feature segments or parts of a user journey things like that and you could plan i mean i hate to say gant chart but to a certain degree you'd say okay they'll be doing this then they're going to roll off, they'll have launched this feature, they'll have accomplished this goal, and then they can move on to the next thing, or we can trade some people around. But now you're having to deal with model drift, you're having to deal with constant maintenance of something.

42:10And really, you own something and you have to continue to look after the existing experience as well as the evolving

42:17Randy Silver: experience. Does that change the way you think about longer term team disposition? Or has that come up yet? So I think, I think the question that you're asking is with this constant shift that's happening, how do you build sustainably an organization around it? And I think for me, it's actually defining areas, the areas will stay the same. So let's say I have an area that's really accountable for the end to end response that the user sees. Let's say I have a person is really an area that's really responsible around just the overarching sort of like consumer product, the framework of it. So I have an area that's really responsible around like long running journeys.

42:58The areas for now are at least sustainable enough for a period of visibility that's not like three months. It's sustainable. I actually think these are areas that we're investing. Then within that area, it's okay that the way you build and solve the problem is varying. The model technology is varying. The technical approach is varying. Again, the challenges are evolving. That's okay. But at least your area is somewhat sustainable because then you can formulate an opinion of where you're going and what good is. I want some amount of longevity. Now, I can't tell you if that longevity is two years, three years, but it's absolutely longer than six months, nine months, because I want people to think about the problem space, have accountability for it, go deep in what they think success looks like.

43:48And so that kind of continuity really matters to me as I do the org design. And as I then related to that, organize the SWAT teams that I want, because those SWAT teams build muscles and expertise in each of their areas.

44:02Randy Silver: Sufi, the time has flown by. We've gone to the top of the hour. I'm sure you've got 10 other meetings you're supposed to be at. So thank you again. This was fantastic. Thank you so much for having me. I really enjoyed the conversation today. The product experience hosts are me, Lily Smith, host by night and chief product officer by day. And me, Randy Silver, also host by night. And I spend my days working with product and leadership teams, helping their teams to do amazing work. Lou Ron Pratt is our producer and Luke Smith is our editor.