What is AI product management, really? Jonathan Evens (AI Product Lead, Google DeepMind)

0:00Feeling stuck in the middle of your career? The Mid-Career GPS Podcast helps mid-level professionals navigate leadership, office politics, networking, interviews, and their next career move. Across more than 350 episodes, host John Narrell and his guests share a weekly dose of practical guidance to help you stop feeling stuck and start moving forward. Search for the Mid-Career GPS Podcast wherever you get your podcasts. Follow the show and listen today.

0:55business, career, or team. Find your next useful idea. Listen to the Business Takeaway podcast wherever you get your podcasts. Click the link in the show notes.

1:09Jonathan Evens: If you train the large language model or an AI on all the data up until 10 ,000 years ago, right? And then you kill all the humans from 10 ,000 years ago and you let it run. and then you then have the real world where we didn't have AI and then you look at both outcomes. Is our world much more exciting and interesting than the world where AI was alone? And I think the answer is always going to be yes.

1:41Hi, Jonathan. It's so great to be chatting to you again. Last time I saw you, it was on the main stage at Mind the Product. How are you doing?

1:49Jonathan Evens: I'm doing great. enjoying the summer, getting a tan and yeah, it's been great. Well, we're going to carry on the conversation that we kicked off when we were at the Mind the Product earlier this year. But before we do, it would be great if you could give our listeners a real quick intro to who you are and what you do in product. Yeah, absolutely. So I've been in product for a little over a decade. My background is mostly in general engineering science. And I really got into product thinking about, you know, what would be a job where I could use my technical skills, but not do those all day. And at the same time, use some of my social skills.

2:32Jonathan Evens: And then I kind of read this job description about what a product manager was. And as I read it, I just thought this is a job for me. And that's kind of how I got into it. And then, yeah, over the last decade, I've really been focused on trying to apply machine learning AI to different industries. So I first did that in the smart grid at a company called AutoGrid. Another industry I did that in was the geospatial industry at a company called Planets, where the idea was, how do you detect buildings and roads automatically across the world with a constellation of satellites? And then more recently, I've been doing this at Google DeepMind.

3:14Jonathan Evens: First, I started on recommender systems. and then I worked on Google Search, the transformation with AI overviews and AI mode. And now I do a few different things, live avatars, things of that nature. Awesome. And I feel like you're the, you know, when we talk about AI product managers, like you have more of a claim to this title than most with the history that you have. But also you've launched some incredible AI features, as you mentioned, AI search at Google, but then also with the Evans Foundation as well. Do you want to kind of briefly touch on that? Yeah, so I would say there's, you know, as I was saying basically on stage in London, in the Barbican, there are mature products like Google Search, where the idea was really to launch an AI feature on an existing product that actually was itself being transformed.

4:11Jonathan Evens: So it's more an AI feature within an AI product, actually, that had already started the transformation. And then at the Evans Foundation, it was much more around how do you build an AI product from scratch where you don't really have resources, basically, and you kind of have to figure it out with limited resources. And there the idea was really how can we try to build tools that strengthen democracy? And so the tool that we focused on or that I focused on in this talk was the idea of a digital citizenry, which would be, you know, if you think about it as a civic sandbox where you can kind of drop a policy and you would kind of get a pre-run of how the public would likely react, whether the interest and the values of different individuals in society would be reflected in that policy and why they would or wouldn't agree with it.

5:03Jonathan Evens: So that was kind of what the talk was about. But I can also answer your question about product management and, you know, what I think it is, if that's of interest. Yeah, that's exactly where I was going to go next. So let's do it. No worries. Yeah. So, you know, I would say that it's interesting that the term AI product management for me is quite weird because I see it pop up on my LinkedIn feed just all the time. and I would say up until maybe 2018, 2019, I even hesitated to put that on my LinkedIn or in my resume. And I remember thinking, I think there was a transition basically where you would actually call it machine learning product manager or modeling product manager.

5:53Jonathan Evens: And then suddenly the term AI started becoming a little bit more used, but just in the industry, not like today, obviously. And that's when I kind of always put the slash. So I put machine learning slash AI product manager, you know, from 2018, 2019 onwards. And yeah, I just thought it was interesting. And then actually the years before that, you know, there was a difference between saying you were a machine learning product manager versus a deep learning product manager, because deep learning was really the more consequential technology. so yeah so you know just historically i just find this fascinating that now everybody's talking about you know ai product management what do you think qualifies someone to be able to say that they're an ai product manager today because it feels like everyone is getting familiar with and learning ai tools and the rate at which technology is changing you know everyone's learning so you You know, is there a level of technical knowledge or whatever that people should be having in order to claim that they are an AI product manager?

7:00Jonathan Evens: Yeah, I think that's a good question. I would say there's two types, to be honest. The first type is really what I was doing before. I would say before it was called, you know, before the last three years, basically, which I would just call a modeling product manager, basically. And so when you're a modeling product manager, you're really responsible for what is the model supposed to do? How do you measure whether it's doing what it's supposed to do? And then what kind of error rates are acceptable for that model? And I think that in the Gen AI world, for me, it really is about finding, defining, measuring these core capabilities of the core model.

7:48Jonathan Evens: And what do I mean by core capabilities? I mean things like factuality, you know, whether it's handling long context very well, whether it's able to reason really well. And so that modeling PM role that I'm talking about, I think, is really centralized in the frontier labs at this point. And so I would really, and then by the way, you even have, as opposed to how it was before, because the model has really become this complex thing. you don't have just one modeling PM. You have a modeling PM for all of these capabilities. You have a modeling PM for pre-training. You have a modeling PM for post-training, et cetera.

8:32Jonathan Evens: And so I would say that's kind of the evolution of the previous role that I had, but in this new gen AI world where you have these highly capable models. So I think that's the, I would say that is the real claim to what I would call a modeling AI product manager. And then I think what's changed is really the fact that now any product manager can use AI as part of its toolkit. Right. Now, the reason why everybody talks about AI product management so much is because the tool is so powerful that it completely changes the experience. And so essentially, if you want to build your product, AI is becoming the center of it.

9:20Jonathan Evens: Right. Yeah, I want to ask you, so it sounds like you're making a distinction between people who are building AI systems and then who are building products and features using AI. And especially on the latter side, you've already touched on there's machine learning, there's LLMs, there's Agentec, there's RAG, there's any number of other things. And I think what most people are doing is they're just opening up, you know, co-pilot or Claude or something like that and using it as a thinking partner or go build this for me. So when you say somebody, you're someone who worked on the Maker's Manifesto and contributes to that as well.

10:00So I'm just curious, when you're talking about an AI PM or an AI enabled PM, what does that actually mean? And is it enough to just say, I use an LLM to help me with my thinking or to do some of my tasks for me? Or is there something more traditional? Yeah, let's start with that. Yeah, absolutely.

10:18Jonathan Evens: So I think there was this first category, first type I talked about, which is the modeling PM. And I think for that, you need technical skills in a sense of you need to understand machine learning to some extent. And I think, you know, the same skills that you needed in the previous era, you still need those in the current era to be that type of PM. But again, the number of roles available are limited because it's all in the frontier labs, right? Then I think what's exploded is basically the AI product or the AI feature PM, basically. And in that case, it's basically AI is a technology in service of your product, of your application.

11:00Jonathan Evens: And there, I think the superpower is really to be close to the end use case, to be close to the vertical, the domain expertise. And I think it's more important than ever, actually. And I feel like people are not talking about this enough to understand all the intricacies of the workflows of your industry, of your vertical, of your use cases, and then know how to use AI to automate or do or augment some of those workflows. I think that there's a tremendous superpower there that I think people are not talking about enough. and I think that there are lots of things that don't change in that world, right?

11:46Jonathan Evens: So for that type of AI product or AI feature PM, and that is that you still have to design the whole experience around the model, but with a user interface, with a user experience, etc. And while the original model can have 80 % of the capability, it's maybe the last 20 % that you really have to work on to get it right for your use cases. And then you were mentioning the AI-powered PM or AI-enabled PM. For me, I don't even think we should really talk about that, actually. Maybe it's, I don't know, I have a reputation of being pretty direct about this. But for me, that's basically using AI for knowledge work, whether you do it in product management or as a financial analyst, it doesn't really matter.

12:41And I think that was the question I wanted to get to. Is that person an AI PM or are they just a PM who is using AI as part of their tool set?

12:49Jonathan Evens: No, they're just a PM using, I would say generally they're a PM that is using these tools to improve their productivity. That being said, there are new skills that I think are needed or can be really helpful in the second role that I described, the AI feature, AI product manager, where, you know, for example, vibe coding or, you know, learning how to prototype pretty fast. Or another one is, for example, if you want to, So one of the core responsibilities of a product manager and one of the added values of a product manager is to bring the right data to the conversation, to turn intuition into something that is backed by a little bit more evidence.

13:41Jonathan Evens: You need the product sense, you need the intuition, but your ability to bring data to the conversation to guide that intuition is quite key. and now you kind of have a data scientist you know in your pocket that can just go and grab that data and do some things for you whereas in the past you would have had to rely on you know a lot of other people even if it won't be perfect it'll still give you a lot of insights and I think there's still skills there that you can apply so those skills are part of an I would say just have a pm actually just a pm yeah i love that we've spent like 15 minutes almost on the very first question that we had but it's great to it's great to have this conversation um one of the things you kind of touched on there is measuring success and kind of you've obviously worked on two very different ai features into very different businesses and products google which is a huge organization with like billions of customers and then the Evans Foundation I'm not sure how many customers you have but I imagine it's much smaller yeah early access program at the moment yeah so so you know different sort of ends of the spectrum for sure when you're thinking about measuring success though I imagine you kind of approach it in a fairly similar way and you know measuring success for those AI features and that the AI capability because there's so much sort of personalization or variability in the output depending on the data that you're putting in so there's been lots of talk before about like evaluations and you know making sure that you can evaluate your the output of your AI capability like within the feature in the product but as someone who has got lots of experience of this like how have you thought about it like how have you approached it in those two different situations?

15:35Jonathan Evens: So the idea of measuring success, you would say, or? Yeah. Yeah. So I think of it in terms of metrics, right? I would say. And I would say that the thing that hasn't really changed, and maybe I'll speak in the case of Google search, is, you know, what are your North Star metrics? That hasn't really changed necessarily that much, I would say. So it's still, do you have active users that come back daily or weekly or monthly? It's things like, does your product actually fulfill user needs depending on how you measure that from a North Star metric perspective? So if you're in the case of Google searches, are people coming back because their needs are fulfilled by Google search?

16:27Jonathan Evens: If there were YouTube, for example, So it might be, you know, did you find the video that you watched satisfactory? Did it satisfy you to watch this video, right? Or do you regret it, for example? And then there's also, are you satisfied? So that's more on the Google search side, I would say. Are you satisfied with the information that you got? Is it of high quality? Do you trust it, right, for example? So those things don't really change, I would say, the North Star metrics. But I think what's really changed are the proxy metrics underneath because your whole product is transforming, is changing.

17:06Jonathan Evens: No matter whether you have an existing product or if you're starting to build a product from scratch, it's a new tech stack. It's a new way of approaching, of using the capability, right? And I would say in that case, what you're looking for are things like side-by-side win rates, right? You want to basically continuously evaluate the outputs of your models and see, is it better than the previous one? Is it better than this competing product? Is it better than a baseline that I've established myself, basically? Then you also want to kind of measure the user ratings and the user responses. Are they copying the answers?

17:50Jonathan Evens: Are they following the recommendations? Are they clicking on the links? Are they giving it a thumbs up? if, for example, you're doing AI-generated music, are they listening to the song all the way until the end, for example, right? So all these signals are extremely valuable, especially for what we call the reinforcement learning loop or reinforcement learning from human feedback loop. And then one aspect that I find interesting that, you know, we hear all the time, like, evals is all you need. And anytime I start a new, like I work on it, which has happened quite a bit in the last year and a half.

18:28Jonathan Evens: I've worked on quite a few different new products, let's say. And every time the thing that comes before the evaluations are what are the product principles that your evaluations are going to build on, basically. And so before even the evaluation, the real secret is what are your product principles, right? Like what is it that you're trying to create as an experience? And then the evaluations are the first translation of this into something that you can use to teach the model, basically. Let's stick into the principles for a second. You're talking about Google and YouTube. These are established products, established areas.

19:17And I think people might be surprised to hear that the underlying principles might still not be clear on some of these things. So what was the experience? Can you be a little more specific about a time when things were a little confused?

19:31Jonathan Evens: So you mean confusing in terms of product principles? If the first thing to do is to get the principles aligned and nailed down. I'm assuming that means there was a time when they weren't. Yeah, absolutely. Well, so one example that's pretty obvious in the case of Google search is why is it that people went to Google search in the first place for, let's say, you know, 2000s to 2022, right? And I think that the reason people came to Google was because it had almost always, of course, not always, but it almost always had trustworthy information ranked first. And you were able to get to that information pretty quickly.

20:17Jonathan Evens: Right. And so there's this trustworthy aspect to the product that is quite key. And at the same time, large language models hallucinated a lot. Right. So going back to the product principles, one of the questions that came up early on was, how do you reconcile these two things? You want to basically build on this amazing capability that large leverage models have, which is summarizing, synthesizing information that's scattered across the web. Amazing. I won't have to do this work. But at the same time, we have a massive issue if, you know, one out of 10 times or, you know, one out of five times, there's a problem there.

21:04Jonathan Evens: And I think, you know, I think the principles there are actually machine learning principles, like machine learning product principles that we had already before, which is you have to make an educated guess based in the case of Google searchers, you know, a lot of history behind it. so you can really look at a whole history to figure out what those numbers should be. But the important thing is the error rate. You want to kind of know what is the error rate that is acceptable for your product to be released at what scale. And so that's really, I think, the approach that Google took. First releasing to power users in what was called the search generative experience.

21:45Jonathan Evens: And then, you know, just getting better, you know, over time at this factuality problem and studying this problem and seeing examples of where the model fails. There are some examples that nobody thinks about, which is you might ask, for example, for the date of construction of the monument. And then over the web, you'll have five trustworthy sources that will basically tell you three different dates. and some will overlap. They'll give you ranges and some overlapping. So what should Google tell you in that case? What should be the answer? And so it's like, how do you design with product principles in mind around these nuances so that in the end, the answer matches that principle of trustworthiness?

22:33And Jonathan, in your talk at MTP, you referenced kind of working with AI research, like doing the sort of AI research as part of working on the features both at Google on search and also at Evans Foundation. Is that the kind of the types of things that you're looking at when you're doing AI research? Is it essentially sort of an MVP and then working with real users to get some proper feedback? Or is there something else to the AI research?

23:05Jonathan Evens: I think it goes back to this separation or the different types of AI product managers that I talked about, I would say. So when you're in frontier research, what you're really trying to do is a breakthrough is often defined by whether the model is beating a benchmark. That would show basically that you've reached state-of-the-art on a certain capability. But the reality of large language models is that they don't, while they generalize, if they didn't generalize, then we wouldn't have all this fuss about large language models since three years or four years. So they do generalize, but they don't generalize as well as it is claimed in the public sphere, in the media, right?

23:49Jonathan Evens: And, you know, that means that there are going to be real world situations where you would expect them to exhibit this particular capability, but where they actually fail, right? And they don't exhibit that capability. And so I think that when you are a product manager that has access to this research, often your job is to think about that model confronting reality and the messy reality. And then I think that the way that you're seeing that it's different is because we have the need for forward deploy engineers, basically. Right. And so that's a really good, that really shows you that the AI research is key, but at the same time, the use case and the messiness of it requires a lot of applied work.

24:42Jonathan Evens: And then the other thing is like, if you look at, for example, math problems, there was just actually some news about math recently with 10 problems that were solved, right? Right. But generally, I think with what we've seen with math problems is that, yes, the model is able to solve math problems, but we have a lot of benchmarks with those. But at the same time, it's still struggling to help mathematicians solve problems that are the ones that they're actually focused on. And so, yeah, that's kind of how I would think about it. Jonathan, you talked a moment ago about messy reality. And I know one of the things you've experimented with in the past is, you know, you're working on things that are massive in scale and doing research at scale is difficult.

25:27It's always been difficult. So you've been playing around with synthetic users and and using research at that kind of scale. Talk a little bit about that. How do you avoid, you know, the problems of smoothing things out of you generating really reflecting the messiness of people when you're trying to use synthetic approaches?

25:46Jonathan Evens: Yeah, so I talked about this a little bit in the talk, because actually in both cases, we have synthetic users, so both for Google search and personalization, but then also for the synthetic citizenry at the Evans Foundation. I would say where they generally are helpful and add value, the synthetic users, is when you have cold start scenarios. So basically you don't have much data and you don't have a product or you don't have a feature yet. So you're kind of starting from scratch. And the other really important case, which I talked about in the talk actually, was when you have privacy concerns.

26:34Jonathan Evens: Because actually with synthetic users, you can kind of do whatever you want because they're not real. So you're not running into these privacy issues. The other thing that's very helpful is a little bit like auto raters, which basically are large language models that help you rate or judge large language models when you train them. Synthetic users can really help you with automated regression testing, basically. So they can really help you understand whether if you change something, you've lost on a particular capability or you're doing worse on a particular use case. And that's why they're really good for these like edge cases that you want to make sure you're doing still well.

27:24So are you going to do you primarily use them to to help inform you and help refine things? I guess the question is, would you ever feel comfortable shipping something that had only been seen by tested with synthetic users? Or do you always go to humans as a final pass?

27:41Jonathan Evens: I think with trusted testers, but not trusted testers, consumers. So, you know, B2B. Yeah, I would feel fine if, you know, there's trust with those users and you're trying to get feedback. then yes I wouldn't have a problem with that because I would consider I would consider the PM actually as the the first user basically the first human user that will really poke with it and and really chew it as much as possible before they put it in the hands of these real users and then that helps you iterate faster so and that's actually what we're doing in the case of the the digital citizenry actually yeah and again kind of like talking of trust um that's like such a key element when you're introducing ai features either from a b2b point of view or a b2c point of view what's been your experience with kind of making sure that you are keeping the trust of the the audience like do you have any sort of tips or tricks for people to take away and consider when they're building AI into their own products?

28:49Yeah.

28:50Jonathan Evens: So I guess I'll still lean on the historical context of all of this, actually, because I just don't think that things have changed as much as we think. So when I was at Planet and we were building a road and building change detector, right? So you basically had satellites that were scanning the earth. You get all this imagery. every day you get the whole world imaged and you so think of it as a frame and then from day to day right you have new frames of the same place right and you're trying to detect changes right but then you have all sorts of real world messiness coming into the picture and those would be things like clouds or you know maybe the atmosphere is a bit different or maybe the the shade is different coming from the buildings you know that are that are coming across the road, for example.

29:45Jonathan Evens: So there's many, many factors that come into play. And we were running into these problems, which was that the accuracy of the models, which was to detect these new roads and these new buildings every day or every week, right? We're trying to basically pinpoint them for mapping customers and so on. And the issue was our customers were analysts in those companies or these public institutions. and they wanted to know whether they could rely on the detection or not, which is very similar to what we're talking about now with hallucinations. It's just that it moved from this select few analysts to the general public now because everybody's using AI.

30:28Jonathan Evens: And the answer was you have a product card that tells you how to deal with the product, basically. When can you really trust it? When does it kind of fail? What have we not fixed yet? and they would have this card and they would be able to kind of have a better sense. And then they can use other tools to decide whether they want to dig deeper, basically, into the detection. And I think today we actually have a very similar problem, but we just have to adapt it to these consumer-facing products. And so that means that trust for me in the case of Google search, It's really about factuality and grounding, ensuring that the sources are, you know, that the information is coming from trusted sources.

31:17Jonathan Evens: When you go to Google, you know that the AI, you know, is really taking the information from the sources. It might still hallucinate, but it will not come up with it from its parametric memory. It will take it from the documents. Then there is a user experience aspect, which is how do you show citations? How do you kind of, as a user, are you able to quickly, you know, with a quick glance, are you able to see, can I trust this information or not? I think the UX aspect of this is very important as well. For example, if something is highlighted, you will have higher confidence in it, right? Being true.

31:52Jonathan Evens: Whereas if it's not, you might think, oh, okay, maybe I need to, you know, verify that if I'm not sure. And then where the citations are placed and which website is it coming from is also quite important. And then the last one is when you ask questions that are politically sensitive, where the answer is not necessarily super clear, it's about punting or redirecting to links or saying that there are different views on this topic, which is in itself also a problem, by the way. Because in some cases, you might give a measured response, but actually one of them is not the truth and one of them is the truth.

32:31Jonathan Evens: So there's still work to be done, but I think it's with this approach that we can get there. Jonathan, so you've had these vast range of experiences in how you build products. And at the Evans Foundation, it's small. I know you don't even have a full-time dev working with you on this initiative. and at Google you have, at least from the outside, infinite resources, infinite tokens, infinite people, or at least incredible people in team sizes. But you've also been doing this for a number of years and the way that tools come in, we've got some people theorizing that the ideal version of a product development team is going down to a one-person team.

33:11Other people think it requires a certain number of people for friction and discussion. Other people are keeping the two-pizza team but increasing the size of the responsibility for the team. I'm just curious a little bit about your experience with this between Google and the Evans Foundation. What does a team look like these days? What is the difference in your approach to building? Where do you think this is all going?

33:35Jonathan Evens: Yeah, I think that's a good question. I'm not sure I have the, I don't think I have a definite answer, to be honest. And so maybe I'm just going to be, you know, in French you say when you don't answer yes or no, you're from Normandy. That's the expression. So I'm kind of going to give an answer for Normandy. But I actually think it's probably the truth. I mean, that would be my bet, is that you'll have kind of a gradient, to be honest. I think there are some problems where you can build it all on your own. And if you're a developer already, or I should say a senior developer, so you have more than five years of experience and you've worked in production systems before and so on.

34:21Jonathan Evens: I think that you can get very far all by yourself, to be honest, on some problem. I think if you're trying to do the digital citizenry that we're doing at the Evans Foundation, I think it's going to be a bit harder for you to do it all on your own. And that's what we thought. And that's because you don't just need the dev skills. you also need a principled approach. You need to think about what the product is going to look like. You need AI research because it's an unsolved problem. So there's a lot more iteration going on. But from our side, from the foundation's perspective, basically, so Matt, who is basically our tech and democracy lead, he's de facto acting as a PM with some cloud magic dev skills, right?

35:16Jonathan Evens: Or other large language models. And, you know, he's been able to do quite a bit. And that means that we only ask for AI research when we need the help, because we've identified problems that we cannot solve ourselves. At a larger company, let's go stick with the Google type thing. At larger companies, where do you think this is going to go? You said you've been working on a range of initiatives over the last year. Have you seen a change in the way that team sizes being approached or team makeup? Yeah. So the interesting thing I saw over the last, let's say, you know, two, three years is really the move from what was traditionally machine learning engineering, which is hyper parameter tuning.

36:02And yeah, basically, you know, really

36:05Jonathan Evens: being a model, a model engineer, right. To a lot can be done with prompting. A lot is about, writing these IPython notebooks and just going through, iterating on your prompt and going through the outputs and building the eval set and seeing where you're going with that. And that's where I do agree that the roles really blend, basically, because engineers will handle an important part of the evaluations, but not all of the parts of the evaluations. And that's when we go back to this product principle approach, because the product principle approach is true product management. And that has to influence the evaluations, right?

36:51Jonathan Evens: So maybe I'll try to make it concrete. If I'm trying to build a generative media product, which, you know, I won't get into the details of what it does, but let's say it's visual, you know, it's video based and so on. and some of the evaluations are going to be about, you know, how realistic is, you know, is this part of the video, right? How, like, is the image coherent, right? So those are things that are a bit more technical and that engineers can really, I would say, handle. But then there are other parts which are more related to this product sense, which would be about what, I'm just trying to think about something without sharing too much confidential stuff, but it would be more about, for example, what kind of movements should we prioritize, right?

37:47Jonathan Evens: Because those movements are key to those types of industries and can really create a difference for their use cases. And so that's where I think the PM product sense and product principles comes into play. Anyways, and so going back to your question about the team size, I think there's a lot of overlap and I think there's just going to be a lot more prototyping going on. That's what I think. I'm looking forward to coming back to this interview in a year and saying, oh, that's what he was talking about.

38:23Jonathan, this has been so good. Thank you so much for sharing your time with us. I know you have a household full of screaming children behind you. So I'm super impressed that you have not been distracted by them. They're probably all AI product managers in the making now.

38:41Jonathan Evens: Machine learning product managers. Machine learning, yes, that's right. So I wasn't paying attention. I guess one more kind of quick question before we go. Of course. you know, for everyone sort of listening to this, what would be your ask of them when they're looking at building AI features into their products or, you know, considering how they use AI in the future? To my kids? Oh, no, not to the kids, but to the audience. But now I want to know what you would ask the kids to do. Okay, maybe I'll answer the first part to my kids and to the audience. I think for my kids, I would say, I think the two most important skills are critical thinking and being able to adapt and learn quickly.

39:31Jonathan Evens: I would say those are the two things that I think will not change. I mean, unless you think that the jacked frontier of artificial general intelligence is going to eventually, you know, supersede us in everything, which you know in my personal opinion is still an open question to be honest um i don't think anyone has the answer some people have beliefs so i'd say those two things are still um critical and then another normandy answer for that one well i i would be more on the i guess i i think there's more to i mean anyways we're not going to get into the you know the whole philosophical debate, but maybe I'll just state it like this, which is something I said actually two, three years ago, which is true intelligence is if you train the large language model or an AI on all the data up until 10 ,000 years ago, and you said it was super intelligence, and then you kill all the humans from 10 ,000 years ago, and you let it run, and then you then have the real world where we didn't have AI, and then you look at both outcomes, is our world much more exciting and interesting than the world where AI was alone?

40:49Jonathan Evens: And I think the answer is always going to be yes. That's what I think. So in that sense, I think we'll never reach true, complete super intelligence. That is the philosophical part of it. I'm just glad we're not going to actually run that experiments no we're not we're not we're not uh so so yeah so and then for the for the audience uh could you could you remind me the question again sorry because we got into the philosophical yeah so you know if they're if they're kind of working on ai features or you know trying to develop their skills as a as a product manager that is having to work with ai in their products like what's the sort of like what should they be thinking about considering to ensure that they're they're doing a good job basically yeah so i would say continue to have user empathy and really focus on the these deep user problems and or deep user pain points and to start with the problem i would say if you're going to be the the second type that i mentioned which is the ai feature or ai product pm i think this is essential and that's basically what i was kind of doing from behind on the Google search AI mode personalization, for example.

42:09Jonathan Evens: And I talked about this quite a bit in the talk, but I think it's really key. I think the flip side of this is you're going to build a gimmick feature and I don't want to do trash talk, but we're all going to have to live with a Meta AI logo on WhatsApp and never know what to do with it and just be bothered by it. So we just have to be, you know, so I think don't build that, you know, build something that, you know, is actually solving a pain point, I would say. Then I think we shouldn't forget about how, like, basically building a great product is about how elegantly does it really solve a real human problem?

42:50Jonathan Evens: And how does it solve, how does it eliminate the friction? I think that doesn't change. It's still the case. And the last one is you mentioned the maker manifesto. I would say I think I would just follow the Maker's Manifesto. I think the principles are quite correct. You know, purpose over possibility, you know, the human accountability over full automation. These kinds of principles, I think, are quite key, and a lot of people have been thinking about those. So I would follow the Maker's Manifesto. Awesome. Jonathan, thank you so much for joining us. I really, really appreciate it. My pleasure.

43:27The product experience hosts are me, Lily Smith, host by night and chief product officer by day. And me, Randy Silver, also host by night. And I spend my days working with product and leadership teams, helping their teams to do amazing work. Lou Ron Pratt is our producer and Luke Smith is our editor.