Evals, Analytics, and How to escape the feature factory: Francois Lopitaux (SVP Product Management, ThoughtSpot)

0:00Randy Silver: Feeling stuck in the middle of your career? The Mid-Career GPS Podcast helps mid-level professionals navigate leadership, office politics, networking, interviews, and their next career move. Across more than 350 episodes, host John Narrell and his guests share a weekly dose of practical guidance to help you stop feeling stuck and start moving forward. Search for the Mid-Career GPS Podcast wherever you get your podcasts. Follow the show and listen today.

0:55Randy Silver: business, career, or team. Find your next useful idea. Listen to the Business Takeaway podcast wherever you get your podcasts. Click the link in the show notes.

1:09First and foremost is the numbers, right? How many people are using your product? How is it adoption? How many people like go back to use your product? And this is for us is an amazing matrix. And for me, it's pretty much the only one that matters.

1:30Randy Silver: Hey, welcome back to the product experience. I'm Randy Silver. And today we are talking about building agentic systems, how the product lifecycle has changed. It's the thing that I think is on everyone's mind. It comes up in every conversation I have with product people. and we went and got someone who actually knows what he's talking about for this. So Francois, do you mind doing a quick introduction and sparing me from mangling your last name? Yeah, sure. So my name is Francois Lepito. I'm SVP of product at ThoughtSpot. And I've been in a product for quite some time now. I had the pleasure to start to be a product manager at Salesforce about 18 years ago.

2:13So my company in Paris has been acquired by Salesforce. I had pleasure to move to San Francisco. And since then, I just love the job. I work at Salesforce, so quite big companies on the customer service side. So building like customer service application, then more like on the analytics and AI side. Then I joined two small startups. And since about two years now, I'm working at ThoughtSpot.

2:39Randy Silver: Fantastic. So tell us a little bit about ThoughtSpot before we go any further. Or just what's the background? What is ThoughtSpot? Sure. I mean, ThoughtSpot, you know, is a company that's been created about 10 years ago. And the premise was you should be able to search your data like you search on Google. So it was really like the core technology was a search-based analytics product. And obviously, with the raise of LLM, it enabled us to go much further than that and really like make it self-service, make it really easy for people from anybody, any profiles, to find their data, find insight, take action now with a combination of the search index and the NLM technology.

3:19Randy Silver: So you're doing this natively, and there's a lot of things. Moving from the traditional software development lifecycle to an AI-enabled one, things are changing radically for lots of people. And I'm seeing huge differences in the development process with the people I work with and also everyone I talk to, but we're also running into some of the same problems that are coming up again and again. We're able to prototype faster, but getting to production is still hard. We can generate code faster, but code reviews are still a huge gateway. What are the problems that you're seeing come up? What's changing in the development lifecycle?

4:00I think the first point is really the opportunity, right? what is bringing and what is the value we get from this AI world. The first one, obviously, is the time to prototype, the time to do testing, the time to validate hypotheses has got reduced a lot. So now a product manager can kind of like, you know, generate a prototype with a rich code. Even myself, for example, I have generated two applications for the company. I had a background of developers, but it was like 20 years ago. I was a bad developer. So that's why I become a product manager because I went to myself, I told myself like I will never make a good career on the development side.

4:45But the reality is because of this AI, I'm able to generate things, right? I'm able to generate things that either can become a product if they are kind of simple in terms of architectures and complexity, or I can build like very complex prototype in a way. And I think this is really a game changer for product manager. Now, you know, are we going faster? Yes, we are going faster. But I don't think like if you really want to have a strong product, you still need to spend time on product innovation, right? You need to be able to create a strong architecture that is going to be sustained in the time, that have the right dogmail, the right trust.

5:24And for that, you know, the architects, the developers are still like much well welcome and have a strong hold to stand there. And I think also that actually organization and discipline become even more important because now everybody can vibe code, everybody can generate things. You need to have a discipline to be able to decide what is going in, what is not going in, what do need to work, what don't need to do of work. So I think it's even like, you know, the AI is creating a little bit of chaos. And if you are not organized and if you are not disciplined, then the chaos is going to kill your company.

6:06So I think it's super important to have this discipline, even more important with the AI world, which is kind of contradictive because you may think like because of AI, I can do everything, you know, product become builders, everybody can ship code and everything like that. But I think if you really like push this at the maximum, right now with a level of the technology, you are, you know, it's a recipe of disaster.

6:27Randy Silver: Yeah, definitely seeing that, you know, we've been trying for years to let teams be as autonomous as possible. And that's created, absolutely created chaos because everyone wants to work in a slightly different way. And what we really need is a minimum viable bureaucracy where people work to some basic standards. And I was talking to a friend who's at an enterprise today, And as they're migrating their teams onto more of an AI development lifecycle, what he said he's seeing is the teams are adopting and they're being forced to adopt to a standard. And it may not be something that's changing the way that they work on a day-to-day basis, but the development lifecycle that underpins what they do is becoming standardized.

7:10Randy Silver: So are we actually enforcing better discipline through the back door by migrating this way? I think we are definitely removing, you know, like, you were speaking code review. Code review now are done by agents, right? Simple bugs can be fixed by agents. So there is definitely accelerations, and it's going to go even further out with the improvement of the LLM. But I think the discipline is super, super important to be able to create consistency, to be able to create something that is not going to be demo well on day one, but it's also going to be sensible for the future, you know, because your product, you are going to add more and more features and your foundation needs to be super strong.

7:53And you cannot be like, you're just things, you know, added to each other that become some kind of like globes that nobody really knows how it's working. So I think that, you know, discipline definitely is super important. It was before, but because before, right, everything was going more slowly. So you had time to kind of catch up before it was going bad. Now, because everything goes faster, you need to be able to control this fast speed Formula 1 that is driving on the road. Before you had your small car, slow car, like your Mini Austin, it was okay. You had time to see it coming. Now, you don't see it coming, so you need to put everything in place to be able to catch it.

8:36Randy Silver: Is everything actually going faster? I definitely see the getting to prototype stage, you know, that first 80 % of figuring out what is it that we want to do, being able to describe it to people well, getting to not a PRD anymore, but a prototype so that we can work from that and communicate better together. But that second 80 % of the project, actually getting something robust, scalable, reliable to production and doing it consistently, that's still really hard a lot of the time. or are you seeing it work differently? I mean, we see it going faster. Like we think like, you know, we go between two times and three times faster for sure in our development.

9:19I think the tricky part is the fact that the way that we are building features is different now with AI, obviously, with code and stuff like that. But also the type of features that you are building is different because now you are not building any more predictive features. Before it was like, okay, I'm inside customer service applications. I'm going to create a flow to create a case. And this case is going through different processes. And then when the case is closed, it's going to mark it as done and I'm going to move on to the next case, right? So the feature building was much more easy to do because it was completely predictive, right?

10:01Randy Silver: Right. Now, at the same time, you have to think about not anymore because you have to take into account the probabilistic nature of NLM. And the fact that your feature is not anymore like step one, step two, step three, and I'm done, is going to be basically like creating guardrail to the probabilistic nature of NLM. And it's much more complex because first, you don't know which feature is going to work or not. It's all about hypotheses. is you think that this is going to be great or not so you have to do much more testing and after you don't really know like if it's the best way to address this problematic right you have a problem you need to solve i am taking the best road to solve it and so that's also like i think the the two components changing at the same time is kind of like interesting in the industry right now so let's let's ground this in in how you're actually working right now this as you said We used to work in a very deterministic way.

11:00Randy Silver: We have a hypothesis. We know what the customer problem is. We want to solve it. We're going to draw up some screens. We're going to draw up a path. And we're going to say, can the customer get from A to B to C to D in a realistic way? Do it quickly. Not too much friction. Not send too many people to support calls. Things like that. Now, we want them to solve it. We've given them a chat interface in many cases. and we can't control how they're going to get from A to anywhere. We have no idea where they're going to go next. So how are you even starting the process of designing the feature and figuring out, just walk us through what it's like for you to develop a product or develop a feature these days.

11:45Yeah, sure. I mean, I think that you need to be, first, you need to be very strong about what is your target, what is the end goal, and you should not be too much attached to how you achieve this goal, what technology you are going to use. If you are going to use LLM or something else, and, you know, like if you connect to the news, right, the JEV models that just get on the news last week is even changing the game, right? Now you are going to think about replacing some stuff that you were doing with LLM with a JEV and then is going to, you know, all your classification questions that you used to use, LLM is going to be done through this new process.

12:27So again, I think it's really important to think about the end goal and the mean may change a lot, like we see now, right? Now we are at the company level, right? We are now looking at all our different features that we have. How can we optimize with this latest technology that has been out on the news, which is super interesting. The second thing is, and this is where it becomes super important, is the eval system. Because your eval system is almost becoming your PRD. The new PRD is an eval system. Basically, you want to validate what is the behavior you are looking for. Because again, you don't control the UI because it's prompt-based, right?

13:08As you mentioned, it's conversation. So you are, but you know what is good means, right? You know what looks like, what you want to achieve, what goods mean. you know also what exactly you don't want to accept, what you want to avoid. And that's why, you know, the new eval is really a way to control a little bit more what you are looking for. And I think the eval is obviously like super important, not super easy to create all the time, but it's really important. And I think the last point, which is really important is, and it was true before, right? When we were saying like, when you ship a feature, it's GA done you move to the next features a good PM obviously don't take like that and they continue after the features getting GA and and and you know continue to work with customers to be sure that the mission is fulfilled and and they like what they have built but I think with AI is even more it's you know it's increased because you don't even imagine what people are going to do with your technology and you need to be able to observe you need to be able to continuously like what people are doing with it to be able to adapt super fast, super quickly, and change the behavior that you are seeing.

14:20So that's, I think, the new way to build stuff.

14:24Randy Silver: Okay, so you mentioned a few different phases there. So let's walk through them a little bit. Again, a more traditional way of designing apps, products, experiences, things like that started with, you know, we looked at the data and the content, but we looked at the way that people interacted with it through a UI, a static experience that we designed. And now we've got something that's completely out of our control in many cases, where the data and the content often is the interface. So making sure you've got a quality layer there before you even start is really critical. How are you scoring and evaluating?

15:01Randy Silver: Do you have what you need to provide a good experience in the first place? I mean, we spend a lot of time to build our Eval system because it's critical, right? You have so many parameters that you can change in your system that is going to impact a lot the result. So we spend a lot of time to look at Eval. What is good things for us is we have also public benchmarks that are available. So, for example, for our technology, you know, because we are on agentic analytics product, we are things like BERT. that we can use or spider that we can use where we can run through a typical question, what is the expected outcome, be able to validate each answer in terms of consistency, in terms of pertinence, accuracy, and so on.

15:53And so we are just using all the time. Every time we change something, we have to run this eval because there is new NLM models going out because we are building more features. And one of the biggest features that we invest a lot is the context, right? Because of the nature of the LLM, which is purely probabilistic. And because analytics per definition need to be deterministic, right? When you ask one question, you all the time want the same answer. And the question are, you know, I love LLM for the creativity, but when you ask like, what is my revenue this year? You don't want LLM to be creative, right?

16:27You want to be like deterministic. And so the big stuff that we have created on top of Eval is to be sure the time is to create the right context and as much as possible context that the LLM is kind of like, you know, going to the right direction most of the time. And also obviously is to use LLM when we need to use LLM. And I think this is really important is really pick the right technology at the right point to have the right outcome. If your outcome is purely probabilistic, then you don't really care. You want the most creative answer. You should do LLM everywhere. If your answer needs to be deterministic, then you should be careful because every time you add LLM at a specific point in your algorithm or in your features, then you are creating some divergence.

17:13And then you have to create something against it to be able to control it as much as you can to be able to really like arrive to the right region. So I think it's, you know, it's super interesting to be able to play all these different technology, but the Aval is staying like the reference if you want to be sure what you are building.

17:33Randy Silver: So how are you doing this for analytics? Because, you know, there's all these lines about lies, damn lies, and statistics, and liars figure and figures lie, and all these things. We've always been able to be creative with the numbers that we've got available to us. We can always interpret them in different ways. And especially when you're working with something that, and I don't want to answer it from Morphize, but it's trying to be helpful. It's trying to tell you what you want to hear up to a point. And if you've got everyone in the company asking questions of something that it may start in the same place, but the way they interpret it, the way they use it may be very, very different.

18:15Randy Silver: How do you keep people aligned and make sure that the stuff is reliable, that it's accurate, that it's actually useful rather than just nice? Yeah. Yeah. And I think, you know, this is what is interesting with all AI product is you, you know, everybody can build a demo, right? And demo will look amazing. But you, this is not how you make it your product successful. You make your product successful with people using your product. Because if they use it, it means like they will trust it. And trust is like the cornerstone of everything. And to get the trust, because you are using NLM, you need to counterbalance that.

18:52and the best way to counterbalance that is first to use it only when you need LLM. And secondly, is to really like work on what we call the context, the semantic layer in our space. So be able to describe as much as possible how is your business working. You know, when you describe what is, what mean a new customer, how you define a new customers, how you define revenues. Because your LLM is like a new intern, right? It's amazing, you have amazing potential, but if you don't know how you call your business, how your business is working, what is your definition in your company, that is going to make poor judgment and poor analysis.

19:33So once you have this defined, this context, anthology defined in your system, then this is where your system is going to create trust because people are going to be able to understand that you're doing the right things. But also this is where you will have a consistency among your users, where everybody asking the same question will get the same answer because, again, it will be grounded in your company knowledge in a way. On these old contexts and semantic layers that, for example, in our case we have built into our system. And that's why our system is super accurate and also where people can trust it because of this layer.

20:11Randy Silver: And how do you measure success of this? So, again, we used to use very traditional analytics of how many times did people come to the page? How many times did they click on this button? How many times did they use this feature? But now you're trying to do something that is both quant and qual in this. You're looking at, are they happy with the experience potentially more than how much did they use it? Or what is the way that you're evaluating how successful you are with these products? Yeah, I mean, there is multiple ways. First and foremost is the numbers, right? How many people are using your product?

20:48How is it adoption? How many people go back to use your product? And this is, for us, an amazing matrix. And for me, it's pretty much the only one that matters, right? If your product is successful, it means people see values and they are going to use it more. And it's a validation of what you are doing. Now, obviously, we are doing other things. What we are doing, for example, is we are sampling. we are anonymizing obviously like the different adoption metrics in our platform but after this anonymization we are driving metrics like how much this answer looks good how much this answer looks to answer the questions is there any follow-up question to your previous answer so have some kind of like numbers about your overall usage of your product to be sure that you are going the right direction and every time you are pushing a new features you are pushing a new feature that is improving your system.

21:44So it's really like, if you summarize it, that's two levels, right? Level one is having values from customers and customer validation. And value two is on our side. We are continuously monitoring our system, actually, to be sure that we are delivering values to these customers. And we can also react very fast if we see like early warning system.

22:05Randy Silver: Quick one for the product folks listening. AI can draft the PRD, prototype the idea, and help you build almost anything. But knowing what to build? That's the hard part. Jira Product Discovery helps product teams bring customer insights and ideas together, prioritize what's worth building as a team, and create living roadmaps that connect directly to delivery in Jira. So when more ideas are possible than ever, your team can focus on the ones that matter most. Try JIRA product discovery for free today at Atlassian.com slash MTP. Yeah, I always find this stuff really interesting because the quality of a conversation is always an interesting one.

22:48Randy Silver: You can have a short conversation as you got to the answer that you needed really quickly, and that's value. But it doesn't show a lot of usage per session versus you can mandate that people have processes in a company that inflates the number of sessions. but it may not be the actual value that people are getting. Lots of things like that. So, yeah, I always find this stuff really interesting about the right way of focusing and making sure that you've got that early warning system in place. I guess things tailing off is the biggest siren or the biggest warning. And I think also, you know, we are enterprise customers, right?

23:25So we spend a lot of time with our customers also in more like interviews, feedback loop, conversation, which also is more like, you know, qualitative, but it's also very interesting to feel the temperature of our customers. I think there is also like something that's super important for us is because half of our business is really like analytics for employees, but also we are powering a lot of companies that don't want to rebuild commercial analytics, but they still have a product where they have data they want to share with their own customers, right? So we are providing what we call embedded analytics.

23:58And this is becoming even more important because here, customers who buy our product, they are going to ship it to their own customers. So for their own customers, they become responsible for it. And so it's an even bigger trust that they need to be able to achieve because they are going to take the accountability about the service that you are providing, which means obviously it's even more complex And the trust is, it's much harder even to get their trust because they want to be sure before they roll out something to their own customers, not going to, you know, do bad behavior because it will, at the end of the day, impact their brand on top of our brand, obviously, but it will impact also their brand, which is really important for us.

24:47So that's why the trust aspect is super important, especially in our case with this kind of like second level of customers of customers.

24:56Randy Silver: So that segues really nicely into where I wanted to go next, which is the next step of the product development lifecycle. Go to market. So as you said, you're essentially B2B2C. It's beyond white labeling. It's disappearing into the client product. And it is part of an overall experience. You can't say, hey, we've turned on this new button, this new feature, this new page. It's we have a new capability baked into the interface that you've got. How do you handle go-to-market on something like that? I think it's, you know, it's also first is like from the product event development is complex, right?

25:34Because from the corporate development, everything you build on things, obviously you want to brand it in your mark, in your brand, right? So we have Spotter, which is our agenting solutions. And of course, we call it Spotter everywhere. And so when our agent answers, hey, spotter or agent spotter or the technology, when you do white labeling or EM, they want to bring their brand, right? So which means like every feature that you build, they need to be able to make it their own features. So you need to think all the time that when you kind of make it your own, you need to make it possible to make it their own at the same time.

26:14And also it means like sometimes you don't want to push too much on the old branding because you want to keep it kind of like high level. So that's quite interesting. The other thing which is kind of complex also is every time you push new features, and you know, these days, it's going, as I told you, like at the beginning, much faster for us. We've been much more capabilities. Every time you need to let people the time to adopt these features, right? Because they have to be able to control the rollout of every capabilities. They need to be able to potentially train their customers to use the new capabilities.

26:51So we need to also be careful when you are pushing new stuff to make it like opt-in base in a way, or to be able to make it seamless or design it super well that you can avoid training or things like that. But it's just like adding another layer of complexity when you are building for others, when you do the white labeling and the OEMing, because this is all the components you need to take care of. And even if you're super excited about new features and you're pushing to everybody, some customers may say like, oh, no, I don't want that. I want to control the rollout. I want to control the experience.

27:29Let me time to do that at my own pace. So it's just like a lot of complexity added to what we are doing.

27:36Randy Silver: That's one of the points that came up when we were drafting the Makers Manifesto was that specifically we have the ability to move at speeds that we never could before, but that's no good if our customers and our partners can't adopt the features that we're doing. We can only move at the speed that they can consume it and find value from it. So how are you measuring that? How are you making sure that you're aligned with them on that? You know, we obviously spoke to a lot of them, but the complexity also is we have a very different type of customers. we have customers which are like startups who are moving fast and we want to have the latest all the time.

28:21And we have more traditional vendors where, you know, their customers, for example, are not as savvy as a startup world. And so you need to be able to adapt to the different pace of people. So, you know, very tactically speaking, what we are doing is there is a lot of flags in every direction to let people to really decide when they want to turn on these or these features because you need to be able to go at the speed of your customers, go fast with the one who wants to go fast and go at more reasonable pace for the one where you don't want to go as fast. So you have to adapt and be flexible.

29:02Randy Silver: The next part of the development lifecycle post-launch, we're used to you launch a feature and you do a bit of a shakedown, But after that, you're pretty much only looking at it from a bug fix perspective. If we need to do an upgrade, if we need to do a migration of stent at some point. But one of the things that comes up with this is models drift. They need to be tuned. They need to be, you know, you need to pay attention to them over time. And that fundamentally changes potentially how you and your teams are being staffed on this. because how much of your time do you need to relegate to reserve to make sure that you're constantly keeping the models up to date and doing the maintenance on products that are released?

29:47Yeah, and I think that, you know, definitely like before, when you recreated a button on the page, once a button is created, you move on, you go to the next things. Now, your feature is never finished. You never know if your feature is really going to work until people are using it. and providing you feedback. So you really need to spend a lot of time on the after once you have shipped to be able to see with customers, like, is it working as expected? Do they get the values? Because honestly, you may create the best eval. Evals are still like, you know, a guess that you are creating on your side about how people are going to use your product.

30:27But like people are obviously very different ideas that you may have. So it's required a lot of time to, you know, continue after the shipping. And it's kind of also interesting, but it's, you know, it's obviously like taking a lot of time. But it's a great interaction. I think it's also, you know, like what we see today is like, hey, you know, everybody can do everything, right? A product can do it with developers. A developer can do a product work. A designer can do a product and build a product. But the reality is also people have some affinity, right? Some developers, they really want to build features and they may not want to interact too many times with the customers.

31:13Some product guys, they want to build features, but they really like to interact with people. And I think that this is also where the role is really important because product managers, by definition, are customer-facing, right? by definition they need to interact so with this new type of ai product building that is also a chance for us because we can interact more with our customers because we have to there's no more choice right before you were able to hide between your behind your features now there is no debate if you if you don't go to them if you don't speak to them if you don't try to understand everything you are not going to do your job really well let's get into that but before we do and just

31:54Randy Silver: want to follow up on one thing. With teams having to continue to monitor and maintain the things that they've released, does it change the way that you as a manager, as someone who looks at the organization, is budgeting for resource allocation? Do you have to say the amount of time, you know, team A to release something, now they're free to move on to something else? Or are you really thinking, okay, how much more can they take on because they're maintaining two or three experiences and they have to make sure that they stay strong. Yeah, I mean, the way that we have designed our organization is not really per features.

32:34And I think this is super important, right? Because, you know, you have different ways to organize a product team. You can do by stack, you can do by features, or you can do by outcome. Or what I'd like to say is in our side, what we have designed is we have designed what we call by track. And each track is responsible for a product. And so their responsibility is a product. Their responsibility is every feature that they should build for this product to make it successful. Which means like, when they are building a feature, it's a feature not like, you know, it's a feature about this specific product.

33:15So when they speak with a customer, for example, they can speak about the features they have built like last month and they can speak also about the upcoming one that they are going to build. So it's not so much that it's going to change the workload or anything like that, because the way that we have designed the organization, it makes sense. They don't move from one thing to another thing. It's a continuum. It's just like the continuity of the product that they're working on. And so it's not really adding new stuff or like, oh shit, I need to look at my other features that I built. It's nothing to do with my new ones that I'm building.

33:49No, it's all like in the same directions. So it's become completely natural as a follow-up. You would say like, hey, you know, we deliver this new AI context feature that is generating context on top of your columns. Now we are doing this. We continue to improve the system. By the way, what did you think about the previous one? So, you know, it's completely like natural conversation, which I think is really important. And also what is really important is because you create these kind of product organizations, then they are the best to decide what should be the next things to build. because they are in the weed.

34:21They know exactly their customers. They know exactly their product. They know the pain point. And they know what is going to go next versus me, which is managing multiple product area. I should not be the, I am not the best person to know exactly which feature to create next for each track.

34:37Randy Silver: What about retiring and sunsetting things? Is that still, you know, it's so easy to build now. Are you paying attention with the teams to pruning and making sure that, you know, the surface of the product is the optimal size? Yes, I think that's, I love to prune, by the way. It's my, I love, I'm a freak in terms of organization and I love to kill stuff. I love to remove previous features. So, you know, you have no choice. Like for example, for our starter, our core sectional ethics capabilities, we have created V1, we have created V2, we have created V3. And so if you don't prune your stuff, again, you are going to end up with some kind of like monster where every time you add something here, it's going to break something there.

35:24So pruning, I think, you know, pruning was important in the previous world. But pruning is becoming even more important now because, again, it's back to the organization and, you know, the fact that you need to be super strict because the code generation is much more easy to do. You need to be careful that, you know, it's like a tree, right? You want to have a branch to grow in every direction. You need to know when you have to cut a branch that your tree is growing to the right directions.

35:51Randy Silver: François, you talked a couple of minutes ago about, you know, an engineer being able to do the product manager's job and vice versa. And I'm definitely seeing that. But my take on it is that, you know, I can do 80 % of an engineer's job and I can do 80 % of a designer's job and the CEO can do 80 % of all of our jobs. and the fundamental difference is, you know, if you give me, me and a designer and a developer the same tools, we're going to use them differently. We're going to do different things with them. I think it's more of an attitude than anything else. And I'm just curious, you know, we used to say a product, the debate a couple of years ago was how technical does a product manager need to be?

36:31Randy Silver: Do you need to know how to code? What is the right question now? What is the core skill set that you're looking for and are you hiring product manager people are you hiring attitude people are you hiring people from different what what do you want from your team so i think that's my requirements one of them never change i strongly believe that you need to hire a technical product manager because if you don't know the basic so i'm not saying like you should be able to code or whatever but if you don't know the technology if you don't know how it's working, it's going to be super hard for you to help your engineering to deliver features and to understand the nuance and to understand why something is complex or not complex, what is possible, what is not possible.

37:18So I think technicality for a product manager, for me, has been a requirement number one since day one, since 20 years ago, never changed. Now, what changes, and you change, I will say, I'm not sure yet, actually, But the second thing that I'm really looking for is curiosity. Like my PMs are the one I'm hiring. I need people who are curious, who want to understand, who want to learn the latest technology. We're all the time thinking about how they can improve some things, how they can feel like what customer needs and so on. And I think if you want to be able to really understand like customer pain, what they want, you need to have a lot of empathy.

38:00you need to have a lot of curiosity to do a good job. So I think that really the two main aspects I'm looking for a product manager when I'm hiring. And very recently, actually, I hired a new associate product manager. And what I put on LinkedIn, it was very simple. I said, like, please, you know, send me your Git repo because now everybody can code. So every PM should have a Git repo. repo. Secondly, send me a video that is explaining to me why your latest things that you have coded on your Git repo makes sense, solve the problem, and it's kind of like, you know, your product, right? And I think like doing this way, this approach, you are really focusing on the fact that the person is technical, you are focusing on the fact that the person is obviously great at public speaking, because he's going to present his project and he's going to pitch it.

38:55And also great to understand that pain point and to solve it. So I think that's really what I'm looking at for PM. But big technical has been number one for me for a very long time.

39:06Randy Silver: And we talked for a long time about the optimal size of a team being a two pizza team. And now we can do, I'm seeing two different trends. I'm seeing teams shrinking in size a lot, or sometimes at larger companies, I'm seeing teams stay the same size, but the attack surface, the amount that they're expected to cover is much wider. What do you think a good team looks like? What are you trying to build these days in terms of an organizational unit of a team? I think the team is going to shrink because you want the bottleneck. Coding is going faster, designing a little bit also, But the biggest issue that we need to solve now is what is meaningful for customers?

39:52What feature is going to move the needle? What is really going to make us win the market? And this, you still need time to think about it. And this is really like the job of the PM. And so because execution is going faster, I really think that the team are going to decrease. And the ratio PM developers is going to decrease. and the two pizza team is not going to be the reality anymore. And also even like some developers may become like, you know, the type of PMs. And some developers definitely I see in my team like are very connected and kind of like do the, you know, part of the job of the PM.

40:32But I think these days now it's becoming even more important to know what you want to build because then after that, you know, you have to maintain it, you have to support it, you have to heal it if it's not working. So that's why, you know, spending time to be sure what you are doing is meaningful is becoming critical.

40:49Randy Silver: Yeah, I've seen some people argue for the one person team or, you know, the Happy Meal team. And I don't think that's quite right either. But I haven't come up with the right food metaphor for the size of a team that's small but not one person. Francois, this has been fantastic. Thank you so much for your time. I think we've got time for one last question. So I'm going to ask you to put on your wizard hat and help predict the future a little bit. We're seeing things change a lot in terms of cost. You know, the cost of LLMs and AI-based development has been hugely subsidized. It was very cheap for a while, but we're also spending a lot on tokens.

41:28Randy Silver: Different models are coming out with different approaches. We're seeing frontier models at a certain tier. We're seeing open weight models come. We're seeing people build their own things and start their own data centers again. Where do you think this is going? What's the trend that you think is going to happen for the next few years for teams and companies that want to minimize their spend but still get real value? I think that flexibility, I think, is a word. You want to be able to switch models super easily because today is like fab on. tomorrow is going to be like an open source model. We saw with Jeff, Jeff now can solve problem of classification at a fraction of the cost of any other model.

42:18So, you know, basically you want flexibility. You want to be able to adopt any type of technology that is going to make sense for your use case as fast as possible. So I think that more than ever, you know, it has been all the times a debate about mega vendors versus niche product. And all the times a bad side of the mega vendor was like, you are stuck with one mega vendor. The good side was everything is integrated and blah, blah, blah. I think for the LLM and all this technology, you want to really be able to be flexible, pick wherever makes sense at the time, be able to switch super fast because, you know, So years of 2025 was to prove AI is working, to prove that value was there.

43:082026 is about going to be, now you get the value, you want to get a good cost in front of it, right? You want to have an ROI that makes sense. You don't want to spend more on some things that don't bring you so much values. So I think that 2026 and 2027 is really going to be about optimization of this cost. And now with all the open source model, all the new techniques that, you know, with Jeff that we saw like this last Friday, it's really like possible. And it's, I mean, you know, the opportunity is there. So you need to be able to catch the opportunity to use all the time, the best option for you.

43:47Randy Silver: Yeah, the more things change, the more they stay the same. We went through this with cloud optimization. We went through this with any number of other things. Yeah, it's just that time in the maturity cycle, I guess. Yeah. Fantastic. Francois, thank you so much. I've learned a ton from this conversation and I really appreciate your time. Thank you very much. The Product Experience hosts are me, Lily Smith, host by night and chief product officer by day. And me, Randy Silver, also host by night. And I spend my days working with product and leadership teams, helping their teams to do amazing work.

44:24Luron Pratt is our producer and Luke Smith is our editor.