Four questions to ask before building AI into your product — Kendra Vant (Chief Product Officer, Tapi)

0:00Randy Silver: Feeling stuck in the middle of your career? The Mid-Career GPS Podcast helps mid-level professionals navigate leadership, office politics, networking, interviews, and their next career move. Across more than 350 episodes, host John Narrell and his guests share a weekly dose of practical guidance to help you stop feeling stuck and start moving forward. Search for the Mid-Career GPS Podcast wherever you get your podcasts. Follow the show and listen today.

0:55Randy Silver: business, career, or team. Find your next useful idea. Listen to the Business Takeaway podcast wherever you get your podcasts. Click the link in the show notes.

1:09I've always believed that product people will be more effective product people if they have a really good understanding of how software is built, which is not the same thing as being able to securely build it themselves. And I think that the tools have made it so much easier to gain that level of understanding.

1:31Randy Silver: Hey, welcome back to the Product Experience Podcast. I'm Randy Silver. And one of my favorite things to do on this podcast is when a former guest recommends a future guest. So Adrienne Tan of Australia, of Brainmates, of Leading the Product Fame. We talked to her and she recommended Kendra Vant to come and talk to us. And I didn't know Kendra at all. We've had a quick call to chat before this and she had some fascinating things to say, but Kendra, welcome to the product experience. How are you doing this week? Oh, thank you so much, Randy. It is absolutely lovely to be here. And yeah, the power of connection, hey, amazing that a dream can put us in touch from the other side of the world.

2:09Randy Silver: We've obviously had the chance to sit down and chat a little bit, get to know each other a tiny little bit ahead of this, but most of our listeners, I'm guessing, haven't heard of you, do not know you yet. So if you don't mind, give us a quick introduction. So what do you do these days and how did you get into the world of product in the first place? So what do I do these days? Well, I've just come to the almost the end of a very long process of building a house in the middle of nowhere, which does fill up a lot of my brand space, but not what I do for a living. But I do hope to move in soon. It is a very pretty house in the middle of a very remote and beautiful area of regional Victoria.

2:41For anyone who hasn't been to Australia, you should visit. What I do for work is most of the time I am the chief product officer of a lovely little get up and go Aussie Kiwi company called Tappi. And what does Tappi do? We provide maintenance as a service for people at the moment, largely people who have rental properties. So your tap is dripping, your tile is broken, your tenants are being a pain in the ass. Bring Tappi and we will take care of that maintenance for you. So I've been in product. Oh gosh, I will date myself. It wasn't called product management when I started doing it. I guess what's always fascinated and drawn me to it is the ability to solve problems, to see something that does not yet exist and say, I believe we can build that.

3:29And I've been doing that now through multiple waves of technology. And the bit that's always really fascinated me, because everyone seems to specialize within a specialization, is how do you take the data that flows through a platform, a product, and turn that into a better product? So that is probably the through line of my career. I've always been drawn to companies where you look at it and you think, oh, now you have that many users. Now you have that much information. How do you take that and actually turn that into additional product value?

4:00Randy Silver: So the thing we're talking about today is something that you did a presentation on at the Leading the Product Conference in Australia. And it's very much related to the kind of business that you do. So if you just said Topi focuses on maintenance, and one of the key things there is reliability. And yes, it was a very cheesy segue, but I'm going to stick with it. And so you did talk about the reliability layer and how that plays in with AI and how we're using it. This is one of the things I'm really fascinated about because what I find from talking to lots of people, you know, AI is magic in some ways, but using it isn't.

4:42Randy Silver: It doesn't magically solve everything. What I've seen people go through quite a lot is the first 80 % of the product development process of getting to prototype, of creating the specs, essentially a living, breathing PRT, something like that used to take weeks to months. And now it's hours to days to maybe a week or two at most. And you can get something, you can discuss it, you can refine it, and it's brilliant. But that second 80 % of product development of actually getting to something that is production ready, that's scalable, that's reliable, that still takes a long time. But you found a couple of distinct patterns where AI is used that create friction in the organization.

5:28Randy Silver: Can you talk a little bit about those and what your experience has been with this? Yeah, absolutely. So you're clearly not drinking the Kool-Aid, mate. I thought AI was magic and it really did get us to AGI. My experience is the same as yours, that there are some things that the advent of generative AI-based tools in software development have almost magically changed and made a lot easier. What I'm very curious about, though, is how come for some firms and companies, they can build extraordinary products, which even I, terribly cynical, pick up and go, this is magic. I can do all sorts of things that I used to have to work hard at and now I can do them really easily.

6:14But then those of us who build other types of products just intuitively know that I'm just not going to get that to work. That's not going to work for my product problems. That's not going to work for my users. And this is what made me so curious because I have no interest in being the person who the computer says no. and that's where I think a lot of product people find themselves today. They interact with executives, sales focused people, board members who bring in the beautiful demo that they've just seen. You get that sinking feeling in your gut and you go, I'm going to have to say no and you feel that you're in the position constantly of being the one to spoil the magic and this happens so often that it made me really curious as to what is the disconnect here?

7:03Why is one set of people experiencing something. And even I experienced that when I use some products, but then when we come to apply these in our own domains, we just know that the same magic won't work. And after staring at the problem hard enough for a while, and you let it cogitate in the way in the back of your head, I read a brilliant paper from two guys who your listeners may know as the gentleman who wrote the original substack called AI Snake Oil. They've now changed the name, but the new name's nowhere near as memorable. So I'm just going to keep on using AI snake oil. But there are a couple of researchers out at Princeton and they do a lot of research around not building the tools and making them better, but taking the models and understanding how they really perform in real world circumstances.

7:48And they published a paper that's caught my eye and it basically said, you know, the models haven't got very much better. They're not terribly reliable. Now that's a beautiful eye jerking headline. And when you go and read it, you think, hang on, that doesn't make sense. I really respect you guys. You're great researchers. I've read your stuff for years. But that doesn't align with my feeling, which is I use Claude Cowork. That's just my personal flavor every day. And I find it really, really helpful. But I went and read more deeply what they'd said. And they said, the models have become much more accurate.

8:20However, they have not become more consistent. And I thought, well, that's interesting. And I dug deeper into the paper. And effectively what they were saying is if take the same task, ask the same model to do the same thing five times, and it will not give you a consistent answer every time. And it was a light bulb moment for me because I thought, oh, that's actually really interesting. When I think about what I do when I use Claude Cowork, I tell it when it's wrong. I change the thing that I asked it that was a little bit ambiguous. I give it another go. I point out to it why it hasn't done what I want and I change it up.

9:00I still leave feeling that that product is magic and I pay for that product. I really do think it is extremely useful, but it really brought home to me. I'm providing the reliability layer for this product. I'm incredibly motivated to tell it what it's doing wrong because I want it to do it right.

9:18Randy Silver: But we've always done something like that, whether it is UAT and writing the acceptance criteria, or whether we're doing test-driven development and now we're doing it with devals. Is there something different that you mean? You're talking about the team who's creating the product. I'm talking about the user. I pay for this product. I'm not building it. I do not, thankfully or sadly, work for Anthropic. I pay to use a software product that I think is transformative to my workday. And yet when I look at the product, I realize I am holding its hand. And I am telling it every time it is wrong. And I am chuckling, you know, like a parent and saying, oh, silly machine.

10:03How did you get that one wrong? Think about that and the experience that many of us as super motivated users of co-work or chat GPT or co-pilot are prepared to invest in a software tool that we're paying for. and then think about, in my case, the users of my software product. Okay. They are not engaging in co-creation with my software. They just want my software.

10:26Randy Silver: They're hiring it for a service to make it just work. Just work. Every time, first time. And that was a really big aha moment for me, is that we are taking two forms of possible AI products. products, most of us in our day-to-day as consumers of products are exposed to the first kind. We're exposed to the kind which are designed around incredibly motivated users who will help the product out every time it does something slightly silly. But what we're building in our day jobs is still software that somebody expects to just work. So we're building for perfection and we look at the magic products and they're not building for perfection, they're building for highly motivated users.

11:07And for me, that was a huge moment to say, oh, here, I am the reliability layer. I, an unpaid human user, in my product, I need to build the reliability layer. And it just gave me a much better way to explain to my non-technical, non-software-pilled colleagues why the experience that they had when they used co-work or Copilot or Glean or Cursor was not easy to replicate for a user who did not give two hoots about the software and just wanted to get their job done.

11:44Randy Silver: And the more I use these tools, and I actively use Cowork as well, and I use a few other things alongside it, but the more I use these things, the more I think the people who are building it have an expectation of their users that is different than, you know, their level of expertise and understanding and using these systems is different than that of their end users. Is that really what we're getting to? Is the difference between providing a tool which people can use any way they want and they have to provide the reliability around it versus providing a solution? I think that's the key. But I think that's what a lot of us miss is you said a tool that will do almost anything, which is how it appears, but then you added, but the user adds the reliability.

12:33And I think what we see is executives and board members who are non-product people, they see the bit which is a tool that can do pretty much anything, and then they say to the product and engineering folk, well, this tool can do pretty much anything. Surely your team can build something that can do this one thing, but the difference is that you as the user of that tool that can do anything is actually doing a lot of the work the user of my tool which can do exactly one thing is paying for that tool to do the thing properly and is not interested in doing half of the job along with you so it's not

13:11Randy Silver: a matter of simply writing a an eval for it this is don't hallucinate to do this the same way every time be reliable do it right oh yeah i'm sure it is yeah i missed that and I think it's just so interesting right because it's almost that the simpler the thing you want the thing to do the more work you have to actually do to make sure that it does it because you take a super creative super multi-skilled super talented algorithm and you do actually want it to do one simple thing but then that idea of simple also means structured I actually want to make sure that this outcome always happens for this set of steps.

13:54And you end up wrapping so much code around your generative AI-based model in terms of guardrails, evals, and checks that that code, your reliability layer, becomes where your engineering and product teams actually need to spend all of their time, which is not wrong at all. It works really well in lots of instances. But I think once you realize that's what you're doing, it gives you two things. It helps you explain a lot better to a skeptical, non-technical colleague why it takes the time it takes to do the thing. And it also helps you cost it a lot better. Because if you don't realize that you need to design and engineer and run the reliability layer, you run the risk of thinking your product is going to be a lot cheaper to scale than it actually is.

14:40Randy Silver: So when you're working on this, you're working at the CPO level, you're working with executives on this as well as your teams. How do you put this into practice? I think you have four questions that you like to ask people and make sure that are answered as part of this. Walk us through those. So I should say I'm a CPO of a very little company, so I have a lot of hats, which is a pleasure. And I think for me at the moment, it feels like a real privilege because I think it's super hard to be a senior executive who is no longer a practitioner and understand and adapt fast enough to the change that is coming into practitioners lives so I'm really privileged in that respect yes I am the CPO we are a very little company and I love getting my hands dirty so I do and I think I would really exhort people to find a way to get their hands dirty to really get their heads around the changes.

15:35So for me, I think what a lot of people say when they pick up an AI product for the first time or a product where they're going to build generative AI inside their product, they say, can the model do this? That's the wrong question, in my opinion, because can the model do it in perfect circumstances if the user's asked exactly the right question and you give it a one-shot answer is quite possibly yes. The question I think you should ask is, where is the reliability layer in this user's experience and can I afford to build it? So to try to sort of help you say, all right, I know that what I should look for is what's the reliability layer?

16:14How do you pick that apart? What are the practical questions that I apply and I try to get my team to apply? One of them is, well, when you use an AI like the one you're thinking of, so when you use a product which has a similar thing built in, be that smart complete, be that structured diagnosis, be that summarization. What are you doing as a user to provide that papering over the gaps? And my suggestion to you is if you go, no, no, I'm not doing anything, Kendra, actually go back and use the products again, because I think you'll find that like my experience with co-work, you don't think you're doing anything to paper over the gaps.

16:56But once you start paying attention to it and you go back and do it again, you're like, oh, no, I am. Yeah, yeah, no, I am. That's true. So that's one of them is what are you papering over when you use AI like this? And will your users in the instance that you're building paper over that for you? So my users are very busy property management professionals who want to spend their time keeping landlords happy and putting new tenants into rentals. Maintenance is a payment house. And they do not want to have to tell a bot, no, I said the upstairs toilet and you're fixing the downstairs toilet. So I have no leeway in my users wanting to paper over for my AI.

17:37Second question I get people to ask is who provides the usability, the reliability layer? When you're using Claude, you're the reliability layer in your product. Are your users prepared to do that? if no is engineering building the reliability layer is product specifying it or is actually nobody yet paying attention to it because for me if no one's yet paying attention to the reliability layer we're at the point that you said we're at a demo we're not at a product third question I ask is when the reliability layer fails what's that going to cost you in terms of customer trust And really, really critically, I'm not saying when the AI fails.

18:17I'm saying when the reliability layer fails. So if you've got something that's built in that is supposed to stop your AI being jailbroken, being able to go and answer a completely irrelevant question, that's what I'm talking about. You can't build perfect software. When your reliability layer does fail, will it fail gracefully? Or will you get customers saying, yeah, that's not ready for prime time. I'm not prepared to pay for that product again. And then the really critical one that unfortunately I think a lot of teams are missing at the moment because we're also new to what it really takes to build generative AI-based products.

18:53What's it going to cost to maintain your layer, your reliability layer, whatever it be, be it wonderful humans like in the background, making sure that the AI doesn't do anything too silly, be it engineers running five different models over your model's answer to check that it's only doing what it's supposed to. what's it going to cost to continue to run that layer at scale? Because I've seen a lot of teams build reliability layers with evals, with model checkers, with humans that work at demo scale and cost more than the product takes to run when you're running it at scale scale. And I've seen too many teams fall into that problem of being like, oh, no one's ever going to pay what it actually costs to run this product.

19:36Randy Silver: Quick one for the product folks listening. AI can draft the PRD, prototype the idea, and help you build almost anything. But knowing what to build? That's the hard part. Jira Product Discovery helps product teams bring customer insights and ideas together, prioritize what's worth building as a team, and create living roadmaps that connect directly to delivery in Jira. So when more ideas are possible than ever, your team can focus on the ones that matter most. Try Jira product discovery for free today at Atlassian.com slash MTP. Is there a way that you recommend that people look at this and say, is my reliability layer robust enough?

20:21Randy Silver: So you've got these questions, but is there something in there that helps me to say, I've asked this and I've gotten some good answers and some bad answers and some things that are a little bit in the gray area. How do I know if I'm robust enough? I think that absolutely starts with some, even if it's a gut level understanding of how will your users react when it's wrong. So one back in the day, back in the day of traditional machine learning, I learned a trick from a colleague. It was very common back then to say, I can train this model to be correct 85 % of the time, 92 % of the time. And he was right.

21:01and he taught me, you never say that to a business colleague. You say, I can train this model to be wrong 15 % of the time. It's the same number, but it sounds much worse when you say that the machine's going to be wrong 15 % of the time. And I tried to ask those questions very early in the conversation so that I was pointing out to the folk I was co-creating products with, it's impossible to build any kind of machine, be it generative AI or be it more traditional machine learning that will be right 100 % of the time it will be wrong some percent of the time what's going to happen when it's wrong and so for instance I was building a long time ago I was building a product that had to do with invoice financing and we were keen to be able to try and take that money out of the person who had financed from an invoice as soon as we possibly could and so the question you have to ask is if I remove the money from the account before it's cleared, what is my fallback plan to putting it back in if it proves that it was not a payment for the invoice that I was financed against?

22:05And there is an answer. And you can build a fallback plan that works in that instance, and then you can build it into your cost. So perhaps I'm not answering your question very precisely, but I think that's because it's a different question for every product. I would say, figure out the way it's going to go wrong, figure out how you will gracefully recover from that, and then you will be able to better answer the question of whether this is a product that your users are going to delight in using, even built in when it's wrong.

22:34Randy Silver: I think you actually just demonstrated it really well. The follow-up I wanted to ask was, a lot of times the fundamental problem that I'm seeing with these tools is that they don't have the capacity built in, or they're not, I'm going to anthropomorphize for a moment, they don't want to say, I don't know. Oh, for sure. And I think that is probably one of the ways of failing in a nice way of saying, being able to say, I'm not sure, I don't know what the right answer for this is. Is there a way that we can, I think that would actually increase reliability quite a bit by being able to declare a fail state or an incomplete seat.

23:16Randy Silver: Is there a way to improve that, to add that into your tools, to say I'm confident about this and I do not know about this? You only ask for nirvana, don't you, Randy? I mean, one thing I would say is as a user of large language models and the new iteration of them, which are very much tool-assisted large language model conglomerates, I think they're getting better at that. if I think about using the base large language model as offered by a vendor four years ago, it was absolutely always completely confident, even when it was talking nonsense. And my test for nonsense at the time is I would ask it to write a biography of myself.

23:57And it would be partially right and very confidently wrong in many places. I think they're getting a lot better at that there's a lot built into those tools now, for instance, web grounding and chain of thought and checking themselves, that they are already far better at not always being entirely confident. The only way that I know of to add your own additional checks on top of that is either to have context that you are checking against, so your own set of facts, your own set of knowledge graphs or and to run multiple models and see if they agree and the challenge with that because it is one of the ways i think that people can wrap reliability layers around their products is that it makes it three times as expensive and so that's why i'm always like you don't want to just build can the model do this in a happy path you want to say with unhappy paths baked in how much is it going to cost me to run this because i think we're also seeing still a massive shift from seat-based licensing to usage-based licensing.

25:01And it's super hard to convince a user base who have grown up with seat-based licensing that they should start doing usage-based licensing for non-guaranteed outcomes.

25:13Randy Silver: Almost like hiring a consultant. Imagine that.

25:20Randy Silver: I wanted to ask you as well about times when you might want your user to provide the reliability layer, at least in part, because the way you were talking about strikes me, this is the way that people customize products, that they create some intentional friction, they increase the engagement, and that creates some lock-in with the customer. So you were talking about the property manager who, you know, when you're developing Toppy, you might have something that works 95 % of the time, but there are some people with unusual configurations or unusual ways of working, and they want to customize it a little bit.

25:59So that engagement, that friction that comes in, they've now built something that works

26:03Randy Silver: really well for their specific use case that you could not have imagined in the first instance. And now they're never going to leave because the tool works exactly the way they want. And the friction of setting up, of moving to another platform would be too high. is this something where you might intentionally create the user as the reliability layer sometimes or at least to a limited extent? Yeah, I really like that idea. I think it's a beautiful articulation of how you can allow customization. I think that when I approach that, and I must sound like a broken record, but that's what comes with being a CPO.

26:44You have to not only build the product, but make sure that you can sell the product for as much money as it takes to build it. when I approach that my concern will always be can I build that yes every time I build customization into a user experience I make it harder to scale my product to the next thousand into the next thousand and the next thousand so I think that when when I wanted to do that that's also what pushes us towards tiered pricing and enterprise plans so that you are almost saying to the person, I would love to lead you to the point where you like my product enough to tell me what you don't like, and then either join my customer advisory board, or how can we work together to set up a situation where you can do that customization in a way that's cost effective for both of us, which I think often does lead to tiered pricing plans and customization above that into an enterprise grade.

27:46Randy Silver: Makes sense. Okay, I'm going to go in a different direction now. So for the last, oh God, five, 10 years, one of the questions that's come up again and again and again has been, do product people need to learn to code? Do we need to be developers ourselves? And I think with the advent of these tools, that's kind of a settled answer. We don't necessarily need to learn to code, but we need to learn how to work with code really well and be able to ask questions. I know in my own things, I've had to go through an evolution and take something that was a side of desk project and to make it production ready.

28:23Randy Silver: I've had to rebuild an entire software development lifecycle and multiple repos and things like that and all that kind of stuff. So, yeah, it's been really interesting. But I think the question now is everybody else in the company, do they all have to become makers and builders as well? Do they have to understand the difference between a prototype and a production system? It's a really interesting expansion on the question, because I thought I knew what you were asking, and you took it in a different place. So I'm married to a software engineer. So let me assure you, software engineers do not think that product people can code, because there's a difference between being a script kitty or being able to ask Claude to build a thing on Netlify and being able to code.

29:08And I think that's been a distinction that a lot of people have misunderstood for a long time. Because if building reliable, secure applications was just the bit that involved running the application code, software engineers would run a lot more application code. So I'm not ever going to be presumptuous enough to suggest that what I can do is coding or that using Claude Cowork would take me to that point. I've always believed that product people will be more effective product people if they have a really good understanding of how software is built, which is not the same thing as being able to securely build it themselves.

29:48And I think that the tools have made it so much easier to gain that level of understanding. So for me, I wouldn't hire a product manager who was not comfortable and confident in interacting with Git and learning from the code base many of the things that they would previously have asked an engineer. As a product person, I find that amazingly freeing because I can actually go and ask the code base things that previously I would often have needed to wait for a day for an engineer to bother to tell me. So I think that's amazing. So to me as a product person, I would make a really clear distinction between I'm not suggesting you become an engineer, unless that is genuinely what you would like to do, in which case these tools give you a path to learn much more rapidly than has previously been the case.

30:36But if you were laboring under the misconception that you didn't need to understand how software was built to be an effective product person, then I don't think you can labor under that misconception very much longer because it has become so easy to be very, very literate and how your code works and hangs together, that you're not going to remain or become someone who's going to be given a lot more new opportunities if you're not showing that you can take that yourself. The next nuance of that of should everyone else learn to code, I guess the thing that makes me a little bit nervous about that is the difference between I have a lovely little customized personal health plan tracker that sits open in my browser and reminds me of all the things I've committed to myself to do to make sure that, you know, my 50s and 60s and 70s are as good as my 30s and 40s.

31:30I'm under no illusion about the fact that that could be given to anyone else. And I think that's really key. So I think it's incredible if people who work and make their living in teams who build software can get closer to understanding what software build means, because it will lessen the friction between I need to roll this out in market, I don't understand what is taking you lot so long, because people have an appreciation of what it means. I'd be super nervous, though, about someone who doesn't see the gap between a prototype, as you were saying earlier, where you can get 80 % of the way towards something that looks marvelous and can't then appreciate why there's still 80 % of the work to do to get to a piece of software.

32:21So I think for me it's a measured yes. I always think it's incredible if you can get closer to what your colleagues do. I'm always keen to understand the dynamics of go-to-market, for instance, because I find it endlessly fascinating. I'm not saying that makes me a born salesperson. And I think keeping the humility of where you have deep expertise and where you have a good enough literacy to have really competent conversations is really, really important.

32:49Randy Silver: Let's get into the practical a little bit. Tell me about how you're working at Topi. How is the team actively doing this stuff on a day-to-day basis? yeah so i think myself and the product team are doing pretty much what i just said which is delighting in the fact that we can now run five experiments in the time it would have taken us previously to have one conversation so i love claude design i'm sure my design colleagues do not appreciate that i love code design but I learn by doing and so I find it immensely useful to be able to push a slightly crazy idea that I have in my head through figuring out 15 ways in which it won't work before I need to go and have that conversation with someone else and take their time so I think the answer would be on the product side we are using it to iterate and think out loud much much faster than we've ever been able to before on the engineering side you would definitely be better off asking my cto but he is a person who believes in software factories and believes in every engineer building their own harness so he is focused on the belief that generative AI-based software tooling hasn't lowered the bar it's raised what's possible so we believe that professionally built software will become richer and deeper and much more functional because once you give people some powerful tools if they are people who love to build they will just build far more wonderful complex and powerful things are you seeing a gap at all between you can experiment and prototype much faster you can ship faster as well but i think the the multipliers are different you know the prototyping multiplier is much higher than the shipping multiplier in most cases and I'm starting to see companies where the idea what the executives are seeing what's in the boardroom is potentially years ahead of what's shippable in some cases and the gap keeps widening is that something that you've experienced or seen yeah I think I've seen a slightly different, not orthogonal, but a slightly different idea of what you're saying.

35:17What I've really noticed recently is that the tax of having a really large install base or a big legacy code base is heavier than it was prior to the advent of these tools. Because what is it that kills most SaaS products? It's not that people stop understanding how to build great software, it's that you have a big install base a big belief that this is how your software works and software that was built with the best of intentions on tools and code and patterns and languages that are no longer state of the art and so what I've really noticed and you know say this maybe I shouldn't because I am privileged to be in a small company there's never been a better time to be in a small company because a very few people who have the luxury and the privilege of not having a large base of code that they need to maintain can move extraordinarily fast.

36:15Probably my biggest concern, because I'm the mother of three 20-year-olds, I am seeing a gap between what a deeply experienced team can do and what a team who carries less scars can do. Because the coding agents are extraordinary. the ability to write PRDs by not having to write them myself is extraordinary but I bring a lot of previously failed attempts to the way I use the tools and I don't know how you replicate that if you are if you haven't had the opportunity to accumulate that learning if that makes sense so that's probably a gap that I'm worried about so that's what some people would call product sense

37:03Randy Silver: or product judgment, but in practical terms, it's the ability to call bullshit on the tool, right? And just be like, yeah, I know why you think that, but my practical experience tells me that that will work now and will not work later, right? And certainly I think also I see on the engineering side the ability almost to move beyond a specific language and think much more in software patterns has become much more of a differentiator than it was, which is kind of interesting because when I started my commercial career about 20 years ago, I'm shameful to say that we would say don't hire from the computer science department, hire from the software engineering department because the software engineers come out of university knowing how to write commercial code and the computer scientists come out of university knowing how to write algorithms.

37:58I think that's shifted a little bit in the last three or four years because understanding the fundamentals of computer science and understanding the fundamentals of what great system design means has become more of a differentiator because that's what the tools can't do. They can write code faster than many software engineers, but they are not yet able to do system design structure for a product that is only conceptually in the heads of the humans so there's huge changes in how software is built and the skills that you need to stay top of the game i guess the product sense is the one that i'm curious about because i genuinely don't know and i don't think anybody does is i learned that through working with other humans who were better at it than I was making mistakes and accumulating lots and lots of lessons.

38:59So I can apply the judgment. I believe I can because of 20 years of having done it before and seeing what doesn't work. I'm not sure how you accumulate that now, if you've just come out of school and I think we'll figure it out. But to me, that's a huge change in how software is built.

39:17Randy Silver: a moment ago you were talking about the advantage of working at a smaller company of having uh less legacy code and things like that a few months ago i was listening to the ceo of a pretty large company uh talking about the fact you know they're spending a lot of money on ai they're doing a lot with it they're getting a lot of value out of it but an incredible amount of that value of that production is essentially running to stand still. It's rewriting the code base to keep up with the state of the art to fix problems that were there before. It's not necessarily moving forward. Are you seeing that in the industry, both in your experience and from other people you talk to?

39:59That's really interesting. That is a particularly perceptive CEO. I mean, to my mind, what that person just described is in many ways a good outcome. Because if you are actually keeping your legacy code base up to date through investment in generative AI-based coding rather than through investment in enormous numbers of engineers, you're probably winning. I think that's a positive outlier. No, I would say I haven't seen that reflected in large industry in Australia yet. I have heard that there is an enormous amount of generative AI spend going into the larger enterprise. But if I had had to guess, I wouldn't have thought it was restoring legacy code bases.

40:47So, but hopefully there's a positive sign of things to come.

40:51Randy Silver: Kendra, this has been a fantastic conversation. Thank you so much for your time today. I've got time, I think, for just one more question. So let's leave on some practical advice for people. returning back to the topic at hand about reliability, if people wanted to focus on this more, if they want to make this a focus of what they're doing and bring it into their day jobs, what's the one thing that they can start doing tomorrow, the top thing that you'd recommend that they fold into their practice? This is probably a little unfair and portrays my origins, but I think it would be asking better questions.

41:28And let me give you a concrete way in which I think you can move towards asking better questions. I spent 11 years training as a scientist. And one thing that science teaches you to do, two things that science teaches you to do really well is one, be okay with struggling with really large, hairy problems that you don't initially know the answer to. I did a doctorate and it took six years. And so that was a very big, hairy problem. And it teaches you to be good at that by breaking it down to smaller problems. so I think you can train yourself to do their five whys in a slightly less annoying way than a toddler and when you force yourself to not be satisfied with the first answer that you give to a problem I think you get much better at articulating the problem space and that's the flip side of reliability if you haven't yet forced yourself to think through really quite hard and in quite a lot of detail about when you say you want it to do this thing what does that mean to five levels of why and how will it go wrong you probably haven't yet really articulated your problem space so for me I'm I often put down my computers and put down my clothes and put down my things and I've been often known to say I learn by I I think by writing so I really recommend people to ask themselves better questions force themselves to write down the answers and then maybe my next step in terms of making sure you understand the problem space well enough is to explain it to someone else so ask yourself better questions write down your answers and then go and explain it to a least mildly interested colleague and see whether your what you thought in you holds up when you try to talk to someone else about it.

43:27And I really would say, don't talk to an AI about that, because as you mentioned earlier in the conversation, maybe the AI will tell you that everything you said makes sense, even when it doesn't.

43:37Randy Silver: I love that. I've heard this before in two ways. One is the, can you explain it? You really understand something if you can explain it to a six-year-old, number one. And number two, the way that a lot of developers work, especially when they're on the maintenance side of things, is they've got the rubber duck that they pass around the office. And being able, if you can't talk to a colleague, at least explain it to the rubber duck and listen to yourself talking. So if you don't have someone, any inanimate object, any toy that you've got in front of you is better than nothing. See, that'll be why I let my cat sleep in my office.

44:15Randy Silver: Fantastic. I hope the cat gives you the answer. She usually does after a while. Kendra, thank you so much for this. Is there a place that people should be following you online to get more of your wisdom on these things? Oh, I am quite active on LinkedIn. I would love to say that I am religious at updating my sub stack, but I'm not. But I have a sub stack. Data runs deep, which I faithfully believe in myself. I will get back to writing one day, but mostly at the moment, I have to admit, busy with Tappi, busy with the house. I'm on LinkedIn maybe several times a week. Fantastic. Thank you so much, Kendra.

44:51It was my absolute pleasure, Randy. Thank you. The Product Experience hosts are me, Lily Smith, host by night and chief product officer by day.

45:01Randy Silver: And me, Randy Silver, also host by night. And I spend my days working with product and leadership teams, helping their teams to do amazing work. Lou Runpratt is our producer and Luke Smith is our editor.