Quality-Driven Product Development with Realtor.com’s Erika Chestnut

0:00Season three of the Built Right Podcast is right around the corner, but we've got one big change coming your way. The Built Right Podcast is now the Talking AI Podcast, and we've got a lot to talk about in AI. In the Talking AI Podcast, we'll be having in-depth conversations with both AI experts and early adopters of AI. That way you can understand how the technology works and how early adopters are beginning to implement and, more importantly, get value from AI. Our guests range from AI research scientists to founders of AI products to industry leaders putting AI to work in their business. While you're waiting for season three, go ahead and subscribe on your favorite podcast platform so you don't miss an episode.

0:41And make sure to leave us a comment about the AI topics that you want to hear about. So get ready to talk some AI in the new Talking AI Podcast, coming your way August 6th.

0:58Welcome to Built Right, a podcast by Hatchworks where we help you learn to build the right digital product the right way. In each episode, we'll deconstruct the layers of successful product development, break down popular trends, and offer real advice to help make sure your product is built right? We may not have all the answers, but we've built a lot of digital products across a lot of industries, and we've seen a thing or two. Let's get into it.

1:33Today, we're chatting with Erica Chestnut, a true champion of quality. She's been building and leading quality teams for around 15 years now at places like Cabbage, Turner Broadcasting, Calendly, Realtor.com, to name a few. And that's an awesome list there, by the way. Erica, before that, she's led development teams. She was even a developer herself. So I know you're going to fit right in with our Bill Wright community here, but welcome to the show, Erica. Thanks. Thanks for having me. Yeah. Excited to get into this topic. This is one we haven't gotten into yet on the Built Right Podcast. Today we're getting into the often overlooked and sometimes undervalued topic of quality in software development.

2:17And the topic of why good processing culture are really at the foundation of good quality. And PS for everybody listening, stick around. We're going to get Erica's take on her perspective of generative AI and how it's impacting the discipline of quality. I know everybody's talking about it. So we want to get Erica's take on that as well. But to set up the problem, Erica, you talk about organizations who want to improve their quality are often focused on the wrong thing. So what is this wrong thing and what can they do about it? Yeah. You know, quality is always, not always, that's a poor statement.

2:56Oftentimes, when people think about quality, they think about the end state and therefore they think about the thing that happened right before the end state, which is oftentimes testing. And so when they say our quality is not good, they say our testing is not good, or we are not investing in the right type of testing, i.e. manual versus automation, or we don't have enough coverage. We don't have enough code coverage. We don't have enough functional or non-functional testing. But the reality is actually that it starts much further up the stream. And you started to hear about this when the industry was like shift left with testing.

3:36But then just like most buzzwords, right? Innovation, innovative, right? It's not unpacked. And so now it's like shift left testing. Okay, well, what does that genuinely mean? And what is the impact of that? Right? And real quick, shift left testing, that's kind of meaning moving quality further up the value stream towards more of the beginning of the process. Is that right? Just for listeners? Yes. Yes, but it's not moving quality up. It's moving the, well, I mean, it is moving quality up, but it's really about moving the validation, the checks, the awareness, right? What is impacting our product quality?

4:21And so one thing that I always love to say is that process creates quality. Process results in quality, good process, because process creates consistency and continuity, which results in quality. So when you say moving quality left or further up the chain, people are still thinking testing. Oh, we're testing the requirements. Well, are you checking your process? Are you checking your communication flow? Are you checking your documentation? Does everybody have what they need? Are you checking to make sure that the quality team is not starting the new sprint at a deficit because the engineers didn't start stopping before they started finishing, right?

5:05Like you've got to shift the idea of what impacts quality, what creates poor quality, right? And it's not just testing. Yeah, and you make a good point because quality is at the end for all intents and purposes. That's kind of the last thing. Let's check everything, make sure it's good to go. And a lot of times they can be the scapegoat when something goes wrong or doesn't get delivered. And I love how you hit on this concept of process. But to clarify, like a lot of people think like process, they think like tools, but it's not about the tools. Like tools are kind of often, you know, over, you know, put on a pedestal in terms of, oh, they'll fix everything.

5:46But it's not, it's not about the tools. It's the underlying pieces in the process. And I love how you talk about the culture element that comes into play as well. Yeah, there's definitely tools and it's testing that's at the end. It's the testing. It's quality is the entire thing. So actually, I have to retract my statement. It's not that it's not about moving quality. It is. It's quality doesn't need to be moved. Quality is everywhere, right? Yeah. Opportunities to lead with quality are everywhere. So it's not about moving it left or right or up or down. It's about acknowledging that there are opportunities to improve quality in everything.

6:29Is it improve quality in our process? Is it improve quality in the tools that we use and how we leverage them? Are they the right tools? Are they answering the right question? Are they implemented in a way that it's cohesive, the not cohesive, that it integrates into our system in a meaningful and impactful way? All of that is quality. All of that produces quality at the end state. And they all come together like they're it's not just testing. It all comes together to produce quality. Quick break in the pod. If you're listening to this podcast, chances are you've been thinking about how to actually use AI inside your business.

7:09And that's exactly why we built the AI Opportunity Finder. It's a free tool that helps you uncover high impact, tailored AI use cases based on your business, your goals, your pain points, and your industry. No fluff, no generic use cases, just real ideas that fit your business and the ranked by ROI potential. It takes about three minutes to run and it's like having your own personal AI strategist for free. If you wanna try it for free, check out the link in the show notes or go to hatchworks.com backslash AI-opportunity-finder. So to make this more real, I am curious because you've been in a lot of interesting companies from small to large and you do some like side consulting stuff in the past.

7:52But what examples do you see? Like what are those common pitfalls that companies have, whether it be process related or just in quality in general? Is there anything that's like, I see this every time or this is like a big thing that typically happens a lot? You know, one thing that I see that's always, it's a, it's, I think a pain point for, not a pain point. It's, yeah, it's a pain point for me is that, or a pet peeve, that's actually a better word. There you go, pet peeve. It's a pet peeve of mine, and I see it all the time. The structure of the business is not clear. And it's fundamental quality opportunity that is missed.

8:34When it's the structure of the business is not clear to the teams or the business flow, like the whole, like, what is it? what are the boxes that make up the business and how does it flow left to right? What are the, you know, what are the little exits along the way and what happened is not fully unpacked for the team. And then when we go through like hiring companies are going through like this massive hiring, these windows, and then we're throwing people in and we're saying, hey, listen, go to your team they'll help you the team has blinders on the team has blinders on and they're like this is our little world but we're not providing this big big big picture view for for people to understand at the top level like this is our this is our business this is our structure this is how we talk about ourselves internally and this is very clearly how it moves down into the organization from a structure and from a business flow, like the actual product.

9:49And so I find that those are missed opportunities oftentimes. And they don't recognize, leadership doesn't recognize that it's impacting quality. You know, I'll go into teams and I'm talking to teams and they're like, I don't know about this. I don't know how this integrates with this other system. I don't know. I had one manager saying, my area, my enterprise area doesn't integrate with this other area, this main area of our product. It does. It did. And they didn't know it. right so like we put on these blinders and you're like hey i've got my area and i'm good it's like what are you thinking about how your area integrates with these other areas and what the impact is and do you understand and are you mindful of that so that's the thing i think it's missed that's interesting especially i guess when you get into larger scaled organizations but it gets back to like you know we talk about a lot about at hatchworks connecting to the outcome and understanding the outcome and knowing that it all layers of the organization.

10:58It's so important because you have to understand, you know, what is the business outcome trying to be achieved? But I love your point around the connection between multiple teams, right? And having that quality understanding between the different organizations. Let's hit on this topic. So quality and innovation, you know, in my mind, I feel like they can sometimes be at odds. Quality is very much like, you know, process driven, like, you know, rigid's not the right term, but you do want like foundational process and how things are structured. And then on the innovation side, a lot of times, whether it's like business model innovation or, you know, anything like that, you're thinking of breaking process and norms.

11:43How do these two play together and how do you create balance between quality and innovation? Yeah, most companies struggle with that as well. That balance of quality to innovation, because obviously the business is running after innovation. That's the they want to stay ahead in the market. They want to be first to make that next big change. They want to be the unicorn in the space to do that. Sometimes you're running fast and you are focused on like, what's the new whiz bang feature that you have? You know, but that can be that can be a struggle. It can it can it can be at the expense of quality.

12:25And if we're not looking at it, if we don't pay attention, we're like, listen, just just innovate. Get these get these new products out. Get this out. and you might have the teams, you might have the quality team saying, there's a problem with our quality. There's a problem with our architecture and we're building these new features on top of it. And so these new features are nice and shiny, but we're putting them on top of something that doesn't smell so great. And eventually the new shiny thing will wilt and it will also smell, right? Because we're not considering that we never fixed the actual problem.

13:03We never cleaned up the smelly stuff, right? And that's part of quality, but it's not just bugs. It's what might create delivery problems. What might create inefficiency? What doesn't allow us to roll back quickly if we have problems? How long does a problem linger out there? How many open issues do we have? just even just needing acceptance criteria, the turn of how frequently, you know, how long it takes to get something delivered. And then we're making, we're taking shortcuts because the requirements weren't 100 % clear. And so we had to go talk to product a lot. And then we went back and forth and we made changes.

13:47And then all of a sudden, something that we developed two weeks ago that was actually pretty well-baked has now been hacked at the very end and were released out there. And then there's an edge case that we didn't know about, but it's like extreme edge case, right? So it's that innovation. Like when you think, are we innovating too quickly over quality? What is the impact of our innovational quality? Did we release these new features that we, you know, this new functionality, there's new innovation. and do we see a high level of defects? Did our defect density increase? Did our CFAC scores go down because our customers are like, yo, this is broken.

14:26You told me about this new hotness and now I'm coming here and it's just broken. Like, well, that sucks. I don't want to use your product anymore. I don't want to tell somebody else about your product. Right? We have to balance that. But it's oftentimes a struggle. It's almost like quality in a lot of ways is the enabler for innovation. If you don't have that foundation set, It makes innovation that much more difficult, right, to actually do that. And I love the, I got a visual in my head. I have a, you know, a one-year-old baby. So when you mentioned the doesn't smell so great, that's bringing up some bad memories from last night.

15:02Things were thrown away. I don't want to get into it. But the, you hit on some other things though, like, you know, in our business and we've done kind of a foundational shift as of late, really focusing on what are our leading indicators versus our lagging indicators? Like, what are some good leading indicators in quality? And you mentioned, I think, like some time to resolution and things like that. What are you looking at, whether, you know, lagging or leading, that are indicative of either, you know, things are going good or maybe I need to like hone in in a certain area? Yeah, like that defect density, right?

15:40What does our release health look like? And do we see a lot of releases that are going out and we're seeing our health dip? Are we seeing a lot of releases returned, a lot of rollbacks, reverts, incidents? What's our mean time between failures in production? These are all alarms. These are red flags that we can look at and say, maybe we're innovating too quickly. Maybe we need to slow down. what's causing this. Maybe we need to look like, were our requirements not fully baked? Were our acceptance criteria not clear? Where was the failure? Did, you know, did we push in something really late that increased effects?

16:22Did we not, is it, what type of failure is happening? Is it a back-end failure? Is it a load capacity issue? These are all things that like, when we begin to unpack that and we say, hold on, we're seeing an increase here. Let's look at it and understand what the problem is so that we can target it, fix it, and then go fast again. Right. But often times they don't. That's such a foundational. Yeah. Yeah. That's a foundational piece is knowing what those metrics are. So you kind of have your, you know, dashboard for lack of a better term of your indicators. And then when something's off, you kind of know where to dig into.

17:00And I heard you mentioned the defect density. Is that just kind of like volume of defects or is it hitting on something more specific? Yeah, it's volume of defects. So let's say that, you know, we we've identified 200 defects in production. And please don't start me mathing because mathing is hard. We've identified 200 production notes. It's Friday for us. We're not getting into math. It's Friday and it's been a Friday. Yes. Not in a Margaritaville kind of way, although maybe it needs to be very soon. Yeah, that's that's next. Yeah. Yeah. But like the number of defects and then let's say that we, you know, we have kind of a trend.

17:41We see that we have, you know, maybe some spikes here and there. But we start to recognize that those spikes are happening every time we release to production. That means we're recognizing that we are introducing in every release a spike of issues that then we're having to work back down. How do we improve that spike? Like, is there a correlation? And then what is causing that? What are we missing? Do we not have enough automation regression? Are these regression issues? Do we not understand our system well enough that we understand the impact of the changes that we're making on downstream areas of the system?

18:24What is creating that spike, which is costly? Because especially if it's like a critical area of the system, generates an incident, you've got no less than 10 people jumping into that conversation. You've got the eye of the CTO, the eye of the CMO. So you've got executive leaders and you've got senior leadership. I mean, this gets really expensive and they're just looking at it and waiting. Right. And I mean, they're jumping in, but, you know, they're engaging in the conversation. And then you've got the management, you know, the middle management layer. And then you've got the ICs that are implementing potentially the change.

19:02You've just got a lot of people in that. It's costly. and so it's like hey we're seeing we release and we spike and then we spend on top of the innovation time to get those spikes back down or we're leaving them out there and the customers begin to deal with death by a thousand cuts because i don't know when it wasn't big issue but there are a thousand of them that everywhere because it's like next right you're like yeah like walking into a death by a thousand bugs yeah i went to school down in in uh south georgia georgia southern so i'm used to the the natch you can probably yep sympathize being in lana georgia native can i get around you yes yes that's a good point though you talk about it's like how do you help the business recognize the value in the impact of processing quality yeah um it's almost like it's before it's too late, I think is the key thing.

20:01It's like, how do you help them recognize that value? What is, where have you found success or what are some good things to hone in on to help connect it to business value before it is too late? And then you got all the C-suite breathing down your neck, kind of like you mentioned as a scenario nobody wants. Right. It's telling that, it's identifying the story of quality within your organization. So hearing from, I love to like, when I come into an organization I really want to hear. I host what I call my what the bug meetings and I'm meeting with different people and I'm asking them some similar questions, depending on where they are kind of level wise.

20:40Some are a little bit more detailed questions, some are more strategic, but they're still kind of in the same vein. And then I'm looking for those categories. I'm looking for the sentiment in the conversation. I'm looking for the themes to surface to help understand where are all the problems? Because the thing about telling a story is you want it to be compelling. You want it to be interesting. We've all picked up a book before and gotten, you know, maybe a chapter or two in or watched a new series and got into like the second half, halfway through the second episode. It was like, this just isn't my jam, right?

21:17The story of quality is no different. You have to tell a compelling story. You have to explain it in a way that attaches and connects to the business heart, to what leadership is interested in, to the value of the business, which is the customer, which is our revenue, right? You've got to connect it into that conversation. And that takes time. That requires a lot of like moving parts and pieces. But when you understand the sentiments, when you get that feedback, you're at least able to say, ooh, you're worried about availability. or who you're worried about, you know, SEO tracking or, you know, you want to understand our customer sentiment.

22:01OK, well, how can I get that and surface that information? How can I make that visible through the lens of quality and say, hey, listen, we're tracking this and we want to hold the teams accountable to it and start to drive that conversation So you're taking the heart of what the business is interested in and you're moving it through the lens of quality and pushing it back to the teams to say, this is something that we need to look at. How are you helping to improve this? You're like a quality marketer. And I promise that's one of the reasons you've been so successful in your career is being able to connect that story.

22:39That's so cool. I love that. It is not an easy thing to do, right? No. I'm still telling you. But it's interesting. It's not. It's interesting. But people don't always think, you know, the thing and the interesting thing, though, is the frustrating thing is that it's not like explaining that I have to go through that. I can't like it's not common. It's not a common expectation. And so, yeah, I'm like wandering around sometimes. What data do you have? What like what is the data? And people are like, but why? Like getting this data is hard. And I was like, I know. And I don't really have a why for you yet.

23:12I'm actually just trying to see what you're tracking. what do you think quality is and what do you measure? Because now I want to pull it together into a single cohesive conversation and be like, now when we look at this across the board, hey, we have a problem right here. Should we focus in on that? Yeah, that's how you connect the dots, right? And one thing you mentioned earlier, you talk about acceptance criteria. And I'm curious your perspective on this. When should quality members on the team, whether it's a QA engineer or, you know, whatever role it may be, when should they be engaged in understanding the user stories, requirements, or whatever it may be?

23:55At the very beginning with everybody else. Here's the thing. With quality, quality team members have the benefit of constantly exercising the entire system. if somebody knows the ins and outs of your house and you have a problem or you want to make an addition to your house wouldn't you call them first yeah exactly who's constantly you know i'm thinking we just had a problem with our ac and we call somebody that this the it was the same guy that came out and fixed the ac problem we had last time i'm not going to talk about the shadiness that that feels like but we clearly said he was like yeah he's talking to my husband and he's saying because i wasn't out there but he was talking to my husband and he was like yeah this is what we talked about last time here's this that and so forth and so on like he knew the problem which made getting to the resolution or understanding that like just that knowledge push in made it so much quicker to get to the resolution and therefore cost us less money because he's out here last time, right?

25:06Yep. That's QA. QA is constantly exercising your system from the customer perspective, which is who we care about. Yeah. QA is connected to the heart of the customer inside the business. It's the health of it, right? And I want everybody to like pause for a second, just so you don't miss this point. if you're a scrum master product person or whatever it is, bring your QA folks into these ceremonies early on. Because to your point, they can save you a lot of times. They're going to be thinking about something from a different angle that you may not thinking about one. And they're going to be given additional context when they actually are doing the testing, which is going to make their job a lot easier.

25:50So anybody that's not doing that, bring your QA friends into those conversations earlier on. And I will, I will point out like one thing that I often have had to do when I'm going into new orgs, when I have a new team, I have to coach inside of my team because the QA folks can be wallflowers at times. Some of them can be wallflowers. And so they will come into a conversation and are like, yep, I'm listening. I'm actively listening. It sounds odd, but okay. They know what they're talking about. And so I'm just going to wait for to come to my desk and I have the context. But the QA organization, it's one of the things that I love talking to the QA community about.

26:33It's like we are more than just testing and that single step in the delivery lifecycle. We provide that value. We need to speak up. We need to provide the, here's a gotcha. Have you considered this? Have you turned the box in this way? And when team members, when we're brought into this conversation, ask for that. Pull on them. Get the, you know, request that feedback. Like, hey, what do you think? Like, I'm, these, this kind of, these are the boxes. I love to talk about things in the form of boxes. So like, this is the box of the flow. These are the boxes of the flow currently. We want to shove one right there.

27:11What do you think about that? You know, like, what's going to happen? How does that help or harm the journey that you experience and go through and think about from a customer perspective. Is that good? Is that bad? Ask those very specific questions specifically to the QA team to draw them out and get that insight. And that's a facilitator tip there. If you're a Scrum Master product person, one thing that we do in a lot of workshops is we'll always ask around the group, hey, Lisa, do you have any clarifying questions or anything like that? Bob, do you, and you go around the full room and it's funny, a lot of times you'll have people say no, but, and then they'll go into what's on their mind.

27:57So that's a good tactic to kind of get those, like you mentioned wallflowers to speak up because they, they do have an opinion and it's a valuable one a lot of the times. All right. So the hot topic right now, everybody's talking about it. Everybody and their mom, generative AI, you know, we're playing around with, uh, GitHub copilot and some other tools at Hatchworks, but I'm curious, what is your perspective, thoughts, theory, whatever it may be, a prediction on how generative AI will impact the quality assurance discipline positively, negatively, how it evolves? What's your hot take? It's significant.