How to Build AI Workflows You Can Trust | Infactory’s Brooke Hartley Moy

0:05Welcome to Dev Interrupted. I'm your host, Andrew Ziegler. And I'm your host, Ben Lloyd Pearson. This week, we're talking about the latest in a series of ThoughtWork memos on AI and the increasing divide between executive employees on their AI usage. But I think, Angie, you had something you wanted to bring up first. Oh, yes. So a little birdie told me that you got a lot of attention on social media in the last week or two. And when you and I chatted about it, it seems like it happened when you didn't have any phone service. Do you want to talk about it? It was a little fun. So I posted this article about Sam Altman saying that U.S.

0:41copyright law is holding back AI. And, you know, I designed it to be a little inflammatory. I'm going to admit that. But I still think I made a good point. But it's funny because I posted it right before I left on vacation and checked my cell phone many hours later when it had hundreds of comments. And, of course, had to turn off notifications. But, you know, my controversial opinions on copyright set aside. You know, I seem to have waded into like the front lines of this brewing conflict that I've seen not just on my own page, but elsewhere on social media. And that is between creative professionals and software developers.

1:16So, you know, people that are in these creative lines of work, like designers, musicians, artists, they are really starting to witness a lot of the disruptive effects of AI. But unlike professions like software development, they don't really seem to be receiving a whole lot of the benefits that it brings along with it. So, you know, I think about software developers, like they really are experiencing a lot of like things that accelerate, that help improve quality, that help them learn new things. And the benefits have been way more substantial. But, you know, in the background, we've also kind of covered in the past how like Chinese companies are now entering this space and, you know, they have zero obligation to respect Western copyright.

1:55So I kind of have every reason to believe that this disruption that creative professionals are feeling right now probably only going to continue to grow, like even if Sam Altman and others can't train their models on copyrighted content. You know, I wore my fixed copyright shirt today because this really matters. This is a matter I can talk way too long about. And most people, their eyes just glaze over. But I hope one thing we can't actually agree on is that today's copyright laws really do need to be rethought because I just simply don't think they're going to work in the AI era. Like the AI was never anticipated when copyright was created.

2:32isn't it funny how timing works how you know you can post any other time while you're at work or dedicated to being online and you don't get much attention but then you post something and yolos throw your phone to the side and you disconnect from the world and you come back and suddenly everyone's pointing at you and you have all this unwanted attention you know and i thought i wasn't even i thought i wasn't even going to post this one and i just did it because like why not i was like it's content that i wrote might as well post it something you mentioned about people in creative work, they're feeling the disruptive effects of AI, but they're not really feeling a lot of the benefits.

3:05I think this really underscores a lot of the disconnect right now that I see between techs and creatives. As someone in more of like a creative role who works in tech, I feel like I'm often stuck between two opposing worldviews a lot of the time when I talk about how I work or what I do. And I know many others are in similar positions. And something I've noticed is that when we use AI for knowledge work in tech, you know, we say context is king. You know, that's what drives good copywriting, good code, good decision management across, you know, multiple iterations. But when you're using AI for creative work, cohesion is king.

3:43That means that the changes we want to apply with AI need to match the rest of the piece or what we had in mind for it. Otherwise, it's really hard to iterate. And this is a completely different set of problems and solutions. You know, in-painting is one of these ways that technologists working with AI to do creative work can change things, but there's still so much to tap into and explore. That cohesion ultimately is what relates that piece with everything else. You see a Van Gogh, you know it's Van Gogh because of the brushstrokes, and you know it's an impressionist painting because of how those brushstrokes relate to each other.

4:18But since tech is really loud and proud about sharing their AI usage in the open. We're better at facilitating those strategies for context. But creatives, they're in a world where they're not talking much about that usage, unless maybe they're dismissing it or disparaging it. There's a large polarizing view on it. As a result, they don't share those strategies for cohesion. They aren't actually able to utilize AI as an artistic assistant in a parallel to how we can use it in tech. So that's just something that I've kind of been noodling on as I think about what is the root of these differences between how folks use these tools.

4:53Obviously, there's other concerns here involved in the conversations around copyright and the models that do diffusion and what they're trained on that are the root of a lot of issues with how artists view the usage of technology. And that goes back to your copyright. They're seeing a different world than we are. And I've been wondering, like, is this almost like a macrocosm of similar trends that we're seeing in software development? Like some teams are really quick to embrace the benefits of AI and have accelerated their work because of it, while others maybe that are doing things like tasks that are more like programming rather than software development, they're seeing more disruption and fewer of the benefits to the point where, you know, I don't think AI is going to take all software developers jobs, but there are many jobs within software development that it certainly will.

5:40And if your entire job is one of those things, then you may feel type of disruption without the benefits of AI. Very much. Very much. But speaking of AI disruption, let's talk about our next topic because it comes from a former guest on our show that I think has brilliant ideas. And I always love reading her work. So what's this story about? Oh, yes. So this story is the latest memo, actually, in a series of memos from ThoughtWorks. This latest memo is from Brigida Boeckler, who, like you mentioned, was a guest on Dev Interrupted back in November. She talked about how we are making sense of agentic AI.

6:15Brigida's back with an article that explores the evolving role of developers as AI-powered coding assistants become more capable and more prevalent. It points out how they can enhance productivity, but it requires constant oversight. And if you've ever been listening to Dev Interrupted, we've heard this again and again from leaders in the space that are using this tool creates so much opportunity, but it also creates so much disruption. And it can really impact things like your time to commit, your team flow, and the maintainability of your code. You know, there are a lot of examples in here in terms of actual applied usage that the author used as she went through different iterations of different tools and what were the problems in her usual process that she encountered along the way.

7:00And it's really a hallmark of, of course, ThoughtWorks memos. It's very easy to understand, but it's also cut straight to the core of how an actual technologist would use these tools. Ben, do you want to maybe talk about some of the things that stood out from the article to you? Yeah, and I want to call out that our producer Adam mentioned, I believe this is the first guest that I ever interviewed on Dev Interrupted. So topic definitely holds a place dear to my heart. But the thing that I really love about Brigida is how she always focuses on very practical examples about the current state of AI.

7:32So this is like cutting edge perspective on what AI is capable of. And why that's so important is because this stuff is shifting so rapidly that I really think she is one of the best voices at keeping everyone in the know about the progress that AI development platforms are making. And there's a couple of points that she made that really stand out to me. And the first is by no stretch of personal imagination will we have AI that writes 90 % of our code autonomously in a year. So I love that really bold statement from her. She does also say that, you know, maybe it will assist in writing 90 % of your code in some circumstances.

8:10She claims that about 80 % of her development is AI assisted, which is like really awesome to hear. And I expect she's probably pretty far on the leading edge of that, just given the nature of her work. And one of the critical recommendations that I picked up from this that she recommends stepping away from GPT sessions when they start to feel out of control. And this sort of echoes my experience. Like most GPT sessions are ephemeral to me. I spin them up when I need them. And then the moment my problem is solved, I forget about them forever. And so, you know, and I think really what we need to take away from this is that engineering leaders, like you can't get complacent about AI assisted code.

8:47You need to maintain discipline around basic code quality standards. It's important that people are trained on those things and that they understand and agree with the practices that you're implementing. Big plus one to the idea of throwing the chat away when it stops helping you or it stops making sense. That's one of the biggest strategies and pieces of device I've given to people always about using a tool like AI is that ultimately if you're fighting it and it's not helping you, step back, start over. And that's a really great observation here from her. It really kind of underscores how, you know, those LLM basics really carry you through this whole process.

9:24And everyone needs to be doing the upskilling work to really understand how this stuff works. So really great article. Really looking forward to the next memo on the topic because I know it's an evolving subject. Yeah, definitely. All the listeners need to head to the show notes. Read this article. A lot of great advice. And I do also want to mention that we didn't get to cover it in this episode because it's a little bit bigger than what we were prepared to cover. But the ThoughtWorks technology radar just came out to twice annual publication, always has lots of great insights about the who's who of software development tools.

9:55So go check it out. Just came out, I think, last week. So what's our what's our last story for today, Andrew? Our last story for today is looking deeper at the divide inside of companies between executives and their own employees about how they perceive their AI usage and its impact on the work that they do. Some really standout quotes and statistics from this article. There were a bunch of posts about it as well from a LinkedIn survey that found that, you know, less than half of employees, 45 % of them think that their company's AI rollout in the last 12 months has been successful. And what makes this such a powerful statistic is that that's compared to the 75 % of C-suite who thought that it went well.

10:37There's a big divide here between employees and how they're perceiving the usage of their AI tools within their company and the impact of it and their executives. Only 57 % of employees say that their company even has an AI strategy, but 88 % of C-suite believe that their company do. So there's also clearly a misalignment between what's the reality within these companies. So that's what, about a 50 % divide between employee perception and executives? Like, that's huge. Yeah, it's pretty stark, especially when you realize, as part of this survey as well, as it goes back to this 2024 LinkedIn report, that said that 53 % of employees were still hiding their AI usage from employers for the fear they would make them look replaceable.

11:23This really underscores how within large companies, small companies alike, there's still a really big disconnect between these two worlds. And there's still so much work to be done to bring them closer together with conversations about how this technology is actually impacting us. Were there other things that stood out to you as well, Ben, in this article? Yeah, well, I love the hiding AI usage from your employer. You know, as somebody who formerly maintained a whole bunch of shadow IT infrastructure, you know, I can't officially condone that practice, but I do understand the motivations. But, you know, I really it's like what I see is this like massive stark divide between employee and executive perception on AI that frankly is shocking.

12:04In this article, I read like there's pushback from employees on AI specifically over a fear of being replaced. Like that's that's that's kind of gross. And there's backlash against executives that claim that AI bots should be in the company org chart. Like which, you know, let's just put this aside for a minute. And I think that's like a gross misunderstanding of how AI is actually going to work in the typical organization. Like it's not going to be this all-encompassing employee that does everything that a human can do. But then there's also like employees sabotaging AI rollouts by just refusing to adopt it.

12:39My favorite quote from this was, it said, we're all sick of the effing chatbots. Don't ask me to use another chatbot. Hey, that's pretty understandable. You know, everything getting shoved into a chat interface these days definitely adds a lot of fatigue. Yeah, I'll get to that in a minute. Chat is not a great, it's an okay interface for AI. It's not the best. But in this survey, actually, it applies to a pretty wide range of knowledge work. But I really think it is representative of something that we also encounter in the engineering world. And that is unstructured AI adoption. So when you give your team AI tools without fully understanding their needs or providing training on how to use AI, you're setting yourself up for chaos, like what is described in this article.

13:22Like I mentioned, chat is not the best interface for AI. Instead of chatbots, you should be thinking about how to deploy agents that are custom built for your workflows. So I call this like the tokenization of knowledge work. So focus on breaking work down into atomic units that are small enough for purpose built AI agents to solve with high precision and high accuracy. And of course, you need guardrails on AI too. There's a robust deterministic processes and tools that out there that come in handy. Use CICD to enforce quality practices, implement static analysis. Don't rely 100 % on human judgment to maintain AI oversight.

14:03Really well said, especially about breaking work down into the small units. I know you've seen a lot of success with it. I have as well. That's one of my biggest pieces of advice for folks right now is knowledge workers. Look at the things that you're doing that you're repeating over and over again. Those are the things that you can solve with these ephemeral one-off types of solutions or chats. There's a lot of potential there for saving time. Yep, exactly. So tell me about our guest today, Andrew. Oh, yeah. So after the break, we're bringing Brooke Hartley-Moy on the pod. She's the CEO and co-founder of InFactory.

14:34And Brooke's work focuses on improving the trustworthiness of AI. Because after all, what good are tools if you can't predict how they'll work? So stick around for this fun chat. It's really going to be a great one. Stop asking if AI can write code and start asking if it improves your workflow. At Linear B, we've supercharged the developer experience with AI-powered enhancements that cut through bottlenecks and save your team hours every week. Linear B now provides AI-generated PR descriptions to eliminate manual summaries and provide instant clarity, AI code reviews that catch issues early so your team can focus on what really matters, and AI-powered iteration summaries that turn retrospectives into automated, data-driven, actionable discussions.

15:19We're already using these features ourselves, and the results are faster delivery, less cognitive load, and better development habits across the board. Want to see it in action? Head to the show notes to schedule a demo or join our upcoming live platform showcase. We hope to see you there. Today, we're diving into one of the most critical challenges for engineering leaders, how to build accurate AI systems that actually work in the real world. Joining me is Brooke Hartley-Moy, CEO and co-founder of InFactory. Brooke has an impressive background spanning companies like Humane, Samsung, and Google.

15:53And now she's leading a company dedicated to solving one of the toughest problems in AI, trustworthiness. Brooke, welcome to the show. Hi, nice to be here. It's so great to have you, and we have lots to discuss, so let's go ahead and jump in. And the first thing I want to talk about is the speed of everything. AI is moving crazy fast, and it feels like some businesses just aren't ready for it. Week after week, we're seeing new businesses and industries jump into LLMs, maybe without even having solid data practices or clear goals. Sometimes we don't even get a clear sense they understand what the LLM can or can't do.

16:30And I know you've seen this firsthand as well. What is the biggest misconception that you think companies have around AI right now? Yeah, I mean, I think we've really been at the front lines of this. I certainly have in the roles I've played before in factory and now in terms of seeing companies trying to use LLMs to solve every single challenge that they might have. as a business. And, you know, to use a trite expression, it's a little bit of a square peg into a round hole situation in some cases. You know, LOMs are this really magical technology, like I'm fully AI optimist, excited about what LOMs can do.

17:10But they also aren't the end-all be-all solution or the silver bullet I think that people have come to treat them as. And not every single AI challenge or even just business challenge to get out of it, not every business challenge should be solved directly with just your run-of-the-mill LM or even a super powerful LM. And when you get down to brass tacks, what do you think an LM is actually really good at for most businesses? It's in the name if you think about generative AI, which is really the kind of AI that we're talking about that LM's power. You know, AI is sort of a catch-all now, but we're really talking about one field within AI or within machine learning.

17:49Generative AI, LMs are good at the generative side. They're very good at being creative. They're very good at content creation. We've seen that in, you know, models that produce images and videos and how much those have progressed in a short time. I'm sure many people have already experimented with things like email writing or blog posts or, you know, the New York Times just announced within the last 24 hours that they were going to use their own internal tool for helping with article creation, research, doing headlines, not going so far as to produce an entire article directly from AI, but to support that process.

18:24They're honestly a very great creative helper. If you want to produce something sort of new and kind of go off into territory that feels unfamiliar and potentially coming straight from the artificial brain of an LLM, you're going to get pretty good results at this point from any LLM-based tool. I agree. It's really helpful for getting over the blank page and getting started with some kind of problem, right? Like some kind of critical mass to get you going. Sometimes you just need to see what you don't like to know what you do. And when, you know, I think you and I can both agree that people playing around with AI and innovating with new things, there's no shortage of ideas or things that you could do to change your workflows with the tool.

19:05But when you get the result from it, how can you really start to build trust in what you're getting out of your LLMs or your AIs? Because, you know, to us, it kind of just seems like a black box. Completely. And I mean, that's somewhat by design. And this kind of goes into the topic of what are LMs not good at, which is they are inherently to the black box problem. They're not really that explainable. It's not very easy to understand where a decision came from. And in some ways, you wouldn't necessarily want to tamp down on that if you're thinking about the use cases that we just talked about.

19:38Like if you want the LM to give you that creativity help you with the blank page in front of you and to start to sort of spark ideation and all of that. You don't want to have it completely locked down to only work with, you know, completely vetted, verified answers and obvious sort of delineation between, you know, source data and other things. But when you're talking about other use cases that people are now increasingly trying to apply AI for, particularly in the enterprise, and across much more critical use cases, that's where the black box problem really becomes an issue. And that's where at the end of the day, it's really about data.

20:19If you are going to trust the output of an LM or the output of any AI tool, you wanna know that it's being trained on the right data, that it's doing any kind of rag flow, any kind of additional grounding, that that data itself is trustworthy and verified. And you wanna be able to have some exposure to like, what are those decisions that it's being making behind the scenes in order to come to its answer? We're seeing this more and more like, you know, operator, the, you know, newest kind of forms of deep research that OpenAI has put out on the anthropic side. All of them are trying to expose more of at least the LLM's thinking.

20:57It still doesn't solve the problem completely of trust, though, because at the end of the day, the LM is still going to be inherently probabilistic in its response. Right, exactly. So what I'm hearing is that the stakes and the quality of the data, if the stakes are high, then the quality of the data needs to be high. You know, I've heard this from other guests as well. And that seems to be a really big trend where when you're establishing trust, you know, if the stakes are high, the data quality and the amount of information that's feeding into that decision making, it also needs to be high. So when you're talking about large enterprises, maybe ones that are typically really regulated and have a lot of restrictions.

21:37Maybe we don't typically view them as like a digital company. There's a very traditional enterprises that are sprawling. And you see them picking up these tools now every week. We talked about Goldman Sachs and how they adopted AI practices within their corporation. And so when you talk about the data quality and being able to use the data, how can enterprises that don't typically have a strong data practice actually get good usage out of LLMs with their data? Like, where do they start? Yes, it's a great question. And it's funny because it's a question that we've been asking for a really long time that certainly predates anything related to LLMs or even, you know, predates people knowing what an LLM even was.

22:21Big data was sort of the moniker of its time. And I've been around the software industry long enough to have heard many conversations around how do we turn what is essentially this massive cost center of messy, siloed, not particularly well-managed and clean data into something meaningful. And there's been various stages of technology that have come around to try to address that. I think the advantage we have now going into sort of the AI era to help these companies that you mentioned that are often more in the legacy side or have not potentially even gone through the full evolution of moving to the cloud or using things like data lakehouses or the other solutions that have come to pass is you don't necessarily need to clean up every single piece of data in your current corpus.

23:10You can actually leverage AI, and it's something that we've really focused on in factory, to really understand what data is worth investing in. What data is going to answer the queries that are most meaningful to your business? Where are the nuggets of gold hidden in what is essentially like, you know, this desert of otherwise valuable, but maybe not so valuable information. And I think a big focus on that is then how do you set that up in such a way that the LLM can understand that data and another application can understand that data that you can use it and get the information that you're looking for.

23:45That's a big focus on what we're doing in factory, which is essentially on the data preparation side, making LLMs enterprise-friendly by helping with this sort of data layer and addressing how do you make this data more useful in an AI context, recognizing that there are huge advantages to that now, but you have to at least take that first step to get them into that stage. Right. And so not all data is made equal and some of it is going to be more important. I think that's a native instinct most people have when they approach data, especially engineering data or things around like finances, like some things are just more important to look at and understand.

24:23And you really highlight, I think for me, what I think is the anxiety when people think about the evolution of AI rapidly within large organizations. And, you know, you talk about like the big data and being able to analyze what you have. And that was already a problem that we had before AI got on the scene. You know, companies were already sitting on troves of data more than they knew what to do with. And they were already going through processes of, having the data warehouse, the data lake house, go through all these different evolutions of how to access it. When you're talking about the nuggets of gold, and maybe you're somebody who's trying to find those nuggets of gold.

24:59Obviously, they tie back to your business, but how can you really start a data forward practice within your organization to make sure those nuggets of gold are always getting found? It doesn't require nearly the heavy lifting that I think people expect. I think part of it is recognizing that there are probably a hundred different queries in a larger organization that covers 90 % of what a business would actually want to know. And so when we talk about big data, I often kind of laugh a little bit about it because yes, there is a lot of data. Corporations have more data than they know what to do with, and they're producing more every second.