Making Sense of Agentic AI | ThoughtWorks Birgitta Boeckeler

0:00And I haven't really ever in a demo video or for myself seen a tool really solve the original problem. There's something weird going on with these marketing videos where even the use cases that they choose for the video often don't make sense.

0:22Hey everyone, I'm your host, Ben Lloyd Pearson, Director of Developer Experience at Linear B. And today I'm delighted to be joined by Brigida Bokela. She's the Global Lead for AI-assisted software delivery at ThoughtWorks. Vegeta, thank you so much for joining me today. No, thanks for having me, Hyben. So we're going to spend a lot of time talking about agentic AI today because, you know, it's been a big part of your research in recent months. And one of the things that I sort of stumbled upon while I was learning about some of the work you've been doing is some of these articles or memos, as you've been calling them, that you've been publishing over on Martin Fowler's website.

1:01and I've seen like it looks like a series of experiments that you've been doing to to test out some of the cutting edge of AI so maybe let's just start there like tell me about like what's going on there and like the types of experiments that you're doing and what you're finding out from them yeah I would say I mean first of all like I'm so I'm a developer by trade right developer architect you know practitioner right so I'm not an AI or machine learning expert and the way that I see my role at the moment is that I'm a domain expert at effective software delivery because I've been doing it for over 20 years.

1:37And, you know, and it's not just like coding, but it's also like effective teamwork and stuff like that. Right. And so that I see as my domain expertise. And now I'm trying to apply AI to that domain. Right. So how can we use AI to be better at coding, to be better at software delivery, teamwork and so on. Right. So and but of course, to do that, I have to understand the technology below the hood, like to a certain extent to understand what are the possibilities and understand if a tool is claiming a certain thing that it can do you know do I think that's like viable like do I want to try that or not like or where do I see this going where do I see this potential right and that's why one of the things I was trying of course like one of the hot topics right now is like how do you use agents or like agentic applications to help with software delivery as well right so that was something I was trying but also a lot of the things that I'm trying is not even me myself trying to build a tool, but just trying to use the tools out there and seeing if I can come up with an example where it's like a, as realistic as possible workflow that I would usually see on a team and just like try to stitch those things together and see if it actually gives me value.

2:44Right. So another one of the things that I wrote about was I used like an issue and like an open source tool that is actually like a business application like tool. And it's like very old code base so you could call it a legacy code base right and I was trying to see okay how would I usually try to find out how to implement this issue this ticket and can AI help me with that right so that's kind of like the the stuff I'm doing so I'm experimenting but I'm also talking to our team so I work for a consultancy right so we have teams in a lot of different domains in a lot of different situations a lot of different tech stacks so I talk to them and you know what what are you using what's working what's not working then I try things I tell them about it I talk to our clients.

3:24So that's kind of like my role right now. It's a lot of different things. And the fire hose of AI, like of change in the space is actually still going. So even with the full-time role, it's very hard to keep on top of everything. So nobody has to feel bad when they feel overwhelmed by what's happening. Yeah, and I love the approach of, you know, you mentioned how you're sort of focused on figuring out what's viable, right, in this day and age. Because, you know, I think we talk to a lot of companies right now that are just, they're like, I don't really know that anyone has figured out how to adopt Gen.

3:55AI yet. They're all like, everyone is experimenting at this point and trying to just learn, like, what are the tools that actually were today? How can we implement them in our organization? And I love the, you know, sort of the legacy case that you brought up, because that actually is one that I think has potentially a lot of promise, right? but also a lot of interest because there's just a really high demand for something that can help manage that. So yeah, so what have you learned? Maybe do you want to dig into that one a little bit? What did you learn about as you were exploring generative AI for a legacy system?

4:30What did you uncover from that? Yeah, so this memo that I wrote basically, so what's always a challenge in this space is as long as I'm not on a team, I need examples. And I need examples that are as realistic as possible. And so we have a team in ThoughtWorks that, like we've had this for years, that is working on an application called BAMNI, which sits on top of an open source application called OpenMRS. And BAMNI is open source as well. And it's like a hospital medical record system, open source. So this is used in countries a lot or in hospitals that can't afford to buy like expensive medical record systems and so on.

5:10Right. So this is like a much more realistic example for me, for my use case, because the business application. So it's not like a Python library or something like that. Right. Because what I'm interested in, like for our clients is like, how does this work for a team building an actual application? Right. Not for like a library or something. So I use that as an example. And what's nice is that all of their JIRA tickets, their Confluence documentation, everything is public and open. Right. So I also don't have to worry about data confidentiality. And I'm not familiar with this application. So I'm actually a user exactly in that use case, right?

5:42So there's like an application that has gone through like, I don't know how many Java versions. I don't know how many different developers, you know, and different approaches, right? And I'm new to this application and there's a ticket and I need to figure out how to implement it. And I don't know the domain terms, right? So it was mentioning things like HI types and FHIR. So all things that I didn't know, right? And I wanted to see, of course, I can search their wiki for this, but can I also ask the code base if it can explain these things to me together with like what an AI model, a large English model would know about these things, right?

6:18So I used a bunch of like different tools to like actually ask the code base, like or based on the code base. And actually, you know, it actually helped me explain some of the domain terms. And also what it's quite useful for is it's like a new form of more powerful code search, right? Where I can go beyond just searching for a string in the code, but I can actually ask for concepts or feature descriptions. And in some cases, I think maybe I would get the same result as a string search if it's like a very specific term. But I can now ask like, where is the, you know, where is, where do we have a list of HI types?

6:54These were like health information types or something, right? And it was actually pointing me to like an enumeration or something like that. So I could find those types of things. That has a lot of promise, I think. Like it wasn't perfect, but I also know that technically there's still a lot of potential how to make these searches better, right? Like some of my colleagues, for example, are working with clients on loading code bases, abstract syntax trees into knowledge graphs and then enriching them. And in this way, making the retrieval augmented generation that is doing the code search, even better, right?

7:27So I know there's a path. I can see a technical path how these things can get better. But then there are other parts of this like journey when I have a code base that I don't know and I need to implement a ticket, right? So now I found the place where it's supposed to be implemented, but now I have to reproduce the current behavior so I can implement the new behavior, right? And for that, usually I would have to do things like either run the application and kind of like find the click path to where I need to do this, or maybe it's about an HTTP endpoint, so I want to run that and try it out, or I want to write a test, or I want to find the right test to change, right?

8:06And this is where it got a lot trickier. So in this case, this is a very old code base. So this was Java 8. This was still using Vagrant. I don't know if anybody remembers that. I think it's not that long ago that we used Vagrant, but a lot of maybe younger developers today wouldn't even know that. So it's still like AI can only help me there only so far, right? And like a complex combination of different old technologies. So I would still have to rely on the documentation. You know, maybe AI can help me debug some things, but I didn't quite go down that rabbit hole. But it was like a realization I had that was like, oh, you know, like even running like older code bases isn't that straightforward, right?

8:49And there are other problems there and setting up my machine and so on that maybe AI is more in the way than helping me. And then the next problem I ran into was this test problem. So for this particular part of the code base, there actually was no test yet. So this is also quite common in these older code bases. Or I hear from clients a lot, oh, we have this old code base that doesn't have any unit tests. Can we just add tests to it now with AI so it's safer for us to change this when we have to, right? And so in this case, I did not have an existing unit test. So the problem I ran into was that the code wasn't very testable, very unit testable, right?

9:31So the AI tools were suggesting test code to me that wasn't really useful. Like it was stuff like the AI suggestions were mocking a lot. So even mocking to the point that it was mocking the things that I actually wanted to test. And then when I was trying to actually use real data inputs, real data structures, I noticed that the data structures used in the code were very convoluted and like a deep hierarchy of objects. So I couldn't just mock that away. hey, I wanted the real data, but AI couldn't help me set up my test data properly because I think it was like the network of these objects was too deep.

10:10It just wasn't quite figuring out the full path. So I kept running into null pointer exceptions and I never succeeded like to write a test with the help of AI that would actually help me. So it was like a great example of, you know, if you don't have unit tests in a code base, probably AI cannot help you with that unless you first refactor the code base so it's easier to test. I've found that a lot of these models just don't work very well if you don't have a concrete example for them to reference as like the baseline. You know, so if you have zero tests, it's like you really kind of have to go from zero to one before you could even expect like most of these models.

10:47Exactly. Test generation works a lot better when you have some examples already and you have like maybe a setup, you already know how you want to do your mocking and all of those things, right? Yeah, but I think one theme that I think I've picked up here is that, you know, particularly when you were talking about asking the code base, it's like there seems to be a lot of value in going from like knowledge acquisition to some sort of practical application of that knowledge, right? You know, because you think about conventional search, like text search, that's how you just find knowledge. But then the human element still had to like bring together multiple knowledge sources to craft like a solution for whatever situation they're in.

11:25So it sounds like that's been successful at the very least. And I also think that this, exactly this, like bringing knowledge together, the right knowledge for the context and kind of like putting it together for the prompt to the large language model, right? The retrieval augmented generation. I think that's where a lot of the improvements in the direct future will come from in these tools and not from like bigger models or like if only we trained the model to even better at Java or something like that, right? but these tools and how good they can be at that. So there are some products also coming out now where they really try some much more sophisticated approaches to pull in your documentation and your JIRA tickets and your code and put it into these knowledge graphs and really categorize that knowledge and then try to figure out the intent of the person working in the tool, right?

12:15What do they want to do and what would be useful information? And the degree to which these tools can do that for us effectively I think that's the big potential, right? Like when we don't have to do all of that anymore. Yeah. And, you know, I've found myself thinking about, so let's maybe use this to segue into like the core of what we want to talk about today. And that's agentic AI. And I think part of the challenge of this is that everyone sort of comes into this discussion with like a different interpretation of what this means. So, you know, a lot of days when I think about agents or AI agents, you know, I'm thinking about having specific GPTs that I've prompted to behave a specific way so that, you know, I do a lot of content, for example.

13:01So I've got the content strategist that gives me ideas. I've got the writer that produces the copy. I've got an editor prompt that will help me edit it. So, you know, as a, like from the non-developer side of me, because I kind of viewed it that way. But then there's also this like agentic AI tools, like developer tools category that seems to be emerging. So maybe we can just start with like, how do you define like agentic AI? Yeah, as a non-data person, right? Like you said in the beginning, yeah. So for me, like in my head, the simplest definition of like what I think of when I hear agent is that you have a model and you tell the model that it has some tools available, some actions that it can take.

13:46And then the model will tell my application, oh, I think we should now run some tests. Please run tests for me. You told me that you can run tests. That makes sense for it. If you think of it as a conversation between my application that is the agent and the large language model, right? So that's like the minimum that you'd give it some kind of tool, right? And then, of course, people talk about multi-agent systems or where you actually have multiple of these applications that then talk to each other. Almost like one agent is a tool that you provide to the other one, right? And that's when it gets a lot more complicated, right?

14:21And I think, yeah. So I think when people say agent, we always have to investigate what do they actually mean. It's one of those terms like service that is very quickly becoming very overloaded. So we always have to ask each other. There's no wrong or right definition, I think, at this transitionary time also when we're still figuring out the terminology. We always just have to ask each other, what do you mean in this context when you say agent, right? The same we have to do that with service or unit test for that matter. Yeah. So given that we're now sort of on the same page about how we view this technology, what do you think are the most promising agentic services or tools that are hitting the market right now?

15:00I mean, there are all of these tools that call themselves like software engineering agents or developer agents. And a lot of the coding assistants are starting to build these in. So like GitHub Copilot has GitHub Copilot Workspace, which I think is still in private beta. I'm not, I keep losing track of what is generally available and not. There's, I don't know that Amazon's product has something built in. There's a tool that is very focused on test generation called Codo. They are also experimenting with agents like that. There's open source things that do that. This is Benchmark, SWE Bench. I think it's from Princeton University, if I'm not wrong, where they're trying to benchmark these tools.

15:40But in the Gen AI space, benchmarks should always be taken with a grain of salt as well, because you always have to investigate, like, what are they actually testing? What are they comparing? So we also can really rely on that, right? So I've watched a few demo videos. I've tried a few of these myself, right? And a code base that I have, and I know, oh, I have to do this little extension, and it actually uses a lot of the existing components I already have. So it needs to create a new React component, it needs to create a backend endpoint, and then it needs to stick those two together, let's say.

16:12So I always think about, okay, what is something where I already have code, but it's kind of relatively straightforward? Because that feels like a good sweet spot, maybe, I think. So it's not quite totally simple. So something that I could do myself, but it's also not like super complex and new and there's no example. Right. So these types of sweet spot things, I was looking for my code basis and then tried these tools. And at the moment, I am not very like excited about them, to be honest. Like one problem that I have is that they seem to often be advertised as like, oh, these things are going to be our agents that can solve like any type of problem for software.

16:50Right. Which is maybe a marketing problem, you could say. Right. Because I don't think that that promise is fulfillable. It's just too much, the scope, right? But what I do find interesting is, are there any specific types of problems that maybe, yes, in the next few years, we could actually get to a point where these can help us, right? Like what I was just saying, like maybe things where we already have code, it's relatively straightforward. Or, you know, what I was writing in the memo is about like a tech stack migration that is not quite as straightforward as a Java upgrade, but has kind of like this fuzziness.

17:24So it's still like quite a lot of work manually. Like a good example today is like migrating enzyme tests to React testing library because it's a very common use case of something that's deprecated right now. So maybe there are these areas where this will work better sooner, right? But then at the same time, when I look at it, I've had these experiences of like, I was talking about this standard case of I want a React component that calls a backend endpoint. I have lots of examples, right? So, and then like an agent is like, I see an agent kind of like go through, you see the text come through the text generation, right?

18:00Here's a plan. This is what you should do. And then I see these one or two things where I'm like, oh, it shouldn't name it that way. That's not a good term how to call this variable or this concept or something like that, right? And we still have to edit this code in the future, right? So we still don't have AI that also takes care of the maintenance for us later, right? So humans still need to be able to maintain this and change this. So it's important to me that certain namings are right. So now I have this big plan that I waited for two minutes for the AIs to spit out, and I have these two little comments.

18:31I want to change the name of that and the name of that. So then when you say, please change those names, it goes and does the same thing again for two minutes, right? It's just like an example of how it seems very tedious at the moment in the user experience for me as a developer to work with the AI to change the plan. So when I look at what Copilot Workspace is doing, it looks like a relatively okay user experience of how I can edit things, right? But this is like one of the things that I'm wondering about. When we have the AI solve these larger problems, how can we still like tweak the little screws and how is that still a good developer experience for us?

19:06So that's one of the things I'm wondering about. And I haven't really ever in a demo video or for myself seen a tool really solve the original problem. There's something weird going on with these marketing videos where even the use cases that they choose for the video often don't make sense. There's a bunch of videos on YouTube of people ranting about that. So I don't know. It still seems very immature. So I'm holding my breath. I think the pie-in-the-sky vision of this is being able to go from a Jira ticket to code that's in production using completely an AI workflow. But just from the experience I've had working with these tools, It's like really the more focused you have each model on a specific task, the better.

19:52And you really kind of have to have at this point in time, you still have to have a human in between each one of those points to validate and make sure that, you know, the input for the next part of the workflow is actually going to be successful. And, you know, I think, yeah, I agree. The marketing around this stuff has been wild, probably because it's so hyped. But I remember one statement I saw a company have on their website. I think it just said, unleash an army of junior developers under a code. And I'm like, on the surface, that sounds amazing. Because you could do so much if you had an army of developers.

20:25But then you think about how much supervision those junior developers are going to need. It's like I said, a promise of a threat. Exactly. No shade to the junior developers out there. Like you're not in an enviable situation right now. So is there anything like today that you think people should be paying attention to as like, like it's here now or if it's not, it's close enough that you should start figuring out how to adopt it into your offer delivery? Yeah, I mean, the coding assistant products that are out there today are actually quite useful already, right? Like for things that we can do today, right?

21:04So GitHub Copilot is a good product. There's this IDE called cursor, right, that I always call it like on my slides, like the developer favorite, you know, because a lot of developers really like it. And it's also like a great tool to look at what are the next upcoming ideas, because they put a lot of interesting things in that, you know, maybe sometimes don't quite work yet, but that gives you an idea of, oh, what if we could make that work? That's really interesting. And they have interesting ideas about user experience. All of these coding assistants now at least have awareness of your local code base that you have open.

21:34So they index the whole code base and you can ask questions about that code base. So that's super helpful. There's more and more things emerging in terms of like what is often called context providers. So that you can say, for example, like a bot, right? So that you can say at Atlassian in your coding assistant chat and it will, and then ask questions about what's in Atlassian, right? So there's like all of these, it's almost like an ecosystem emerging now with like all of these extensions and these tools integrating into each other. So again, that you can have more context and like better context orchestration and better curation of what's relevant in the moment.

22:10So that's really interesting. And what I also would like to see in more products is this idea of like sharing prompts with the team. So there's this open source coding extension called Continue, which is, by the way, also really nice if you want to try coding systems with different model services because you can plug in local models, you can plug in, I don't know, your Google Gemini subscription, your Anthropic subscription, all of these different models. So it's kind of nice for that as well. But one, like maybe not so like flashy looking feature that I really like that I haven't seen in that many of the other products is that you can create a folder and put in prompts that maybe on your team you're using a lot, right?

22:51So just the other day I was trying this thing where, let's say you say, summarize all of my local changes as like a change summary, right? Like let's say it's a basis for release notes or something like that, right? Because you can point it at your local git div. or you can ask it for code review, of course, like these models are not that great yet at code review, right? But you can do that, right? And then you can have these shared like prompts where maybe you have some of your coding conventions that you always want to check, or this is the format of the change log that we want to have, and you can have that as custom commands in the chat, right?

23:24And this way you can kind of codify some of the practices or conventions that you have on your team in these prompts and like share them with the model when it's helping you, right? So I think that's quite promising. Also, such a simple thing, right? And it gives you, as a team and as developers, it gives you a lot more transparency about what's actually going on. Because these tools often do all of this orchestration under the hood. And sometimes it feels like magic, but you don't really know what's going on, right? And sometimes it's very easy. You just say, okay, I want to point at my local Git div.

23:54Here are my instructions. Go. And you know exactly what's happening, right? Yeah, I mean, you know, a year and a half ago, none of us knew anything about writing prompts for these things, right? And today, it's like, we're all sort of collectively trying to figure out what it takes to do it the right way, you know? And it's actually a practice that my team has adopted as well. Like, when we come up with interesting prompts that help us be more productive, that's something we've started sharing them with the team and commenting on how we can get it better, you know? And if something only gets us 80 % of the way, how can we get it 85 % the next time?

24:28I think that's a really great practice for teams to take away. And this term prompt engineering is also kind of annoying, right? Because it's another one of those where we're not quite sure yet what do we mean by that. And the way it works changes every day, it seems like. You know, the models update. Yeah, and the word engineering, again. And, you know, so some people, like sometimes when you say prompt engineering, you just mean the way you write your prompt when you type into chat to PT or something. Right. But then there's like a much broader definition where it's like all of the retrieval augmented generation engineering that you do around it.