Simplifying the AI Stack | IBM's Dr. Maryam Ashoori

0:06Welcome to Dev Interrupted, everyone. I'm your host, Andrew Ziegler. Joining me today, as you know, is fellow host, Dan Lines, co-founder and COO of Linear B. Always great to have you here, Dan. Yeah, awesome to be on. Thanks, man. Of course. You know, listeners, we have a great episode in store for you today. It's featuring a conversation with Dr. Miriam Ashuri. She's the head of product for Watson X AI at IBM. She had a lot of really great insights on how AI is reshaping engineering work, and I'm really excited to share them. But before we get there, you know, Dan, I know you've listened to Dr.

0:41Ashuri's conversation and you have a lot of insights too in your role at Linear B about how AI is reshaping engineering work. Yeah, absolutely. I would say I talk AI, so maybe per week I'm meeting with 10 customers or prospects that are utilizing Linear B and we talk AI probably eight out of 10 times, nine out of 10 times. So like, absolutely, I think it's top of mind for everybody. Yeah, unavoidable and unavoidable topic for sure. It's really kind of hard for it to not come up anywhere. Sometimes when you have a conversation, AI doesn't come up, then it's kind of becomes a delight or a rarity these days, right?

1:22You know, let's dig into some of that. Some of what Ameriam is going to be talking about, about how particularly how AI is going to be impacting the developer experience and developer productivity and how AI is ultimately shaping the skills that developers need to be picking up in order to be more effective and to get their work done. I wanted to get a sense of, you know, how does that line up with things that you see when you talk with engineering teams and you see how they're realigning? First of all, like, you know, what she was saying was very interesting, but I can just give my perspective and I'll give it from two different ways.

1:56So one is I'm talking to developer experience teams. So you got developers and you got DevX teams. DevX teams know that they should be or need to be using AI to gain productivity for the business. So that's kind of understood now. Okay, that's my mission. I need to use this AI in some way, shape, or form for my developers. But also what I'm hearing is it's still fairly unknown or unproven. they're having a hard time saying it definitely is improving productivity or not. So for example, the last customer that I talked to about it was like two days ago. And they were like, yeah, we rolled out, you know, like co-pilot to the entire engineering organization, did a survey of the teams, like ask people.

2:47And it was like one team said, okay, I think it's effective. And most of the team said, I don't know yet. And so I think that's what DevX teams are facing. Kind of like, hey, I know I need to be using this stuff. I know it needs to drive productivity. I'm unsure if it is or not. So that's one problem to solve. But I think the other thing about it, and I know that we'll get into it, is like being more, I think, intentional and specific about how it's going to make you productive. From the developer side, I'm also hearing a range of things. Like we rolled some out at Linear B. We rolled out our AI code review to our developers.

3:28Some of the feedback is like amazing. Hey, we need to do this AI code review everywhere. It's already finding bugs and stuff. Like we should keep it. And then other times like you might hear, okay, it still needs a little bit of work. Like it suggests it's, you know, something different or wrong, right? So it's kind of like still in the work. The most advanced thing that I've seen is we had a developer that's actually managing a team of AI bots. So it's like, okay, now I'm a developer and I have three bots that are getting the story done or the issue done. I have one creating tasks. I have one writing code.

4:07I have one QAing. So I know that sounds a little bit advanced, but I think for developers, there's a big range right now of where do you fall? Are you like commanding a team of bots? Are you still learning like prompts and that kind of stuff? So yeah, that's what I see in the field. I completely resonate with that. That's what I hear from folks as well when they talk about the folks that are really getting ahead, that are utilizing this to kind of get ahead in their productivity. They're splitting their work into a bunch of little agents, a bunch of little workflows. But most importantly, they're managing those outputs.

4:38They're being in charge of the decisions and the information that goes into them, but they're not spending as much of the time doing. Which is what Dr. Ashwari really focuses on in our conversation is that shift from being the doer to the decision maker, the decision manager, and how AI kind of gives you that focus. I really like that part around, I think she said like act doers or like decision makers versus doers. It actually made me think of something pretty interesting. like when I was developing and I think it still is today it's like sometimes you have a good manager and sometimes you don't have a good manager and there is kind of this like age-old problem of sometimes companies or people would take their best developer and be like you're the manager now because you are the best developer you're now the manager and you're going to lead a team of developers and oftentimes that goes really bad yes and it's because you're trying to get a great doer and make them a manager and then also like doers and managers there's just this natural conflict yeah that's the classic engineering problem that's exactly right but now when i was seeing like the one of our developers at linear b managing these bots i thought to myself okay we have one of our best engineers who would never want to actually be a manager like manage all the people stuff don't want to do that actually managing bots really really well so she was like yeah i'm the doer through these bots i'm also the decision maker of what they're doing and i'm getting a ton of work done maybe at like 2x 3x i don't know maybe even 10x and i'm like building a complete app so it's almost like i'm just wondering maybe you can combine the best of both worlds now within one human with many bots.

6:31I think that's what we're going to see. The way that you're describing it of maybe someone who doesn't want to evolve into manager role. They love being the doer. That's why you have the manager tracks, you have the staff tracks, and that's why those things are very separate in what they do because of the mentality of how the folks within them approach their work. But when maybe a lot of the issues that make it difficult to become a good manager don't exist when you're working with an AI agent or an AI workflow, just because of the lack of all of the social everything that's hanging around it, right?

7:05You can boil it down to something that's more straightforward and repeatable. So then suddenly you get folks that maybe don't have the traditional manager skills or the managerial output evolving into those skills because they can do so and it meets them at their level. Yeah, it's just I know it's a little bit out there, but like imagine one senior developer that's like, I'll just use our company for linear B. Yeah, I lead 18 bots. I lead three teams of six, but they're all they're all bots. So I work on three products in parallel and they report to me. It's like that could be, you know, the world that we're going instead of like, hey, I lead a group of 18 people.

7:47right and it well becomes important that is managing all of that context you kind of become the living context for all of those uh workflows they keep it aligned with um other co-workers who also all have their own little army of workflows um i think it's going to totally change how companies will even start to structure themselves from the beginning we have even seen really what the new structure of you know a ai native tech startup might look like in a year or two from now, at least on the personnel side. Yeah, pretty crazy to think about. Very cool. We're talking about using these tools and about applying them to do wider level decision making.

8:27Another really great thing that she called out is about not chasing after them per se, but really understanding your goals and what you're trying to go for. You know, what did you think of this? You know, you and I just talked about something kind of maybe it's in the future and just in a little bit, it feels a little otherworldly. Let's bring it down to earth like right now. I think she's absolutely right from the customers that I'm talking to. And this is the classic mistake in engineering. It's, hey, I want to go try a new technology just to do so. But I don't know what output or outcome I'm actually looking for.

9:06That's what I'm hearing when people are like, yeah, I just deployed co-pilot everywhere. Now my business is asking me what it did for me and I can't even answer that question. That's way, way different. And I'll just, again, use the Linear B example of saying, you know what? I measured and I see I have a problem in my code review process. I then went to my CTO and said, hey, look at this data. We have a problem in the way that we review code. It's causing a big bottleneck for us. I am now going to deploy Linear B AI code review or whatever code review you're going to use to solve that problem with AI and prove back to the business that this was a good move because we gained productivity for all the engineers because everyone's doing code review.

9:54That's two different worlds, right? One is like I'm just trying stuff and I'm doing it willy nilly and I'm hoping for a good result, but I can't even measure or say what I'm going for. and the other one's super pinpointed. I know the ROI I'm looking for. I know the output I need. I'm reporting it to the business. I pick an AI tool that's really good at that thing. That's like, I think, high maturity, high responsibility. Yeah, it really speaks to like, you know, when you have that tool adoption, you need to make sure that you're managing it well and picking things that have high impact that are going to lead towards the outcomes you want and don't necessarily play into the hype.

10:29And so you have to understand upfront what you're trying to pinpoint. that way you can pick up the right technology that's going to work for you and still be able to move quickly. Exactly what we just said is what the best DevX teams are doing. Like the developer experience teams that I'm working with that I think, oh, okay, you're like leading edge is doing exactly that. They pinpoint exactly where they're going to help developers. They use AI or some type of automation or bot to do so. And then they report back on the outcome. They can track it. And even better than I can see, okay, they're starting to put rules and governance around it and making it like in the workflow really, really effective.

11:09Like that's what killer elite DevX looks like to me right now. and when you talk about these elite teams that you see and all the great things that they're doing it kind of actually calls to me a little bit about what dr shory was saying about the lack of general expertise and skills around this technology is still for a lot of the industry a lot of the market that she's exposed to we're talking about ibm so she has a really big surface area she had a survey that she did even we're going to talk about it in our interview with her where she talked to over a thousand US software developers and found that, you know, less than 25 % or only about 25 % of them actually felt like they had a basic level of AI comprehension.

11:52Right. And those are, these are folks that are actually building AI tools and implementing AI right now. Um, and so there's a, there's a really big skills gap, um, that she's calling out. Yeah. First of all, I think it's really smart and I think it makes sense. And I'm, I'm kind of I'm asking myself, is this any different than any other technology or skill set game? Yeah, of course. Everyone already said AI is changing the world. That's impact. But that's not different than, hey, I need to learn how to use AI just like any other technology skill that I need to learn. And I think the gap is real.

12:29And I'm sure there's probably already companies doing trainings on it or prompt engineering. I hear that terminology all the time. But basically, it's at the end of the day is let me coach you on how to use this AI stuff to make yourself more productive, grow your career. I think it's the same as anything else. Yeah, ultimately, it's just going to demand a new skill set and an understanding of what those skills need to be. So it goes back to everything that she said and what we're going to talk about in just a moment, all the focus that you can get from AI. And so after the break, we're bringing Dr.

13:00Miriam Ashuri on the show. You know, we recently discussed on Dev Interrupted how IBM is shattering market expectations around Gen AI. And after IBM's Q4 earning results, shares of the tech company, they jumped by 13 % in just one day. And did you know that a staggering 80 % of IBM's tech revenue actually comes from consulting, talking one-on-one with software developers about their problems? And you're about to get a slice of that insight for free right here now on Dev Interrupted. So you don't want to miss this one.

13:33Let's talk about something that's been on every engineering leader's mind. How do you measure success beyond just Dora metrics? That's exactly what Linear V tackles in their latest guide on how top teams track developer success beyond Dora. If you're relying on deployment frequency and lead time or cycle time, you're missing the full picture of how engineering actually drives business impact. This guide breaks down real world metrics that matter. Stuff like developer satisfaction, cognitive load, and engineering impact beyond just chipping fast. Because what's the point of going fast if you can't see where you're going?

14:10If you're serious about improving your team, go grab your free copy now. The link's in the show notes. I'm delighted to be joined by Dr. Miriam Ashuri of IBM. Dr. Ashuri, welcome to the show. Thanks for having me, Andrew. Of course. I want to start by sharing your awesome background with our audience. You're the Senior Director of Product Management for Watson X AI at IBM, where you're focusing on simplifying the AI development lifecycle, expanding it for enterprise applications. Your background includes leading engineering at Lyft, bikes, and scooters, pioneering user experience design for AI and even quantum technologies at IBM Research.

14:52And you hold a PhD in systems design engineering, two master's degrees in artificial intelligence. You're on a board seat for the Virtual Science Teachers, which is a nonprofit for the mission of making science learning equitable and providing free resources for teachers and students. and you're an adjunct professor at Waterloo. And I really don't know how you find the time to do all of that. That's an incredible resume, and we're really appreciative of your time today. Thanks for having me. Of course. As a former teacher, I'm really excited about your mission to make science education engaging and accessible.

15:27And today has lots in store with us, so I'm just going to jump right in. One of your biggest focuses is on simplifying the AI stack for the enterprise. Why is this so important? And what are the biggest hurdles that we're facing? Listen, there's been a lot of excitement in the market over the past 18 months around generative AI. But the market is overlooking the complexity of what it takes to deliver on that potential. Behind every intuitive application, there exists an immensely complicated AI stack that our developers must master and take advantage of and harness to unlock that value, right?

16:10So because of that, we've been focusing and looking into how can we simplify this AI stack for our developers? This is an area that is evolving rapidly. There are many tools popping up every day. There is a lack of AI skills for our developers that we can talk about. And what is the role of IBM in terms of simplifying this AI stack? And we looked into our WatsonX portfolio, starting with that, which is the core IBM AI engine for enterprises to overcome some of these existing development challenges. Okay. So WatsonX is focused on that entire tech stack and not just the application layer. And we certainly hear a lot from how people are adopting it, that it kind of becomes a Jenga tower of systems and designs that combine and you might add or remove things and you're afraid it's going to fall.

16:59And we're moving towards, you know, starting to standardize this, but it's still very early days. So what does it mean that Watson X is focusing on that entire tech stack versus the application layer we might be thinking of most days? Yeah. If you think about a year ago, what excited the market about Gen.ai was the idea of an then accelerated delivery and development of enterprise application. Historically, with traditional ML, the AI lifecycle was starting from data. So basically, enterprises collect a bunch of data, and then a data scientist or ML engineer goes in, creates a model, they train the model, and then they hand it off to developers to incorporate that in AI application.

17:43With generative AI, in theory, a developer in less than five minutes can use one of the LLMs that are hosted somewhere behind an API endpoint and bring those AI capabilities into their application. So the impact and the potential of this technology is significant. That five minutes access to the model that is hosted behind API gives you an aha moment, an initial wow factor that we saw the market earning for or looking for when they were exploring Gen AI and investigating Gen AI. But the majority of the market at this point has moved to from that exploration to production and scale, where the story is different.

18:28When you go to production and at the scale of enterprise, optimization comes into the play. What model are you using? What is the size of the model? The largest general purpose model might be very powerful for a wide range of downstream use cases, but may not be necessarily the best option in terms of being cost effective, right? So if developers really want to capitalize and optimize their investment on the AI, they need to go down the stack, not just application layer, to be able to gain control and optimize the stack. And because of this reason, we've been also looking into the entire stack, not just the application layer, to see what are the points that we can help and make available to the developers to help them optimize every layer and build up on them.

19:18Great. So it's about using it as a tool at all different parts of the software development lifecycle and the process of providing good software at scale and using AI as a tool and the lens to really get in there and solve those hard problems. You've done an exploration about the complexities of generative AI and enterprise, and you spoke with over a thousand developers and uncovered some really interesting challenges recently that you talked about in a LinkedIn post. Can you elaborate on some of these findings for our listeners? Yes. We recently surveyed about a thousand developers in the U.S.

19:53who are charged with building AI applications for the enterprise. Survey participants were coming from a range of development roles, including application developers, software engineers, data scientists. And what we've heard from them was actually reinforcing the challenges that we've been also seeing in the market and hearing from enterprises. The top two that I like to highlight and call out here, one is the rapidly evolving market. Like on a daily basis, a piece of technology is popping up, a new tool, a new model, and then just pair this rapid excitement and the speed of innovation with the skill level of the developers, which is my really the second point that makes the whole application development extremely complicated and frustrating experience for some of the developers.

20:46On one side, the complexity is expanding on the day-to-day basis. On the other side, if you look into who are those developers, there is a wide range of difference in terms of their generative AI skills. The developers that we talked about, too, it's interesting to see that I would say the app developers in our audience, only 25 % of those app developers self -associated themselves with generative AI knowledge and expertise. 25 % of application developers. That's a huge number. And these application developers are all building AI applications for enterprise. So just think about this skill set, the need for simplicity to understand the AI concept, the expectations from the market to go down this stack, really optimize this stack and deliver on the potential of generative AI puts a lot of burden on developers.

21:50And that's one of the highlights that we saw on the survey. A lot of burden, a lot of pressure, and it puts them in this really difficult spot where they have to take advantage of the productivity gains or otherwise start exploring AI now. They can't afford to wait, but they lack maybe the understanding or a way forward or a way of taking that first step. And that comes to a gap in skills. And that might also come from a sense of being overwhelmed. You hit on a really good point that it's evolving every day. Every day you can go on product hunt and find a new litany of AI-powered tools that you can start exploring.

22:25And the implications of this is that there's a lot of tool sprawl happening right now for developers that are building any kind of application. It doesn't have to be AI-powered. It could be a tool that they've been using since the beginning of software development for them, and now suddenly it has AI capabilities. And harnessing and using those tools, it's very different from being able to apply them into your entire tech stack. And so how can developers effectively manage the tool sprawl that we're all experiencing right now? What's the strategy? Well, developers, they need to master a wide range of tools.

23:03Like the developers that we talked about, in average, they are using between five to 15 tools to create AI applications. So five to 15 tools that are evolving every day and a small percentage of them stated that they are willing to spend more than two hours in learning a new tool. So limited time, evolving market, a series of increasing number of tools that are available to them to learn and figure out if that's the tool that they want to integrate or not, and then the burden of integration with an evolving stack to their legacy system. So that's what they are dealing with. Parallel to this, also the developers that we talked about focused on enterprise.

23:49They always stated the need for performance, flexibility, ease of use, and integration as some of the top essential qualities that they are expecting from AI development tools. But then if you look into enterprise tools, these four characteristics are the rarest to find, right? So not available in the market, evolving, and a number of tools that are emerging every day. So because of that, we've been looking into our own stack, figuring out how to simplify the stack in a sense that we are offering not just point solutions to these developers as an integrated stack, providing guidance in terms of what are the available choices and options to you at every level of the stack and some sort of guidance on where to pick what and trying to abstract the development from the underlying AI frame version model.

24:48So sort of hiding away the complexity of what's going on behind the scene, but helping you to pick the right model and an opportunity to customize the solution that you need, which is really what the developers need. Right. And that's really core to the enterprise problem as well, because you have to scale it, but it has to uniquely fit your organization like a glove so that it's accurate, so it's compliant even in some cases, and you can actually count on it to help get your work done. When it comes to enterprise, you have the scale that you mentioned. You have the regulations, especially in highly regulated industries like finance, healthcare, insurance.

25:28You can't just put something out there that you can't trace or have transparency and traceability onto how it's performed, which makes it even more difficult. And last but not least is the complicated, often legacy systems that enterprise has that now needs to be integrated with the modern AI stack. Those legacy systems are, I think, the killer part of this equation. The old hulking machines that you're carrying from the past, those algorithms maybe written in a long dead language that you only have a few experts for within your company now. Those are the things that you have to also reimagine when you think about reimagining enterprise with AI, which is a very difficult kind of obstacle to kind of cross.

26:12And you mentioned that when developers look for tools that they use, and this is kind of a universal thing for tool adoption is they're looking for ease of mastery. They want to get good at it, get better at it, and become an expert in it. They want performance and flexibility, the ease of use, and they want that integration, the openness to make it do what they want to do. And when you talk about AI and how it's kind of like scalable, like thinking and computational, like output of an idea, then when you combine it with these things of like, oh, is it easy to master? is it performant? Is it flexible?

26:43It becomes uniquely tied to the use cases that they're learning. So everyone is learning in these little silos of how to use AI for their specific use case. And it's about surfacing those universal truths in AI application development, bringing it into your tech stack. And so how do you expect that AI developer experience to change in the coming year? I would say the highlight of that is productivity. 99 % of the developers that we talked to and we surveyed used cognitive and really coding assistance in some capacity. So this is like using AI for code generation, code explanation, even using AI for root cause analysis, figuring out what are the root cause of the issues and bug fixes, like recommendations on how to optimize the code.

27:31That's what we have been seeing the developers using. We were told by the developers, the survey that we did, that it typically saves them three hours or more per day for a typical development. Just imagine the efficiency that a developer software engineer can take out of using coding assistant tools to generate, expand the reach of what the application can do. Because now, instead of spending the time on figuring out the root cause of a bug and bug fixes, there are recommendations made to the developers who can just approve and move on and spend that energy on thinking about how to optimize further or add new features or reach a wider audience with a better experience.

28:19So I would expect the coding assistance to continue evolving and massively change the experience of AI developers. One thing that I keep thinking about, about how the software engineer's roles and responsibilities are going to be changed in the future is software engineers that are using code assistant, are using AI for code development, are going to replace software engineers that are not using AI today. Yes. So you're hitting on a really good point about how software developers are using AI now in this cognitive partner, right? When creating code, when working with code, you pointed out a really key thing of like asking it for a code review or to explain what a chunk of code is doing or otherwise to talk with you.