AI Agents Fixing Your IT Before You Even Know Something Broke | Erhan Giral & Ryan Manning, BMC Helix

0:00What's that going to do to IT staff? It sounds like it's going to get increasingly automated. That space is changing pretty quickly and that workflow is changing. Egentic AI pretty dramatically. Right now AI, if you think about it, AI is only learning through someone else's description, right? People will start exposing these models to the real world more and more so that they can create, they can gain that first person view of things. Once we detect something, we always ask the data, why, why, why, until we get to a point where we can take an actionable step to mitigate or perhaps even remedy that issue.

0:34Why don't you introduce yourself to listeners? My name is Arhan. I run the AI office for BMC Helix now. I work with Ryan on applying AI to various different problems in service ops space. So service ops is essentially you can think of that as any digital business these days is really a conglomeration of IT services you can imagine. And service ops is essentially an attempt to automate operational activities around these services as much as possible. so that we look at enterprises as a combination of software, infrastructure, and human beings operating on these assets. And we provide various products and nowadays agents to automate different workflows around these services.

1:32My personal background is in monitoring space. I've done early work on monitoring applications and infrastructure and then combinations of them, which means you have to process a lot of machine data. You have to be able to reason about high throughput streams of data that these data centers and these machines are constantly emitting. So, yeah, that's what I do in Helix. We're going to talk about Helix and IT service management. Can you begin, one of you, by talking about how the landscape has changed with the introduction of AI agents? What the process for service management was before and then what it's migrated to today?

2:27Yeah. Thank you.

3:06yeah uh and and you know just thinking back in the old days it was jira right you would you'd open up a jira ticket uh it is uh it is does uh your solution work alongside jira replace it just for people that aren't that familiar with. Yeah.

4:04Thank you.

4:25And you guys, we'll give the background of BMC first. You were talking about that before we started.

4:50Yeah. Yeah. So, I mean, service management, the discipline is complicated. The explanation of what they do is pretty simple. It's the front door to IT. So if, you know, I'm a new employee trying to get my laptop provisioned or if I'm having an issue with an application, I might go to a portal, send an email, call the service desk and try to get that remediated. And that space is changing pretty quickly and that workflow is changing with Agentic AI pretty dramatically.

5:54Yeah, no, that's, Jira does have it. So our competitors, you know, go beyond service now into Atlassian, into Freshworks. The market is a lot of players. The interesting thing about the service management market is that unlike the CRM market, to broaden it out for a more general audience. Describe what IT service management is and how critical it is to enterprise and the economy in general. In the high enterprise where we operate, we'll say the Fortune 2000, you submit a ticket, you're either submitting it to ServiceNow or to BMC Helix. Down market, Jira is a very popular tool. Freshworks is a very popular tool.

6:44I volunteer a few others. Yeah, I like to think about BMC in four eras. Era one, remedy rules. A lot of folks know remedy. That was an acquisition that BMC made years ago and was the ServiceNow and Service Management before ServiceNow. It was the on-premise ServiceNow. Era 2, about 2010, the music began to slow a little bit, lost our way a little bit in terms of hopping on the cloud train. The company ended up getting bought by Bain and taken private. It sort of continued the same mission. then there were three KKR came in and said hey ServiceNow still only has one competitor it's VMC Helix what if we took a different approach what if we brought in a technical team in the operations of those companies it's kind of the critical Broadcom I'm in Syed, Ali Siddiqui were asked to come in and rebuild the platform from the ground up you know, AI first.

8:09And that, that journey, you know, took us to close to where we are today where we, you know, went through the natural progression of the heartaches for our customers and for ourself and rebuilding a platform and still servicing some of the biggest companies in the world to now being able to innovate and separate ourselves from the competition with the innovation that we got access or permission to build through KKR's ownership.

8:51Yeah. Yeah. And there's the service management being the front door to IT can encompass a lot of different requests that need to get fulfilled. Some of them are very simple you know i need a new iphone um some of them are hey the quoting tool is down it's the end of a quarter um we're in trouble and there's all these workflows that sit behind that request some of them more complex um when those more complex general model issues come in talks to the engine room i think everyone everyone keeping those services up what we've been trying to do is there's a process to remediate that issue and so what we did in that transformation i talked about is we brought those two disciplines together service and operations naturally in category service ops because when you because when you end up sharing data ryan was talking about communication you have to be able to quickly process all the telemetric data You don't have to worry about lots of outages taking too long to restore that outage.

9:58Understand that graph and those relationships. And certainly less regression to the same thing happening again. Natively. And in IT, it's very common that you have to constantly deal with deep relationships. Because something that runs on a computer turns out it's actually not a real computer. It's a visual computer that is resourced by a physical computer somewhere else. So there is all these resource dependencies, resource and transactional dependencies. in the fabric that you have to understand before you can be, you know, you can produce anything actionable and useful. Because on the operation side, what we want to create is not, you know, on the service management side, a lot of disruption, what we want to eventually create is we want to be able to create new execution plans that other agents, that other missions can pick up and then one of Microsoft teams type something in a virtual agent.

10:51And that does require to be able to do that in a reliable sense. You have to create repeatable, predictable infrastructure from these models. And there is so many relationships between investment in that regard to keep that service option and running together and viable. CMDB where they store all their assets with asset information. Can you walk us through an 18 million relationship amongst those assets? And so going to that space, it's not like I was just plugging LLM in. And Iran, hopefully you can expand on this. And you get your answer and you get that remediated. From outside in, it's actually very simple.

11:32So you first need to understand, hey, do I have a problem? To get to root cause. Why is that happening? Should I mobilize anything at all? Do I have a problem? Awareness and identifications of problems is one use case we constantly deal with. And that requires our agents to stream this telemetric information. This can be your machine logs, time series information, alarm information that they emit. So something needs to be there and constantly monitor these things. And we have a funnel that deals with that. Let's take the observable data and run that through various machine learning models so that we can understand, oh, that spike that you see right now at 8.30 a.m.

12:17on a Monday on the bank's ATM network is actually kind of unusual. We don't expect that spike to appear at 8.30 perhaps. Maybe it's supposed to come in later. so something needs to essentially constantly reason about, hey, you know, what's unusual or given the time we are in or given the workload we are dealing with, given the circumstances in general we are. So that's one part of the big use case we try to do. And of course, once you detect an anomaly, once you detect an availability problem, you then first ask, okay, so why? You know, what is the root cause of this issue? You know, why is this router so busy, you know, where it should be, you know, close to idle state, for instance.

13:01So then we call that the root cause analysis. So essentially something that needs to dig into all that telemetry data, dig into all that observability data to say, ah, you know, it's actually, it's not that, that's a symptom. what's really going on is such and such file system or such and such network queue, such a broker somewhere is now full and dropping the traffic or what have you. So we always ask, once we detect something, we always ask the data, why, why, why, why, until we get to a point where you can take an actionable step to mitigate or remedy that, or perhaps even remedy that issue.

13:46And while doing that, of course, you know, data centers and IT is very target-ish. All kinds of stuff happens all the time. So you need to be able to do this by sifting away all the noise, all the background noise that typically happens in that environment. But it's like there's always this cosmic background noise that you have to essentially first detect and then push away to then focus on the real signals in the data to understand what the root cause is. And then we also do impact analysis. Like, okay, so is this a big problem? Because sometimes technically challenging problems are just that, you know, no one cares about.

14:29Maybe it's a QA environment, maybe it's a staging environment that's being modified at the moment. So essentially from an economist's point of view, we always ask, okay, we have a problem, but is this an impactful problem? Should we mobilize human beings? Should we spend more resources in mitigating this issue? So our system constantly makes these decisions in the background by looking at real-time status of systems while also constantly comparing the state to its past self. And we also make plans of action based on plans of actions that were executed in the past by human beings or by bots. And this, well, first of all, this is all being done within an agentic layer.

15:24It's not, and with reasoning models, to look at the problems and try and figure out. It's not kicking questions out to the human team who then sit around and try and figure it out. It's doing this internally, right? That's right. So some parts of what I just described that is actually taking that gigabytes of gigabytes of monitoring data and then reducing and then essentially pushing that noise away. That's typically we employ proprietary technologies to do that because there's so much data to process. We couldn't really hope to expose all of that to the generative models all the time. So we take all that sparse data and run through various machine learning and statistical analyzers first to basically find the needles in the haystack, if you will.

16:29But once those needles are found, they are collated, combined, correlated, causally analyzed. And then the LLM, you know, the LLM is given a pretty comprehensive, causally described, like a patient chart, like a set of x-rays of the system. and then we ask LLM, okay, I mean, we ask various questions to LLM, but most importantly, we ask the question why and what needs to be done. And that level is completely agentic, just like you said. And there we use reasoning models, just like you said, maybe with a twist, because we want our models to reason just like one of the employees of our customers.

17:25Because, for instance, if you go to an open AI model or a generic off-the-shelf model, they'll always give you plausible responses, accurate responses based on the documentation, based on what they know about the space. But it's often the case that when you go to a big organization, the rules of engagement around these systems are very different. Meaning you can say, oh, go to this file and make these edits and everything should be fine. but then that bank's knock, you know, network corporation personnel will probably say something like, well, you know, that sounds plausible but that's not how we do things.

18:02You know, I need to first, you know, run these additional tests. Maybe I need to talk to some other team, get their approval. You know, I need to make sure, you know, there's a backup procedure. I need to estimate this. There's lots of lots of things that they need to account for and then that's often specific to the enterprise, to the day they work. So what we do is we take the reasoning capabilities of these models and we fine-tune them so that they can reason just like that employee of that particular enterprise. So we found that when you tune these models, especially the planning aspects of these models around the subject matter experts' practices, they become extremely, extremely capable.

18:56They really produce actionable and relatable results for the users. So just to point to your question about reasoning, yes, we rely on reasoning, but we try to guide that reasoning towards economic ways of solving that problem. That's another thing great. So a lot of times when you ask complex IT questions to these models, they'll come up with plausible plans. But then is the plan actionable? Essentially, if the solution is, you know what, tear down everything, rebuild everything, this is a rewrite situation, then the customers won't be happy. They'll say, that's too expensive. I can't really do that.

19:39And so then we train our models and we reward them in ways such that more economical, more expedient ways of lower risk solutions are preferred over others. So there's a lot goes into the guidance of that reasoning. But at the core, just like you said, we do rely on LLM's reasoning capabilities to figure things out, to do the orchestration especially. Yeah. And so that whole reasoning workflow, is that one module? I mean, we were talking before we started recording that you guys build your modular. So you have these microservices that work together that presumably then you can configure or pick and choose among them, depending on the use case.

20:39Is all of that reasoning one module? Yeah, yeah. So we follow the similar philosophy of divide and conquer, you know, create good encapsulations around capabilities. So we follow an architecture pattern called mixture of experts. Yeah. What this is, is it allows you to, you know, these pre-trained models are trained on piles of piles of publicly available textual data, if you're talking about language models. and what we found was actually what we not we didn't find this but a meta sort of led the way into this architecture in our in our adoption they they described they had this wonderful people paper to talk where they talked about you know how how these models can be further tuned for different purposes while encapsulating that training in a specific set of weights and biases called experts.

21:42So what that is, is you take one of the open weight models. Open weight model means a model that you can basically download your computer and run as much as you want based on, of course, depending on the licensing agreements. So what you do is you take one of these open source models, and then you identify a problem data set and a reward function. So you essentially, you basically devise a clever way of signaling the network, hey, you know, here is, when you give me a good response, here, you know, I'll give you cookies. If you don't, if you fail, you know, you'll get the stick. So you basically train these models on additional data sets that allow you to focus on that problem.

22:33And then you capture these as what we call experts. So experts are, just like you're saying, they're mini, I mean, I wouldn't say they're services, but they're encapsulations of that training. They are the artifact of that training cycle. and within this mixture of expert architecture, you also specify gate activation values during your training, which means the model then knows when it's prompted with a question whether to activate that training set or not. Like, for instance, if you ask our model, you know, what's the weather like in France today? It will give you an answer. It will do some tool calls and figure out the answer.

23:15but in that prompt, nothing will signal that, oh, they're asking me a question about like a root cause analysis on a mainframe Z system. So it will detect that and it won't activate that part of its training. But if the question is, hey, I'm in big trouble, how do I restart, how do I reinitialize this L part on my mainframe? Then that activation layer kicks in and says, oh, okay, so it looks like they're talking about this other thing that I was trained on, and it loads those weights and biases that you create during your training. And then those weights and biases sway the network just enough so that whatever generates plan or output summarization or what have you is now affected by that training.

24:09We found, and it's not just us, but also this is throughout the academia and now also in the industry, the best way to generalize a data set, a training data set, is really to go through these trainings so that the model can reason about the native in a model-native way. So that's why we've been pursuing this for some time now. Yeah. And then so once the reasoning is done and the system has identified a problem and designed a plan of action, is that then passed on to an agent that executes that plan of action? Yeah. So when we first started, we just started with a textual recipe that we would hand off to the user and say, OK, so here's what you need to do.

25:09This is the recipe I need you to follow. And maybe there's like eight to 12 steps in there. And then we call it a day. But of course, you quickly realize, well, OK, so first of all, some of these actions are very automatable. Like once we break it down to a problem, a problem into little steps, it already, you know, automatically suggests automation. So in that regard, what we first said was, okay, so a lot of these are first, let's verify the problem. Let's do additional diagnostics. It's actually a lot of read-only operations. So we categorized, we thought, you know, we trained our model so that it knows how to categorize and label these actions as, well these are diagnostical so you'll never break anything by just doing these so we started automating those as you can imagine some of the actions are on the remediation side of things which means you know oh you need to go to this configuration and actually change that port number or go to this machine and allocate more file systems so it's actually things that touches the system We still require a human in the loop for them to approve the plan before any such automation is actually called upon.

26:31So right now, essentially, we are in an assistive capacity, which means these human beings are now fed these recommendations and AI-generated plans of action. and as they do things and as they adhere to the plan or deviate from the plan, we actually track that lifecycle, Craig, so that if any of our recipes are executed and everything is fine, that's great. That's great feedback for us. But we actually learn from our mistakes more, meaning if, let's say, for instance, we tell them, hey, you need to do A, B, and C, but then if they do A, B, and X, then we actually learn that once that ticket is closed so that we can, even if that subject matter expert is not really donating and writing a lot of descriptions of things we want to essentially understand their intent and the real way they fix the issues and we train our models as that flywheel turns as they respond to tickets and leave little digital traces of their actions.

27:45The human in the loop. With a lot of these autonomous systems, there is something called automation bias where people become accustomed to accepting the machine's recommendation because it's worked the last 10 times, and so the 11th time you don't even think about it. You just say, go ahead. I mean, is there a point at which this will be fully automated? And, yeah, and the human users will sort of be monitoring outcomes but not necessarily approving every action. Yeah, that's certainly where it's going. There's definitely huge economic pressure in automation, of course. One thing that you said is automation bias is very true.

28:54to address that we have built various quite interesting fingerprinting techniques of incidents and issues essentially when something goes wrong we keep a very detailed record of what the machine signals were what human beings have done and locality of that issue how that issue transpired like what was the first domino that toppled over what was the second domino. So we create these causal traces of incidents and issues in the enterprise so that we can say, oh, you know, what's happening right now is something that just happened last week. And it seems like every Friday you guys are going through this.

29:40Hence, it's highly likely that this is that issue. So it's actually to combat that and also give confidence to people that this is something they can actually tackle with automation. We have done a lot of work on fingerprinting, fingerprinting issues. And that is actually in IT, in this sort of problem space, it's very important to know whether you are dealing with a brand new problem versus something that happens with some frequency. There's tremendous difference between them. Essentially, being able to say, you know what, I don't know, I'm throwing the towel in, you really need to pay attention to this is a great asset.

30:32That's not a failure of the product. When it does that, we are extremely happy that we've been able to do that because one thing that you've noticed probably, LLMs will never tell you, oh, you know, I don't know something, right? They're very positive. They love to generate. So it's actually to get them to say something like, well, you know, this is really out of my bounds is difficult. And that's something we've been working on ever since we started this. But of course, if you can sufficiently fingerprint these incidents and issues and can sufficiently confident to say, hey, this is actually something we dealt with and here is how we solve these problems, then there is a lot of economies there.

31:27So you can imagine that the pressure in the market is always towards, okay, so can we fix these reoccurring problems, which tends to be like 80, 85 % of the problem space? What can we do to automate these as quickly as possible? That's essentially what people like Ryan and me think about all day. Yeah. Well, one of the things that people have been doing is they build this kind of a committee of LLMs that check each other's work and then come up with a consensus about, you know, which, what is the optimal output or plan, whether that's a plan in action or, yeah. Are you doing anything like that?

32:18Yeah, we absolutely do, which is interesting. So we have this new agent called Deep Root Cause Analysis Agent. So this is something that you unleash on big problems in a postmortem sense, meaning problem already happened and you're trying to understand why that was. And you have a lot of time, so you have a lot of research time. It's actually, it's not a, you don't have to be, you know, counting milliseconds necessarily. So when this thing is launched, it quickly formulates an opinion, a theory about what it is based on all the available data, all the snapshot data that was handed to it. But it has agency.

33:06So, in fact, when we say agentic AI, what that really means is, hey, do you have a system that can take initiative to dig into data and come up with an outcome? Come up with an outcome that is towards, that is in the trajectory that you think it should do. So, for us, that's root cause analysis, meaning we task our agents to say, hey, here's all the telemetry data that we have. Here are all the data connectors that you might want to pull in, you know, in an agentic manner if you want. And I need you to get to the bottom of this and do a deep root cause analysis on this. And when you test this agent, like I said, it formulates a hypothesis.

33:55And then this hypothesis is then passed to other subagents in the system. We have an agent that is expert in log analysis, analysis of log streams and log data in general. So when this hypothesis is passed down to that agent with the task, because our planner says, okay, so I suspect it's going to be the load balancer again, so somebody needs to go study the load balancer logs, and it hands off to that agent. And that agent says, okay, I guess this is what I do now. It goes to the log source and then starts analyzing the data. But we give that agent enough creative space such that if it finds a refuting evidence or supporting or refuting evidence in the data, it can signal that back to the other agents and say, hey, you know, what you told me was to look for this.