Engineering AI Systems for Autonomy and Resilience with Krishna Sai

0:00Enterprise IT systems have grown into sprawling, highly distributed environments spanning cloud infrastructure, applications, data platforms, and increasingly AI-driven workloads. Observability tools have made it easier to collect metrics, logs, and traces, but understanding why systems fail and responding quickly remains a persistent challenge. As complexity continues to rise, the industry is looking beyond dashboards and alerts towards agentic AI systems that can reason about operational data, reduce toil, and take action when things go wrong. SolarWinds offers solutions to monitor, understand, and remediate issues across complex distributed systems.

0:43The company began as a leader in network and infrastructure monitoring, and has evolved to support modern applications, cloud environments, containers, and AI workloads with a growing focus on reducing operational toil. Krishna Sai is the Chief Technology Officer at SolarWinds. He joins the show with Sean Falconer to discuss how SolarWinds is rethinking observability in the age of AI, what it means to design agentic systems for mission-critical environments, how AI-assisted programming is reshaping engineering workflows, and why the future of operations depends on building platforms where humans and autonomous agents work together.

1:25This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.

1:43Krishna Sai:hi welcome to the show thanks sean it's great to see you and meet you big fan of the show so thanks for having me here oh well thank you so much that's nice to hear yeah i'm looking forward to this as well so i wanted to start off talking a little bit about you know solar winds and kind of set the stage there? Because I think a lot of people know SolarWinds maybe as a single tool that they used years ago, but you guys do a lot of different things. So given where you are today, how would you describe what SolarWinds actually is today to someone who hasn't looked at the space in a while? No, absolutely.

2:17Krishna Sai:If you take a step back and say how IT and ops teams who typically use SolarWinds products have been using SolarWinds for the past 25 years or so, a product portfolio broadly expands three domains, observability, incidence response, and service management. And to put it simply, IT and ops teams use us to help detect and remediate issues across a variety of workloads in their environments. Network and infrastructure, which is where we started and have been a leader for a very long time, but also applications, databases, containers, ML workloads, et cetera. And so our solutions cover this from a horizontal perspective, meaning give you the ability to look at the general basic health of the typical workloads, compute, storage, network, et cetera, but also vertical cross -cutting concerns like performance, reliability, cost, security, and so on.

3:15Krishna Sai:And what happens is typically IT and ops teams are accountable for SLAs and SLOs, and that kind of drives your day-to-day behavior. More mature teams, of course, manage error budgets at scale, and they have nuances of that same dimension. But all of this is much simpler said than done. I was talking to a CIO who was part of a customer call recently. He's here of a major system integrator responsible for running big managed global services for an organization. And he said it well. He said, I'm responsible for SLAs, but honestly, I can't tell you everything that contributes to an SLA. It is a statement of complexity in these environments, but it's also increasingly these teams have to deal with large microservices, distributed systems, etc.

4:08Krishna Sai:And so complexity is very real. And so what we target is, especially in the context of AI and so on, our goal is to reduce toil. We've all been there waking up at 3 a.m., alert storms, and getting into a war room. And the problem with that is that even today, a lot of the tools just ingest a whole lot of data and show you a lot of dashboards with red lights and so on. But they still finding out why something is red is still a big challenge. So we've been thinking about this challenge. So when we think about AI assisting with this, traditionally, you know, we've gone from statistical approaches, things like anomaly detection, machine learning, basic stuff, to now there's a very clear shift to agentic AI, not just in our industry, but just across the board.

5:00Krishna Sai:And so that's something that we want to focus on and increasingly index on. And the way we talk about that is we just call it Solomon's AI more broadly, but in particular, the agentic portion of it we call Solomon's AI agent, it often gets confused with AI observability, which is something that comes up a lot. And the way we think about AI observability is that as a more of a vertical use case, right, rather than a lot of a horizontal thing. But that's also something that we're starting to do here. Yeah. So I think there's quite a bit to sort of unpack there. I definitely want to talk a lot about how the use of essentially AI agents are starting to sort of impact the types of use cases that your customers are generally interested in.

5:46But one place I wanted to start off with, because I think a lot of people listening to the show, and it's certainly a lot of businesses I talk to are all interested in understanding what are some of the leading technology companies doing when it comes to leveraging AI. So I wanted to talk briefly about AI-assisted programming. So to start off there, how heavily has SolarWinds invested in various AI-powered programming? Is that something where every engineer has now got a junior engineer in their pocket where they're leveraging something like cloud code? Where are you with that?

6:20Krishna Sai:No, we are. We're investing very heavily. All our engineers use AI-assisted coding. Actually, it's super interesting because yesterday we were kind of reviewing the progress that we've been making through the year. And all engineers have enabled using AI-assisted, both in terms of co-pilots as well as agents and so on. And what we're broadly seeing is that in general, we're seeing, of course, increasingly, you know, increasing code being generated with AI, for sure. The percentages vary from organization to organization and how you measure it and so on. but we're seeing increased commit velocity.

6:58Krishna Sai:Our commit request velocity has gone up like 25, 30 % north of it sometimes. But part of it also, what we're seeing is through deployment, frequency has gone up as an example. So we're seeing some improvements in lead times. But what we're also seeing is that tools are maturing, which means that both the acceptance rate as an example of generated code is significantly improved over the last year as both the models have improved and the agents have improved. But the shift is, of course, now more on the code review side, right? Like a lot of a code review portion is still, the bottleneck has moved there.

7:36Krishna Sai:So that's something that we're starting to address and look at how do we make it better and so on. But broadly, I would say the signals are positive. Of course, the concerns are the usual concerns around code quality and flaky test generation and security guidelines. et cetera, like those things still are very much front and center. But, you know, one, maybe this is kind of where you are going, is that when we think about agent engagement, coding is actually a very good baseline because we've all been working as an industry for a long time. And the nuances of, okay, how does the application or the use of coding agents now extend to broader enterprise, like software use cases, is something that's front and center.

8:22Krishna Sai:And that's where we've also been playing both sides of it, which is using coding agents and building our muscle around using how agents work and so on, while at the same time offering them through our products to our customers. So there's that context as well. So one of the things that when I think about this notion of how do these agents apply in the enterprise software use cases, if you take a coding example, for example, If you ask a coding agent, go add an OAuth support to my function or whatever, or refactor this module to be async as an example. These types of things are very, very, instead of taking an example of taking a single code snippet, breaking the task down, the coding agent goes and reads the repo, inspects dependencies.

9:15Krishna Sai:There's a lot that's going on underneath, editing multiple lines, running tests, so on and so forth. So this notion of setting the intent and then the system deciding what actions it needs to take to drive towards that goal turns out to be a very good mental model to baseline on in terms of how we think about agents in the context of enterprise software. Yeah, I think that's a very astute point. I think that one of the reasons, I've been thinking about this a lot recently, I actually wrote an article about it. I think one of the reasons why programming has been such a tip of the spear is, like you said, there's all this sort of history associated with it.

9:55But it's also like a hard truth environment, I would say, where even though the output might be non-deterministic, there's kind of deterministic ways of checking the correctness. And then that becomes almost like a reinforcement learning cycle because I can compile the code. I can run it against unit tests. And most environments don't have that. So I think for other sort of non-coding environments to be successful at the level of coding, you need to be able to create those similar sort of deterministic guardrails where you can actually evaluate the outputs. in some reasonable way so you have confidence that's actually generating something correct.

10:27Krishna Sai:No, absolutely. And I think that's why that analogy kind of makes a lot of sense to me when we try to internalize it as like, what is the intent, right? And what is the intent and how do you measure it? Which is why, as an example, if you, a lot of systems like, which is in operational practices, things like SLOs and SLAs come in super handy because at the end of the day, what you're really trying to drive is towards a certain healthy operational state, which is actually well-defined in the non-agentic world, a very human-driven world as well, because a lot of your practices, et cetera, incident response is an example, traditional setup, a threshold fires, page goes out, engineer wakes up, engineer does a series of things, checks dashboards, pulls logs, inspect traces.

11:16Krishna Sai:It's like a very, very disciplined type of an approach that the industry itself has matured along those lines. But the way for an agentic system to kind of then, let's say, mimic it, so to speak, and to make it a lot more effective, efficient, autonomous later on when we talk about, one approach is to say, how do you not remove that logic or the set of practices, but how does an agentic system say, absorb it, so to speak, right? And that's where I think a lot of the implementation design challenges, et cetera, come in. So a system, the way we think about a typical, like an agent, as an example, right?

11:54Krishna Sai:If you have an SLO, a system could be observing raising error rates, as an example. Notice that, okay, this is isolated to a specific service. And then it correlates that to a deployment that happened 10 minutes earlier. It observes a trace pattern that happened during a previous incident, concludes that this is a bad config change or whatever. All of these sets of steps is very, very, I would say, there's a lot of historical knowledge and actions that an agent can learn from. And I think that's where I think the analogy with a lot of what happens or how do coding agents work in a coding use case versus an operational agent working in an operational use case, there's a lot of similarities there.

12:40Yeah, absolutely. And you have a tremendous amount of experience in, I guess, like traditional sort of enterprise infrastructure, having worked at a number of large, successful organizations. How much of you see building sort of agentic systems as something that's brand new versus, I don't know, like a rebranding of some of the typical things that we would do with any software application?

13:07Krishna Sai:Yeah, no, that's a great question, actually. And if you think about this evolution of operational systems, if you think about it, like traditionally they were monitoring systems and then monitoring just things that were polling and observing the state, so to speak, manual thresholds and UAC, like in the first generation, so to speak. Then they evolved into things like observability, where the system expressed its state through multiple ways, and then there was a way to correlate them across these multiple signals. Then there was this concept of AIOps, essentially, which is essentially using these signals and coming up with ways to correlate those signals and making decisions.

13:51Krishna Sai:still very, I would say, very early on, but that's where it started. Then we had started to see cases of like co-pilots emerging, where there's assistive tooling that is sitting next to all the human decision-making that's happening. And now we're starting to see the early green shoots of agentic AI, where there are agents that can actually act, even autonomously at times, within boundaries and so on. So there's an evolution of how this industry has gone through all of that. And some of that is, I would say, the natural push and pull of technological evolution. But also, a lot of what has made that almost an existential need is just the sheer exploding complexity and tools brawl and data.

14:40Krishna Sai:And it just, at some point, we all realize that it's just a human is not going to scale in terms of maintaining the health of these complex environments, right? Like that's the evolution that I've been seeing. Yeah. I mean, we saw a similar evolution in the world of biology too. There's just the sheer amount of data that exists in biology. Like people started sequencing the DNA and stuff like that 30, 40 years ago at this point. So technology has served a massive role there. And a lot of people say that the 21st century is going to be the age of sort of biology because of the fact that now we have powerful enough computers.

15:19We have these really powerful models to kind of assist in the data crunching involved in essentially evolving that science. Because no one human, no matter how gifted you are, could possibly keep all those things in your head. And I think we're seeing a similar evolution in technology because there's just any complex enterprise environment. There's literally thousands of different data systems that might be sending important signals all over the organization. It's like, how do you sort of start to be able to parse that? And most businesses are sitting on these terabytes or heaps of unused data where they hold on to it because, you know, they might be useful some day, but they don't have sort of a way to unlock that use.

15:56Krishna Sai:That's right. That's right. And a lot of that, if you think through, for example, if you extend that and then you ask yourself, why? If you think through that, in an operational context, this was an aha moment that we had probably a couple of years ago. We've always kind of loosely had this strategy, but it came more front and center a couple of years ago. If you think through how the operational systems have evolved in digital environments, there was a monitoring observability industry, which was just only focused on getting signals, showing dashboards, and giving alerts, as an example. And then you had this incident response or more DevOps-y types of environments that came where you really saw all those signals, but then it adapted to how teams were operating with incident response, being able to maintain SLOs of services, manage error budgets, and so on.

16:49Krishna Sai:And then you had the IT systems off on the side where there were very, very ITIL-driven, service delivery-driven, very, very, shall we say, structured processes that enabled enterprises to scale at these practices. But what all of that did was create all of these silos in terms of IT operations management, IT service management, et cetera. And the silos happened in organizations, but they also happened with data. And to your point, when you have all of these massive amounts of data ingestion, which data ingestion is something that we've mastered very well now, but they ended up creating all these massive data silos.

17:29Krishna Sai:So what happens is when you want a Gentex system or any kind of an AI system to work, when you have data silos that becomes incredibly complex and expensive, results in you have this third wheel of separate AI systems that then ingest all the data and then having to process data on top of that. And then you have multiple dashboards and consoles. And so when you're dealing with operational situations and war rooms, There are just so many different consoles that are spread around everyone's desktop. All of these challenges, which really almost became important, critical, that we really take another look at how these things work internally in SolarWinds.

18:15Krishna Sai:As an example, in our design discussions, we use the left brain, right brain analogy, right? Where if you take the human brain, it's actually one of the most, we're talking about biological systems, right? It's the most wonderful biological system for observability ever created. We walk around the face of the earth ingesting all kinds of signals through our five senses. And you can be in a crowded mall with a lot of noise and someone says, Sean, and you're instantly paying attention to that, right? With your name. And the subconscious seems to have this phenomenal mechanism of being able to process all those signals, decide what's important, what's not, and being able to surface that, shall we say, actionable signals to the conscious where you as a Sean can say, hey, is that really meant for me?

19:05Krishna Sai:Oh, maybe that's a different Sean. I can go about walking around, getting cookies in my mall, so on and so forth, right? So you can do all those things. So when we think about, you know, extending that analogy to an observability use case, you have this system of observability on the one side, which is optimized for your mean time to detect, so to speak. And then you have those right brain systems, which were conscious systems in the past, where there were actions and runbook automations and workflows, etc., all optimized around remediation. And these two come together as a unified system. And then you have kind of the analogy of the subconscious and the conscious, where in the subconscious is all this processing that's happening and the conscious is where the human element comes in and increasingly the conscious actions which are the actionable things are also becoming dimensions or various dimensions of autonomy right so this analogy of the human brain is something that we talk about a lot internally when we think about these systems how does that impact the way that you think about using agents for observability you know i think traditionally, as we've talked a little bit about, like observability is a lot of dashboards, it's metrics, it's maybe some level of statistical-based ML to highlight certain issues like anomalies and so forth.

20:28But it's really about giving visibility in the systems for humans. But now with AI agents potentially playing a role there, it's really about giving inputs to machines, not necessarily dashboards for people. So how does that kind of think about how you would build these systems and how is SolarWinds approaching this problem?

20:45Krishna Sai:No, I agree. I mean, there's a lot. Maybe it may help. It may be useful to maybe baseline on a couple of different examples of how we've seen this evolution take place. And then maybe I'll go into more of the design choices that we've had to make. So if you think about, for example, in this evolution of how Gen AI essentially, right, started to help out with the problem solving, diagnostics, resolution, so on and so forth. We saw this phase where there was this co-pilot phase. where there was this Gen AI essentially becoming a very capable interface layer sitting next to the system, not necessarily inside it, but next to the system.

21:26Krishna Sai:So when you're debugging a production issue, the co-pilot then helps you summarize things and explain an error pattern and help you write a query across multiple metrics and traces and so on. It was still very useful, but fundamentally still very reactive, right? So, for example, we have a couple of examples. Just to illustrate this, we have a configuration agent as an example. And the configuration agent is interesting because it is one of those configurations, one of those highest leverage, highest risk surfaces in modern systems. If you think about the number of outages that have been caused by poor configurations, it's pretty massive.

22:06Krishna Sai:And the other thing that's super interesting about that is, if you think about DNS misconfig, which we hear about every other week these days, are certificate expirations and overly permissive security rules and so on. What happens is a lot of them don't necessarily result in a crash, so you don't get an exception that you can go look at, but there's a subtle degradation of surface behavior. You don't realize that the outage actually happened because nothing specifically crashed. The system is just executing to your configuration. What typically happens is in a co-pilot world, that configuration failure would be detected, discovered after the fact.

22:45Krishna Sai:An engineer would get paged and a co-pilot will summarize all of that. But what we're increasingly doing is changing that behavior to where a config agent is continually looking at how the service itself is degraded. And then when there's a service degradation, having all the information that is required to be able to go and correlate that to a config change and then be able to make a very effective choice about whether I want to do a rollback or something else, right? Like I'll bring a human in the loop, so on and so forth. So we're starting to, I would say, see these types of use cases increasingly.

23:26Krishna Sai:Now, what happens is when we think about engineering for a lot of this, there's a lot of, I would say, very important considerations that come, right? And the hardest problems actually tend to be architectural in nature. And that happens, especially when you're dealing with production systems, mission critical systems. How do you think about building AI software that can act in real-world environments and not just a co-pilot? How can you kind of start to build that autonomy? And that becomes a very, very important kind of design choice. So we have this internally, we think of this as AI by design, which is not AI first or AI everywhere, which tends to get kind of misunderstood a lot.

24:11Krishna Sai:But how do you design a platform from the beginning with the assumption that AI-driven components would exist, would evolve, and eventually operate autonomously. I think the mistake that a lot of teams end up making is to treat agents like a feature that you bolt on after the fact, rather than... Then what you end up is you have these powerful models, which people have built billions of dollars building, sitting behind super brittle guardrails. And then you have this problem where, hey, I have this best model. Why is it not giving me the results that I want? Then you realize that you don't have the basic system that is really designed for these things to act.

24:55Krishna Sai:So we have this statement that we use, the model can propose, but the platform must dispose, meaning treat the model for what it does. It's a great reasoning component, but make sure that the platform's a safety boundary. And so when you start to build out these types of systems, then you have to have specific architectural platform components in place, right? And these become very concrete design choices. So, for example, LLM gateways is a great example, right? Like early on, when we started experimenting, you know, teams were experimenting with like wiring logs directly to an LLM as an example, right?

25:32Krishna Sai:It works great in a demo. Executives are super impressed. But then you start to say it immediately runs into problems the first time you try to put anything close to it in production. Because cost spike unpredictably, sensitive data gets into problems. Different teams hard code different models. And then suddenly you can't change providers, enforce policy and so on. So one of the initial design choices that we had to make was actually bring all of that together in a platform service, which is an LLM gateway, which then handles everything from model selection, abstraction, PII masking, rate limiting, auditability, and so on and so forth.

26:12Krishna Sai:So a lot of those shared concerns across expanding your EI cases are kind of isolated. So that's a very good kind of choice. The other one for us is also around, you just can't throw melt data at LLMs, right? Like the logs thing that I mentioned about, it's one thing about generating a whole lot of logs. Logs tend to be super noisy. And so we have to think about how do you feed logs to an LLM or a model for decision making? You know, you can't have, so you need to really think about logs. Are you going to compress them, deduplicate them, summarize them before you ever expose them to a reasoning layer in your platform?

26:54Krishna Sai:So instead of asking a model to read like a half million lines of log lines, you have a compact representation of what changed, what's anomalous, you know, what's new. And then that's a kind of a classic systems approach that we've applied in other use cases, but then you need to bring that into something like an AI system. The same thing kind of also is, yeah. Yeah, so I wanted to go back to a couple of things that you said there at the beginning. So you talked about these like co-pilot experiences where even a really great co-pilot, you're still finding out after the fact that there's some sort of problem and it has to essentially wait for a user to prompt it.

27:35And it sounds like you're trying to move to a world where you have kind of these more ambient agents that are always on. And I really believe I'm a big proponent of this. Like, I think that's going to be the next evolution of the use of agents. You know, there's been a lot of success in the B2C world with AI, especially with ChatGPT. And I think that has been fantastic, but it's also locked a lot of companies into thinking that the only way these interfaces work is it's a chatbot, you know, it's some sort of co-pilot. Maybe there's an agent behind it, but still just a chatbot waiting for a user to ask a question.