From Kubernetes to AI maximalism | Stacklok's Craig McLuckie

25:36And it's really about inverting the question, you know, can we do this with AI to why can't we do this with AI? Show that it cannot be done with AI first. Yeah, you know, I've actually, as I personally have been adopting AI, I've found a lot of situations where, you know, early on, like hallucinations were a big problem. And they still are today, but I feel like that's, you know, as context windows get bigger and as the technology improves, like that becomes less of an issue. But particularly early on when I would encounter an issue where I was trying to solve something with AI and it would fail, taking that sort of approach of like, why can't AI do this?

26:12And then actually just asking my model, like whether it's chat GPT or whatever, like, why are you struggling with this? And like, help me help you unblock this issue. Then suddenly you're getting to more of a productive state where you're actually like starting to unpack the challenge. And you can almost like piece by piece understand a problem of like, why is AI struggling with each individual component of this and then just solve it piece by piece, you know? I think that's a really essential, you know, one of the key learnings I've had has been, you know, I've been in distributed systems for a long time.

26:47My first project I worked on was clustering for Windows NT351. If you want to like really date me, that kind of tells the story of how long I've been working in distributed systems. And one of the observations that, you know, I've made about myself is that as I started to approach this technology, I thought I understood it. I thought I was clever. I thought I was smart, right? It's like, well, I've built a few things and some of them have been okay. You know, it's just another distributed systems. It's just another technology asset. But it's not. It is a stochastic system. It's a probabilistic system.

27:23The way that it works is it's a natural source of entropy in your environment that enables you to unlock new capabilities. Yeah. But it comes with these very specific limitations. and the sort of you know the the thing that really caused me to you know sort of you know start trying to invert the way that i was i was thinking was i just didn't intuitively understand a lot of the the properties of this class of systems when i first encountered them i remember having this vociferous argument with one of my um you know kind of principal researchers about you know the use of of synthetics in um in in training and refinement and i couldn't get my head around you You know, I'm like, I'm a kind of information theory geek.

28:03And from simple information theory perspective, you know, you add entropy, you start to reduce these things down. The results should get worse, not better, but that's not how it works. And so, you know, I think one of the kind of key observations that I've certainly seen for myself, and I've seen a lot of others kind of embrace is that the value creation narrative around the use of generative AI systems is somewhat inverted. You know, people tend to approach it with a platform engineering mindset. and that idea that like, well, I understand the system conceptually. I'm going to need a vector database.

28:35I'm going to need this. I'm going to need that. I'm going to plug them together. I'll get it basically working. And then I'll just, you know, tune and refine the system until it produces the results that I want, which is how we built things like Kubernetes. Yeah. It's just not, it doesn't work. Yeah. You've got to kind of invert it. You've got to figure out what works first. You've got to figure out why it works. And then you've got to figure out how you can, you know, kind of improve it over time. And it does involve this inversion of thought. which is, I think is very important. Yeah. And, and, uh, you know, you mentioned like this natural system of entropy, like, you know, getting back to like my, my comments on like hallucinations, it's like that both is like, kind of like viewed as, I mean, that's primarily viewed as like a failure of, of an AI system when it like hallucinates or it like that probabilistic system, like goes down a path that is unexpected or is viewed as like being wrong or a failure.

29:23But that's actually like the way it's designed fundamentally. Like it's supposed to behave that way because there are many times that it actually does that and it does it what exactly what you want it to do because that's what it was intended it's like you know there's a lot of papers being published these days that like you know i don't know if you saw the paper a couple days ago around you know hallucinations effectively um an artifact of the reward system during training yeah you're rewarded to produce a result you're not rewarded to say i don't know so of course you're going to do that but i mean at the roots of it the core operating sort of principle here is that you've You've extracted the semantic meaning of that information and created a set of embeddings.

30:01And then you've created a set of attention heads that are, you know, receiving a certain amount of context and then working to collapse down a probability field one token at a time to produce a result. It is stochastic. It is probabilistic. You know, things are going to change. And, you know, attention is everything, you know, like the Google paper attention is all you need is great, but it's also finite. You know, and like at some point, you know, attention, self-attention mechanisms do collapse under their own weight. At some point, things will start to hallucinate. And you won't know when until you try.

30:32And you won't know how to kind of maintain a system until you do. And I think that's why you really have to make this jump in with both feet and to start to challenge your own assumptions around the use of these technologies. One of the problems I think a lot of people face is that, you know, they apply this probabilistic system into really what is like more of like a deterministic problem set or deterministic problem area. But also, you know, I think a lot of people look for deterministic ways to sort of constrain and control these systems. So do you feel like a push and pull between the probabilistic nature of this, along with a need for more deterministic control systems?

31:16Does that play into your maximalist system in some way? Yeah, I mean, I think it's important to recognize what it is and what it isn't, right? And it's interesting, you're not going to know what it is until you try it. So I'll give you two kind of examples of how we embraced an AI maximalist view in our organization, one that succeeded and one that failed spectacularly, just to give you a sense of it, right? So the thing that succeeded spectacularly for us was building our own knowledge management server, right? So basically what we did was we built a server that would go and capture documentation from Google Drive, would go capture content from GitHub, Discord, all of these things, you know, generated a system that would go, process that, create a set of embeddings, put a semantic engine in front of it.

32:04You could query it and you could ask it things using any of these existing systems and would come up with very lucid answers. And conventional wisdom would say like, hey, I'm an engineering organization. That's a tool I can go buy. Why would I build that tool? Well, the answer is like, it shocked me. It's what we built took about two weeks and it's shockingly useful. It's just changed the way that we work. And we would never have known that unless we'd actually gone and tried to build it. And it was just a fantastic success story for our own kind of self-actualization on the AI journey. It's not something we got to plan to commercialize, but it's just, wow, it's a really cool system.

32:40And then we tried to do the same thing with Prometheus data. We're running Kubernetes clusters, and we're starting to ask questions like, hey, pod goes into crash loop back off. can we use these models to do some initial RCA so that ISREs, instead of ISREs getting the page, getting some kind of RCA that reduces TTR times recovery for these systems. And the answer was no, actually. We started playing with it. We tried to build these systems. We generated, like we got some data. We generated a whole bunch of synthetic data. We ran it. And what we discovered is that when we started just presenting raw time series data to these things, They would just collapse the context, the self-attention mechanisms.

33:23And we were getting far, far better results by just performing basic Bayesian analysis for looking for correlation between Prometheus streams than we would ever get out of the transformer. And, you know, not every organization needs to know that. But until you actually spend a couple of weeks giving it a go, you just don't know. Yeah, the root cause analysis has been such a wild one with AI. because like, and sometimes, like, you know, I have seen a lot of people give you the advice that like just paste your stack trace analysis into AI and it will tell you the answer. And it's like, sometimes it does just like right off the bat.

33:57And it's like, and if it happens the first time, you might think, wow, this is amazing. And maybe we'll do this all the time. And then the second time you do it, it's like, oh, your Kubernetes cluster has a memory leak. So just allocate more memory and it will be solved, right? Absolutely. No, it won't. And like, no, it's just going to crash in 15 minutes. Yeah, yeah. Yeah. Well, instead of 15 minutes, 10 minutes or something. Yeah, exactly. So like, you know, it's funny how these things work. Yeah, yeah. So, you know, so something that we've seen that you've mentioned in the past is, you know, how some of this is impacting like team roles, team structures, that type of stuff.

34:33So what are some of the most like surprising or like counterintuitive changes that you've seen to like the roles or the structures of teams that are particularly among teams that have successfully integrated AI? Like how is it impacting a team structure? Well, I think there's a couple of things here that are sort of, you know, and I drew these sort of curves, you know, for folks. And that's something I like to draw, which is, you know, the first thing to recognize is even in an AI maximalist world, you know, where we're doing everything we can, we're using every tool, every trick we can think of to kind of drive our own productivity.

35:08You know, when you look at the cost associated with producing great code, we've seen perhaps a 20, 30 % reduction in the cost to produce great code. And this is, yeah, we're writing the unit tests using these technologies. We're doing everything we can to get there. Turns out the cost to produce bad code is infinitely cheaper, right? Because I can produce bad code like overnight. I can produce bad code, you know, using cursor to, you know, solve anything. And so I think one of the key things is, you know, it is definitely changing the narrative, right? The ability to use the distinction between roles and the work associated with a specific role.

35:49There's nothing stopping a designer, instead of producing Figma, producing a scaffolded prototype that you can click through. The class of work product, the way people start to self-identify changes. But then there's also some invariance. Bad code takes work to become good code. You can certainly use these tools to shorten that path, but it's going to take time. And the other thing is to say great code does not equal great product. So, you know, in terms of the concrete things, radical improvement in the ability to iterate and produce disposable code. Yeah. The need to actually dispose of disposable code, because disposable code can be free like beer or free like a puppy.

36:29And if you choose to hold on to it, it's more free like a puppy. It's going to cost you a lot more to maintain and turn into great code over time. um you know there's obviously you know tensions that emerge you know with an organization where there's a sort of temptation to vibe code um you know these the sort of the natural tension where someone like what the hell you just vibe coded a 1500 line pr like you know like why am i reviewing this for you this makes no sense so yeah definitely some kind of cultural you know kind of um reinforcement necessary that you know you produce the code you own the work artifact and so definitely the way that people work and and sort of you know radically improving that that sort of fast turnaround loop and changing the way that we think about disposable code as an asset for that early iteration cycle.

37:12And then the rigors associated with actually disposing of the code that's being produced is one thing. The second thing I'd say is kind of back to that knowledge server example, these systems are fantastic at synthesizing and summarizing data. And like, you may not get actually, you know, sort of perfect results every time they occasionally miss a document, they may, you know, you know, to your point, there's occasional hallucination if if they get into a situation where that sort of self-attention systems become a little bit overwhelmed or there's just not enough parameterized data to render a great result.

37:42But bringing all of that data to teams in a way that they can introduce it into the data work so that they don't have to context shift, they don't have to context switch to Slack and interrupt another engineer to get an answer to a question is fantastic. Being able to flatten the organization significantly because you don't have to have managers as a primary arbiter of the flow of information and information summary, kind of up the management chart, because this information is not democratized and it's immediately accessible. I don't need to go to an engineer manager to know what an engineer is doing.

38:13I can just ask the knowledge server that, you know, like, hey, what did so-and-so do last week? And it'll give me a pretty decent summary. It won't be perfect, but if I really care, I can go look at their, you know, their PRs. I can go and actually ask a manager to get in some more details. But it's just creating this running downhill experience from an engineering management perspective because a lot of things that were hard are just suddenly free. Yeah, and I love that in particular because so much focus when people talk about AI, even still today, which still kind of surprises me, is on generating code.

38:48Because that second half of your answer there is very distinctly not about generating code, right? And that potentially being a big impact that can improve a developer's life without actually touching anything related to writing code, right? And personally, that is a big area that I've seen, like knowledge and skills acquisition, as just a thing that we all have to do as knowledge workers. That has been one of the biggest, I think, often overlooked things that AI does that really helps a lot of people. Yeah. But I want to touch on the concept of disposable code because you're not the first person that has discussed that concept on our show before.

39:28We've actually, Andrew and I, you know, the host here at Dev Interrupted, we have talked about this concept quite a bit in the past. We actually have referred to many of a lot of what we do internally, both at Dev Interrupted and Linear B, as disposable code because we are in an era now where, you know, a marketing team can build their own internal app in an afternoon that doesn't have to serve any purpose outside of their own team or even an individual moment because now you have an AI that can just build it for you. So I'm really curious if you have any sort of like success stories that you can share of where you've seen someone like take a more like disposable approach to code.

40:09And even like specifically, like I really like, you know, how you're saying like, you know, it's one thing to generate disposable code. It's another thing to know when it's time to get rid of that code and to have like a process for like efficiently and like, you know, effectively getting rid of that code when it's time has come to an end. And yeah, I mean, I think for us, you know, probably the biggest success story around disposable code was, you know, having an engineer vibe code system, like having an idea on a Friday, vibe coding over the weekend, showing up with a POC that we could look at and, you know, get a real sort of acute sense of and basically say like, yeah, you know, that's actually, that's a really good idea.

40:53I'll give you an example of this from my previous life, right? Like, you know, when we were getting Kubernetes going, the whole thing that caused Kubernetes to click in my head was this guy, Brennan Burns. He's one of the most creative, amazing engineers I've ever worked with. And, like, you know, I was having this conversation with him around, like, let's, you know, we were playing with ideas in this area. And, like, I was pretty keen on Docker as a container sort of, as a container framework, you know, for like, you know, packaging software. Yeah. And I described this thing to him and he went off and coded it over like, and he's super fast.

41:29You just jam it out there, right? And I came back and I hadn't really connected the dots until I saw what he built. And I was like, holy shit, you just built a personal work sub. I was just like, you know, I was like, this is it. Like, this is the thing, right? And, you know, it would take someone like Brennan Burns to do that. Like he was the kind of guy who's just this creative genius. He would just produce things. Like you could walk around with this guy and like, I swear, if a VC just followed him around for two weeks, there's 10 fundable ideas that'll just drop out of his mouth, right? Like he's just that kind of guy.

42:03But his ability to execute quickly and produce prototypes is a superpower. And not everyone had it. Not everyone has that power. And that's, and like, I've seen this in my own organization time and time again, where, you know, people come up with ideas. You know, like we built a system we call Toolhive, which is something we are all in on as a company. And Toolhive didn't spring into life as a fully formed concept. It was just a great engineer on my team having a weekend fling, vibe coding, something that we looked at, and it was great. And we've also produced probably 20 other things just like it, which didn't stick the landing.

42:41And so I think that sort of early ideation, rapid ideation has been just wonderful as an enabler for the organization. I've also seen having a designer go off and vibe code something makes it much more real. Like the ability to actually touch it and feel it and reason about it is fantastic. you know from the marketing side of the house like a lot of a lot of the content we create you know we'll give the devs a week to go off and just vibe code some crazy ideas together to show how you know you can use the system to create you know outcomes and some of the greatest ideas aren't coming out of you know the like you know actual working prototypes aren't being generated by the developers they're being generated by other members of the organization so there's just so many examples where bringing creative energy into the organization that's provable through an artifact you can see touch work with and then if it's great invest in and turn it into great code or if it's not just throw it away and just don't have to live with it yeah so do you feel do you sense that more of the push is coming from the non-engineering side or is it more coming from the engineering side like like do you feel like the real potential is like non-engineering folks coming in getting their hands on ai and being like look at this prototype that i built i think like maybe it's a marketer maybe it's a sales rep they're like i built this thing i think i can market it or sell it um you know here's the prototype i need an engineer and a product person to see if maybe we can turn this into reality or is it the engineers doing kind of the reverse of that or is it a mixture of both like what do you sense you know like i mean it's this is an age-old story right like um and there's you know shadow it has always been a thing that's existed uh you know for you know since time i used to manage shadow it so right you know like and it's like i remember like and if you've ever run an it organization you know like you'll eventually have this moment where the business comes to you and they went ahead like they were just they weren't happy with you as an it you know kind of provider and so they secured some budget and then they hired some consultants and they went off and built this crazy janky thing in two weeks that they start using right and it gets to a point where you know eventually you get called in and they're like yeah we um we're using this now what are you using it for oh we're using it to you know do our xyz and it's not working anymore and like and i'm like you know and then you're like well why is this my problem and it's like well you're a biot person was like and then you look at it you're like holy crap like what were you thinking like this is never going to scale past a certain point.

45:19You've done this kind of crazy thing. And now you're left living with this thing that someone else owns. So I think there is this very real problem associated with kind of sort of weaponized shadow IT, where you're going to start ending up with a lot of systems that are going to be showing up with production implications that are going to become a natural part of a team's lifecycle because they can simply build them and they don't know any better. So there's definitely risks to an IT organization from that perspective. And it does beg the question, you know, like if the team that you're serving can produce this system in a certain amount of time, if you are not able to kind of embrace a certain amount of agility where you're serving the needs, they will self-service.

46:02They will ultimately absolutely do that. We may get to a point where these technologies get good enough that code is more like an intermediate language than actual code. We may well get to the point where they're actually producing sufficiently robust capabilities that things will work well. But the reality is they're not there yet. We just aren't there yet. The reality is that we don't have reasoning systems. We have systems that are emulating reasoning through planning loops and iterations and other pieces. Context is everything, but it's finite. Eventually, the self-attentional mechanisms collapse.

46:38Things become too complex. And then you left hand, you know, kind of holding the bag for the team. Yeah. Yeah. You know, there's a lot of hype around like artificial general intelligence. I think really what we have now and maybe even for the foreseeable future is simulated intelligence. It's like something that looks exactly like intelligence most of the time. Yeah. And then it doesn't suddenly. I think, you know, the way I'll probably get yelled at by AI purists for describing it this way, but I keep coming back to this book by Kanahan, Thinking Fast and Slow. Okay. Right. And so Kanahan describes these two systems in the brain.

47:11The first system is, you know, the sort of system one, which is like the fast thinking system. Yeah. And then system two is the sort of reasoning system. And you kind of need both, right? Like you need to be able to look at, you know, process visual information. That's a chair. Yeah. You know, to turn visual information into something that has semantic meaning. And the closest analog that we have, you know, in terms of generative AI is it's effectively kind of, you know, like that sort of fast thinking system. It's not a reasoning system. It's emulating reasoning by making snap judgments to generate planning and then running kind of internal iterative cycles.

47:44And the essential thing that's missing is the bridge between system one and system two. You know, like, you know, when we see a chair and if our brain says it's a chair, but it's not a chair, like something will like think us like, is that really a chair? Is that really a something? And we actually have the ability to catch ourselves. And then we have the linkage where the reasoning starts to program the fast-thinking systems. Those don't exist. So I think you're right. It's emulated reasoning. It's not actual reasoning. And because it's emulated reasoning, it'll work within the boundaries of a certain set of operating attributes.

48:22But at some point, things do tend to fall over. A listener out there, their company is just starting their AI journey, or maybe they've made a little bit of progress along it, but they're still struggling along the way, or they're still learning a lot. What's some practical advice that you would give to somebody who's still early on in that experimentation stage, and they're maybe not quite at full-scale adoption, but trying to get to that phase? What is the advice that you would give to someone who's in that stage right now? I would say that, you know, first of all, you have to recognize that a lot of your intuitions and instincts are going to be wrong.

48:59And so there's this learning arc that you have to embark on, which you have to embrace a certain amount of patience with. And I think for me, the learning arc that worked for myself and my organization was, you know, first use the tools just to understand, you know, what the art of the possible is. Then actually attempt to build the tools and optimize the tools. And once you've got to a point where you're successfully feeding yourself with tooling, you then may be positioned to start building general products. So go buy a cursor license, start using it. Use it a lot. Figure out where it falls over.

49:34Figure out how to make it better. Figure out how to connect cursor to a variety of existing systems using MCP servers. Great. Get to a point where you can say, well, okay, what would a knowledge server look like? Don't buy it. Just build it. Just throw yourself down the cliff. Try it. See what happens. At the end of the day, it's a lot easier to... take a great engineer and, you know, get them to the point where they have that intuition, that instinct around what works and doesn't, than it is to take someone who has, you know, has learned the intuition and instinct around what, you know, like how these things work and turn them into a great engineer.

50:07And you just kind of have to be patient with yourself and your teams and look at it as a path towards understanding through use, then optimization, and then delivering actual, you know, sort of systems. Like there's a lot of work associated with building long-running stateful agentic systems. It eventually will deconstruct into a distributed transaction problem. I promise, workflow always does. So unless you're willing to kind of work your way into it just over time, you're going to get hurt. Yeah, you've used the phrase vibe coding a lot, which gets a lot of negative attention. And we've addressed that on this show a lot.

50:44So our listeners are very familiar with our take on it. And you've used a little bit harsher language than I typically use. I think you said throw yourself off a cliff with it, which is, I like to say vibe code until the wheels fall off. Like you should do that at least once, you know, just so you see what happens. It's faith-based engineering. Throw yourself off a cliff with a bag of parts and make sure you construct a plane before you hit the ground. Yeah, just so you know, like this is what happens when it all falls apart on you so that you're prepared so that you know when you're serious about it, like you can protect yourself and be ready for it, you know.

51:18It's a sharp new tool. You got to treat it with a bit of respect. Yeah, yeah, exactly. Like it's, you know, like sidle up to it, use it a bit, learn it, understand it. You know, just chartering a team to go off and like, you know, hey, build this long-running, stateful, agentic subsystem with reflection patterns and memory and, you know, kind of tool calling and contextual treatment and all that. I promise it's not going to work. You know, you'll get there eventually. But I promise, like, unless your engineers have actually walked the journey, you're going to be disappointed. There's a lot of frustration.

51:51A lot of frustration. And frustration leads to negativity. And you just don't know, like, why is this breaking? I don't understand why it's breaking. Well, you just add a new tool. Well, what happened? Well, you just bumped the thing over a threshold where, you know, like, it's not rendering too much context. And it just is not performing the same way. Or, you know, like, hey, you just updated a model. You've got to realize the prompt and the model and the orchestration pattern are all tightly coupled. These things, you know, like, they cannot be easily decoupled. Right. You know, behavior is optimized for probabilistic outcome.

52:18Yeah. And until you get your head around that, it's going to be a bit of a bumpy path. Yeah. So, you know, we've talked about how, like, you know, there's efficiency gains around generating quality code. There's efficiency gains around generating bad code. But beyond efficiency, like, what do you think are, you know, the biggest benefits that, you know, or even challenges that engineering leaders are going to face as they start to integrate generative AI and agents into their core processes? you know particularly if you know of any like unexpected ones that people maybe don't typically see well i mean it's there's the bill at the end of the day well everything's everything's subsidized by venture capital right now so the bill is a lot cheaper than you know like a lot of people are like you know i speak to a lot of very smart people and they're like i don't worry about it like you know like you know like the you know like using sparser matrices and this and that and all these other things are gonna you know inferencing costs coming down it'll be fine um You know, like, no, the bill is going to continue to increase.

53:18I think the biggest thing I see as a problem for organizations is this, it's effectively change management in the context of something, right? So it's not like you start to get to a system where you've, with care and love and iteration and running, you know, kind of running, you know, gradient descent for your prompt optimization, all of these things, you produce this relatively optimal system. The problem is that, you know, the minute something changes, and it could be like, hey, I'm using a tool call system with MCP, and, you know, I'm starting to kind of, you know, project tools that are now available, the tool description changed.

53:58All bets are off. There's no guarantee that it's going to continue to invoke that tool in the way you want. You know, I laid it on five more tools in my registry, and it was doing dynamic tool binding. and suddenly, you know, I've laid in 100 tools and suddenly my context, when I start to just present a tool list, has been blown up from 10 ,000 tokens to 150 ,000 tokens and all bets are off again, right? So I think the, you know, the fact that, you know, you really need to kind of, it's not enough to just get it working. You also need to understand why it's working and what's going to be introducing entropy and sort of inconsistencies in the system over time is really big.

54:34And I think that's been probably one of the more surprising things I've seen is just how things are fine until they're not. And that's something I'm surprised to be. Awesome. Well, Craig, I appreciate you very much coming on our show and sharing your insights with our audience. If our audience wants to follow you after the show, where's the best place for them to tune in and keep up with you? Hit me up on LinkedIn if you want to have a conversation, Craig McClucky. or just look at the toolhive.dev website if you want to learn more about some of the technology that we're working on. Awesome. And to our listeners, thanks for tuning in today.

55:10If you're not subscribed to our sub stack, make sure you sign up for that. And thanks for joining us this week.

55:27you