AI isn't for cutting costs, it's for multiplying impact | Super.com's Matt Culver

0:05Okay, from the top. Welcome to Dev Interrupted. I'm your host, Andrew Ziegler. And I'm your host, Ben Lloyd Pearson. This week, uh-oh. Whoa. We're professional. Yeah. All right. This week, I'm sitting down with Super.com's Matt Culver, who joins the pod to discuss why the common view of AI as a tool to reduce headcount is a misguided accounting mindset that ultimately destroys trust. Instead, leaders should see AI as an opportunity to reinvest in their teams and improve the entire product development lifecycle to generate more value and empower developers. But first, we have some news. So what's up this week, Andrew?

0:45Yes, in the news this week, we have an interesting roundup of both AI skepticism and excitement. So one of the things that came across our desk was Anthropik's report about model poisoning, also paired really well with Charlie Waugh's timely article on intelligence drift with these models and how folks are using them. Is that real? Is it fake? Is the phenomenon on either of those, doesn't matter. And speaking of real or fake, we're talking about AI usage and water consumption. We found a really interesting article that breaks down the context of this number and an emerging trolling as a marketing playbook in Silicon Valley.

1:21We're gonna look at a little closer at what this means. And for the web dev nerds out there and the lovers of things like semantic HTML, myself included, we're taking a deep dive on a fun little HTML tag that's maybe been left in the dust for a long time in the HTML spec, and finally will have its moment in the spotlight. So a fun little roundup this week. What do you want to dive into first, Ben? Yeah, let's just start right at the top with Anthropic and how a small number of samples can poison LLMs of any size. Yeah, so Anthropic is leading the charge on all things AI model research and security, and they release these really fantastic reports in combinations with AI research organizations.

1:59In this case, the UK AI Security Institute and the Alan Turing Institute. And they found in this joint study that in as few as 250 malicious documents can poison a large language model regardless of its size or its even training volume, training data volume. So what this means is that like a huge model, like something with like a 600M model can be backdoored with these articles with like only 250 of them. What this means is that there's only a small amount of documents in any model that an attacker has to control in order to poison or alter the model's behavior. So in their example, they use some really interesting approaches that generate some documents with gibberish and were able to poison all sorts of models.

2:48What did you think of this one, Ben? Yeah, I think that example that they used of testing it where they convinced a model to output gibberish whenever it encountered a certain term was a really great illustration of how this works. And, you know, past research in this area has shown, or in similar areas, has shown us how, you know, you can do things like, for example, using a lower grade model to judge the output of a higher grade model to spot like deception, for example. So I'm almost wondering if there's like some way that you could accomplish something similar here, like using a lower grade model to determine if the higher grade model might have been deceived by something that has been injected into it.

3:28But it is a little shocking just how easy it is to inject bad information into these models. And maybe this will, one of our later stories will, you know, maybe have some insights that tied to this. But I think what really what we're seeing here is that, you know, just the fundamental nature of software security is changing. So now information itself is an attack vector. So if you're using models that ingest data from somewhere for their training, that is now an attack vector into your organization. And I think this is where stuff like AI observability is really going to become more and more important just to make sure that models are successful at the things that you expect them to be successful.

4:09But, you know, I'm just going to continue sitting in this chair here preaching the good word of data provenance. You need good data input into your systems. You need to trust that data. And it needs to be enough to give adequate context to your models. And, you know, I honestly think that is probably the biggest challenge that practically anyone that has an AI initiative is facing right now. you know, just getting good data into your models, whether it's your own data or it's a model that's been trained on data that you don't control. Yeah. The data element of, of context engineering and bringing that in, that's critical defining element of the engineering challenge that folks build when creating these at scale.

4:52And we learned about that all last week too, when we were on site at Dreamforce, you know, we talked with all sorts of leaders and learned from them about how they're tackling this type of context issue around owning their data and working with it where it lives, right? But I loved what you said about information itself as an attack vector. I think that's a really powerful takeaway from this article for folks to realize in terms of operating with them. And related to it, you know, we read this article, Revisiting Intelligence Drift by Charlie Waugh. And it's an article about, you know, folks complaining about LLMs getting dumber or changing their behaviors at different times of the year.

5:26And this article explores the different categories of these theories and arguments that folks have made over the years since LLMs have hit the scene. Some of these include like a cost-cutting theory, like, oh, they are downgrading or changing the hardware that the models operate on so they perform less in certain times of the year. Some of them are more based on like training data, like thinking that the model's on winter break because it's ingested so much trainings data about people taking it easy during the, you know, Or, you know, they're not actually post-trained correctly or they're using old training data.

6:04And all of these different theories, the core thread amongst them is that it's a different kind of perception based upon people's expectations from the model. But it also is related to new models coming out and ultimately kind of like fighting hype versus reality of operating with these new types of tools. I think we encountered this recently when GPT-5 hit the scene. And a lot of folks had mixed results and mixed opinions about the model to the point where OpenAI even made 4.0 available again due to the backlash of folks missing the older model. Ben, what did you think of this one? Yeah, so I've got I've got a little bit of a hot take on this.

6:40I actually have not noticed a significant decline in model performance. Like, in fact, I think over the last six months, my workflows have improved dramatically. You know, I mean, I do notice slight changes in behavior like that happen relatively frequently. But like from my experience, those usually get ironed out either through like better system prompts or through better context or something like that. It's usually a workable problem. And I think we've also gotten to a point where it's like, like GPTs are so good at writing system prompts for other GPTs now, like they've even gotten much better in recent months that you can just dump like an incredible amount of context, like into the system prompt itself now, which is pretty amazing.

7:25but yeah i mean gpt5 i did i did kind of at first feel some of that you know in in the sense that like i didn't feel like it was doing quite what i wanted it to do at first i don't know if that was the model getting worse or if i just my expectations about how chat gpt works didn't align anymore with the latest model you know like maybe the new model just behaves differently and i have to adjust yeah um so yeah and you know like one great example we've been joking about this in our chat andrew but how like whenever i ask for technical architecture things it speaks to me in the most like elite hacker way possible it's like it's like yeah yeah it's like almost like sophisticated elite speak and i just half the time i don't even know what to do with it yeah gpt5 like when it goes in the engineer mode it it uses the bullet points it talks to you like a 10x engineer who has no time for your conversation it's actually quite funny and it makes me like almost it's like it wants me to feel dumb too but oh the thing is you know what i what i learned is that it's actually really good at building architecture designs for other gpts so even though i may struggle to to understand what it's what it's like asking me or the things it's telling me to do if i work through all the things that i know with it then it gets me close enough to like a prompt that I can feed into like cursor, for example, to start building stuff for me.

8:50So, you know, it's, it's not necessarily worse. It's just, I've changed my approach to, to using it, but there is one sort of like elephant in the room that I think we, we also just can't leave without discussing is, you know, a lot of these services are trained on, you know, really wide range of data, whether it's like books that they've harvested or internet articles that they've consumed but also like social media i mean i it's not clear how much of like reddit for example is being ingested into these uh like the flagship gpts but you know if it's ingesting conversations from reddit that were generated by other ai bots just trying to like promote products or you know you know whatever they might be out there doing like that is that is a potential to have, similar to our last story, data that is poisoning the flagship models themselves, and that may affect their ability, their performance for certain types of actions.

9:46Yeah, really well said. Although I can't say that I, on the same line as you about, you know, the models changing when new releases come out, because GPT-5, when it came out on the scene, I felt like so many of my old prompts just really stopped hitting the mark. I really did have to change them, adapt them, like you said, in order to kind of find that new rhythm. But in many cases, I still like some of the older models, you know, different kind of models. They're trained on different stuff, like what we talked about in the last article, too. And so in their own ways, it's like even if they're trained by the same company and trained in the same way, ultimately, they turn out very different.

10:25And so people just need to be aware of these differences and understand the different uses for them. Like I learned a lot about this from watching Claire Vo. She talks a lot about making specific model choices for the specific task at hand. And that really resonated with me. It made me be really self-aware of like using auto mode and cursor. Like don't do that. Like intentionally choose the model for the task at hand. So really well said, especially about the consumption of sources from social media. Like think of Grok. Grok is a foundational model trained on things on X that largely are also generated and posted by other bots on X.

11:04And so it's interesting to know how that would affect something like model collapsed. But jumping into our next story, it's a little bit more on this AI up and down hype train. And this one's about AI profiteering and marketing now looking as indistinguishable from trolling. And what I mean by this is like a trolling as marketing playbook that's hit Silicon Valley in recent years. We're talking about billboards that say things like stop hiring humans, the Friends AI companion pendants that have been scattered around all of New York City subways, San Francisco subways, and also heavily graffitied.

11:36And there was this really great article that covered kind of like the different elements of this marketing strategy that really pokes at the vulnerability felt in the market right now by folks. And so it was a really interesting deep dive from Bryant Merchant on Blood and the Machine. Ben, what did you think of this article? Yeah, I mean, first of all, I think this whole like higher AI, not humans, sort of like sloganeering type thing, it's just pretty gross, to be frank, because there's two issues with that. First is that it kind of tells me that the company itself doesn't really understand how AI or LLMs and GPTs actually work.

12:15Like, I think they're sort of caught up in that, like, the bubble hype of like AGI and stuff like that, rather than like the actual real world practical application of GPT technology. And, you know, to be frank, like the more I work with AI, the more I realize that there are certain reasoning capabilities that they are just simply lacking. Like they have, they're like a facsimile of intelligence often. And a facsimile, it may be an exact copy, but it's still a copy. In many situations, you can tell a paper copy of an image is not necessarily going to look at all like the original image, even if it is an exact copy.

12:55And that's kind of how I've always approached AI. It's like there are many ways that it can emulate human intelligence, but it isn't actually demonstrating all of the advanced reasoning capabilities that we have. and as a result like basically every notable experience i've had so far with ai has been using it to augment my workflows rather than just replacing my job or replacing productive work like yes it replaces large volumes of effort but it still needs experts to wrangle it to keep it on the rails to make sure that it's contributing in a positive strategic direction and like technically speaking, it could do things that we would hire other people to do.

13:38But in the past, in the days before AI, we simply wouldn't have hired those people because the effort would have outweighed the benefit. But now that equation is changing. So the effort for many challenges is now approaching zero through AI. And when we start to think about hiring, it's not about figuring out who can just do a job. It's about who can leverage all of the benefits that AI provides to us. And if anything, it's making the human more important because it allows us to focus on those higher order challenges that we tend to just spend so much of our time in the weeds that we don't really have time to focus on those higher order things.

14:19And then just one last point I'll make on this is I think there is a cultural movement right now across the board that is around AI backlash. It's seeing things like these provocative marketing announcement or statements that say that humans are being replaced or CEOs out there saying 50 % of our code is being generated by AI. I think those sorts of things are creating, along with the mismatch of what developers' expectations are around it, so developers aren't really seeing all these transformative things that the market is telling them they should be seeing, that I think there is this AI backlash that is brewing in a lot of organizations.

14:59So don't ignore that is the point I want to make. If your organization is experiencing that, you really do need to dive into it to understand why that's happening. Yeah, really well said. And it's definitely important to have the conversations about how people feel about the tool, get the shape of the adoption within your organization, but also the resistance and really validly listen to their concerns and be able to meet them where they're at. And, you know, many of those concerns, they range all over from the creation of the models to even how they're run and arguments around their water consumption, right?

15:31And there was a really interesting article too that came across our desk that tackles this water consumption issue that many of us have heard around AI usage, data centers popping up to support the usage of AI. It dives into the actual numbers of AI water usage for data centers in context of other things within the U.S. that use and consume water. And in this article by Andy Masley, it breaks it down by comparing water consumption to other industries, like even things like pharmaceuticals, but major parts of the U.S. GDP, like livestock and crops as well. So I thought it was really well written.

16:08Ben, what did you think about this analysis of like the actual AI usage of water? Yeah, you know, it's really easy to get caught up in the details of stuff like this. But I think anyone who's like really deeply concerned about AI's water usage is kind of missing the point. Now, one thing I will say is there are a lot of concerns around like all these data centers that are popping up all over the country. And, you know, I've been trying to get our producer Adam to let me rant about that on an episode, but we just haven't found the right story to cover yet. but and those are valid concerns right like you know you can't have a data center opening up in the middle of a community and consuming you know a substantial portion of their power and water and forcing them to pay for it like you know that's that's that's not a great relationship to have ai is going to get more efficient over time i think a lot of the the water consumption problems we see are related to all these companies that are chasing like more advanced models which I think that, you know, I've said on here in the past, like how I think that is the real bubble that's happening here is all these companies that are trying to get AGI with bigger and better models all the time.

17:17And I think that's where a lot of the water consumption goes. But if we were to look at what we have today and just how we can apply existing GPT technology, over time, this is going to get more efficient. So water usage, energy usage, it's all going to just go down over time. Like that's just how technology tends to develop. But also more importantly is like this stuff also can help us solve really big societal challenges. So if it can help us solve big problems, the water consumption and energy consumption make it worth it. And I'm aware there's all types of, you know, some people might view them useless ways to leverage AI.

17:53And that's certainly a thing you can critique, I suppose. But there's also a lot of benefits. And, you know, but like this really just makes me think of an article. It's an old article that I come back to almost every year. It feels like written by Aaron Swartz, which is a really interesting human being for anyone who doesn't know who this person is. You should go check him out. But many years ago, he wrote this article titled Life in the World of Pervasive Immortality, the Ethics of Being Alive. And in this article, he basically points out how in modern society, it's actually almost impossible to be a perfectly moral human being.

18:30Every decision you make has a butterfly effect and creates downstream weight. It's almost pointless to get caught up in these moral dilemma discussions because if you're using the AI to do something that benefits your life, then the water consumption ultimately may not matter. wow i thought we were going to just like talk about like water consumption of almonds that's deep ben you know yeah we were joking about like what can we make a conversion calculator you know can i convert one hamburger to a number of ap of ai requests like we need that so i can just like make that decision in real time you know yeah or or we all need to get those like suits that they wear in doom that like makes you like completely water efficient you never lose your water we're We'll just do that for everything.

19:16You only get as many AI requests as you generate water off your body. That's brilliant. Yeah, that's a great sci-fi idea. So if anyone writes that book, be sure to credit me. Now that we've covered the philosophical approach to how our technology impacts the world and leaves our ripples in the existence that we have, let's lighten it up a little bit and talk about a fun article about the HTML spec. This came across my desk from Den Odell, and I loved this article because it's a really fun, bite-sized piece. And it's about a tag in HTML called output. So everybody knows input. We use input as buttons and fields and, you know, ways for users to send information into a server.

19:55But output? Output has largely been sitting on the wayside. You know, us React developers of the world and otherwise have just been embracing other types of libraries and approaches to do the kind of thing that output in the HTML spec was always meant to do. And what was it meant to do? It was an ARIA supporting field that announced streamed changes live to the user after they were completed. And this is actually a really fundamental part of solving a lot of accessible AI tooling and user interfaces for the web. We're talking about, you know, talking to a chat agent. It's streaming its response back to you and using this output tag to properly flag this information for a screen reader and also read it out loud.

20:40So this makes, you know, using LLM technology, which is getting deeply embedded across the agentic enterprises of the world, much more accessible. Imagine going to a website, you're on Williams-Sonoma, you're asking for recipes for your crockpot you just bought, and it would allow you to use like a screen reader to be able to get the streamed response from the LLM. So a really fun look at something in the HTML spec that's always been around, A semantic HTML is always amazing to embrace. So check out his really fun and simple HTML examples. Yeah, I got a lot of respect for people who build foundational specs for core technologies on the internet.

21:18You know, it's kind of a thankless job. And when they plan it right and build cool things that maybe it's not relevant the day they build it, but eventually it's time comes around. It comes around. No, it's like the founding fathers, you know, when they wrote the constitution, whatever, they thought about all these different edge cases. You know, the IETF, when they sit down and they write those specs and those proposals, they think of how this stuff is going to evolve for decades. And it's really impressive to see, you know, the diving catch. Yeah, absolutely. All right. Well, that's that's our news for today.

21:50Stick around for my conversation with Matt Culver from super.com. Are you tired of slow, inconsistent code reviews? Meet LinearBee AI, your new AI-powered workflow assistant built to supercharge your team's review process. With LinearBee AI, you'll get automatic PR descriptions, AI-generated review suggestions, and instant insights that help your developers fix issues before a human even looks at the code. No more waiting, no more guesswork. Just faster, smarter, and higher quality reviews powered by AI. Visit LinearB.io to bring AI into your code review process today. Matt is a senior engineering leader at Super.com, where he champions the principles of data-driven servant leadership.

22:38He is a recognized thought leader in developer experience, management, and the science of measuring engineering organizations. Matt, welcome to the show. Thanks. I haven't heard that read out before, but it all sounds very impressive. We like to hype up our guests here on Dev Interrupted. But hey, let's dive into it. We chatted a little bit. I'm really excited to talk about some of the things that you're here for. Some engineering leaders view AI, which is a topic that everyone is talking about these days. But a lot of engineering leaders view it as a way to save time. And as a result, a lot of organizations view it as a way to reduce headcount often.

23:19But you have a different opinion and you think that might be misguided in a lot of situations. So let's start there. Yeah. So my take on it stems from something I've been thinking a lot about as I kind of study and admire other companies that have become ultra successful. In particular, NVIDIA really focuses on a metric of revenue leverage for employee. And they have one of the highest proportions of this in the world. And so when I started thinking about the value that AI brings to our organization, and hearing a lot of the conversations from different people interested in the value of AI generating or improving efficiency or allowing us to produce more lines of code, that kind of thing was, this shouldn't be a conversation about tuning for efficiency, this should be a conversation about how we can take the time that we gain back and reinvesting it, or how can we leverage ai to make one developer have a much higher value generation leverage essentially is the way i think about this um and every time people get myopically um sort of focused on the development loop um i immediately start to think what about all the other other inputs to this process uh because the value generation loop uh the end-to-end product development life cycle if if you will, in most software engineering companies, starts with a human writing an idea down, articulating that idea or looking for an opportunity in the market for something they should be creating.

24:52And that process is time-consuming and that process is complex and that process is often wrong. There's a lot of opportunity, I think, to apply AI there, considering, again, the whole value generation cycle. And some of those problems are a lot easier to solve, too. like again we can make you get you get like just a a straight one-time um step function improvement like developers are 20 or you know get 20 of their time back and that's great and that is valuable but is that as valuable as hey we cut the entire total product lead time from having a notional idea about a product to being able to write a spec to be able to get that spec ready broken down into tickets and estimated then start writing the code if that process took us 20 days before and it takes us 10 now, that's crazy valuable as well, right?

25:40Because that means that it can start realizing its revenue potential earlier. And so when I look at this problem, I'm really still trying to consider it as a holistic problem where we have to just throw away the old paradigm and reinvent it in a new AI first paradigm where your product developer or owner, your developer writing the code, and your product designer are all sitting down collaboratively now, starting with AI first to generate an artifact they work from instead of working in, you know, parceled work streams through Jira tickets in that traditional way that we have for however many, how long has Jira been around?

26:21It's like, I don't even want to know, but yeah. Yeah. And I think you're hitting a really great point because, you know, with AI, this is a top, a theme we've hit many times on this show. Our listeners are probably sick of hearing it at this point. But with AI, you can do more than ever before. And if you're doing the right thing, that's amazing. You're doing more of the right thing than you've ever done in the past. But if you're doing the wrong thing, we're doing something that doesn't have value to your organization. You're just doing more waste than you've ever done before. And really what you want is for that more to result in more impact coming out of the engineering organization.

26:58So, you know, If all your engineers are doing is using AI to migrate some deprecated system that none of your customers use anymore, then why are you even using AI to begin with? So I really appreciate that you have more of a focus on making sure that AI is helping you reinvest into the organization in a more productive way. Yeah, like, people keep asking for, I think, a really interesting thing that maybe we should be a lot more vocal about that we're coming to realize is, our journey as a company has been one where we have, you see what our one of our original founders, founders, talking about this all the time, but AI has been in, has been in our ecosystem.

27:46them. And we've been like aggressive, bleeding edge early adopters. And as a result, we've been able to ship more and more product growing our company every year by this huge, huge margins. And we have not been growing a number of engineers doing that work every year. We've been relatively static as a headcount for the past several years, not as a strategic cost saving decision, but because we were able to get more and more and more value generation leverage, because we were getting smarter at like our entire tooling pipeline and removing friction for developers and using tools like linear b for developer enablement and speeding up that that iteration loop and getting stuff out of people's way ai is just another tool having arrived on the scene that allows us to further optimize that yeah and let's let's dive into that because you've emphasized that you know any change that that you implement that impacts your engineer ultimately needs to to improve developer experience.

28:44That should always be your goal. So how do you see AI reshaping the way that organizations prioritize developer experience? There's actually kind of a hazard here that I've spent a lot of time thinking about this week, which is what I'm hearing and what I'm seeing is that the interaction surface area between the developer and the AI in the development tool chain right now isn't making developers' jobs more enjoyable, more interesting. It isn't It isn't in many cases. In fact, yeah, it's not making their job better. It's actually making it worse. And we have a lot of cases where this is true. For example, a really common hazard that I'm seeing all over the industry is someone will take AI, they'll point it at their code base, and they'll ask the following question.