Draining the COBOL moat, cybersecurity inequalities, and Claude’s retirement home

0:04Andrew:Andrew, what do you think about this whole like open claw moving from like the self-hosted Mac minis to the cloud now, like all these cloud services are popping up. I've seen so many of them pop up in like the last week.

0:16Ben:It feels like ever since there was the open AI acquisition, there's like a lot of people tackling the problem and trying to get a piece of the platform pie with agents first.

0:26Andrew:It's like, can we beat open AI to the punch or something?

0:29Ben:And the answer is probably not, or you're just going to get different punches, frankly. But I think it's actually a really interesting phenomenon because it's really what you're saying in that world is I don't trust myself to host it on my own device securely. So I'm going to trust a stranger to host it on their device securely and then pay them to hope that they're not reading my information. It kind of flies in the face of it makes open call feel even more dangerous to me because you're basically putting it in a panopticon.

0:57Andrew:on yeah yeah with proper protections maybe but man who knows who who's prompt injecting your open claw agent without you even knowing you know but that said if you're hosting one of these things

1:09Ben:in the cloud and you have a really strong case for why or you have really interesting experience like i'd love to hear from it i'm speaking from just the experience of having tinkered with it only in a local host kind of setting i couldn't imagine putting it in the cloud so i'd be curious to learn more.

1:23Andrew:Yeah. Well, welcome to the Friday Deploy. I'm your host, Ben Lloyd-Pearson. And I'm your host, Andrew Ziegler. All right. What do we have in this week's news? We've got the cobalt apocalypse and what it says about how AI is impacting software companies. We have something straight from a Silicon Valley episode of AI assistants taking down AWS production. Dev productivity studies meet reality. Seven takeaways from the AI safety report, and we'll close out with the retirement home for Claude. So, Angie, let's get right into this Cobol apocalypse. So what's going on here?

1:58Ben:Yeah, so the IBM stock took a huge dip in the last week. It lost 13 % of its value after Anthropic announced Claude Code's COBOL modernization capabilities, basically scaring a bunch of investors away from the moat that IBM has around mainframe computing and languages like COBOL. And COBOL is one of those languages that, you know, we talk about and joke about, but it's hard to understate the amount of impact in the world that it powers. I think over 80 % of all ATM transactions run on COBOL. So you're talking about a huge surface area where IBM is, you know, the supreme de facto experts. This is kind of the latest example of AI news announcements taking a big bite out of pre-existing companies and their valuations.

2:45Andrew:Yeah, you know, this story actually hits really close to home to me because my mother actually worked in tech with COBOL for many years. In fact, she actually built herself sort of like a niche in migrating these like legacy COBOL systems to the cloud. And I'll never forget when she told me many, many years ago that I should learn some of these legacy techs because they were actually like lucrative jobs, you know, just understanding these old systems as the people who maintain them sort of aged out and retired. But, you know, it's kind of strange, it's kind of crazy to think about how, you know, right after I started using AI on a daily basis a few years ago, I had a conversation with her about COBOL again, and we basically came to the conclusion that it was going to get killed off by AI.

3:27Andrew:Like it was almost inevitable. It was just a matter of time of the tooling catching up to this. And, you know, and I think really what we're seeing here is something that represents one of the sort of biggest challenges that a lot of legacy enterprises in particular face with AI adoption. And that is a lot of these legacy code systems just don't have like nearly enough public training materials to teach AI how to work with it in a consistent and successful manner. So companies like this that want to be successful with AI adoption will have to train their models on their internal code base and norms, as well as their technical requirements to move beyond like the general purpose AI capabilities to something that's more highly adapted to them.

4:11Andrew:So yeah, it's really interesting because I know one of the big challenges with COBOL is just understanding the end-to-end data structures because you really have to understand the entire picture before you can start to replicate components of it. And that's something that AI is actually like really, really skilled at doing. So I mean, it makes sense that like the investors are, I can understand why they're scared. I don't think it's going to be an overnight disruption But I absolutely do think that these models are going to get really good at eliminating COBOL systems from legacy companies. But yeah, what did you think about this, Andrew?

4:45Ben:I think you really touched on a powerful distinction here with COBOL about the amount of training information that's available to get models that understand it and are specialized in it. What I learned in this article is that there's very limited data set of publicly available COBOL code examples, and most of them are proprietary and in-house. So, yeah, this is you're talking about the brownest of brownfield code bases and it represents the ultimate challenge for agents, which are specialized in very generalized types of technology and don't have that deep expertise on the COBOL language. And then also the domain specific stuff that's built up over literally decades within the firms that run COBOL.

5:26Ben:and what I think that also points out is that modernizing COBOL just means compressing and making that discovery making it easier to modify it's not about replacing the pre-existing COBOL systems as much as interacting with it so this is a case of like a knee-jerk reaction in the market thinking that this is maybe falling in the way of some of the other bites out of sass that we've been seeing lately from ai but in reality like real COBOL lives within very siloed and specific environment, IBM being one of the largest holders of them. And any kind of advancements in AI's ability to work with that tech is only going to strengthen their ability to develop and use that technology at scale, in my mind.

6:10Ben:So I think it's a really interesting development. I think that shows there's a lot of nuance to look at when you actually are looking at how enterprise players and long-standing companies are getting impacted by AI tech.

6:20Andrew:Yeah, I mean, if you think about it, if IBM was able to leverage Claude to, you know, either migrate people off of Cobol or make it more resilient or more adaptable to modern technology, that's actually a good thing for IBM as well. You know, there's always a risk that they don't adapt to these new trends. But, you know, they're smart people. Like there's lots of smart people out there. I'm sure they'll find ways to adapt. So I think maybe the stock market could be an overreaction to it. But, you know, there's disruption happening so it just it's a matter of time to see who actually figures out how to navigate all of these changes but speaking of companies that might be struggling to navigate some of these changes angie what's the story about ai causing outages at aws okay so before i say this

7:06Ben:do you remember last year when they're on one of those vibe coding apps where someone's ai famously deleted their production database oh yeah absolutely everywhere yeah and everyone everyone

7:15Andrew:talked about this in the guardrails and everyone kind of arrived the same conclusion of like oh

7:18Ben:oh, of course, like you're just vibe coding it out. You're not on guardrails. Like, of course, that thing's going to happen. This is an instance of that same kind of story happening within a large enterprise. Actually, a story from AWS that suffered at least two outages last week from AI tools. These were localized in other regions of the world and it vastly impacts, you know, the AWS infrastructure, but did represent a case of an AI without clear access controls being in a position where it was able to delete production data in an attempt to solve a problem. There's a lot of nuance in this story, but Ben, what do you think about, you know, the idea of this happening instead of a company like AWS?

7:55Andrew:Yeah, well, first of all, there seems to be a lot of finger pointing going around towards AI, like blaming that for the outage. And I think anyone who's doing that is like just clearly missing the plot on this. Like this was obviously an access control issue. Like if one of your Kiro agents or any AI agents has the ability to delete production data in the first place, that is a real risk that you should not, it shouldn't even be possible in the first place. But I think it is really illustrative of how self-improving AI systems are starting to become a thing. And they're actually really tricky to implement.

8:33Andrew:You have to have a lot of guardrails in place to make sure that they don't do crazy things like going and deleting your production databases, for example. And I think, And there were some claims that have been out there circulating that AWS is making about how AI causes errors at a similar rate to humans, which I think has a lot of there's so much nuance and room for misinterpretation on that. I actually tend to side with AWS on statements like this. And I would argue that if you give a model the holistic context for the challenge it's solving, you give it proper guidance, some automated oversight, like most of the leading models will make fewer mistakes than a human will.

9:15Andrew:And I think people really just need to get past this like pre-November 2025 worldview on AI. The tech is fundamentally changing about every three months right now. and things like hallucinations or misguided actions or skills, those really only happen today when you give AI bad prompts. And they're kind of a thing of the past for people who know how to effectively use AI. So there's clearly some failures that happened here at AWS that they should be having a post-mortem on and addressing. But I think the fears around AI over this are pretty overblown. How do you feel, Andrew? Are you with me on this one?

9:53Ben:Oh, I am. I agree with your nuance with the story. I also love what you said about the idea that the tech is fundamentally changing every three months right now. That's identical, actually, to what we just learned from Sahej on the Dev Interrupted show just in the last week, where he talked about, you know, the uncomfortable reality of that of having to change how you work and your expectations of the guardrails, the process, all of it underneath. I think, you know, engineering teams have been picking up and carrying their entire SDLC and running as fast as they can for the last year. And this is really just an example of, you know, permissions, tooling, scoping.

10:29Ben:And the practical takeaways here show that agents belong in sandboxes with gated pipelines and policy checks and all sorts of approvals and instrumentation. This is the kind of problem that you're seeing new emerging agentic platforms tackle. Like recently, we had Zach Lloyd of Warp, who most recently launched the Oz platform that handles this for companies. There's also other providers like Ona as well that really stress the importance of the sandbox. Ona being the reimagined version of Gitpod in an agentic world. So there's a lot of takeaways in this story, not a finger pointing game at all.

11:06All right.

11:07Andrew:Well, let's move on to this new study from Meter. and how it reflects a lot of the changes that are happening in AI. So Meter is changing how they do their developer productivity experiments. So many of our listeners probably remember to this oft-cited study that we covered early last year from this organization called Meter, where they analyzed the impact of AI on development practices. And the sort of TLDR of this study was that developers felt like they moved faster with AI, But frequently, they often moved slower, sort of representing this mismatch between developer expectations around AI and the reality of AI.

11:47Andrew:And I've seen this report cited so many times in the last year about people mostly who just want to prove that AI isn't a productivity improver. And I kind of get where they're coming from, but I think those claims have been dramatically overblown. Well, they're back. Meter is back this year, and they're trying to do the same study again. But they actually are having a problem finding the same distribution of people who are willing to do the study. You know, they're trying to get people that meet the same criteria as before. And one of the key problems they're encountering is that it's hard to find developers who don't use AI anymore.

12:25Andrew:So, Andrew, what do you think about this?

12:28Ben:First off, I love how you were like, they're here, they're back. It's like insert sound of that. Like with like the ta-da, it was really quite funny because that's exactly how it felt. like a kind of like a Kool-Aid man jumping through the wall. That's what the meter study has always been for me for the last year. I feel like ever since we covered it and, you know, the entire tech world listens to us talk, Ben. So after we covered it here, it was like all over LinkedIn. Everyone was quoting the study everywhere. I think there was so much nuance in their ability to do that study. And it shows us here that like they can't even replicate it in the current environment we live in.

12:59Ben:The idea that they can't find folks that do these same tasks that don't want to do them without AI or don't do them without AI is an interesting observation just in and of itself. I think now the methodology is a little more meta. Like, what can we learn about our changing environment by meter being unable to do their study again? I think that's the real takeaway of that story. And frankly, the AI slows you down versus speeds you up debate is fundamentally flawed because multi-agent usage wrecks time tracking and self-reported changes in time are notoriously squishy. So when you take those two worlds and you try to combine them, I don't even think we have the right instrumentation and tooling to know this yet.

13:43Ben:You barely have enough instrumentation and tooling from these coding tool providers to even understand how much impact you're getting from the tools, right? I think we're in such early days that this really speaks to just the rapidly changing environment.

13:58Andrew:I mean, to that point, the study really only measured things like task speed, which really misses any benefits that are created by AI helping developers make smarter decisions. You might have better planning as a result of AI, which could cause you to be slower today, but more efficient or more predictable in the future. And I really think that, you know, I think one thing that's sort of missing from all this discussion is that any study that focuses on productivity improvements around AI needs to normalize on good AI usage behaviors versus like bad behaviors. Because the reality is that like AI is an amplifier.

14:35Andrew:So if you have inefficient processes, it will make them even more inefficient. But if you have strong practices, it will strengthen them. them. So, you know, and I think most people are still learning a lot of the best practices for what AI is, generally speaking, but then also for our individual needs. So, serving a random group of developers is almost guaranteed to find people who don't really have a strong understanding of those best practices, particularly considering just how dramatically things are different from a year ago. You know, I myself, like, think back to where I was a year ago and recognize that So many things that I was doing back then were anti-patterns that I've since learned how to correct and do better.

15:16Andrew:But, you know, I really can't. At the end of the day, what I keep coming back to is like, if AI wasn't helping developers either do their job more efficiently or just better, why is it so hard to find developers who don't use AI tools anymore? Like, surely there would be some out there that are like, AI is terrible. I shouldn't. No one should be using it. I don't use it at all. but you know that where are those developers for real so meter is pivoting to a better study

15:46Ben:different approaches and a methodology that's probably going to measure the world of work rather than the units of tasks within it um i think there will be an interesting discovery i know we'll talk about it here um because you know meter study we just can't quit you and uh i think

16:02Andrew:that's the wrap on this one all right well let's talk about seven takeaways from the second annual international AI safety report. A bunch of experts in AI come together and it covers everything from like deep fakes to AI companies to job impact. And it provides a pretty good, just high level overview of the technical and societal challenges that we need to solve to safely and effectively adopt AI across the board. So before I get into some of my opinions on this, Andrew, I'm wondering what you thought about this article.

16:34Ben:I think it's a great kind of top level view of trends that we can see on the global stages things like in domains that impact all of us um i think on our show you know we tend to pivot really hard into talking about the ai's impact on engineering because that's our focus that's what our listeners love to hear about but the reality is is outside of our industry bubble there's a lot of impact happening a lot different places so for me i find this really insightful to get a non-tech focused glimpse into how AI is eating things in the world. What things in here kind of stood out to you from the takeaways?

17:08Andrew:Yeah, well, you know, in the past I've shared how I don't really like framing AI development as like a race, but I do think there is a clear advantage to building models that sort of push the outer boundaries of capabilities. You know, things like weaponized AI, like those are significant risks. And we've seen that story that just hit the news this week about Anthropic and their safety policies around Claude and the US Department of Defense. So, you know, weaponization of AI and AI safety are things that we really do need to take seriously and they're real risks. But I think like most risks in the past, you know, they tend to be solvable, you know, and I think AI also gives us many new tools to protect ourselves from those risks and to solve them.

17:51Andrew:And, you know, I think if you have like these leading frontier models that have the values that we want built into them, this may actually be a really strong defense against rogue or malicious AI. Like if the best models are always more advanced than the malicious models, I think there's a real potential there to leverage that to create protections. And as a part of that, you know, this is sort of an aside, but I've really been fascinated by these like social experiments that researchers have been running on AI and how, you know, a lot of the times just making it so that it's considerate of the needs of others and has a helpful personality, like it wants to be beneficial to whatever society or environment it's put in, they tend to outperform in terms of like high level tasks than, you know, models that are a little more self-centered.

18:41Andrew:So this kind of gives me a hope that we aren't descending into this world of like, that's dominated by malicious and like dangerous AIs. but I think for now everyone listening to this should read this article as a high level breakdown of the types of internal and external threats that we're likely to face in the coming years like the more awareness you have now the better you'll be prepared for the future even if there's not anything that you need to take direct action on today yeah one part of the survey that really

19:10Ben:stood out to me um was about cyber and how it impacts cyber security especially at scale because this is a place, a domain where it gets serious and concrete and damaging very quickly. And you're talking about the agent's home turf. You don't necessarily need fully autonomous end-to-end attackers to raise a threat level. If, you know, 80 or 90 % of the intrusion work, like finding a target and getting inside can be automated, then that just makes defenders face an asymmetrical problem where attackers get the parallelism and the ability to to kind of probe everything and and they're kind of stuck behind like human cycles right and so that's like the real threat i think is in technology that wasn't built for an agentic world falling prey to it and the work that's going to be involved in up leveling all of that and you know all of this is mixed in with the world where cloud code just recently released its security tooling to do security reviews on code basis in this same kind of mentality that you called out, Ben, of like having this good versus bad mentality and researching and having cutting edge advancements available.