SED News: OpenCode, AI Code vs. Shipped Code, and the LiteLLM Breach

0:12Gregor Vand:Hello and welcome to SED News. We are your hosts. I'm Gregor Vand. And I'm Sean Falconer. And as I think a lot of you do know already, this is a slightly different format of Software Engineering Daily, where we pick off some of the big tech headlines that you might have seen in sort of more mainstream news we're going to dive into a bigger topic in the middle and then we just look at some of our favorites from hacker news towards the end which usually ends up in sort of weird and wonderful rabbit hole type projects that developers have been working on so as usual we just like to kind of catch up on what we've been doing what has been in your sphere sean since we last did a scd news last month?

0:55There's been a lot in the last month. So I moved into a new house, which has been great. But while I also moved in a couple of weeks ago, I actually have traveled each week of this new adventure in this new house. So I was in DC and then I was in Seattle for our data streaming world tour last week. And then my wife was away this week. So we've been here almost three weeks and there's never been a completely full week where everyone as a family has been in the house yet so i'm looking forward for that happening and then confluent also the company i work for was officially acquired by ibm about a week and a half ago so that's exciting news that's all completed then yeah yeah everything's been completed so lots and lots of conversations now going on with those folks over the ibm side figuring out how we're going to work together which is exciting lots of stuff happening and then you're looking ahead to april my kids are going to be on spring break which means that everybody's going to be on spring break for vacation.

1:50So we have some travel coming up and then I have some work travel to Cloud Next. And then I'm also going to India. So lots and lots of stuff happening over the next little while. But what's going on in your world? You're in a different location than usual.

2:04Gregor Vand:Yeah, well, I'm in Scotland. So that's where I'm from originally. I try and come over here for a month or two of the year. When I do, I'm up in the far north in the highlands of Scotland. So it's always kind of fun. That's what technology makes possible. you can still do a podcast from all the way up here where there's i actually looked it up the town i come to is a population of 80 80 people so i thought i was from a small place yeah so i didn't actually grow up here but it's sort of where i've decided is where i come back to you know when i come back to scotland so yeah it's always quite a change from singapore yeah but that's i think by design but yeah no i mean work wise has been kind of exciting actually obviously small plug here for super base of course but yeah we've just launched so stripe we've been doing this interesting project called well called stripe projects which is a cli tool effectively so you install it through their cli and basically there's a bunch of partners it's not just super base you'll find like all the usual names on there but it's been really great i've been kind of overseeing that one and it got out yesterday so we're recording this on a friday and this became kind of public in preview yesterday it's just awesome when you see something like actually hit the public sphere that you've been working with engineering on for the last like we've really been crunching on this last couple months and nice to see that go out so yeah check it out and yeah i'll be in sf later this month as well that's for sessions that's stripe sessions so for obvious reasons as i've now just disclosed is like we're working a lot with stripe and stuff so yeah if anyone is at stripe sessions do look me up and say hello always great to meet new faces as well yeah that's exciting that'll be our first opportunity to meet in actual person in real person yeah in real life yeah yeah sure yeah for anyone listening sean i have never actually met in person so yeah this is gonna be cool and you're coming to singapore at some point as well yeah so we have two opportunities there we go yeah whether we'll do an in-person episode that's that's a different thing because actually recording in person means like a whole studio setup i think or something whereas people are surprised i've done a few in person it's there's a lot more technical complexity with doing in person recording on your computer remotely exactly yeah i've had a couple of guests on se daily just say oh can we go to a studio they're like well we don't have a studio se daily is a remote podcast that's how we do it so right moving on to the headlines so some of you that do listen to this sed news a lot we have talked about arm in the past, but more so the lack of ARM.

4:35Gregor Vand:But I think the headline here is ARM is back in a big way. And this all kind of focuses around CPUs, because the headline stories of any of this compute inference, the story has really been around GPUs. And we did talk a little bit about TPUs, which are Google's derivative of that. But we are really going back to CPUs. So yeah, Sean, what did you pick up on this one? I mean, it's like everything in text, Like the old thing is now the new thing again. Like it's always cyclical. I think part of it, the reason we have GPUs, we have TPUs and now, you know, CPUs are back. Like it's just, you need a lot of compute to run all this stuff.

5:12If you're running models, you want probably GPUs, TPUs to be doing sort of the matrix multiplication there. They're really good at that. But CPUs are also good at certain things. You know, they're good at logic and branching and task switching. And I think given now that people are running these agents on their desktop and not just running one, like running multiple agents. I talked to a startup that I worked with recently, and I was like, how many instances of quad are you running simultaneously? And they're like, well, I start to lose track of things when I get up to like six. So that's a lot of compute that you're running locally.

5:43And I think essentially that's what's kind of driving this is that people are running these agents locally. I don't know that that was something that we necessarily foresaw in the industry a year or two ago, that so many people would be kind of running these agent experiences on their local computer versus the cloud or something like that, where you maybe you're using some other type of compute but if you're going to be running these locally for things like open claw or some other version of claw that now exists or you're doing it with the agentic engineering tools that are out there you're just going to be consuming up a lot of cpu and a lot of memory so there's now a lot of i don't know appetite i guess in the industry to find

6:20Gregor Vand:more of this compute yeah and i think one of the interesting pieces here is that arm is actually going into more of the actually manufacturing their own chips not fabbing exactly because you fab being effectively the sort of manufacturing plants that can do this and those are incredibly there's only three really players that have those i believe is tsmc the taiwanese is the main player intel do have that capability and samsung is actually the other huge one that has that capability not entirely clear who's fabbing these for ARM but the difference being that ARM traditionally was supplying the designs of chips like that's their whole thing virtually every chip that is created has some design piece in it that comes from ARM and so they basically take a royalty if you want to call it that off almost every single chip that is created in the world but they hadn't really gone full on into hey this is like an arm chip with an arm logo on it for example and this is kind of where they're going and yeah as you've called out sean it's the rise of agentic tasks and it really has just accelerated as we touched on i think in the last scd news and in the last like two three months that suddenly everyone's doing agentic stuff not just engineers like anyone seems to be quite interested now in getting an open clock running and like leaving it on overnight on their mac mini or something so there's clearly just a lot of appetite now for having these independent environments that have their own cpu to run a specific agent and i think that's kind of what we're seeing here yeah and i think that's only going to going to continue like these companies whether it's anthropic or others are verticalizing a lot of their success around cloud code cloud desktop to be geared towards finance and geared towards lawyers and whatever profession that you have.

8:15So every sort of white collar job eventually is going to have people who are running these things in some sort of environment. So you need compute. You probably need some sort of containers and security parameters around this. So there's not just, there's, I guess, like a ton of people's startups all the way to large companies that are really trying to meet the needs of the market right now. And it's something that like everything now, when it comes to AI, it's exponential growth. it's not this kind of like you know nicely growing linear thing it's like as soon as something takes off it's like extremely viral and companies have to move really fast to essentially meet that demand

8:50Gregor Vand:yeah i mean nvidia are like are in this space as well you know they offer cpu racks but i guess they're still known i guess on the gpu side but yeah it's kind of i would say like quote great to see cpus coming back i think the fact that they've powered so much to this point in time from a technology standpoint and then everyone kind of thought oh they're like yesterday's technology almost but they're incredibly powerful but for specific tasks and you don't always need like a incredibly expensive gpu running every single task as i think a lot of people have decided just for the sake of if they can afford it that's what they're going to do but it isn't actually needed yeah when i did an scd episode unfortunately i can't remember the person's name but he was a turing award winner in supercomputing one of the topics of conversation that we had was how The industry's focused so much on workloads catered towards is essentially the math behind model crunching and inference has led to essentially poor performance in supercomputers for certain types of tasks.

9:47So there is always like a tradeoff, right? Like if you overly specialize in something, GPUs are really good at a certain type of math. TPUs are even more biased towards being really good at that particular math. you're also giving up something that might be important for other types of logic or compute that doesn't essentially fit the profile that you're going to run on one of those other pieces of hardware like a GPU. And when you're running agents, it's not like it's just model crunching. There's all kinds of other things that they're doing where you want to be able to ideally use the right compute depending on what the profile of the task is.

10:22Gregor Vand:Exactly. And it's just the volume here. like there isn't really a sort of realistic world for like an average person at the moment where every agent has its own GPU. So even for most people in tech, like the numbers don't add up if that's like what it takes. So if you want the number of agents that you think would get your tasks done, you're probably looking at CPUs to run most of those logic tasks. So yeah, very cool to see ARM back in the fold. And yeah, I read a book about ARM kind of just before, I think it was just before they IPO'd or something like that. And I just, I really fully at that point and understood like, wow, they have just behind the scenes powered so much of what we take for granted with all these designs, but never really got the recognition for it.

11:07Gregor Vand:So it's kind of, yeah, it's fun to see them actually stepping into branding their own CPUs as well. So exciting. moving on to this does sort of hit the main tech headlines in the sense like this was a lot of reporting by tech crunch on this one and i think especially anyone in software engineering has maybe seen it on hacker news as well but i do think this kind of really crosses into main news in terms of our world the light llm hack now this was a takeover of a dependency effectively that then extracted the credentials of many services so light llm basically gave nice api access to a multitude of llms and helped with like things like metering and so on so forth so it was a pretty used tool by so many developers out there so that kind of makes it unfortunately incredibly ripe as a target for someone to think about what can we do if we can get the credentials to that.

12:05Gregor Vand:And I think it's just the focus here, LightLM themselves were a ex-Y Combinator. And then yeah, the kicker here is the fact that another Y Combinator company, Delve, they were already in a bit of hot water for potentially fabricating. So they're a compliance startup. Sock2 reports giving, you go through the checklists and you get the, Sock2 is still a funny industry where it's actually accountants that do the final stamp on things. That's a whole other story but they apparently auditors as well yeah exactly they're your auditors and it just it still seems like kind of strange that you've got effectively a financial company saying that your security of your technology is up to scratch but that's how that system was put together and yeah there's been allegations that delve have been fabricating the sort of the rubber stamping of these reports unfortunately delve had given one of their reports with a clean bill of health to LiteLM before this happens.

13:05Gregor Vand:And it doesn't, I think that people are pains to point out, getting stock 2 does not mean they have checked every single process that is going on in the developer's workflow within LiteLM, for example, but it should be a pretty good indicator that they're following best practice. And while something kind of wasn't going by best practice here and it's had a catastrophic effect on a lot of developers. But yeah, that's, I guess, the sort of slight security-minded focus for me, but what did you make of this one sean yeah i mean i think it's just another example of where like compliance isn't necessarily security compliance is really a series of kind of check marks where you're saying yes we are following the best practices but every company pretty much that has some sort of data breach has some sort of hack is in compliant they pretty much all have like sock two or whatever various isos like whatever sort of security check marks that they need.

13:57And I think in a lot of ways, like compliance is really about insurance, while security is actually about trying to stop the attacks. So there's a very big difference between, I would say, like the companies are really like secure by default, and they take a hard stance on this versus people who are, hey, we have to make sure that we have these badges, so that when we're going through procurement with a vendor, they're not going to immediately just say no. I also think the interesting hearing here is where a lot of times when we think about hacks historically or breaches, it's like, what is it that people are trying to go after?

14:31Well, they might be trying to go after someone's password. They might be going after credit card information, banking information, personal information for identity theft. And here, I think there's now this new industry that's going to be the value is, hey, I want your open AI API key or I want your Anthropic API key. And then not only can they run up bills on you, but they could be leveraging those models to be doing something malicious against a bunch of other people too. So they can spin these things up using your precious tokens, run up a big bill for you while also using that thing to attack somebody else.

15:04Gregor Vand:Yeah. So this was discovered by a guy called Callum McMahon of Future Search, which is a company that offers AI agents apparently for web research. but he like documented how this all unfolded basically ended up with his machine being shut down so that's a pretty clear indicator that something has really gone wrong here in terms of how deep this sort of hack has gone it's something we've covered on se daily with guests obviously of companies that help to mitigate this the one that comes to mind is whiz we had rami mccarthy from whiz and we actually covered a case that he'd worked on that is exactly this it's like a dependency has a sort of leak within the maintenance of it and that just cascades through so many other packages and this one particularly just the number of developers right now using light lm is just shown to be such a problem i think these kind of like dependency supply chain attacks are hard to catch and it's easy thing for attackers to kind of exploit and then in a world and we're going to talk more about this during sort of the main topic, but in a world where we're using AI more and more to generate code and maybe not always looking at that code that closely and is picking up packages in order to do some type of work on our behalf, we might not have as clear a handle on what those dependencies are.

16:28We're kind of just looking initially at the outcomes of that. Does it pass the tests? does it kind of look correct if it generates some sort of ui based on what it is that we're trying to accomplish we commit that thing and then we get it running production and then boom

16:41Gregor Vand:we have some sort of problem yeah so the next headline here i guess again this might have crossed more slightly into the hacker news from a headline perspective but i think it was interesting that yeah open code really made a lot of noise in the community here and this really goes to like the fact that agentic coding has just taken off but most of the time we were talking about either cloud code or we're talking about codecs from effectively chat gbt and that means you're buying into again one of the big names and here was a product that's able to say hey we're actually fully open source you've got access to local models you've got access to free models or if you really want you can hook it back up to opus or something like that but i think that's just the amount of opinions that were then given about it is kind of really interesting.

17:31Yeah, I think we're going to, of course, see more and more of these open source alternatives. In a lot of ways, it reminds me of, you know, if you looked at the IDE market, maybe 15, 20 years ago, it was very similar to where open source IDs started cropping up as alternatives to the mainstream ones. And the thing that I've always thought about, or since people start, even with GitHub Copilot was, I think we've always gotten to a place where developers don't really like paying for their tools. And then you end up with like open source alternatives. And then with the IDE market in particular, the IDE very much became like a commodity.

18:06And there are certain IDs that survived that and people pay for, but predominantly I would say most became open source or were given away. Even things like Visual Studio that historically had been like a big moneymaker for Microsoft. So today, people are able to monetize these agentic engineering tools because they are incredibly valuable. If someone took away my quad code, I'd be very upset about that. So there is a lot of value there. But as you have more and more of these open source equivalents that are maybe, even if they're not as good, they might be as good. But even if they're not as good, maybe they're 80 % as good.

18:40People are happy to use something that's 80 % as good if they don't have to pay x dollars a month to use some other tool so then you get into a world where people are willing to pay for the tokens but people aren't willing to essentially pay for subscriptions at scale to have those tools so if you want to monetize that market like how do you create a big enough value and a big enough moat around what it is that you're providing so that people don't just go to the open source equivalent yeah and it was kind of interesting just to look at

19:09Gregor Vand:some of the conversation around it in hacker news as well and i guess what maybe even i hadn't sort of fully appreciated is the fact that like clot code okay it runs in your terminal but it is actually an electron app i didn't even realize this and like that's how you kind of achieve all these amazing visual niceties effectively like yes it's in a terminal but there's a ton of stuff going on from a rendering perspective like terminals are yeah being hacked a bit in this way I love the visuals, but then I was kind of starting to think, how is this even possible inside a terminal? Codex apparently is done in Rust, and people say that that is a tangible speed difference.

19:49Gregor Vand:And one of the things that has been questioned about OpenCode is the fact that basically you're using up a gig of RAM basically for a terminal, and that it's a pretty chunky TypeScript code base that's running this thing. So this is it. When you're designing these kind of tools for developers, well, guess what? You're going to have some pretty, I think, spicy opinions on like, how did you put this tool together in the first place? So I think that's going to be an interesting adoption data point that they have to think about. In mobile application security, good enough is a risk. GuardSquare uses advanced, multi-layered code hardening techniques and automated runtime application self-protection and mobile application security testing, combined with real-time threat monitoring to deliver the highest level of mobile app security.

20:40Discover how GuardSquare brings all these together to provide mobile app security for your Android and iOS apps without compromise at www.guardsquare.com. Today's episode of Software Engineering Daily is brought to you by Unblocked. Your coding agents have access to your code base. Maybe you've even connected other tools via MCPs. But access doesn't mean context. Agents can't reason across MCPs. They don't know your architectural decisions, your team's patterns, or why the API was shaped the way it is. So agents look in the wrong place and deliver bad outputs. Then you spend time correcting, turn after turn.

21:19Unblocked is the context layer your agents are missing. It synthesizes your PRs, docs, slack, and tickets into organizational context that agents actually understand. So they make better plans, write higher quality code, use fewer tokens, and require fewer correction loops. If you are running Claude Code, Cursor, or any agentic workflow, Unblocked is worth a look. Get a free three-week trial at getunblocked.com slash sedaily. You know, there's a lot of these tools now available. You have Claude, you have the mainstream sort of commercial tools from OpenAI, from Anthropic, and from others. You have the open source equivalents, like you have Klein, you have OpenCode.

21:59There's a whole plethora of these. How many can the industry really support? I feel like sometimes one of these tools crops up, kind of catches fire for a period of time, gets a lot of GitHub stars and love and people start using it. And then it kind of like tails off as the new hotness hits the market as well. So I don't know that that's like sustainable in the long, at some point, I think some of this stuff starts to level out. And they'll emerge like a couple of main players that dominate the market. And maybe there's a couple alternatives, but I don't think you can have like 100 alternatives.

22:29Yeah.

22:30Gregor Vand:And just to round out the headlines, this sort of, I guess, broke cover maybe a couple of weeks ago, but I think it just sort of is useful to touch on a slightly wider story that we're seeing play out, which is how OpenAI and Anthropic are kind of, how are they playing their markets now? and there was obviously a lot of noise around how the pentagon they wanted to the department of war or justice or whatever they're called these days wanted anthropic to allow clods for what they call all awful purposes and that it does include things like surveillance and autonomous weapons and anthropic refused and they didn't get the contract and then apparently open ai quotes swooped in and signed a deal and there's a lot of i think user backlash around this and it did make Anthropic look like the real human winner there if you want to use one of these technologies like why would you use the one that really wanted to get in with the government the US government and if we kind of then just like play that through to like well where have we seen OpenAI versus sort of Anthropic like where are they positioning themselves like it does to me at least look like OpenAI started to becoming this kind of super app type construct which which we've touched on before, Sean, like in sort of other deeper topics and Anthropic, Cloud, they're really going like full enterprise and in a sort of say in a positive way, but they're really going and saying, we can be the enterprise tools that you need.

24:00Gregor Vand:If you look this back to the whole Pentagon thing, well, a lot of companies really don't want to be associated with a company that they think is like aiding the government with say surveillance or as we speak the iran conflict is on right now and we don't know how the technology is being used there so just kind of staying away from that i think is what companies are looking at and it's just a very interesting development i think in this sort of the landscape of the two companies yeah absolutely i mean i think that it ends up kind of highlighting i think something where those two companies are already somewhat have like opposing views of what is the right thing to focus on when you're building you're sort of responsible for building these models and even this Dario and some of the other founders at Anthropic were open AI at one point and they kind of split off because at least the story goes that they didn't agree essentially with the philosophy that open AI was taking towards like building models or releasing models.

24:56They wanted to make sure that as a company that before they released the model, it was really well tested and really like security and trust was sort of the number one value. So they splintered built Anthropic. And I think this whole thing that's happening politically has been probably some of the best marketing that's ever happened to Anthropic. Like you saw people leaving or canceling their open AI subscriptions. I think even under the best of circumstances, like with politics, people, companies, individuals might not be that comfortable with a company supporting something like surveillance of individuals or automated warfare or anything like that.

25:34But on top of that, we're also in a place in the United States where politically things are very polarizing right now. And there's a large chunk of the country that doesn't want to be associated with the government at all. So this is a very, very polarizing issue. And it's kind of Anthropics kind of chosen to take one stance on this. OpenAI has taken a different one. And as a result, I think it just highlights the differences between the two companies even more. Yeah.

25:59Gregor Vand:And it's not like we've seen things like this probably appear in the past where a company potentially anchors itself a bit closer to a government or not. I'm not going to bring up any specific examples. But what I will say is like, in the past, okay, would one have refused to use this? I'm just taking pure examples that might not be true here. But would I refuse to use Google, if I knew that they were aligned with a certain government? I'm not sure because like okay it's a lot of your search history and that kind of thing but now that we're talking about what LLMs are used for well it's quite often people are divulging a lot of their personal information beyond just oh this is what I purchase or this is where I'm located right now which is you know it's personal information of course but people using LLMs they're kind of writing about their deepest darkest thoughts and in companies especially people are just literally hooking up whether through official integrations or full-on just pasting in slack conversations and documents and that's it like where does that information end up then ultimately and i think that's why this is such a concern for companies especially yeah absolutely there's something i think more personal about chat than necessarily search i think people have kind of exactly like taught themselves that okay well maybe i shouldn't put my social security number into like Google search.

27:18I don't know that people think of that necessarily with chat as much, or they, you know, paste documents in that are incredibly sensitive, not really thinking that through because of the value. There's just so much value with using these tools that people kind of throw out maybe their reasoning skills sometimes with like, should I actually be sharing this? What is the consequences of sharing this information?

27:38Gregor Vand:Yeah, I've definitely noticed that people that previously would have sort of said, I will never divulge certain information to a tech company. And then the next day, oh i've had this amazing conversation with claude and told them all these things and i don't comment on it i just think that's really i'm just observing it's very interesting that this is someone for example that i know would not have done this in the past but actually just they've felt some trust there that's a whole topic for a different day but i think it just sort of underscores like why who you give your information to especially between the you can call like the two main players here anthropic and open ai as soon as one then anchors himself to a entity that people disagree with that's like adds a whole extra layer so we'll watch how this plays out i guess and yeah we'll sort of see does this affect open ai enough that they have to walk anything back or are they kind of going more like palantir style they actually think there's too much value to be had by being very close to the government and helping them achieve things like i guess we'll see how that happens yeah and i think from like a consumer standpoint a lot of times when there's so much value behind whatever the product is, consumers end up turning off a blind eye to maybe things that they wouldn't normally agree with.

28:48You know, you can take any major retailer in the world that's probably manufacturing goods in developing countries where maybe the standards of work are not necessarily great. People kind of sort of know that, but they also like their inexpensive, you know, t-shirts or whatever, you know, and they're kind of willing to turn that part of their brain off because they're focused on like their own personal outcomes and i think you could potentially see something similar with you know even if you don't in conversation agree with what open ai or any other company is doing there's so much value behind their tools you kind of turn it off when it comes to your own personal objectives yeah so it's that time within sed news

29:29Gregor Vand:we're going to move to the sort of main topic, deep dive. And today we are looking at effectively writing code versus shipping code, and especially the emphasis on shipping code. How does this look in the age, if you want to call that, of LLMs now that, you know, probably at least 50 % of developers in a given company are going to be using LLMs to write code? What are the kind of maybe cascading effects on how that code actually then ends up to be frank on main branch you know what does main branch look like from a speed perspective and actually getting things merged in and we kind of anchored this around there was a circle ci as you know a company that many people know you know in the cicd space and they did a 2026 state of software delivery report so that was just back in february we're recording this just at the end of march they said that they analyzed this is gonna be quite data heavy this main topic so get ready for some numbers because thinking about when we talk about writing code versus shipping code it's basically all numbers like lines of code and how many mergers are made and so on and so forth so they said that they analyzed 28 million ci cd workflows across 22 000 orgs in 149 countries so that's a pretty good spread this is the interesting anchor points here that the daily workflow throughput was up 59 9%.

30:53Gregor Vand:But it's kind of misleading that the top 5 % of teams nearly doubled the throughput, whilst the median team increased only 4%. So that basically suggests that super high performing teams could super accelerate their throughput, i.e. things that got merged into main, again, just to kind of bring it back there, they could double their throughput, the top 5 % of teams. But the median, just call it like, you know, the average, I know average and median is different, but when people think about it that to only increase four percent on throughput for most for like just call it the majority that's pretty interesting because you know most developers are saying oh i'm so productive i can you know i can do so much more per day now if i use an llm but if actually from a team perspective and a shipping perspective the majority are actually only increasing by four percent that looks way off what we might have expected i think from this yeah You know, there could be a couple of things going on.

31:52One is that, you know, not everyone has reached the level of the most high performing teams. Adopting some of these tools is still relatively new to many parts of the industry. So you're going to have people who are kind of trailing behind, maybe they're kind of as a company, maybe just, you know, dabbling, or this is like fairly new efforts. So they're kind of getting used to how do we get value out of this. but I think the other thing too you could say is like well maybe the teams that are shipping these like you know top performance maybe they're shipping a lot but are they shipping also a lot of bugs and they're just you know okay with that whereas other people are you know trying to control that so there's a bunch of stuff that it would be nice to kind of be able to dig into the details of like why is it that some people are at you know four percent increased throughput whereas you have some people who are you know double this and there could be many many different reasons for that to be the case.