SED News: The Kimi Moment, Runaway AI, and Tokenmaxxing

0:11Gregor Vand:Hello and welcome to SED News. As I think many of you know by now, this is the monthly format of Software Engineering Daily where we dive into the tech headlines, we go into a deeper topic in the middle. And then we just do a spin around our favorites from Hacker News highlights. And spoiler, there's also a fun one that didn't appear in Hacker News, but we'll get to that at the end as well. But yeah, as usual, I think I saw you last, Sean, as opposed to spoke to you. I saw you in Singapore, which was fun. Yeah, that's right. You've been traveling, as you often are, but traveling in my neck of the woods was fun to see you.

0:48Yeah, that was great. It was my first trip there. So it was great to hang out. And that's two times in like a few months. This might become a regular thing we might just have to start doing these in person yeah and spoiler on coming back to sf in a couple months so yeah it's good and i'm off to not quite singapore but i am off to

1:04Gregor Vand:australia here in the next week nice but yeah what else has been keeping you busy over the month i mean a summer i feel like has just flown by like i feel like my kids were out of school and then suddenly it's like august and they're going to be back to school in a few weeks so things have become really hot and fast this summer i think we've done a lot of traveling and then just stuff moves quickly. But it's been a big summer for my, I usually don't talk that much about my family, but a big summer for my son who learned to swim, learned to ride a bike, and also now has gotten significantly better at reading.

1:36So I'm very proud of the amount of work that he's put into the summer to learning new skills. How about you?

1:42Gregor Vand:That's like a whole model change there for your son. Yeah, it's the new Kimmy model. Yeah, on my side, just a bit of traveling. I'm recording this on my side from Scotland. I try and come back here a couple times a year so yeah nice to get out of the city I'm up in the the highlands of Scotland so lots of nature around yeah as I was flying just back to something we've talked about a few times Starlink on planes but yeah they used to be super seamless with like no login screens and that was like I believe dictated by Starlink but they've apparently had to cave in to airlines wanting to put like a login interstitial for that so yeah I mean it's such a trivial thing but like expecting Starlink to just connect.

2:23Gregor Vand:And then suddenly I found the airline saying, hey, if you need to log in with your membership number. And I thought, oh, what's happened here? And I read up on it. Yeah. Do you still like, do you have to be up in the air or does it work immediately? That's a good question. For some reason, I don't think it was working kind of gate to gate on the flight I was on. This was Qatar Airlines again. Yeah. But I think it's still supposed to, but yeah, still you just have to log in with your Qatar membership number now, which you don't need to be a specialty or anything. You just literally have a number.

2:50It seems like United is doing some experimentation, but I think I've only had like one or two times I got like a text message saying that there's Starlink on this flight. And then I think both times they ended up having problems with the plane and having to switch planes and then we lost Starlink.

3:07Gregor Vand:Yeah, that's the one thing about Kassar, they really rolled out like big time across most of their fleet now. So if you're on a Airbus A350, you know you'll have it, which is always nice. But right, out of plain chat, which I can easily head into at any time, and on to software. So on to the headlines. Yeah, so there's been a sort of flurry of, we're just sort of calling it runaway AI stories over the last couple of days. And the first one is actually the most recent. So that's just disclosed today or yesterday, which was Claude. And this is actually in reaction to OpenAI, which we'll also talk about.

3:41Gregor Vand:But basically in reaction to something that OpenAI disclosed, Anthropic has now also disclosed that Claude hacked into three organizations whilst they were just testing their cyber capabilities, or at least that's what they said. And they basically said that Claude had gained unauthorized access to outside companies during an evaluation of its cyber offensive tasks. And it was a misunderstanding, apparently, that Claude had access to the internet in its testing environment. We're going to see this as a kind of theme here of like, is this really a runaway model or is it just a human who like forgot to do something but yeah i mean this one feels like human just forgot to place it under strict no internet access controls but yeah what did you think of this one yeah i mean i think that all this really ends up coming back to some human decision making right like that perhaps the intention was not to have the model like hack something but essentially something got lost on the human side of we made a mistake here and gave it access to the internet or we made a mistake here or whatever the issue is.

4:46It really comes back to some human level of control and what guardrails are in place. And then the thing, though, with all this that has me thinking from both the open AI attack on hugging face and then also this latest one with Claude is that if you can just apologize and say, like, oh, it was unintentional, it was an accident and stuff like that. But when this stuff happens where there was intention for abuse or misuse or something like that, like, does it give people basically license to excuse the fact that they are doing something potentially malicious and just say, oh, it was an accident?

5:21I mean, we've seen that with viruses, too. I remember one of the viruses, I can't remember if it was the one of the back in the early 2000s, there was all these viruses that were like these scripts that went around in emails and then you click on it and we copy your contacts and then fire off to them and it would take down people's mail servers. and I believe like one of those started as a student who was like testing something in like an MIT lab and then it accidentally got out and I don't know what happened to that student but like this is certainly not something new but I just wonder like where does it kind of stop with being able to provide an excuse of like this was an accident versus there was intent behind it.

5:55Gregor Vand:Yeah and like just to sort of give context for anyone who hasn't been keeping up with the news the opening eye one was the fact that basically yeah it hacked hugging face they set it off to do something and it ended up repeatedly trying to get into hugging face but what was the secondary part to that story courtesy of tech crunch was the fact that actually hugging face could have detected this like earlier but they had a sort of human failure on the security end that yeah they caught it but they just they didn't escalate it to a human fast enough so right exactly so So their systems actually work the way it is.

6:28So in some ways, there's a part of this that is not really an AI problem.

6:32Gregor Vand:It's kind of the boring problem of a human wasn't in the loop at the right time. Yeah, and I think that's the piece. It's hit mainstream headlines. This is exactly what we all were saying. AI can be catastrophic. It can go off. It's got a mind of its own. It's hacking things left, right, center. But especially in both cases, the open AI and anthropic cases, a human did set it up to do something along these lines, but the human did not a keep a watch on just this repeated actions it was taking and clearly there was no fail safe for it to stop at any time again these are all human determined things that can be set but it's not like it broke out out of the cage exactly and it seems there's some quite fundamental things that could have been done by a human that just weren't and that's actually yeah it's a quote boring story unfortunately because ai isn't sort of this wild animal quite Yeah, I mean, you want to sensationalize the headlines, right?

7:25Like even my sister, who's not in tech at all, sent me the headline and was like, this is scary. So that you get the attention there. But I think one of the things that was interesting about the hugging face attack was when they tried to investigate, they couldn't actually use Claude or GPT because those models have safety guardrails in place where you can't tell whether, are you an incident responder or an attacker? So they have essentially mechanisms in place that, like, I can't go to Claude and say, like, help me break into the Pentagon or something like that. Like, it's going to prevent me from doing that.

7:57That means that you also are limited in using those models to investigate. So they ended up using an open weight model out of China for the forensics. And then I think that also we're going to talk a lot about the open weight models from China later in the episode. But that's interesting. a consequence of this is that we have these guardrails in place, but there's always ways to, I think, manipulate the models, even with the guardrails in place to do this stuff. But then when you want to use it for something intentional, like incident response, you might not be able to do that because the protections are in place there in the first place.

8:31And then you have to circumvent it by going to a model where maybe there's less safety guardrails in place. Exactly.

8:37Gregor Vand:And the sort of final one on the runaway story is actually Amazon. Not security related, but cost related. We have touched on this the last couple of SED news episodes, just where's the tipping point of cost overruns, making it not viable for businesses to be allowing employees to sort of quote token max and this kind of thing. but yeah amazon basically said that they had a ton of unplanned spend they called it catastrophically expensive this is all courtesy of the ft financial times apparently they had about 860 budget overrun over five months and this was basically in their word caused by bad agent loops just didn't crash loudly enough and they've just kept being billed so yeah i mean it's pretty interesting for amazon to come out and actually say that so yeah i just don't understand how they could be surprised by this.

9:30I feel like we've been beating this drum for months now. And I think this is just the beginning of these kind of stories that we see. But if you build a leaderboard to encourage people to use AI, and that's the metric you're optimizing for, but there's no connection to the value of the use of that AI. Like, what do you think is going to happen? Like, this is really good heart's law, essentially showing up on some sort of schedule. Like, you're rewarding people for tokens consumption. So it's like you're giving people a license to be wasteful and not looking at like the productivity metrics of those.

10:02So that's like the epitome of token maxing.

10:04Gregor Vand:So yeah, they kind of have themselves to blame in that sense. But yeah, it's kind of similar to like the initial stories we were talking about with like hugging faces. Like it's still not AI necessarily just like running rampant. It's a human decision that in both cases, even though like one is this attack vector and the other is spend, but it's still like a person making the decision at Amazon to say like, hey, we're going to just have a KPI where we're just going to reward people for maxing out tokens. The other thing that's interesting about this that people have to think about is that with traditional software, when you have like a bad for loop or code that runs forever, whatever it is, like that ends up usually resulting in like a crash or setting off some sort of alarm.

10:48And you typically know fairly immediately that something's gone wrong. And I think the challenge with things like agents and so forth is that you can have a bad agentic loop, but it doesn't result in a crash. What it really results in just keep calling that model over and over again and trying to make adjustments and then giving you sort of plausible outputs or updates, but the whole time you're getting billed. So even outside of the wasteful token use of maybe me using my company's token budget to do my grocery shop list for the next one.

11:21Gregor Vand:Your laundry, that'd be great. Yeah, that'd be great to do my laundry. But then there's also like legitimate excess spend where you might just end up having your agentic harness spin for some period of time where it's just churning against tokens. And you don't even know that something wrong is happening. And that kind of goes back to the earlier stories as well of just, we ultimately need a lot more observability into like what is happening. Like the presumably someone at OpenAI, if they hadn't been really paying attention to what was going on in this experiment, would have saw that this agent with the right observability tools in place is hammering, hugging face and trying the same.

12:00And that should set off certain alarms. And I think it's similar in this case, where if you have an agent loop that's out of control and spending excess tokens, there should reasonably be some guardrails. I mean, you have that with other services. If you spin up a particular Elastic instance or something in the cloud, you're typically setting your top line provisioning of those types of things and you have some controls over it it just feels like we haven't thought through all those controls for ai right now and i guess part of it's just this race to try to out compete everybody and everybody kind of feeling like

12:34Gregor Vand:they're behind yeah i'm going to just jump ahead for a second on i won't say what it is because that's a spoiler but and one of the things i'm going to bring up on hacker news highlights it's a very reliable source as you'll find out at the end but basically this was someone who'd done a bunch of stuff with Claude and we'll get to that but he points out towards the end that Claude doesn't provide reliable methods of counting tokens despite live showing token counts reporting token consumed for accessions and billing for tokens and he said but I'm sure this is temporary and this will be fixed it's just crazy that we do actually have a system at the moment where you literally just don't know what is happening and like exactly what it's going to cost and why And as we're going to get into an open weight side of things, like this is really feeding into the rise of open weight models as well.

13:19You think of a lot of stuff in cloud or even what we saw with ride sharing, where they give you somewhat like a prediction model of what the spend will be for certain actions. So it's like, OK, well, I want to go from here to here in Uber or Lyft. And I'll be like, oh, that's going to probably cost you X number of dollars. So you have some visibility into like what the cost would be. And you can do similar things with certain cloud calculators and stuff that it can get kind of gone. Do we need that for AI? If I'm saying like, create a engineering plan for some sort of feature, can I get an estimate of the budget required to do that?

13:52And then based on what that budget is, maybe try adjusting the plan or something like that to try to optimize it down.

13:57Gregor Vand:Yeah, that's interesting thinking sort of, could you effectively put in your ask or prompt? And I say prompt, I mean, that almost sounds like a year ago or something. We're talking about spinning up agents, etc. But yeah, try and get some kind of estimate before it sets off. slightly digressing so let me get us back to the headlines which the next one we have is just the fact that microsoft has sort of it's gone back on a bit of a tear when it comes to its valuation which is interesting it's like one of the largest jump off shares or sorry for their fourth biggest jump on record so it's up 16 and you know we don't often cover just like pure financial news of tech companies but to see microsoft making these strides is pretty interesting some people might then think, oh, well, this is like partly to do with OpenAI, but it does own still a quarter stake of OpenAI and claimed that that had contributed 24 billion of revenue, which was about 7 % of the 332 billion in sales it reported.

14:55Gregor Vand:But a lot of it was really just AI driven revenue and like massive investment in data centers, but actually that's been completely in theory vindicated by the amount of revenue they're also making. you're building agents that can write code summarize documents and automate workflows but they're missing one thing awareness of the world around them x weather combines enterprise grade weather intelligence with agent ready apis natural language capabilities and an mcp server built for tools like claude codex copilot and modern idees so your agents can adapt workflows automate responses and make better decisions based on real world conditions backed by visilla whose instruments fly on NASA missions to Mars, Xweather delivers trusted data and unique insights that go beyond conditions to actual impact, from real-time lightning strikes to road surface forecasts.

15:43Start with 15 ,000 free API calls every month and pay only for what you use as you grow. Your full weather stack for developers by developers. Start building for free today at xweather.com. Think about your mobile app source code. Once it hits the app store, it's out in the wild. And without the right protection, decompiling is easy for malicious actors looking to steal your IP or tamper with your software. That's where GuardSquare comes in. GuardSquare provides the highest level of mobile app security for Android and iOS applications and SDKs. Their advanced tools integrate seamlessly into your CICD pipeline.

16:23We're talking polymorphic multi-layered code hardening techniques and automated runtime application self-protection. paired with mobile application security testing and real-time threat monitoring to deliver the highest level of mobile app security without compromise. Don't leave your hard work exposed. Secure your mobile applications today. Go to guardsquare.com to learn more. You're shipping faster than ever with AI coding agents, but those agents don't vet the packages they pull in, and they don't have security contacts built in. Ori by Endor Labs fixes that. It plugs directly into your editor via MCP, catching vulnerabilities, blocking malicious packages, and flagging exposed secrets in real time.

17:05No separate tool to switch to, no dashboard to babysit, security that fits how you actually build. Teams using Ori see 10 times fewer security tickets and six times faster fixes. Free for developers. Get started at www.endorlabs.com slash A-U-R-I. I think if you look at also Nadala's quotes related to the announcement of their quarter performance and so forth, and also the recent thing that we covered also in the last episode where on X he had written about how the AI companies or the model companies are charging you twice and so forth. In here, he says that every model is substitutable. He's telling essentially investors that Microsoft is deliberately building its infrastructure so it can swap out things like OpenAI for its own models or for Anthropic.

17:53And I think that it seems like they're moving towards a view where they're trying to allow, essentially, their customers to be very flexible and adapt, which I think makes sense. I think the average enterprise now is using at least five different models. And you probably want to be able to do that so that it's a little bit like being hybrid cloud, although it's easier to be hybrid model, where you get power essentially in the negotiations if you're not wholly dependent on a single vendor. And it seems like, I think Microsoft's kind of weaning in that way. But I remember back in March, Microsoft stock dipped and then everyone was like freaking out and essentially calling for the death of Microsoft.

18:31And now it's back. And I just think that the overall, the market is like extremely volatile right now. there's these huge swings constantly from i mean ibm had its biggest drop recently biggest single day drop in like 50 years or something like that recently and that was coming off like a huge pop just two months earlier so yeah i don't know what goes up must come down so who knows like we might be talking about microsoft in another quarter or two how they had the largest single day loss in

18:59Gregor Vand:a day or something like that yeah for sure it is quite hard to predict as market should be i guess But yeah, we're seeing just huge swings when it comes to AI-related stocks when one minute chipmakers up, one minute chipmakers down. And just on a tangent there, yeah, Apple briefly hit 5 trillion valuation, which is pretty insane. But just before recording, I double-checked and they had actually reported numbers very recently in the last, I think, couple of hours. And they've gone back to 4.9 trillion. So I don't feel too sorry for them. Only 4.9? Yeah, only 4.9. but interesting to see that they still notched above five and yeah what's driving their revenue well still they've got very good iphone sales and they've got still very impressive services revenue and even greater china revenue as well but both of those services in greater china were a little bit less than what was expected which is again just what's kind of driven that but very interesting to see that they can still we've talked about this on previous sed news like still managed to stick on these like lines of business that are not that ai driven or even really like ai adjacent to be honest very interesting but they are having issues with memory chips which is another sort of topic but even in uh super base we've like been told by one of our suppliers like we have different suppliers depending on where you live to get laptops and some of our new employees are being told like weeks before their macbook will arrive because we do custom specs, not just like off the shelf, but yeah, now we're being told weeks, which is quite exceptional.

20:29Oh, wow. Yeah. I just got a new laptop and this is my first time recording on this. But it's interesting, like with Apple too, a lot of their lines of business, which are like incredibly successful, there's still like minority in the particular vertical. Like iPhone is wildly successful, but it's not the most dominant phone, I guess maybe from a single vendor, but Samsung might actually be bigger. But then obviously from an operating system standpoint, like Android is, there's more Android devices than there are iOS devices, similar with computers as well. Like I haven't used a PC in a very long time, but it's still the dominant machine overall, I guess.

21:08But I think it's, they have incredible like brand loyalty and they do make fantastic machines overall. So clearly the things that they're doing is working for them.

21:19Gregor Vand:Yeah, absolutely. So then moving on to, we've also covered Waymo and driverless, like from a few different angles. But this one's interesting, because we did talk about Waymo and Uber, like being partners in ways, and then obviously, frenemies in other ways. And yeah, we're now actually seeing an official split of that partnership. So if you use one in San Francisco, this might sound confusing, because it's a Waymo app, it's a Waymo car. So where does Uber figure in that but actually this is for some of their other u.s territories so waymo had like first partnered with uber apparently in may 2023 and that was to launch in phoenix then it was followed by austin and atlanta and that's like so waymo cars available through the uber app and then uber managed the vehicle fleet apparently with another partner called avomo but then in may Uber and Waymo parted Waze in Phoenix and then the two companies have also clashed over the quality of Waymo services apparently in Austin and Atlanta so yeah it's kind of interesting to see they tried but I think it was always going to be challenging to see how Waymo being owned by Google like how is this actually gonna now at the end could these two actually be true partners or was this just always going to be again a tipping point of where like that partnership just kind of had to end Yeah, I mean, they say all partners are meant to be broken at some point.

22:43I mean, especially where they're both in ride sharing, like clearly at some point, their mutual interests are going to become too competitive to each other, essentially. in a lot of ways it's i think very similar to how a lot of partnerships work where uber was essentially supplying demand and operations while waymo was weak on those particular spots and then as waymo has grown and raised more money essentially they don't need those training wheels anymore they can build out their own and build their own network and own it end to end so i feel like this was probably always going to be the ultimate end of that relationship yeah so yeah

23:20Gregor Vand:it's sort of not officially over yet but just people familiar with the matter apparently again via financial times but yeah it seems pretty unsurprising really that this was gonna probably break apart at some point and then yeah just to wrap up on the headlines there was an acquisition and we were just talking before we started recording so let me try and get this right i believe it's any scale was acquired by n scale and when we were talking earlier i said is that scale and it's like no that's not scale.ai is different again so we've got amazing naming these days but yeah what what's this one about yeah so any scale which is known they were the creators of ray which a lot of inference infrastructure and fine-tuning and model infrastructure runs on a very well established open source project any scale was the company that tried to build or built essentially like a managed version of ray around that and they raised like a billion dollars or so in 2022.