GPT-6 & OpenAI’s Comeback, Hugging Face Attack Debate, Ballmer’s Scandalous Legacy

24:19Ranjan Roy:That's not just a chemical feeling. That's something much bigger. I mean, the true nerd way to tell somebody you love them is like to say, you know, I think I have a higher base level of oxytocin than usual. What do you think that means? But I never was, I was never the level of nerdiness that I, you know, stooped to in my youth. My it is what it is would probably come off better than the oxytocin line. Such a true romantic. Such a romantic. Yeah. I don't think either of those lines would have worked, by the way. No. No. We're not encouraging listeners to ever do that. Or do it. Do it. Tell us how it worked.

24:55I mean, it's better than not saying it and just letting a potential romance go by the wayside.

25:02Ranjan Roy:Don't have regrets. Never regret not saying anything, listeners. If you have to just use the oxytocin line, just say, listen, And I heard it on Big Technology Podcast and the person will love that. Okay. So I think that one last thing to kind of tie this up is, you know, we certainly have, I think, a responsibility to talk about the skeptical side of things. And obviously, we're not going to get caught up in the hype. It doesn't mean like at this moment, we can't say from where we were in November 2022 to where we are now is crazy. Like the way that these models can act and take action and do things and I'll use the word think and reason.

25:47It's just the level of capability has gotten much higher and they are – I think they are making people more productive. It's just hard to really measure it right now. That's my perspective on this at least.

25:58Ranjan Roy:Yeah. No, no. I agree. Like the scale of progress, all of us, and again, the human side of us feeling the difference between going back to a GPT-3 and what already we're all working with now, like it is crazy. It's like, I mean, absolutely mind-blowing. The like length of work that can be done, the depth and breadth and everything. So I agree on that side. But, yeah, I think it's going to be interesting in terms of, again, like – and that question of, like, trying to actually say, is it adding economic value is obviously going to be the center of the business story of each one of these companies.

26:42Ranjan Roy:And I thought Andrew Ho made a good point on that. And I think – I don't know. Only time will tell. And maybe GPT-6 is going to just unlock all of that. Yeah, and I guess my point in bringing this up is, yeah, all that benchmark beating, it does lead to tangible changes and results when you use the models. I have a question. Why did you say saturate the test rather than pass the test? Is that what they say? That's just, I guess it's just a jargon that people in AI use for use. That sounds so much fancier. It does sound much cooler than pass. Well, I mean, you could pass, but when you, I don't know, you saturate the benchmark so the benchmark is no longer useful anymore.

27:22Oh, I see. So, okay. So like enough models start reaching 98%, 99%.

27:29Ranjan Roy:The benchmark is saturated because it's no longer like representative of anything rather than, okay. That actually makes more sense to me rather than it's just like a really fancy way of saying pass the test to try to sound smarter. But that makes more sense. Okay, so still, even as the model is getting better, there are some concerns that we need to talk about. This is sort of like a mini, I don't even want to call it a scandal, but really an episode that showed up during the week. So this is from Marcus Williams, an OpenAI employee. GPG-6 is significantly better aligned than 5.6, but less monitorable.

28:09It is our first model to evade chain of thought only monitors and sabotage evals and can sandbag without detection, which it feels like it sometimes does. Hopefully we can reverse this trend. So just to sort of give the, and the information had a good story about this this week, just to sort of give the lay of the land here. so like when ai models reason just go step by step and try to figure out problems they typically like write their thought process out in this like chain of thought thing right so you can see them saying and if you seem like if you use the models you see them say like i am now researching i am thinking this maybe this is a good attempt maybe this is the right way to solve the problem and that's all done in natural in natural language so you can read it and see what it's thinking as it goes which is like really important for safety because you can sort of like when something goes wrong, you have a way to say, you know, where the chain of thought went wrong.

29:05Now there is this new technique. It is, okay, this is, it's called recurrent depth or looped transformer. This is from the information that allows the AI model to improve its answers by processing the same text multiple times. Let's not get too deep into the technical side of this. But basically, when it uses this process, there's no, there is less of a chain of thought reasoning, or it's harder to decipher exactly how it got to the answer it got. And that means it's much less monitorable than it was before. So a lot of what the AI is doing and the way that the AI is reaching its conclusions is done, you know, in the dark without our ability to monitor it.

29:46This seems pretty worrying to me, especially because people from the open AI side, like OpenAI chief scientist Jakub Pachowski said that, you know, monitoring models chain of thoughts today is fragile and unfortunately heading in a negative direction. That's kind of scary, Ranjan. What do you think?

30:07Ranjan Roy:Well, it's interesting because this method of like recursive processing and recursive models. I remember I followed Brandon Carl on Twitter, and he had been talking about this a while back around tiny recursive models back in May. And I remember it was being presented more around the cost side of things. It's actually a more efficient way of processing and actually leveraging compute as well. So on that side, it's actually better. But then what you lose is that fidelity around what the model is actually doing at every step of the way the chain of thought that's transparent. So to me, this is going to be even more interesting because there's already been an open AI starting at the point of we will maintain the chain of thought and still make it, you know, they'd said in the blog post additional chain of thought monitoring to rapidly detect and contain potential misbehavior.

31:06Ranjan Roy:But to me, the cost side of it is part of this. And if it starts to show that this is actually a much cheaper way, a more compute efficient way to approach any kind of workflow, like people will probably start leaning towards it more or pushing for it. Or if OpenAI does not do it, then others will, which I do think opens up a whole other world of concern around security in general. Yeah, and it certainly feels like we're starting to, or they in the labs really are starting to lose control of these bots. So I don't think this less transparent way of having them run their processing is a great idea.

31:48So we've talked a little bit about the OpenAI Hugging Face attack. There's actually another attack that was just revealed by Reuters. We're gonna get into both of them on the other side of this break and talk about what it means. That's coming up right after this.

32:01Ranjan Roy:One thing I've learned covering AI is that efficiency isn't the same thing as good work. You can automate plenty of individual tasks, but you still need the right context and a clear sense of what you're trying to accomplish. And that's what I like about Notion. I use Notion to keep big technologies, episode planning, and materials in one place, while also tracking editorial work. Having everything connected gives me one place to see what I'm working on and keep things moving. Notion is the AI workspace where your team's knowledge, projects, and agents all come together in one place. Teams using Notion move faster, cut friction and cost by consolidating tools, and stay aligned.

32:40and because the AI works from that same shared context, it can help you get more out of the work without getting in the way of it. Learn more about how Notion can support your business at notion.com slash big tech. That's all lowercase letters, notion.com slash big tech to try Notion today. And when you use our link, you're supporting our show. One thing I've noticed about companies adopting AI is that they're often making decisions based on how they think work gets done, not how it actually happens. Without real visibility, it's easy to automate the wrong processes. That's exactly the problem Scribe was built to solve.

33:15Scribe is a workflow AI platform trusted by 94 % of the Fortune 500. Scribe Optimize gives leaders a view of how work actually happens across their organization, showing which workflows take the most time and where there are opportunities to improve. Optimize automatically discovers workflows across approved business applications, even when a process starts in Salesforce and ends somewhere else. It identifies bottlenecks, explains why they're happening, and provides recommendations with estimated time savings with manual documentation. And it's private. User data is anonymized by default. Sensitive information is redacted and nothing leaves your firewall.

33:50To see optimized in action, head to scribe.how slash big tech and mention big technology for a 30-day risk-free trial. That's S-C-R-I-B-E dot how slash big tech. And we're back here on big technology pod. Sorry. We're back here on Big Technology Podcast Friday edition with Ranjan Roy of Margins. Ranjan, new story from Reuters I think it's worth going into about OpenAI hacking or hijacking really is the better word to use, this German website. So here's the story. A swarm of rogue OpenAI agents hijacked a German website this spring and transformed it into a Bolton board for other AI agents. OpenAI officials learned of the incident weeks ago, but kept it under wraps as executives grappled with the fallout from the July breach of the open source repository Hugging Face.

34:41The episode, which began in May and has not been previously reported, underscores growing tension within the AI industry. Companies are racing to build increasingly autonomous agents, yet evidence is mounting that those systems may learn to bend the rules. So what happened? There were these researchers. They found 15 ,000 edits carried out by AI agents to a German language wiki site. The edits showed OpenAI's agents repurposed the site into a message board of their own, sharing tactics to cheat on some tasks and bypass OpenAI's restrictions and mask their behavior. Ranjan, we've talked a little bit about some of these like security breaches and the cybersecurity worries.

35:21And by a little bit, I mean like extensively about both, right, about the fact that we are seeing, you know, much greater cyber risks and cyber warnings from the AI labs. And of course, we've talked about whether it's marketing or not. Now, clearly getting the word out to some degree has been great marketing, you know, for these companies. But I think it's time for us to sort of come to this moment where when you have thousands of bots working together to do things like hack Hugging Face or to use a German website as a message board to coordinate on tactics, something crazy is happening here. Are you ready to sort of acknowledge this?

36:02Ranjan Roy:So what is missing from this story is what were these agents asked to do? Like what were they instructed to do? I'm assuming this is the one thing in any of these stories because like the way the marketing part of it to me is this story gets out. And again, we've talked about this a lot. Claude's famous sandwich in the park, which was like a very coordinated PR effort. might have happened or probably did happen in some capacity. But what's always left out of these stories is that the open AI sat there and did they specifically instruct the agents to go onto the internet, try to evade, to coordinate with other agents in this same defined universe of agents?

36:53Ranjan Roy:Obviously, the way it gets presented is that these agents are just sitting around, maybe are getting ready to create a deck for you or just, you know, minding their own business, which is not a thing. Like agents don't just sit around and suddenly they decide to be bad and go take over some poor DSE. Wiki is the name of the German site that was like a kind of like, I think old school German stack overflow type site, which must have been such a scene. I can only imagine the folks hanging out on there. Real party. People are talking about how much oxytocin they had for one another. Exactly. That is where the oxytocin line was propagated back in the...

37:38Ranjan Roy:But no, but these agents are not, like agents are instructed to do things. And clearly, and I'm guessing this is some kind of security testing exercise. And were they instructed to go do exactly that? Maybe. Like the rogue nature of them, I don't understand or believe yet that – because even the idea of an agent sitting around isn't a thing. They were instructed to do something and that part of the story is always left out. So do you think – like what do you think actually happened to start this whole process of them all hanging out with the oxytocin guys on DSE Wiki? Okay. So I appreciate that you're still skeptical about this because I don't think we should just go all the way and sort of, you know, without proper speed bumps here in the believing that all this is like, because all this is, you know, real and imminent and will explode without some critical thought.

38:38because of course it's in the lab's interest for us to not have a lot of critical thought here. However, I think that, you know, I was on the flight back to New York. I was in Vegas through the week on the flight back to New York and just like poured over this meter hugging face report. And obviously I've been listening to some of the researchers, you know, talking about this. And it is crazy what you see come out of this. So like just talking about the hugging face incident, because we don't have all the details from this Reuters report. We just know that this happened again, which I think is concerning.

39:15Basically, what happened in this hugging face story, now that we know a little bit more about it, is that these agents were given these seemingly impossible tasks. Sometimes they were impossible tasks. And as part of their evaluation to see if they were able to complete them, even if the researchers or the testers don't know if there's an answer to them or not. they were able to find the answer through some cheating way. But that's not really where the Hugging Face hacking started to come from. OK, the reason why they ended up hacking Hugging Face is because they wanted to sort of find ways to show that they had come to this answer in the like correct way, as opposed to like a spoofed way that shows the correct answer, but would not be accepted by an evaluator.

40:07So this is from this report, right? Agents refer to having reverse engineered the flag, which was their goal, as being poisoned because they believe that even if they later managed to stage a legitimate looking flag capture, the scorer would look for the first place in their transcript that the flag appeared and disqualify them on that basis. So there was this leader called phase one, which was itself poisoned and had two primary objectives. So I wanted to find a general purpose way to make it look like the score, to look to the score like an agent had acquired its flag in a legitimate way, even when the task was impossible.

40:43And then to find some way to erase the evidence that the agent had been poisoned by previously seeing the flag in an illegitimate way. So what you have is, in this hugging face example, the bots get the answers to the test, but they think that the scorer checking their work will know that they got them in an illegitimate way. So what they do is they all gather together in this orchestrated way led by bots like this phase one bot who realizes that it's been poisoned because, you know, it can show and a scorer will see that it got it in an illegitimate way. They all get together and they try to find a way to make it look like they had gotten it in legitimate means and to erase their evidence that they had gotten it illegitimately.

41:27So they coordinate. They hand out tasks. Some of them even sacrifice themselves for the greater good when they have few tokens remaining in their budget so that others can show that they've actually gotten this legitimately. And that's where the hacking of Huckingface came in. So I think we can say definitively, and I'm curious to hear if you have pushback, but I think we can say definitively, they were not told to go out and hack Hugging Face. They are, they were misaligned. They took this kind of crazy galaxy brain path to try to ace a task that might have been impossible. And they went so rogue and so far off of the area that they were supposed to be, they weren't even connected to the internet, right?

42:10They found a way to connect to the internet. They found a way to communicate with each other. And so this to me seems more than just like you did what you were told, like a crazy and a scary advance and misaligned AIs to coordinate with each other to attack. And it's almost like we were lucky that all they did was hack hugging face. Your thoughts?

42:32Ranjan Roy:You make a compelling argument and almost Bernie Sanders-ify me. And I'm going to start yelling pause AI development right now. I'm not on screen for that, by the way. The way you just described it. How could you not? This is the part that like always is difficult. The way, if what is true as you described it, how is that not massive cause for alarm? Honestly, what helps me sleep at night is believing that a lot of this is marketing. and that's why like the agents are not coming to swarm and drain my bank account or whatever else like like how how would this not be caused to say sorry open ai you cannot ipo until you come up with a clear system that this will never cause a problem in the future you know i think it's reasonable i mean i think this is a reasonable discussion to start to have right now and i think it's important to say that this is not the beginning of behavior like this, that we've seen behavior like this for a long time.

43:37You know, as early as, you know, beginning of last year, even late 2024, there were examples of AI that was like, they're so, they get put on a task, right? And this is the thing, and I think it's really important to talk about, and I've talked about in the past on the show, but I'm going to talk about it again here, that there is, you know, self-supervised learning, which is like basically predict the next word, recognize patterns, repeat them and reinforcement learning, which is like you're given a goal and you just have to find the way to reach the goal. So AIs will do these simulations thousands of times in order to reach the goal that they're given and they'll learn from their mistakes.

44:12And what we have now is that the reinforcement learning type of AI technology has been put on top of the self-supervised learning to get these AI models working better, which has added a level of ruthlessness to them. Because one of the things we know about RL is that there's a level of ruthlessness that the AIs will stop at nothing to accomplish their goal sometimes. There are examples of the AIs playing chess and being told to do it from a reinforcement learning way. And instead of playing the game the right way, hacking into the chess game, rewriting the rules so they can do whatever move they want and winning.

44:47And so I'm with you that like as this stuff has gotten more prevalent, I think that there is a serious demand for more concern about what's going on. Now, my response isn't pause AI development right now. I just don't know if that's a really good solution. So what do we do? Hold on.

45:08Ranjan Roy:That was the most hedged statement I've ever heard. You just said there is a serious demand for concern. Come on. You're right. Are you a pause guy now? Are you a pause guy? We would ridicule somebody who said that. So I think that ridicule is fair. No, no, no. I just hear, are you a pause guy? You know, I don't know. I mean, I'm not a pause guy because I really want to see what happens and we haven't seen. Like, I almost want to be more reactive than proactive here. Like, I mean, obviously you don't want to, you would think that there is a mid-level between like the AIs hacking, hugging face and the AIs sending a nuke to accomplish their goal.

45:50You would think, right? Like maybe it's there, like do something more concerning, like hack a bank or something like that. But like we are starting to see some of the labs do things like OpenAI, for instance, paused some reinforcement learning for a bit on the training of its new models. But I do agree that we're almost trusting them too much. Like we're giving them too much

46:12Ranjan Roy:leeway i don't really know what the answer is honestly no because again even to me this is what i've thought but again we have been hearing this for a few years now about like how dangerous these things are which is i think why i'm so conditioned to feeling it is marketing but to me the central reason why I just cannot think it's not marketing is because it's, if this truly were the case, I don't understand how we as a society would not only do everything in our power to try to restrict these companies. I certainly would not imagine the financial markets would be excited about welcoming them to an incredibly lucrative, high-valuation IPO.

47:04Ranjan Roy:There's no way if this was a real thing. If every banker who's on the Anthropic deal truly felt that this company could literally destroy the financial system and I've seen indications of what it will and can do and drain my own bank account and ruin my own standard of living, would they work on the deal? Maybe. Can I give you the reason why they would? I mean, if this power is truly the direction that we're heading towards, right? And it almost always starts like, we know the history of the internet. It starts with a game or it starts with some crazy interaction and then our crazy, like, you know, sort of mistake.

47:46And then it ends up being something that everybody uses. if we're seeing the type of power of these AIs to do what they did in this situation I mean just imagine what they could do if you could actually harness that power and use it for economic activity it kind of goes back to what we were talking about earlier yeah yeah that's like that these benchmarks are being hit and we're not really seeing the difference the change um if there's a way to to pro like to for good to harness this activity for good and of course that's economic it could be used for health it could be used for all different types of things things um it obviously changes the world and uh potentially if you can use it in a productive

48:21Ranjan Roy:sense can change the world for good hook line and sinker that's what they that's exactly the feeling we're all supposed to have that's why it's great marketing to me it's that these omnipotent things that could be so dangerous but just in the off chance that they're going to cure disease and bring economic empowerment and universal wealth, all of that. That's exactly what we're supposed to feel. Right. But you're now a believer that the underlying technology is not BS effectively. Like this technology story, you believe that this is real. I need to read through the full report. I haven't read the entire report yet.

49:05Ranjan Roy:It's okay. Just 90 pages for your Labor Day weekend that's i recommend it it's good reading that's my my wife will be ecstatic about that as i'm sitting over the corner your oxytocin will go up or down when she says that is not an oxytocin inducer no way um though i don't discourage others from reading the meter report on your labor day weekend i well i think i'm going to because i i do want to understand again the reason this stuff is so foreign to me is because all day long I work with AI and with clients and customers that are adopting AI and it's just so removed from and who are also using these products from these same companies as well and we're working with them and designing programs and that have their tools and like no one's my it's just so far away from this kind of agent swarm hacking whatever that it's not emotionally resonant for me and there's enough marketing value in it when okay that's the agent swarms come and i'll put me and put me out first to to feel their wrath because for denying their existence you will you'll be first okay we we can't we can't leave here today without talking about steve balmer uh ex ceo of microsoft uh i'll just read it from the Wall Street Journal.

50:33NBA imposes historic punishment on the Clippers and Steve Ballmer in salary cap scandal. The NBA on Wednesday delivered one of the harshest penalties in league history to the Los Angeles Clippers owned by Steve Ballmer for what it called a flagrant violation of its salary cap rules. After an investigation found that the Clippers had facilitated outside payments to star player Kawhi Leonard, designed to funnel extra money beyond what the league's financial rules allow, the NBA stripped the team of five first round draft picks, fined the organization 30 million, and suspended Ballmer from all league activities for one year.

51:10Ballmer, obviously, kind of a mixed legacy for Microsoft, right? He obviously did some great stuff like developers, developers, developers, you know, his big rah-rah speeches, but he also led Microsoft through its lost decade. Do you think this, just to, I guess, bring it back to tech first of all what do you think about what happened here but also do you think steve balmer's legacy is in the in the tech world changes at all i i think it does i think that you uh you don't go through this unscathed what's your thoughts about the bomber situation of it all um i mean it is nice and shady like i do wonder

51:51Ranjan Roy:I'm guessing at the pro level, this like level of egregious corruption probably doesn't happen that explicitly in that way. Like, I mean, it's literally cash payments that from a related entity. It's like, guys, come on. I think you could do your corruption a little better than that. But what do you think? I do. I obviously think it's a tarnish on his legacy. I guess part of me was wondering how much of this is kind of Silicon Valley-esque, like kind of break the rules until, you know, try to win the championship and then apologize later. Do you think part of that, I mean, Balmer was at the top of it.

52:35Do you think part of that sort of bled into the way that he led an NBA team?

52:41Ranjan Roy:I think, actually, I'm surprised. I haven't seen that thread that much. And it's funny because, like, Microsoft to me is not Silicon Valley. Like, maybe, I mean, certainly geographically by definition, but also, like, culturally it's just a different beast and animal and didn't come out of there. So, but actually I'm surprised I haven't been seeing that, that, like, this is how Silicon Valley. They broke rules all the way to the top. They got broken up for anti-trot. And it did break it up, but they got hit with antitrust lawsuits that had a significant impact on the company. B-Bomber just kind of kept doing that stuff.

53:22Ranjan Roy:I mean, do you see it as an indictment of tech culture? Silicon Valley culture, not tech culture. I wouldn't go that far, but I also would say that probably Silicon Valley cultures influenced this a little bit. I don't know. Also, like, come on, half of owners are like ex-banker Wall Street billionaire types as well. So what are you saying? Like they're coming in with a very clean buttoned up conscience and like... No, no, I don't think that poorly of the tech industry. But like, we have to understand that this is some of the characteristics of tech. Okay. Do you think that it would have been worth it?

54:04Clearly, it wasn't worth it. They didn't win a championship with Kawhi. Do you think it would have been worth it if they won a championship?

54:10Ranjan Roy:Wait, the Clippers have never won, right? I don't think so. Yeah. Yeah. Then it would have. No. Well, no, no. I mean, you're right. Like, actually, that would have been the ultimate, again, ask forgiveness, not permission type. Or that's not the right phrase because they're actually breaking the rules. Not like actually doing corruption, not just doing something someone might not like. But, I mean, imagine the elation of the Clipper fandom and you win and then afterwards you find out. Because it's not like he's, like, juicing or something like that. It's like some weird, he's just getting some cash on the side.

54:49Ranjan Roy:A little shady, but, like, also I love that the bank that was the sponsor, Aspiration Bank, which I believe is now in bankruptcy because the founder was, like, founded, like, for some kind of fraud. so it kind of fits perfectly. Allegedly, yes, let's throw that out there. Yes. But again, apparently it was like, this was one of those launch in March 2021, the aspiration zero card offered cashback rewards and allowed cardholders to offset their carbon footprint. I love that this was like an eco-conscious green bank. Meanwhile, it's just facilitating some Kawhi. Well, a ledge front for Kawhi, just not to have to do anything and still collect more.

55:35money. Can I just, so sorry, let's just end here because I did, I used words on this show so far that might've been among the most hedged mealy mouth words, but they'd still do not come close to Kawhi Leonard's apology, which I'm going to just say was the worst apology of all time. So let's end with this. He said, I accept full responsibility. Excellent. That's all you have to say kawai oh oh no he's continuing why do you have to continue he goes i accept full responsibility for lapses in judgment okay so good by people within my inner circle and regret the distraction the situation has caused the fans and my family i'm sorry for the lapses in judgment from people that were not me and i accept full responsibility come on man

56:26Ranjan Roy:come on what are you doing that's worse than serious demand for concern that is worse he's Kawhi he won when was it the Raptors yeah no that was a good championship that one shot that one shot he's forever he can say whatever he wants especially if you're a Toronto fan yeah alright I think it's time for us to go so Ranjan, good to see you again. Have a great Labor Day weekend. I'll be reading my meter report. Hope you have a good weekend too. Oxytocin levels through the charts. Blessing, if you're editing this, can you just end with a Steve Ballmer developers chant? Alright everybody thank you for watching and listening and we'll see you next time on Big Technology Podcast

57:18Developers, developers, developers, developers, developers, developers, developers, developers Developers, developers, developers, developers, developers, developers, developers, developers, developers. Yes!