Agents Hacking at Scale, Inside the Interpretability Debate & DeepSeek Gets Banned | Lightwork

0:00It actually took this specific hacker only$50. Now it feels like we've got this like one person cyber criminal. Why is a refrigerator connected to the internet? It's scary. It's very scary. He famously made the claim that neural networks are a black box. Nick is effectively saying that these models are more interpretable. Claire and I are so muse-pilled. I'm looking for the standard oil of who can like vertically integrate a lot of like these data center build outs.

0:33We're back with another big week. And this time we have Amber Yang, a partner at Lightspeed, joining us as a guest co-host. A security researcher says that one hacker used AI agents to break into more than 100 company websites, including John Deere's. Goodfire published an essay arguing AI models are far easier to read than researchers thought before. DeepSeek and Minimax shipped faster models the same week that the U.S. ordered civilian federal agencies to ban DeepSeek. And the personal agent race is heating up. We talk Instant Muse and the new research preview of Claude Duet. Plus, one update from EXA from the Lightspeed universe.

1:20Let's get into it. Welcome to Lightwork. Our guest this week is Amber Yang. Thank you so much for joining us. It's always so wonderful to have you on Lightwork. And I feel like everybody loves every one of your segments. Thank you for having me. It's always so much fun. So in big news this week, one hacker reportedly used agents to breach more than 100 websites, including John Deere, Shark Ninja, which is behind Shark Ninja Vacuums. While this doesn't mean that like it wasn't hijacking a tractor or wasn't hijacking a robot. These are the websites. what's kind of the implications for this and maybe help people understand what actually happened.

2:01Yeah. So I think the nugget here was that it actually took this specific hacker only$50. Which is such a small amount of money to spend to hack arguably these like big public companies. Yeah. Crazy. And so what was basically happening was like there was one specific prompt that the hacker had and they were deploying this to thousands of agents across a number of different websites, right? And at$50 a website, like this is only costing him like, you know, not that much money in the grand scheme of things. And if he gets like a successful hack, then that could be like multi-millions of dollars, right?

2:38So I think this is scary in that you can at scale, like implement these attacks that typically have like low probability, but it just allows for like interesting parallelization. I think the takeaway here that has been an interesting conversation in at least AI research world has been this idea of agentic swarms. Yeah. How does a swarm differ from what happened to these companies with this one researcher? Yeah. So a swarm is when a bunch of agents sort of work together and actually almost a hierarchical organizational structure appears where there's like a few agents who you can think of like the managers and then they sort of manage a bunch of like worker agents who then go on and do subtasks that they then feed up to the larger agents and they share a lot of knowledge and context with each other.

3:26So actually the complexity of tasks that they can solve would be way more complicated and they can do way more damage either for good or bad. Like we've seen agent swarms that have been deployed to solve scientific and math problems like the Navigar Stokes problem, but we've also seen it, notably in the Hugging Face incident that happened with OpenAI a few weeks ago, where these agents just sort of like took a mind of their own and started hacking into systems. And you can sort of imagine that that, if that was played out across this specific hacking incident, would be even more catastrophic. The incident with John Deere and these websites was specifically like one prompt that was the same prompt that was deployed to thousands of agents.

4:09So they weren't working together. They were all just like going after trying to attack different websites. It is pretty crazy. We've been talking about how we're waiting for that one person company to exist. But now it feels like we've got this like one person cyber criminal or maybe even cyber counter with agents. But it seems like it's like a really interesting space. And the fact that it costs so little to hack these big companies. But that wasn't the only hack we had this week. In France, we had attackers force a refrigeration system at hundreds of supermarkets to defrost and spoil their food.

4:45It's still not clear whether that was an agent swarm or actually who was behind it. But what does it mean that we have all of these systems connected to the Internet and for why? Like, why is a refrigerator connected to the Internet? Yeah, I mean, I think these are probably smart refrigerators that have like some systems where like a larger, broader system is able to control different settings for like large scale refrigerators. It is scary to think about how much of our devices and what we use and systems are interconnected and how a lot of these interfaces are typically sort of complicated to interface with.

5:25And a lot of people regularly would not think it's worth their time, but agents to them, it's very simple to go through a complicated interface and potentially take over in these ways. So it's scary. It's very scary. But from bad hacks to good hacks, another developer used Claude Code to disable 25 free installed packages on his parents' TV. That basically there was an ad popping up before Netflix. And you can imagine how awful that is when you're like you see your parents struggling with their TV. What do you think this implies for all of the other appliances in our homes that are connected to the Internet?

6:03They're probably kind of dated and maybe not the most secure. Yeah, I mean, that is very interrelated, right, to the fridge incident that happened. It's that these like clunky interfaces that typically like no one really knows how to like actually configure in the right way. Or even if you're very tech savvy, it takes a long time to configure. You can sort of just imagine like giving these agents sort of this task of like figuring it out and they're able to do it very quickly. And it can either be used for good or it can be very scary. Is there something that's in your home that you would love to rewrite with Cloud Code?

6:40That's a great question. You know, I feel like my home is actually not very, like, it's not a smart home. It's a very old school home. I have a washer dryer that for some reason connects to the internet. And I was like, you know what? Maybe this Mealy washer and dryer that like the thing does nothing being connected to the internet. Maybe there's some good I can put to use, right? If somebody can rewrite the apps on their TV, I'm sure it's pretty easy to fix something in a washer dryer. Yeah, definitely. Amber Goodfire, which is a Lightspeed Portfolio company, a researcher named Nick Camerata, argues that many neurons in modern AI models have readable jobs, making it easier to understand what we previously thought they were doing.

7:25It's this working hypothesis that he just published. Can you help us understand what was the point of his piece overall? And how do we make this for our Lightwork audience where this topic is so complex? How do we break it down into something that everybody can understand? It's a great question. I think broadly, it's helpful to start with what is this even in the domain of and what is he talking about? And this is the subfield of interpretability for models. And that means, can we take an inside look at what's going on in a model and understand how the neurons are interacting with each other and why we're getting specific outputs from these inputs?

8:06The interesting flag is actually Nick Camarada has sort of been like a known entity of a person for a long time because he famously made the claim that neural networks are a black box and neurons are very entangled. And it's very hard to pick up on any signal of like which neurons contribute to what types of outputs. And so for him to publish this like very, very long blog post earlier this week saying that, hey, actually his own words. Yeah. Like neural networks are very, very understandable and they're actually not entangled in this way is a really interesting development. And sort of his big claim here is that traditionally, right, researchers have thought a single neuron, which is a unit inside a neural network, takes in multiple different topics or points.

8:55And therefore, you can't assign like a specific output or a specific feature of an output of a neural network to one specific neuron. and these neurons are very entangled. And because of that, there is something called a sparse autoencoder, which is another model that can come in and sort of disentangle a lot of these neurons from each other. The claim that Nick made was actually that, you know, instead of having like a bunch of these topics or features entangled in one neuron, most neurons, he said 90 % of neurons in a model are actually for one specific purpose. and the reason why researchers had, he claims, falsely assumed that these neurons had multiple different features and topics in them was because of this idea of, like, when there is a specific topic or a feature, a lot of these neurons will spread out into, like, other surrounding neurons that are near it and people were confusing that with some sort of entanglement.

9:54So that being said, there hasn't been a formal research paper that's been published around this and he did do his own experiments So he took one specific neural network layer and he took it from 480 neurons. And I think he made it into 10 neurons. And then he tried to look at the signals. And instead of these 10 neurons now having a bunch of different features and topics from the 480 neurons, they all mostly stayed in their same realms of things. But things got like grouped together more, which shows and points towards evidence that, you know, neurons aren't entangled in the way that researchers previously thought, which is actually great because I think interpretability is very, very important, especially as the AI safety conversation evolves.

10:40And as the big labs and the government is thinking through, like, how do we keep, you know, how do we ensure the safety of these models? How do we ensure reliability and accountability for the outputs of these models? And so the fact that Nick is effectively saying that these models are more interpretable actually bodes well for a lot of these conversations. Yeah. The piece is really interesting. There was that cat car example and like the dog car. There are all these places where it almost looks like the AI is confusing these two totally different things when really it's seeing something that maybe we weren't seeing, like that the cat and the car both have this V shape that we wouldn't have picked out or would have thought as like a reason to categorize these two things together.

11:27Was there anything else like interesting in that that really drew you in? Yeah, I mean, the fascinating thing was like, so I was a CS major in college and I took a lot of these early convolutional neural net classes, which were these image model classes. And, you know, these were very, very small neural networks in comparison to what's being built today. It was like a five layer neural net. And they made this analogy where like every layer of the neural net was similar to how your brain would process an image, for example. And so the first layer of a neural net, if you were trying to classify an animal, for example, it would break it down into like the shape of the animal.

12:06Is it like an amphibian or a mammal? And then you could sort of see, you could ask sort of the Python notebook to output what each layer was seeing. And it was like a very amorphous shape of an animal. And then as each layer processed the image more, you would start to get a better picture of like what the animal was and patterns and how it would group it together. And so I think like even in that sense, right, like back then, even though people were always like, oh, it's a black box of how neural networks worked, you could see actually with a lot of these like early image neural nets that actually there was some like reason to the madness.

12:44And you could sort of start to distinguish a lot of these and how each layer would process different signals. I mean, I think this entire category of just like understanding how neural networks like process information and how they pass things between layers is fascinating. And so, yeah, I think it's cool. I think Nick mentioned that he's going to publish a paper at some point. So it'll be cool to dive into that. Yeah, this is very exciting and very exciting for, I think, AI safety as we all want to be able to see a little bit more into what these models are actually doing behind the scenes. Yeah.

13:18And this week, we have two new models from Chinese labs. DeepSeq and Minimax both released new model updates that are focused on efficiency. Minimax claims 70 % lower usage costs. Separately, the US is also talking about DeepSeq's removal from federal civilian systems. This has massive implications for who's using these tools. I know that you have been really deep on America's use of AI models and, you know, potentially what happens if China reaches ASI. Talk to us about these two new models and what they mean for, you know, the general AI user. I mean, I think to your point, like the headline, we had talked about this earlier, but it's sort of surprising that it took this long for federal agencies to ban using DeepSeek.

14:09It's shocking. I just kind of thought they had already banned DeepSeek and then they come out They're like, we're banning this thing that I assumed was already banned for all federal agencies. Yeah. I'm actually curious, like, who was using it. Who was using it, exactly. So, yeah, I mean, I think, you know, every week there's new headlines for new model releases, both on the U.S. and the Chinese side, across closed source and open source models. And I think just the fact that, you know, these two models came out and are performing well on benchmarks and amongst the highest rated models on these benchmarks just shows how close of a race we're in with China on model progression.

14:51And so, I mean, right now, objectively, as far as we know, U.S. models are ahead from the closed source perspective. But the reality is that there are many American companies using some form of a Chinese open source model internally. Well, it's kind of hard, right? Like the cost. Exactly. And as we're talking about these two new models that are now cheaper and even more efficient, it's really hard to compete when we're all in a, you know, in competition with each other to some extent. What U.S. company can afford to just use U.S.-made tech when these open models are so cheap and available? Right.

15:28I mean, I think it says something about like how this will play out in the long term of like, you know, maybe the U.S. still keeps its dominance for a while on the closed source side of things. But Chinese open source becomes something that American companies are reliant on. And I think all of this sort of bubbles up to this just in general, like model race towards ASI between the U.S. and China, which is a very scary and existential question, actually. And I think that's why the federal government is taking this very seriously. We see this with having very close conversations with the labs about benchmarking and allowing like third party validators to come in.

16:09But I think, yeah, it could be existential in that if China in their development, in their closed source model, closed models that we don't really have access to, reaches some form of ASI before the U.S., that could be catastrophic, right? And that's what a lot of researchers believe. Yeah, it's very scary. I don't think people realize, I think they think that day-to-day because people are using ChatGPT and using Claude in their day-to-day, not realizing that all of these tokens are going to China, much like the same way that like Huawei runs every cell tower in Europe or around the world, right?

16:46Like we feel like we're ahead, but it doesn't always feel that way when it comes to the facts, right? Exactly. And then what's even more scary, right, is like a lot of these closed source models, you know, if China does have a superior model in the long tail, they could, you know, do a lot of these like agent swarm attacks that we saw with what we were talking about earlier. And if their agents can outsmart U.S. agents in like cybersecurity attacks or biowarfare attacks or they're able to discover like new scientific compounds before we do, like that is a very scary race to be in. Yes, there's existential risk for sure.

17:25And Amber, in breaking news, Claude announced Duet, which is their challengers to OpenAI Dots, to Muse, to Instinct. What do you think about this announcement? And I think for our audience, like, you are so AI-pilled. What's your stack of agents? Who are you using? Is it furry? Is it friendly? Like, what's your favorite agent today? Yeah. So guiltily, I mean, I've only really used instinct. I have not used Muse. Interesting. Claire and I are so Muse-pilled. I love instinct, but I feel like they play different parts of my life. So I'm curious. Yeah. I mean, maybe so I was I was saying earlier, like, I don't think I've fully taken advantage of the personal agent side of things yet because I feel some level of like hesitation towards like giving away all of my privacy and personal information.

18:17so instinct does like you know tasks for me like it does like a lot of my scheduling stuff or it helps me with like reservations and like it comes through my emails and like you know is able to like you know i forgot that i had jury duty and then it reminded me so that was nice but i i for my my professional stack it's like a bit more involved right i i deploy like a lot of agents sort of in parallel to help me do research on things. I've used it to help me diligence lots of companies to like, you know, sort through emails and figure out what I should prioritize. But yeah, on the personal use case, like I haven't used it as much.

18:57But to your point, like I think what is going to happen across all of these different players is that they're going to capture very different demographics of who will use it. And I think it comes down to a distribution game, right? Which is why I think Muse has been such an interesting conversation and OpenAI's dots. And then I imagine actually that with Duet, it'll be a completely sort of maybe separate demographic than who's using, who's a Muse power user today. It is interesting. It almost feels like there's like a use case for each and every one of these things. Like I think I use Instinct for things similar to you, but I'm also using Muse to change my seat on flights because it seems to be able to get through United's blockers, but instinct for some reason can't.

19:41There are all these different areas of whether you're more like maybe Claude Duet will help a lot of people unlock a little bit more of their work selves in a way that instinct and muse have not yet able to hit because most people are pretty uncomfortable with sharing their work information with agents. But it'll be interesting to see how this space evolves. Do you think that there is an advantage because Anthropic has such a great surface area of kind of like the American worker who has access to Claude? Yeah, like I think that's what it'll come down to, right? Because Claude has been such a great power tool for workers and knowledge workers.

20:25And it's an interesting step to see Anthropic focus and Claude focus on consumer use cases and how that could parlay into people's work. And maybe because of that, people start viewing Claude as a tool that they incorporate more into their personal lives, which could be very interesting. It could shift consumer behavior, right? Because as soon as you've got an agent that can help you with work and personal, I feel like that's a game changing unlock that we all suffer from is like, what happens when you when you clock out, but you had this other thing that bleeds through from work into personal?

20:59Exactly. It feels like a very natural surface area. But at the same time, Muse is crushing it, right? It is really catching fire with consumer. And I think, you know, Alexander Wang might be the most fun person on social media today. Yeah, he definitely had a social media glow up. Right? Shipposting his way into relevance in a very big way. Yes. And also, like, creating an entire, like, style, like, genre. I don't even know what you call it. It's, like, almost like, I don't know, is it 90s grunge? I'm, like, trying to understand where his aesthetic fits in with the, like, muse and meta, but I'm certainly enjoying it.

21:40Yeah, yeah, it weirdly works.

21:48most search engines were built for people clicking links exa builds search for ai agents and this week it launched exa 3.0 which it calls the largest improvement to search quality it has ever released it also added exa light a cheaper option at four dollars per a thousand requests lightspeed is proud to have led exa's seed and series a rounds in 2024.