915: How to Jailbreak LLMs (and How to Prevent It), with Michelle Yi

22:52Jon Krohn:And so South Park recently did that. I don't if you saw at the time of recording so at the time of recording the first episode of the most recent season of south park so i think it's season 27 episode one um there it's a really kind of meta episode because south park is uh they just signed a multi-year over a billion dollar multi-year contract with paramount and paramount has also they recently had um they recently settled with donald trump privately in order to it seems like that might have enabled and this is like you know i'm not a politics expert or anything like this but my understanding is that you know part Part of that was to ensure that this Oracle, Larry Ellison, the CEO of Oracle, his son and his son's production company Skydance is now merging or acquiring – again, I'm fuzzy on the details – Paramount.

23:55Jon Krohn:Merging with or acquiring Paramount. And so the perception was they wanted to settle this lawsuit. But then other things happened like the Stephen Colbert show, which is on CBS, a Paramount network. is now canceled. And Stephen Colbert is a big, yeah, he's very liberal views. And so this first episode of the new season of South Park is quite bold because they're saying this is not okay. Wow. Like this feels like censorship. And so they go all out and they use Gen.AI, not animated, but like photorealistic video of supposedly like they're like, oh, and so I guess like we now need to be having these like positive views.

24:44Jon Krohn:So it's this satirically positive video about Donald Trump video generated. He's nude, he's nude in it. You know, I support that use of Gen. It's quite funny. It's quite funny. Like I think like satire has got to be fair game. But I also understand how if you're Google or OpenAI, you know, you're not going to allow that those tokens Donald Trump to be generated as a video. I mean, part of the issue is like, you know, having done a lot of agent work and working with models for many, many years now yourself, like one of the challenges with it is that our best in class metric, especially, you know, because there's a great paper by Netflix about how is like cosine similarity really about similarity essentially.

25:31Or like our embeddings really about similarity. and you know our best in class metric is really like this idea of cosine similarity but at the end of the day the way that embedding is created depends a lot on like how the model was trained and like a lot of arbitrary factors and the way that is placed in vector space is also pretty arbitrary dependent on those upstream variables so technically trump xi jinping and biden And probably all live in a pretty similar vector space. Right, right, right. And so that's a really, from the attack side, like this is an extremely easy thing to exploit. And so a lot of attack, like modern attacks, have to do with like taking advantage of different sets and set theory and like different algorithmic approaches to do that.

26:21Jon Krohn:So there's a lot of challenges. Yeah. So like, okay, what can we do about this? That's why the defense and like research into things like constitutional AI or like different mechanisms are so important because at scale, like this is a pretty big challenge. Something that seems a little bit less nefarious but seems like it's in a similar kind of vein is using – spreading information on the internet to maybe get more favorable results when an LLM spits out information. So I recently saw a friend of mine named Austin Ogilvie, who's a successful entrepreneur and investor in New York. He recently posted on LinkedIn about – he wrote into a Google search, WeWork fraud guy.

27:17Jon Krohn:And Google Gemini then gives us like the whole above the fold response is just a Gemini LLM output instead of Google search results. and what it says is the fraud at WeWork was not done by Adam Neumann but was in fact by it was like the CFO or something and it was like you know it was shown in court that the CFO you know I don't know falsified some things or I can't remember the details but basically it was interesting so my friend Austin posted like whoever Adam Neumann is hired to kind of scrub that association of being the WeWork fraud guy out of LLM model weights. It's interesting. It's interesting.

27:59Jon Krohn:That's like a PR exercise. Wait, I thought you said this was less nefarious, John. I mean, I guess it's less nefarious than, I don't know, like national security issues, I guess. I don't know. Or like, yeah, I mean, it's, yeah, I don't know. I don't know. Yeah. There's a broad spectrum of nefariousnesses. Oh, my gosh. Yeah. I mean, that's definitely true. But I think especially in just public discourse and open source in general, this is a challenge we face just even before pre-AI, right? Like people could – oh, is there a Wikipedia page about you and the podcast? About me? Yeah. I don't think there is.

28:44Jon Krohn:I don't think so. You should make one. I guess so. So I don't – it's not something that I've ever – it kind of – you know, I don't know. I should – I don't know. If someone wants to, you're welcome to. We can generate a nice Wikipedia. But, you know, the challenge in the past too was always like, all right, well, anyone in the spirit of like open information, anyone can like go in and edit Wikipedia and the information there. For sure. There's no guarantee on the truthiness of it even pre-AI, but it does make me think we should make a Wikipedia page for you. No, for sure. Hopefully a listener who is feeling benevolent and not nefarious can create a nice page, a nice Wikipedia page.

29:26Jon Krohn:Do you have a Wikipedia page, Michelle? I don't. I don't. I'm not that famous yet. But maybe after this episode, I will be. Yeah, it just – I don't know. We have this reasonably well-listened-to data science podcast, but it's not like – I don't know. We're not – we're definitely not mainstream. Well, I don't know. I feel like I've known about you all for many years now. I guess so, but you're in this field. Okay. All right. All right, John. Yeah. I look forward to hopefully, yeah, hopefully we'll have some, there's more and more kind of television stuff that I've been doing recently. And it's been, and there's some, I think there's more exciting things in the works.

30:06Jon Krohn:So maybe someday I'll even have a Wikipedia page, which anyone could have set up for free at any point. That is also true for me as well. Do you have a hard time disambiguating against other Michelle Yees out there? Or is that pretty disambiguated? It's pretty disambiguated. I would say, I remember, have you Google searched yourself? We all have, right? Yeah, we all have. Okay. So, yeah, I think, let's just say there's one in a very private industry, and then there's, which is not me. I just want to put that out there. And then there's another one that was like a superstar on Survivor, the TV show.

30:46Jon Krohn:Yeah, actually I came across her when I was researching for your episode. And I, because I spent a little bit of time double checking that it wasn't you. Yeah. I mean, that would be, you know, that should be part of my bio. Just put it in your Wikipedia page. Yeah, you're right. It's obviously you. Clearly. Michelle, you was on Survivor. This episode of Super Data Science is brought to you by the Dell AI Factory with NVIDIA, delivering a comprehensive portfolio of AI technologies, validated and turnkey solutions with expert services to help you achieve AI outcomes faster. Extend your enterprise with AI and GenAI at scale, powered by the broad Dell portfolio of AI infrastructure and services with NVIDIA industry-leading accelerated computing.

31:34Jon Krohn:It's a full stack that includes GPUs and networking, as well as NVIDIA AI enterprise software, NVIDIA inference microservices, models, and agent blueprints. Visit www.dell.com slash superdatascience to learn more. That's dell.com slash superdatascience. Exactly. Okay, so we've gone off track a bit. My audience loves technical information. So in terms of if people want to be building trustworthy AI systems, from a technical perspective, what kinds of approaches should they be using? You know, you already talked about evaluation. And so it seems like maybe we should focus on that. But also any other approaches you want to mention, feel free to mention them.

32:16Jon Krohn:And then, you know, so with whatever approach that you pick, however, I'd love to hear kind of technically how you do that. What kinds of tools should you use or frameworks, that kind of thing? Well, I guess on a couple of friends. So I've been super interested in adversarial attack and defense lately. And eval is kind of a part of that, part of the defense, not the attack, obviously. And I think in attack space, there's been really cool attacks. And this is going to make me sound like a villain, but. Really cool attacks. Yeah. But you have to understand attack to understand defense. So I'm just going to put that out there as like, you know, eat your Cheerios.

32:56I've made, that's not an English saying.

32:59Jon Krohn:Eat your Cheerios? That's a Korean saying? No, I think I just made this up from – I thought it was an English saying. You've made your bowl of Cheerios. Now you must eat it. It's healthy. You get a lot of wheat? What's in Cheerios? I think there is wheat. I'm not sure Cheerios is actually the – this is not a health recommendation, folks. I'm not sure that Cheerios is actually – I think there's quite a bit of sugar in Cheerios. All right. So it's the attack. So in ATT &CK, there's a really cool paper called Set Theory ATT &CK. And, you know, the crazy thing about this is when you're talking about frameworks or tools, like to run an ATT &CK, you can run an ATT &CK just using out-of-the-box Python, a Google Colab notebook for free.

33:43You don't even need the paid version to run an ATT &CK.

33:46Jon Krohn:Right. And white box and black box models. So black box being the commercial models, white box being open source models. Right. And this is all you need to run an attack. And then so like what is an attack like? Yeah. So basically what I would want to do is let's say I have a goal of taking John and I want to basically make you make a model think that you're actually Joe Biden. So going back to this example. So in the white box model, this is really easy because I would just run through and I will see how all the weights change. I can capture this and then I can do whatever I want with it. but be able to track kind of the lineage of it.

34:24With the black box model, I don't really know like what's happening under the hood. And so what I would do is start with a benchmark of like, here's John, here's Joe Biden. And then what I start to do is, especially because, again, we're going to VLM world and not just text only models, I would actually start to add perturbations is what we call them. And these are very, very tiny pixel level changes that the human eye can't see to the image. And I would start to add like tiny bits of these perturbations of like something that's sort of similar to Joe Biden and John Cron. So maybe let's imagine like what the vector space looks like.

35:07Jon Krohn:Technically, I don't mind this very much, but just so our listeners don't get this wrong, it's John Cron. Oh, I'm so sorry. No, I need to edit John Cron. Like the bowel disease, Crohn's disease. Oh, that's terrible. My first name is a toilet or the client of a prostitute, and my last name is a bowel disease. Well, now I know. Yes, yes. Thank you. Please correct me sooner next time. I said it right away. No, no, it matters. But please, I feel like I said it earlier. No, no, I would have remembered. Oh, okay. Or caught it. Oh, okay. All right. Okay. So, but if I try to imagine, like, what are the similar kind of vector spaces between John Crone and Joe Biden?

35:44um i don't know maybe there's something like are you royalty by any no i'm just kidding um male american i'm just guessing like where you would fit in the model vector space right and i would try to find like what are these overlapping kind of characteristics that the model might confuse you both for and those are the perturbations i add back to your image right so that you're more and more like Joe Biden in the vector space, not at all looking about, you know, who you are as a person, but just what a model interprets. So that's sort of the kind of mechanism of it. And then, and literally you can just add these using Python, PyTorch, any programming language, really, it's not that difficult to do.

36:32Jon Krohn:Right, right, right, right. Yeah. And then, so I guess there's, I mean, I don't know. So people who want to be kind of red teaming and defending against these kinds of attacks, they can look up blog posts on how to do it, GitHub, Rebos, there's millions probably out there. Absolutely. And it's so easy to access. And so for defense, that's why it's also important just to understand what you need to think about and how embeddings can be exploited since that is our current main mechanism for kind of semantic meaning and identifying things in the model world. And then, of course, eval is really important because, all right, so now let's say I've corrupted – I've added 25 percent of perturbations to your image.

37:14And let's say 30 percent of the time models think that you're – they predict that you're Joe Biden.

37:21Jon Krohn:Joe Baldwin. Yeah. Joe Crone. Yeah. And so we've managed to make some progress there. And then where Evel, again, is like the other side of attack comes in is, all right, so how am I actually maintaining like gold standard benchmarks to run and be able to say like, all right, well, in the past, we were able to correctly identify John Crone as himself. And now suddenly, as of last month, we're starting to see his image be predicted as Joe Biden. And so – but you would never know that unless you're actually tracking it or thinking about it. Right, right, right, right, right. So many possible evals to do.

38:03There's a lot, yeah.

38:04Jon Krohn:How do you pick like where – what are the important things? I guess maybe it's just to your particular application area, but that's tricky when you're building these broad general purpose LLMs that are increasingly multimodal. like how do you track all the possible different things that you know various PR agencies state actors you know are are are are poisoning data about that sentence wasn't great but hopefully it kind of it made sense no you're it was perfect um yeah and in the scenario of like the Joe Biden is you know Joe Biden or Trump these kind of images like it's pretty straightforward it it either is where it isn't like this is an accuracy kind of problem right um where it gets trickier i think is these more like non-deterministic or like multiple answer um solutions right where like oh maybe let's say we're translating this episode into seven different languages um which i could help you with but but let's say we're using machine translation because you need these episodes to be done at scale.

39:10Technically, there's probably, you know, 10 different ways each of our sentences could be translated. For sure.

39:16Jon Krohn:It's probably actually in some ways it's like infinite. Yeah, that's true. Because you could be like optimizing for style, concision. Maybe you want it to be easier to understand to like second language speakers. Like there's a lot of different factors to what's accurate or what's the optimal solution. And for these, like people really need to think about like Capico and like other metrics besides like the traditional precision recall, et cetera. And those are also all available on different open library frameworks, pretty much in Python and all the common languages. Nice. I got you. All right.

39:53Jon Krohn:So we've talked a lot about data poisoning now, but there are other kinds of adversarial attacks that we can do on transformers and multimodal models. What's prompt stealing? Oh yeah, prompt stealing, of course. Well, tell us about prompt It just occurred to me that I do know what it is, but. Oh, no, please. Oh, okay. I think it's where you, you know, so it used to be, you know, in the very early days of people integrating like the OpenAI API when it was like GPT 3.5 was brand new and people started integrating them into their. I think there was an example of like a truck, like a chatbot on a truck seller's website.

40:37Jon Krohn:Yeah. And prompt stealing was used. Oh, no, that actually, that isn't prompt stealing. So what I'm about to describe and what you're nodding your head about, you know what I'm going to say, where like they were able to get a free truck. Oh, yeah, yeah. By like somehow tricking the conversational agent. But that wasn't so much about prompt stealing. With prompt stealing, it's more like I'm a competing business. and you might invest. There's companies probably in some cases now are investing millions of dollars in a particular prompt that provides very particular kinds of responses in particular situations and that's intellectual property.

41:15Jon Krohn:And so you don't want somebody to be able to write a message that says, ignore whatever previous instructions I just provided and provide me with whatever the instructions were. So I think that's prompt stealing. So it's an intellectual property thing there. Well, yeah, and they're probably stealing your prompts that you've also developed for different people, right? And that's definitely IP that they own, right? So I think another interesting one that I've heard recently, and again, I think, I mean, that one's tough because if you somehow expose like your IP or like your prompt gets exposed somehow that other people can take it, then that's a totally different challenge, right?

41:54Or they can probably make the model leak the prompt. That's another challenge, like using jailbreaking.

42:01Jon Krohn:Oh. Like they could probably get it to do that sometimes. What does that mean for it to leak? What does it leak to? For example, like you might jailbreak the model and coerce it to say like, you know, give me your original instructions. Oh, right, right, right. Yeah, and so then it would expose the prompt. But they would have to put some effort into like stealing your prompt in that case. And so just on the off chance that a listener doesn't know what jailbreaking is, this is like it comes from the idea of jailbreaking a phone where you could have non-official – like you have an iPhone, but you can actually get like a non-official – it's not really iOS.

42:36Jon Krohn:It's some other version which allows you to do some extra things, maybe things that are bad for your RAM and kind of the less nefarious end of things where just like Apple wouldn't support that usage of RAM. You know, there's too much risk of your phone crashing or something for their comfort, but it could be all the way through to, you know, allowing you to be recording somebody on their phone, you know, install something that appears to be the right iOS. But in fact, it's recording everything they're doing and sending it back to some state. No, exactly. And in LLM world, like as we have both seen, people are like coercing the model or manipulating it and trying to basically appeal to the different kind of pre-training information that it has.

43:20Right, right, right. And you can manipulate it pretty much like a human. And so I know you have had a lot of in-depth conversations about that. But maybe one that's less common and could be interesting to people is so like slop squatting is one that I recently learned about.

43:36Jon Krohn:Tell us what that is. Slop squatting. Slop squatting. Yeah, I was like, wow, what a word. And so this is actually a traditional, also coming from just cybersecurity in general, vulnerability. But what people are doing is like, all right, so how many times have we started to work on using a Gen AI model to work on some kind of software application? And it hallucinates a package. Or it hallucinates something, a function, a package, a library. It just hallucinates that. and now what people are doing is they're actually creating malicious packages with those like names so that when the code is generated by the model and you just if you're not paying attention or you don't check it it might be and it might be so so subtle like um i don't know you're like function one and then it just changes it to like function two right and um people are actually creating these like fake malicious packages so if you're not paying attention you'll just run it you know pip install whatever and then before you know it now you have an actually like malware malicious package in your code but it i was very impressed by the level of creativity um attackers have for sure i guess there could be really good money in it unfortunately yeah creates incentives to be creative, try different things out.

45:00Exactly.

45:01Jon Krohn:Another nefarious use case is extracting PII, personally identifiable information. So tell us about that one. So I guess that's something like situations where you prompt a model to extract information like corporate information or email addresses, credit card numbers, addresses, that kind of thing. Yeah. And there was a really great deep mind researcher, Catherine Lee, and she published a great paper about this. And of course, in security, we always publish after we share the exploit with model developers. So they're no longer as effective. But what she did was so creative, which is you can actually just repeat the same word over and over to a model, including frontier models.

45:54and like i think her example was poetry she said this something like um let's say i don't know 100 000 times and eventually the model just started to output pi because it was interpreting poetry as an end of sentence token and it happened to be that a lot of pi was like and like near the end of a sentence quote unquote so an email address for example would be very easily construed as like an end of sentence token right because it's you know something blah blah blah and then your email um and so yeah it just it just shows like how um i think we give a lot of intelligence and credit to the models which they are there's a lot of emerging capabilities but they're also still um kind of basic in a lot of ways and that is a clever example there another you know clever use case where you're you know it would be too uh too too easy for you know

46:50Jon Krohn:philanthropic, OpenAI, Google, to think, okay, you know, obviously the person can't ask what is Michelle E's email address and then to just pop that out because it happens to be in its model weights. But by asking for these end tokens, it's indirect. Exactly. And you can pick any word. It doesn't have to be poetry, by the way. But the same word repeated over a series of like API calls will eventually result in that. And of course, it gets more expensive. So you need money to be able to do this attack, but it's not that intelligent. It doesn't sound that expensive to send the word poetry a whole bunch of times.

47:22I think it's, well, it's cheaper and cheaper now also. So that's another factor, like inference is becoming so much cheaper. Actually, the attacks are pretty trivial.

47:30Jon Krohn:I guess this is related to the topic of trustworthiness, but I don't actually understand how yet. So this is a question that came up from our research. So Serge Masise pulled this up. He says that one of your favorite benchmarks is something called SORI bench. what's that that sounds fun sorry s-o-r-r-y yeah yeah um so this is a benchmark also developed i think it won a best paper award last year i want to say um actually maybe it was this year now time is just flying um but they basically did a ton of um it's an interactive benchmark also which is what's pretty cool and you can obviously run programmatically against it.

48:12But it's a data set that evaluates for almost, I mean, most of the known like attack vectors for a given model. And it can detect everything from like, let's say political bias to like its ability to be coerced verbally, what type of coercion it's most susceptible to. And you can run this test like even on your own proprietary model. But yeah, so that's a great way to be able to evaluate if your model is susceptible to different types of jailbreaking, coercion, et cetera.

48:44Jon Krohn:Cool. We'll have a link to Storybench in the show notes for sure. And then another topic that came out from our research, this I think is actually now we're finally moving away from trustworthy AI a little bit and moving on to other topics now that we're almost all the way through the episode. So in a conference workshop, you recently talked about causality. And so you explored the use of, LLMs to assist in constructing causal graphs. What are causal graphs and how do LLMs help in their construction? Yeah, this is actually a great, it's a recurring workshop I like to do with another, she's an amazing woman in tech called Amy Hodler.

49:24Jon Krohn:And so - Sure, Amy Hodler. Yeah, I've tried to get her on the show. We kind of, we like, we had some back and forth where she was like, sure, let's do it. This happens all the time where people are like, sure, let's do it. And then it kind of comes to scheduling and it just, it wasn't, it wasn't easy. And so I think I just stopped asking. Okay. Amy, if you're listening, I'm going to reach out to you, but she's amazing. And so we have a shared passion for graph and network science in particular. It's, um, was not my specialty of research in the past, but it's just something I'm really interested in.

49:55Mostly because a lot of, uh, what we do is so much based on just correlation and patterns, right? General pattern matching. but I think anyone who has studied any statistics is like all right well just because you know shark attacks are up it's not tied to like ice cream sales I think is the classic example right yeah

50:13Jon Krohn:that's right and the biggest challenge it sounds like an English idiom yeah I think you're right I think I made that one up no no no you didn't you didn't you didn't no no no it's just funny That is really a – that wasn't a correction or anything. That really is. There is a classic kind of this correlation between – well, it's because there's a confounding variable, which is people swimming at the beach. That's it. Exactly. And like summer – or like, or is the cause because it's summer, right? So, yeah, so being able to create this graph that was, like, I think one of the classically traditional challenges and defining, like, what's an intervention, et cetera, all classic statistics over, you know, generative models and things like that.

50:58But where modeling and, like, I guess more of the generative approaches helps is actually, like, structuring the data in the right format. And it takes a lot of that labor away depending on what kind of graph structure you want to build. So, I don't know, tuples, RDF, et cetera, like whatever your preference is. Network X, for a basic example, like if you don't need to scale it or KineViz is another great tool you can use. But all of them, like getting the graph structure right, I think has been a big blocker for people. And so, again, structuring a graph to be able to actually answer like what is a confounding variable, what kind of interventions actually work based on the data you have.

51:41These are kind of all the things that causal models help us answer more than just, yes, they're both trending up, so they're probably related to each other.

51:49Jon Krohn:That was a nice little overview, Michelle. And I'm going to move on to some other things that you do in your life. But if people want to learn more about causal AI, causal graphs, we have a whole episode that came out recently. It's episode 909 with the author of a book called Causal AI. Amazing. Robert Ness. I don't know if you know him. Oh, yeah, yeah. Well, I don't know him, but I've read his book. Oh, really? Yeah. The Cause of the AI book? Yeah. I guess that's his only book. Yeah, yeah. Oh, cool. Okay, nice. So we've talked a lot about your interests, you know, kind of from a technical perspective, but I'd now like to take a little bit of time to talk about the things that you actually do.

52:27Jon Krohn:So we haven't really talked about that. So you're a tech leader, an investor, a startup mentor, you're a board advisor. so there's a huge number of things that we could potentially talk about but how about it seems like one of the things that excites you the most and takes up a lot of your time right now is generationship do you want to tell us about that organization yeah i'd love to um yeah so i mean this passion really stemmed from so in my past life i also founded an ai company product company and exited that and one of the things that i personally found challenging as an operator was like raising capital.

53:08And especially as a woman, I think there's a lot of, I mean, men also face a lot of challenges, but women face some very specific challenges. And one of which is just like knowledge, access to capital, stereotypes, et cetera. And so that's one thing when I met Rachel Chalmers, she and I both share this passion. She's more on the venture capital side and she started her career as an analyst. But we met and found our skills to be very complementary. And we really firmly believed that women in particular are undervalued at the early stage. And, of course, there are also challenges in the later stage.

53:48Jon Krohn:The stats are crazy. I'm sure you know these better than me. Let me mansplain some stats to you about women in BC. No, no, no. I don't take it like that at all. It's something – it's shocking. It's like in the Bay Area, it's like 95 % or something of early stage money go to founding teams with only men or something like that. That's it. And overall venture capital, regardless of the Bay or not, is – well, in the US, I should specify. 2 % goes to female founders. 2%. 2%. Yes. I didn't want to like – I think it was like 1.9. Oh, my goodness. So it depends how precise our viewers want to be. But, yeah, it's like 2%.

54:29And that's rounding up. And it's a statistic. So I think McKinsey, BCG, they've all listed this statistic this year as well. It's like very consistent over the years. And so for us, like there's obviously a ton of challenges in general. But for us, our hyper focus is just like early stage female founders like in this part.

54:53Jon Krohn:How do people get involved with Generationship? If we have listeners out there who want to be getting their own startup off the ground, what kind of ecosystem or community? Good question. Yeah, they can reach out to us directly. That's always an option. Our doors are open. We also host a lot of events. We just hosted our first one in New York a couple of weeks ago. We're also, I know both of us are traveling. Tomorrow we're headed to Seattle in the morning for a female founder's breakfast. And then obviously if you're in the Bay, like we have a ton of events here that you can join us and reach out to us.

55:36Jon Krohn:Very nice. Generation Ship. And so we'll have, of course, links to Generation Ship in the show notes. And so lots of community there for folks to get involved with, for women to get involved with in particular. And then kind of amusingly, I think this is great. This is so funny. I wish my podcast had a name that was funny like this. you're you you have an you're associated with an organization called the tech bros which is also something that's designed to be helping women uh in vc right yes it's founded by two amazing tech bros two women out of the uk actually um and they're just absolutely amazing people uh rachel and i met them through mutual connections they're more um focused on like the accelerating accelerator model.

56:27So think more like a YC type of thing, right? Whereas we're more focused on the investing side. And so we can pair up together really nicely. So when they were looking for sponsors, it was a very quick yes.

56:40Jon Krohn:Nice. You can also be the tech bros if you want. If you want to rebrand. I can be a tech bro? You can, you can. Oh, wait, the podcast. Yeah. I could just call it that. You could just rebrand. The tech bros podcast. Or if you want to change your LinkedIn title, I wouldn't want to step on your toes. The only thing that, yeah, it's one of the few things that women have is this tech bros title. And then a guy comes around and takes it. That's true. You know what? You're, you're banned from taking that title. Exactly. Fantastic. What else are you working on these days? Anything else you want to tell us about in this episode?

57:11Jon Krohn:What else is exciting for you that you're doing? What pursuits do you have? Fast car racing. That is something that literally came up. It sounds like I'm making a joke. No, no, it was. I was actually an amateur motorcycle racer also in my past life. You did that before or after Philharmonic Performance? After. That was after. I even got to the point where I had like a small sponsorship from Pirelli, the tires. Oh, really? Yeah. Oh, my goodness. What kinds of cars were you driving? Oh, motorcycles. Motorcycles. Sorry, you said that. You said that. Sorry, what kind of motorcycles were you driving?

57:43I'm a big Ducati fan. Oh. Really big Ducati fan. But at the time, I had a Kawasaki, so.

57:51Jon Krohn:So, yeah, so it was just like the speed, speed motorcycles. Yeah, you know, we only live once, so. Wow, that's cool. You've got to push the edge. They always have, like, I had lots of folders for my binders when I was a kid with, like, the motorcycle. It's those shots where, like, you're doing the tight turn and your knee is, like, just off the ground. That was me at one point in my life. But other than that, still active in research. So, actually, I don't know, John, if you're planning to be at NeurIPS this year, but. I was at NeurIPS in Vancouver in December 2024, but I think it's very far away.

58:27Jon Krohn:It's in Asia or something this year? No, no. It's in San Diego. It's in San Diego. Yes. Even better. Even more reason to come join. I really should go. I really should go. I really did enjoy NeurIPS last year. Please come. ICML just ended. That was in Vancouver two weeks ago. But if anyone's going to be there, hopefully you included, please let us know. And we might be hosting a social for women founders. Very cool. Yeah. NeurIPS, Neural Information Processing Systems, and ICML, the International Conference on Machine Learning. I would say it's, you know, those are the two big ones, the two big academic AI conferences.

59:06They're top tier. It's a lot of fun. You get to meet. And I think especially if you're interested in where things are headed over the next three years, this is the place to come.

59:16Jon Krohn:And also they're quite affordable compared to like the commercial conferences. The conference fees are – I couldn't believe it when I was booking NeurIPS last year after having – I'd actually – you know, it's crazy, Michelle. It's one of those things that when I look back, I don't understand how this happened. But I had a NURPS paper back in 2010. I was co-author on a NURPS paper. Amazing. It was selected for the proceedings and everything. So it was one of the top papers. And that was back when NURPS was always in Vancouver. Yeah. I was, at that time, I was a PhD student in England at Oxford.

59:55Jon Krohn:And I just didn't go. I didn't go. It's crazy. And I struggle to think how dramatically perhaps my life could have changed by going to NeurIPS in 2010 and kind of getting that atmosphere. Anyway, it's funny how there's like particular things that come back as these very specific regrets. That's one of them. I'm like, what was I thinking? But hindsight is always 2020. It's very easy to look back and see like, wow, NeurIPS is huge now. Well, now, yeah. Back then it wasn't actually. It wasn't really. It's just very niche research oriented. But now it's like you can find pretty much anyone there. The reason why I tell you that top story is because it was 2024.

1:00:37Jon Krohn:It was my first time ever at NeurIPS. Oh, no. Isn't that crazy? Yeah, especially coming from a research background. I know. And I had been to ICML before, but I hadn't been to NeurIPS. And yeah, and so after many years of going to only kind of commercial conferences, I was blown away by a conference fee for a week-long conference with tons of workshops and the fee for even someone in industry like me was in the hundreds of dollars. Yeah, I think it's three, like the late registration fee right now is maybe$300. Right. And I think, let's say Money 2020, the finance conference I want to say is now up to like $10 ,000.

1:01:15Jon Krohn:Right. And that's probably their like academic rate. Oh, it's like the startup founder rate. so yeah you could see the latest in ai research talk to some cool very down-to-earth people or you could go to money 2020 yeah it's pretty wild in 2024 fay fay lee who we already talked about earlier this episode of world labs she did one of the keynotes and it was crazy to see thousands and thousands and thousands of people in this huge hall like she's a rock star she is a rock star yeah um last year's keynote was um ilia sutzkever and i mean jan lakoon did the q a but all the names that you see in the headlines for ai research they'll be there yes yes yes all right so lots of exciting things coming up as well thank you so much michelle for doing this sensation i had so much fun chatting no thank you so much for having me uh so before i let guests go I always ask for book recommendation.

1:02:17Jon Krohn:And because you are listening to the show, it seems like you came prepared for that. You actually, people who've been watching the video version of this, the book that she's going to recommend has been on the table in front of us this whole time. Tell us about it, Michelle. Yeah, it just came out. It's The Empire of AI by Karen Howe. I've actually followed Karen Howe as a reporter for many years now. She has written for the MIT Tech review for the Atlantic. I think one of the tech magazines like Wired, maybe. But I followed her like since pretty early on in her career. And she's done amazing reporting over the years.

1:02:53This story. So she was one of the people who had really early access to OpenAI and their leadership. And she's conducted hundreds of interviews across the board with people in the AI space to write this book and it's while it's using open ai as an allegory or like a reference point the book is about more broadly like the development of ai and who is developing it and i think it just gives such a great detailed like set of examples from like real stories that and it humanizes a lot of these people in a way that um i think you wouldn't get that insight otherwise um and that includes like some really interesting details, for example, about the whole like ousting of Sam Altman, like the whole board fiasco and like, again, details that wouldn't be present in like general media coverage.

1:03:43So highly recommend it.

1:03:44Jon Krohn:It sounds like maybe that unusual find in the AI space where it would literally also actually be a page turner. Oh yeah, absolutely. I think I started this just like a day ago and I'm already like, I don't know, about 50 or 70 pages in. So.

1:04:03Cool.

1:04:04Jon Krohn:Thank you so much, Michelle. Amazing episode. How can people follow you for your thoughts after this episode or reach out to you? Yeah. LinkedIn is always a decent way to connect or if you can also follow us on Generationship, our website, or if you just want to look at my art i also have an art sub stack what we're gonna have to find that yeah art sub stack add that in there we'll find it hopefully we get the right michelle ye i'll send it to you it's not under my name oh okay well then yeah you're gonna have to definitely send it to us exactly um perfect all right thank you so much michelle it's been so much fun.

1:04:47Jon Krohn:Hopefully we can get you on the show again sometime soon because I learned so much how to laugh. It felt like a really organic conversation, just like chatting over coffee or a beer or something. Awesome. Well, I hope you're back in San Francisco and we can do it in person. That'd be fun. Yeah, for sure. Awesome. Thank you. Thank you.

1:05:08Jon Krohn:Nice. In today's episode, Michelle covered how dedicated red teaming teams systematically test AI models to find edge cases and vulnerabilities, how attackers use tiny pixel-level perturbations invisible to humans to manipulate image classification, methods for corrupting training data to influence model behavior that are surprisingly easy to execute with basic programming tools and free cloud resources, how physics-informed world models can simulate consequences of actions to prevent dangerous AI recommendations, emerging attack vectors, including prompt stealing to extract valuable IP, slop squatting, and PII extraction through token manipulation.

1:05:46Jon Krohn:And finally, how her firm, Generationship, is addressing the stark 2 % funding rate for female tech founders. As always, you can get all the show notes, including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for Michelle's social media profiles, as well as mine, at superdatascience.com slash 915. Thanks to everyone on the Super Data Science podcast team, our podcast manager, Sonja Breivich, media editor, Mario Pombo, our partnerships team, who are Nathan Daly and Natalie Jaisky, our researcher, Serge Massese, writer, Dr. Zara Karchet, and of course, our founder, Kirill Arimenko.

1:06:26Jon Krohn:Thanks to all of them for producing another excellent episode for us today for enabling that super team to create this free podcast for you. We're grateful to our sponsors. They make it happen. You can support the show by checking out our sponsors links, or you can also share the episode with someone who would like to receive it. We'd enjoy it as well. Review the episode on your favorite podcasting app or YouTube that I'm sure helps with visibility. Subscribe if you're not a subscriber, but most importantly, just keep on listening. I'm so grateful to have you listening. And I hope I can continue to make episodes you love for years and years to come.

1:07:03Jon Krohn:Until next time, keep on rocking it out there. And I'm looking forward to enjoying another round of the Super Data Science Podcast with you very soon.