Nightshade: Data Poisoning to Fight Generative AI with Ben Zhao - #668

0:08All right, everyone, welcome to another episode of the Twimble AI podcast. I am your host, Sam Charrington, and today I'm joined by Ben Zhao. Ben is a professor of computer science at the University of Chicago. Before we get into today's conversation, be sure to take a moment to head over to Apple Podcasts, Spotify, YouTube, or your listening platform of choice. And if you enjoy the show, please give us your greatest rating and review. Ben, welcome to the podcast. Thanks, Sam. Happy to be here. I'm really looking forward to digging into our conversation. We'll be talking about your research into the intersection of security and generative AI.

0:47And to bring us down that path, I'd love to have you share a little bit about how you came to work at that intersection. Sure. I've been an academic for, gosh, it's almost 20 years. For the last 10 years or so, I'd probably say, I've been focusing much more on computer security. And for the last few years, it's been much more about computer security of AI systems and really mostly defending machine learning systems. But in the last few years, I think it's been increasingly important to me, visible to me, that AI systems and machine learning systems are open to misuse and abuse. And so a lot of my more recent attention has been spent on understanding what that looks like and building systems to protect people against misused and even weaponized versions of AI.

1:33That's what got me here. And I guess I think in the last 12 months, it's really been all about human creativity and defending human creatives, whether it's artists or choreographers or singers and musicians and writers from invasive uses of generative AI that, you know, not only trains on their life's work without consent or compensation, but also seeks to replace them wholesale. And has that represented a big shift in your work? I know a lot of work at the intersection of security and AI in years past has been focused on like protecting models against adversarial attacks and bad actors, whereas you're taking a human-centric approach as opposed to a model-centric approach.

2:14Is that new for you or kind of a continuation of the thread? A little bit. I mean, I think, like you mentioned, I worked on adversarial examples and how to defend them for a number of years. But also, you know, more recently, I think as early as 2017, I think we had a paper on what happens when you can basically fool humans by writing online product reviews that are as good as human-ridden ones. And people couldn't tell the difference. But this was, you know, years before gender of AI. And then 2021, we had a paper on what happens when people can mimic human speech and synthesize human speech in the voice of a particular target.

2:50And again, fool humans who couldn't tell the difference. But that was also the year before Generative AI got there. So yeah, I mean, increasingly, I think it was quite clear that this line of work was going to be important. But I think Generative AI really just took everything up a notch real quick. And one of your first projects in the area was Glaze. Is that the correct order? Can you talk a little bit about that project? It started, I mean, honestly, in 2020, when we did an early project protecting humans against invasive facial recognition. So how do you modify your own photos so that they don't get downloaded by companies like Clearview.ai and then use the training models to recognize your face, right?

3:31But then we had that tool that was out there, got a lot of press. And then last year... Well, before we go much further than that, let's spend a little bit of time on that one. How did you approach that problem? I've seen and I've interviewed some folks that have worked on analogous problems or adjacent problems or approaches of like camouflage clothing and masks and different kind of signals that take an adversarial example type of approach. I'm presuming you're talking about digital images and maybe injecting some kind of noise or something. Is that the idea? It's a little bit different. So, you know, the ugly sweater thing from Tom Goldstein and others, for example, that's more of a untargeted attack, right?

4:14That's a, I'm going to screw with your classifier so that you can't even detect that I'm here type of thing. Facial recognition is a little bit different. And facial recognition is a pretty straightforward task. There's lots of models out there. They all sort of have mutual transferability. So Fox basically took one of the existing models and said, okay, we're going to interpret how most models are going to see your face. basically the compute and optimization where we find out the minimum set of pixels that we can alter to change your photo in such a way that it looks pretty much the same to you.

4:45But in the fish recognition feature space, you look like, I mean, make me look like Denzel Washington, right? Or someone completely different. And so what that does then is that if a model trains on this, right, I'm going to take all my selfies and modify them all the same way, then it's honestly going to associate the name Ben with somebody who looks like Denzel. and then somebody on the street takes a real photograph of me, sends it to the model company, they're going to come back with a blank because the actual feature space where I should be, there's not much there or there's lots of other people there.

5:17So it'll come back with someone else's name, but it'll think that Ben actually looks like Ben's own. So you give me a normal sort of Asian looking guy and it's not going to come back with Ben's name on it. Okay, super interesting. Was that approach implemented or was it, How far did you get in terms of implementation and getting people to kind of take it up as an approach? Yeah, no, we built a tool. It was downloaded over a million times in a year. After that, we stopped tracking. It got a New York Times article. It got a lot of global press. I think a lot of people used it. I don't know. We don't keep track of how much now.

5:51But we still get occasional emails on our mailing list, people just asking, hey, have you done any updates to this? We haven't. It's been a while. But what's interesting is that after that paper came out, I think at least six or seven follow-up papers came out from different groups trying to do similar kind of things using slightly better models or more transferable or different approaches. So I think that was a big positive was the fact that we sort of pointed people's attention to this new problem. And I think a lot of people jumped on it. And they're still writing papers about it today. So that's a plus.

6:22That's, I think, the lasting impact from that. And there's a bunch of these tools out there now that people can use. I'm curious if there would be a way to somehow measure the broad impact of a tool like that on the facial recognition software providers. You're shaking your head. You've thought about this. No, no, we've done it. You've built it. You've done it. We built it. We measured it. And even at the time that we, you know, as part of the research paper, we tested against Amazon recognition. We tested against, I believe, which other model. We tested on at least like three or four public commercial models.

6:57And we found that you could fool them. Absolutely. I guess I was thinking less like benchmarks and more like, I'm curious, like what percent of images in some large providers database have been like masked or fooled or something like that? And if there's a way to reverse engineer that out, probably not. But it's kind of an interesting idea that I was curious about. Yeah, that's a very interesting question. I never thought about it along that dimension. Difficult to know because on an individual photo basis, I think it's pretty challenging to figure out whether the feature space position has been distorted or not.

7:32There's probably some techniques, but none that come to mind immediately that really work well at scale, right? To where you can actually do this with a bunch of photos. So yeah, but that would be an interesting question for sure. That was a while ago. We're talking about 2020. That was a whole pandemic ago. That was a whole lifetime ago. I mean, man, we all look different. We all felt different. We all lived different back then. And then last year, we had some artists reach out to us, basically said, hey, your tool Fox looks super interesting. Can it protect art? And I was utterly confused. I was like, what do you mean?

8:03I mean, come on. It says facial recognition. You know, what part of that connects to art? And so we wrote back this very sort of confused email, said, well, thank you so much for your interest, but like, I don't know what you mean. And that was that until about like three, four months later, when all the press hit about all the image generators, Dolly 2 and Mid Journey, all that. My student, Sean, was like, literally like we had a light bulb moment in the lab and he was like, Ben, is this what we're talking about in that artist email? And we said, oh, maybe that's it. So we reached back. You know, we still had that original email.

8:37We get it now. Yeah, we're like, is this what you mean? And this was the artist from the Netherlands, Kim, who still had that old email. And she's like, yeah, but if you guys don't know what's going on here, jump on the Zoom call. And sure enough, we did. We're like, all right, We don't know what the heck is going on. We get on this call, 10, 15 minutes, we get on and it's packed. It's like 500, 600 people, all artists, all on a single Zoom call and just crazy. Just everybody talking about how their life is literally upside down using, because of generative AI, what people are doing with generative AI models.

9:09People are finding not only have their entire life's work, their entire gallery's been downloaded and scraped and trained on. but then that other people are specifically using these models to literally replicate them and then use their names as a prompt so that they can draw whatever they want in the style of you know artist x and artist x doesn't even know spend some more time on like as best as you understand it like what that means for an artist's life to be upside down i think for people are creating art they're probably not monetizing that art because nobody pays for art and their art got sucked into these databases and yeah, they're probably pissed, but like it doesn't really impact them.

9:50Yeah. Unless they're like the top 0.05 % of artists. But here you have 500, 600 artists that probably aren't all the top 0.05 % of artists that are really impacted. How are they impacted? It is so interesting. I mean, I did not understand the art world. And, you know, even today, I mean, it's been a year, but I still only understand snippets of it. Right. But there are so many different kind of artists, concept artists, illustrators, graphics artists, and they all do different things. But a large majority of them work on a freelance basis and they're happy doing it. It gives them freedom of scheduling.

10:25They can pick the gigs they want and they can support themselves well. And it's largely through basically word of mouth. They put their galleries online of their best work and people will see it. People know about their work. Oh, you did that album cover for Tori Amos. Oh, you did that book cover for so-and-so. love that stuff. Can you work with me on this new upcoming movie poster? That kind of stuff. And so that signature style is their livelihood. Absolutely. And they work for decades to get this stuff, right? So they're very unique. You can a lot of times look at one single image and be like, oh yeah, that's Sam Yang or that's Carla Ortiz or something very distinctive.

11:02And so for them, it's devastating. I mean, I've never seen anything like it. It's not like me, you know, someone saying, oh, I took Ben's research paper and I did a thing with it. I'm not married to that thing. It's cool. But this is literally their identity, right? This is their calling card. When somebody sees that art, they're like, oh yeah, that's Gregory Taus. Look at that dragon. But then what happens is someone else takes that style. Literally, it's like wearing your skin and taking your entire identity and then doing things with it that you not only can't control, but sometimes you don't even know.

11:36And one day you discover there's this whole trove of art that's out there that bears your name, right? Because it's part of a prompt, but it has your name on it. But obviously it's not you. You never do it like that way. And it, in most cases, looks like crap, looks like really poor imitations of your art. The emotional devastation for some of these artists is so real. Like self-harm is a thing. People are depressed. People are just, it's dark. And it gets to the point where like, for example, just this past week, Kelly McKernan is one of the plaintiffs in the stability lawsuit. And I've gotten to know her over the last year.

12:09She's one of these artists who's been attacked a lot in this dimension. And one of the things that really upset her this week was the fact that she realized when you Google her name in the image column from Google, what comes up is not her art. It's literally an AI generated copy of her art. When your identity has been completely subsumed by AI, that's when it really hits. And she's not even the first one. I mean, I think Greg Rutowski had that happen to him months ago, but it's a whole level of sort of identity theft. And economically, it's devastating. I've now gotten to know so many artists who were a year ago were highly successful, had no worries about supporting themselves and their family, and now are literally scrounging.

12:49They're losing their life savings. They're just trying to hang on because all the gigs are disappearing. People are using AI to generate really cheap. If you have anywhere near a sort of an artist's eye, you'll see just so many problems with this generative art. But that's what people are doing because it's cheap. And so the gigs are drying up, the freelance jobs are disappearing, and you see on a weekly basis, people announcing that, hey, it's been a good 15 years, it's been a good 23 years, but I'm gonna start driving for Uber tomorrow. That hits home. That's devastating to see some of the most amazing artists basically say, I'm so good, but I still can't make a living.

13:27I'm going to go start driving cars tomorrow. The thought that comes to mind is like, we've got fair trade for coffee and products and things like that. Is there a movement to have like certified human created art? Is that a thing yet? It is. It's rapidly becoming a thing. We went so fast from the utter fascination of, wow, look at that glossy thing. That's so real. And I can't believe I I produced that, to now, it's literally people look at it and they're like, yeah, that looks so plastic and sort of soulless. And so people are really, whether it's the book cover industry, whether it's people who do Wizard of the Coast, fantasy books, and a lot of these publishers are now getting very specific about if you're using AI, you better tell people.

14:10And if you're using AI, you're probably not going to get one of these human gigs. So now there's a delineation and some of these AI people who are creating AI prompts and art, quote unquote, they're going off and claiming themselves to be human artists. And they're literally scamming people. They're now people who go on Twitter and tell stories of, I paid a good commission for this and I was promised this is human art. And then lo and behold, someone can prove to me that this is actually a mid-journey output. And so they're very upset. There's a lot of that going on. It's totally unrelated, but I saw a tweet and I guess you can't really believe the tweets that you see any more than you can believe the art that you see.

14:49but someone kind of uncovered this conference that is a tech conference and they had a couple of, you know, out of their end number of male presenters, they had three female presenters and it turns out that they were AI generated images and the women did not actually exist. Yeah, that's a thing now, right? I mean, talk about like diversity through what? Forgery? And this is just the most recent incarnation. I mean, months ago, we already had cases where magazines were including people of color, models of color, and turns out they were AI generated. Already underrepresented, everything stacked against them, and now to be replaced by AI just because it's cheaper to do that than to find real models of color and real representation.

15:37So yeah, it's all kind of gross. There's not a whole lot of good about it. Yeah. And it's just striking me in this conversation, like the speed at which like entirely new classes of social problems are coming to becoming real is nuts. Yeah, it's, you know, in many ways, it's really exacerbating the worst of what we already had. Underrepresentation, bigotry, bias, the wealth inequality, all these things are getting blown up and magnified by this race to the bottom, right? You have this technology that's really centralized in terms of who's producing it. And there's going to be real reach, a real gap between those who can afford human things versus those who are basically going to get AI things.

16:26And some of these companies, they go out there and they say, hey, look, it's going to be great. We're going to be able to produce AI doctors. And then you ask them, well, if you had a choice, who would you take? They'd be, of course, yeah, now I would prefer a human doctor. In many cases, since people can't afford human doctors. So let's give the people who can't afford human doctors AI doctors, right? And that's the problem, is when the AI doctors are hallucinating, when they're wrong, when they're making bogus recommendations. But then, you know, the argument is, maybe this is better than nothing.

16:54But man, that's a slippery slope, and it's a dark, dark place that we're heading towards. And magnifies a lot of problems that already exist, as you pointed out. So how does this get us to glaze? Right, so glaze is really the first step. What I saw was just tremendous damage to real creative people where they were being impacted, you know, viscerally in front of them by these models that were being fine-tuned on them. That's where you take a stable diffusion model. You just say, I'm going to literally learn the fine-grained details of a particular person's style. And when you zone in and hone in on a single person, it can get very, very specific, very, very quick.

17:35So people were doing this. People were, you know, training on individual artists. and then sharing and even selling some of these models. Like, hey, would you like a Sam Yang for five bucks? Right, you could do that. And tens of thousands of people were downloading these little mimicry models. So the first thing we wanted to do was to stop this. And so Glaze does that. Glaze basically kind of does what plays the same trick we did with Fox, except on art. And so it takes the style, it takes your image, digital image of an artwork, and it understands what the current AI models would see when they look at this particular piece of art in terms of style, in terms of composition.

18:12And then it goes in and says, mathematically speaking, how do I compute the smallest set of changes in such a way that it would look pretty much the same to human eyes, stylistically, artistically, whatever, but to the AI model, it would look like a dramatically different artistic style. And that's the question that it does. The AI answers every single time you fit in an image and will take in a realism charcoal portrait, for example, and produce a Jackson Pollock-like style. And so what that does is it makes small changes to the art. For the most part, it looks pretty much the same. For some styles, it's actually really, really hard to tell the difference, even if you've got it blown up side by side.

18:52It's very subtle. But you put this online, and the whole point was artists can do this with their art, and then they could upload images online and feel good about it because they know they won't be trained. If someone does download that glazed art, they try to train the model on it to try to subsume your style, they'll just get it wrong. They'll get some other style that looks nothing like the original Target. And that's how it protects people. And artists were really stuck in a hard place where they said, well, either I could stop posting art, in which case I have no more income because that's just how people get known.

19:25That's my promotion vehicle, right? Exactly, right? I mean, this is how you get commissioned. So either I don't do that and just shut myself off, or I upload art and I risk basically being sucked into a model without my permission or knowledge. And so Glaze sort of gives them a way out, right? So now they can upload art, they can feel good about it, they can know that even someone were to download these images and try to train on them, that they would get something wrong and they would fail. So that's what Glaze does, yeah. Got it. And implementation-wise, I'm envisioning something akin to white box adversarial attacks.

20:03You mentioned that there are representative models and the approaches that you would take against them generalized. So it sounds kind of white boxy. And you mentioned injecting perturbations, which sounds like adversarial attacks. Is it that kind of idea? Yeah, kind of, except it's a poisoning attack, right? It's not an inference time attack. It's modeled poisoning. Yeah, it is white box, but it transfers real well, Well, particularly since Stable Diffusion is what most of these models use when they try to mimic, because it is the open source version, right? You can't take Dolly or Imagine or MidJourney and fine tune it yourself.

20:38So people use Stable Diffusion, and that's what we use to generate Glaze as well. So that works fairly well. It is white box, it is fully accessible, and it transfers real well to the target because the target is also basically using the same model. Okay. In terms of challenges that you ran into trying to adapt what you did previously to this new challenge, this new world, or some of the biggest challenges slash lessons learned? I think the first thing was just trying to understand how do you isolate style? Because it's such a vague thing. And that was mentally our biggest challenge. And I was most concerned about that.

21:16Turns out you can use style transfer to solve this problem. So we take your original art. We give it to literally a local copy, Stables of Fusion. We say, transfer this, do a style transfer, and give me a version of this that looks like Jackson Pollock, right? It'll do that. And we use that output then as the training target, as the projection target for our perturbations. So we can make it look in the feature space like that Jackson Pollock version of your art so that we make small changes to the original. It still looks like the original, but in the feature space, it looks like a Jackson Pollock version of that same thing, whatever that you drew, right?

21:53And so that gets past that problem. But we had a lot of other issues that I think it's just part and parcel of trying to actually do something for real. Developing software that people can actually use is obviously a challenge. But to the credit of all these artists, we worked together with so many artists and got so much help. Our first research paper was such a success because we had almost 1 ,200 artists sign up to help us in a massive user study, right? I mean, you don't ever see that in a user study. You get 100 and 200, you're like, okay, that's legit. No, we had like 1 ,156 artists, all either part-time or full-time professional artists, all join in on this user study.

22:33They didn't want to be paid. They're just like, okay, we just want to help because this is going to potentially save our careers. Practically, what does that mean? You set up some interface, they can upload their images and use your tool and then assess how well it performs in some dimension? That would have been probably a little bit too much just in terms of timing and getting everyone on the same page. What we actually did was we just ran a representative set of a few artists that everybody knew. And then we ran their art through and we said, okay, here's the before and after, before the change by glaze and after the change for different levels, different knobs, different strengths of protection.

23:12And then we show them, here's the protection effects. If you don't get protected, this is what you can do with mimicry. If you get protected, this is what happens after you've had the protection. And, you know, we asked them lots of questions like, okay, what level of perturbation are you willing to tolerate on your own art? Now that you can see what it does to these three different or four different art styles, right? And people are pretty good at sort of understanding what that looks like, and they can mentally project that onto their own pieces of art. And then, you know, we show them the output.

23:40We say, okay, this is the protection. At what level do you think this is sufficient? Is this sufficient? Is this good enough protection? So yeah, so just tons of feedback, very positive. And so that went into the research paper. But, you know, even with all that help, we had artists who volunteered to help design the app that we wrote. You know, after a while, we had glaze out there and people were so enthusiastic. We had artists taking money out of their own pocket to fund social media campaigns, advertising for Glaze. That's how excited they were. We had people, of course, advocating for it on social media, on Facebook, on everywhere.

24:16You can imagine trades, organizations, guilds, all sorts of these kinds of organizations, held seminars about Glaze and lots of keynotes and all these things. So it was tremendous. But still, at the same time, we had so many things to learn. For example, for months, I thought, okay, we're going to do this tool and it's going to be for GPUs and these artists are going to make use of this. It's going to be great. And then it took me like a whole three months to realize most artists don't have anywhere near that income to buy a desktop computer to run GPUs on it. I'm like, what? And so these artists are like, I can barely afford an old tablet.