Stealing Part of a Production Language Model with Nicholas Carlini - #702

0:00The way that GPT-4 works would be functionally identical if the entire field of adversarial machine learning dropped off the face of the earth. like nothing that we learned at ever so machine learning impacted the way that we designed the biggest models today like no one trained them to be absolutely robust open ai and google changed their apis they broke functionality to make their apis a little bit less functional because of this attack because they have viewed the attack as sufficiently advanced that they want to make sure that this can't be done

0:44all right everyone welcome to another episode of the twimble ai podcast i am your host sam charrington today i'm joined by nicholas carlini nicholas is a research scientist at google deep mind nicholas joined us early last year to discuss his groundbreaking paper extracting training data from large language models, and his most recent work in the area stealing part of a production language model earned him a best paper award at the 2024 ICML conference. We'll be digging into that paper as well as his other best paper winning work focused on large scale application of differential privacy in machine learning.

1:23Before we get going, be sure to take a moment to hit that subscribe button wherever you listen to today's show. Nicholas, Welcome back to the podcast. Yeah, thanks for having me here. I'm super excited to jump into our conversation. I will refer folks back to our last convo for your full background, but I did notice something in your bio that I hadn't noticed last time, and that is that you like to write interesting coding puzzles like tic-tac-toe games in a single printf call or doom clones in JavaScript or a fully functional CPU on top of Conway's Game of Life? What is that all about? I feel like programming should be fun.

2:07And most people get into computer science because they find joy in writing fun programs. And I wanted people to understand that people can both do serious research and also have a good time with things that have absolutely no practical relevance but at the same time are are just entertaining and like you know like it's it's a thing that you the people should should have fun with the things that they're working with and so yeah i wanted to to call it some of the things that i do that are interesting and fun but have they don't they shouldn't exist it's like there's not no no one asked for this to happen uh it's just it's entertainment that's awesome awesome uh so as i mentioned the last time we spoke we were talking about some of your early work in exploring this intersection of security and large language models large language models as a field has exploded since that time maybe a a good way to ease into the conversation about the specific paper is to have you talk a little bit about that explosion from your perspective.

3:21Like how has the landscape evolved since you started studying it? Yeah. So yeah, in the last year and a half, so the last time we spoke was just after chat GPT's launch. And so in some sense, very little has changed because in the field of adversarial machine learning, we're still worried about the exact same problems. We're still worried about attacks that we used to call adversarial examples and people now seem to call prompt injection. We're still worried about poisoning where someone manipulates the training data set. We're still worried about data stealing where someone tries to recover data model was trained on, we're worried about the same set of problems.

4:05But on the other hand, everything has changed because it's no longer a field that is focused on problems that may exist in some hypothetical scenario. It's now a question of attacks that someone actually would want to implement because people are using this in real production environments. And so a bunch of our recent work has been trying to think about this problem from this angle of why would someone who is an attacker actually want to achieve some goal and not why might a security researcher want to write a paper on some problem that they think might be interesting in the future. And so the problems have been the same, but the applicability of these problems to the real world has gone up by quite a lot.

4:55and i inferred from uh the paper that the this idea of model stealing has become its own kind of standalone field of research yeah so model stealing as a field has been around for for quite some time uh in in machine learning the very early paper on this early machine learning world i think was 2016 by one of the co-authors in this paper florian tramer and his collaborators where they showed how if someone deploys a simple machine learning model as some prediction service, and I interact with the model, that I can actually steal the model the person has trained by making API queries. And the way that they did this was they focused on some very, very simple classifiers, like a linear classifier that takes the input, does a dot product with a single linear sort of scaler, and then returns you the output.

5:54some very simple classifiers like this, and they showed that it worked in those settings. And there has since then been a very long field of work that tries to make this more and more practical. And so the way that this tends to go, that the field basically has two directions. There's one line of work focused on trying to steal a model that is roughly as good as you can by making queries. So what this looks like is you have your model and I make queries on like the same data distribution I care about. And then I do some kind of distillation, or I use the fact that your model is good to train my own copy that's like as similar as your model as I can make it.

6:39And here what all I care about is getting high accuracy, like on the same data distribution. There's another line of work that tries to do something more like model extraction, where i'm trying to actually get at the internals yeah like the exact weights of your model like i want a bit for bit copy of what you have on your disk and it turns out that this is this is actually impossible like i'm in in that there are many different models you could have that represent the same function you know you could scale one layer by two and the next layer by one half right and so when you when you perform your computation you'll initially have your computation going one way, you'll multiply by two, then divide by two immediately afterwards, and the output will be identical.

7:23Like I can't tell. And so I can't actually recover the same bits that you have on disk, but I can recover what's called a functionally equivalent model that like up to some invariance is as similar as possible that I could hope to achieve. And so in prior work, let's say In 2020, we and another one of the co-authors on the same paper, David Rolmich, wrote independently papers that showed that if you have a neural network with only ReLU activations that only has, let's say, under 50 neurons that works in floating point 64 precision, that I can feed in arbitrary inputs that i get full arbitrary outputs then it is possible to steal this model in a functionally equivalent way okay but no one uses models that have like 50 neurons that work in float 64 and all these things yeah and so after we both wrote this paper at the same time florian the person who originally did model stealing david the person who wrote the other paper concurrent and i got on a call together and we're like okay like what can we try and do how can we extend this work and we tried to come up with some ideas and one of the ideas we came up with was actually the idea that eventually became this paper from six months ago um but the so we had an idea but the idea had no practical utility yet and so we just like we didn't we said okay fine like we can't we can't find something to do let's sort of like let's wait um and then you ask sort of what at the beginning, like what's changed?

9:04Well, the language models became a thing. And it actually now is the case. Like if you think about it, it's now actually a thing in the real world where someone will deploy a model that you care about, that you can make queries to, but you don't actually know what the model is, right? This is what a language model API is. You know, all of OpenAI and Google and Anthropic and everyone like have these production language models that do great, but they're not going to tell you exactly what the weights on the models are. They're not going to give it to you. And so we went back and we started thinking about this model stealing problem again.

9:41And we came across this idea we had from probably 2020, 2021, and realized this is now an important problem that we had some ideas on how to make some improvements. So that's roughly how this paper happened. Now, was the idea problem focused or approach focused? Yeah, both. Right. So, so we had the approach. So, okay, so maybe let me tell you what we can do in this paper. In this paper, we show it is possible to recover one layer of a language model. So language models, like, you know, like any other machine learning model are a sequence of layers. and what we can do is we can steal the last layer of the model which turns the internal representation into a sequence of tokens or words or whatever you have and in many models the first layer and the last layer are the same they're shared and so this also gives you what's called the token embedding matrix which turns the input tokens into the hidden state now not all models are this way but many models are they do share these two layers and so what we learn is the last one and if the last one's the same as the first one we also learn the first and a couple of comments yeah one is the to contrast the paper that we talked about last time again this is february 23 this was focused on extracting training data as opposed to the the model itself Yes.

11:07And I guess I want you to speak a little bit to the like the relationship between those two and the way you think about them and the approaches and their utility. Yeah. So so they're they're both about stealing, but they're about stealing entirely different things. So one of them is. Although like presumably the way you're the way and the reason you're stealing the training that is that it is somehow embedded in the model. also extracting training that is kind of model stealing yeah okay sure yes okay so from this from this perspective yeah but so maybe let me say talk about the motivation so in security this is a question of like who is being harmed by an attack like when i'm doing an attack like who am i trying to get at when i'm doing model stealing i'm generally harming the person who trained the model because like i have spent you know whatever millions of dollars training my very fancy model and someone else can try and get a copy of it without having spent that much money.

12:08Versus training data extraction is more privacy concerning your... Exactly. I'm worried that some hospital is going to train a model on some patient records and they're going to try and do a good thing for the world and make it so that some small hospital in the middle of nowhere can feed in their medical scans and get a reasonable diagnosis when previously they had no specialist on hand. And that would be great if they could do that, But it turns out that given access to the model, I can learn about the people the model was trained on. And so there's concerns of patient privacy. And so I'm not harming the hospital directly because they've actually published their model.

12:46I'm harming the patients who participated in this scheme. And so this is the big difference on the motivation here. And then the techniques are also fairly different. So when you're doing data stealing, for the most part, the way that we do this is we try and mimic the kind of data the model may have been trained on. And we try and get it to emit that kind of data at test time. whereas for for training for sorry for the model stealing piece what we try and do recently is use sort of we treat these things as mathematical objects and we treat this like a cryptographic problem where like you have this thing that exists and we try and steal it and so like so this is what we tried to do so the paper that i mentioned that we published in 2020 we published our paper at Crypto, which is like the conference where people submit new block ciphers and new theoretical encryption algorithms and whatever, these kinds of things.

13:53And the reason we put it there is we tried to draw this analogy between model stealing and cryptanalysis of ciphers. And this is the way that we framed the problem. We framed the problem not as some machine learning thing, but as you have this mathematical object. It happens to be a language model or a vision model or something, but like what it's doing internally is some matrix multiplies with some non-linearities like this is a thing that is just a mathematical object you can study independent of how it was derived and cryptographers are really good at that kind of thing and so we tried to frame this as that kind of problem to get some of that community focused and working on this problem the second question i had you were talking about the extracting this um this final embedding layer which often correlates to the initial embedding layer you know there's a lot of excitement about embeddings embeddings uh have are like increasing importance does that mean that like extracting that piece in and of itself uh is outsized outsized has outsized value relative to other parts of the model that you might be able to extract yeah we don't know um this i think is a good question so the the question of what you would do with it so we have some things that some applications of what i could do with stealing this part of the model um you can speed up some forms of generating adversarial examples in some ways.

15:16You can speed up some other attacks that need the ability to make arbitrary queries to the model. But a big part of what we wanted to try and do was to say, previously, the literature has focused on, let's imagine a hypothetical threat model. And then let's see, what is the strongest attack that I can make possible there? and like you change the threat model so that your attack becomes really strong. And this is, you know, I said we can steal a full model as long as it had this long list of assumptions that were all false in practice. What we wanted to do is we wanted to say, maybe on occasion we should write papers that look at what is true in the real world and then try and see what is the best that we can achieve in the real world.

16:09And obviously, it's not going to be nearly as strong as we can achieve in some hypothetical future where like various things were true but like this is the world we're living in now and so let's let's look at that world and try and see what the strongest attacks we can do are here because we know it will have some real world impact and that was like the setup for this paper is maybe this model is so maybe this layer is very interesting maybe it's not but it's it's the best that we can actually do right now and my hope is that or maybe my hope isn't i don't know depending in which you're viewing this, you could extend this to a second layer, right?

16:42You know, once you've done one, maybe you can do two. Once you've done two, maybe you can do three. And this is how the attacks that we wrote previously worked. Previously, we would go first layer, second layer, third layer, fourth layer, and we'd sort of cover them one by one. This time we're going backwards. We've got the last layer. Maybe you can use that and get the second to last layer. Now, our techniques are not going to help you there, but someone else who's smart might be able to come up with a way of doing that. And this is the general question and the way we want to try and reframe the field a little bit so that we could get some people looking at some real world problems.

17:11The implication there being that you were very much targeting that last layer as opposed to you had an idea of an attack and, oh, look, it produced the last layer. So this is where it comes to the two. So we had this idea in mind of let's try and attack real things. And I think even in the last time we talked, I might have mentioned this a little bit, where this is the thing that we've been trying to think about for a while. and in general the way that i try and find research ideas is i try to first ask what do we want to achieve and then ask what are the things that we can do and then you just like compute the cross product you look at all the things that we're able to achieve we'll look at all the things that we actually want and then we try and see which ones match and so we had this idea in 2021 or so of a way to recover part of a model and then we had this sort of realization that we we want to be able to to steal these these production language models and so we already had both these ideas we just needed to connect them yeah and that's what happened sometime uh almost a year ago today and then we we spent some time writing the paper up and then i got the paper into ICML.

18:34I guess I'm really trying to get at like the, you know, I want to steal part of a language model, a production language model is different than I want to steal the embedding projection layer of a language model. And so was that just a low hanging fruit or was that what appeared as you started attacking or did you look at the model as a white box entity and say, i can probably get that thing great okay um maybe to be more clear stealing the last layer is the thing that we learned how to do in 2021 ah okay so like the the attack idea that we had in 2021 was exactly this of how to recover the last layer of a model that had certain properties okay and no interesting model at the time had those properties and it was not interesting to recover any part of this model if we could make one up and then when it turned out that like these language models became a thing and we tried to think about stealing we went like well what are all the things that we know how to do and was like oh here's this idea that we came up with in 2021 that was not very useful at the time and it has to be adapted because at the time we were looking at value networks and language models aren't value networks and so there are some tweaks we had to make but we had the idea for how to steal the last layer of a model with certain properties at that time and then it turned out that many of those properties happened to be true some not that we had to adapt some that some that were that we could just transfer directly over uh so how does the attack work yeah okay um so uh i will hopefully not scare anyone away um but you will need a little bit of linear linear algebra knowledge here uh so let me just try and say this in the simplest possible way.

20:26So a language model emits, when you make predictions from it, a probability distribution over a bunch of tokens. So it says, you know, if you have like, my name is, it will have very high probability on valid names and very low probability on like the word cat. Okay. So many models have, let's say, 100 ,000 possible tokens that they know. This is roughly like words or something like this. And so you can visualize the space of outputs from a model as a 100 ,000 dimensional space. Now, one thing you could ask is what happens if I query on many different random inputs and plot the outputs of the model in this high dimensional space.

21:18One thing you could imagine is maybe I'm drawing points randomly in this hundred thousand dimensional space. And that's what people usually think about. It turns out though, that's not the case. Actually, despite the fact that these these logit vectors live in 100 ,000 dimensional space, they actually all lie on a much smaller dimensional subspace, let's say like, maybe 8 ,000 dimensional subspace. The way you should visualize this is, you can have a bunch of points in three dimensions, and you can have them all put on a plane, a two dimensional plane. So you can have a bunch of points in some very high dimensional space, but they all live on some lower dimensional subspace of that.

21:57And that's the case for language models. And there's this very particular reason why, which is what language models do is they work in some small hidden space. The hidden space of the language model is the width of the network. And this is roughly 4 ,000, 8 ,000, something like this for various models. And the final thing that the model does is it projects from the hidden space up to the output embedding space with a single linear transformation. And if you remember your linear algebra, you cannot increase the dimensionality of a space by a linear transformation. And so what you do is you take the hidden space and you just linearly scale it up into an 8 ,000 dimensional subspace of this 100 ,000 dimensional vector space, which is the outputs that the model produces.

22:44OK. Now, this is true. And now I'm going to skip over a bunch of details, except to say first one thing. First, let's observe that this lets you learn the size or the width of a model fairly easily. This is sort of the first warm-up attack that we have. First thing that you can do is you can learn how wide the model is. And the way that you do this is you make many, many, many different queries to the model. And then you just plot them all. And you look, what is the dimensionality of the subspace that all these points lie in? and it will tell you, let's say, 8 ,000 or something, or 4 ,000 or something.

23:24And now I've learned a single number about the model which was secret. Now, this number in itself is also a little bit interesting, because the size gives you something which people don't publish, right? No one knows how big GPT-4 or GPT-3.5 or any of the cloud models are. But if you run this attack in this way, you can learn not the exact size, but you can learn the width. Only the height is missing. And the height and width are usually highly correlated. And so I can learn some significant information there. And I think this in and of itself is also interesting because, okay, so for example, we worked with OpenAI to run this attack on their models.

24:05We got their permission. Lots of excitement there of like, you know, asking Google lawyers to ask if I can steal OpenAI's models, at which point they say, tell me no. and then like you know okay um but uh after we got we all agreed on this um i ran the attack um and what this means is there are now two groups of people in the world who know how wide gp 3.5 is there are like current and former employees of open ai and me like the way like this is like an actual thing you would want to learn and like actually the best way to learn it is through this attack which is something i think is interesting because this is not the case for many other attacks in security right or in adversarial machine learning where there are like other other ways you can achieve the same goal but like here like the best way to achieve this is through this attack and so we we have this piece of knowledge you talked about um querying the model.

25:05What is the nature of these queries? Are they, you know, human readable, a collection of human readable queries, a collection of nonsense that has statistical properties? Like, what are we talking about here? So it actually doesn't matter. In the language of cryptanalysis, all we need is known plain texts. I don't need to choose the inputs that I make. I just need to know what they were. So all that I need is a bunch of different inputs. And so in practice, what I did, because if you had a large user, you could passively observe their queries and responses and perform your attack without actually executing new queries, or at least this initial part of it.

25:46With one caveat, which is that I need to know the entire probability vector for a given input. And so most language models don't reveal this to you. They only give you, let's say, the top five probabilities. so what i have to do is i do have to make many queries to the model with the same prefix with the same prompt and ask the model to reveal to me the probabilities of all the tokens one after the other so i can't just do it entirely passively but which prefixes i choose don't matter so you're asking the model uh you're presenting a fair degree of repetition so that you can reliably extract the distribution of yes output tokens correct so i'll take the same prefix um whatever the prefix is and i'll ask the model for the output probabilities and i will do this you know a hundred times or a thousand times so i can recover the probability distribution over outputs And there's like the majority of the technical work in the paper is exactly in how you do this efficiently.

26:57Because there are lots of strategies you could take depending on exactly what APIs are made available. And do you also, presumably, I guess, you know, you also need to. So that's one dimension where you're like crafting your sequences. But then another is like some degree of coverage so that you can broadly scope out the distribution. So that matters less. Oh, really? Okay. Yeah. So all we need is to be able to just, because models are highly nonlinear. And so as long as you make any arbitrary queries, you'll cover basically the whole space. Dig into that a little bit more. How does the nonlinearity translate to covering the whole space?

27:39I guess when I hear something is nonlinear and you're trying to extract information about it, my intuition would have been that, well, you need very broad coverage because learning something would be very local. Okay, yes. So this is a good distinction. We are recovering the last layer, which is just a linear projection of the hidden state. So what we are learning is just a linear thing. All we need is inputs that in the dimensionality of the final hidden state are sparsely distributed around. And because the rest of the model is very nonlinear, we can rely on that to give us points that are essentially randomly sampled from the embedding space.

28:26But we're only learning the final linear projection. And that's what makes it easy to do that piece. Got it. So the linearity of the final layer kind of says that you only need a you know small number of parameters like if you can learn all the points of a line with you know your m and your b and then the internal non-linearity says that you're going to get a broad sampling on this whatever that space is you don't need to craft that yourself exactly right yeah you're relying on the non-linearity of the first piece to give you the sampling and you're relying on the fact that there are not many parameters for the second piece.

29:07Now, okay, in a simple world where you're doing a line, yeah, this is like mx plus b, you know, you do this in pre-algebra kind of things. What we're trying to learn is a matrix times some x plus some bias b. And it turns out that the techniques you could do here are essentially the exact same things as the things that you learn in algebra. They're just over some big giant matrix and it turns out that there's an algorithm for it it turns out the singular value decomposition gives you this it's not immediately obvious why but that's like very in the weeds it turns out that this is the thing that you can compute and and you can recover it in this way so the attack is a matrix inversion essentially yes yeah what we're trying to do is we're trying to invert some matrix and yeah there are reasons why it makes sense to try and do this and um no It's one of these things where the attack is not hard because of the math.

30:09The attack is hard because of the scale. Yeah, right, exactly. You need to get all the things working in the right way. Most of the technical work, as I said, was recovering these probability vectors because no one wants to give you these directly. You have to do some analysis on how to actually recover these things. And yeah, and so that's what makes things a little bit tricky. And then there's a lot of analysis that happens in the paper where we try and formally characterize exactly what's going on here. And this is where it's nice to have theory people as collaborators. I am not a theory person.

30:44My knowledge does roughly stop at singular value decomposition. But all of them are able to tell me, here's this attack that works. I go to them and say, why does this work? and they can do the analysis and explain the query complexity and here's what you should in principle want to be able to do. And then we can go and take that and actually implement it. That's awesome. And so getting from the collection of query responses or the pair of queries and responses to the matrix that you need to invert, what's involved there? Or is that an obvious thing? No, yeah. So it's not obvious, but the answer is nothing.

31:25So once I have the, so I'm going to form a sequence, I'm going to form a matrix. Here's what my matrix looks like. For each row, it is the logic vector for each token, one after the other after the other. So I sort of look at the token for the first token, second token, third token, fourth token. I place the probabilities of the logits in each of these for a row and I form the matrix that has many rows. Each row corresponds to a different query and then this is the matrix that I invert. On this matrix, I compute the singular value decomposition, and it turns out that this gives you what the final projection matrix is.

32:09And yeah, I won't work through the math there, but the 10-second explanation that should convince you of this is this is a low-rank matrix, and the singular value decomposition gives you the low-rank approximation If it actually is low rank, it gives you the actual low rank values. That's all I'll say for people that this made sense for. And this didn't necessarily make sense to me a year ago. I had to like brush up on my map. Low rank is essentially the dimensionality of your embedding space is much shorter than the number of samples you have. Yes, it means that these vectors live, even though that they are in a 100 ,000 dimensional output space, they actually live on a small, low dimensional hyperplane in the outputs.

32:53Yeah, so there's some fun math here that I hadn't touched. So ostensibly, I have a math degree. I haven't used all that much of it in the last, let's say, 10, 15 years. And so it's fun to occasionally go in and get to actually use some of that stuff. and so a lot is known about uh at least for gpt2 like the tokenization scheme and things like that like do you rely on any of that knowledge uh no we don't but but they actually publish it too um the reason why is because you want to know how much you're going to pay on a per token basis and so most providers give you a way to tokenize and compute what the cost would be of a given query OpenAI actually just gives you the full tokenization directly.

33:46They have this library called TickToken that is open source, and they just give you the full tokenization of all their models. And so this is not usually where the secret sauce lives. But if it were to be, we have another paper that shows how to recover this new. If it were secret, it wouldn't be harder for you to do what you're doing. Yeah, it's not. We use a couple of facts about the tokenization, but not a lot. but people would in general want to reveal this anyway. I was going to ask about a mixture of experts, which is kind of an emerging architecture for these large-scale models. Do we know enough about those architectures to say whether this attack might or might not work?

34:27Yeah, so it should work identically because we make no assumptions on the internal representations of how those are computed for the model. As long as the final output projection is a linear transformation, everything else will behave the same. and you will of course need to make very different approaches once you try and start to scale this attack up to recover let's say the layer before because if that has some mixture of expertise kind of thing that might behave very differently but as far as ours goes we don't make any assumptions about what happens inside the model you wrote about this you have a paper someone can't go and like perform this attack because turns out there's a remediation that you shared with OpenAI and Google, like talk a little bit about the remediation approach.