Ensuring Privacy for Any LLM with Patricia Thaine - #716

0:00I think that a lot of the misconceptions around what is easy or hard in AI when it comes to thinking that it's easy or hard to think it's easy or hard to think it's easy or hard to understand it's hard. high-quality representative data. And then the misunderstanding how huge a lift it is to get a product in production that is scalable, that also has a lot of corner cases in mind.

0:45All right, everyone, welcome to another episode of the Twimble AI podcast. I am your host, Sam Charrington. Today, I'm joined by Patricia Thain. Patricia is co-founder and CEO of Private AI. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Patricia, welcome to the podcast. Thanks so much, Sam. Great to be here. I'm looking forward to digging into our conversation. We'll be talking about some really interesting things you're doing with regard to AI and privacy and more. To get us started, I'd love to have you share a little bit about your background and how you came to start Private AI.

1:24Yeah, sounds great. Personally, my background is in doing research in privacy-preserving natural language processing. Specifically, it was originally focused on homomorphic encryption. So that lets you compute on encrypted data. and co-founded the first version of private AI on that topic. I have an academic hammer. Let's go find all the nails. It didn't work out. It turns out that's a very bad way of problem solving. And then started the second version of private AI really with the focus of building a privacy layer that's built for developers that can actually help them comply with data protection regulations.

2:01And now there are, I think, a couple of ways for us to dig into this topic. The first, I think, is to dig into what private AI is doing for AI developers and how those capabilities are expressed. And then also how you're using ML and AI to do all of those things. But let's start with the use cases and what you're trying to enable. Absolutely. So the core technology is understanding what's in your data. What kind of personal information you have, protected health information, payment card industry information, confidential information, all of these can play a really big role in whether or not you can end up getting data for your AI projects, whether or not you're going to be able to use third party tools.

2:51What exactly happens with regards to the output of the training data when it comes to sensitive information being displayed to your users or employees or otherwise? And so really what the product does is identifies this personal information and is able to redact whatever it is that you don't need. And that's really the concept of data minimization, which is core to a lot of data protection regulations. Got it. Got it. Minimization from the perspective of not having data that is not necessary in the process. Exactly. And it sounded like we're talking about data that comes into contact with a system in three different places, maybe even more, but certainly input and output and maybe storage.

3:47And can you kind of talk about data flows the way you see them? Absolutely. So if you look at large language models in particular, for example, you're going to be sending data maybe to a third party. in which case whatever information you're sending you're losing control of when you're sending it external to your organization or to your own laptop and so it's really important to think about what kind of information you're sending through what can you remove and then once that information is removed it can very often be reintegrated into the resulting output of the model and so you can still have a fairly seamless interaction with the model without that risk of sending that data externally.

4:28And then other aspects of it are even if you have a model internally within your own private environment, there are a few things to be concerned about. One, are you collecting data that you shouldn't be? Because whatever data that you do collect puts you at higher and higher risk of a more significant data breach down the line. That's part of the interesting part about data minimization. You keep what you need. And you don't keep what you don't need, meaning that you strip out that risk from your data in the very likely chance that at some point there will be a data breach. And then there are other aspects to it with regards to training.

5:09When it comes to trading models, I think this community probably is aware that that data can be quote unquote memorized by the model. And leaked out. Yeah, and as a result leaked out. And there's lots of research showing this, initially for character language models, then for GPT-2. And then, you know, there are more and more coming out around exactly the risks around training or fine-tuning on personal and confidential data. And we've got some episodes we can link to in the show notes about that topic. Perfect. Oh, yeah. I saw that you interviewed Professor Carlini. That's, yeah, he, yeah, great, great person to talk to about that.

5:51then there's also the risk of having personal and confidential data in the contextual data that you're using to create embeddings for RAG, for example. And that's the one where a lot of people actually don't think about. The assumption is that if it's an embedding, you know, it's generalized. It's kind of opaque, right? Yeah, exactly. But that is completely untrue. And there's research showing how if you have, for example, word embeddings that are created from healthcare data, the names and first names end up closer in distance to last names and disease that somebody had ends up closer in distance to the relevant name.

6:38and so even just if you can't inverse the embedding does that mean that is subject to leakage or it's subject to attack like i can get attacking that but yeah leak you know kind of passive leakage seems more difficult to less likely i guess no subject to both it's subject to really but i like that you're bringing that up because i i was doing a deep dive around uh inversion attacks of embeddings, right? There are some that are showing that when you invert an embedding, even a dense embedding, you can get up to 92 % of the original context. So that's an attack, right? That's an attack. However, what is more likely is that you are going to be leaking that information as part of the output of the model.

7:32So if you, for example, have embeddings that contain information about people's salaries, for example, that can get leaked out. In one of the examples for a data leak for one of these OLMs was finding out the salary of a CTO at a company. This can happen with embeddings just as much as with fine tuning. Interesting. Interesting. And was the example, the CTO salary example, that was an embedding leakage? I don't know. Okay. I don't know if that's public info. Can you talk through an example of how an embedding leaking data would work? I guess I'm imagining a scenario with regard to a particular condition and somehow the name that it uses as an example is a real person's name.

8:27Is it like that kind of thing? Yeah, so these embeddings can be more than just word embeddings, right? They could be representative of paragraphs, of documents. They could be chunks that are basically easy to search. And then when you send in your query through your large language model, it's going to go through an embedding model, look through your database of embedding, see what's closer to the query, and then produce an output with that information. And so if that query is asking for something sensitive, that sensitive piece of information could be leaked. And it could even be leaked accidentally, even if you're not asking for something particularly sensitive, but it is relevant to the context of the question.

9:10But then we need to distinguish between sensitive information being in the corpus of chunks that the embeddings are referring to and leaking information from the embeddings themselves. So we need to distinguish between the leak, if you have access to the embeddings directly, and how you can pull out information from there, whether or not it's an inversion attack, whether you're doing distance measurements or similar. And you have to distinguish that from output or risk with output from models that are using a database of embeddings to create responses. Does the risk require that the sensitive information still be in the data or can just the fact that the sensitive information was used to produce the embeddings, even though the database has been scrubbed, cause problems?

10:21So if the sensitive information was used to produce the embeddings, the chances are that there's sensitive information left over in the embeddings themselves. that could be output um and so that's why the the recommendation there is de-identifying data before you create embeddings or understanding what data you have in the first place to not include that personal information uh period um and just stripping out that entire entire piece it depends if you need the context or not uh around that personal information and the interesting part is that a lot of the times you don't need the personal information to get value from a lot of NLP applications.

11:02If you think topic modeling, sentiment analysis, and many, many other ones, the sensitive information is generally superfluous. Unless you're doing something very explicit in HR or need specific customer information for things like your master data management system, you're not going to you often don't need that personal information and in addition to that the personal information can lead to more bias in the responses and so that's something else to watch out for more and more research is coming out about how things like physical attribute or the country that you're from or the political affiliation etc that can all lead to biased responses from the model.

11:49And so by removing that, that removes the or reduces the risk of having biased responses. Got it. Got it. Got it. So that kind of brings us full circle to how folks might use private AI. So one is you are interacting with third party LLM and you've got sensitive information that you don't want to be exposed to that LLM. And so private AI would sit between your system and the LLM and kind of cleanse that data as it's passing through. And then there's another part that is about if you're training, doing fine tuning or training embeddings or some aspect of using data to build a model, then you can also cleanse that data kind of in, is it like in place or is it as part of the training loop that you're working with that data?

12:44It's an API. So it runs in your environment. You send a post request through and then out comes the redacted data and the list of personal information and sensitive data that was found. And so you could put it anywhere in your software pipeline. That makes sense. Got it. And are you typically doing it request by request or like are there batch or bulk APIs? There's a batch mode and also a screening mode as well. It sounds like a foundational part of what you're doing has to be kind of entity recognition, identification in this data because the user is not expected to tell you specifically the entities that are, well, what information does the user need to provide?

13:35This may be a better place to start. Yeah, the user can provide the, so we provide a list of entities that they can identify. and they could either choose to by default remove all of them they could choose which ones to remove if they don't want all of them or they could choose based on regulation that they're trying to comply with so if it's HIPAA or if it's GDPR CPRA you could choose which one you want and then it'll auto select which entities make most sense for that regulation thinking about entity recognition in general, like that's been a problem for a long time. And on the one hand, because it's been around for a while and there are lots of different solutions, I think it's tempting to think that it's like a solved problem.

14:23On the other hand, you know, if you have any experience trying to get LLMs or, you know, even traditional tools to do this, it is very hard. Talk a little bit about your experience with the entity recognition part. Yeah, I think that a lot of the misconceptions around what is easy or hard in AI, when it comes to thinking that it's easy, it tends to be around misunderstanding the huge lift that it takes to get high quality representative data. and then the misunderstanding how huge a lift it is to get a product in production that is scalable that also has a lot of corner cases in mind and part of the part of the problems that you have to deal with when building a named entity recognition system are you're going to deal with optical character recognition errors you're going to deal with automatic speech recognition errors, depending on, of course, your use case.

15:25But if you're dealing with a lot of data, like many of these large organizations are, the different types of contacts is huge. And then the different languages that it needs to apply to as well. And then all of the different entity types that you need to be able to perform well enough on for it to be good enough to comply with data protection regulations. So it's a huge balance between speed, accuracy, and then multimodality in addition to that, because it needs to generally deal with results from files and of course, text, transcripts, chat logs, but audio as well. And if you put all of that together, that's a giant problem space to deal with.

16:14So it's named entity recognition itself is an important piece of the puzzle. Everything else is also a huge lift. For the named entity recognition lift, if we double click on that, you have to think about the 50 plus entity types that you often need to recognize for some of these regulations multiplied by the number of languages that you need to function on. And you need to do that in a way that's also speedy, right? So we've created data pipelines that are very efficient with regards to adding new entity types. We've created data pipelines that are efficient with regards to creating synthetic data to add to our current data.

17:01Because there's no real personal information that you could go out and capture in the wild. you have to basically either beg customers for data or you have to create your own. And then finding out what kind of data you need to add to the model in order to make it more efficient or more effective while limiting the decrease in efficiency in your data process with the more data that you add. Is the customer training the entity recognition model? Or it's supposed to be zero shot with regard to the customer data? Yeah, absolutely no training by the customer. And for PII in particular. And here's why.

17:50It takes a lot of training to know how to annotate properly. if you introduce errors in the annotation process that can have bad effects in the models then we can't help you debug the model because we don't know what kind of data you put in there the in addition to that the

18:14basically we won't be able to provide the same model of that's that's consistent for the kind of corner cases that you have that might be relevant to other customers as well. Meaning if a customer's fine tuning, you know, what you're doing, then that it impacts repeatability across customers. And then are you as part of the service, like, are you making some type of guarantees to the customer or anything like that? We do have a warranty that's optional that folks can buy, and that's provided by Armilla AI. And it has a certain accuracy threshold that is guaranteed that we reach, if not your contract money is returned to you.

19:07And we have a certain time period in which to fix anything that falls below that threshold. And so your ability to honor kind of warranties like that is going to be impacted if you're customizing the model for each customer. Yeah. However, we do deal with extremely sensitive information, often things like credit card numbers for PCI compliance or health care information where folks do not want to have to go into the health care data and look around to see whether anything was missed. And so our mistakes are pretty high when it comes to accuracy. Can you talk a little bit about like how about generalizability and like how you've built the system to be able to deal with generalized data?

19:55You know, without without training or tuning to specific customer use cases, I would imagine there's a fair amount of even though we're limiting ourselves to entities, I would imagine there's a fair amount of variability in the types of files and documents. And that could be problematic. Correct. um lots of data is the answer to that okay lots of data collected over the years uh where uh oftentimes it's our customers saying hey you missed the stuff uh fix it and they send us a chunk of data and then we'll add synthetic data to that uh and do a lot of rounds of testing um But yeah, it's a big lift to be able to have a generalizable model like that.

20:42You mentioned multilingual as well. Is that also solved with more data and the models deal with it on their own? Or are there specific challenges and things that you have to do in order to support multilingual? There are specific challenges. and oftentimes it could be things like when you have multilingual data conversations, for example, so code switching. That can be fairly tricky to deal with. Also, the ability to detect what kind of language is being used to be able to provide the best model. That's something that we put in place as well, so a language detection system. the ability to also based on the language detection system provide different labels in the appropriate languages is also something that we've worked on and so there are more data definitely but also how do specific models deal with logographic languages like Chinese characters for example, versus alphabets like English.

21:57And being able to find models that are suitable for each of these, in addition to finding optical character recognition models that are suitable for each of these, and training them with the right data is very tricky. And you've mentioned OCR a couple of times, is the presumption that your customers are not responsible for the OCR themselves. They're giving you whatever their raw data is and PDFs and scans and all that, and you have to deal with it for them? It depends on the customer. But oftentimes, we have to deal with it. And OCR is another one of those things that is like, oh, that's a solved problem.

22:38And it's not.

22:44mainly again because of data complexity because you've got documents within documents or images embedded in documents and being able to it's not about just surface level comprehension right it's not just a summary that you're providing it's this is really detailed work that's being done and that's meaning you need to be able to find the name in the chart in the section and the you know on a page in an image basically exactly yeah and and when folks say oh i could just do that with a large language model it's really not taking into consideration uh the amount of lift it actually takes for the more detailed work at a large scale uh in in actual production when you're handling so many different variables at once.

23:34So large language models are out of the box, very good for things like summarization, for giving you ideas, stuff like that. But when it comes to really detailed work, the data is still incredibly important. And so as opposed to large language models, you're using more traditional approaches to these various problems, entity recognition and OCR and A, that question, but also like, are you using off the shelf open source or commercial things? Or have you rolled your own for most of the underlying models? Like, how do you think about your model supply chain? So can we think about it in the sense of what our customers really need?

24:24And they're dealing with massive volumes of data. And so large language models tend to be too slow for them and the amount of additional accuracy that one might get with a large language model with where we're trained on the data that we have and adapted for their tasks is often not worth the speed and cost increase for the customers and And so we spend a lot of time focusing on making the models as accurate as possible in a smaller package. And so there are multiple ways you can do that. You can start from a traditional base and kind of build up from there. Or you can take a large language model and distill it down to smaller language models.

25:17Is there a general direction that you prefer for these kinds of tasks? It's a continuous experimentation process. We're always trying different things to see how much juice we can squeeze out of this. You mentioned synthetic data a bit. Can you dig into that a little bit more and some of the ways you're thinking about that? I can a little bit. A lot of synthetic data is helpful when you don't have enough data for a particular problem. I think one thing I want to point out there is some folks might think that synthetic data is the right way to go for training their models, which might be true, and it worked for us.

26:00And the reason that it worked for us is because we also have a good balance of real-world data in addition to the synthetic data, and we've informed that synthetic data with the real-world coordinate cases. when it comes to fully synthetic data, the hesitation there is that if you're able to create fully synthetic data for a problem, that problem might, in some cases, depending on the problem that you're trying to solve, that problem might already be solved because you were able to create that data. In other cases, you might lose context where you might need things like conversation flow information or did a particular customer service agent need training in a particular case and medical information that's related to one another, to different parts of a different time, a full timeline of a patient, for example.

27:02And so the type of synthetic data that we use is usable because we have access to high quality original data. And we also have a very specific task where this works out well for it. And are you, like at what level are you primarily using synthetic data in the process of training the entity recognizers? Or are there other parts of the system where synthetic data comes in handy for you? We also generate synthetic personal information as part of an option for the output. And that's one of the interesting pieces where if you want to keep things like conversational flow or topic modeling and things like that, you still can because the majority of the context is still there.

27:52Meaning as opposed to masking, you can generate synthetic data that takes the place of the original tokens. Yeah. Interesting. Okay. Yeah. And so very similar to masking, you can still keep the context of the data. We've talked a little bit about OCR and how are you using that as one example of multimodality. Can you talk about other ways that you're using and incorporating multimodal data? Yeah. So we also do provide the ability to redact information generally for PCI compliance from audio recordings. And so you can bleep out the personal information from the audio recordings themselves. Meaning I've got a call center and I'm recording for quality assurance and people are talking about personal, they're incorporating PII into those conversations and you want to redact it before you store it, for example.

28:46Correct. That's right. And that's really important for PCI-D. assess compliance, where you do need to remove things like credit card numbers and account numbers. And the way that we do that is very much by, you know, we bleeped out the PII. One thing I will point out is that there is some research on anonymizing the voice within a recording. And that's interesting because if you anonymize information, it falls outside of certain data protection regulations like the GDPR. And for those listeners who don't know, the GDPR is the General Data Protection Regulation in the EU, and it basically covers what you're allowed to do with personal identifiable information and consent and all that jazz.

29:32So if you anonymize data, which basically means there's a very, very low risk of re-identifying the individual, then you don't have to comply with that regulation because you're no longer dealing with personal data. But anonymizing the voice doesn't remove responsibility to anonymize PII that's mentioned in the... Yeah. That's correct. And so the voice is a biometric identifier. And you can, I'd say at your own risk, modify the voice and perhaps call it anonymous, but there's not enough research out there, in my opinion, to say that a voice modification cannot be reversed. and so there are certain assumptions you might have to make with regards to whether or not you can consider an audio recording actually anonymized but data minimization is not just about anonymization it's about limiting the risk of the data and in situations like removing credit card numbers you're reducing the risk of fraud for example and if there's a data leak reducing the risk of how much information is going to be part of that data leak.

30:44And are you doing this on stored audio or do you have the ability to do it real time? It could be both. We are not an ASR company. That sounds a lot harder. Yeah, yeah, it is hard. It is hard. But we are not an ASR company, to be clear. So we can do it on audio. But if anybody needs something that's more specialized, like, you know, things like diarization of audio or things that, yeah, if you need actual control of what's going to happen to your audio aside from de-identification, there are lots of good companies out there like Deepgram, which I saw that you interviewed, Assembly, and Twilio and so on, who can do a great job at that.

31:29You have an audio stream coming from your call center. People are mentioning PII and credit card numbers and the like on your audio stream. You can real-time redact certain types of PII as it's happening. Is text being used as an intermediary or is it happening real-time on voice? Yeah, text is being used as an intermediary. If it were to happen real-time on voice, you'd be talking about probably keyword detection. and then you have to train a model specifically on keywords. That's just a very large search space of keywords to be looking for. Got it, got it, got it. I mean, you could train it for digits, for example, and probably be okay, but that's a big problem.