How private LLM inference actually works

0:04James Wilson: Hey everyone, I'm James Wilson and welcome to this Risky Business Features episode about private LLM inference. That is the ability to use cloud-hosted AI where the provider itself cannot see your interactions with the large language models by design. Joining me for this episode is Tanya Verma. Tanya is a cryptography engineer and is one of the co-founders of Tinfoil. You can find them at tinfoil.sh. They are a provider of private inference and they've pioneered a lot of the work required to create a cryptographically verifiable private AI in the cloud. Now, this is not a sponsored interview, and while we will talk about some of the novel work that goes into running Tinfoil and creating that service, this episode is about the paradigm of providing access to LLMs where the customer's privacy is preserved in a way that can be verified.

1:00James Wilson: This is also quite a long episode. It's about an hour and 20 minutes, and it has two distinct parts. The first 45 minutes is technical. We start off at a very high level intentionally explaining how private influence works for sort of an executive or senior leadership style audience, but then we increasingly delve further and further and further into the detail of the incredible engineering and technology that goes into making this possible. Then at around the 45-minute mark, we pivot to the social and non-technical aspects of this. Should this level of privacy be as prevalent and as assumed as HTTPS and TLS is today?

1:41James Wilson: Or is it only for nefarious actions? Do you really only need this if you're doing bad stuff? And what sort of governance, be that KYC, guardrails or safety classifiers would need to be in place in order to make private inference a net positive for all of us? And I guess the biggest question out of all of this is, is this just a niche for open weight models right now? And does it remain that way? Or is private inference how we should all be accessing large language models and AI going forward? We'll cover all of this and more. And look, I went into this not really understanding how private inference was even possible.

2:26James Wilson: I knew it centered around this concept of what's called a trusted execution environment and that there's a lot of inspiration from Apple's private cloud compute. And this trusted execution environment, we'll refer to these kind of in a couple of different ways throughout this episode. You'll hear T or TE or TEE. And just know that that refers to this foundational concept of a trusted execution environment. But as you're going to hear, this episode was really a journey of me discovering exactly how this technology works that I wasn't familiar with, nearly to the depth that I wanted to be in. And I owe a huge debt of gratitude for Tanya for taking the time with me to really walk me through this and correct my wrong assumptions and to educate me as to how this works and why, particularly in her view, that this is as important and foundational as just the level of privacy that we expect from our interactions on the internet today.

3:27James Wilson: So I'll drop you here into the chat where Tanya talks about, again, starting off at that very high level, just taking us through the advancements in CPUs and GPUs that have made the concept of a cryptographically verifiable private AI inference in the cloud even possible. I hope you find this as enlightening and as interesting as I did. Enjoy. So recent generations of server hardware come with this capability that allows you to create a region on the hardware where like all the memory is encrypted. And even if you have access to, you know, the machine, like you're the one that's like operating the service, you actually can't look inside what's going on here.

4:12And so up until a couple years ago, like this sort of like technology was only supported on CPUs. But as of like a couple years ago, NVIDIA's latest generation of GPUs also allow extending, you know, this technology onto the GPU. And so that's kind of like how you like, you know, isolate the data and like keep it confidential. But then, you know, these secure enclaves or these secure hardware regions also have this other property that basically allows them to produce a fingerprint or like, you know, produce like a proof that says like what is like the hardware that's there, like what's the security configuration, what's the code that's running, you know, the model, so on and so forth.

4:55And And this process called attestation, the hardware can basically produce a signature over all of this stuff that says like, hey, this is kind of like what is going on in here. And then you can combine that. So what we do is we open source all the code that runs inside this enclave and we actually produce these measurements offline or this fingerprint offline. And it's published to a transparency log. And the client, when it connects to any of our, you know, like models or any of like our other applications, they're basically able to pull the proof of like, you know, the code that we claim to be running, then get the actual live proof from like, you know, the server that's actually running it and check that the code that we claim to be running is exactly what's running.

5:41So when you combine the confidentiality property and then the attestation or the verification property, you can get a very strong guarantee that I'm speaking to this particular model, running this particular code, configured in this way securely, and tinfoil or the GPU provider has no access into any of my data. And this is all provable to the end user.

6:07James Wilson: And so, again, staying at sort of the high level, like explaining this to an executive, there's, I guess, terms out there that people will be relatively familiar with now. Like if you're in the Windows ecosystem, you've definitely heard of TPMs and that sort of form of that trust. In the Apple ecosystem, the secure enclave was a huge development that allowed us to do things like Touch ID and other sort of biometric storage of data. But is this the same thing, but just transplanted into GPUs or are there fundamental differences around this? And again, I'm sort of approaching this from an executive mic.

6:42James Wilson: Oh yeah, I've heard of this. Same, same, right? But is it? Yeah, it's very, very similar, except that this can run like an arbitrary program in a way that like Touch ID, TPMs, like they can usually only like, you know, they can produce a key, like they can generate some keys or they can sign something or maybe they can encrypt some value. These environments that are similarly very locked down do have the capability of, you know, running any program with, you know, some caveats here and there. Right. And hence the term trusted execution environment, right? Exactly. It's the combination of encrypted memory that can be used for actual program execution.

7:24James Wilson: Does the programs sit in the TEE or is it sort of still on disk outside? Like I'm just, I'm trying to establish in my mind a high level understanding of the boundary of what this environment actually creates. Yeah. So the program does in fact run in the, in the TEE. Yeah. Right. Like it's, it's loaded, like, yeah, that's the program memory is like all inside the TEE. Okay. So again, from high level, what we've explained here is that we've got, it is primarily a hardware-based solution. That hardware-based solution has sort of, I guess, echoes back to TPMs and secure enclaves. And now this is taking it from just storage and, I guess, production of keys and or signing capabilities to actually executing arbitrary code in there.

8:08James Wilson: And so the solution then, I guess, the next thing to dive into is, well, what runs in that trusted execution environment and why is it special? So you've got this capability, great. What do you do with it? Yeah. And I think that's a harder question than it sounds at first, because I think for a number of years, people really struggled with figuring out, especially as these were relatively restricted environments, what to actually put in there. And so a lot of the solutions that you saw were almost using it for some sort of compliance enterprise checkbox, especially if you look at like some of the stuff from like, you know, clouds and so on and so forth.

8:51And there are like, you know, some things in like the blockchain space, but like, I'm not like too sure about that. So for us, when we started doing it, you know, as my co-founder likes to say, like inference was created as like, you know, the perfect use case for secure enclaves, because it's a stateless process, again, with caveats, But and it's getting like, you know, more and more and more complicated, especially when you have like agents and like you have to make like a lot of like, you know, external actions. And there's there's a lot you can do there. But yeah, like in in in general, what are we currently running in there?

9:27We're running like, you know, like large open source models. So, you know, you can think of models like Kimi or GLM or DeepSeek, Gemma, so on and so forth. And the idea is that when you interact with like, you know, like 10 false inference API or like, you know, like our chat, like you have this guarantee that like, you know, we can see your data, no one can see your data except you. And so that's kind of one thing. But there's also like a lot of people that are like building applications that where they don't want to see their end users data. So like, you know, like, for instance, one of our customers, they kind of have this, what you would think of as like a very invasive product where, you know, it's like constantly recording your screen and like, you know, analyzing and surfacing insights and so on and so forth.

10:15What they started out with was using only like local models for that. But as they're expanding to like, you know, different platforms and like, you know, trying to like make things like more efficient, you can't do that on like a phone, for instance, and you can't use like, you know, better capabilities. And so what they kind of think about is like, you know, how do we like extend like the trust boundary from, you know, your local device to the cloud in some way while still making sure that even though they're operating the backend, like they've like locked themselves out of the data and obviously tinfoil cannot see the data and so on.

10:48And so it's kind of like this like shift a little bit from like how people thought about enclaves like for a long time. I think people like, you know, especially these like hyperscalers kind of like pitched as a solution to like remove trust from the cloud. But like I think now you can almost think of it as like removing trust from the service provider, the application operator. And that's that's that enables like a really wide variety of use cases that that weren't really possible before. and yeah and like one one other note on that is like you know i think ai in particular is interesting here because there's there's so much incentive to look at your data and like benefit from it in a way that maybe didn't even exist with like the cloud providers because like you know that may not have been their business but here it's like there's there's so many companies out there where you know like a large part of their revenue stream so like you know their main product is like whatever it is and then a large part of the revenue stream is actually just kind of like harvesting that data and like either like training models or like you know more often than not like packaging it up and like selling it to like you know these like frontier labs or like other like you know companies like scale ai or so on and so forth that like you know kind of collect and package this data for like again selling to the labs so yeah yeah it's almost like the uh the

12:07James Wilson: data broker ecosystem extending into into ai um there was a point you made there that i think is worth just unpacking a little bit how inference is like one of the perfect use cases for trusted execution environments because it's stateless and like I remember early on in my trying to wrap my head around this emerging thing called large language models the big the big aha moment for me was when I went from thinking you know there must be some you know gigantic amount of state being held in the cloud because when I type the next message into chat it must have a session up there. It must be storing that session.

12:40James Wilson: That's how it knows how to compute. But it's this bizarre moment when you realize, my goodness, this protocol is so naive. You send everything and then you get the next response. And then you send everything plus that last response and then you get the next response. It is, I cannot think of a protocol in the networking sphere that mimics this kind of behavior. We have spent a long time gravitating towards, I guess, almost like thin client on the edge of the device where there's minimal state, just enough to display the UI to the user, and everything else is held in the cloud in a very stateful way.

13:17James Wilson: Now, inference changes that model entirely, where again, as I said, it's almost like there is, well, no, actually, in fact, there is zero state held in the cloud other than logs and holding onto the data that you mentioned. that it's this odd duck of a protocol where the entire chat is sent up for every message and then you get the next response back and so on and so forth. And so that I think is the important thing to explain to folks when you say it's stateless because the penny doesn't quite drop until people realize, oh, there can literally be none of my chat being held in the cloud and this thing still works.

13:52James Wilson: Is that what makes this the perfect use case for trusted execution environments? Is that what you meant by that? Yeah, yeah, exactly. So like trusted execution environments have no persistent state. Like, you know, everything is in memory. And like, you know, when you shut it down, like everything goes away. So, you know, you can think of it as like, you know, more of like an ephemeral thing. And even when there is state, it's again like, you know, like kind of like in-memory state. So for instance, like with inference, like you have things like KV caching and other kind of like components. But yeah, like I think the coolest thing is that there's like you don't need to write anything to like disk or like persist things like, you know, like long term if you don't want to.

14:35Like it's so like self-contained and like it's this loop. And even when you like start thinking about like, you know, kind of these sandboxes and like code execution and like, you know, agent actions and whatnot, you can still for a lot of these, like you can still maintain this sort of property of like, you know, like a closed loop system. there are some places where like you do need like state like I think memory systems are are one place where you really have to like think about like you know how how do you manage that like client side how do you like you know sync that across like devices and so on but uh but yeah like it's it's a really it's a really compelling application for for tees and in general even if there's like you know like rough edges like it's it's kind of crazy like how how well it like fits.

15:22James Wilson: And also, and again, I'm intentionally trying to, I so desperately want to delve into the details of it. I'm trying to keep it high level and then we'll get more technical. And I think one of the other points that I think I would be asking if I was sort of approaching this from like a C-suite person evaluating whether we need this, what benefit does it provide, is this notion of the trust boundary, right? Trust is such an overloaded term, right? As an industry, we've gone through zero trust and now thanks to all these AI and LLM attacks we seem to be going through, let's do zero trust, but for real this time.

15:55James Wilson: But in a network and application sense, the trust boundary often comes down to thinking about creating bridges between environments, right? You might have like your on-prem network or your office network and you want to be able to talk to your apps you're hosting in your Amazon data center. If you don't want that going over the internet where you're gonna have to set up things like vpcs and vpns and network transports and things like that you know trust boundaries sort of like you've got the boundaries and then you're working out how do you very selectively create the pipes between those boundaries in a safe way but i get the impression here but i'm not 100 sure but i think the trust boundary here might be a little bit different right if you were explaining it again to sort of high level leadership as to what that means is is it the case the tinfoil gets almost like a vpn sort of sort of typical way of thinking about the trust environment or is this a different way to think about that trust boundary i think that um yeah funnily enough zero trust is like the actual right term for this and like i understand that like zero trust is like so overloaded that it almost like means nothing at this point but like that is maybe like the best best way to think about it I think VPN less so because like um yeah like VPN less so because it's not just like there's like a hop and then it's kind of like going somewhere it's like here's a closed loop system where like someone else is running it yes but like they're not seeing like anything that's kind of like you know going through it.

17:33I think like maybe like the best way to think about it is like, you know, if I have like an on-prem system or if I have like, you know, like my local, like, you know, my laptop or whatever, I'm kind of like extending like what, extending like the computation that I can do onto the cloud without like giving anyone else in that path, like any access to the data. So that's kind of what I think about is like, you know, kind of extending the trust boundary of like, like, you know, my local device or like, you know, my on-prem, like, you know, data center or whatever. Yeah.

18:06James Wilson: And I think that's what I was trying to get at is that people shouldn't think of this as just like it's private insofar as there is an encrypted path to get to the inference engine. It's more and like that would be the traditional sort of like, you know, we've rented servers in the cloud. Okay, great. But is that private? Well, you know, yeah, but let's get, you know, we'll establish like VPNs between the head office and the cloud so that it's going over our path. Or maybe we've even got dark fiber, right? That would be the traditional way of thinking about that trust boundary. What I was trying to get at is I think the way you articulated that there, which is this is extending the trust boundary at an execution environment level, right?

18:48James Wilson: This is not like networky VPN sort of ensuring that there's private links. It's a deeper form of creating that trust boundary that, I don't know, I'm struggling to wrap the right terms around this. How do you help people understand this? Because it is difficult to. It's really, it's a challenging thing to like, you know, understand. And I think maybe another way to think about it is like, you know, right now you can spin up like a serverless instance on like, you know, like, say like Cloudflare workers or, you know, any number of like, you know, the sandboxing companies that exist. this is like if you had the guarantee that Cloudflare would not be able to see what's kind of going on inside your inside your application or the dark fiber angle is interesting I think that's that's actually like you know maybe in in practice like that's closer to like you know the technical like you know details of like maybe what what what you get because like with dark fiber you know like no one on the public internet can see it um the caveat is that like you know like whoever is like operating that um that data center that you're like connecting to even if it is with dark fiber like you know operating the service you're connecting to like the service provider can still see what's going on so you know if you're using like cloudflare or something and like you have like a private like you know channel with them cloudflare can still see like what's what's going on and in tinfoil's case like or with like you know things that you can build secure enclaves even they like even we cannot see what's what's going on I don't know if that's helpful yeah I think it is and look I think we'll move on from this point but I just wanted to you know when we go through threat models we're always asking you know even if even if even if even if right and this is the next even if which is previously even if you had encrypted the data on disk and and you know you're on the fly and decrypting stuff off the disk and running the program in memory even if you were doing that if you had physical access to a server and you could snapshot it then you're going to be able to get that data this extends that boundary up a level where it says in a trusted execution environment even if you can get physical access to the device it matters a hell of a lot less.

21:11James Wilson: Okay, so having tortured the trust boundary there, but hopefully that gets people understanding why this is different. One aspect of how a trusted execution environment in this context is different to things like a TPM or a secure enclave is just the sheer capability of it, I guess, right? If you think of a secure enclave and those sorts of very dedicated hardware in a device, they tend to be slow by design. They're very limited in capability. They've got a small amount of data they can hold, but they hold it very securely. This is different because, as you mentioned, the entire large language model exists in and the inference is occurring in the trusted execution environment, right?

21:50James Wilson: So we're not talking about a tiny pocket of memory over here. Is this the entire GPU's memory? Is it, you know, I'm trying to get a sense of what's the, where does this sit? Is this like a portion of the capability of a GPU, right? Take, for example, you've got a Blackwell class GPU. Is the trusted execution environment the entire addressable CPU and memory space of that card? Or is this still like a little carve out? It's as many resources as you give it. So you can give it like all the memory and CPUs or CPU cores, or you can give it like a smaller space. I think I don't want to like go too deep into this because there's like, you know, there's there's strategies that you can do where you're like almost like a testing like the entire the entire machine itself.

22:46though that's like you know less fleshed out but for the most part like you can think of it as like you know kind of like the primitive that you use is a confidential VM and like you know that's it's kind of like a regular VM where you can have like you know multiple VMs on like you know the same machine but in this case it's like you know a confidential VM which is like you know all its memories encrypted and then on the GPU side yeah this would extend to like you know like the whole GPU. So on Blackwell, like, you know, you can attach like one GPU. So usually like, you know, a Blackwell machine will have like eight GPUs.

23:23It's like, you know, one node has eight GPUs. They're kind of like connected to each other. So you can like, you know, have like inside one secure enclave or one confidential VM, you can have like one GPU or two or three or eight or whatever. And then it does cover like, you know, the entire space of that GPU. Yeah. And there is like, you know, I mean, some, some overhead, like there's like a 20 ish percent, 20 to 30 percent overhead on like, you know, like inference in terms of like, and it depends on what exactly, because like, you know, certain things are much less affected than others, like, you know, say like time to first token would be like differently affected than, you know, inter token latency.

24:05And this depends on how large the model is and so on and so forth. We have a very detailed blog post on our blog that you can check out. So there is some performance overhead, some of which would go away in the next couple of years as certain improvements come out. But yeah, that's kind of the space. Yeah, okay.

24:23James Wilson: No, that's good. I understand this better now. I was trying to just get a sense of, is this like a subset of the hardware that supports this? Or is this like a, like you say, a virtualization layer that sits on top of this that essentially treats the, I guess, the physical cores and the RAM as resources available to a layer that is then mapping multiple potentially trusted execution environments within that same unit such as that. I mean, is this inherently a multi-tenant solution as well? Like, do you take that one node and you can configure, is it unlimited number of trusted execution environments?

25:00James Wilson: Like, let's say you've got one node, 10 customers all using it for private inference. what's the actual primitives that get created there to support that multi-tenancy? Yeah. So for inference in particular, so if they're all using like, say like our public, like, you know, inference endpoints, actually it's all being served from like, you know, like one, one node or like, you know, one trusted execution environment because inference is stateless and there's like, you know, properties you can prove that like, you know, one person's like request does not really intersect with another. They're kind of these independent things.

25:36So it's not like we have a separate model loaded in for every single user. I mean, we have dedicated inference for people who want it, but that's not maybe the way to think about it. However, there are some restrictions. So for instance, one GPU cannot be split into multiple enclaves. So at least in the way that like nvidia confidential computing is implemented that's not a property that's uh that's available like they do offer this like you know outside of confidential computing it's called uh nvidia migs i think um i i forget like what it what it stands for multi something uh i'm not sure but uh yeah like when you're doing like a like a confidential vm or like you know creating a secure enclave it has to be like one whole gpu that you connect into it so you can't like fractionalize that.

26:29And I can get into the reason for that, like, you know, like really quickly. So with like, with the CPU, the memory is encrypted. And so like, you can have, you can kind of like, you know, carve out these like portions on the CPU. And like, you know, you can dedicate certain cores and like, you know, certain parts of the memory for one. On the GPU side, the memory is not actually encrypted. It's just all the connections to it that are encrypted. And the memory doesn't need to be encrypted as long as it's single tenant and cannot be like, and nothing else can be co-located. Like it has to be part of like the same boundary.

27:08And you still can't like tap into the memory because like there's no path from like the host machine that will like let you like, you know, directly connect to or like, you know, directly pull anything out of like the GPU memory or like, you know, VRAM. Yeah, without going through like, you know, this like CPU channel. So and there's like other reasons for that as well, which is that like, you know, if you can take the memory out, if it's like socketed, like, you know, then you can do things like cold boot attacks, and like read off the bytes, but with like GPU memory, like, you know, it's all like on board.

27:42So it's like, you know, the kind of like, think of it has like you know stuck to the board and like glued in um and like in order to like read anything off of that whether it's um in whatever state it might be and you would have to like shave like you know like the chip itself and like no one has ever really done a successful attack with that like

28:00James Wilson: that's extremely extremely expensive certainly not while it's powered on and serving that's for sure um okay so yeah so now i've got a better understanding of this i think so it's not So if I sign up for a private inference service, I'm not getting James's trusted execution environment provisioned for me. What I'm paying for is access to trusted execution environments that have already been set up in a way that because they are inherently stateless, there's no need for me to have my own, you know, James's specific trusted execution environment, my own little petition, etc. because I'm essentially gaining access to, it's a shared environment, but it's shared in a way where the sharing is not causing a cross of trust boundaries because my data is not commingling with others because there is no data to commingle.

28:46James Wilson: It's not persisted, right? Because I think that's the sort of mental model that a lot of people would have going into this, which is if I buy an EC2 server off Amazon, then my trust boundary is that that's been created on some hypervisor and there's an abstraction layer and that's my little space. But in a trusted inference environment, I don't get my own little space because the thing that is my own little space is kind of the end device that I've still got in my control. What I'm gaining access to is a trusted execution environment where that inference process can happen in a way where no one can take a peek or look into what that inference engine is doing, right?

29:25James Wilson: Is that the right way to think about this? Yeah, yeah, absolutely. Absolutely. You can go like audit the code and everything. I think that's one other part of it, which is like you can trace through and like see if there's like, you know, like any writes that would happen that would like, you know, commingle like your request with another users. and like, you know, there's basically like none of that happening. And there are like, you know, one or two places, especially with KV caching, where there can potentially be like a timing side channel that you can create. But like we take steps that we've again like written about at length that like, you know, prevent that from happening as well.

30:06James Wilson: Yeah. All right. So let's finally take this down a level and get into more of the really interesting technical detail. And I want to start with the analogy I've used before is that a large language model, it gets so much airtime and talked about so much. But if you took it as the car analogy, the large language model, right, that big chunk of weights and numbers, it's the engine block in the car, but it's certainly not the entire car. There is so much more around that. You've mentioned a few things already. KV cache, the key value cache that helps speed up the inference process. There's the inference engine itself, right?

30:45James Wilson: You don't download glm5.2.exe and run it. You download the model. The model is the thing that gets persisted into the GPU's memory so that all these massive amount of matrix multiplications can happen. But the point I'm making here is that there is a massive amount of software and executable machinery that happens around a large language model itself to allow this inference to happen. So from a security and privacy perspective, help me understand how does the trusted execution environment get extended into those other things like the KVCache, like the inference engine, tell me other components that are mixed into this to maintain this idea of this entirely private inference space, despite the fact that there's actually a lot more than just a large language model that's sitting on a GPU involved with doing the inference.

31:39James Wilson: a lot of this is still being is the domain of the cpu there's still a lot of stuff sitting in traditional memory these are traditional apps they just happen to be leveraging a gpu in a really novel way to do what they do so so help me understand the overall ecosystem and how you manage the privacy of that as well yeah so there's there's a couple components there um that we can maybe start to break down so um what i think like maybe one thing to think about is like, you know, what are you, what are you proving like about, about the system? And there's like, you know, you, you can prove something about like, here's like the hardware that it's running on.

32:19And like, you know, here are like the security configurations with it. And there's like, you know, like kind of the firmware, like, you know, all the system, system code above it. And like, you know, you really want to like, make sure that like, you know, you're using like, you're reducing the size of like, you know, like the code that you're trusting there. Because like, you know, how can you have vulnerabilities and so on and so forth? Like, you know, there can still be like software vulnerabilities, right? Like there can be like, you know, compromises at like each level of the stack. And so like the less code that you have to trust, the better.

32:53And so, you know, like maybe you want to make sure that you're only pulling in like, you know, certain packages from like, you know, certain like trusted like vendors and not like, you know, just like some random, like third party, like dependencies and so on. So yeah, like next step is like, you know, shrink the size of like, you know, the system software that you're trusting, get rid of like, you know, like random things and like the user space. And then you get to like the application code. It's very interesting because like, if you look at things like, you know, GCP is CBM or like, you know, like Azure for sure, they don't really extend this like trust boundary into like the application code itself that's running or like, you know, say like the model weights that are running.

33:37Because that's another like very important component of this whole thing. Like they kind of like, you know, stop at like, some of them stop at like, you know, this hardware firmware layer, some of them stop at like, you know, the system layer. So anyways, like we really like, you know, wanted to like attest to like the entire stack. So you can actually prove that, you know, this is exactly what's running, like, like the inference engine, like, you know, that's a tested, like the, the model weights, that's, that's a tested and pulled in, in like a certain way where like, you know, the hash of the model weights is like committed to and like, tell you like exactly like, you know, what like the hugging face like commit is that we pulled from.

34:15And like, you know, that's something that is like, bit for bit reproducible. And you can actually, you know, generate that offline and check that like, you know, tinfoil is serving you exactly like, you know, these, these sets of weights. So, so yeah, there's, there's like, I think, like, you know, like kind of the biggest, biggest piece there is just like, you know, making sure all of that is attested.

34:33James Wilson: The next, the next thing there is, like, how do you make it maybe like, usable to like, you know, like, different people that are like trying to build like services. So like, right now, we, we generate like, you know, like, so we started out by just generating like a TLS certificate, like inside the enclave. And like, you know, when you would connect to it, like you would terminate your connection like directly inside the enclave and that's how like you know like tinfoil doesn't have the private key like we can't decrypt your connection it's just like standard tls so instead of it you know like ending at like our regular servers it's like ending in the enclave and that's why we don't have this ability but like that makes it hard for like you know people that are like building um applications of of some sort and like you know they need to do like billing or like you know proxy the connection through their back end or Or like, you know, they like really simple use cases, like, you know, they just have like an API key that like they don't want to put into their client, even though the connection should be private from like the client to the enclave.

35:30So like, how do you think about that? So, you know, like we kind of had to like design this like almost like stripped down version of like, you know, Oblivious HTTP, which is like, which basically like encrypts your inference request. So like, you know, like the request response with a key that is like generated on the enclave, but like the headers can stay transparent to like, you know, like a proxy server. And so like you can use that header to like, you know, put in like, hey, like number of tokens that were used or like, you know, like here's how much you should pay or like, you know, inject the API key.

36:04So that's like, you know, some other kind of things that you have to design. um another uh interesting thing here was um uh I guess thinking about like you know again like how do you do uh prompt caching is something that like everybody cares about they want to pay less for like tokens we want to like you know cash prompts but um if you do that then then you run into this problem where like you know technically you can have like a timing side channel um where like you know like someone else can like probe someone else's prompts and like see like you know what exists in the cache and like be able to reconstruct things.

36:36And so we kind of like, you know, do this thing called like KB cache salting, where like, you know, the client injects like a secret that Tinfoil doesn't know about. So Tinfoil can probe it or like, you know, any other user can probe it. That's separate from the API key, which is like, you know, what Tinfoil does know about. And then that lets you create like a cache namespace that's like isolated to that particular like end user. And like, that's kind of how we like, you know, mitigate like you know that that one other like um side channel that can happen so there's there's just kind of like a number of these like different things that you have to ensure i think like other things to think about is the the boundary of like um what what goes on uh here like like sometimes like a way to think about it is like you know you can do like whatever like stuff you want inside and like it won't like affect the privacy as long as nothing can like leave like whether through like a network request or like a disk write.

37:31So like enclaves are great because like there's no persistent state. So there's no like disk writes that can happen. But you can still like, you know, like have like network connections. And so here, like, you know, you can think about how do I limit like egress or like how do I harden egress to the point where there's like such minimal like, you know, logic that exists that would like ever like be able to like, you know, break out of this thing. And like, you know, maybe I have like a specific set of domains and then just very little networking logic that would be bypassable. There's, yeah. I don't know if that's like, you know, I'm trying to give you a sense of kind of the different aspects of how you think about it, but I'm not sure if that's kind of what you were envisioning.

38:13James Wilson: That's really good. I think there was three things in there I heard which are really important to call out from a security practitioner and privacy perspective. The first is the TLS session itself is actually getting terminated in that trusted execution environment, right? And that's what I was trying to get at is I was trying to understand how much of this is more like a traditional server where there's an element of you've just got to trust us, right? We'll attest to how this works and we'll explain to you all the versions and et cetera. But I was trying to understand, is there a point where the TLS connection terminates?

38:42James Wilson: The session is visible in some sense, in a traditional computing sense, and then it gets handed off into this trusted execution environment. But actually, it's direct TLS termination into the trusted execution environment. And so there is zero clear text visibility of the actual LLM transcript outside of that TEE, which is super important, first thing to know. Now, that gets us into the trusted execution environment with our transcript, which is stateless. Great. We know that's important. But the KV cache is the next thing to really pull apart, right? This thing gets mentioned in passing, but I don't think people quite understand why it's important and why it's so pivotal.

39:18James Wilson: because that transcript is stateless and there's no state persisted of like a server-side rendering of that that would allow it to pick up the next session easily. What would happen without a KV cache, and you tell me if I'm getting this wrong, what would happen is the entire transcript would be re-encoded into sort of the token space representation such that then the GPU can work out, well, given all these tokens that have come before me, now what is the next and next and next token to create for the next response? that would be hellishly inefficient especially when these contexts grow and grow and grow up to a million tokens so the kvcash sits there and says well um you know you're sending in this transcript and i've already seen that before and i've got this key value cache where i've already cached the token values great i get them out of the cache i don't have to re-encode them and then i can then hand them off to the gpo and allow allow it to just start doing the next message right that's the efficiency gain we get but the problem from a privacy perspective is there's the there's all manner of things I could do here.

40:16James Wilson: Like I could send a request and work out, well, how fast did that come back? And I can know what sort of tokens then must've been cached. I could get from that, I could get a sense of who else is using my TEE, trusted execution environment here based on what's coming back faster. I could, it's like, basically I could, there's all manner of ways I could poke and prod at that key value cache to get an understanding of what other people are doing in this execution environment. And so that's the second thing you called out, that you've solved that by essentially having a salt there, which is not known, you don't see it, but that automatically petitions that entire key value cache space, right?

40:53James Wilson: Because my salt's not going to be the same as your salt. Therefore, the fact that some tokens seem to return or some style of messages returned faster isn't an indicator of anything other than that salt happened to match a key value cache, but that's going to be because of my prior messages, right? So I'll stop there those two is that have i got that right so far yeah yeah exactly yeah uh and i think like you know i mean caching and like site you know timing side channels have just like always been a thing like it's not just like a trusted execution environment thing or like you know like a kv cache thing like there's there's like a couple things that are like always like you know like a trade-off that we like that compromise security and privacy and like caching is is is one of those because the whole idea of caching is you're doing something faster, you have some state, and then that can be exploited by someone else to figure out what is the thing that you did faster.

41:47So it just applies throughout internet infrastructure history. Right, right.

41:52James Wilson: If there's a precursor set of actions that make a subsequent step different in behavior, then those are all the ways that you can infer what those precursor steps were. Okay, so I would love to delve deeper and deeper and deeper into the technology here, but I think this is an important part to almost come up for air and say, look, the solution is technically brilliant. And it's super interesting how this, sorry, I should say before I get there, there was a third point that I wanted to mention. So you've solved for the fact that the TLS terminates directly into the TEE. The key value cache, which is the primary caching thing here is solve with the salting.

42:28James Wilson: But then the third thing you called out there is that this trusted execution environment deliberately by design or actually by physical constraint does not have the ability to persist or make egress. Like what's the actual mechanism that prevents that from happening, that prevents things from getting out of the TEE persistent? Yeah. So there's, there's nothing that like actually prevents like network egress, like every like trusted execution environment does technically like, you know, let you like send like network requests out. It's, this is where you almost have to think about like, you know, sandboxing techniques a little bit or like, you know, like firewalls and whatnot.

43:15I think like the biggest thing that it gives you is like you can you can kind of like specify like here are like, you know, like the domains that I want to connect to or here are like, you know, the package managers that I maybe want to fetch from. You can audit that and you can verify that this is exactly what's going on. And maybe the other part is like you can like shrink, like you can get rid of say like, you know, like a ton of like there's so much like stuff in the user space that does like network parsing or like, you know, that can be used as like a side channel. I think like, you know, when, like you can think of what happened with like the recent like OpenAI hugging face like incident where like, you know, like the model was able to like, you know, use like a vulnerability and like, you know, just kind of like, I believe one of the package managers and kind of like, you know, like exfiltrate its weights out.

44:03but like if you shrink how much like network code or like logic there exists and you make it like really dumb like you know what it can actually do then you can like you you're basically like hardening like you know the the surface and like hardening like the boundaries and like you know punching like very specific holes that are like hard to like you know kind of like um use in in

44:25James Wilson: different ways um and then yeah and so we're not talking about something that is like um this is not a hardware bound control. It's not like there is no physical access to a, you know, ethernet adapter, or there's no physical access to a disk subsorage system or a storage subsystem. Well, disk is different. Disk is different. So there is no persistent access to like a disk system. Like an enclave is an in-memory thing. You can write things to disk if you want. The way that we've like structured our CVM image is that that fails like you know at the station and like there's like no way that you can actually do that but like you can design systems where like you know like that is that is allowed but like it just gets like really complicated it's not like a first-party thing because again like trusted execution environments for the most part are like they're like an ephemeral thing like they're like an in-memory thing yeah okay that's interesting so I so my my understanding was a little bit off there but I think what I'm what I'm hearing from you is that the it is a bit of a combination of um i guess hard and soft um yeah controls here right there might not be a disk ride subsystem but by virtue of the fact that you've got the tls connection being terminated directly into the te it's got to be able to do networking of some kind um which would give rise to those egress okay so i want to come up for air just for for a bit here and say look technically a brilliant solution and some really amazing engineering has gone into this.

45:58James Wilson: But who's this for? Why do we need this? And I'll almost get to the, maybe we start at the pointy question first. I only need this if I'm doing bad stuff, right? I think that that is, it is less and less true. Like, I think that, you know, like people, people said that about like Signal, like, you know, people said that about end-to-end encryption. People maybe even said that, like, I wasn't around for that. I haven't read the literature. But like, maybe people taught that about like, you know, TLS over the internet that like, oh, like, you probably only want to hide your your financial credentials from your like, you know, ISP or like the Starbucks Wi Fi system, just because you're doing bad stuff with your credit card, right.

46:44So I'm not sure that that's like, you know, like, I think of this a lot more as like, you know, just internet encryption, Like having internet encryption allowed people to use the internet in vastly more, you could say, like, you know, real ways. Like you could sign like zero data retention agreements and stuff with like every ISP and every like, you know, like Wi-Fi operator in like the plane or at a Starbucks or whatever. and maybe you know maybe that is a world that we should have lived in but I don't believe that that to be the case I think it's it's best for everyone to just have like internet encryption and you know who you're talking to at the end of the line and you can get there securely and like you know you you don't have to like rely on the on the goodwill of like um you know all these all these wi-fi people um operators and like you know isps to like you know kind of channel that so so that's that's kind of my take but I think like AI is like even crazier than that because like you know to some extent this was still all like very structured information I think with AI like especially as you get into like agents and you're getting into like um you know things that are just like always on like always always online always listening um like you're um like maybe at some point like you know you have a chip in your brain or something and like you know you have AI on there that's like, you know, sees like everything that you see and does, and yeah, like has all the context that you need.

48:16Like, and, you know, again, with AI, the more context you give it, the more useful it becomes, like, do we really want this context to be shared with like, everyone that it lies in this particular path? And like, you know, for instance, like, do you, if you, if you use Neuralink, and you know, you have a chip in your brain, like, do you, really necessarily need like, you know, Elon Musk to have the ability to like read your thoughts.

48:42James Wilson: Well, I think that's a very, that's a very pathological example. I think a lot of people would say no, but I take your point here that from, from a, you know, view of the global world and the global benefit. Yep. I get it. It's just like an encrypted internet and encrypted transport. But again, and I'm intentionally pushing back here because at the moment, like right now, this is still uh you know there's performance overheads that that's going to mean cost overheads um things like internet encryption we've gotten to the point where that's just like so it's on for everyone it's just assumed you're not having to make a choice of do i want to actually pay a premium for this anymore but in this case this is still in the realm of a deliberate decision that someone is making to say i want to pay a premium or i'm okay with taking the performance et cetera, I'm okay with all the caveats and carve-outs that come with this because I need, for whatever reason, this level of privacy.

49:38James Wilson: And I still think that's a little bit different from just the overall encryption is good, because this is still a tool and a solution that someone is deliberately choosing, paying more for when they could have used something else. And so I just want to push a little bit more on this. I still feel like there is an argument people could say that, you know, at the moment, because private inference is not just the way all inference works, because these are niche players that are providing this, because it is a trade-off of performance and cost, it is still something that is, you know, you probably don't need to pay that premium unless you're actually doing something bad that requires that level of privacy.

50:16James Wilson: Like how, I guess, put it another way, how do you think about the defensible argument here of why should your service exist right now? Mm-hmm. Yeah. Yeah. So I think that when I mentioned like, you know, things about the performance overhead, to put it into perspective, like when I say like there's like a 20 % performance overhead, for instance, like on like the inter-token latency, that means like you're going from like a couple milliseconds to maybe like an additional millisecond. And like for a lot of use cases, like that is really not the bottleneck. And like, you know, there's like other things that are like much less affected, like, you know, like in terms of throughput and, and so on and so forth.

51:00So unless you, you need like, you know, extreme fast latency, like blazing fast, and like, you know, you for for the average user, like, you know, like the, that is like, not really perceivable. And like, you know, again, like, we can only really run this on like, on like the latest, like flagship NVIDIA hardware. So like, you know, a lot of inference providers that would be serving you tokens aren't necessarily like, you know, serving it to you on like, you know, the flagship like B300s where we kind of have to. So even when you kind of look into that, like, you know, like the, yeah, like the speeds are really not like that big of a problem.

51:40And even then, like, you know, there's like specific reasons why that happens. It's not like an intrinsic property of the hardware. And you can like make changes to like, you know, some of the kernel mod, like the the GPU kernels, the way that they're written, like rewrite it. And there's like very specific interventions you have to make and you can get that overhead, like down to like, you know, like the, a couple percentage points. So, which is like, you know, something that we do and for, for like our model. So yeah, like not out of the box, but, but, but yeah. And then in terms of pricing, our pricing is like maybe like 10 to 15%, like higher than like, you know, like other inference providers and like a large part of that is because we want to like validate whether people are actually interested in paying more for privacy and like right now our biggest bottleneck is that like it's really hard to find compute and like and like you know we're not able to like keep up with like you know like the growth that that we're seeing so um okay so i i was sort of i mean maybe

52:47James Wilson: as you're talking, I think I came up with a better way to articulate what I'm trying to push on here, which is that let's look at cryptocurrency. Brilliant amount of engineering, like just incredible. But because of all the friction involved in using it, right, it's slow in terms of transaction volume. It's difficult to even acquire the cryptocurrency in the first place because you've got to go through all these KYC sorts of things, right? It's a brilliant technology, that because of both, I guess, intrinsic and extrinsic friction around the system, its sole use case is crime, essentially, right?

53:25James Wilson: Yeah. And I was trying to, I guess what I was trying to argue with you is, convince me that private inference is not just the next cryptocurrency, where it's like technically brilliant, but its purpose is crime. Yeah. Well, so that's an interesting way to maybe think about that. I think that this is maybe, at least at a personal level, this is maybe the first, one of the few places where I really feel like I need privacy. like when you are interacting with like, you know, a model, like, I think like, you know, there's, there's places where like, I personally don't, don't care as much, like all of our code is open source.

54:11Like, I don't really care if like, you know, like someone on the other side is like looking at like, you know, like all, all, all the code. There are companies that care a lot. And like, you know, we have like customers that we work with where like, you know, they have to hide a lot of like credentials and stuff. And like, they have to kind of almost build this parallel system. And it's like so much easier to just like, you know, use like private inference and like, Like, you know, just never have to worry about redacting credentials or like, you know, passwords or whatever. There's like obviously like, you know, like we have like number of like enterprise customers.

54:43Lawyers love to love to use us for for for various reasons. And like one of them is like, you know, your your chats with like chat GPT or Claude or like subpoenaable in court. And like that's that's obviously like kind of like a problem. then obviously if you have like sensitive IP, that's like, you know, maybe you're like a financial, like, you know, you're like building like, you know, like high frequency trading algorithms or whatever, like, you know, you, you want to like reserve that data. And then, you can also think about like, you know, on the flip side, like what are these model companies like incentivized to do with, with that data?

55:17Like they're, even if they say that they don't train on it, like, you know, they can, they can analyze it in like various ways. They can use that to understand, like, you know, like almost using it as like an eval set where like, you know, they can understand like, you know, what are like the places where like, you know, certain things are failing and whatnot. So there's like a lot of like, you know, information that you can gather when like, you know, people are like putting like their entire lives like, you know, into these models. So I think that's all. And then there's also something to be said about just like, you know the amount of power uh that that having like access to like everyone's data like gives like a couple of different you know companies um the governments uh in this world um like that's something that we really like care about which is like you know like concentration of power like you know avoiding that thinking like you know like a real like democracy like you need to have like the freedom to like you know workshop things in like private like you know have like you know like discussions and stuff.

56:19And as more and more people interface that with AI and like collaborate with AI, like it's kind of scary to think about like, you know, like, hey, like I'm trying to like, you know, like, like do like some sort of like antitrust lawsuit or like, you know, like some international treaty coordination about like, you know, stopping AI or something. And like, I have to send, as I'm talking through like an AI model that's like hosted on like, you know, US soil and like, you know, one or two companies, they have like a full view of exactly like what I'm doing, like, you know, everything that I've kind of discussed.

56:48So, so like, it just gives, yeah, there's just like so much concentration of this information in like the hands of one or two players that that's another kind of thing that we really care about.

56:58James Wilson: I think we are when, not if, when the first breach of a frontier lab occurs, perhaps that's when the public reckoning with this comes to life. Because you're right, it is so convenient today to throw anything and everything into a chat GPT or a similar sort of LLM. I don't think people realize just how much of that is being captured and used. But okay, so again, if we break down my argument there of the comparison to Bitcoin, it's not so much that the intrinsic properties of private inference make the friction such that it's a big complicated step up. It's not like we're paying, it's not like you're charging five times more to get one 10th of the performance and therefore people would only do that if they were doing the worst of the worst things, right?

57:46James Wilson: It's actually, it sounds like the intrinsic friction here is actually quite low. It's still as performant. There's a 10 % or so premium being paid. So it's not outside the realm of people making a reasonable decision of being happy to take that little performance hit and a little bit extra financial hit to get that privacy. On the flip side though, how do you feel about things like KYC and the level of governance that you've got to maintain to make sure that you know? and I loved your post on Reddit when you were like, hey, we're 10-4, we do private inference. And it was like four or so comments down where someone was like, please accept Bitcoin payments.

58:21James Wilson: And people were like, no, no, no, no, no, no. What's your view on this for governance and KYC? Yeah, before I get into that, another thing I wanted to reiterate is like, we really care about UX. Like we have cared about that from like, you know, day one. that's why like you know we started off with open ai compatible sdks that just like that work that do all of the verification automatically like our our chat in our browser like you know that we had from like you know day one really like that did all the verification like automatically in the browser and just to explain that to perhaps a non-coding developer audience what that means is you can take your code base and swap out like import open ai sdk for import tinfoil sdk and Maybe there's one or two other line changes, but that's it, right?

59:12James Wilson: You've actually gone to the point of replicating the entire ecosystem around that. Compatible SDKs that do the attestation and all those checks for you, right? And that's a big deal. That would have taken a lot of work. Yeah, yeah, exactly. So we really want to make this not something that people who are only extremely motivated have to configure. And it's like configuring your whole tails, like a store set up or something. Like this is something that like, you know, when you use Signal, when you use iMessage, when you connect to like, you know, some some internet server, like you don't have to like necessarily know the details of like end to end encryption and like, you know, like TLS, like it just works.

59:52It's secure. And like, you know, and if it doesn't work, like it fails, like, you know, very loudly. So, yeah, going back to like KYC and governance. So, yeah, I think so anonymity and privacy, like it's there. There's separate threat models. They're like, you know, separate things that you kind of have to think about. We don't try to like, we're trying to do one thing well, which is like, you know, provide privacy and confidentiality. We, at this point, like do not try to provide anonymity, partially because it's like extremely complicated to do so in like an accurate way. because like, you know, it's like the last mile of that just like introduces like severe amounts of like complexity.

1:00:39And like, we think that it's better to have like a very clear, like, you know, precise threat model and really like try to execute towards that in like a clean way than to like, you know, kind of like increase your threat model a lot. And like, you know, then just kind of do like a slipshod, like, you know, like a solution that doesn't like kind of like cover like all the all the pieces so so that's one part of it other governance so you know like it's funny you mentioned crypto payments like a big part of that was like we just did not want to like deal with like you know kind of setting that up it also creates like fraud abuse like harder to like you know like manage but Stripe recently introduced like just like an easy way to do like crypto payments so like we we enabled that but it is good that in in case of like you know something going wrong like you know at least like guess like stripe has that data potentially so so there's there's there's parts of that um I think um I am not at all opposed to kyc actually I think that we're not at all like like we're actually very interested in like you know exploring like safeguards that can be like transparent and like publicly verifiable I think like what's very like it's it's interesting because like you know like you can basically co-locate like you know classifiers or like you know probes or like you know however you want to do your like safeguards um like in in a te um and like you know just kind of like run everything and like you know you can have like a public policy like you know like um anthropics written a lot about like you know constitutional classifiers i wish that there was like more like work out there um that showed like how to build good classifiers and safeguards.

1:02:21I think that is very interesting stuff to explore. OpenAI released GPT-OS as a safeguard. We actually use that as a privacy filter for search queries that go out into our search provider. We've kind of written a lot about that as well. But yeah, that is something, especially as the models get more and more powerful and you're using it for more agentic tasks, Like we're still like exploring this space. I think like the biggest bottleneck here is like, how do you build like good, like, you know, classifiers? There's a surprisingly like low amount of like open source work in this space where like you can just kind of like take something like GPT-O says safeguard is maybe like the best one out there.

1:03:04And like we're already using it. But yeah, like you can co-locate this in the TE and like, you know, make sure you run everything. And like you can kind of build like interesting properties. So like, you know, one thing that you can do is like, you can say like, hey, like, we're going to like, if you violate this thing, like, you know, like more than like three times, like, you know, your account will be banned. And like, if you want to like, protest that, then you can like reveal your transcript or something. And like, you know, we can, we can talk about that. I think there's like excellent like governance mechanisms that you can do in like a privacy preserving way that people are just like, surprisingly, like not exploring very much.

1:03:40And like, we've been talking to like a couple, like, you know, larger companies that are also interested in like working with us on this. So we'll report back. But yeah, like, I'm very interested in like, you know, publicly verifiable governance or like publicly verifiable safeguards. I think the transparency is actually like a really cool like property that you get. And then one side note on that is like, unlike, because like, you know, there were all these debates during like, you know, with like end-to-end encryption about how do you do like, you know, detection of like, you know, certain things.

1:04:10And there, that actually involves like weakening the threat model because like now you have to like send stuff. Like, you know, Apple had that whole controversy with like, you know, the CSAM. Yeah. But in this case, like you actually are not lowering like any of like the security guarantees or the threat model, like you're already trusting the TE for your inference. So if you co-locate like the safeguard thing in the TE, like you're not actually lowering like, you know, like the boundaries of that. So, yeah, I think I think that's that's all like, you know, very interesting stuff. And I think, like, KYC is probably just going to become, like, more and more important in, like, conjunction with that.

1:04:54James Wilson: Well, so that's an interesting point there around safeguards being possible, right, guardrails, et cetera, being possible, classifiers being possible in the T itself, right? Because I think the other thing that I think people assume here is that private inference is synonymous with open-weight models, which is synonymous with un-guard-railed models, uncensored models, which is synonymous with you only need it if you're doing bad stuff again, right? It comes back to that old argument. And I hadn't, until you mentioned it there, understood that actually things like classifiers, guardrails, right, input classifiers, output classifiers, models that are examining the outputs of the primary model to make determinations about safety and risk, et cetera.

1:05:34James Wilson: You could have those all in the TAE, which is a neat way to explain that a private inference provider is not just serving up unguarded models and have at it. They can bake in their own guardrails. I would imagine, though, that you would be running into the problem of how do you monitor how effective those guardrails and systems have been given that you don't get to see the data, right? What do you do there to make sure that your guardrails are actually beyond just good faith? Yeah, I think that is the hardest part of doing like, like building like private systems in general. Like, I don't think that that is restricted to private inference.

1:06:17Like, for instance, you know, like a bunch of people at Signal, like, you know, have spoken about like, you know, how hard because they don't have any like metrics and like they don't like, you know, track like a lot of different things. Like, it's very hard to understand, like, you know, what is failing and what is breaking and so on and so forth. So, yeah, like that is always like the challenge here, which is why we really believe that, you know, like because we're using like very standard open weight models, we want there to be like more work on developing guardrails and like security, baking in like security into them that like other people can do.

1:06:57And like, you know, again, because like if we were training our own models, like we would need like very bespoke, like handcrafted things like around it that, you know, maybe only work for like our set of use cases. But because like we're we don't train any models, like we don't like, you know, we don't serve. We only serve like models that are publicly available on like other inference providers that have like, you know, like large adoption. we think that there's like ways to do that like you know externally through external data sets that's kind of how we did like when we were trying to develop like our privacy masking filter for like you know like search queries that that go out we there's like a number of like data sets on like you know like on like you know private information that like you know contain private information you can like you know run like evals against those right but basically like a huge

1:07:47James Wilson: A range corpus of like, this is what a URL looks like if it includes an API key or something sensitive, right? Exactly, exactly. And like OpenAI also recently released like a privacy masking filter model, which we've been using as well. And like, you know, so we almost like, you know, call upon like other organizations to do like more of this work. Like I'm still a little bit like surprised at how little there has been. Like, you know, people talk so much about open weight models being like, you know, unsafeguarded and like unguarded. Like you have to understand that most people are not like running these in their basements.

1:08:27Like these are like really large models to run, like, you know, and like most people are using this through like inference providers. and like if we can like you know um create like you know better like a better ecosystem of like safeguards and and guardrails like or even like you know publish more about like the practices that you need in order to like create this stuff then like a lot more like um companies like you know not just private inference but like you know like a lot of like other inference providers may be like interested in kind of like you know running like more more of that so yeah that's

1:09:00James Wilson: kind of like our stance there so that's an interesting point right you're intentionally trying not to create any bespoke models here because you want to, I guess, benefit from the attestation of this model is exactly the weights that are out there in the public domain, which is great for the base level model. You've got increasingly more and more choice, but without that sort of, I guess, open development of, or I guess, open weight classifiers and open rate guardrails, you're kind of hamstrung by what's available. I guess maybe naively the first reaction to that would be, well, of course, there's not a lot of public publishing of models that are classifiers because that will only make it easier for people to work out how to get around that classifier.

1:09:39James Wilson: But I would think that that's kind of the point, right? It's like, yes, publish them. Let's pick them apart. Let's find where the weaknesses are and let's build better ones. Is that sort of where you think this needs to go? I've never believed in security through obscurity. I've always believed that it's like, if you think about it, I mean, And like a lot of like the publishing that has like come out of Anthropic versus OpenAI on this is like, you know, Anthropic did a really good job with like, you know, building these sorts of like constitutional classifiers that weren't like nitpicking like specific instances of things.

1:10:15It's almost like, you know, you align the model to like act a certain way. And then like your classifiers are also like relatively like they're more like rule based as opposed to like exception based where they're like trying to like, hey, if you see this thing, like, you know, freak out. And like those are like easier to jailbreak because like, you know, as opposed to like more like semantically like, you know, I think like the other good thing about this sort of like approach is that as the models get better, you expect this to be like, be like harder and harder jailbreak, which is again, like, you know, something that like these, these, these companies have like spoken about.

1:10:53And like, you know, like we've kind of seen just generally happen. So I think like that's that's one thing that's interesting. And I think like, you know, OpenAI for a while, like the way that they were doing some of this work was a little bit more like like I just remember reading like a lot of like the early like jailbreaking like research or like, you know, like the attacks that people would do. and like they would target like these like specific like very you know tightly crafted instructions and then you can just get around it as opposed to like more yeah and and like another way to think about it is like you know like the public domain like what what do we have like we have laws that are public the constitution is public like you know the set of rules that you have to follow as an individual are public like don't don't kill people don't do x don't do y like you know and like yeah like there are people that like fall through the cracks and then you get Like case law and like all that sort of stuff.

1:11:47But like the things that you're really trying to prevent for the most part, they can actually be public. So, so I think like, I think I'm maybe like, you know, like oversimplifying it in some, some aspects there. And like, I think that there's like, you know, a lot of like very interesting techniques behind besides like, you know, just kind of like, yeah, telling it what to do and then picking that apart. I think another thing is like, you know, probes and like, you know, looking at the activations of of weights like i know that a good fire for instance like you know like kind of like open source like a lot of like tools for that but i know that um you know both like anthropic and open ai like use like probes as well in addition to just like you know like uh standard like classifiers um in order to figure out like you know like certain things about model behavior and responses um and there's like some like open source like projects like around that as well um that kind of like do like more of this like interpretability type thing, like checking activations and stuff.

1:12:45So I'm like, I'm hoping that there's just like more like work poured into that. And like, you know, like, yeah, there's already been like a lot in like the last like few months. So I'm just hoping to see that become like more, you know, commoditized, so to speak, and like, you know, easier to like, kind of just like roll out and deploy.

1:13:02James Wilson: I guess this is sort of the last thing I wanted to touch on as we wrap up is around open weights, We're at this amazing point in time where we've simultaneously got the most powerful open-weight models hitting the market from Chinese frontier labs, as well as what seems like the strongest indication that there might be regulatory interventions soon that could prevent US companies from using these models, hinder the Chinese ability to train them. We don't know what form they're going to take, but it's just a funny moment where it's simultaneously so exciting to see these developments and also a little bit nerve-wracking for the moment to see what the future is of this.

1:13:39James Wilson: Now, I don't want to just say, hey, your business model is founded on open-weight models. How are you going to survive? I think there's something a bit more interesting here, which is private inference in general. Today, it is the domain of open-weight models being served in a very novel way. but does that remain the future for this or what what happens if a closed weight um you know provider gets uh very good at doing private inference and then everyone just flocks to that and that just kind of weakens the need for people to even need open weight models does that drop off the demand i'm asking you to sort of step back and take more of a an industry view here and say like what is the ongoing shape and form and importance of private inference when we've got this almost fork in the road now of open wait versus closed wait.

1:14:30Cool. Okay. So I think that open waits allow you to do private inference in kind of like, if you're able to run this locally, that's kind of the best place to be. The second best place to be is if you need to run it in the cloud and like, you know, that's when you need quote unquote, like private inference. I think that if closed wait providers started offering, you know, verifiably private inference, which I don't believe that they will, there's like no incentive to do that. We've seen them repeatedly going in the other direction. So for instance, with Fable, like, you know, there is no zero data retention even that can be given to enterprises.

1:15:17James Wilson: That's okay. It's useless for anything security related anyway. That's also a fantastic point. It's something that we run up against like, you know, every day. Yeah, I think that I don't see open weights going away. I see them being harder to access maybe in certain countries. I don't think that the rest of the world is suddenly going to like, you know, give up and stop training models, even if it's just for their own sovereignty. China is already building a lot of their own chips, like, you know, they, like, a lot of the recent models have been trained to some extent on Huawei chips. China is extremely good at building hardware there.

1:16:07Even if they don't have like, you know, all the specifics, like, you know, fully fleshed out yet.

1:16:12James Wilson: I think I'd just push it back a little bit and say, I think China's very good at designing hardware. I mean, like chips can't quite build them themselves at the moment, right? That's the sticking point. Right, yeah. But I think that that's not something that is going to be impossible for them to like overcome over time. Because again, like with AI, like you can build very restricted subsets subsets of chips that are still extremely powerful and useful. Like you don't have to build like, you know, every single capability all at once. So it's not like, you know, some like super general purpose thing that you have to build.

1:16:51And like, they do have access to like cheap power, which is like another big, big component. They have, you know, access to a lot of like, you know, like other like manufacturing equipment and whatnot. And like, you know, better supply chains like that, that the U.S. like, you know, also sometimes struggles with. So I, I think that it's more a question of time, as opposed to like, there will never be open rate models if like, you know, like the U.S. bans them. I feel like I, I, I think that there's, there's a lot of different camps and like a lot of different people, some of them who have like, you know, kind of their their own agendas where like, you know, it benefits them from like their business model front.

1:17:37And I think some who have just kind of like, you know, blinders about like what other facets of this means. So, you know, like, let's, let's get into like, you know, for instance, I think concentration of power is like, is a big component here, right? If there's only, even if you have private inference, and there's only one or two companies that are like, you know, kind of doing that, you still don't alleviate the sovereignty thing. You still don't alleviate the concentration of power thing. Because at any moment in time, like, you know, if they have that much power, they can turn, you know, quote, unquote, turn that off or like, you know, just break that in some way, shape or form.

1:18:18And I also don't believe that they're going to make everything open source and like, you know, have like a fully verifiable chain. And there's like so many places that you can kind of like attack that. So even giving like people outside these like one or two entities, like some degree of leverage, even if it isn't like the absolute best frontier capability is still extremely valuable because like for a lot of things that people are actually trying to accomplish, like you don't necessarily need like the absolute frontier capability. There are certain things where like you kind of like hit saturation and And I think for certain coding tasks, you absolutely need that right now.

1:18:54For certain long-running agent tasks, you do need that. But if you're trying to compose emails or just kind of brainstorm about certain sorts of legal structures and whatnot, maybe you don't need Fable+++. So I think that's where they can be extremely powerful. I think the other counterpoint there with respect to like safeguards, I think it's really useful for these models to have like, you know, built in safeguards. I wish that there would be like, you know, I really wish there would be more publicly available like research on how to do that. I think like, you know, a lot of, you know, you hope that like, you know, a number of these like companies, whether they're Chinese or not, like would be interested in at least like understanding like how to build this.

1:19:42Like nobody wants to accelerate like bio risk. Cyber security, maybe less so. Like here, I think like we've almost like leaned too far into the safeguard territory to the point that we're like actively like harming defenders substantially and like kind of making the world more insecure place in some ways. and I do believe that there will be like a period where this is maybe kind of like a more controversial take but like I believe it's okay for us as a world to go through this like period of like you know an increased amount of like cyber attacks and and and so on and so forth if it kind of wakes everyone up and like people start like patching a lot of this software and like hardware and like you know we end up in a place where like things are like a little bit more like robust but you know security is like never 100 % solved it's always an arms race like it it's been since like you know the this the start of time so I don't know if that's uh yeah

1:20:38James Wilson: yeah I think what I'm hearing from you I think what I'm hearing from you is that it's not about it's not about open weight versus closed weight in the private inference space right because it is I think it's another example of where a closed weight is not a drop-in replacement for an open weight for so many reasons and what you've articulated there is actually a case I hadn't considered, which is the implications on private inference. Because if you take the example of private inference serving an open-weight model that is bite for bite, the exact same model that's publicly available, there's an additional level of trust there and there's a durability of knowing that it is always going to be that model that behaves in that particular way.

1:21:15James Wilson: You can depend upon it not just being private, but what is private is dependable in terms of its, I guess, properties. Now, a private, a closed-weight company, there's no technical reason why they can't do private inference. It's all there for them, right? But what I heard from you there is that the incentives don't line up for them, first and foremost. That's why they don't do it. But also, even if you could get a private inference offering from an Anthropic and an OpenAI, it's not the same thing in terms of overall benefit, because you're essentially getting private ability to execute something that's unknown, right?

1:21:51And that's still a significant

1:21:54James Wilson: multiplier to having the equation. Yes, it's private, but it's private execution of an unknown model and an unknown level of durability. It could change. They could put a new guardrail in. They could change the model weights. They could quantize it. You don't know. But that's what open weights give us, right? It's the ability to not just, it's not that open weights are an enabler of private inference, but the two of them brought together gives you end-to-end attestation and durability. Is that a good way to sum it up? Yeah. Yeah. I think transparency is maybe like, you know, like even more, yeah.

1:22:28Like I think transparency is a very like important property, especially because a lot of people have like, you know, just like distrust some of these companies, like being able to like verify everything, having some level of like, you know, access into like hey this is what I'm getting this is this is how it kind of came about like you know these are kind of the properties that it has and like not having like you know the rug pulled under them per se like I think that's that matters a lot and I'm not sure if like the closed frontier labs are like you know really moving in a direction of like transparency and like you know I feel like they're only more so moving in the direction of obscurity and like complete opaqueness.

1:23:10So yeah. Yeah.

1:23:11James Wilson: Agreed. Agreed. I find that a troubling trend. Well, Tanya, you have done an amazing job of helping me understand all things private inference. It's far more complex and nuanced than I'd even imagined going into this. And I thank you very much for spending the time with me today and for having this conversation. Yeah. Thank you so much, James, for the excellent questions. This was really fun. my pleasure so that was Tanya Verma co-founder of tinfoil and I've been James Wilson for risky business features