Nat Friedman Leads $15M in Superintelligence Insurance Company | AIUC

0:00No one really today knows how these AI agents fail. You might have seen the OpenAI launched their chat GPT agent, and Sam Oldman on the stage puts out a big caveat, like, don't use it for anything important. We don't yet know how this works. Certificates and insurance is a way to build confidence for companies to adopt AI. What it does is we've built a standard, AIU-C1, that outlines what are all the things that can go wrong with an AI agent. And for each of those risks, what are the things an AI company should do to mitigate that risk? We are coming out of stealth, having raised their$14 million seed round from Nat Friedman emergence and terrain.

0:32Your investors are super ingrained into this base, especially Nat Freeman. I mean, he's now a part of the superintelligence team at Meta. What kind of advisement are you getting from them? Because you're an underwriting company that does insinuate you're a financial institution, how do you put dollars against this? Could you break down what is a kill switch in AI? It might be what's called a loss of control scenario, where basically the AI agent becomes so agentic that it starts to take action that we no longer want. So a kill switch is like...

1:11runo welcome to sorcery thank you thanks for having me i think we have some big news today we do can you share this big news we are coming out of stealth having raised a 14 million dollar seed round from nat friedman emergence and terrain wow this is big how big was the round What was the process like? It was$14 million. It was a very quick process. We got an inbound term sheet, preempted, and that prompted us to look up. I knew I wanted to work with Nat. Got introduced to him. A week later, we had a term sheet, and we got it done. That simple? Yep, pretty much. Okay, given that, I think we should understand more about how we got here, how it became that simple.

1:55Yep. So what exactly is your background? I know you're very impressive. Most relevant to this conversation, I started working in AI when I joined Anthropic back in early 2022 as the first person to work on product and go to market. This is before ChatTBT. Is there that moment in time where it just become obvious that something radical was happening here, but the world just did not know yet? And I joined Anthropic to help think through, like, where is the value going to accrue? How does Anthropic sit into that? how our businesses and individuals are going to interact with this AI. So now you're working on the artificial intelligence underwriting company.

2:33That's right. Bit of a mouthful. Bit of a mouthful. Very intense. Super fucking intense, yes. Yeah, we insure and certify AI agents. So what does that mean? Basically, no one really today knows how these AI agents fail. You might have seen actually yesterday OpenAI launched their chat GPT agent. And Sam Oldman on the stage puts out a big caveat, like, don't use it for anything important. We don't yet know how this works. It's kind of like, if you're ordering a burrito and you're not that hungry, go ahead and use this. This is an awesome tool. But when you start putting this in meaningful context, if you run a hospital or a bank or any kind of big business with real customers, no one really knows whether they can trust any particular agent.

3:17And certificates and insurance is a way to build confidence for companies to adopt AI. What it does is we've built a standard, AIUCA1, that outlines what are all the things that can go wrong with an AI agent. And for each of those risks, what are the things an AI company should do to mitigate that risk? And spilling that out in enough detail across hallucinations, jailbreaks, all these terms that people have heard about, that you can have an independent third party go in and audit a company and test that agent to see how likely it is to fail. And if you do well, you pass the certificate. The insurance comes on top of that.

3:51If you meet the certificate, you may want to go a step further in taking risk off the table from your customers and insure them in case it fails. So if it hallucinates a refund policy, if it spews Nazi propaganda, if it leaks your customers' data, if it discriminates against the people you're hiring, the enterprises are going to be on the hook. And so everyone's asking, who's going to pay? And that's where insurance can step in and help pay the bill. How did you come to this model? Yeah, so it started with a deep trip into the rabbit hole. And throughout history, how have markets solved these collective action problems?

4:31And the most interesting example starts back in 1752. Ben Franklin, all-American hero, was living in Philadelphia at the time. Philadelphia's population was growing 10x over that century. And the house was just getting closer and closer. And they were just burning down frequently. so the whole city would burn down. And Ben Franklin wanted to solve that problem. So he started an insurance company, the first fire insurance company. And all of a sudden, he was on the hook financially when houses burnt down. So he had a direct financial incentive to go and reduce fire risk. So they asked themselves, how are we going to make sure that all the houses are built in a way that makes them less likely to burn down?

5:12They built the first building codes saying, oh, how far should houses be apart? Can we make sure to not build houses under trees that are likely to catch lightning. And then they wanted to make sure that people obviously met these standards. So they built the playbook. That probably wasn't the term he used, but the playbook for fire inspections, really the audit mechanism. So this triangle between insurance standards and audits was something that he stumbled into and called the incentive flywheel. And that then shows up again and again throughout history. So when the light bulb came out, starting like around the year 1900, light bulbs also led to the houses burning down the insurers again were like we don't want to pay for this so if you look at any of these light bulbs they will have a little UL certificate the underwriters laboratory funded by the insurers that like specified how do you need to test light bulbs and toasters to make sure they don't burn down later on cars killed a lot of people as they were getting really like everyone needed their own car post World War II and it was the insurers that were taking the charge on looking at the data for what's causing accidents, figuring out what are the interventions that we need, what is the kind of car crash testing we need to incentivize the car manufacturers to build safer cars.

6:29And a lot of their research is what then eventually became embedded into laws like seatbelt laws, airbag adoption. Before that was law, that was often incentivized by the insurers. You get cheaper insurance if you have a car with an airbag. So that model of insurance, building the standards and running the audits to both build confidence in technology and incentivize security is really battle tested. And that's what we think is missing when it comes to AI. I think it would be good to take a step back. AI has evolved so fast. Like we really weren't talking about agents and like a tangible aspect until maybe a couple months ago.

7:10Like now there's real applications and usability with it. So I'd love to get your perspective of the evolution of AI and AI safety. Some companies baked it in from the start. Some just progressed and were all gas, no brakes. So what was that evolution like? Yeah, I think when you started all the way back in like GPT-2, it's kind of obvious what was coming. Like already at GPT-2, you could start to see what GPT-3 for the agents, what are they going to look like? And there are kind of two schools. One was like, there's no risk today. GPT-2 was like literally could not spell its own name. And so like there's no risk.

7:50Therefore, all gas no brakes. And then there's another school that's like, we can obviously see this is going to be a really big deal. In short order, they're going to be able to like help people hack. They're going to be able to help create bio weapons. weapons. They're going to be able to do all these kind of things. And we need to think about that in advance. And that debate kind of played out for a long time without really any evidence either way, because it's just too early as the models were just getting better and better. And now we're getting to this point where everyone agrees pretty much that this is getting pretty serious.

8:23So you have companies like Anthropic have always been in the camp like, hey, this is really risky. Companies like OpenAI have gone a little bit back and forth on what they think. And with their released yesterday, they're saying like, wow, now the risk levels are really high. They are proactively sharing that with the world that they're now like enhancing the security levels. So I think we thankfully haven't had the big incidents yet, but everyone's agreeing that now they're sufficiently good that they can do real damage. When the models are not good, they're not going to do any damage. But as they get really good, we're going to delegate like real work to them, real task, real critical systems.

8:56And no one has sufficiently good answers to say, oh, there's no risk here. Right. So there's companies like, I call it super safe, super intelligence. I don't know why. Extra double safe. I just add more supers to it. Yes. There's companies like that. Ilya takes it super seriously. Yep. Again, super. And then with Anthropic, maybe we could start with Anthropic and then go to SSI. They have different levels of safety. So what do those levels mean? And could you break those down? Yeah. So they call them security levels, security level one, two, three. Really, it's a process of saying when models get sufficiently good and therefore sufficiently dangerous, we need to match that with the levels of security we're implementing.

9:42What are the level of tests we're going to run? What is the level of cybersecurity protection we need to have in place? What are the kind of policies that match each of these security levels? And so we're slowly climbing through the temperatures now. Ananthropic, you're talking just before we got on here about whether security level three is going to be a thing. And I think the short answer is, yeah, it's just a question of time. And is it going to be this side of Christmas or after? I don't know yet. But it's really like companies' ways of matching proportionally the risk with the mitigations.

10:14And then what about super safe super intelligence? They've spoken much less publicly about it. And also, their whole strategy is one shot to safe super intelligence. So they haven't put out a product yet. And so therefore, a bunch of the risks that come when you put the product out in the world, they don't have to deal with. There's still some risk when you're building it as you're building it in-house. But these things are much more relevant to Anthropic and OpenAI, where you have thousands, millions of people using this every day. We're assuming some companies have that. We also assume a lot of companies don't have that.

10:46One of the largest companies is telling people, caution, watch out, don't put your sensitive data in it. That's where you come in. So could you explain a little bit further on AIUC, right? I'll use the acronym. There you go. What we do? Yeah, yeah. Like, where do you come into play and how do you start integrating with partners? Yeah, so the starting point for all this is if you're a business looking to adopt AI, you face a hard choice. On the one hand, if you're a bit nervous, you face that your competitors are going to basically make you irrelevant by moving ahead. or you can lean into this new technology wave as everyone's saying they're doing and then you risk making headlines.

11:26And so we talked about some of the examples before of like hallucinated refund policies, brand disasters happening, data leakages, etc. The question they're all trying to answer is like, can we trust this AI agent? And where we come in is creating the confidence infrastructure between the AI companies and the people that are adopting this. So we work with the AI companies to earn the confidence of their customers by saying, hey, these are all the things you guys need to pay attention to, helping them meet that standard, these best practices for what businesses really need to have confidence in it.

12:00And then as an extra layer of protection saying, if something goes wrong, we will pick up the bill. So if you work backwards from that conversation between an AI company and an enterprise, it's really like, how can you equip the AI companies to say the most confidence building things in a sales meeting? and typically the problem they have is that they all want to say how responsible they are but enterprises look at them and be like we obviously can't trust you when it comes to financial products you obviously have third-party auditors even in cyber security today if you get a pen test you can't do that yourself no one's going to trust you to grade your own homework so you need a third party that can basically say whether you're up to scratch that's the role we play we're the independent third party that can kind of broker the confidence between them you mentioned your SOC 2 for AI agents.

12:47For people who aren't familiar with SOC 2, can you break that down a bit? Yeah. So SOC 2 is a security standard that says, if you're selling a SaaS product, what are the risks that you need to look out for? Things like privacy and security. And what are the things that every company must meet to get SOC 2 certified? And SOC 2 certified is basically like a gold star you can put on your shirt when you go and you meet your customers and say, hey, we meet all these best practices that we know that you'll need to be able to adopt this. because of course enterprises have real security requirements and SOC 2 is just a way of summarizing that into one kind of trusted badge.

13:20There's a question about like how trustworthy it is, but the purpose is really to broker trust between software companies and their buyers. But SOC 2 doesn't speak to all these new concerns that are coming up with AI agents. It doesn't say anything about what are you doing with my training data? What are you doing with my data as it comes to your training models? How are you thinking about measuring hallucinations? oh we read this blog post about jailbreaks what are you guys protected against that what mitigations do you have in place do you have a plan when your ai agent inevitably fucks up all of these kind of new questions suck to a silent on and so we've seen a lot of demand both in the ai companies but also from the enterprises being like who can create this like language for what are the risks that really matter and actually like down to the level of a code base what do you need to do to mitigate that risk?

14:13How do you go down to the code base for this? Because things are moving so fast. How do you model this out? Core of the standard is we actually just have to test how the AI agents perform in real-life settings. So if you're worried about, if you want to know whether an AI agent can get jailbroken, the best way to figure out is to try and jailbreak it. And the field is moving really fast. So, what's really important is that you build this connection between the latest research papers and what the standard does and the kind of testing that's required. So, where some of the old standards get updated every 5-10 years, we're going to update our standard, AIU-C1, every three months to make sure that as new kind of threat vectors come out, as new capabilities come out, that the testing that's being done reflects what's possible.

15:01Because you're an underwriting company, that does insinuate you're a financial institution. Right. How do you put dollars against this when you're underwriting? Like, what are you underwriting and how do you actually cost that out? People are probably looking at AI, of course, as a trillion dollar industry. And so if these companies are growing so fast and are, you know, like$10 billion, now there are hundreds of billions of dollars. They're approaching a trillion dollars in value. How do you underwrite this? Yes. The first thing is, what are the risks that you want to cover? And so if you're backwards from the problem of we're trying to build confidence infrastructure between AI companies and businesses, you start from what are the things, what are the risks that are keeping the businesses up at night and are holding up adoption?

15:44And those are the things we talked about before, hallucinations, jailbreaks. Are you going to mess up the refunds? Are you going to infringe IP, et cetera, et cetera? For each of those risks, you have to figure out what is the, and insurers think about this in terms of what is the frequency? How often does it fuck up? And what's the severity? Like how bad is it when that happens? The core of underwriting these systems is testing. Really getting a robust sense of any particular agent. How likely is it to fail? And you can only do that by like just getting it to fail thousands and thousands of times over.

16:18And then you want to connect that with some of these things that are actually not new. So what is the cost of a brand disaster? or brand disasters are not new. Maybe AI generated brand disasters are new, but we have tons of data on like, what does it actually cost? And so connecting these two up is the core to underwriting it. And then frankly, there's going to be some trial and error. This field is moving. There are going to be mess ups that no one will have predicted. So there's also a way of like, how do you structure the risk so that you don't lose the skin on your nose when that happens? Back to the amounts of, you say, like trillions of dollars.

16:53We obviously don't have a, it's not a$14 trillion seed round. And we've raised... This is also insinuating there's an AI reinsurance company. Yes. Coming up next, next podcast. In the first instance, we work with existing insurers. Okay. So when we help companies get insured, ultimately the guarantee of the money comes from established insurers, some of the oldest ones in the industry, that have the balance sheets to provide this. What these insurance companies are missing is, Frankly, they don't know how AI works. They don't really understand the risk and they don't know how to run these tests.

17:31So they bring the balance sheets and some of the rigor in how to think about the risk. And we bring the technical capabilities to saying, like, what do we think the risk is here? What is the kind of data? What is the kind of monitoring that's required here? And when you put that together, that's what unlocks this industry. You've mentioned a couple scenarios, a couple use cases. I mean, there was the whole Gemini situation early on with their image generation product. There was the Nazi thing with Grok. There's like all these instances for that. But I'm not sure many people have talked about the enterprise level of this and what kind of data is being affected at the enterprise.

18:08So could you go into that a little bit further? Yeah, they're actually kind of related. I think what those examples are showing is that it's actually really kind of hard to build a powerful model that is never going to say something outrageous. And so if you're an enterprise and you're deploying a customer support chatbot, for example, builds on top of those models, someone is going to be able to get your customer service chatbot to say something outrageous. And so that's something that keeps them up at night. you'll have seen lots of people have questions about like oh is opening eye training on my data great enterprises have teams dedicated to privacy that all they're doing is checking what is happening with our customers data and so they have the same questions like what is actually are you training on our data um data leakage it's a big problem you've seen some of these chatbots leaking other customers' data.

19:10So all of these kind of, you might think of them as kind of mundane concerns, but they just really matter for business. Like a bank and a hospital makes really strong promises to their customers and it has to flow through the entire chain. So before they can work with an AI company, they need to know that those promises are going to be met. That's the beginning of that answer. I'm curious what the state of adoption is in enterprise, if you have any sort of like inside look onto that. I've heard rumors about, you know, the AI legal space and there might be huge high flying companies that rhyme with Schmarvy, but, you know, not many actual lawyers are using it in-house.

19:48So like implementation inside is low, but adoption is really spread across. How deep is AI like actually implemented within the enterprise and how long do you think that's going to take? Yeah, so we're transitioning from this phase of everyone for a couple of years have known that they must say to the world that they're using AI for everything. Every board will have said, like, we have an AI mandate. We're going to adopt or die. And so that means everyone talking a big game about how they're using AI. In practice, there have been a lot of proof of concepts, sandbox, enterprise deployments, internal deployments, kind of in low stakes scenarios.

20:22And that's enough that you can say you're using AI. but there's still a gap for this too. When you look at what is the number of lines of code that are being contributed to JPMorgan's code base that's generated by AI, as of very recently, that's extremely low. It's now starting to happen. We're starting to make our way into real use cases, both hospitals adopting this, banks adopting this, and business adopting this. But it feels like it's really in the last six months that you're moving from toy projects to kind of real adoption. But still, that's just the frontier companies that are, like when you look at the long tail of businesses, they're still like, still lots of the world runs on paper, right?

21:03Like there's still a long way to go. Sorcery is brought to you by Brex, the financial stack trusted by more than 30 ,000 companies, including one in three venture-backed startups in the US. Nearly 40 % of startups fail because they run out of cash. Brex is literally built to help founders avoid that. Unlike traditional banks that let your money sit idle, chipping away at it with fees, Forex's designs help you spend smarter and move faster. Their all-in-one solution combines checking, treasury, and FDIC protection into one powerful account. You can send and receive money globally at lightning speeds, get 20 times the standard FDIC coverage through their partner banks, and even high yield from day one.

21:46With same day and even same-hour liquidity. Access your funds anytime. Companies like Scale AI, DoorDash, Service Titan, HIMSS, Anthropic, Flexport, Robinhood, and Plaid trust and use Brex. Start today at brex.com slash sorcery. That's B-R-E-X dot com slash sorcery. As we think about superintelligence, superintelligence has become like the next word. I feel like we've already skipped over AGI. AGI is out. We're over it. We were tired of hearing about it. We skipped to superintelligence. And now that we also don't know exactly what it means, but it sounds really important. Sounds really important.

22:29So what do you think about superintelligence? Let's start from what it is. I think AGI was kind of like, can we build a system that's as smart as a human across all the domains? and everyone was like, that's the milestone in the future. When that happens, everything's going to change overnight. And now we're kind of in this situation where we have systems that they still fail in a number of dumb ways, but it's starting to like, they're really, gosh, they're really smart. They're really good at coding. They're really good at science. They're really like getting pretty good at common sense. They're not very good at robotics.

23:02They're not very good at a number of things. But in a lot of ways, when you talk to something like O3 from OpenAI, You're like, this one is just a lot smarter than me. And so in some ways, it's like we have AGI now, and yet the world hasn't completely changed. So then what's next? And superintelligence is like this idea of like, well, if you plot very roughly AI in a curve over time, how is the IQ going up? We've now hit human level for a lot of use cases, but it's not going to stop here. Now you can use AI to build AI, great, this is just going to keep going up. And so superintelligence is just the idea that you will have something that is like radically smarter than us, where we look like what monkeys look like to us.

23:49And so far, it's a little bit hard to pin down reasons why that's not going to happen and why that's not going to happen in like within a decade. And that seems like the most important fact. You were an early employee at Anthropic. I'm sure you're still friends with people there and you're very connected to community. Are there like actual concerns on the inside about the rapid adoption or I guess the rapid advancement of these models I mean clearly you're building the company in here but like I'm curious about the other externalities around it whether it's like jobs or yeah I guess like let's start with jobs yes short answer I think everyone who's building these models is in like a kind of a dual state where everyone's like wow this is just so exciting we're just making new breakthroughs this is going to advance science is kind of like uh yeah this is going to allow us to cure cancer short order like who does not want that and the field is moving so fast and so it's just like an electric space and at the same time everyone's also like wow this is just deeply uncharted territory both at the model level but also at the societal level and just no one has good answers for what's going to happen to employment everyone is just doing their best at guessing really and so those are those are real concerns and i think what a lot of people are kind of responding to that is like everyone had those concerns when like back when the industrial revolution was going to happen uh it's like what are going to happen to all the jobs the computers came out isn't that going to eradicate all the jobs and it turns out it didn't the question is is this time different i think there's really good reasons to think that this time is is really different and in particular because when you look at the marketing claims of all the aging companies the the value proposition is we're going to do the work for you and so everyone's like well so what's going to be left and so I think that's just a very real risk Anthropic have made some pretty bold statements as to what they think is going to happen to white collar jobs I can't remember the details but by the end of next year you're going to have models that can automate something like 50 % of all white collar jobs and that's like a terrifying prospect you're now starting to see Congress waking up and using terms like AGI they haven't made it to super intelligence yet but that seems to be about next month thinking about this is going to just rip the social contract and that is not a problem that the model companies are going to solve.

26:09Yeah. I was actually just at the RAISE AI conference in Paris. It was amazing. Like everyone was there. I don't know if you were there, but maybe you were. Nope. There was no sign of super intelligence talk there either. And so I was like kind of joking. It's funny about your joke about Congress, but I was joking. I tweeted like degrowth is real. We're not even talking about super intelligence in Europe. They're way behind. Absolutely. I mean, I grew up in Denmark. Yeah, they live in a, most of them live in a different world. You'll see a lot of commentary based on like people who tried their first version of GPT-4 and they're just confident that this thing cannot do math.

26:51Yeah. And it takes about 15 seconds on a mobile phone to be like, no, it can really do math. and at the same time i also have a lot of respect for like if you build products with these models every day it's also hard to believe that agi is going to happen by end of 2026 the super intelligence like it's it's actually still kind of hard to get some of these things to work in practice as reliable as you need it to for this to like i don't know uh scheduled doctor's appointments with the level of precision like life and death scenario and so i think it's like actually really useful to embody both of those perspectives both like wow the real world is really freaking complicated and the last mile really matters and like the models can't do it on their own yet so there's a lot of work to be done and hold in mind like oh two years ago thinking about whether we could like get the precision that's required to base life and death decisions on it wasn't even a question it was terrible they couldn't write marketing copy That was really fucking awesome with marketing copy.

27:52And it's a real question about whether it can run tasks that life and death depends upon. And that is just insanely fast. So you need to hold both of those perspectives in at once. In today's high-speed business world, staying ahead means using the smartest tools possible, including the powerful capabilities of artificial intelligence. Meet Turing Intelligence. Turing builds customizable AI systems designed to solve your mission-critical challenges, no matter your industry. From expert guidance to tailored projects, Turing helps top companies realize AI that's more capable, more adaptable, and more effective.

28:26With Turing, discover how AI can accelerate your business growth. To learn more, visit Turing.com slash sorcery. Spelt S-O-U-R-C-E-R-Y. That's Turing.com slash sorcery. One of the craziest things that has happened recently that gives it more tangibility against like, oh, no, this is really moving super fast is Google's launch of VO3. Because now we're seeing what deep fakes like actually look like. Like these videos are they're hilarious. Some of them are just so scary. Yes. Then I think what was that show? I think Mountainhead came out. And so like they were kind of like modeling the premise on the story.