From Idea to $650M Exit: Lessons in Building AI Startups

0:00What we're going to talk about today is how my company built an AI app that was so good. we're able to bring it to an exit for$650 million and how you can do that too. All right. So really we're talking about three big ideas today. The first is what ideas to pick. How do you decide what to pursue? Second is how you actually build it. And third, and honestly often overlooked, is how you take that thing that you built and market and sell it successfully in the market. But before we dive into this, a little bit about me, so you know who's talking to you. I grew up a coder. I've been building stuff since as long as I can remember.

0:43It's probably the same as basically everybody here. A bit of a side quest for me, but I fell in love with law and policy, and I became a lawyer. And I had a pretty conventional, though brief, legal career. Law school, clerkship, you know, big law firm, etc. I think like anybody who builds stuff and then goes to one of these old professions, like law or accounting or finance or whatever, the first thing you find out is, I cannot believe that they are doing it this way. And so I immediately left that and founded a company called K-Text in 2013 when I think a lot of you were about turning eight. And maybe as a side note, that's about how long it takes sometimes for these companies to be successful.

1:21So I know you're 18, 19, 20, 21, 22, whatever you're old right now. Be ready to sign up for one of the most amazing adventures of your life when you start a startup, but it takes time. At KSTX, we've been focused for the vast majority of our experience on a deep conviction that AI, when applied to law, can make a huge difference. And by the way, it wasn't even called AI when we started focusing on it. It was called natural language processing, maybe machine learning. But one of our AI researchers who is here today, Javed, saw an early application. As soon as the BERT paper came out, attention is all you need, et cetera.

1:58It's like seven years ago of how AI technology could apply to making lawyers' lives better. For example, making search a lot better. Because we were so focused on large language models and were researching deeply in this space, we got really early access to GPT-4. like summer 2022. We were like$20 million in revenue. We were doing great. I had like 100 people and we stopped everything that we were doing and said, we're going to build something totally new based on this new technology. And that became a product called CoCouncil, which was the first ever and I think still the best AI assistant for lawyers.

2:35For reasons I'll go into the rest of this talk, we were acquired by Thomson Reuters about two years ago for$650 million in cash. By the way, that feels like a big number, But I think for a lot of folks in this room, you're going to look back at this talk and be like, I can't believe that was a big number back then. You guys are going to build things that are so much more valuable. I really believe that. And I think that's because what AI is going to unlock for all of you is the ability to build amazing stuff for this world. So, okay, how do you pick an idea? How do you know what to work on? It's actually one of the hardest and most consequential problems.

3:09Since the beginning of YC, they've had this saying, make something people want. And the reason they had that saying is because it's genuinely difficult to know what people want, especially in like the old world of bidding software. You kind of like have to build something, get it in users' hands and try and fail a lot of different times. And you just hope that it's something that people actually want to use. So that's why the saying for Ycomodeer is make something people want. I actually think it just got a lot easier because what do people want? Well, what do people want, for example, things they're paying for right now?

3:41People are currently paying people to do tasks, right? In this case, it's one to very unhappy, like customer support people or something like that. But we already know what people want because they're paying people to do it. This includes a lot of work like customer support or insurance adjusters or paralegals or things you do in your personal life, like personal trainers or executive assistants or whatever. That is what people want. And so the problem of choosing what people want just got a lot easier because now you just have to look, what are people paying other people to do? For a lot of those problems, either traditional AI like LLMs can solve many of the problems that people work on right now.

4:20And if not that, then robotics can solve a lot of things that people are working on in the physical world. And what I think you're going to see as you decide what you're going to build, you first pick an area to target. It really kind of falls under three different categories. One is like assistance, where, say, a professional needs help accomplishing a task. That's what we built at the co-counsel. Lawyers need a lot of help reading a lot of documents, doing research, reviewing contracts, marking them up, making red lines, sending them to opposing counsel. So that's one big category, is assisting people doing their work.

4:51The second big category is just replacing the work altogether. People currently hire lawyers. What if we just became a law firm powered by AI? People currently hire accountants and financial experts and physical therapists and people to fold your laundry, whatever it may be, you can just replace that task using AI. And finally, the third category is you can do things that were previously unthinkable. For example, at law firms, they would have hundreds of millions of documents. And they would never think in a million years, I should have people read over every single document and categorize it in certain ways and summarize it and index it, et cetera.

5:33It would be insane. It cost them millions and millions and millions of dollars. But now that AI is here, you can have thousands of instances of Gemini 2.0 Flash or whatever, read over every document. The previously unthinkable is now thinkable. These are basically the three categories of ideas to choose. And what I think is incredible about this is the amount of money to be made with these new kind of categories each has gone way up. It used to be that what's called the total addressable market, which is basically how much money you can make from your product, was the number of like professionals, for example, number of seats you can sell times the dollars, like$20 per month or whatever, right?

6:12And by the way, a lot of many billion dollar companies are built selling seats to X number of professionals. But today, the actual amount of money that we already know people and companies are willing to spend is the combined salaries of all the people they're currently paying to do the job. And that number is like a thousand X bigger. You pay$20 a month to solve a problem, for example, you know, a typical SaaS kind of subscription, but you might pay$5 ,000 or$10 ,000 or even$20 ,000 a month to certain professionals to solve problems for you. So the amount of money that you can make with your new applications with AI has gone up by a factor of$10 ,000,$100 ,000, or even$1 ,000, compared to what it used to be.

6:58I want to take a quick moment because that might sound like pretty dystopian. like we're talking about taking all these salaries and these become, you know, your addressable market. I think it's kind of the opposite. I think it's beautiful. I think the future is beautiful for two reasons. The first is that you're going to unlock a future. When you replace or substantially assist certain jobs, like people used to, Sam Altman wrote about this in a recent essay, people used to have a job called lamp lighters where we didn't have electricity and lights. So people go around with a matchstick, lighting all the lamps at night on and then turning them off at night by putting out the candles, right?

7:37That's what things used to be. And we couldn't even imagine the kind of stuff we're doing now because that's what we were stuck doing in the past. So you were going to unlock a future that we can't even imagine today when we move past the roles that we're currently doing right now. It'll feel antiquated 10 or 15 or 100 years from now to do the kind of things we're doing today because you're going to help us move past that. But as importantly, what I think some people don't think about with this stuff, which I think is very true, is you're going to democratize access to things that used to be really, really hard or very expensive.

8:12In the field we worked in in law, over 85 % of people who are low income don't get access to legal services. It takes way too long and it's way too expensive. working with human lawyers, right? But if you can help make lawyers 100x faster and 10x cheaper, or, you know, frankly, just provide those services yourself as a new law firm powered by AI, then all of a sudden saying where lawyers have to turn away clients because they did not have enough money, you can now say yes. And that applies everywhere. Everybody should get the world's best financial assistant. Everyone in the world should get the best executive or personal assistant.

8:49Everyone in the world, you know, can already have the best coding assistant in tools like Cursor and Rinse Surface, et cetera, right? I do think that despite the fact that I'm telling you how to pick an idea is you should potentially replace jobs. I think you're going to do something really amazing for the vast majority of consumers and enterprises by unlocking a better future and by democratizing access to things that used to be only for the value of very wealthy. Okay, so that's how to pick an idea. Pick a job, replace, assist, or do the unthinkable that was previously unthinkable, and build a better future.

9:21But how do you actually build this stuff? I'm going to give you a quick outline of how we built it. What's kind of nuts to me is everything I'm going to say right now may sound very simple and commonsensical and maybe even obvious. But the craziest shit is nobody's doing it. Like, nobody's picking ideas the way that I recommend it in terms of picking job categories. There's very, very few companies out there doing that. And even fewer companies are doing what I hope will look like pretty obvious and simple things to building reliable AI. I put it reliable and underscore for what it's worth, because that's going to be the key for many circumstances in terms of getting from a cool demo, as Andrew was saying earlier today, something that actually works in practice.

10:05Here's like four quick points about how to actually build this thing. The first is, think about like making an AI assistant or an AI replacement for, say, a profession. Ask yourself, like, what do people actually do? What does a professional in this field actually do? What does a personal trainer or fitness coach do? If that's the app you're deciding to build. What does a financial assistant do or financial analyst do? And be like super specific. I'm going to say this a few times, but it is really helpful to actually know this answer, not like make it up. It was helpful for us that I was a lawyer, my co-founders were lawyers, 30 to 40 % of my company, even the coders were lawyers, because we actually lived it.

10:47That may not be the case for you. Just go be like an undercover agent somewhere. Really learn what happens at these companies, right? What do these people do? Other way to do it, by the way, is you might be the tech talent, and you may find yourself a co-founder who has some deep expertise in the field. But whatever way you get there, you know, find out what are the specific things that people do that you can assist or replace. And then ask yourself this question. how would the best person in that field do this? If they had like unlimited time and unlimited resources, like a thousand AIs that can all work simultaneously to accomplish this task, right?

11:28How would the best person do this? And work backwards from there, right? What are the actual steps that somebody might take to accomplish a task? To give you an example from our legal field, we did a version of deep research two and a half years ago. As soon as you got access to GPT-4, it was like the first thing that we did. And we asked, what was the best lawyer going to do if given this research question? It wasn't just generally research. What does that even mean? They broke it down to steps. First, they get a request for this research project. And they say, okay, well, I need to understand what this really means.

12:05I might ask clarifying questions, quite like deep research today if you've used it. And then they might make a research plan. They might execute dozens of searches. That might bring back hundreds of different results. They'll read every single one of them very carefully. Kick out the stuff that's not relevant because search results are sometimes ever relevant stuff. Bring in the stuff that is relevant. Make notes about what they're seeing, right? Why is this relevant? Why is this not relevant? Where does this fit into my answer? And then based on all of that, put it all together in an essay. And then maybe even have a step at the end where you check the essay to make sure it's accurate and actually refers to the right resources, et cetera, et cetera, et cetera.

12:42These are the kind of steps that a real professional might do when doing research. So write them down. Now we turn to code. Most of these steps for the kinds of things you'll be doing end up being prompts. One or many prompts, right? One prompt might be read the legal opinion and decide on a scale of zero to seven how relevant is it to the question that's being asked. One prompt might be given all these notes I've taken and all the cases I've read so far, write the essay. One prompt might be like, here's a footnote in the essay. Here's the original resource. Is this thing, you know, accurately cited or not?

13:21The reason why many of them are prompts is because they're the kinds of things that would once require human-level intelligence, but now you're injecting it into like a software application. So now you need to, you know, do the work of turning it into a great prompt to talk about in one second to actually do that human level intelligence. By the way, if you can get away with it not being a prompt, if it's like deterministic or it's like a math calculation or something like that, that's better. Prompts are slow and expensive. Tokens are still expensive. So when you're breaking down these steps, some of these things might just be good old software engineering, right?

13:58Do that when you can. And then here you make a decision. When you find out how the best person would approach this, if it's pretty deterministic, Like every single time they always do this task, they always follow the same five steps. Simple. Make it a workflow, right? It's actually the easiest outcome for you. And to be honest, a lot of the stuff that we built while building co-council was exactly like this. Every time you do this task, you're basically going to take the same six or seven steps. And you don't need to have, frankly, like fucking Lang chain or whatever. Just Python code. This function, then the output of this function goes in this function, output of this function, boom, you're done.

14:34Right? Simple. Sometimes it's not so simple. Sometimes how an expert would approach the problem really depends on the circumstances. Maybe they need to make a very different kind of research plan, pull from different resources, run different kinds of searches, read different kinds of documents, whatever it may be that you're doing, right? That's when you get to something that's a little bit more identic. That's harder to make sure it's good. But maybe what you have to do, right? Okay, underscore this again. In doing all of this, having some form of domain expertise, somebody who knows what they're talking about here, which, by the way, you can also acquire this by talking to a lot of people.

15:09There are lots of different ways to get here. But don't fly blind. Don't assume this is the way that all government employees in this field do X. Really, no. Okay, so that's the basic way you can build these AI capabilities that start to round out. And that's it. Right, simple. The hard part, frankly, isn't building it. The hard part's getting it right. Like, how do you know the research was done well? How do you know it read the document right? How do you know it edited, you know, it did the insurance adjustment correctly? How do you know it made a correct prediction about whether to buy or sell a sock or whatever it is that you're doing?

15:43This is where evaluations play a very, very, very large part. And this is the thing that I see most people not doing because they build like demo level stuff that frankly is like 60 to 70 % accurate. And if we're being honest, you can probably raise a pretty good round of capital by showing your cool demo to VC partners. And you can even possibly sign on your first few customers with a cool demo as a pilot program, right? But then it doesn't work in practice. And so all that excitement and VC capital raised and pilot program excitement, et cetera, falls apart if you can't make something that actually works in practice.

16:21And making something that works in practice is really hard because LLMs, like people, you don't have your coffee that morning, you wake up on the wrong side of the bed, it might just output the wrong stuff for prompts. I'm sure you've all seen this before. Even if you just use ChatGPT, you've sometimes probably been blown away with its brilliance at times, and at other times, shocked by how incredibly wrong it was about code or some informational lookup or just hallucinating when George Washington's birthday was or whatever it is, right? So how do you deal with that? I'll tell you how we dealt with it.

16:58This is not the whole answer, but a big part is evaluations. YC's next batch is now taking applications. Got a startup in you? Apply at ycombinator.com slash apply. It's never too early and filling out the app will level up your idea. Okay, back to the video. This all begins again from domain expertise, which is like, what does good look like? What does it mean to do this task super, super well? If you're doing research, for X given question, what is the right answer? What must the right answer include? For X document, and you're asking a question that's a document, what must it pull out of that document?

17:38What pages should it find that information? What does good look like? This is true of the overall task, like complete this research for me, but also each micro task necessary to complete the overall task. like which search queries are good search queries versus bad search queries. Here again, not sounding like a broken record, but it's good to know what actual professionals would say about this, right? So what does good look like? And then those become your evals. My favorite thing to do when I'm writing evals for things that are like, you know, when possible is to turn into like a very objectively gradable answer.

18:13For example, have the AI just output true or false or a number between 0 and 7 or whatever. Because then it's really easy to eval. For the given document and given question, the answer should be 6 on a scale of 0 to 7. That's how relevant it is. Not a 7, not a 5, it's a 6. And if you have that, then you can set up an eval and a framework, I like PromptFu. I don't know if you guys use that. It's like open source, runs on your command line. There are many frameworks out there that you can use to put together these evaluations. doesn't really matter. At the end of the day, it's like, for this input and this prompt, the answer should be six.

18:53Make a dozen. Try to match what your customers are actually going to throw out your program, right? Make a dozen and then try to get it perfect on a dozen, then get to 50, then get to 100. And keep on tweaking the prompt until it actually passes all the tests you keep on throwing at it. If you're reading really good about this, have a holdout set. And don't, you know, look at those while you're writing your prompts. make sure it also works on those. You're not just fine-tuning the prompt just for your evals. What you'll find without any technical fine-tuning, you can go so far with just prompting.

19:28If you're being really careful about this, you will find that the AI gets things wrong predictably. You're ambiguous as part of your prompts. You're not giving clear instructions about doing one thing, or maybe it just constantly fails in a certain direction, and you have to give it prompting instructions to pull it back from making this kind of error. You give it examples to guide it away from certain classes of errors. But it's not going to be a surprise why or how AI fails. Once you start prompting, you'll start to see patterns that you can prompt around and give instructions around. And what I like to say is the biggest qualification for success here is whether you or whoever's working on the prompts of your company is willing to spend two weeks sleeplessly working on a single prompt to try to pass these evals.

20:16If you're willing to do that, you're in a really good place, right? It just takes such a grind because the thing is, you're going to do these evals and at first you're going to pass like 60 % of the time. And at this point, most people just fucking give up. They're like, hey, I just can't do this task, right? They're like, I just can't. I'm not going to do it. And then you'll spend a night prompting and you're going to be at 61%. You're like, oh my God, the next group of people will give up at this point. What I'm here to tell you is that if you spend solid two weeks prompting and adding more evals and prompting and adding more evals and tweaking your prompt and tweaking your prompt, you're going to get to something that passes like 97 % of the time.

20:51And the 3 % is kind of explainable. It's like a human would, it's like a judgment call almost. Humans make similar kind of judgment calls. Once you're there, you can feel pretty good about how this might interact in production. What I recommend is like pre-production, maybe in like beta, Get to 100 tests per prompt and 100 tests for the overall task. If you're passing like 99 out of 100, again, you should feel pretty good about where you are. So that's just a rough guide. If you can beat 1 ,000, that's 10 times better. Do that. But it's hard. It's actually really hard to come up with great evals.

21:25So I'd recommend just at least 100, go to beta, and put it in customers' hands. And set the expectation, by the way. This is not yet perfect. That's why you're in a beta. and then you listen and learn. Every time a customer complains, either you have their data because that's how your app is set up or you ask them like, hey, can you share that document and that question you asked to see why it failed? That's a new test. We've added much more evals at this point from real things that happen to real customers than the ones we came up with in the lab. And that's going to, your customers are going to do the dumbest shit with your app, okay?

21:59They're going to do such dumb things that you would not predict. But that's what customers really do. If you've ever seen a real person's Google queries, they're barely legible. And I'm assuming the same thing is true of ChatGPT. They see a bunch of stuff. Like your prompts probably look pretty smart. Most people are like, burrito, me near, how, ouch, or whatever, like, what do you do with that, right? But you have to try to bring back a great result and determine what they're actually trying to say with these ridiculous prompts, so do it. Those become your real tests. And just keep iterating.

22:31This is not a static thing. New models will come out. Try the new models. PromptFu and other frameworks make this really easy. Add a new model, it'll compute how well it does against your prompts so far. Keep tweaking your prompts. Sometimes the addition or subtraction of a single word might move you up a single percent, but that's a very big deal if you're working in a field like finance, medicine, law, where single percentage increases in accuracy really matter to the customers you're serving. Keep iterating. Never stop. There should be a new GitHub pull request like every other day or every day on your prompts.

23:03And I'm telling you, if you just do those two last slides, you know, how do professionals really do it? Break it down to steps. Each step basically becomes a prompt or piece of code. And then you test each step, test the whole workflow altogether. If you just do these two things, you'll be like 90 % of your way there to building a better AI app than what most of the crap that's out there. because most people never eval and they never take the time to figure out how professionals really do the job. And so they make these kind of flashy demos on Twitter. They maybe even raise capital and they may even be some of your heroes for a minute, but be careful who chooses your heroes.

23:39The real people are behind the scenes, quietly building, quietly making their stuff better every single day. If you just do these two slides, you're going to be 90 % of the way there and better than most of the things that are out there. That's the craziest part. Okay, now the hardest part, honestly. And the part that, frankly, we are still trying to figure out post-exit, you know, at a multi-billion dollar company, it's still going to be really, really, really hard. And I'm going to give some tips about marketing and selling AI apps in this new kind of world where you're maybe replacing or assisting a job, things that we've learned along the way.

24:11But the first thing I'll say, this is a little bit counter to what I think is out there in a lot of the VC kind of, a lot of people like say like the most important thing is sales and marketing. A lot of people really think that. When you guys raise Series A's and Series B's, you'll have people on your board who say, product doesn't really matter that much if you're really good at marketing and selling. And they've seen some examples of this working out really well. I think it's fucking bullshit. For 10 years, we had an okay product at first. We went through different marketing and sales leaders, some of them super well-qualified, et cetera, and they did okay.