Min-Kyu Jung: No offense, kid

20:21Min-Kyu Jung:And then number two, if you're looking for the, I don't know, like the cosine, semantic similarity between your query and the documents, you end up with this issue where quite often the legal meaning of a word is different from the ordinary English meaning of the word. and the way we assess our accuracy is against the quad data set the the contract understanding etiquette data set which is the state-of-the-art legal data set um we've actually found that ivo is more accurate than that data set because we identified human errors that in the data set um but the most difficult test case for us was most favored nation provisions and mfm provisions are interesting because they can look like anything in an agreement uh and if you're just looking at like the semantic similarity of different words you would you never actually figure out a contract has an MFIN provision.

21:09Min-Kyu Jung:So what we've had to do is we've had to actually read the contracts. And the obvious reason why nobody else does that is because it's incredibly expensive and you literally bankrupt yourself as a company if you try to do that across over a million contracts. And it's also pretty slow because you're reading all of these agreements. So I think the real technical achievement from our engineering team is figuring out how we can do that while still having great latency and being able to keep costs under control.

21:38Ivo:What other non-obvious bets have you made?

21:40Min-Kyu Jung:I mean, there's a lot. One of the product decisions we made early on, and I think it took us maybe too long to figure this out, is that there are real trade-offs that you need to make depending on which kind of customer you want to serve. And I think this is a challenge that every product person needs to figure out, which is just maybe to give you a thought experiment let's suppose that you had access you had early access to this incredibly advanced ai model and you all you have to do is type in a prompt and it will generate the perfect product for you it'll generate whatever product that you ask for and there are just either no resourcing trade-offs or you it just builds what you want even in that scenario you still want to be you want to be you want to have a point of view on who your ideal user is because there are trade-offs you have to make where certain features are just going to be better for certain segments and worse for others.

22:34Min-Kyu Jung:Certain features are just diametrically opposed. And I think the challenge is you have to figure out what is the subset of out of the universe of potential users, what is the subset that you want to serve where you can still be the best product on the market because you're sufficiently narrow while also being broad enough that you can build a really big company. And the decision that we made there is we want to serve in-house legal teams rather than law firms, because we think those are two very different markets and the best in class product is really different for both. And there are a lot of maybe obvious reasons for that.

23:07Min-Kyu Jung:In-house legal teams serve one customer rather than many customers. In-house legal teams tend to review many relatively repetitive agreements, whereas law firms tend to review bespoke agreements across a large set of clients who all have different business objectives and different needs and requirements. The reason I say that's non-obvious is because I think basically every one of our competitors have made the decision to focus on both. Any competitor that you can think of, they decide to serve both law firms and in-house legal teams. And they're actually just some examples of why I think in some ways the product has suffered as a result.

23:48Min-Kyu Jung:So for example, we were the first AI legal company to have the constant of playbooks to review contracts when reviewing contracts with AI. And the reason playbooks made sense for us is because if you're an in-house legal team, you're probably reviewing a lot of repetitive agreements. You're serving a single client. You have a clear idea of what your business objectives are, what are the fallback positions, how you handle different corner cases. But that kind of feature doesn't make sense for law firms. So we were the first company to even build that feature. and we've gone far deeper into building that feature than others, whereas perhaps for other companies they're thinking, okay, why would we spend time and effort building out this playbook functionality when three-quarters of our customers are law firms who see no value in it?

24:36Min-Kyu Jung:One of the observations we had when we built the product is that lawyers are actually surprisingly willing to tolerate inaccuracy. So one of the other things we've done is build an incredibly accurate product, but of course AI is not 100 % accurate and contracts are demanding where the cost of failure is high but one of the things we discovered is actually if you draw it down into it, lawyers are comfortable tolerating some degree of inaccuracy but they're only willing to tolerate their inaccuracy if they really clearly understand why the outputs are being generated they understand the relationship between the inputs they're put in and the outputs they're getting back and the reason that's important is if you can easily verify an output and you can understand that the output why the output was generated and therefore whether the output is a good output or a bad output you can dismiss it really easily whereas if you have to spend time like trying to figure out why the ai did something and then only once you figure that out then can you finally dismiss it and replace it with whatever you would have done otherwise um you've actually just wasted more time going through that verification process and you would have if you could dismiss the the the incorrect answer in the first place.

25:47Min-Kyu Jung:The reason that's counterintuitive is I think the design decision that most product managers are taught when they work at a company like Google or Meta is you want to have the fewest number of clicks possible. You want to have the most direct path between the user making a click and the output being presented to them. Whereas actually, if you take the philosophy that you want the user to understand why the outputs are being generated, you need to make the process a little bit lower level. You need to kind of guide the user through multiple steps and then the user understands through a step at a very granular level what they're doing and then why the outputs are being generated in the way that they are.

26:22Ivo:To circle back to the first point you made, why do you think that all of the competitors are wrong and your bet will continue to compound in the right way?

26:33Min-Kyu Jung:Why they're wrong in terms of deciding

26:35Ivo:to service both in in-house lawyers and legal teams versus focusing simply on in-house teams

26:44Min-Kyu Jung:i mean the immediate answer to that question is the same as the answer i gave before which is we think we can build a much better product by serving in-house legal teams um i think the the longer term answer is maybe the the kind of an interesting version of that question is, will the best company be the company that serves law firms or the company that serves in-house legal teams? And the reason we decided to serve in-house legal teams is we think that contracts are ultimately business documents first and legal documents second. We think the ultimate opportunity with contracts is beyond the legal team.

27:18Min-Kyu Jung:And indeed, many of our users now are outside of the legal function. There'll be the procurement team who needs to review vendor contracts, of course with legal oversight or in some cases it'll be the sales team if you believe that contracts are business documents um then the way to frame this is either in many ways it's kind of like docusign but rather than helping you sign contracts where the product that gets you to a point where you already signed the contracts in the first place and that that adjacent opportunity is much more obvious if you're serving in-house legal teams versus if you're serving law firms I think the other part of this is if you want to build a generational company in this space, if you serve law firms, you know, having this always been incredibly successful, in some ways you're cannibalizing your own market because part of the value proposition for us now is we're helping in-house legal teams reduce the outside counsel spend, right?

28:10Min-Kyu Jung:So the bet that we're making is if we become one of the most important companies in the world, there'll be a world where also law firms are less important than they are today.

28:20Ivo:If we play that forward, do you imagine a world where law firms don't exist for contracts?

28:27Min-Kyu Jung:I don't think so. I think law firms always exist. But the modern era of incredible number of lawyers that we have is a little bit of an anomaly. In the 1970s, there was a huge explosion in law firms. There's no rule of nature that that always needs to be the case forever. I don't know if I'd go as far as saying there will never be any law firms. I think law firms will exist for contracts for a variety of important reasons. But I think if you extrapolate out the trends that we're seeing now, which is that in-house legal teams are able to reduce your dependence on our side counsel, and if you assume any degree of consistent LLM progress whatsoever, then I think at some point in the future, we will see law firms being less important than they are now.

29:19Ivo:I suspect you're seeing, you're bearing the fruits of focusing on a particular ICP in bake-offs. I'd love to know how you've been going with respect to selling into these larger enterprises.

29:33Min-Kyu Jung:Yeah, we've been incredibly successful at selling into large enterprises. We have something like an 85 % hit-to-hit trial win rate. overwhelmingly when companies consider ivo against our competitors we win especially once they've had a chance to to actually use their products um and and this is for all of the kind of notable well-known competitors you can think of but much more than that this is an incredibly crowded space we quite regularly go into competitive deals where there are 10 or even 20 other vendors and i think the takeaway for me is usually that's a surprise to these large enterprises they'll say wow i'd never heard of you guys before or you're a um you uh i i we you know we thought we'd throw you into the ring because we heard good things about you from somebody else but we didn't expect your your product to win um and we we hit it yes it's so good fortune 50 company who said they evaluated 10 vendors before they even heard of vivo um they didn't like any of those 10 vendors then they tried to build their own internal solution and that didn't work out uh then they found us and then within a few weeks it was it was very clear that we had the best solution and and and i think the the thing that i i think that worse in our favor is that lawyers are really smart and they have incredible bs detectors and you can't like trick lawyers to saying they like your product more because you're in the news all the time or or and you've raised a lot of funding the ultimate test for the the lawyer is that this tool genuinely helped me and did it help me more than the other tools?

31:08Min-Kyu Jung:And if the answer to that question is yes, it almost doesn't matter how, like all external factors almost don't matter at all. And certainly we've been in situations with large customers where all of the trial users say they prefer Ivo and when they're all literally unanimously, they'll say they prefer Ivo. I mean, maybe the CLO or the GC will still prefer some other solution because they've heard good things about it from some law firm. But ultimately in that situation, the end users win when the contrast is as strong as it often is in these backoffs.

31:41Ivo:I want to talk to you about the disadvantages and advantages of not being as well as known in market as some of the US competitors in particular. Before we do that, I'd love to just learn about how your go-to-market has evolved over the last few years.

31:58Min-Kyu Jung:The thing that makes it difficult to answer that question is that the market itself has evolved a lot. It is an absolutely crazy pace of change in the market. So a lot of what we've done from a GTN perspective is almost just trying to keep up with the pace of change in the market. And I'll give you an example of this, which is in 2023, when we first launched, our product was already just well suited for large enterprises more so than smaller companies. But that wasn't now icp at the time and the reason for that is back then even though there was an enormous interest in ai large in-house legal teams were not in a position at scale to adopt legal ai solutions especially not from random c-stage companies with you know run by a bunch of kiwis uh it was it was a lot easier to to sell your tool to some small you know one or two person legal team where the only thing that needs to happen to make the sale is a general counsel needs to think your tool was pretty cool and they did their credit card out um and i actually suspect that's the reason why early on a lot of the companies that experienced a lot of growth in in our market and especially in the house legal space were companies that served smbs and had more of a prg kind of driven motion and whose products were just more oriented around um kind of getting immediate value out of the product without needing to focus on any kind of implementation to make the tool more effective.

33:29Min-Kyu Jung:And then what happened is starting about midway last year, the market started changing in a big way where you start seeing these top-down mandates. So one of our customers is Shopify. And Shopify's CEO put out this very famous memo where he declared that the entire company needed to be serious about adopting AI. He threw down this mandate that every single department within the organization was expected to be proficient in AI and figure out how to use AI to improve their processes. And that's just not a thing that was happening in 2023. Maybe there were some very forward-thinking leaders who were doing that, but it didn't really start becoming something that would impact us until midway through last year.

34:12Min-Kyu Jung:So as that happened, our company was in a very good position to capitalize on that. We pivoted our GTM model to focus on large, sophisticated enterprise companies. And the tailwinds have carried us through. How did they find out about you? A mixture of things. Quite often, it is through referrals now. So people will hear about us from a friend of theirs who works in another big company. And one of the things we've learned is that the legal communities across different areas are surprisingly small. And, you know, I spend a lot of time going to different legal events. And, you know, and it's quite often the same crowd of people over and over and over again.

34:53Min-Kyu Jung:So that part is really, really important. Other than that, all of the usual, all of the usual GTM tactics that you can think of, performance marketing. We get a lot of them down that way.

35:08Ivo:I guess I'm also curious about the intersection of the product, especially in a market where lawyers are by nature skeptics and how you have thought about building that trust in your sales cycle and where that trust begins and how that is built up so that you do achieve a win rate of 85 % post-trial.

35:33Min-Kyu Jung:I think one of the things we're very intentional about is we try to define the product beyond the software that we make available to our users. Really, every interaction that the customer or the prospect has with our organization is part of the product experience. So that includes, as you point out, the sales process as well. And I think one of the things that's unusual about lawyers is that they're incredibly, incredibly skeptical as a group of people. I read this interesting study, this research paper, I think it's called Why Lawyers Are Unhappy. And one of the arguments they make is that lawyers are unusually pessimistic compared to other professions.

36:13Min-Kyu Jung:In fact, there was a sort of study on different students where they assess the performance of different students against their personalities. And they found that for basically every group of student, pessimism was negatively correlated with academic success. And the one exception to that was law students. And it kind of makes sense because often many of the best lawyers that I know have this sick sense of, hmm, I feel like something might go wrong here. So just in case, let's put in a protection here for that minuscule downside risk scenario. And of course, in aggregate over the course of millions of transactions or thousands of transactions, it turns out that that really saves you as a company that somebody thought to put that provision into the agreement.

36:58Min-Kyu Jung:And I think that kind of pessimism that serves lawyers really well, you kind of turn it off. So when it comes to evaluating products, again, they come in with this incredible skepticism. And I think there are a few ways to handle this. One is that the moment you start talking in marketing jargon, you start talking about how you want to revolutionize this or whatever, you can almost see the eyes glaze over and they just stop caring. or they start looking for holes in what you're talking about and it becomes a little bit adversarial. I think it's much better to be very straightforward and then make sure you really understand your stuff.

37:39Min-Kyu Jung:So if a lawyer asks you a question like, what makes you better than a competitor or how are your competitors better than you? Don't try to spin it. Tell him as factually as you can what the answer is. And then also even better is you should be able to go very deep into explaining what makes you better. So if they're asking you technical questions about how you have you done this or that, you should be able to explain that in quite a bit of detail. The other part of this is I think even if you are an incredible salesperson and you do all of these things really well, there is still some element of this is a salesperson.

38:12Min-Kyu Jung:So I don't know if I completely trust them. So as much as possible, we try to include lawyers in every part of our customer facing processes. So we have lawyers included in our sales processes. We have laws included in the customer success and implementation. And we very intentionally not incentivize these lawyers with any sort of commission-based incentive like you would with a salesperson. Because these lawyers, their job isn't to try to sell to the client. Their job is to help the client and build trust with them and be the trusted advisor and partner. And we found that goes a long way to building trust.

38:48Min-Kyu Jung:Can I ask what your sales cycles are? Yeah, it depends. It seems pretty wildly. Our enterprise motion is fairly new. From what we can tell so far, it's somewhere between three to six months. So our biggest deal in company history was about six months. Most are a little bit less than that. How do you feel about enterprise sales? You know, it's funny you ask that because I am absolutely not a natural salesperson. I'm very... I think very few lawyers are natural salespeople. um uh sales you know for me i don't maybe it's a kiwi thing i don't like asking people for things and it's an austin kiwi thing um i even when i'm following up on people i i feel rude and i i makes me feel a little bit gross but you know one of the things i never thought i would appreciate as much as i do now is is how fun and like creative enterprise sales can be um i actually really like doing enterprise sales now.

39:51Min-Kyu Jung:And maybe the thing I didn't appreciate is the degree of craftsmanship that just goes into being a great salesperson. When you see a great salesperson in action, I think it's really, really impressive. And it's not just about being charismatic and saying the right things on the call. It's about making sure you understand the customer or the prospect's business problems, making sure you understand how to solve those problems. And then also managing the internal way of politics and this entire complicated organization and how they function and work, I think is an interesting puzzle as well. Because when you're working with a big company, you have the direct point person that you're working with and they have their own incentives and goals and dreams and hopes and desires and all the rest of it.

40:33Min-Kyu Jung:But then there's also all of these other people within the organization who are also part of the decision-making process. And being able to navigate that, I think, is really fun. I think the other thing that's fun about it is enterprise sales is a competitive process, right? You're directly head-to-head against your competitors. And I think there's just something very thrilling about going head-to-head against a competitor. And one of the great things about startups is the only way that you can win is by having the best service. And people need to choose you out of their own volition because they believe that you have the best service.

41:10Min-Kyu Jung:And I think there's just something very satisfying about seeing that sort of process through.

41:18Ivo:How have you been creative in an enterprise sale?

41:21Min-Kyu Jung:So we have one of our customers is a Fortune 100 company. It's one of the great American companies. And I'm very proud to be working with them. And on multiple occasions, they actually told us that they weren't going to go ahead with Ivo. And one time they jumped on the call with me and they said, look, MinQ, I'm sorry. I really like you guys, but we're not going to go ahead with Ivo. we're going to go ahead with this other solution. And they gave us a number of reasons. Some of them were kind of internal political reasons, and some of them were product-related reasons. There were certain features that we needed to have that certain people in the organization deemed to be critical for this particular product.

42:06Min-Kyu Jung:As soon as he told me that on the call, I jumped off, I spoke to our engineers, and we worked on the thing that they asked us for, and we literally released that feature over the weekend. And I remember one of the things that he used to always tell me is, you know, we really care about working with partners who are going to be long-term partners with us and you can be receptive to our feedback and all the rest of it. And we put together this feature. It was a pretty complicated feature, but it worked. And we released it over this weekend and I sent him a Loom video and I said, look, we've built this feature.

42:42Min-Kyu Jung:I would love to talk to you. I'd love to fly out and show you our roadmap and where we're going to be in the next three or six months. And he replied and he said, all right, fine, we'll take you to a pilot. And of course, we ended up winning the pilot. And that actually isn't an unusual story. There's been probably two dozen of those stories of the core survivor history where there's been some kind of important deal. We had to throw out a Hail Mary. Maybe it was me, maybe it was a champion saying no, and then I had to send a cold email or a cold call to the CLO and I had to beg that they give us a chance.

43:22Min-Kyu Jung:And it's worked. More often than not, that actually works. When companies see that you're willing and capable of pulling out these miracles, they start to believe that you will do anything in your power to make sure that they're successful as a client. And I think that's fun because I think if you have this sort of PLG motion, you can't really do that. You can't pull out the stops to make sure that you win some critical client. And I think that part of sales is really fun. Did you ever try the PLG motion? Sort of, but not really. We never really committed to it and we don't rule it out as a pathway in the future.

44:07Min-Kyu Jung:I mean, the problem we have right now is we have too much demand to handle, which sounds like a fake problem. It sounds like when you have an interview and you say your greatest weakness is you work too hard. But it is actually... No, I believe you. One of our greatest problems where we had, you know, last quarter, we had actually all of our sales reps at least doubled their quota. We had one sales rep who did over a million. He's not even an enterprise sales rep. He's a mid-market sales rep who did well over a million an hour just by himself and split across like a bunch of$10 ,000 to$20 ,000 deals.

44:44Min-Kyu Jung:So you can imagine just how many calls he had to do to close those deals. So I think where PRG could make sense is unbottlenecking our sales team, like the humans that need to walk our customers through these sales processes from a lot of our sales activity. But I guess, again, there's this trade-off thing. We're making your product really good at serving lower-end customers who would benefit from a PLG motion. I think there are real trade-offs for larger customers.

45:17Ivo:How do you figure out who to sell to in an enterprise? Is it the GC? Is it the lower-down? Is it the COO? How do you figure that complicated web out?

45:27Min-Kyu Jung:Yeah, it's the legal team. I mean, we're lucky because we do very little outbound. Almost all of our revenue comes from people signing up for a demo on our website, right? So there's no kind of discovery process where we, I mean, there is, but the champion is the one who signs up for a demo and books a call on our website. And that changes from legal team to legal team. Some companies don't even have a legal operations team. Legal ops is a relatively new function. Sometimes there's a senior lawyer within the organization who's taking a particular interest in AI, and that person will be the one driving the project.

46:07Min-Kyu Jung:So the unsatisfying answer is it depends. I mean, maybe the lesson for us is with enterprise sales, every deal is a new puzzle that you need to crack.

46:19Ivo:What do the best performing salespeople at Ivo have in common?

46:25Min-Kyu Jung:There are a few ways that you know that an AE is going to be successful. One is they have a deep curiosity about the product. I think a lot of sales reps, particularly ones who come from larger companies, they're used to a solutions engineer doing all of the demos for them. They don't really need to understand the product really at all. And they see product understanding as something outside of their job description. whereas the best AEs have a deep curiosity not just about the surface level here or the features that we have but the machinery underneath that like how do these features work? What are the specific engineering decisions we made that make us different from the engineering decisions some other company has made?

47:05Min-Kyu Jung:And many of the AEs who I really admire I often see them walking over to the engineers in their corner of the office and asking them questions about how something works. So I think there's that aspect of it. Another thing, and this is sort of a general point across the entire company and not limited to salespeople, but I think there are salespeople who have just this internal sense of urgency about everything. They understand that deals die all of the time and they understand that if they instill a sense of urgency in themselves, then that almost transmits to the prospects. If they're answering emails fast, if they're scheduling calls out a couple of days or the next day rather than a week or two out, if they are following up after calls immediately rather than waiting until the next day, like all of these things compound.

48:00Min-Kyu Jung:And the person on the other side, they sense that sense of urgency as well. And that compiles them to move faster, too. And I think it's just such a known goal as a sales rep. if you move slowly and you lose because you want to lose some percentage of your deals that way just from answering an email a little bit more slowly than you could then the third and final thing is um i think the best sales people have a very good orientation towards client service one of the best things i learned as a lawyer was that client service is incredibly important and you should do basically anything within your power to make your clients happy and the way i've um i think i've seen our vp of sales describe it is if you're sending a one million dollar deal if you're working on a$1 million deal, then the customer on the other side, they're expecting a$1 million experience, right?

48:46Min-Kyu Jung:You can't write sloppy emails with typos. You can't send like half-assed, like word products to them. You're asking the customer for$1 million, which is a lot of money. And the service you offer has to match the price tag. Nothing to make use, just understand and appreciate that perhaps at a more intuitive level than others do.

49:12Ivo:You mentioned earlier that, I don't know if you mentioned who the customer name was, but they said something along the lines of like, oh, we went through 10 vendors and we hadn't heard of you, but you ended up winning. Does that type of distribution where the world knows your name matter? And does it matter to the extent, like to what extent does it matter? And are you worried about it? are you thinking about how do we how do we get louder to ensure that we are at that top of funnel or for you is it really just about compounding the product into the enterprise sales motion

49:48Min-Kyu Jung:it definitely matters um so in the universe like right well maybe the way to put it is if i had the option to click a button and clicking that button uh made it so that we were the most well-known company on the planet then like obviously i should i should press that button yeah so so the end goal is definitely we need to become that company that the the universe where we become one of the most important companies in the world is one where everybody knows who Ivo is so I I think the question is more like what is the most direct path to getting there is the most direct path to getting there like having a lot of creating a lot of hype around what you're doing and you know and then the the the product like follows after that or or the reverse we have this incredible product and people spontaneously recommend your product to other people because they love it so much um and and uh your reputation spreads that way if those were the two options then i'd rather select the second option um that isn't to say we couldn't do more around becoming a more well-known brand i think there's a lot of work work we need to do there and we have some very some initiatives i'm very excited about uh for the next month or two but But if those were the two options, then it is much better to have a product that's great and that people genuinely love and will recommend to their peers.

51:10Ivo:How do you measure their love? Frame differently, like what product metrics are you looking at that illustrate the love that you've described?

51:18Min-Kyu Jung:Yeah, so there's a few different things. The most obvious one is are they using the product? like if they you know our product we at the moment our products are really productivity software so people are incentivized to use it if they think it'll save them time and money and they won't use it if it isn't accomplishing that goal so you know usage is a good proxy for whether people are finding value from the tool which is a related but distinct concept from do they love it I think the love aspect for us is a bit more qualitative at the moment. The reason we can get away with that is we aren't a PLG company.

51:55Min-Kyu Jung:We focus on relatively small, you know, like we have a few hundred customers and we spend a lot of time with our customers. We, like with many of them, we meet them every single week outside of the weekly meetings where we're talking to them, we're texting them. You know, I know everything about many of our users. I know their kids' names, where the kids go to school, what they like to do on their weekends. And I think if you know your users that closely, the kind of qualitative feedback you get when they explain, this is how we think about your product and this is my opinion on it, I think that's many times more valuable than the quantitative data.

52:39Min-Kyu Jung:The third and final thing is we try to keep a very close eye on referrals. If, you know, the larger the percentage of users or companies who come in through the top of the funnel through referrals, the more confident that we are, that we're doing something right from a customer perspective. It's actually, it's kind of like a convoluted NPS score because the NPS survey is, would you recommend this product to other people? And NPS is great, but it's even better if rather than just looking at, are they, whether someone says they would recommend a product, are they actually recommending the product?

53:23Min-Kyu Jung:I mean, we see a lot of that and that's also good evidence for us that we've built something special.

53:30Ivo:I know you're incredibly, you have an incredible attention to detail and I'm curious how the, in part I'm kind of surprised there weren't any product like metrics to give you the feedback loop to say that these are the details that matter in our product and you've been able to achieve that because of the love that you're seeing from IBM, Netflix, Lassie and Figma, Canva, Notion, Reddit. The list goes long and it's a beautiful list and And maybe there are some product principles that you can share that if I was in your product team, I would be like, these are the things that this product team really cares about.

54:09Min-Kyu Jung:So one of the things we do that I suspect that our competitors aren't doing is we have an incredibly low threshold for what we consider to be a bug. So for us, a bug isn't just necessarily like something is broken and doesn't work. is whenever Aibo generates an output that is worse than what a human would have generated. So that is our threshold for whether we consider something to be a bug or not. So one of the metrics we really look at is what is the ratio of bugs to not bugs in the red lines we generate, as an example. And part of the reason this is important is our product sits in a domain where it's interacting a lot with these messy real-life use cases.

54:59Min-Kyu Jung:Maybe an extreme version of this is self-driving cars. So our head of engineering, he used to intern at Waymo. And one of the things he would talk about is at Waymo, one of his main jobs was just hammering down this long tail of issues that would arise over and over again when you put these cars in the messy real world. Like, you know, in this like perfect setting, you can have the self-driving car, do whatever you want. But in the real world, they're like complicated issues like, OK, there's a specific intersection in San Jose where when the sun sets at a certain time of the evening, that kind of looks like a traffic light and the car stops moving.

55:37Min-Kyu Jung:Or somewhere in Arizona, when the car is driving, you'll see a cactus and it'll mistake the cactus for something else and the car will get confused and make a wrong turn. And there's no like simple, like singular, beautiful algorithm that fixes all of these problems. It turns out you just need to work really hard and hammer down this long tail of each cases. And I think the thing we've really done well with our product is we have this feedback loop where our users just tell us through the product whenever they see an output that they're unhappy with. So we can actually track what the quality of the red lines look like.

56:09Min-Kyu Jung:And you can have a million evals. You can have whatever evals you want. there are certain areas where the evals are unlimited in the usefulness because there's an element of taste um for whether a human would prefer this red line over their own and i'll give you a few examples so when it comes to to redlining agreements one of the things that's really important is when you put a red line into the contract are you formatting it correctly right like are you if the contract is certain like list numbering are you like preserving that numbering and yeah are you preserving the like bold headings and all the rest of it um and that is an incredibly it sounds like such a stupid problem but but it's actually really important because the details matter yeah you're not going to be delighted from your your redlining tool and again there's no like simple uh like uh i don't know like algorithm you can build to to permanently fix all formatting issues it's just there's literally thousands of corner cases that you have to deal with one after the other after the other after the other and you hammer these down over time and your product is incrementally better like another example would be um when you generate the red lines there's this additional step we have to decide where are you going to add the deletions and where are you going to add the insertions like how are you going to call how are you going to present the red lines to um to the user and if you just get chachapiti to do this or or cord or whatever it'll be pretty dumb like if you if you get the lm to do this out of the box that'll just like delete a word and then insert another word and delete a word and then insert another word and no human red lines like that um so we spent a year and a half just building our own coalescing algorithm to handle a coalescing red lines in a way that seemed organic and human like um and then we shipped it and then it was a lot better but then there were still some issues and we had to um do all sorts of clever engineering work to handle all of these edge cases that we discovered as well and i think from like a product quality perspective you know so so we have our own evals which are really really important but maybe the even more important one is sort of having this intolerance or if a human says this is worse than what i would have done if i were the one redlining this and then having maybe more importantly a willingness to to deal with all of these issues um even if there are thousands of them and it takes literally years to to get to a point where it's noticeably better than than the alternative of doing nothing so at the beginning you you use

58:33Ivo:the word atomic unit for contracts. And there is an implicit opinion that you have that will then be represented in the database that sits behind the product. And whether you're aware of it or not, each time that you add functionality, you add data, you're obviously building the data schema or the database is growing. And at the center of it is the atomic unit of the contract. And each additional moment that like, if you started somewhere else, you're going to have a totally different set of outcomes that feed off whatever that other atomic unit was. So like, for example, with Slack, their atomic unit was channels so that like memory was searchable like workspaces um were identifiable whereas like um hip chat was like by person um and you weren't able to work in streams so you couldn't remember things as as easily so the product experience with slack was much better on that basis because it was just how we worked so maybe that brings the example to life a bit more i think one of the things we've really started

59:48Min-Kyu Jung:to appreciate is just how many processes within an organization depend on contracts to some degree right so for example if you're um if you're the cfo it's really important that you understand what percentage of agreements have a price increase provision or maybe if you're the um you know you're leading the security uh division of your of your business may and there's been some with security breach, maybe it's really important that you understand that what is remedy in the event of a security breach. And then these are real kind of examples that we see very frequently. And I think one of the interesting things about having contracts just be the source of truth for everything that you do and recognizing that you can now actually extract important intelligence from your agreements is there are actually very interesting workflows that are possible within in your organization that weren't possible before.

1:00:45Min-Kyu Jung:I think the CLM solutions are interesting because the promise of CLM solutions was meant to be that now that you can extract all of these insights from your contracts, we can set up all sorts of interesting workflows where if a salesperson wants to create a contract, we have these workflows that are set out. And then at the end of that process, we can extract useful information from those agreements. I think, unfortunately, CLMs haven't really lived up to that promise. And I think the main reason they haven't lived up to that promise is that contracts are complicated. And maybe one of the comments you said is like, what is the data schema for a contract?

1:01:21Min-Kyu Jung:But it's really hard to do that because contracts are unstructured data. And I think the mistake that the CLMs made was they tried to have these very rigid rules to accommodate agreements.

1:01:33Ivo:what are the core bets that you have over the next year two years three years that we can we'll be we'll be able to see whether you were right or wrong in a year's time i think one is like the

1:01:45Min-Kyu Jung:very obvious bit is that large language models will continue to get better um i think a lot of people were saying that my drop these were you know with plateau at gpt5 was even in self-plateauing And I think clearly that's not the case. So maybe the more interesting question there is, what does that imply for companies building products on top of these models? And I think the most obvious one is, increasingly the kind of user experiences that you need to build are changing in the direction of more autonomous multi-step workflows. And that's something that we've tried to do. And as we build the product, one of the things we keep thinking about is how can we orient the product in such a way that we're in a position to capture the gains of these improved models over time?

1:02:36Min-Kyu Jung:So, for example, the contract intelligence solution was useful for a variety of ways. Number one, because we become not necessarily the system of record, but we become the center of gravity for all of your agreements. Now we can do really interesting things on the contract review stage. stage. So when you're reviewing an agreement, we can proactively make recommendations for you based on the way you've negotiated agreements in the past. And that's sort of an example of a capability that maybe today is moderately useful, but as model capabilities get better, that becomes incredibly useful. Another example on that particular product is that it was at one point too cost prohibitive for us to run that product at all or make it commercially viable.

1:03:21Min-Kyu Jung:and they changed over the last six months and we expect that to continue to change going forward. The token maxing approach we're taking will be increasingly vindicated as models get cheaper. The second bet, and this is one that I've spoken about quite a bit already, is that we think that in-house legal and law firms are different markets and we're already seeing some degree of convergence where companies that were predominantly focused on one space are trying to focus on the other as well. and I think when the dust settles and the markets ossify because we're going through this sort of consolidation process at the moment where many of our competitors have been acquired by CLMs and other companies and companies are all coming together but when the dust settles we expect it to be a company or maybe a small group of companies but most likely one company that is dominant in the law firm space and then there'll be another company that is dominant in the in-house legal space and then you'll see a typical power distribution of outcomes.

1:04:23Min-Kyu Jung:I don't know if that'll be clear within the next year, but certainly within three years, I think that's a very falsifiable claim to make.

1:04:33Ivo:What have you learned from being in San Francisco that you're not sure that you would have learned, especially around company building, that you're not sure that you would have learned just from being in New Zealand? Was it the right call? It sounds like yes, and then taking that a step further, What are some principles or learnings that you've treasured?

1:04:51Min-Kyu Jung:Yeah, so it was definitely the right call to move to San Francisco. I don't think there's anything. The value of being in a different place is not necessarily distinct things that you learn. Because if it was just straightforward, you wouldn't actually need to move to that place. You just read about it in a book somewhere and there'll be sufficient to transfer the learning across. It's more that there are some things you might intellectually understand because somebody told you and it seems to make sense. But you don't internalize that lesson in a place like the Bay Area. One example of this is I really value speed.

1:05:31Min-Kyu Jung:I think speed is so important. I think one of the main jobs that a founder has is to instill a sense of urgency across the entire organization and make sure that nobody puts their foot off the gas at any point. But I think the degree to which speed is important or the degree of urgency that's required, I think needs to be a little bit more uncomfortable than what people are used to. And the great thing about the Bay Area is that there is just like a palpable sense of impatience from everybody here. And I think when you're in this environment and you have people who literally think the world is going to end in the next couple of years because of AI or whatever.

1:06:10Min-Kyu Jung:And there's this sense of something great is happening. And we have this limited window of opportunity to make something really special happen. You just can't help but have some of that rub off on you. And I think that's a huge net benefit from a company building perspective. And one of the things I really like about New Zealand and about Kiwis is that we're very relaxed and easygoing as a group of people, right? And I think that's a wonderful quality in many ways. But I think when it comes to building a startup, you do need to be able to move quickly.

1:06:45Ivo:What is one operating rhythm or ritual that you feel like is unique to Ivo?

1:06:51Min-Kyu Jung:Every single day, you're confronted with decisions that you need to make. And just on a personal level, you want to orient towards making these decisions faster and just doing more things and doing them faster than what you might do normally. And I think when it comes to working with your colleagues, it's about being in a space where you feel comfortable challenging them, not just me challenging my colleagues, which is in many ways easier for me to do, but our colleagues, because we all understand that speed is so important, our employees challenging their colleagues to move faster as well. And, you know, those are little things, just like if somebody says I'm going to do this thing on Friday, like, can we do this today?

1:07:43Min-Kyu Jung:Like, is there a way we can cut down the time that's required to get this thing done? Is there a way we can ship this feature faster? Is there a way that we can figure out this important problem that we have sooner rather than waiting for some event to happen in the future. So there's no sort of quirky thing that we do. It's more, I think, when everyone understands that we're in the precipice of something really special as a company and we have an opportunity that, in my opinion, maybe we'll never see again in our lifetimes, which is this incredible platform shift. And then everyone internalizes that.

1:08:24Min-Kyu Jung:I think we push each other to do more than we would otherwise. By the way, one of the interesting things there is that I think founders have to have the opposite mentality of lawyers. With lawyers, you're thinking about the worst case scenario and how things could go wrong. As a founder, ideas are very fragile and you have to think about the things that could go right. And kind of ex-ante, the probability of startup success is low. so you need to think about the best case outcome and focus on maximizing that.

1:08:55Ivo:Well, look, I'm so grateful that you joined us on Wild Hearts and I just deeply admire that you built such a special product that is helping so many in-house lawyers. I think that's really, really exciting and really, really hard to do and it takes a tremendous amount of guts and I'm stoked to see where those guts take you over the next couple of years. Thanks. Really nice to talk to you. Thanks for having me on.

1:09:48Ivo:Blackbird team and day one. The show is produced by Camilla Herring and Melia Rayner at Blackbird. Our marketing genius is Eva Telemachus and our editors are from day one, Annie Jones and Sanjay Chabria. Thank you all so much for listening and we'll see you next week.