Why the Future of AI May Be Smaller: The Rise of Domain-Specific Models

24:38But majority of human beings, he says that using narrative thinking. For example, you enter a room, you see a scene, you immediately connect these things and you build a story that how time and events have developed at this scene. And that's through the narrative. So I was thinking that can we build narrative machines? Can we build a machine that can see dispersed or maybe contradictory pieces of information and generate a cohesive, plausible narrative. And I was thinking if we can do that, we build the AI. So I spent a year when I started my PhD and learned that with symbolic AI you cannot. By the nature, it is too rigid to allow for these plausibilities, possibilities.

25:20What we now says that LLMs are probabilistic, they look at the context and they come up with the things that we probably express as human experience in written form on blogs and internet. And that's how it's trained. So it was a good experience, but failed miserably.

25:40Matt Paige:You were ahead of your time. It sounds like narrative machine and whatnot. It sounds a lot like a large language model and concept in some ways. You just didn't get the transformer architecture figured out yet. Yeah, I think it was way early even with the deep learning at the time. But when we start looking into this case, I just had a flashback. Oh, that thing which was so rigid and always just, it doesn't have these different possibilities and modalities of output. Maybe that's what we want in the legal work, specifically when we are looking at like business transactions. Because if you go to litigation, that's a whole different, very complex creature.

26:19But in business transactions, things are very rule driven. and you always expect to follow this. It was interesting. My co-founder, Mark, spent like the past three years because he has this conceptualization that any legal work is to mitigating unforeseeable risk. That is the way that I think. It said that when you hire a lawyer, when you are purchasing - Unknown, right? Yeah, exactly. When you hire a lawyer, you're purchasing a house, you want to mitigate what are potential risks that might come. You try to predict those surfaces of risk and protect yourself. So he said that, okay, can we just define what is a knowledge graph of what are the risk surfaces in every aspect of a business transaction?

27:02And we can codify that as certain like rules and positions that you prefer to take and we call it the risk profile of a company or the risk preferences or appetite of a company. For example, if you're dealing with patient data, probably your risk tolerance in terms of data protection is very low. You don't want to risk anything. But if it's a minimal use, you are not collecting anything. Maybe your risk tolerance is higher because you are not collecting any user data. So he spent some good amount of time. And then when we're talking, he said, okay, this is what we call it in the symbolic AI ontology, which is a representation of the domain that we are working in logical terms.

27:43And there is a well-established technologies and standards for there. So we can do that and then we can delegate all of the reasoning about legal risk to this deterministic engine because we know what is your preferences as an organization. It become an instance of a global ontology which instantiated means that it got the real values of your risk preferences. Every time I ask it your net payment terms it is net 30 days under these contracts goods and manufacturing it might be i don't know 15 days whatever so we can do that and then we take out the major component of reliance on llm which is the reasoning out of the equation with two benefit cheaper faster compute and always deterministic outcome so this actually has a name called neurosymbolic ai so that you get the neural component of llms combined with the symbolic component of ontologies and symbolic AI, you have a semi-deterministic system that still can benefit from some of the aspects of leveraging large language models.

28:54But the certain places you want it to be deterministic outcomes, it can be built into it.

29:00Matt Paige:Neurosymbolic AI, I'm going to start using that. I'm learning something new every day. I'm curious though, the interesting thing to me is so much with the legal field and a lot of fields out there is it's not always binary. Like there's a sequence of decisions and events and things you could do. Like, yes, I could do this thing from a legal standpoint, but it may have these other related consequences that may lead to some other thing down here. And my mind immediately went to another episode I just recorded the other day. And we were talking about the recent OpenAI and Hugging Face debacle and whatnot.

29:39Matt Paige:And the thing that was interesting, It wasn't necessarily the zero day attack per se, but it's like how it did it. Right. So it like sequenced together multiple vulnerabilities, failure points and all of these things, ultimately getting to where it was able to break into Hugging Faces infrastructure, ultimately just to do its benchmark, basically cheat on the benchmark. But it was that sequence of events that it was able to go through, which was just super interesting to me. And I got to imagine maybe not so much as much in procurement, but I could see in the legal field, like there are just in number of possibilities based on a decision you make.

30:20Matt Paige:I don't know if there's a question in there, but my mind's just kind of like connecting the two things together. I'd be curious your thoughts. Yeah, actually, this is something we were discussing because you said that nothing in that experiment in the open AI was in any way asking the LLM go hack another company outside of its sandbox. Exactly. They just gave it a goal of you're doing the benchmark, right? Yeah, and that's another argument that sometimes I have that you should not use like an intercontinental ballistic missile to hit, I don't know, a fly when you can use a fly swagger. And in some cases, I was thinking that because these frontier models can go beyond, they have these extreme capacities and you give it a goal, all of a sudden he tried to find shortcuts.

31:09Or if he cannot, he tried to other, that unpredictable behaviors. That is another argument why we need to use the smaller domain focused language models. that they are dumb enough that they cannot do anything beyond the scope that I defined for them. Or if they want to do it, they do such a poor job that actually medicate the risk. For example, if I have a 9 billion parameter model that I find you in domain adapted for legal, it could never hack its sandbox way out and try to go steal like the other parties' information hacked to their systems.

31:44Matt Paige:Oh man, yeah. But that can be a case. And if you are in a legal case, I don't know. I would not be surprised if we see that at some point. Yeah. So our agent went straight and hacked to the computer and get the privileged communication between the lawyer of the other party. Because it's interesting to that point, like you're setting the goal, but there's things you're not considering that if you're not giving the LLM context to certain things, like you can still achieve the goal and there could be all kinds of alternate impacts and effects that you're not really considering, I suppose, is the issue there in a sense.

32:22Yeah. And again, I'm sure that all the engineers, they're offering, creating these harnesses, they put the guardrails. But because of the nature of these models, we cannot predict all of the possibilities that might come out. I'm sure that probably now they learned and OpenAI have much better safeguards in place until we know the next time that this thing escapes and there is another opportunity. So that's why I think that in terms of both efficiency, safety, and just building something that is properly sized to the size of problem, I argue for small language models. They are still very powerful.

33:02I remember. In 2020, indeed, we built some early transformer based on birth came out to extract attributes from job seeker resume and job description. and it was in 60 different languages. And that was a massive success. And we built it on the models. We were like, now it looked like toys, like 150 million parameters. And they perform well for the task. They reach like 93, 94 % recall and precision for those specific tasks. So that's enough, in my opinion. We don't need to run like a massive 3 trillion parameter model. And those models, if you think most cases are mixed up experts, that the active parameters are 30 billion.

33:47So it depends on which road it goes through, which area of expertise it takes. So that's why if you're talking about legal contracting, that area of the knowledge is so small, you don't need that. Probably a 9 billion parameter model is more than enough that can capture all the knowledge you want in that area and perform well. And I think we have good frameworks for knowledge distillation. You can benchmark. This is what we are actually doing. We do knowledge distillation against the leading model to get it up to the point. And beyond that, we look at these new methods such as reinforcement learning way of self-distillation.

34:26It means that the model is going to reflect on its own mistakes from the signal from environment figure why I made this mistake.

34:34Matt Paige:Is that not just reinforcement learning? or is it a form of re... So you can do reinforcement learning with verifiable rewards. So say that, okay, this was the better option. This is the worst option. But now it is what they call it a self-distillation policy optimization. It means that it look at the context reflecting on its mistakes in the past and try to figure out why I made this mistake and then get it as an input in the signal. So you can do that. And then there are the cases that even the frontier model missed. So we human expert labels, we're also going to fine-tune it on those. At that point, our 9 billion parameter model, which runs in one second, sub-second, on a commodity GPU, performs as Fable 5.

35:18But if you ask it about astrology or who was the MLS champion, it just produced garbage nonsense. Which is fine because you don't need it for that. Yeah, and actually that's good. So that's why it cannot hack its way out of the sandbox.

35:32Matt Paige:Yeah, that's interesting. Tons of good insights here. It's funny you mentioned distillation because you have Dario on one side speaking out against distillation specifically with a lot of the Chinese firms and companies. But you could argue that distillation was used in terms of building the bottles in the first place. And then I think Sam Altman mentioned something where it's just a common practice they use even in building their own smaller models and things like that as well. Curious your thoughts on distillation as a practice. Yeah, so I think the argument maybe for the frontier model is that if you're following me and you try to replicate my frontier model with two trillion parameters and you're just a copycat to distilling knowledge, you're not doing the hard work.

36:13but when you're building a very small tiny model which is domain specific i think that's very legitimate use of it because you might use a committee of the way that we're using you use a committee of frontier models to create what we call it silver data sets and then human review to bring it to gold level that even those five six percent errors that they are doing or seven eight percent errors can human experts resolve those and then we have much better data to train these smaller models and again i think there is a lot of argument for how we can use it across so again part of it is also the the impact that we have on the energy consumption the prices of electricity for consumer and environmental impact yeah i'm i think there you're gonna see a lot of businesses turn to small language models, in my opinion, to solve very specific business problems that economically doesn't make sense for them to spend like$15 for a million token output.

37:20Matt Paige:Yeah, this is the model. You get your AI expert, which is you. You get your domain-specific expert, which is your co-founder, and you just go create the AI native service company. So I want to get into this point. You mentioned this earlier, the billable hour, time and materials. Majority of service-based businesses in large part are built on this model. And it's funny, I saw, it probably was like a Harvey or Legora ad I got at some point, I was probably doing some research, but they had the TV show Suits and they were a part of the ad where they'd be doing the thing, they'd pick up the paper and they'd be like, oh my God, I know exactly what to do.

37:52Matt Paige:It's like, you did not read the 100 page brief, but like that's possible now with AI. But the whole unit of measure was the hour, the billable hour. How does the business model evolve when that unit of measure is completely broken in a sense, or is it just you do much more within that hour? I'm curious, what you think will actually evolve? Because the logical thing is outcome-based, but outcome-based is easier said than done, I think, in some forms as well. yeah i know we don't know the dust has to settle and we have to see where it ended up but i think there are some possibilities one is that outcome based and maybe they bring in more machine learning maybe traditional machinery in a sense that okay i look at the case i look at the facts i like who is going to be the judge you look at the track history of the judge rulings the context the jurisdiction and we predict what is our success rate in this case and then we're gonna have an estimate of how much time going to work.

38:58Now we have AI enabled lawyers that can do 10x or 5x of work. And they do the math and says that, you know what, our expected spending on this is, let's say, I'm going to make up$100 ,000 of the legal time. But the expected value means that 70 % of the probability and making$500 ,000 is$350 ,000. So that's a very good deal in terms of expected value. You might lose it because you have 30 % lost chance. So that's in the long run, if you keep doing it, maybe that is one framework that you start moving that they're looking into that. The other framework, which is probably in the short term, more plausible is that you're going to get AI enabled law firms.

39:43So they said that, you know what, you still charge your bill over but we are ai enabled so this writing this summary judgment brief which used to take 100 hours we're going to do it in 20 hours so you're going to get much lower cost so that's another for and i think probably maybe it is not the individuals who are interacting with these law firms but probably the businesses like large companies now going out to their outside counsel says that, tell me how you're using AI. I wanted you to start using it. And I want you to provide me better service at a cheaper rate. And I think competition, I think we touched a little bit about it that some of the states start changing the regulations that now investors can come, tech companies can come investing in the legal practice.

40:37And I think that might introduce new creative business model and pricing models. Yeah. That you have to see how it comes out of it.

40:46Matt Paige:I think the other interesting thing too, because I think humans just naturally have difficulty doing this when you're thinking logically through things, but also in terms of exponentials and whatnot. But there's a massive market for legal services that is completely untapped. And we just don't think about it because we don't see it, right? But think of people that may be disadvantaged for one reason or another, that just, they're just not going to hire a lawyer or they're hiring a subpar lawyer. There's this whole market of folks that are underrepresented in a sense, and legal is the domain we're talking about.

41:22Matt Paige:But I think there's all kinds of different services. We were just talking about accounting and whatnot. I think there's the same angle there, but I'm just curious the democratizing effect of AI and maybe legal specifically, how do you see this playing out over the next, it's hard to do time horizons, maybe five years, I don't know. It is interesting you brought it up. So we start building this AI product, of course, for procurement, big enterprises, they have tens of thousands of business transactions a year, because that's a scale that makes sense for them to start using and leveraging and have better efficiency, faster business transactions.

41:59But when Mark and I were talking and he was because i had a really bad experience with legal processes and costs and i said that you know what i think for a democratic society is exactly what you said it is very important that every citizen has access to quality affordable legal services that they can protect rights because in our society it becomes if you are a very rich individual or a corporation you You can actually coerce people sending a letter. And again, when you're dealing with a 500-pound gorilla that has a legal department with a billion-dollar budget, you can't fight them. You can't.

42:39It costs too much money. It costs too much money. And again, so maybe that make it... So the other aspect I was thinking like public defenders. Now, probably public defenders might have better resources to defend the people who, I don't know, of course, in different cases of criminal law or other cases, when the defendant cannot afford to present themselves. Another aspect interesting, I think two or three weeks ago, New York Times has a piece about that. Previously, courts encouraged people who cannot afford a presentation to present themselves as pro se. And some people could not do it. They didn't know the basic ropes, how to file a motion, how to file a complaint.

43:20and in the past year or so the course i think is skyrocketing pro se representation a lot of people it turned out using literally chat gpt and claude and other ai and part of it is that how can i write a complaint how can i go to this 1999 website of this specific jurisdiction which is extremely non-user friendly to upload my documents, get the things, or how can I read? Someone sued me. My landlord sued me. So how can I read this and understand? And how can I respond? What does it mean in layman's terms? What does it mean? Yeah. In the human language. So we will see, I see there is a massive transformation.

44:01One going to happen at the business level in terms of how legal business going to charge people moving forward, how this new opportunity for non-lawyers getting involved in the space in terms of investment and presenting new business models. And at the bottom level, you also see that AI empowers some of these individuals who did not have resources, maybe by using a$20 subscription, they can at least understand what is this weird legalist language they got, what does it mean, and what are their options to act upon. So I think a lot of these movements happen. We have to see whether it's going to be for better or worse.

44:40but I think it empowers a lot of people in my opinion. Yeah.

44:45Matt Paige:How do I get out of jury duty? ChatGPT, tell me. That literally is what gets me most excited about everything. There's so many negative things that are mentioned about AI, but I think there's so many positives that people aren't necessarily considering. I love that specific use case, but I think you could apply it to almost any domain. It's almost like this new check and balance in a sense across society. At least that's my hope as we go on. There are a ton of things and I can tell you a personal story. So there was something happened with a federal institution and they need to get some information from them, but they say, I have to file a FOIA, Freedom of Information Act.

45:24Yeah. And I said, okay, I don't know. And I went to the website and all of a sudden I wait, I opened my AI agent said that I want to file a Freedom of Information Act, walk me through and it just walked me through. Okay. you have to go here, click on this. And in five minutes, I filed it. So this is, as a citizen, now you can hold your government accountable because now you have something that wide you through. And it was because of really bad setup of the user experience, how are we going to file that? So this is the benefit. The other thing, I am sure AI is going to create a lot of massive displacements in the job and other things.

46:00But one of the positive sides, I see that it also opens up new entrepreneurship opportunities. Let's talk even only in the area of software development. Let's say if you wanted to build this enterprise software, you need 150 people to just build that. Now, if Amazon lay out five of those talented engineers, I think those five can get together. And using AI-enabled development, in six months, they're going to build something that Amazon probably, and they might start eating up into the Amazon share in the market because they are nimble, small, and they can move fast. So this is the opportunities.

46:36And even in the small business, like you have a, I don't know, you wanted to sell baking goods. I think AI might be able to help you to, how can I optimize? How can I reach other people? How can I set up a website? And those things can maybe lower the barrier. I think the main challenge is that how can we tell people how they can use AI? and there are two concerns that they have with AI is that at a high level as a society how do we decide that where the risk outweigh the benefit in terms of like the incident with OpenAI and Anthropic also says that oh they learned they didn't know after OpenAI incident they went look at the logs and noticed oh they hacked three organizations and they had no idea.

47:23That's what the other part is that how can the next generation of kids who are in the school learn to think independently, not fully become reliant on immediately processed nuggets of information from AI. Because this is a challenge I have with my nine-year-old. He's very bright, but he was saying, I'm doing research. I said, I'm talking with co-pilot. And I said, that's not research. You're asking questions and he gives you answers. Because I think critical thinking is still very important to carry.

47:53Matt Paige:Yeah, that's the key thing. And so last topic I want to hit with you. You mentioned how easy it is now to build. And I feel like you're a perfect use case building risk advances. I'd be just, I'd be curious how you would contrast building this today versus what that may have looked like in, let's say 2015, because you're a data scientist, engineer and whatnot, but I'm assuming you've been able to build this business with much faster and with much fewer head count than it would have required 10 years ago. yeah this is unbelievable sometimes i think to myself how is this possible that you build a whole enterprise applications yeah in this compressed period of time and i'm thinking about even like five six years ago we were working this big enterprise and we wanted to do a small release and there was like whole group of people involved and take six months and then i think it is the people who can leverage it that's why i said that opportunities for entrepreneurship is the problems that we have in society and different aspects of business and everything are numerous so we are not short of problems to solve but now we have the opportunity to a small group of people can really solve big problems and that's my speak it's something we can think i'm dealing with some of this government websites or portals or services i see them man And these are so outdated.

49:18And I think, can't we now modernize our government very efficiently at much lower cost with the taxpayer money, but start offering the taxpayers better service for the money they are paying the government?

49:32Matt Paige:That's the innovator's dilemma, though. That's the problem is like as an existing organization, public or private, government or not, it's like it's almost getting out of your own way, I feel. I think that's why it's so much easier starting new because you just don't have the baggage. you're just starting over in a sense that is very true but it is extremely fast and especially again if you know how to now and it is keep changing if you look at even best practices coming out of entropic they said that hey three months ago we were prompting our systems this way but now we learn we have to clean up shorter prompt space context progressive disclosure so you have to stay on top of it, but it is unprecedented in any form or shape the way that now we are developing software.

50:19Yeah.

50:19Matt Paige:Ahmad, it was awesome having you on Talking AI. Great conversation. So where can people, I'm sure we have plenty of people that deal with procurement, whether they enjoy it or not, but where can they learn more about RiskVantage AI and more about you and where can they find you? Yeah. So we are at riskvantage.ai. So you can visit our website. You can contact us there. And you can see a demo of the products we have already launched. We launched our first product, Risfaction Negotiate, on 4th of July. So we are ready for business. Nice. Nice. That's awesome. Thank you for jumping on and talking some AI.

50:55Matt Paige:Thank you so much, Matt. Thanks for listening to the Talking AI Podcast. If you enjoyed the show, give us a follow or subscribe on your favorite podcast podcast. And don't forget to leave us a review. We love this. For more info on Talking AI, visit TalkingAIPodcast.com. quick break in the pod if you're listening to this podcast chances are you've been thinking about how to actually use ai inside your business and that's exactly why we built the ai opportunity finder it's a free tool that helps you uncover high impact tailored ai use cases based on your business your goals your pain points and your industry no fluff no generic use cases just real ideas that fit your business and the ranked by roi potential it takes about three minutes to run and it's like having your own personal AI strategist for free.

51:42Matt Paige:If you want to try it for free, check out the link in the show notes or go to hatchworks.com backslash AI dash opportunity dash finder.