#266 Andy Kurtzig: How Pearl Uses AI + Human Experts In Professional Services

0:00Pearl will give you an AI answer and then you'll get an actual doctor will show up read the conversation, read the answer, and then give it a trust score from one to five. We focus on primarily text and professional services, so medical, legal, veterinary, accounting, tech support, home improvement, these kinds of areas. And that focus yields a very different approach to how we handle things. And that approach results in a significant improvement and quality for the user. We do. We've got 12 ,000 experts that are on our network. Like I said, we've got pearl.com as part of a collection of businesses.

0:42And across those businesses, we have about 12 ,000 experts that are at the beck and call of our pearl.com users. And so they're there to jump in and verify these answers. We've figured out how to monetize it. And that is through real life human expert professionals that the human expert needs to be in the loop, People understand that a doctor costs money, a lawyer costs money, an accountant costs money. Fraction when you do it through AI and online. But people are used to paying money for that. And that creates a business model here that is both supportive, that improves the quality, reduces the risk, and creates monetization opportunities.

1:19Building multi-agent software is hard. Agent-to-agent and agent-to-tool communication is still the Wild West. How do you achieve accuracy and consistency in non-deterministic agendic apps? That's where agency, A-G-N-T-C-Y, comes in. The agency is an open source collective building the internet of agents. And what's the internet of agents? It's a collaboration layer where AI agents can communicate, discover each other, and work across frameworks. works. For developers, this means standardized agent discovery tools, seamless protocols for interagent communication, and modular components to compose and scale multi-agent workflows.

2:12Build with other engineers who care about high-quality multi-agent software. Visit Visit agency.org and add your support. That's A-G-N-T-C-Y dot O-R-G. Visit them today to support high quality multi-agent software. Go ahead then. Well, it's nice to be here and happy to give my background. So my name is Andy Kurtzig. I'm the founder and CEO of Pearl, pearl.com, like a pearl of wisdom. And what we do is we take solo professionals and turn them into AI superheroes using essentially agents to do the sales and the marketing and the customer service and the account management and payments and all that for them so they can focus on what they love, the actual human judgment expertise part and not the other stuff.

3:08And I've been doing this for sort of blending AI and humans together for roughly 20 years now. Wow. And really proud of how far we've come. Yeah. When did you found Pearl? So Pearl is a subsidiary essentially of Just Answer, which I founded 20 years ago. Oh, okay. Great. Yeah, it's a collection of brands, Just Answer, Pearl. I've got a couple other businesses as well. Yeah, so Perl is what I understand is an AI platform that combines LLMs and a network of human experts, excuse me, to verify responses. Can you tell me about, I mean, it must be tough in this environment to compete with ChatGPT or perplexity.

4:03Can you tell me how you compare to those guys and how you manage that? Of course. So first of all, we're very focused on the professional services sector. So we're a vertical, they're horizontals. They're doing everything from coding to video and images and everything in between. We focus on primarily text and professional services. So medical, legal, veterinary, accounting, tech support, home improvement, these kinds of areas. And that focus yields a very different approach to how we handle things. So, and that approach results in a significant improvement in quality for the user, for the consumer.

4:51So specifically, we're 22 % more accurate than ChatGPT and Google Gemini in professional services. And professional services, it runs the gamut from legal to healthcare to, I mean, how broad is that vertical? Yeah. So if you take all the subcategories, so even within medical, you've got everything from oncology to dermatology and endocrinology. If you take all the subcategories, you put them all together. It's roughly 700 different categories that we focus on. And they all have some very similar characteristics. Number one is that quality really matters. And humans are a key element of delivering that quality.

5:43obviously in medical or legal, for example, making a mistake is extremely unfortunate and costly and not a good place for lots of hallucinations. Sure, sure. And how does that work? Then you have this network of human experts. Someone asks an LLM a question, or just describe how it works. Yeah. So, so it's fairly straightforward. You go to pearl.com, you type in your question, it works kind of like a, like a chat GPT or Gemini or perplexity. At the beginning, you type your question, and then it'll go back and forth with you typically more like a little bit more like some of the deep research tools out there do, and qualify and actually take, get rid of the need for fancy prompting, because we find that's a big source of, of hallucinations is, is the prompt that the average person puts in isn't great.

6:45And so then the output isn't great. So we take care of all that. So you just put in your language, what you're trying to do. I have a rash on my foot or whatever it is. And we will go back and forth if you clarify the question, just like a doctor in a live environment would. And then we actually give you the AI answer from pearl.com. That's just the beginning. That's what's 22 % more accurate. And more importantly, actually, than the 22 % more accurate, it's 41 % less wrong. So when you're dealing with medical, legal, these kinds of accounting, these kinds of categories being wrong is what you're really trying to avoid, because that's where the catastrophes happen.

7:27And so that's the first step, higher quality AIL answer. But then the additional twist here is we will then have an actual human expert verify the accuracy of that AI response, of Pearl's own response. So in that case, with your rash on your foot, Pearl will give you an AI answer. And then you'll get an actual doctor will show up, read the conversation, read the answer, and then give it a trust score from one to five. so that you then know okay can i trust this thing or not and if you can great have a nice day and if you can't well then you got to keep digging yeah you know i've there there are a lot of these systems now that have an llm verifier that that looks at the answer from another llm and scores Is any of this automated?

8:30And if not, how do you do it in real time? I mean, do you have experts in each particular field on call, you know, 24-7? Or, yeah, how do you manage that? We do. We've got 12 ,000 experts that are on our network. Like I said, we've got pearl.com is part of a collection of businesses. And across those businesses, we have about 12 ,000 experts that are at the beck and call of our pearl.com users. And so they're there to jump in and verify these answers and read them and give the trust scores. And then if you want to, as a user, you can then choose to pay and then actually end up on a phone call or a live chat with the doctor about it for as long as you want.

9:22So that's kind of step three of the experience. And that's where the money comes as well. That's where the monetization happens. So the AI answer is free. The trust score is free from the human expert. but then if you want to talk with them then you can pay a monthly fee and talk to them all you want and then you pay a monthly fee it's a not per call yeah that's right okay and uh so who are the users in this are they casual users that want one answer uh for one thing or is it's uh other medical professionals or other lawyers, for example, or paralegals that, I mean, is there, is it more a B2C or a B2B product?

10:11It's interesting. We launched it as a B2C product and we're finding a lot of power usage out of the B2B group. So like you said, lawyers or doctors or mechanics or accountants that just need help. So I'll give you a funny story. In the mechanic category, we're seeing a lot of mechanics that might be in a small town like the middle of Alaska or something. And they're the only mechanic in town. And so you can imagine they might know how to deal with a Ford or a Chevy, but then somebody brings them a Volvo or a BMW or a Saab or whatever it is. they don't know how to repair those cars, but they're the only mechanic in town.

10:55And so they turn around and start using us to be able to service those cars as well. Well, that's fascinating. So, and how quick is the response? I mean, if I'm a mechanic among your experts, is it through my smartphone that I get an alert? Yeah. I look at it, and how much time does it generally take to score the LLM response? It usually takes a few minutes. It's not very long. Wow. Yeah. And then you pay the expert for their time and take a commission, in effect. Is that right? Yeah. Yeah, well, so for the verifications, we pay the expert for their time and then give it to the users for free.

11:53So we're eating that cost in the hopes that some customers will choose to then become subscribers. Yeah. And what is the subscription fee? It's about$30 a month. I see. Yeah, so similar to any of the LLM. Yeah. yeah except for that includes actually talking on the phone or on chat live with a human doctor or a lawyer right yeah well that's fascinating yeah yeah uh and and on this back and forth you talk about uh so you ask it a question uh pearl asks these nuanced follow-up questions to better understand your intent um and and are the have you refined i mean fine-tuned the models that you're using with uh answers collected from these uh experts of course so that's a key part of what makes it 22 more accurate than than chat gpt and google gemini is is is the rag of all of our data.

13:08So we've got about 30 million past conversations between doctors and their customers and lawyers and the customers, et cetera, in our database that we can then use to increase the quality of the answers. Yeah. And that fine tuning, what's the base model or do you use multiple LLMs. We use multiple LLMs underneath as well. So we've got we've built our own eval system as well because we've been doing this for so long that evaluates the various different models for various different purposes. And so every time you use Perl you're using at least two different LLMs or happening behind the scenes. It's all invisible to you as the user but at least two different LLMs are being used behind the scenes.

14:01Yeah. And so there's, you know, if you're dealing with healthcare or legal, there's a potential liability issue. How do you handle that? That is an excellent question. So let's start with how do the LMs handle that? How does ChatGPT handle that? How does Google Gemini perplexity, et cetera, handle that? And the short answer is they don't and they can't and they're liable and we've got a bunch of data on that everything from some pre-court justices to law journals to even consumers and their intention uh we did a survey recently and asked consumers if if an lm was wrong you know what would you do about it and uh 40 said they would consider legal action against ai companies and and 50 percent of americans believe the ai platforms are legally responsible for the answers they provide so so that's a giant flaw in the foundational models big game plans uh there's actually three big flaws that's one of the three big flaws we can chat about the other two if you like but we'll focus on risk for now um huge flaw i mean they're starting to see the pain of that right you've seen the lawsuits, that kid that killed himself because AI told him to, and all these things.

15:23I mean, it's just tragic. And the family of that kid and the families of some others that have been given hallucinations, quote unquote, are now suing Google and some of these other companies. And I think they're going to win. They don't have an argument against it. Now, with all that said, there is this thing called Section 230. It's an act of Congress that has protected Google and Facebook and many of these other companies from that exact kind of lawsuit in the past. But there's a key way that Section 230 works. It works for platforms. And a platform or a marketplace is where somebody might come, like even on Google, you go to Google, you get a billion search results, you click on with some of one of them and you end up on some other third-party site.

16:12And then that third-party site is the one that's then giving you the answer and that's how they deal with it. Google can say Section 230, we're just the platform, you can't sue us, and that's worked like a charm for them for years. Until now, where they're not doing that, they're not sending you to some other site, they're not even attributing any other sites most of the time. They're just telling you, you're going to be fine, don't worry about the cancer or whatever it is that you're asking it about. And they're liable for that. And when you say they're not going to a third party, it's because they're letting their LLM answer.

16:47That's right. That's right. And they often don't even know the source. It just, all that data just kind of got sucked in and then they give you the answer and they don't even know if it's true or not. And they also don't know where they got it. And so they don't even attribute it. Not that that would solve it, but they're not even close to solving it. And so that's the piece where Perl actually steps in and can really help out. So the interesting thing about what Perl.com is doing from a liability perspective is, yes, we have the AI answers, but then we're very forward with the human expert verifications and conversations, which turns us back into a platform again, which gets Section 230 protection again, and it solves that big problem.

17:33Building multi-agent software is hard. Agent-to-agent and agent-to-tool communication is still the Wild West. How do you achieve accuracy and consistency in non-deterministic agentic apps? That's where agency, A-G-N-T-C-Y, comes in. The agency is an open source collective building the internet of agents. And what's the internet of agents? It's a collaboration layer where AI agents can communicate, discover each other, and work across frameworks. For developers, this means standardized agent discovery tools, seamless protocols for interagent communication, and modular components to compose and scale multi-agent workflows.

18:27Build with other engineers who care about high-quality multi-agent software. Visit agency.org and add your support. That's A-G-N-T-C-Y dot O-R-G. Visit them today to support high-quality multi-agent software. I see. And the liability, if there is a wrong answer, does that extend to the experts or are they protected by 230? no so that so the experts are are are are the ones giving the answer and so they're they're they're the ones that are held liable in this case but right and then they've got their own insurance and they've got their own ways of dealing with that right they know how to deal with you know medical malpractice or whatever it is that's something that they're used to and have have protections for.

19:22Yeah. And have you tested how uniform are the answers across experts? That's something I've always wondered. Yeah. You know, it's interesting. You kind of, in some ways you want uniformity, but in other ways you don't, right? So when you've got a medical condition, you don't want everybody telling you, sorry, you're out of luck, have a nice life you want a bunch of different angles and a bunch of different perspectives and points of view and sometimes the person says sorry you're out of luck have a nice life the next one says oh there's this new thing over here have you heard of that you might want to try this i don't know if it'll work and so some of the the differences are actually useful especially when you're dealing with with tricky diagnoses and things like that well that's interesting so So if I have a medical question, I go on Pearl, I ask it, I get an LLM answer.

20:18Do I have to prompt it for the trust score? Nope, it'll prompt you. It'll say, hey, you know, it'll prompt you. And sometimes it'll even just automatically do it for you, depending on the situation. Right. And then I get the trust score. And do I have to ask for the, to talk to the expert then? Yeah, if you actually want to have a phone call or a live chat with the expert, you have to ask for that. And you have to sign up for the subscription. Yeah. But can I then talk to two experts, get a second opinion if I don't like the first one? Yeah. Yeah. With the subscription, you can absolutely do that.

20:56And that's one of the beauties of this is you can get multiple opinions really easily. Right. Part of the reason a lot of people don't get second or third or fourth opinions is because it's so hard. You got to make an appointment for three weeks from Tuesday. you gotta drive down there and you show up and you know they're late and whatever else happens you know now you can sort of spam bam bam get three different opinions very quickly yeah actually that sounds amazing i mean did you consider a charging per session or per question because a lot of times people only want one answer. They don't necessarily want a subscription.

21:40Or you think the subscription, can you sign up for a month and cancel immediately, and then you've spent$30 to get an expert opinion, which is extremely cheap. How does that work? Yes. So first of all, it's very easy to cancel. So yes, you can just sign up, get your one or your two or your three opinions or whatever you need. You got a whole month to do it then, and then you can cancel and only pay 30 bucks for all that. And yeah, but what we find is people really like the ability to sort of ask a second and a third opinion. And a lot of these things kind of end up being maybe chronic issues. And so you want to be able to ask them over time as well.

22:25And this is a way that we can price it lower. So obviously if you just, you know, without insurance, showed up at a doctor's office and said, hey, I want a painting, they're going to charge you hundreds of dollars for that, plus the drive time and everything else you have to deal with. You know, we're charging 30, and you can get multiple opinions that way. Yeah, yeah. Okay, the – I'm sorry, I'll cut this out. I just had a message pop up that knocked my train of thought askew. No problem.

23:06I'm reading some of these notes.

23:15Yeah, I'm seeing this note about why Perl is the first truly monetizable AI search platform. platform. Can you go through that? Yeah. So let me frame this in a context and then I'll answer that question specifically. So I believe there are three fundamental problems with the foundational models out there, the chat GPTs and the Google Geminis and perplexities and such. And we've already talked about one, which is risk. They're liable for huge amounts of risks. I mean, they're going to end up with billions and billions of dollars worth of lawsuits from everybody that that they hallucinated to and caused damage to and and it's not a big deal when you're writing a poem maybe but for the family dinner party but when you're dealing medical legal these kinds of categories it is a big deal and they're not taking that seriously enough uh and until they're giving out wrong answers and they're hurting people and they're hurting pets and they're hurting people's legal situations, et cetera.

24:21So risk is a big one. Second issue of three is this quality, the underlying quality, right? So we're seeing hallucinations like crazy. And in fact, with these new versions of models that just came out, they're actually hallucinations went up, not down. This is a fundamental part of how these foundational models work is that they hallucinate. That is just embedded in how they structurally work. And so those hallucinations, again, are a big problem, and nobody's got an idea on how. Everybody's got lots of ideas, but nobody has a clear path on how to get past the hallucinations. They just seem like they're just a part of these symptoms.

25:05And then the third, of course, is monetization. We're seeing in the news left and right, not only these AI companies, but many companies in and around the space are losing money hand over fist right they're spending this much on gpus from nvidia and there's they're getting this much in revenues from you know if they're getting any revenue at all uh you know nvidia is making a lot of money off of this whole thing but but most of the foundation all the foundational models and and uh and big players are losing lots of money on on ai today and so those are the three big fundamental problems with the with the foundational models It's quality risk and monetization.

25:44You asked specifically about monetization. So let's talk about that. They don't have a good monetization method yet. Charging 20 bucks a month kind of works for early adopters like you and me and small businesses and stuff. But that's not the model that's going to end up winning. You know, something more like what Google does with a free model is what's going to end up winning to reach the masses of consumers. And we're seeing the free models out there. that's lovely uh but it's they're not figuring out how to monetize it yet so where are the ads gonna go and what are you gonna do when when when you ask what's the best tennis shoes to buy and the answer is nike but the but the advertiser is adidas right what are you gonna do right and and so they don't know what they're gonna do with that and and it's not like kind of google where you can just say well we kind of suck anyway here's a billion links and a billion ads and you just sort of you figure it out they're trying to tell you an answer and when you start getting into the answer business it's tough to to to both give a great quality answer and uh put ads in there yeah you can imagine asking for the score of the you know nba game and you know one model decides to say, well, before I tell you the score of the NBA game, let me tell you about the all new Toyota Tundra, right?

27:10Nobody would go to that model. They'd go to the next model down the list. And so there's a lot of question marks still about how to monetize these things. And we figured out how to monetize it. And that is through real life human expert professionals, that the human expert needs to be in the loop. People understand that a doctor costs money, A lawyer costs money. An accountant costs money. It's a fraction when you do it through AI and online. But people are used to paying money for that, and that creates a business model here that is both supportive, that improves the quality, reduces the risk, and creates monetization opportunities.

27:49yeah um and before i go on to what's going to happen to these uh llm based search models or engines or services whatever um how do you so i i asked pearl about a bump on my nose that may be malignant. You know, it gives me an answer. I get a trust score, but I want to talk to an expert. This is all within my subscription, I imagine. But once I have that, excuse me, that expert on the phone, uh you know he answers the question i say yeah but you know i have ringing in my ears what should i do about that i mean you can imagine that the conversation with the expert goes on for an hour is uh suddenly this person has an expert on the line and can ask all the questions that they want to ask how do you prevent that we don't prevent that we love that we're trying to help people That's at the end of the day what our mission is, is to help people.

29:05And we're happy to help people. Not only do we help people like that in that kind of situation with that doctor, but we have all kinds of specialists. I mentioned we have 12 ,000 different experts. And so sometimes the person that's going to talk to you about the bump on your nose is a different person than the one that's going to talk to you about the ringing in your ears. And we're happy to connect you with the ear doctor to talk about the latter and the dermatologist or whatever it is to talk about the former or the ENT. And that's all within the monthly subscription. Yep. Yeah. Yeah. It's amazing.

29:38That is remarkable. Yeah. And so what's going to happen to AI search? Well, I think they're going to have to evolve. So, I mean, I think we all have to get used to the fact that this is about as good as the quality is going to get. There's going to be lots of optimizations, but the hallucinations are endemic. to the way these things work. And we're not going to get to 99 % accuracy like you would need for, you know, you'd need even more than that for healthcare and legal and things like that. You have to assume, I mean, our estimates in professional services are that they hallucinate about 37 % of the time.

30:18So you have to sort of recognize, it's going to hallucinate. And that means there's certain jobs that it can do and certain jobs that it can't do. And we just need to sort of get good at recognizing what kinds of jobs to use it for, what kind of jobs not to use it for. So that's on the quality side. I will say we should expect that the prices will continue to come down so that the competition and these LLMs are becoming commodities between all the open source models and DeepSeek and everybody else coming into the market. The cost to the users will continue to fall. Yeah. Yeah. What are there cases where you think AI is can be trusted more than than others?

31:06Yeah, where you're an expert. Right. So I'm an expert about the email I want to write to my friend. I'm an expert about, you know, what kind of logo I want for my business. I'm an expert on these kinds of things, anything that I can evaluate properly. I, you know, so if it gets it wrong, I don't like that letter. I don't like that email. I'm going to change this, this, this, this, and this. I don't like that logo. Try again, try again, try again, right? If I'm a good judge, if I'm an expert myself in that topic, and we're all experts at many things, right? Then it's fine because we can judge it.

31:44We can know how, where to use it, where not to use it. Well, the problem is when we're not an expert, when we use this for medical questions or legal questions or for veterinary questions and AI tells us something. And by the way, it tells us with 100 % confidence every time, oh, no, don't worry about that cancer. You're going to be fine. Just heat some rocks and it'll go away. Oh, okay. I mean, that's what these things are doing out there. But in the absence of that knowledge, I'm not an oncologist. I wouldn't know. And so I get this information and I trust it. And that's where you got to be careful.

32:18That's where the big mistakes happen. And that's where you need a human in the loop, a real life human expert. Are there cases where the human expert says, look, this issue is too complicated or really requires a face-to-face meeting with a professional? Of course. Of course. Yeah. And you can imagine all kinds of scenarios, right? a surgery is required on your pet. I can talk to you about what that surgery might look like, but I'm not doing a surgery online, right? I can't go represent you in court, you know, on Tuesday, for example. I mean, these kinds of things you need to go face to face. Yeah.