144 | Hallucination free business research - Better and fast decisions with AI with Bryan McCann

0:00Hello, everyone, and welcome to another live episode of the Leveraging AI podcast, the podcast that shares practical, ethical ways to leverage AI to improve efficiency, grow your business, and advance your career. This is Isar Maitis, your host. And we've got a very interesting topic for you today that's going to combine the present and the future and what a lot of people do today. So companies and organizations have many departments, right? And each department has multiple roles. And yet there's tasks that go across almost every role in almost every department. One of these roles is research.

0:35People need to research specific client or a specific prospect in order to know what to offer them and what's the best solutions for them. We research, we do market research, right? To understand where the market is going and where our niche is going to know how to plan better from a strategic perspective. We research prospects to know what kind of deals to offer them, what kind of services would serve them best. We research potential investments, either personal investments or investments in companies or new technology or infrastructure. Almost anything you do in business and a lot of stuff you do in your personal life requires us to do research.

1:10But research is not that easy to do because it has multiple steps. First of all, you need to find data sources. These data sources might be internal in your company, different databases that you have, your CRM and so on might be external, but you need to find those data sources. Then you need to find relevant, reliable data within those data sources. And because we have access to so much data, that's not easy to do either. And then you need to analyze the data to figure out what actually helps you in whatever it is that you're trying to do. And then you need to summarize the data. And then in many cases, that data comes in different shapes and sizes and forms.

1:45So it comes in different formats. And so it's not very easy to go through all of them to actually create a cohesive outcome that helps you make proper decisions. That's why most people are not very good at research because it's not easy to do. The good news is AI is actually really good at many of these steps. AI can help you in finding those resources and summarizing data and analyzing it in different formats and so on. But that's just step one. The reality is we're now walking into a future which is now becoming the present of agents within AI. And agents can do a lot of these steps for us. So it's not just we have to apply multiple AI tools in multiple steps of the process.

2:30It will figure it out on its own. Now, our guest today, Brian McCann, is potentially the perfect person on the planet to talk to us about agent and how to use them in research using AI. So his background is multiple years in AI research and AI for research, including spending several years in Stanford, as well as being the lead research scientist at Salesforce. So he has done this both on the academia side, as well as on the business side for more than a few years. And he's currently the CTO at you.com, which is a company that allows you to do research using AI agents. So he knows literally more than 99.9 % of the people on the planet on this topic.

3:16And so So since I think this is one of the biggest breakthroughs we're going to have with AI, and we already have access to it right now, I find this really exciting. And so I'm really excited to welcome Brian McCann to the show. Brian, welcome to Leveraging AI. In the next few years, AI technology will change our world dramatically. Whether you are a business executive trying to catapult your business forward, or just somebody who refuses to be left behind and want to advance your career, this is the show for you. I'm your host, Isar Maitis, a serial entrepreneur and an AI enthusiast. You'll hear invaluable practical tips from innovative business leaders, AI practitioners, and some of the brightest AI minds in our world today on how you can leverage AI in ethical ways to advance your career and grow your business.

4:12Thank you so much for having me. Yeah, I'm stoked to talk about some of the technology that's going on right now and maybe the future a little bit as well. Yeah, I think we're at the point that the future and the present lines become very blurry, where people are talking about, oh, we'll have AI agents and they'll be able to do all these things. And if there's a company like yourself, we've been doing it for a while. I don't see what's futuristic about that. And there's a lot of things like that with AI right now, where many people still haven't figured out the basic usage of AI. I teach courses.

4:44I have a course starting on Monday that I teach to business people. By the way, those of you who are with us right now, those of you who listened to the recording too late, those of you who are with us right now, we have the AI Business Transformation course that we've been teaching since April of last year. We have another cohort starting this Monday. So if you want to still join the course and you listen to us right now, you can join with promo code AI2025 to get$100 off. And this course teaches people how to use AI. But the reason I'm saying all of that is not to promote the course, but to say that there's a lot of people that still don't know how to prompt ChatGPT.

5:18And yet there are tools like yours out there that allows you to just send an AI to do a task and it will figure it out and it will do the whole process and we'll give you a beautiful summary. And so I think from an education perspective, what we're going to do today is going to be extremely valuable to people to understand what's possible today. And like you're saying, okay, when this gets perfect, what does that mean in the next two to three years? Yeah, we've seen a huge change in user behavior. VU.com started in August 2020. So that was when my co-founder and I were excited enough about the inflection point that was happening in the research world that we just had to get out in the wild and start experimenting with what new products would look like and new ways of search, but really more about just interacting with information broadly speaking.

6:16What we saw was that, or part of the hypothesis was that if AI could understand language better, you could do a lot more for them, for the user. So we always talked about moving from being a search engine to being more of a do engine, something that actually does something for you and fulfills that intent end to end all the way so that it's just done. There's various versions of this. There's all these assistants. I think a year or two ago when ChatGPD first came out, people talked a lot about chat as assistants and now the current word is agents. There are some differences between these things and we can dig into those.

7:00But even those lines blur a little bit because once it's cool to be an agent, a lot of assistants become agents and a lot of just software seems to be becoming agents. But really the exciting part about agents themselves is letting the AI decide in a very different way than we used to in the past. But over the course of the last four years, we've seen people's queries change, broadly speaking, go from basic Google-like keyword searches to really complicated prompts to entire workflows. And that's where we live now as we've transformed over the years as well with the technology, trying to get at the heart of a deep research use case where it's not so much about quick factoid answers that you might have gotten from Google before or some of the other chat products out there, even something like ChatGPT.

8:08We're not optimizing our system, which is dozens of models behind the scene, to give you the absolute fastest, shortest, most concise answer to your question. Instead, what we've been seeing with our users, and it dovetails really nicely with where the technology seems to be going, is this desire to have absolutely accurate responses and be slightly leaning towards comprehensiveness over conciseness. And in some cases, that's actually quite extreme. So extreme comprehensiveness, where we're not giving you the answer the way sometimes it seemed almost like Google was becoming the information god before all this started happening.

9:02You go to Google, Google has the answer. That's the answer. You accept it, you take it for what it is. Maybe it's one little line in a box. Instead now we're seeing an appetite for show me all of the information that you possibly can find on the internet or within my company's data or both. go to every tool you can possibly think of and bring it all back. Still synthesize it, obviously, and summarize it to some extent, but provide extreme transparency and attribution so that while we're in this phase of people gaining trust with this technology, you can click on all those citations, you can click on the charts, you can look and see where that data is coming from, Which I think is exciting because it opens up a lot of new things we can do, more complex technology we can deploy.

10:03But also it feels like maybe we're growing up as an information society. Oh yes, it is really important to be able to track attribution and transparency. And even that brings up a lot of questions about where that information is really coming from behind the scenes. So it's a good one for us, I think. I want to touch on two things you said that I think are critical, and then we're going to dive right into, okay, the how this is done and what your tool actually allows people to do and so on. One is really the question of agents versus assistants, right? And I'll say what I think the definition is, and I would love to hear your thoughts.

10:39But before that, I will say my two cents about the last sentence that you said. I think we live in a world where we're getting more and more extreme with opinions, whether it's in politics or thoughts about society or thoughts about many different things. And it's because social media, and with that I include YouTube as well, serves us what we want to see. So we consistently for the past X number of years have seen more and more of the same. Instead of seeing a plural version of the world, which is the real world, which will actually serve us better as a society. So I think moving to a direction where you're going to get a variety of opinions, a variety of sources, a variety of information, and not pick the one that is aligned with, forget about pick, be served the one that aligns with your thoughts is actually good for society.

11:30So that's my two cents. But now for agents, I think to me, the biggest difference is the following. One, it can do more complex tasks, meaning it know how to take a goal versus a very simple task, analyze it and break it down for itself, what the tasks needs to be to complete that, where an assistant doesn't know how to do that. You have to give it the step-by-step process. Two is that it knows how to monitor itself and correct, right? So as it's performing the task, no, this didn't work well. Let me try something else, which some of the assistants knows how to do to an extent, but not really.

12:05And three, which to me is the biggest difference, is it has access to different tools, meaning it doesn't live in the box it is right now. It can go to your CRM. It can go to the internet. It can go to different research sources. It can enter information into a form. it can do stuff that the assistants cannot do at this point. So to me, these are the biggest differences. If you have anything to add or to correct me, I'll gladly hear your thoughts about it. Yeah, no, I think that I pretty much agree with you there. The way that I often frame it for people is assistants are just this technology in some form that we build to help people, right?

12:46Like whether it was Siri, Cortana, all of these kind of like day-to-day tasks or work tasks, To me, that's closer to the user framing, whereas you might use agents to do some of that stuff. Maybe there's multiple agents behind the scenes that are powering assistants. So for me, although the word agent is a little bit more hype-y right now, I actually think the distinction becomes quite blurred and probably should not exist so much for an end user like on the consumer side. but technically speaking I think you're right on assistants tend to be very structured the way that they've been made in the past they're highly structured they don't have a lot of flexibility a lot of times it's mostly rule based or something like that and then there was a wave of trying to introduce LLMs into it and some deep learning but it still didn't look like the assistant was making the decision It looked like the code you were writing dictated what to do, maybe based on a couple models that were classifying which decisions to make.

13:57But in an agentic setup for agents, I feel like the key there is the agency, which you alluded to. It's that the LLM gets to decide. And it's not just a classifier that's set up to decide between a few options. It's this LLM gets to ingest context and decide what to do next in a far more open-ended way. Whether that's picking up new tools or spending more time thinking, whatever it is, the space of possibilities for that agent is much greater. And so it feels like it has more agency to it. Okay, awesome. So now that we had this amazing theoretical conversation, let's bring it down to earth and let's show people how to actually use these kind of tools for research.

14:54And I know you have different examples for us and different ideas on how to show it. So I will let you the stage and we'll allow you to show whatever you want to show. Cool. And by the way, those who are with us live and there's a bunch of people both on LinkedIn and on Zoom. first of all, if you're just listening to this recording, we do this every single Thursday. So you can join us live with people like Brian, who are really at the top of the game when it comes to AI and implementation of it. And so you can join us here either on LinkedIn or on Zoom every Thursday at noon Eastern. So I'll show a pretty standard example for us.

15:31And if you go to you.com, The kind of setup that I'm talking about is really this research setup. I recorded a little video for us here so that we can just zoom through it. We have smart, we have genius, you can play with different models if you want to. But the one that I'm really passionate about myself is this research mode. That's where I'm pouring a lot of our time and energy into seeing what we can do. In this example, we're just trying to see, should we invest in NVIDIA? and one of the things that we try to really emphasize is that users can upload their own data so you can transfer this over to enterprise use cases as well but you also have access to public web data.

16:17So this one's going to show how we're uploading in videos like Q2 quarterly proceedings and things like that. And so what the research agent is going to do is go through that file, go through that PDF. It's also going to generate a bunch of things that for this very abstract question, you probably should be asking or doing. This is part of the agentic workflow for the researcher, at least. Okay, we're going to extract these key financial metrics. What's happening behind the scenes there? We're going to parse the PDF. We're going to parse all the images, parse all the plots. We're going to pull those in, summarize them, try to find the key metrics, probably look online to determine even what those key metrics should be.

17:04So there's a whole suite of models going into just pulling information from these. We can dig into those two later. Then we're going to go do a bunch of research on recent stock performance, because you don't want this to be out of date. Every minute, if you're in a waking hour, these things can change. We're pulling in recent news, we're pulling in things from the competitors, et cetera, et cetera. All of these are not set in stone. These are just generated by an LLM, by the agent. Just a second, for those of you who are listening to this on the podcast, what we're basically seeing is we wrote a question, should we invest in NVIDIA, uploaded a file, clicked go, and it generated a list of 10 things it is going to do on its own based on the very short and general requests that we made.

17:53And it finds ways on the things it needs to do in order to provide us the best answer. And to harken back to what you were saying earlier, this is part of bridging that gap for a lot of users that themselves don't necessarily know how to get the most out of this technology yet. You should be able to, I'd argue you should be able to ask a very basic question like that and then have the agent take care of the rest for you. So moving forward, these workflows, as you can see, are where you see what's happening with the agents. So there's a thinking step, it's researching now and actually going out and getting all of these sources live.

18:35So we're going out to the web and crawling whatever we need. You can see a little list of the sources. It looks like we're crawling about 140 in this case. so we've gone out, picked up those pages, mixed it with the PDF financial earnings like we uploaded ourselves as a file. And then instead of getting a quick answer box or something like that, you're going to get something optimized again for comprehensiveness. And you can see the citations here. These are the things that are specifically pointing back to the file you uploaded because they're all pointing to one, the first source. All of these are clickable so you could actually go into the source and see where we're pulling that information in.

19:21And then as you go further down into growth prospects for AI market leadership, data center growth, now we're pulling in information from source 2, 3, 4, 5, 6. These are coming from the sources that we've gone out and collected from the public web. And you can see that the response keeps going for a while. and it took a few seconds to actually go do that research and crawl all those pages. But the type of answer you're getting is much, much closer to a research report than it is a quick Google factoid-like answer. And it's got that attribution and transparency baked in so that users can build trust with it over time, which I think is really important as you give these agents more control and more deciding power over what's happening in the future.

20:18When it's making use of your credit cards and your bank accounts and all sorts of other things, you're going to want to know and have built up this sense of trust with the agent. So that's one that uses the public web and some files that you upload. And you can upload a lot of files. and if you're in the enterprise setting, you can upload many, many, many, many files. We've had people uploading 20 million PDFs and 20 million CSVs that go with those PDFs on clinical trial data. Then you'll see the agents having additional tools at their disposal, text-to-SQL tools that perhaps we've created or customers have created.

21:00That gets back into that tool usage that you were alluding to as well. I got a bunch of questions. Great. Because this was great. Got a couple more demos too. If you're in any business that gives you an idea of what the things that you can do, because if you give it your own data, but on its own, it knows how to go to search for relevant data somewhere external and it knows how to combine all of it together into a very detailed report that is structured properly, that is easy to follow, that also has links to the sources. That's obviously very powerful. So here I have three immediate questions and then you can answer them in whatever order you want.

21:34The first one is you alluded to the fact that it's accurate and grounded information. And that's one of the biggest problems that people have with these tools, right? The more it feels like the more open the question is, the more they hallucinate and make stuff up because you give them the freedom to do that. And so I'm wondering in a tool like this that actually starts with research, starts with actual factual information, how much do you see hallucination reduced compared to when you're just using chat GPT? That's question number one. Let's start with that. And then I'll ask you the follow-up questions.

22:08Yeah, we have essentially the lowest hallucination rates out there. There are actually a couple independent research papers, academic research papers written that measured us against ChatGPT, perplexity, maybe BingChat, I think as well. And I can find that and post a link here. but yeah it basically shows that we have the most kind of accurate and faithful answers I can talk a little bit about how we're doing that yeah I'll look it up while we're talking about it I just don't want to misquote them but the way that we're doing that is we're tapping into this trade-off we're taking a little bit more time this is a paid version of the service so there's more complex models behind the scenes that are only really enabled once there's some money going into it.

23:05But then it's not just a single rag setup. It's not, okay, we're going to get a bunch of sources, drop it into a context window, and have the LLM summarize things. We're identifying across a huge range of intents for the user. what category of question this is falling into. And that basic intent is used for many decisions down the line. And that intent and understanding system is the system, essentially, that we've been building. Even before chat, going back to 2020, that was the first system that has now seen all this data of user behavior change. and I saw u.com be very popular with developers and students and researchers broadly and financial.

24:00So it's seen a lot of this data on, okay, do I need to go to the public web? Do I need to personalize this answer for this user? Is this a developer type query? Is this a biotech kind of query? And with that information, then we dynamically construct prompts. So prompts for us are not really something that's set in stone anymore. It's not something that even the user is really writing. We're trying to get it so they can ask basic questions and not worry about prompting. The system generates prompts based on all this information and constructs them. And then once we get sources, candidate sources, because we take a little bit more time here, we actually go crawl those pages live so it's up to date.

24:46A lot of search engines or chat services, they're going to use a more traditional search index to pull in information from whenever the last time they crawled was. They're not going to crawl it live and get the most updated information. So that's another first step towards making sure there's freshness of the information itself. Once you have it and you have those different sources, part of what is making research mode hallucinate less is we're not tuning this model for making a decision and making choices between different services or between different sources. It's okay from a comprehensiveness standpoint for it to say, this source says this and that disagrees with this other source.

25:34So that even itself allows it to not necessarily say this is right, this is wrong or make something else up entirely. It's just really tuned, say this is what this says, and maybe these four or five sources agree. So it's doing that work for you, but it's not oversimplifying to some extent. And for any particular fact, we have models that run to determine whether the LLM's response is actually entailed by that fact, and vice versa. does what the LLM say entail that. You try to make sure that there's not any extra information in either direction because that's hallucination. So we're minimizing those as well.

26:26And when you do that across, as you saw, 180, 200 sources, you start to narrow in on these are the things that 90 % of those sources are agreeing on. Maybe there are some outside perspectives, but if you've tuned the model and you prune any part of the response that deviates from being grounded and faithful in that way, you get this kind of research report. Awesome. Great answer. I think it's important for people to understand what's happening behind the scenes in order to get those results. The other question that I have, and actually was asked by as somebody on the chat as well, is sources. And I have two variations of that question.

27:12One, on the external world, can I point it to specific sources? Either on a setup, I only want to look at this, or as I'm writing the request in real time and say, I just want to look at these sources. But also on the internal perspective, can I say I'll give it access through an API to my CRM or my ERP or my customer service ticketing system or whatever other enterprise system that I'm using, can I connect it to it so it's always available to it so I can query that as well? Or do I have to upload documents and do stuff like that? Yeah, great questions. And then I found that research paper. I'll show you that too.

27:51So for internal documents, you can upload yourself through this interface the way I just showed in that demo or in some of our enterprise settings. and this will probably come further and further out into the more out-of-the-box solutions as well. You can just essentially have a bucket or a blob or some sort of cloud storage bucket you're constantly uploading data to and we have all the indexing and permissioning and everything outside of that. So you can have those cron jobs running on your side that are constantly pushing information in. where there's a limit right now is just on the kinds of connectors we have already pre-built.

28:34If you want to connect your Gmail or something like that, that one's coming soon. But if there's some obscure CRM service, that's probably not in our system yet. Yeah, but I'm saying that you can create your own export that every day runs a query, finds the differences, pours it into this data lake or repository or something, and then you know how to crawl that and find information there. So there's still a workaround even if you haven't built a connector yourself. And there are custom kind of integrations or connectors we've built in partnership with companies where they have specific tools that say their biotech researchers use.

29:16And then we point this agent at that as another tool it can use. And we've seen really great results with things like, as I was mentioning before, For text to SQL, if there's a large database, like a SQL database, or if they have a bunch of spreadsheets or something like that, very effective and improves accuracy a lot. And some have their own sorts of dashboarding, plotting, charting kind of technology. And being able to just read from those charts and plots can also make a pretty big difference over trying to do some lossy transformation from that into another format into yet another format before you pass it into an LLM.

29:58And I'll show you real quick this paper that was perhaps surprisingly pointing out that SearchGPT was the least accurate system when using citation. And they go through and analyze all these different situations where there's citation accuracy Sometimes there's unsighted sources. Sometimes there's one-sided answers, things like this. There's you being perplexity in search GPT. And you can see the most green and the least red is over here on the u.com side on all of these different metrics, whether it's citations, actually being able to trust the citations, the kind of sources we bring in, which is really important, right?