#264 Amr Awadallah: Vectara's Mission To Make AI Hallucination-Free & Enterprise Ready

0:00So Victara, what we're about is we allow our customers, which tend to be very large enterprises, to build AI agents and AI assistants that are accurate, so they don't hallucinate. When you're a large enterprise and you have 20 or 40 different teams all building their own DIY rag in a different way, you have a number of problems. In real time, we have a model that's called the Hughes Hallucination Evaluation Model that in real time, in a matter of milliseconds, reads the response that came back and correlates that response back with the needles in the haystack. This movement is all about that. How can I do that over and over and over again for any task that humans are doing?

0:42Build the future of multi-agent software with agency, A-G-N-T-C-Y. The agency is an open source collective building the internet of agents. It's a collaboration layer where AI agents can discover, connect, and work across frameworks. For developers, this means standardized agent discovery tools, seamless protocols for interagent communication, and modular components to compose and scale multi-agent workflows. Join Crew AI, Langchain, Llama Index, Browser Base, Cisco, and dozens more. The agency is dropping code, specs, and services. No strings attached. Build with other engineers who care about high-quality multi-agent software.

1:36Visit agency.org and add your support. That's A-G-N-T-C-Y dot O-R-G. Join other engineers who care about building the Internet of Agents. So go ahead, Amr, introduce yourself and tell us about Vectara. Sure. Hey, so I'm Amar Awadallah. That's my name. I'm from Egypt originally. I came to the US in 1995 and got my PhD from Stanford University in computer science and engineering. And then after that, I started a company actually during that, to be more honest, I started a company in 1999 that was acquired by Yahoo in the year 2000. That was a comparison shopping database engine. And we became a part of Yahoo shopping.

2:27And then I spent eight years at Yahoo, in which my career trajectory shifted more towards big data, business intelligence, and data science. And I left Yahoo in 2008 and started a company called Cloudera. Cloudera was one of the very first big data companies. We went public on the New York Stock Exchange in 2017, if I recall correctly. And then about four years ago, Cloudera got acquired by KKR, a private equity firm, for about$5.3 billion. And then after Cloudera, I joined Google Cloud for two years. I was vice president of developer relations at Google Cloud. And I was fortunate at Google and Google Cloud to experience CHLGBT two years before anybody else.

3:15Because inside Google, they had it already. They had something called MENA, M-E-N-A, short for meaning, essentially. And that said, they were very cautious about launching it because it hallucinated a lot. It made up facts all the time. So they didn't have the courage to launch it because they were afraid of the reputational risk. That said, it was super impressive. Like, just remember the first time when you tried JGBT, how impressed you were. And me trying it out, I right away saw, wow, this thing is going to change the world. However, it's not going to make any impact on the business world until we fix these problems with accuracy.

3:53So there needs to be a company that can make this technology enterprise ready. And that's exactly what Victara is about. So Victara, what we're about is we allow our customers, which tend to be very large enterprises, to build AI agents and AI assistants that are accurate. So they don't hallucinate. And when they do, you know when they are, so you can suppress or delete that result. And even correct it. We have some correction algorithms. So accuracy is our core, core. It's really our core differentiator, if you were to ask me. Second thing is security, making sure that these systems are resilient to what's called prompt attacks.

4:31Prompt attacks, you know, they are like when you try to psychologically engineer the LLM to reveal something to you that you should not be seeing. Sure. Like the same way somebody can trick your grandma to give them her pin code for her bank account, right? So you can do that with LLMs using front attacks, and we suppress that. So we prevent it from doing that. Security is very, very key. And then last but not least, for any regulated industry, finance, health, government, manufacturing, accounting, law, you need to have transparency. You need to have explainability for how did the AI come up with this result?

5:09How do I make sure this result is not biased? How do I make sure this result is not unethical? How do I make sure this result is not going to piss off my customers? Too negative in terms of the negativity content in it. How do I make sure this result is not a copyright infringement of somebody else's content? So these are all additional things we provide as part of this core platform. Now, there is another very key, I'll end with this last part. There is another very key value that we provide as well. And that is the notion of a standardized platform. So there is, and this is actually the talk I have at this conference on Wednesday.

5:46It's going to be partially about that. It's an issue called RAG Sprawl. So what is RAG first? Are you familiar with the concept of RAG? Yes. Or maybe for your listeners, just to remind them. We can go over it. It's an external database that's... Yeah. It's retrieval augmented generation. Right. And the idea very simply is when you generate a response, first retrieve the needles in the haystack necessary to give an accurate response. This is really what this is about. And we call that RAC. And there is lots of companies now doing what's called DIY RAC, do-it-yourself RAC, right? Which is okay. But the problem is when you have 20 or 40 different...

6:27when you're a large enterprise and you have 20 or 40 different teams all building their own DIY rag in a different way, you have a number of problems. First, it's hard to control the costs across all of them. Second, if one of the engineers that built this DIY thing left, this other engineer that built this other DIY, I have no idea how they did it. They cannot maintain it because it's not standard. And then third, the central IT team cannot govern them. It cannot make sure they're all living up to the same accuracy standards security standards explainability standards etc etc bias and so on so one of our key benefits as well is we usually say the aspirin the medicine to cure the rags pro is now we can have a central platform that your it team can govern properly that your developers can learn one and once and keep building on over and over again whether that be on premise or in the cloud.

7:22We actually support both modernities. So that's another very key aspect as well. Yeah. This battle against hallucinations has gone through several iterations. I mean, originally it was going to be solved through reinforcement learning with human feedback and guardrails. And then it moved on to RAG, Retrieval Augmented Generation. And then people seemed to be dissatisfied with RAG for various reasons, and they started fine-tuning models, supervised fine-tuning. And now there's reinforcement learning fine-tuning, and there's test time or inference time compute with something like the reasoning models.

8:18What are the drawbacks of RAG? And do you cover that whole gamut now in addition to RAG? Yes. So first, in my humble opinion, there is no drawbacks to RAG. You absolutely want to have rag if you want to have an accurate response coming out of your system you need to continuously ground the model in the truth in the facts yeah because by definition with all of these techniques supervised fine-tuning or or otherwise you will always have a probabilistic model at the end of the day and when you have something that's probabilistic that means there's always a probability of error and you can see that so if any of you yourself or any of your listeners go on Google and just search for hallucination leaderboard.

9:04The number one result you will get is a leaderboard that actually Victara maintains, which ranks all of the models out there by their what we call grounded hallucination rates. What is that? Let me explain. Grounded hallucination rates is when I already find the facts, I find the most relevant documents for your question, and I give the LLM, here are the facts, here is my question. What is the probability that the response is going to live up to the facts and there's not going to be anything extra in it. And you'll find out that even when you do that, even when you ground them in the knowledge, they will still inject things in the response that could be wrong.

9:43So the best in the world right now is actually Google and OpenAI. They're about 0.8 % and 0.9%, which is okay for consumer applications. And that's which models at Google and OpenAI? That's the O3 model from OpenAI and Google Gemini 2.0. Yeah. So they, and again, that's under grounding when you ground them. If you don't ground them and you ask them to give a response out of their parametric knowledge, meaning the knowledge stored in the large language model itself that was fine-tuned into it, the hallucination rates are way higher, way, way, way higher. They're like 10 % and above, right? And that's why you still always want to ground them in the truth so that it's kind of like what we do with humans, right?

10:22Before we go to our high school exam tomorrow, we refresh our memories, right? We refresh our memories so we give better answers. And you as a reporter, when you finish writing your article, if you want to make sure, extra sure that it's accurate, what do you do? Who do you call up? You call up a fact checker. Yeah, or I am the fact checker. Yeah, or you double read it or you double check it. Usually you want to get somebody else because if you hallucinated something and you read it, you're still going to think it's correct because you made it up yourself. So most organizations that have news organizations, when they have an article written by a reporter, they will have an independent fact checker that will read that article and make sure it does not deviate from the facts to ensure truthfulness.

11:03So that's exactly what we do in our system as well. So in real time, we have a model that's called the Hughes Hallucination Evaluation Model that in real time, in a matter of milliseconds, reads the response that came back and correlates that response back with the needles in the haystack that were retrieved from your ground truth and gives you a factual consistency score. 100 % means you're perfect. Everything was exactly like the facts said. And then if you go down to 0%, it means throw this away. This is horrible right now. right and we'd mark in the like with a highlighter just like a fact checker with which check your article for you they would mark oh this sentence looks off you might want to remove the sentence from the response to the human leveraging the response and this way you avoid all these embarrassing things that we see in the media like you must have heard about the lawyer that leveraged chat gpt and then it made up a number of completely made up citations of prior cases and that that gentleman get this barred because of that or for example, Air Canada, they were using customer support and then it offered the ticket for $1, right?

12:06Because it hallucinated that as, that's an offer I can give you right now. And that went all the way to the Supreme Court. And the Supreme Court, of course, sided with the plaintiff, meaning the customer and said, yeah, you have to honor that. It's not his mistake that your systems gave that ticket. So that's why you cannot take that chance. You need to have a real-time fact checker that is continuously checking the responses. And that's how we keep the accuracy under fault. So all the techniques that you mentioned, the supervised fine tuning, the reinforcement learning, the reasoning, the real-time reasoning with inference computing to think more about the responses, that's all in the LLM domain.

12:42That's in the LLM itself. But even when you do all these things with the LLM, you're still going to have a lacination rate of 1%, right? Like we see with O3 or like we see with Google Gemini 2.0. So that's why around that LLM, you want to have these two things. On the input to the LLM, you want to have a very, very, very, very, very, very, I'm stressing the importance of it, good retrieval system that finds the most relevant of all the documents and all the knowledge that you have. What's the most relevant needless in the haystack that are needed to give the right response? So that's number one.

13:16You need to have that. You give that to the model. You tell the model, now, okay, now think about the response. Don't use the knowledge in your head. Use the knowledge I'm giving you right now in the context window and answer this question or do this task for me And now you're gonna get a response, but that response might still have a chance of 1 % being wrong That's one of you. That's where you want to bring in the fact checker and tell the fact checker check this response against the needles and give us a score And that's exactly what humans do by the way like that's exactly a legal a lawyer before preparing a legal draft will study the prior art first, will prepare the legal draft, and then the paralegal will review the draft to make sure it's correct.

13:51So it's the same mechanisms. In other words, what I'm trying to say is rag will never go away. You will always need to have rag there, and you will always need to have explainability around it that sees when it goes off so you can fix the data. Because the next problem where this, sorry for digressing here, but the next area where problems can occur is bad data. And in fact, that's the biggest problem for all of us right now, by the way, is bad data. And one of the funniest examples I always like to share is when Google launched AI in Google Search last year. One of the most embarrassing examples they had was somebody was searching for, I'm trying to cook a pizza.

14:30And every time I cook it, the cheese is falling off the pizza. How can I make the cheese stay on the pizza while I'm cooking it? And then the AI from Google replied back and say, put some super glue between the cheese and the pizza. And we all thought at the beginning that that was a hallucination. that the AI just made that up. It wasn't. That was a data quality issue. What happened there is there was somebody who posted that question on Reddit a number of years ago, and a very evil human being replied back sarcastically and said, did you try putting some superglue in the middle? But lots of Reddit users found that answer so funny, they gave it thumbs up.

15:03So that answer got like 30 ,000 thumbs up. So the AI looked at that and said, that must be the right answer, right? So if you don't have the lineage of where did that response come from, that's what we we call the explainability, then you won't be able to go and fix that document that was wrong, fix that error so it doesn't happen again. So you need to have very good observability on the output as well, not just fact checking, because sometimes you do the fact checking and it will tell you 100%, but guess what? The fact was wrong. You had the wrong fact and a human would spot that. So now you need to go back and correct that fact.

15:32So that's why you need to have all of these subsystems integrated holistically to get the best quality of response. And this is on a platform that, so it's a SaaS offering. People go on and - It's on-premise as well. So you're right. It's a SaaS-like platform. So it's cloud native. In the cloud, it's fully SaaS. So in the cloud, you upload your data to us and it's completely SaaS, just like Snowflake as a database. But we also have an on-premise option as well that you can deploy on-premise. Because some of them, especially in the government and financial sector, they're very cautious about doing AI in the cloud.

16:06Yeah, sure. Yeah, it's like a double whammy for them. It's like, what if the AI figures our business for us and then opening AI now opens a bank? You see where this is going? So many of them, they wanted to have it run under their control, inside of their premises. Yeah. You know, I've spoken to a lot of companies doing things that sound very similar to what Vectera's doing. Yes. Is it a crowded space? I mean, I had Edo Liberty on a couple of years ago from Pinecone talking about vector databases and RAG. So we are not a vector database. I want to highlight that. So we are not. We use, inside of our system, we use two vector databases, open source ones.

16:48There is one from Facebook called FIAS, the FIAS library. And then the other one, oh, man, it's spacing out in their name right now. Starts with a W. Come back to me. Yeah. Yeah, it's not Pinecone, though, because we needed something open source that can also be deployed on premise. But we are not about the vector database. We're not about that. We are about the algorithms around the vector database that make sure you have the best embeddings inside of it, that you do the vector matching in the vector database, but you also do keyword matching at the same time. That's called hybrid search. And then once vector databases are very good at what's called retrieval, meaning finding the best needles in the haystack, but they're not good at ranking them.

17:26Exactly. I was going to ask that. All of the vectors around the given direction will be equidistant, so it doesn't know which one is more important. And that's where you need to have a re-ranking model. So we have a very specialized re-ranking model that we built ourselves that ranks these facts correctly. And then we have on top of that what's called chained re-rankers, where we allow the customers to define their own rankers. Because sometimes they might say, oh, I really trust information coming from VentureBeat more than I trust information coming from TechCrunch. or I really want to trust information coming from the last month over information from the last year or I want to trust information that had more than 20 thumbs up.

18:03So we don't know what kind of metadata things they want to do that on. So we allow them to plug in their own re-rankers as part of that pipeline. So sorry for the long spiel here. I want to highlight that we are not about the vector database. Yeah, but you answered my question because there's a lot of dealing, building a vector database and applying RAG to a vector database is an art. Yes, exactly. Retrieval, yes. So that's why, like you were saying, there's many other companies doing like what you do. Yes. But what differentiates us are these two things. Retrieval, amazing retrieval. Because the quality of the result you're going to get is going to directly be proportional to did you retrieve the most important knowledge to be able to answer this question?

18:51That is directly correlated and really, really good at that. And then the hallucination detection on the other side. Again, as I said, we are the benchmark in the industry for that. If you search on Google, like all the big models, when they come out, they call us up, please add us to your dashboard. And then our open source model is called the Hughes Hallucination Evaluation Model. It's open source, actually. It has more than 3.5 million downloads, way more than any of the other vendors out there. So that's kind of how, sorry for bragging a bit here, but that's how we differentiate ourselves, is by the accuracy of the results that we provide.

19:21Yeah. And then you do fine-tuning and you do test-time compute. You also do those things. And how much does that add to your accuracy? Yes, that can help a lot. So it depends. It depends on the use case. It depends on the use case. First, I remember the name of the vector database that came to me. Melvis. Okay. And so it doesn't start with W. It starts with M and it's Milvus. I was going to confuse it with VV8, which we also evaluated, but we lined up with Milvus. It's very, very scalable. It's very high performance. We're very happy with it. And it's open source, so anybody can grab it and use it.

19:57But going back to your question, again, we are not about the generative model. The generative model, it's pluggable for us. So rank is retrieval and then generation and then detection on the other end to detect whether you did something right or wrong. We are all about the retrieval and the detection. This is what we focused on. For the generation, we take the best models out there. So we actually plug into all of them. We plug in with OpenAI. We plug in with Google. We plug in with Anthropic. We plug in with Mistral. We have a version of DeepSeq that we're running ourselves. We have a version of QAN.

20:30QAN, in my opinion, is the best open source system out there. QAN is amazing. It's way better than DeepSeq. I think it's unfair how much press DeepSeq got compared to QAN. and we took actually a version of QAN and we fine-tuned it to your point, does fine-tuning help? We took a version of QAN, the QAN model and we fine-tuned it with a different cost function so usually when these models are being built for consumers, they are encouraged to be know-it-alls they are encouraged to always answer always make the user happy and answer, that's what they're optimized for and that can lead to the error rate being higher because the cost function is about giving an answer always.

21:10It's like when you go to a multiple choice exam and there is no penalty for getting wrong choices, right? You're going to answer everything and you're going to get some answers in the middle because they want to maximize your grade. So that's the cost function that most consumer models have built around. But for the enterprise, you want to penalize it for the wrong answers. So that's exactly Mockingbird. Mockingbird is a derivative model that we built. It's based on QAN. That fine-tuned QAN with a cost function based on DPO algorithm that says is every time you get an answer wrong, I'm going to slap you in your hands.

21:41So when you get an answer right, I'm going to give you a cookie. But when you get it wrong, I'm going to slap you in your hand. So that now makes the QAN model more aligned to only answer when it's confident in its response. And if it sees something where it thinks it's going to be guessing a lot, says, sorry, I'm not going to answer this. I'm afraid to be whacked on my hand. That's interesting. Make sense? Yeah. Build the future of multi-agent software with Agency. A-G-N-T-C-Y. The agency is an open source collective building the internet of agents. It's a collaboration layer where AI agents can discover, connect, and work across frameworks.

22:17For developers, this means standardized agent discovery tools, seamless protocols for interagent communication, and modular components to compose and scale multi-agent workflows. Join Crew AI, Langchain, Llama Index, Browserbase, Cisco, and dozens more. The agency is dropping code, specs, and services. No strings attached. Build with other engineers who care about high-quality multi-agent software. Visit agency.org and add your support. That's A-G-N-T-C-Y dot O-R-G. Join other engineers who care about building the Internet of agents. And that's a reinforcement learning fine-tuning. Yeah, using a technique called DPO.

23:15Yeah, and you do that before you hit the rag? Yes, that's during the training phase of the model. Right. So that's in compile time. I usually make this analogy. There is compile time and there is runtime, right? Model training is compile time. And model inference is runtime, right? And by the way, there's going to be way more inference than training in the long term. Today, we have lots of training going on because we're still at the beginning of the market and people are still building the models. But inference is going to be way bigger in the future. And that's why you see new companies now jumping into the inference space like Grok, not Elon Musk Grok, the other Grok.

23:53And Cerberus and Sambanova, you're right. Absolutely. So we do this technique of fine tuning and penalizing for giving the wrong responses. We do that during training time of the model. And once it's trained, we use it over and over again for all of our customers. Now, some customers might come to us and say, we built our own generative model and we fine tuned it on our own data set. which makes it more accurate. And that's correct. We'll take it. We'll plug it in. It's fine. Like, we'll leverage that. So we are pluggable on the generative model side. Yeah. I also spoke to...

24:31Take your time. I'm going to forget the name now. Give me a clue. Guy who spoke this morning. Glean? I've met too many people. Yeah, yeah. Take your time. Anyway, but they're... oh databricks i'm sorry oh yeah yeah and he was talking about how uh you know applying their technology to your uh data and and uh you know cleaning and rationalizing and improves the quality of responses yeah do you guys are they complementary to you they're actually they're partners they're actually investors in us i see so databricks is one of our investors and we are partnering with a number of other organizations as well on the data ingest side, like Confluent, Kafka, if you're familiar with the Kafka framework.

25:22We're working on other similar partnerships with other vendors. Airbyte is another one that we're partners with. And absolutely, as you bring data in, if you can clean it first before you load in, that's great. But in our case, even if you don't clean it, if you just give us good metadata around the document so that we have the metadata of who was the author, TechCrunch versus VentureBeat, what was the timestamp, which group within the company, how many thumbs up or thumbs down, any knowledge, supply chain metrics that you have around this document coming in, previous legal cases that you won, any metadata coming in can be leveraged to do that cleaning on the fly.

26:02Because what will happen then is the re-ranking algorithm will push the dirty data out, down, way down, and will keep the good data higher in the ranking. and the generative model is incentivized, give more attention to the stuff at the beginning of the prompt and not the stuff at the end of the prompt. So this way it provides a good response. It's a good hack to still give good responses without having to clean all of your data a priori. I'm not saying that to say that you should not clean your data. You absolutely should clean your data and in fact that's the biggest problem in the industry today, by far.

26:38In fact, I love the tagline from Informatica who I'm talking to them hopefully to be a partner as well. We're not partners yet, but hopefully they will be. I'm giving them free advertising right now. So Informatica tagline is lovely. I love it. It goes like this. Everybody is ready for AI except your data. And that's absolutely right. Yeah. But where do you think, so vector databases you think are here to stay, RAG is here to stay, oh no. No, vector databases. So it's funny. So actually, I'm an investor in a company called QDrent, which is a vector database company. I should disclose that. But I did that three, four years ago when the market was still at the beginning.