The perplexities of information retrieval

0:05Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at Fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents. So you can launch your app near your users. Learn more at Fly.io.

0:45What's up, friends? Do you remember when ChatGPT launched? I do. It felt like the LLM was this magical tool out of the box. However, the more you use it, the more you realize that's just not the case. The technology is brilliant. Don't get me wrong, but it's prone to issues like hallucination on its own. But there's hope. There is still hope. Feed the LLM reliable current data, ground it in the right data and context, then and only then can it make the right connections and give the right answers. The team at Neo4j has been exploring how to get results by pairing LLMs with knowledge graphs and vector search.

1:22Check out their podcast episode about LLMs and knowledge graphs throughout 2023 at graphstuff.fm. They share tips on retrieval methods, prompt engineering, and so much more. Don't miss it. Find a link in our show notes. Yes, check it out. Graphstuff.fm, episode 23.

1:50Welcome to another episode of Practical AI. This is Daniel Whitenack. I am founder and CEO at Prediction Guard, where we're safeguarding private AI models. And I'm joined, as always, by my co-host, Chris Benson, who is a principal AI research engineer at Lockheed Martin. How are you doing, Chris? Doing great today, Daniel. How's it going? It's going great. I am sometimes perplexed throughout my workday, but generally the week has gone well, and I'm really excited because hopefully our guest today will provide a lot of ways for us to navigate through the perplexity. I hope he finds that punny. Yes, yes, of course.

2:33We have with us today, Dennis Yarets, who is the CTO and co-founder at Perplexity. Welcome. Thanks for inviting me and great to be on the show. Yeah, yeah. Well, we've been wanting to make this one happen for a good long while. Of course, been following the things that Perplexity has been doing. really impressive and inspiring work. And yeah, just really excited to hear a little bit about that story and also this kind of space of answering, giving people knowledge with generative AI, with language models, search more generally, maybe. Could you set us up by maybe talking through, of course, people have followed, everyone's kind of followed this surge of generative AI over the last couple of years.

3:23But there's been kind of a segment of that that has focused a lot on kind of answering questions, discovering knowledge, curiosity, exploring topics, an intersection with search more generally, I guess. So could you kind of help us understand maybe that journey, how that's kind of developed and how also you and the co-founders of Perplexity kind of came upon your approach to that? Yeah, that's definitely been like very fascinating, almost two years, I guess. I think like general like web search, right? So like what do you want to ultimately get is an answer to your question, right? So it's the current iteration that we had like when it was Google and all the other like classical search engines is an approximation to get to this point, right?

4:12So you have your question, you ask it as a form of query, and then you get like a bunch of documents that are very relevant, but you have to still do ultimate sort of like, you have to do like additional work to get to the bottom of it, right? So you have to scan through the documents, you have to for yourself understand what is this a true answer. If it's a not true answer, kind of like have to trust the information. And as you can imagine, it's a lot of work, especially if you're trying to search for something very complicated, maybe things that are not so obvious and would it be nice to um kind of like avoid that step right where you just like answer a question and you ask ask the question and you get an answer right away so that that's kind of like ultimate destination we're trying to get obviously uh there has been um it's been tough to get there over uh last like decade or so it's you know there's been like a lot of work but it never quite worked there is a this like a much higher level of uh hallucinations much higher level of uh maybe not perfect synthesis of the information you kind of like basically get like a frankenstein so like uh instead of like a hearing uh and a nice easily parsable of readable answer you get like some just like basically extracted pieces of the information and just like concatenated together so like not very pleasant and you know um it's funny that when we started so one of our angel investor was uh jeff dean and requires like no introduction and he he was saying you know um google uh actually wanted to always build something like this but it's just because they have like such high um expectations for accuracy because like you know millions of billions of users using google right and like if you like hallucinate like one percent of the time you're gonna get a lot of uh unhappy people and so they would never able to because of the models were like not as strong as they are right now, they were never able to get it to just like 99.9 % of accuracy.

6:16Right. And that's why kind of like this work never pan out. And, but something great happened, you know, 2022. Right. So we, when we started our company, we kind of, you know, both myself and Aravind, my Mike of Cofondren, we come from like academics, right. We've been doing a lot of research on language modeling, reinforcement learning, stuff like that. And he was actually at OpenAI at that time. We've been very clearly following improvements of GPT models, like GPT-2 and then GPT-3. That's where it actually got very interesting. And it became obvious and obvious there is going to be something there.

6:56This was primarily my motivation for us to start a company. we wanted to do build an answer engine from the get-go but it was kind of like very ambitious you like we i remember we would go to the investors and there was a say like oh we're going to build a search engine and they're like start like you know looking at you like you're crazy which is makes a lot of sense they're so like oh there's google already and they had like a fair point but we we still kind of like weren't like very discouraged by that we knew there is like something there and we started like prototyping so we the first version of perplexity we actually created as a slug bot or like discord bot where kind of like it was a very primitive combination of a search engine plus uh at that point it was like da vinci 2 models it's still you know pre-chat gpt and it's kind of like worked much better than i expected it was like very quick like we put it up this demo like in a couple of days and it was like you can already see that it in certain cases it is very helpful like we because this was like very early company we're trying to hire one of our first engineers and we didn't know how to organize the insurance for him.

8:06And so we actually used this bot to ask those questions about insurance because if you go to Google and start asking the questions about insurance, you're going to get a lot of ads and you're just going to get very quickly disappointed. I've appreciated that same thing in founding a company and using these models in that way. It's very useful. Yeah, exactly. And then it was kind of like useful. So we were kind of like been trying it out and like playing with it. But then what really happened, started happening is like, I think it was like a couple of weeks before ChatGPT, they opened AI, released this DaVinci 3 model.

8:44And I literally remember very clearly, like changed the, like literally the model name and started asking the same questions that I was asking before. And I could see right away, just like so much better. it's just like so much better at understanding what you of your intent so much better at like understanding where like what should it say what it should not say so much better like synthesizing the answer and i was just like really like blown away and you know um i was like okay so there is like definitely something there and then obviously like shy gpt happens like a few weeks after and we're like okay so we have to like our initial product was actually because as i mentioned we came from academic background like our citations were like the core component because it was like very clear to us from the beginning.

9:28Like if you want to get an answer, you want to make sure it's accurate. You want to make sure you can verify it. And so the citation was like the sort of like first class citizen in our product. And then when Chagipiti came out, it was like, okay, so one of the biggest point of feedback for them was, okay, so I don't know like if this is accurate information, if it's not, if it's a hallucination, if it's not, how would I verify it? And that's why we like recently decided, okay, so this seems like a good opportunity to release our product. We like literally in a matter of like today's put up a website uh connected to our like back end that we had and just like obviously did not expect that it's gonna uh be um people gonna use it and like the usage is gonna grow as much as possible but coming back to your uh original question i think like what happened it was just like literally this in a matter of like days or month i mean obviously it follows like a lot of uh years many years of research but like it was very clear step function in the quality of the generated answers.

10:23And you can like literally, if you, if you sort of like spend some time playing was if you can clearly see that it's now becomes very, very good. We also realized at that time, okay, so like models only going to get better. Things only going to get like faster and cheaper. So there is something, there is a lot of stuff to build here. For those who are still kind of learning about your organization and what you're offering, could you step back for a second? Like if you were talking to Jeff Dean or another investor, kind of giving them the elevator pitch about what you're doing specifically and how it's differentiated from the GPTs and stuff out there.

10:59How do you define that? How do you describe yourself in terms of the specific opportunity that you're pursuing? The fastest way to get the most accurate answers to your questions. I think that that's essentially like answer engine. Then we're kind of like one of the first people who coined this term. And can you differentiate that from a search engine a little bit there? Yeah, yeah. So basically in a search engine, right? So you get like all the way. It's like, as I mentioned before, in the search engine, you have to search first, you get a document. Let's say you have like top 10 results and you have to scan through all of them and identify the information to get your answer.

11:34Here, we kind of like, we do this step for you. So we take the first step of the retrieval of the relevant documents, and then we synthesize them into a human readable, nicely formatted answer that you can then like if needed if you want to get more information then you can click on the citations that are kind of like nicely attributed for each sentence and then you can like learn more information and so we kind of like wanted to do things like very simple we were like early on identified so there is like two things we care about this is a accuracy and then speed because this is you know you want to get information fast google trained us all to get instant search results.

12:14I think that was very important. Early on, it was kind of challenging because those models were very slow. Our infrastructure was not very advanced to do that. I remember very first version, you would have to wait for three seconds or five seconds to get an answer. It was very slow, but because it was such a better experience than just looking at the search results at Google, so people still would use it. Our ultimate goal was like, okay so can we be as fast as possible so yeah so the the main differentiator is just uh we care a lot about quality so we minimize the chance of things being inaccurate or like hallucinate and we want to do it as fast as possible and so that's that's kind of like distinguish us from google because you know google doesn't for example generate the answers uh even though like more recently they started doing this which is kind of like validated our idea and chat gpt probably primarily focuses on like different things, you know, but I guess also more recently they started doing web search as well.

13:29What's up friends? I love Backblaze. I'm happy to have them as a sponsor. Backblaze makes backing up and accessing your data astonishingly easy. This is a service I personally use. Go to backblaze.com slash practical AI. You get unlimited cloud backups for Macs, PCs, businesses for just$99 a year. You can easily protect business data through a centrally managed admin, protect all the data on your machines automatically, easily deploy across multiple workstations with various deployment options. You can add on enterprise control, including granular access permissions, advanced single sign-on, group management controls, and compliance support.

14:09They even offer multiple restore options, including rapid recovery in the event of data loss or ransomware. That sucks. You can access your backed up data from anywhere in the world using their web app or their iOS or Android app. You can even restore by mail. They'll give you a hard drive with all your data shipped to your door. You buy a hard drive restore, send the hard drive back within 30 days, and get a full refund. Get one year file retention and version history. Over 55 billion with a B files restored for customers so far. Visit MacBlaze.com slash Practically I so they know where you came from and continue to support the show.

14:47This is a service obviously recommended by me, but also by New York Times, Inc. Magazine, Macworld, PCworld, LifeWire, Wired, Tom's Guide, 9to5Mac, and just so many more. You receive a fully featured or no risk trial at backblaze.com slash practical AI. Again, there's support in the show. Go there, play with it. Start protecting yourself from potential bad times. Start today.

15:38So you mentioned a few things there. You mentioned web search. You mentioned retrieval. You mentioned the large language model. So at least in kind of how I think about it and maybe others categorize it differently, there's one element of information that you can get from an LLM, which is I'm going to put in a prompt and it's going to generate text. and that may contain some facts or made up facts or some text, but it may be informational, right? So there's some sort of knowledge that can be gained there. And then in a second case, there's a way to retrieve on the fly external data. So that could be like from your company's documents, it could be from the web, whatever, and then inject that into prompts into the model, which kind of grounds it and like you say would give you a citation there's also you know more agentic approaches to this where maybe there's multiple ways that you could get knowledge maybe doing a web search or searching a certain database that you have access to or any handful of of sources that you've kind of curated as tools and you call them in a more automated way so i'm wondering from your perspective, obviously you are part of a team that have been exploring this very deeply from very early on, like you say, when these models made that jump.

17:03From your perspective now, both in terms of what you're building with perplexity and also kind of how you generally see the ability to get accurate information from these models, how do you view those kind of categories in terms of their utility and what you're relying on for the accuracy element specifically. The way I see it's going to unfold, I think the tools and sort of like agentic behaviors, I think that's where it's going. I think it's going to be the main bottleneck for this right now is just like models are not smart enough yet to take into account and sort of like reason all of the information that is out there.

17:42but uh i think it's going to be like a main component so there's going to be like models they're like very powerful already right now they're like trained on a lot of data like basically internet they have a lot of internal information internal knowledge and they can do like already like very good job of uh synthesizing information there's like certain things that they don't do well and perhaps they never going to do well those things like for example like you know like computation like when you need to do like some maybe around like some code or like do like uh some sophisticated like math computations that's like did the lm architectural i guess transformers is like gonna struggle at that also you know because those models are like so big sometimes it's going to be very expensive to them update very frequently so you need a way to ingest um something some new information that just like happened and it's still not part of the llm weights you have a way to and this is what we sort of like specialized also some private documents as you mentioned right so sometimes if it's like uh enterprise you know you have a some of the documents that obviously the model is was not trained on and you kind of like maybe want to reason about those documents right so and there's like all kinds of other tools that you can yeah eventually there is going to be like agents that is going to do like actions maybe you're going to like book a ticket, like buy a ticket or something like that and stuff like that.

19:08So I think it definitely where it's going, it's going to be like a synergy. Everything's going to come together. We just need a top level, powerful model that's going to kind of reason behind multiple things. And we'll have to need to have like long context windows, maybe like some memory as well. And then, you know, just like utilize those tools as much as possible. How we are perplexed to thinking about it. We are like, okay, so, and there's like multiple ways and multiple sort of like applications, multiple use cases of this general approach. We are primarily focusing on the information retrieval part of it.

19:45So kind of like initially web, web is the main component because there's like lots of things to instruct from web, but also we're thinking about how to integrate a new different data sources. maybe like some of the different databases maybe there's like some more complicated or like more specialized documents like you know maybe there's like pdfs or something like that or like some financial data things like that the other aspect that i'm very excited about and and we're working on is kind of uh do you know right now we can answer we can do like a great job of answering like complicated questions that you can get answers on google but like still not something where you can ask expert and then can get an answer right like what if i have like a question that is just like requires like multi-step of reasoning so it requires like okay like searching web multiple times analyzing information that during the issue retrieval and then like maybe refining the search so kind of like those things that may be going to take some time but like if you do it yourself you would spend like let's say like an hour so here maybe the system would spend like a 30 seconds and it's going to save you a bunch of time so kind of like those use cases that's um that can answer very complicated questions.

20:58And we believe that there is a world for those type of questions. Technology like this is going to be useful. As people are using an answer engine like yours more and more often going forward, and you kind of alluded a moment ago to the fact that LLMs are not the be-all. There are things they don't do well, like mathematics and such, and a variety of other things I'm sure that we could all throw out there. But they're really powerful at what they do, but clearly there is a place and a need for both the LLMs, these largest models that kind of suck up all the air in the news cycles, as well as many smaller models that are specialized, a mathematics model that you plug in.

21:39As we're looking at trying to use answer engines to retrieve information, and that information is increasingly multimodal in nature in terms of what you're asking, how does the architectures of those come together. This is a space, it's not the first time we've asked it here, but it's evolving so rapidly. We're no longer hosting a model. You're now hosting a whole collection, and they may be mixed with models that you're APIing out to and such. How does that look to you as you're building this company at this point? It's going to be always like trade-off between like if you have like one powerful model, I mean, yes, you can do like lots of general things like super well but like it's going to be slower it's going to be more expensive one of our like key principle is just like we want to do things very fast so we can get like that answers as vast as possible that means you have to design your system like orchestration system in a way where certain things will have to uh rely on like customized models like and something that is much smaller much faster but it knows it's like a specialist model so it's not a model that like knows how to do everything but it knows how to do like one task and basically the the challenge here is like how do you balance between this like general models and this like a specialist models and i think we've we've been doing um this like from the very beginning so like when you send the request to perplexity it's just like not one model there is like i don't know at least like 10 different models trying to um do lots of things with your request it's like all kinds of like ranking models, a bunch of like embeddings, all different like classifiers and stuff like that.

23:21And like the other trade-off here is just like, it was general model. Let's see, um, one of the, uh, big, and I think it was like actually very critical component of, uh, why like company like perplexity in the first place became, uh, possible is it's like the speed of iteration. Like you literally can change the prompt and you can just get a new product, like in a, in a matter of like hours like imagine a couple of years ago if you wanted to build something like you know perplexed or like whatever other like genii product you have to collect data first have to train the model launch the product see if this product makes sense does it have market feed or not if it has market feed then you like start collecting data and then and then you just like keep improving so what being possible with gpt models like an api is just like this kind of like flipped over so you very quickly can build a product, see if there is any signs of life for this product, and then you start collecting data, which is, I think, honestly, the most important thing.

24:19And once you collect data, you can distill it, you can build many other smaller models and optimize the experience. So you can make the models faster, you can specialize them. I think this is the key. I think this is the, honestly, was one of the most fundamental changes in the development. And we kind of like took advantage of this thing like early on and then still using. But it's still tricky kind of like imagine like every time you have this like specialized models and you have like if you have tons of them, you have to then like treat each model like care about each model separately. So if you want to like retrain this model, so you have to spend some time on it.

24:58So you have to like evaluate it. So it becomes more difficult to manage. but on the other hand you know you have some great benefits so the key is just like you don't want to go like too overboard with those models like if everything is on like customized models but also you also clearly don't want to have just like one model that's going to do everything so yeah i i have a question maybe related to that which i think is a pain point a lot of people are feeling and i'm i'm guessing your teams have felt which you even mentioned this like you you can make a small change in your prompt or create a new prompt and all of a sudden it's almost like you have a new product which is sort of like amazing in in one sense and really frustrating in another sense because as you were just alluding to it's like oh maybe I have these like 17 different things chained together and they all have prompts that I've worked really hard on and then like tomorrow, Lama 17 comes out or something.

25:58And now it has a different character of behavior than the previous model that I was using. I'd love to use it, but now I have all of this. It's almost like AI model debt that I've got in my system. Do you have any perspective on that or anything that has happened in your experience in this regard? Yeah, yeah. This has clearly been a thing and it's been happening quite often. And if I would guess that's going to continue happening. One thing we realized early on is, okay, so this is going to be the case. There is not going to be one model that rules them all. I mean, even though like for some time it was like GPT-4, but like now we can see there's like particular like Anthropic, you know, Gemini, like Llama.

26:46There is like going to be a future where there's several frontier models, right? Because of that, we decided, okay, so like let's design our infrastructure and our system in such a way that it's going to be model agnostic, right? So, and then that means, okay, so there's like a ways where you can evaluate each component independently. there is a way where you can quickly change things up to adapt for like a new model and stuff like that and that's uh it took some time to get there but it's i feel like it was like very correct decision for us and then so for example one of the advantage we have over let's say like um things like chat gpt or like load or like basically one model providers companies is just like we can seamlessly integrate many different models and like our users can like okay decide this is the they want to use this model or they wouldn't do that more.

27:39Like later on, as we progress, I think we can even like decide based on the complexity of the query or like the type of the query we can like route to like a particular model that does the better job for those type of queries and like, you know, minimize like maybe like some of the queries are super simple. You don't need to run like a very large model for those like to answer. So then you can like can optimize speed and things like that. So I feel like you have to just make a system in a way where it's like agnostic to the model.

28:18What's up, friends? Have you ever had trouble accessing that favorite sporting event or that awesome show or that film even? Because it's not in your region. Well, our friends at NordVPN can help you switch to a virtual location, to a country where it is available, unlocking a world of entertainment. Plus, it's not just about streaming. It's your go-to for online security. Protect your bank details, your passwords, and your entire online identity. If you're traveling, they can shield your data on public Wi-Fi, keeping you safe no matter where you're at. They also have this cool feature called threat protection.

28:55That means you can take a bite of viruses, malware, and phishing sites because NordVPN will protect you no matter where you're at. They're one of the fastest VPNs globally. They ensure no buffering while streaming and no stops for your ISP from bandwidth rattling. It might sound costly, but think again. NordVPN costs less than a cup of coffee a month, and you can use one account on up to six devices. To get the best discount off your NordVPN plan, go to nordvpn.com slash practicalai. Our link will also give you four extra months on the two-year plan. There's no risk with our 30-day money-back guarantee.

29:38Once again, go to nordvpn.com slash practicalai.

29:58So to follow up on what we were talking about before the break there, I know that you were talking about really building around model agnosticism to be able to handle that. I couldn't help but wonder, occasionally as we get a new model out, it breaks new ground on modality, being added in, a whole new approach, that kind of thing. And so how do you as a business builder who is having to try to accommodate all these different models, when you have one that jumps out and has a completely new thing added in that was unexpected prior to the announcement, how do you in the organization, how do you guys kind of pivot to accommodate that and keep the agnosticism and yet provide that extra functionality that's now available?

30:46How do y 'all tackle that problem? The most important thing to be is to not be caught off guard and try to anticipate what's going to happen. I think that's very important. And it's kind of like, for the most part, I wouldn't say it was too hard to predict what's going to happen. But after that, I think it's primarily like product decision, right? So does this new feature benefit your product or not? Do we want to build something in product or not? So for example, one great example was image upload and kind of like multimodality, right? So we knew, it was like last year, we like knew for sure that this is going to happen at some point.

31:22Because I mean, there was like already like some smaller models that kind of like supporting that. We knew that, you know, much better models going to come out. And then because of that, we kind of like in advance start, first of all, like understood like, okay, so this is going to be important for our project. So we can support like this and this and that, those use cases. and we decided to build infrastructure in advance and kind of like anticipate it. Obviously, we didn't fully like predict like how exactly it's going to operate, but it was like very close. So it was like required like little, like couple of days to adjust.

31:56And then we were like one of the, you know, very quickly can release it as a product. So it's having the system that are kind of general enough that can like support those like new modalities. It's very important. get some great deal of uh anticipation but also you know like sometimes you know like certain features that maybe come out maybe they're like not useful for your product you don't you also don't want to like put everything into okay so like this is a cool feature let me just add it to your product so like that that's always not not that great idea in general so only things that make sense and if if those things makes sense for your product you likely already like thought about like how do you how would you implement them in advance so it's just like makes it a bit easier While you were talking, I was thinking there's sort of one axis that you have to navigate here around model releases and functionality and modalities, all of that stuff.

32:47There's sort of another maybe around UI and user experience. I sort of multiple people making the comment, oh, well, like the chat interface came out with chat GPT. So everyone's sort of like focused around the chat interface. Is that the best way to utilize this sort of technology in the long run? There's probably a lot of exploration that's still open around UI and user experience with this type of technology. And certainly chat is is relevant and you know we're using it a lot already i'm wondering from your perspective especially as we see this functionality maybe more embedded in the physical world around that whether that be like in our glasses with with meta glasses or in like kiosks in airports or whatever whatever those things are what is your perspective on how important it is to explore new types of UI or user experience with this technology?