Johnny Ho: The Future of Online Search (Encore)

0:05Hey, everyone. Welcome to Generative Now. I am Michael McDaniel. I am a partner at Lightspeed. And this week, we're revisiting a conversation with the great Johnny Ho. He's the co-founder and chief strategy officer at Perplexity. We talked about his background as a competitive programmer ranking number one in the world, the origins of Perplexity, and how they're navigating an exponential growth spurt in not only queries, but the company itself. Take a listen to this conversation with Johnny Ho. Thank you so much for doing this. Yeah, of course. It's great to be here. Your background, if I understand correctly, is actually in engineering, right?

0:40And research. You were an IOI medalist, if I understand that correctly. The reason I know that is we're also investors in another company called Pika. Demi, the CEO, she's an IOI medalist as well. I saw some tweet online that like, basically, there's this concentration of IOI medalists and AI right now. And if basically, if you're an investor like me and you come across an IOI medalist, you should just invest in that company based on your track record and Demi's. Tell us a little bit about what it means to be an IOI medalist and a little bit about your background. For sure. For sure. Yeah, I think the IOI medalist population is standing on AI, especially like on the founder side.

1:21It's a bit unusual actually, right, to see that technical side show up and, you know, be heavily represented there. So it's really, really good to see that progress. To be an IOI medalist takes a lot of training, honestly. So usually like a few years of just like working on very difficult lead code problems, basically. So lead code problems are usually like 10, 15 minutes, but these problems go up to like hours, right? And it requires just a different mindset. it's like you know you're building up skills every day that that may or may not actually be practically useful but you know you kind of just you kind of just work on them for fun and you just invest in yourself and these are like coding challenges yeah like lead code but kind of more performance based like you have to have basically an optimal solution to get full credit on any of them got it and is this something that happens um in high school in university typically most Most people start in high school, I would say.

2:17Yeah. And then they try to kind of develop their skills over years. And then it culminates at the international competition, which is called IOI, International Olympiad of Informatics. Or there's a related one called ACM ICPC. Did you do that one as well? Yeah, we did that one as well. Nice. You know, in college you have less time, so we didn't prefer as much for that one. And so you were, early in your career, you were an engineer at Cora, I believe, but then you actually left to go back to school, to go back to Harvard. I'm curious to understand that decision, like being in the workforce, building clearly like an amazing company, an amazing role, and then going back into academics.

2:58Like what was the decision like there? The decision was pretty interesting at the time. It was like a great talent pool at Cora. So I love the talent pool that you're mentioning. Also went to Cora at some point, at one point or another, Demi was there. at the time it was like the knowledge-based company to work at, right? Whereas now there's kind of like several cool ones to work at, right? If you're interested in developing humanities knowledge. At that time, that was like the cool place to be. It's the place where everyone learned like how to build products for the user, how to build these building blocks I mentioned.

3:29And then it was interesting to take that perspective and go back to school and understand what actually can you learn that's practical and has real-world applications. And what are some examples of what you learned there that you could apply to real-world applications? Ever mentions big data as a thing. So back in the day, it was actually really hard to get your own big data pipeline started up. And it was very manual. You had to build all these foundational infrastructure building blocks yourself. And nowadays, it's much easier. It's like 100 times easier, probably. But the core lessons ran the same.

4:07It's like, you know, you have to really understand what you're doing every day and how it translates into metrics, right? So if you're optimizing a certain metric, you know, you make that your goal for the month. But then you think about, you know, are you over-optimizing that metric? Is the problem truly only quantifiable versus qualitative, right? So there's all these factors that you learn to balance over time as you get more experience so that you're not just gaining one metric. Right. Yeah, that makes total sense. And then so how do you go from being an engineer, being an IOI medalist, to now, like we were talking about in the beginning of the conversation, it sounds like you've shifted completely into being more about product and strategy.

4:47And that's not a common transition. Definitely not a common transition, but it's good to see the IOI community kind of branching out and trying new things, right? I think at first it was mostly just a passion, right? So it's just understanding the product really well, just using the product a lot myself. Just understanding that in the company there always had to be some motivation to improve the product and think about things from the user perspective. I think over time it's just grown more and more focused on that side rather than the low-level development side. But I still try to stay in touch.

5:25I still try to review the code just to try to understand what every engineer, at least on my team, is working on. Are you committing code yourself at this point still? Yeah, still. That's awesome. Okay, so perplexity. Let's talk about the product. I think you all refer to it as an answer engine, not a search engine. But I think most people that use it, they're probably using it to do something that they used to do in more of a search context with Google. Give us the big picture. What are you trying to accomplish long term? Is this trying to replace the search engine behavior? The big picture is that it's still additive, right?

6:02So it's introducing new behaviors. We have this list of all the kind of verticals and use cases that we think are unlocked with AI. And a lot of them just are not possible with traditional search, right? That's why we term it an answer engine, right? So maybe like 30%, 40 % of queries are longer than a few words, right? Are just not possible, not answerable with old tools, right? And so within those domains, there's still a lot of original, very old search technology that's relevant, right? You know, pulling up sources, trying to find the right graphs, the right visual representations to answer your question.

6:42But the key difference is now you have to understand that the user has very complex intentions. There might be multiple steps in the queries themselves. there might be a more sophisticated, less general answer that you need to render in order to satisfy the query. So it's a much more difficult problem, but at the same time, it's exciting to work on. You mentioned intention. Is that, do you feel like users are coming to perplexity with a different type of intention that they would normally take to Google? And is that a conscious decision you are expecting the user to make? Or do you see it more along the lines of users slowly over time replacing more and more of their behaviors or their intentions with perplexity for something they used to do with Google?

7:29Right. So it's additive, right? So the user has these old queries that they're doing just a few words long. And we do hope to replace those, but we also hope to extend those. Right. And so the user is learning. They can actually start to do their own work with perplexity. They can break code with perplexity. Things like that are just not possible in the old regime. But in the new regime, it's best to have one answer engine that satisfies both at the same time. And like I said, it's a very challenging problem, right? But it's one that we're excited for. If we take the three things, Google search, perplexity, ChatGPT, like search seems to be about scanning the field.

8:09Perplexity seems about getting a specific answer. Where does something like ChatGPT fall along the spectrum? I think perplexity is somewhere kind of bridging both worlds, right? So there is a way in perplexity to find and surveil the links and click on a particular one that you're looking for, if that's your intention, right? But it turns out that that's actually really the intention, right? Because, you know, 80 % of the time you do get a good answer. It's just that last 20 % of the time where you have to go deeper, maybe perform some actions, maybe verify the answer using the source. So having that fallback mechanism is very important to us.

8:49That way, the answer always has this origin, right? That has some trustworthy source behind it. You mentioned about 80 % of the time the user kind of gets what they want right off the bat. And then 20 % of the time they have to go deeper. Is that a ratio that you actually want to maintain because it sort of puts you in a good position strategically between the two? Or do you want to get the 80 % up so that the user can get the answer they need quicker and more immediate, which would obviously lead to like a positive user experience? So we're always going to be trying to push that up, but maybe not in the way that might be obvious, which is to just only give the answer.

9:27Instead of only giving the answer, we think about it more as almost a video game, as our head of design calls it. Basically having ways out of the answer. So you have currently the ability to fall back to sources, also the ability to follow up to your question, as well as click on a few interactive elements. But I think over time, those interactions are going to increase. The user is not going to be happy with just a technical answer 20 % of the time. 20 % of the time they're going to want to follow up with actual actions, things that interact with their personal life, things that interact with the real world.

10:05So it'll be exciting to see how AI adapts to that, right? Because that'll be the next step of AI. Like, how does it actually help my everyday life? So it almost sounds like what you're saying is today the 20 % is there to help the user get the answer they're looking for. but over time you want the 20 % to be sort of more additive or more supplemental and sort of continue the experience and maybe they get smarter or they learn more that they weren't expecting to learn or they weren't expecting to just get in the answer they were looking for. Is that kind of how to think about it? That's right. That's right.

10:37I think that 20 % is going to be hard. There's an 80-20 principle where the last 20 % is the hardest, right? I think just making small headways like 1 % of the time into that direction will be very important and will be the proving factor for whether AI can be really consumer product. Yeah, that makes sense. That's really, really interesting. Speaking of ChatGPT, OpenAI, can you talk a little bit about how this technology actually works? My understanding is you're not training your own models. This isn't a search engine. It's something in between where you're leveraging search, but you're also leveraging large language models to help synthesize the quality of the answers?

11:20Maybe talk a little bit about how it actually all comes together under the hood. So at the core of complexity, there's the ability to ground your answer in the sources, right? So there's a lot of offline crawling of the web, and then there's a lot of online retrieval of the right paragraphs and snippets in order to answer a question. And it's a tricky balance of how much context from those snippets to include, and it's a tricky balance of which snippets from which sources are the most relevant, right? So there's a bunch of machine learning going on underneath the hood before it even gets to LLM.

11:56And when it gets to the LLM, we pick the right LLM for the right scenario, right? So we do actually have an in-house fine-tuned LLM as well. It's called Sonar. And over time, we've adapted that model to, first of all, answer your question quickly and also have the proper formatting at all times consistent with our brand and our conciseness demands. I think having this language model over time will be very strategic in being able to modify and adjust the language model's behavior and direct it to that concise answer experience. The other part of the technology that we're looking at now is how to then also direct that to those actions and those real-world interactions.

12:47And that'll be another challenging problem in itself, but you can think of it as a parallel track. So there's the answer, and then there's also interactive elements that could be presented with the answer. Like going and doing something on behalf of the user, is that what you mean? That's right. Doing something on behalf of the user, what some people hype up as agents, But also interactive elements for the user to interact on. Because as of now, it's actually very difficult for the AI to act by itself without human assistance. Right. So should we think about, before it gets to Sonar or the other LLMs that you're leveraging, should we think about the first step in what Perplexity is doing is actually like a traditional search?

13:29In some ways, it's like a traditional search, right? But traditional search is interesting in that it's always trying to get you to click. So the value function is purely based on clicks, whereas we're trying to have the value be more determined by the user, right? So determined by what the user is looking for. And that may or may not be a click sometimes, but, you know, for our product, 80 % of the time, it's not, right? So oftentimes it's just the most factual piece of information, the most trustworthy source, the most relevant source to the user. And how exactly do you do that search? Are you leveraging Google under the hood?

14:07We're leveraging a variety of APIs and signals that we then combine together to determine what is the optimal snippet and source to use. I think where this is interesting is, for example, in traditional search, you can only get like one URL for an entire massive domain like Wikipedia, for example. So you only get like a very small information, piece of information. Whereas on Wikipedia, there's a wealth of information, for example. across multiple pages, across, you know, embedded very deep within the page. And we can then kind of synthesize and use many of these together. Got it. So the mental model I have in my head is like this two-step process where you're doing some sort of traditional search.

14:50Maybe it's using some APIs. Maybe it's some things you guys have done yourself. And then you're using an LLM to almost like synthesize that information and give it back to the user in a very kind of like succinct and readable way that also sort of exists within the tone and the brand of the perplexity answer. That's right. Yeah. So we actually started putting together some kind of conciseness and kind of brand and kind of user defined objective function for perplexity. That's a little bit different from traditional search. And I imagine also a little bit different from Chachuk Ti, right? We're more focused on the answer being more concise, well-formatted, right to the point, and then offering those jumping off points to the user to expand their answer and to interact with the real world.

15:42Got it. And one thing that, you know, all this makes me think of, especially like this 80-20 rule we talked about a few minutes ago, is how all of this might be impacted by this next generation of models. Like, for example, the, you know, GBT-01 family of models. You know, my understanding is these things are doing sort of a deeper level of, you know, what seems like reasoning in a sense. And I know that's not the same as what perplexity is doing, but as we talked about earlier, that 20 % is sort of giving the user options to kind of do their own reasoning. And so does this new family of models almost do some of the job of what you're trying to do?

16:25And maybe if so, like, are there benefits you get from that or are there actually drawbacks? With every model release, we always see some sort of benefit, right? So for example, as the open source models LAMA came out, we saw a lot of benefit in incorporating those into the products and fine tuning those for our purposes. I think with the O1 model, it's interesting because it's obviously a different kind of mechanism, right? There's a long chain of thought that's not 100 % transparent to the user before it actually outputs the answer. so that between let's say five seconds and 20 seconds of latency for our product makes it hard to incorporate for the average use case i think there might be use cases for example in coding or in more action-based interfaces where then that latency is okay right if you're coding it's probably fine if you're waiting 30 seconds for your answer right and obviously as more companies adopt this technology as more people work on this it'll get faster right so whenever we see that drop in latency that drop in price we see that the use cases for those you know stronger models just keeps expanding right so i wouldn't be surprised if a large percentage of coding answers are powered by something with a longer chain of thought in let's say six months right I think it's just very early right now to call it with the current iteration of O1.

18:02Yeah, that's super interesting. So it's almost like the model itself, well, so today, I guess the analogy I have in my head based on what you just said is the user is almost doing the chain of thought today through the extra 20%. But over time, you're saying the model will actually take care of a lot of that chain of thought for the user, especially in topics like coding. So for context, the O1 model is going to optimize for an objective function that's completely objective, right? Just accurate or not, right? That's why it's so good at multiple choice. That's why it's so good at coding, right?

18:35So I think for objective topics, that'll be the case. It'll be very appropriate to use O1 or very appropriate to use a long chain of thought for legal questions, scientific questions. I think where it becomes difficult is when the user wants to express their preferences or if the user wants to then dive deeper within the chain of thought, I think it becomes tricky because then you have the AI assuming things about the user that may or may not be true. Like if I forget to tell the AI that I live in New York and it starts thinking for like 5 to 10 to 20 seconds, it's not a great user experience.

19:09That's why you need these ways for the user to jump in and interact. Right. So it's almost like either the user has to do the whole chain of thought or the AI has to do the whole chain of thought. But sort of putting those two things together can get really messy. It can. And that's why it's really important to have the understanding, the opinion of which topics are suitable for which language model. Right. Right. And I think that's just only something that we're going to keep improving on and understanding better over time. Yeah. So maybe jumping back to search a little bit. So, you know, if the goal over time is to win more and more of the behavior and go from sort of searching to just more just getting answers, I'm sure you've been asked this question a million times, but it strikes me that perplexity has a challenge to overcome that the traditional search engines have and that they have this sort of embedded distribution in Safari, in Chrome, in all of the traditional OSs and browsers.

20:08Yet, despite that, perplexity seems to be growing really, really, really well. How is that happening? And how do you eventually overcome these inherent distribution advantages that the traditional search engines have? This is a very difficult question to answer in the amount of time we have. But I think the core reasoning behind all of the AI products being successful is that they're disrupted. Right. So having an opinion about which use cases you're disrupting and which cases other companies are not going to be working on as much as you are, like not going to have that same obsession. Yeah. A counter positioning to what they're doing.

20:45Right. And then having that opinion and pushing just as hard as you can on that, first of all, in the product. Second of all, having some ways of being more retentive. Right. So I think there's a lot of traditional plays that you can make like, you know, sending notifications, you know, trying to get the user to read more passive material. But I think the interesting thing is that propensity, it's really easy actually for us to build these building blocks with AI. Right. Because traditionally, in order to have a feed, you would need to have a massive social network, like millions and millions of people.

21:24But we were able to get off the ground quickly, working scrappily with AI's help, kind of hand in hand. A lot of manual work, of course, as well. But working hand in hand to build on the products that wouldn't be possible with such a small team. And I'm really proud of the team for trying just their hardest every day to work what they have, right? Just being resourceful, just iterating quickly, not worrying about the long-term future sometimes. Sometimes it's just you have to worry about zero to one. And at this point, we're more in the polishing stages of some of these products, right? Our Discover feed, our iPhone app are actually well polished at this point.

22:05But you have to remember those products like started off being extremely scrappy at the end. Yeah. Yeah. Oftentimes it's just like one engineer working on them. No product managers, no engineering managers, nothing. Yeah. Yeah. So how do you, maybe from a team sort of goals and culture standpoint, how do you drive that growth? What I'm hearing from you is like, there are no silver bullets. Like you're not going to have default search and Chrome anytime soon. And so you just have to like kind of keep pushing and being scrappy every day. How do you actually drive those results as a team? We have a lot of organic ideas that appear across the team, right?

22:41So recently we had the back to school campaign. I think everyone knows that back to school is an important time. But very few companies are thinking about it from the student perspective and like reaching out to students like scrappily, right? They're all thinking about kind of like running marketing campaigns and spending lots of money on it. But, you know, we took the opposite approach of just going back, understanding that, you know, students love swag. They love competing against each other, right? That's why the programming contests are so popular in the first place. So never losing sight of the users at hand, right?

23:12Understanding their motivations, just having a bunch of individual people, just a couple engineers, one engineer working on the integrations, one engineer working on the UI. Not having a bunch of people managing the engineers, but rather just having the engineers come up with their own thought processes for how to develop, how to iterate quickly. And then oftentimes, mistakes will be made and then you have to kind of recover in that last 10 % before delivery. But it's much easier to let everything grow and then dial down than it is to apply red tape as something is growing. Do you set as a team, do you set, I don't know, like weekly or monthly growth goals?

23:55At this point, the goals are at a very high level each quarter. And then it's broken down weekly. So every week, each individual has their own goal and tries to deliver at least 75 % of that goal by the next week. that way there's that momentum and velocity. I think not every individual has the same attitude of how to accomplish that goal, right? Sometimes they'll do a lot of experimentation and sometimes they'll just try to ship really fast. But I think either way, the cadence is the same. Yeah. So like there's many paths to get to that final 75%. Is the goal sort of more input-based, like ship X feature or is it achieve, you know, Y week over week growth?

24:39There's a lot of qualitative goals as well. It's not just user growth, right? Because user growth at a high level is not completely actionable, right? So you have to break it down into smaller milestones. You have to think about if I do this experiment and it fails, which it will 75 % of the time, right? What did I learn, right? Or if I built this prototype and it's not quite good enough to ship, you know, it's still something, right? there's still a building block that I can then use to develop either slight pivots or just another iteration before it's done good enough. Right. So there's at least concrete milestones that, that get you some satisfaction, even if you're not probably increasing user growth or retention.

25:25Yeah. Makes sense. So let's say you are all very, very successful in growing this product. And, and so far, you know, the, the trajectory suggests you will be, um, Let's talk about the broader implications of that. You know, one of the things that I really think a lot about with, you know, products like Perplexity taking over more traditional searches is that kind of the entirety of the Internet in many cases is works and works for free for most of the users because of the business model of search and advertising. And so the more that perplexity pulls away from traditional search, the more that business model is challenged.

26:08Maybe starting at the top, like how do you think about the business model of perplexity both today and long term? Obviously, you have subscriptions today, but my guess is that is beginning to expand. Maybe you could talk a little bit more about that. The core business is currently subscriptions. So either consumer subscriptions or enterprise subscriptions. I think because it's a very kind of focused productivity tool right now, it makes sense that that's where the majority of disruption is happening, right? Like it's getting your work done a little bit faster. So, you know, for every hour that you're saved, maybe perplexity captures like 1 % of that value for somebody, right?

26:50So maybe we capture like a dollar, like every couple hours, something like that. So it's not trying to kind of extract the value directly from the user on every single query. It's more just kind of how much value do we add across the course of the month. I think the next step will be eventually advertising, right? Because not everyone wants to pay for the subscription, right? And there's actually ways to advertise that are aligned with the user, right? So having good recommendations that are either passively or actively discovered for you, right? And having that mindset and understanding that we're never going to, for example, change the answers.

27:35We're never going to bias the answers. We're always going to be building business models that are aligned with the user and what they want. This is really interesting, actually. So, you know, if you think about Google, more traditional search, the answers to an extent are biased, right? Because you have, you know, certain links at the top that were maybe paid for. In the case of perplexity, you're saying the answer is always going to stay the same. I'm guessing maybe you'll put ads near the answer or around the answer that are related. But the incentive of perplexity feels very, very different than the incentive of Google in that you're trying to get to the right answer.