How To Build The Future: Aravind Srinivas

0:00We released the ability to ask follow up questions. that doubled the engagement time on the site and also increased the number of questions every day. So I was like, okay, there's something here. It's not worth killing and pivoting to enterprise. It was not like I wanna go and kill Google, like that sort of a motivation. It was more like, what is an idea of that scale and ambition is something like this. Today, my view of perplexity is a more intelligent Google search that's really useful in certain scenarios. What do you want me to think of it in three or four years?

0:37Welcome back to another episode of How to Build the Future. Today, we're joined by Aravind Sreenivas, co-founder and CEO of Perplexity, which in less than three years has grown to more than a$9 billion valuation. Thanks for joining us. Thank you for having me, David. How did you get into this world? I was pretty interested in AI, deep learning research. That's actually what got me into the U.S. I was an undergrad in India. I came here to the U.S. for doing my Ph.D. here at Berkeley. Life really changed when I got to do an internship at OpenAI, and Ilya Sutskyer was there. I still remember the day I first met him, and I was very prepared and had all these fancy ideas that I thought were very interesting, and he listened for five minutes and said, all this research is useless.

1:20It feels really bad to hear that. So I got used to hearing the right things, even if they're uncomfortable. And then he told me the only thing that matters is he drew two circles. one big circle, called it unsupervised learning. And then inside he said, reinforcement learning, another circle. And he said, this is AGI. Every other research doesn't matter. This was around the time when they were building GPT-1. They didn't even call it GPT-1. When I saw that research, I went back to Berkeley and said, hey, I was working a lot on RL. That was the rage at the time because of AlphaGo and DeepMind. But that was kind of like chasing the trend.

2:00So, I went back to my professor and said, hey, we have to actually go and study unsupervised and generative models and generative AI. So then I got into that and did more internships at Google. And during my Google internship, I stumbled upon this book called In the Plex. So I would launch jobs during the day, training runs, and then go and read these books in the library because interns don't have any other thing to do, right? And it would feel amazing that, oh, like these guys actually were once upon a time grad students like me, and now I'm working as an intern in their offices. Reading the book about them.

2:34It feels nice. It would be amazing to start a company like that in future where there's a lot of research. There's a lot of like AI. At the same time, it's very grounded in product building. It's very difficult to do that. And I spent a lot of time thinking about it. I even spoke to Ilya Sutskiver about it and he like where we said, there are probably only two problems where you can work on AI and also build product at the same time. One is like search and the other is sub-driving car because all your product rollouts are becoming data points or improving the underlying AI in the product. And that'll make the product even better.

3:12And that'll lead to more users and more usage will lead to more data points and it'll become a flywheel. And it should also be on the AI completeness path. It's sort of a buzzword to say this, but basically what it means is better AI should keep making your products better. So that way you can keep working on your company until AI is solved. And once it's solved, okay, sure, we'll worry about all those implications. But your company gets better as AI gets better, as opposed to your company gets run over by somebody else. Exactly. So search is like one of those problems. So you're at this moment where you kind of have this realization that you want to start a company.

3:49How did you get the kind of activation energy to quit your great job at OpenAI and go do that? How did you find your co-founders? I came across this blog that one of the former YC partners, Daniel Gross, wrote. But it was like how to build the next Google. And I think basically the core idea is like you could do so much more with better query reformulation. So you take a query and you just add some suffixes. So if someone's looking for reviews of a movie, just suffix site colon rottentomatoes.com. If someone's looking for reviews of some new gadget, do a site call in that corresponding subreddit.

4:28You can get away with a lot of these suffixes and special strings to filter results and already make Google so much better, even with the existing Google ranking. I'm not even talking about the ads problem, just simple ranking. And then you can do more sophisticated things of classifying queries. and he was talking about how LLMs could automatically figure out these suffixes. And I was pretty interested in that. Okay, that seemed like, okay, maybe generative AI might be, as in like I used to just call it LLMs or generative models, could be a better way to build search engines too. I also was pretty interested in like trying to do something agent-like.

5:11When DeepMind had this Android environment that they built where like they kind of wanted to prototype mobile app using agent that knows when to use what apps and control the apps. That's when I spoke to my co-founder and CTO. His name is Dennis. We had written the same paper a day apart. So we knew each other and he was a visiting student in my lab. And we used to talk about, we'd bring some ideas of how we can build agents to control the Android environment. So we were definitely chatting about doing a lot of things, but never concretely about any company or product. The first thing that anyone would tell you is like, why?

5:46Why even work on this? of course Google's going to do this it's not even like you go build a better Google Docs and Google will eventually do it because it's a secondary thing for them so companies like Notion can still be funded this is their core crown jewel so why would you even try I think the reason it actually made sense this is again after launching the product we realized this, not before so there's like some benefit to being ignorant ignorance is bliss is that if people stop clicking on links, the ad economy kind of dies. Now you can, there's a lot of nuance to this, but that core insight was only realized by us after launching.

6:29So once we realized that, I thought, okay, we were onto something and that kind of took us the last two years. Yeah. Walk us through like the first iterations of your experimentation. Like I know you did a bunch of demos that were very dissimilar from perplexity. Yeah, so I was adoshes enough to go and pitch to the first seed investor of ours, Elad Gill, that I want to disrupt Google, but I kind of want to do it from pixels from a glass. And I think that's why you're not competing with people typing on the search bar. They're just seeing. But even at that point, you knew in your mind, I want to go after Google.

7:07Yeah, it was not like I want to go and kill Google, that sort of motivation. It was more like, what is an idea of that scale and ambition is something like this. It was also around the time when multimodal models were slowly beginning to work. So I thought, like, if you were on the trajectory of improving technology, you could build something pretty amazing. My investor rightfully said, like, not to work on it in the beginning. So we focused more on searching over, like, specific verticals or data sets or databases, tables, actually. And we were an enterprise kind of focused company, except, like, nobody wanted to give us their data.

7:41I remember I used to, you know, hustle for calls with like PitchBook or Crunchbase because I kind of wanted to build a demo that would first make sense to an investor. That way we can keep, you know, raising some capital and then actually hiring good people and then go and like do the real thing. And so Crunchbase had all this data, PitchBook has all this, but they just don't want to give it to us. And so... Next best step, Twitter. Yeah, Twitter. The pre-Elon CEO moments, academic access was allowed, legal. So we built a database of Twitter. We organized it in the form of tables. We tried to do it with the OpenAI codex models.

8:19This was even pre-GPT 3.5. We wrote a lot of templates. Oh, for these kind of queries, these are example sequels. And then the model would do kind of rag, pull relevant queries in the templates, and then write the actual SQL based on the template sequels. And that was the only way to get it to work reliably. And then we had a lot of callbacks in case errors happen. It'll automatically correct it. And then it will go and query the database and then retrieve the records. Got it. It was very nice. And it was a chat UI. You could chat. You could converse. You could plot. And this was like the first real product or demo that you guys watched?

8:56Yeah, yeah. We did it very fast. It took only a month to do this because there were like three people only. But that energy in the beginning was insane. and they showed it to a bunch of people and they all allowed it. Mainly, there are two reasons. One, something like that never existed before. Like you could never actually go on. Twitter searches. Exactly. To this day. Even today, right? And then also people allowed finding all these, I think the social search of like knowing who other people are following, whose tweets are they liking, whose tweets are they not liking, who did they unfollow this week.

9:30You know, those kind of diffs. It's all like funny. So you launched this Twitter search thing. Yeah. How did you transition from that to now what we all know as perplexity? Yeah, so we had that, right? And then we were trying to do something similar to that for many different databases. GitHub, like if coders could go and search about repos or LinkedIn, if you could like almost be a recruiter, just say, but even now it's pretty hard. I want to say like, I want all the people who worked, who have been YC founders, you and who also worked at a CCOD startup because they would know what it means to be scrappy.

10:06It's very difficult to do these queries. Using the LinkedIn UI to do that. You cannot do that, right? For whatever reasons, people don't want to give their data, their paywall. Such technology, if it exists, we'll be creating way more value. But it doesn't exist for many other reasons. We were beginning to see how, even with the capability of the models at that time in 2022, pre-3.5 Turbo, things were actually pretty reliable to the extent where people would use this and find value. I actually read this Paul Graham tweet, I think, where if you try to, often when you figure out the better solution, when you try to solve a harder version of it, but you end up with a simpler solution that's more general and scalable, that's what we realized.

10:54like, okay, like there's one way of doing these things where we go to each of these domains and like try to build an index of it and put it into specific formats like tables and then have the LLM like read that in a structured language of SQL. Or you could do the other way where you just keep it unstructured and expect the LLM to do most of the work at the inference time, at the time of the query. And don't do all this work in the indexing time. and clearly we knew that if second is where the world is headed, where the models will get smarter and smarter, it gives you an advantage to build it that way because it's more general and you also stand a chance against the legacy system that Google has built, which is a lot more in the first style.

11:42So we thought, okay, we would try to build a more general solution and then we prototyped this thing one weekend, actually. Actually, John Shulman's team had already published this thing called WebGPT at the time. So I was pretty aware of it. OpenAI even had a bot when I worked there called the Truthbot, which John built with this team, where you could ask it a question and it'll go and search the web and then it'll give you an answer with some sources. And it was very slow. And it was built with 175B GPT-3 model, so incredibly slow and inefficient. It was more agentic, like it would actually be like an RL agent that decides if it wants to click on a link and browse it, scroll.

12:22Okay. This is very slow. So what we tried is a very simple heuristic version, but much faster, which is, okay, you always take the top key links that a search API provides you. You always only take the summary snippets that the index has already cached. So there's no scrolling, there's no clicking. And you always feed all those links into the prompt. So there's no selection. Ask it to write a summary with sources in like the academic format. And that's it. When these models were getting to a point, like 3.5 turbo sort of models were beginning to come, this actually started working much better.

12:59Got really good, yeah. The instruction following capability increased enough that you didn't have to do it very, very rigorously. Gotcha, so you kind of did like the dumb approach, betting on the fact that the AI would get good enough that it would make all of that irrelevant. Right timing, I would say. One year ago when John and his team tried, the models were just so much worse that if you tried the dumb approach, it just wouldn't work. And so therefore you would conclude that you need a smarter approach. But then when the models began to be much better instruction following, the dumb approach actually works.

13:35And that fixes a core product UX problem of latency. You are used to links appearing instantly on a traditional search. right? Even then by the way the first version we launched which is the answer version took seven seconds or something because we didn't even have this concept of streaming answers. We would wait till the entire answer was generated. We couldn't control the verbosity so sometimes the answer would be very very big. We even had to hard code a prompt saying only write five sentences or something like that or 80 words. To keep it fast. Yeah exactly. Okay so you launched this when was the first moment that you thought like oh i'm on to something here so we tweeted it okay i i was while writing the tweet i was like um you know people are going to ridicule it it's going to make mistakes blah blah blah first moment of virality came when one uh annoyed uh like intellectual academic came search for herself uh it said she it gave a biography in the past tense.

14:38And she's like, I'm still alive. What the hell? But actually what happened was there was a person with the exact same name and spelling who died. And LLM thought she died and she gave a past tense. I actually thought there was pretty clever reasoning on the model, except it's not even higher order to know that they're different people. So then that got us a lot of attention. People were beginning to start thinking, okay, look, the sources thing is good, but can we really trust the answers these things are saying? And then that got into this trend of people searching for themselves. This is something that keeps happening time and again with all consumer products.

15:19When I got a chance to speak to Mike Krieger that vacation, he said the same, that even though you can click on your own profile icon and go back to your photos, people always love to go to their profile on Instagram by typing their username on the search bar. It's such a human habit. So a lot of people start putting their Twitter handles or social usernames, and then it would mash all their activity across the internet, including stuff they did in the childhood many years ago, and then give these interesting summaries, and they would screenshot it and share it. So I thought there was something there.

15:56So there's something driving it. Yeah, but I still wasn't sure. And then we released the ability to ask follow-up questions. That doubled the engagement time on the site and also increased the number of questions every day. And the number of people, number of questions every day was increasing exponentially. So I was like, okay, this is, there's something here. It's not worth killing and pivoting to enterprise. You have this like initial momentum. And you said earlier, it wasn't until hindsight that you had the idea that like, oh, we actually have a chance of competing with somebody like a Google.

16:26When did that realization happen in this journey? How did that go down? So I never really thought about the Google competition in a serious way, to be very honest, because I knew that like they cannot make this exact product on the Google homepage. It's so hard to know when a query is purely informational or not. And then the Google search page is already like so cluttered. That's the answer box, the knowledge panel. There's some ads. There's some links. There's, you know, like perspectives from socials, all these social cards. it's already too much information. So it clearly feels like fast food and healthy meal difference for using Google and perplexity on even informational queries.

17:09I was more worried about Microsoft in the beginning because they were launching Bing Chat. In fact, on the day, we agreed to have a term sheet, like handshook on a term sheet with one of the venture capital investors, NEA, here in Sandwich Road. after like one week of torturous pitches. And we're just having like coffee. And then the words leak screenshots of Bing chat. And I was like, okay, like there's this 30 day due diligence period, right? And one of the other investors will give me a term sheet. He just increased it to 45 days. You know, you could see the diff. Yeah, right. It was in sneakily.

17:48And I knew why clearly. He also texted, what do you think about this thing? Okay, okay. I get it. I get it. getting a little sheepish. And then the other person I handshook with, like he texted me the night saying, hey, do you have time for a call tomorrow? Okay. Like clearly like this is it, right? So I told my co-founder, okay, maybe they're going to back out or ask us to pivot. So maybe we should just try to sell the company and like get it done. You know, this is not going to go anywhere. The person actually who handshook said, look, I'm not going ask you to pivot i'm not going to ask you to like do anything different uh you guys keep going and uh we already word is word i was like damn that's that's pretty impressive and then the next week actually google also releases a blog from sundar saying they're announcing something called the bard which is screenshots so we knew that like this is going to get pretty big and competitive but we were like look it's at the end um microsoft was never really good at consumer products for a long, long time.

18:50You can't suddenly change that. So they actually messed up the opportunity, in my opinion. Google, obviously, I knew that they're going to have their own problems and challenges. So I felt like there was space for someone else here. Yeah. Having spent almost a decade at Google myself, I see a lot of the culture of the early days of Google, like the things I've learned about Larry or about Sundar. And I see a lot of that in the way that you have built your product. There's a lot of attention to detail. It feels like you are the primary user of product yourself like is that a thing that you deliberately tried to do yeah i did i did deliberately try to do it uh one thing that larry said is like you know we we uh i keep reminding everyone in our company about it the user is never wrong so even today while testing a new feature um it didn't work uh but there was some ambiguity in the query so the person i was talking to the engineer and saying, hey, you know, this is not good.

19:46What else could AI have done here? And you know what AI should have done? It should have come and clarified to me and asked me, hey, I'm not sure either this or this. Which one did you actually want? And then I should have clarified. And then it should have gone and done it. Instead of saying, I don't know, that is the user is never wrong principle. The other way of designing products is like, make the user be a better prompt engineer. Blame the user and tell them to be a better prompt engineer. Teach them, educate them, get them to do it the way that the product wants them to do it. Yeah, exactly.

20:18Enterprise software is more like second kind. Yeah. But magical consumer products are more the first kind. I agree. Right? Like in Google, why should Google handle typos? They needn't have, right? We should have all been great at English. It's like Larry says he was never good at spelling, so that's why. I think the true story is YC partner, Paul Buchheit, he was just annoyed by it. And he's like, someone should build that. Yeah, exactly. And spell check, corrector, it's all there. Similarly, auto suggest. Why is it there? Like easier, right? Similarly, cached results. I was even reading somewhere where Larry wanted the homepage to have the simulation of the weather outside your home so that you don't even need to type the weather query.

20:58It's just already there. So I was very influenced by that style of design, including subtle things, the Chrome search bar. If you've already gone to a site, it's already there. You just have to click enter after typing the first two letters. So that influenced me to to make sure we have the cursor ready to type on the search bar. You don't need to take your mouse and place it there. It sounds like your main metric that you care about is number of queries per day. Yeah, exactly. Which is exactly what Google did, I think, in the early days, right? It's hard to grow that without retention. In the long run, I agree.

21:32You cannot just pay for a user and get that number up. User could install your app, and maybe you can even game it where when they install, that's one query automatically submitted. but a repeat query doesn't need to be submitted. Yeah, I think the only counter example, which I don't think is happening in your case, is the product is not serving their needs, and so they need to issue a bunch of queries to get what they want, which is kind of the opposite of Larry's approach on Google was, you should be on Google as short as possible because we're trying to get you somewhere else to solve the problem.

22:00Yeah, so that's not happening. I mean, sure, I'm sure there are some errors and stuff, but most of the follow-up queries actually we see are completely irrelevant to the first query because they just want to keep continuing the session or questions that they never even knew they wanted to ask, but they want to keep asking. So I presume your team has grown a bunch. You've raised a bunch of money. How do you manage the team? How do you operate your team on a week-to-week or cycle-to-cycle basis? With that, number of queries per day is our primary metric. So every all-hands, we start with that number.

22:34I don't believe in this putting a TV and having a metric being seen every day because I think that's also distracting. But I do think it makes sense to take a look every week, see the weekly growth rates, see the monthly growth rates, and if something declined, then discuss about it, figure out ways to actually freak out if something declines, like we do, and if something grows, look into why, where. So we are very data-driven, and we share it across the company. actually I've been trying to share to the users too so that they feel like uh you know it's an something that's actually happening right in front of their eyes and they want to be part of it there's a hierarchy like if if there is some bug to be fixed if I know some particular person's working on it I can go and talk to the person directly nobody else feels threatened because I'm going and talking to that person there is no feeling that because I'm raising a bug uh uh it's like, oh, they're going to be fired or something.

23:33Because I raised like 50 bucks a day. So if you know, like, it's more like they understand, okay, it's important for the product to feel great. And if it doesn't feel good for ourselves, the user is also not going to feel that. In fact, we have way more incentive to go use our own product, but the user doesn't. So the standards for the user should be even higher. So always feel like a user. I think that culture is there in the company. I love that. And did you intentionally select for that when you were hiring? Like people who are very product centric and in the details? I wouldn't say I explicitly had that as a criterion, but I look for people who cared about doing good work.

24:12If you don't care and you're just treating it as a job, then it's very hard for you to get excited about things. And I think so much of it feeds off of the founders and like your culture, your DNA. And it sounds like you're that type of person that obsesses over the details and you're just going to naturally want to hire people who share that trait. Yeah, I do get pissed off if answers are wrong. And I do get pissed off if people on Twitter are saying like, perplexity is degrading or like, you know, but a lot of the things, some things are actually not true. But I do try to see, you know, leave aside the cynicism, even if it was someone who's like a hater.

24:49But if there was something true there, and I want to still know. Yeah, I love seeing you engage on Twitter with customers. Is that the primary way that you talk to users? Or are there lots of other ways that you are talking to your users? I mainly use X on Twitter. People are just super brutally honest there. And I think in email, people are a lot more polite. Which is okay, too. I like both sides. But I think the brutal honesty brings out the worst bugs and things that people are afraid to say. Oh, and in person is the worst, where you go show someone something and they're just going to tell you good things.

25:25Yeah, yeah. Even if they hate it. I kind of like don't like any, hey, tell me what do you think? You're always going to say nice things. You're going to grow your company, presumably. You're going to need to hire more people. How do you avoid the fate of becoming a big, slow company? Well, it's beginning to happen already a little bit, right? We're not as fast as we used to be. I think some of it is not because of people. It's also because things breaking in production, people start losing trust in the product. like today we deployed some change and then someone got frustrated that there was some front end bug somewhere.

26:01It was actually something back end, but people are just assuming things. I think like fast loading, all that stuff matters and not every new engineer has the full context of the code base in their head. The earlier ones do. There's some tension in like moving fast and breaking things if you do want to like grow to mass market usage. So that's mainly slowing us down, I would say. and we haven't quite figured out the best way to do this fast. I mean, we do have staging, deployment, testing, A-B tests, and all that stuff's happening and that's naturally slowing us down and getting things out to production widely.