Edo Liberty: How Pinecone Revolutionized AI’s Long-Term Memory

0:04Hey, everybody, and welcome to Generative Now. This is a podcast where we talk to the builders who are creating the world's most exciting AI products and companies. I am Michael Magnano. I am a partner at Lightspeed. And for this episode, we'll be featuring another conversation from Generative NYC, the in-person meetup that was actually the inspiration for this podcast. Back in mid-2023, I sat down with Ido Liberty, founder and CEO of Pinecone, one of the most important companies building in generative AI today. Pinecone powers vector databases. If you're not familiar with what those are, they serve as long-term memory for AI and ensure that AI applications are getting the most relevant and up-to-date information as quickly as possible.

0:46This conversation with Ido got into the details of how VectorDBs work, how Pinecone leverages them, and where he sees that technology heading into the future. So take a listen to my conversation with Pinecone CEO Ido Liberty. Let's give it up for Ido for joining us tonight. And yeah, thank you so much for joining us. Appreciate you doing it. You're right down the road, right? Yeah, we walked up here from 36th Street. Amazing. Well, I appreciate you being here. And I think some members of your... Well, we have some other Pinecone folks here with me, so... Hello, Pinecone team. Cool. Well, listen, Pinecone's been such an incredible story.

1:24You know, many people are now just hearing about Pinecone for the first time because LLMs are exploding and AI is exploding. But you've been at it for a while. This is not an overnight success. Would you mind giving us a little bit of background of your personal story and how Pinecone came to be? First of all, I apologize for all the engineers out here where you're offended by buttons. I had meetings today. Usually I'm with a T-shirt. So my personal story, I started off with my Ph.D. in computer science, my postdoc in applied math. I was a data scientist and engineer for a long time. Then joined Yahoo as a scientist and an adjunct professor at Tel Aviv University, kind of in tandem.

2:05Stayed at Yahoo for a long time, moved here to run their research labs. Then moved to AWS to build and manage services in the AI space. If you use SageMaker and other tools, that's, you know, my team and others have built that and built, been in that ecosystem for a long time. and then started Pinecone in 2019. And kind of throughout that journey, the underpinning of what Pinecone does today was growing. It's like, you know, my PhD was on the topic of high-dimensional geometry, big data, the foundations of theory of machine learning and so on. And that has always been kind of at the base of it.

2:46I saw the gradual need and growth in that kind of infrastructure. You could see how that was powering, you know, eventually feed ranking at Facebook and shopping recommendation at Amazon and like ads at Google. And you figure out, wait a second, this like the biggest products in the most AI forward looking companies are ran by one platform that nobody knows about. That was like a big call to action at some point. I said, OK, this is going to be gnarly to develop, but once it's out there, it's going to be massively useful. And that was that was it. It took only four years to put together. And and when did you start when did you start seeing that that, you know, really strong organic pull from the market?

3:34Like you said, you said this is going to be important one day. When did that actually come? So it kind of, you know, it kind of grew monotonically over time. And I kept personally thinking, kind of flipping between thinking, oh, it's way too early, nobody knows about it, and oh, it's too late, we've missed the, you know. Now this is, we can't even catch up anymore, right? But then Transformer models, like BERT came up, and suddenly for the first time, like everyday developers started talking about embeddings, and like it kind of became a topic of conversation. and then I felt like, okay, this is really like, it's either now or never.

4:16Yeah. Taking a step back, you're in a room full of extremely smart people, but for those of us in the room, myself included, who are not engineering day-to-day, who haven't built products in AI, talk to us about vector databases. What are they? Why are they so important, especially in this new world of LLMs? So I'll speak in two levels, all right? I mean, so this is kind of the meta level, and then I'll zoom in a little bit, okay? And I'll try to keep it pretty high level, okay? At a high level, okay, you can think about LLMs and in general, in some sense, AI, as developing computational reasoning, right?

4:59Understanding summarization, building language, building sentences, composing language, and so on, right? you do not necessarily think about them as as containing the the knowledge itself right like when somebody we think about somebody as being very smart it's almost sometimes hard to know whether they're smart because they have a high IQ because they just know a lot if you work in a law firm you know you have the 22 year old whippersnapper who's like clearly the smartest person in the room but they don't know anything right and you have the six-year-old partner who's been And like who knows everything, right?

5:32And they, you know, that's a different kind of smart. And so when we build AI, we want to have both. We want to have the knowledge and we want to have the kind of the IQ. And so you marry a long-term memory. You marry this ability to let the language model access years' worth of information and years' worth of experience and bring it into the generation of language. And so if you work in a law firm, you might ingest hundreds of thousands of documents and contracts and so on and put them at the disposal of the LLM. And now kind of popping down one level, why is it a vector database? And we can talk about that in the technical level if you want.

6:18The language models don't access data with SQL. Like they don't do keyword search. They don't do like key value pairs. They don't represent the world in that way. They represent objects as numerical objects or vectors. And so that becomes the key. That becomes the search. So the interface is just different. So the database that you need to be able to operate and house that information, retrieve it efficiently, is different. It's not a traditional database. Got it. So a law firm ingesting thousands or hundreds of thousands or millions of pages of legal documents, one use case. What are some of the other use cases you've seen emerge on the platform and for vector databases?

6:59So for specifically for generative AI, when you ask, you know, you have some interaction, you want to provide context. You want it to be based on actual data and you want that actual data to be fresh. Like if you update it every day or every hour, you want it to be correct, like stuff you actually know. You want to be able to delete stuff if you want to be GDPR compliant and so on. And so that is a very common pattern. And any conversational agent that needs to be even remotely accurate and fresh and accountable and so on kind of has to do it. And it doesn't matter if it's technical support or legal analysis or, I don't know, what have you.

7:41There are many, many other use cases of vector databases that are still there, right? semantic search is a huge use case. I mean, anomaly detection, deduplication, recommendation, you name it. They're all in some sense search retrieval and machine learning problems, but you kind of have to think about them that way. Got it. Shifting gears a little bit, let's talk about startups and AI. Again, you're in a room full of builders. We talked about a minute ago how you started Pinecone before sort of this explosion of AI. Talk to us about the journey of building an AI startup maybe what it was like a few years ago before it was the coolest thing in the world and what it must be like to i mean you are building a company today what's it like building a startup uh post let's say september 2022.

8:27very relaxing affair very relaxed yeah i'm sure exactly i'm sure there are founders here and people that you know part of very young companies so while right. I mean, this is, you kind of go on this crazy journey with the rest of the world. Funding is easy, funding is hard, crypto is hot, crypto is trash. Like, MLOps is the next best thing. Nobody gives a shit about MLOps. Like, this, you know, and we all go through these cycles together. We can talk about it for hours on how the journey has been. But at the end of the day, it's just, It's just, you know, you kind of have to kind of look far and think fundamentally about your business.

9:12And if it makes sense as a business, then it will make sense. And if like it's very easy to get discouraged because the thing that you're doing today is just nobody gives a shit about. And it's very easy also to get excited because everybody is excited about it today. Right. But that changes tomorrow. So just kind of keep your keep going. Yeah. But also stay, you know, don't get bought into the hype, you know, neither good nor bad. Yeah. Right. I mean, in some sense, I am as much as I wasn't discouraged when people didn't know what the hell a vector was. I'm also like not as excited as other people about generative AI.

9:55I'm like, I'm like, OK, yeah, it's a hype. It's come down. It's going to go up. It's going to come down. And at the end of the day, like this is a foundational layer that, you know, tens of thousands of businesses would need. And we're going to get there. Right. The technology that startups are using in AI right now is so powerful. Like, you know, in the world of generative AI, to be able to build a product that produces amazing images from prompts or, you know, spits out beautiful long essays from just a couple of words. It's so powerful. And in many ways, it's democratized product building in some way.

10:30How does a startup stand out in this ecosystem, in this AI ecosystem, where new products are launching literally every day? In some sense, it's a continuation of exactly the same thing that I said before. I don't believe you build a great product by some shimmy layer on top of chat GPT. I mean, that's not a moat. That's not a business. That's a hackathon. Think about, again, realistically, like how hard is it to actually do what you're trying to do? How many people actually want it? What value you actually bring? That's the same. It doesn't matter if you're building on top of LLMs or on top of compilers.

11:14It's a tool. If you're bringing value, you're bringing value. And so I think it's a new tool that we get to build with, And I think the people here are already ahead of the curve because they're very, very early adopters and they're willing to embrace a very new tool. But, hey, it's raw and it's new and it's half broken. It's got sharp edges and nobody really knows exactly how to use it. And if you found a way to give customers a tremendous amount of value using that tool, amazing. You got yourself something great. and if you're you've just hacked at it then it looks cool okay and maybe that's not quite there yet let's talk about the other side of the equation so we have startups and then we have incumbents big companies microsoft google etc felt like a few months ago we were seeing all this emerging tech break out of the startup landscape now incumbents are starting to ship as well right we're seeing every week it feels like microsoft's announcing something or google's announcing something.

12:15How's this going to play out over the next few months? If all these companies, whether you're a startup or you're a big company, you have access to the same technology, it feels like the advantage really goes to distribution and being able to reach many millions of companies. Do you feel like startups have an opportunity right now or the incumbent's just going to kind of wipe them all out once they start shipping? I think that's always the fear, right? I've heard the same argument about every technology ever. Why doesn't Google do this? They could, they have 80 ,000 engineers, they could do anything that a 10 person startup does.

12:50But why won't they? Because they can't do everything. They're not fighting you, they're fighting Microsoft. off. So like, realistically, it's the answer is that for most of those, for most companies today, the reason why Google doesn't compete with them is because it's not a priority for them. And when it becomes a priority, your business is already big enough that you can probably, you can probably fend them off. Right. So move really fast. Yeah. I mean, and again, you have to be realistic. I mean, you can't, you know, So if you build a startup to go after the number one priority for Google, then maybe that's not great right now.

13:34But priority 5 ,000, yeah. But it's also like that's the unfair advantage of startups. If you were a VP at Google and you start a new product line, and after two years, your product line makes$10 million a year, you get fired. You're a loser. What have you done? I mean, you wasted energy, right? If you start a startup today and your startup makes$10 million in two years, you're a rock star. Like, you crushed it. It's not the same economy. It's like the economics of startups and big companies are not the same. To move the needle at Google, you have to bring in, you know, hundreds of millions of dollars a year as a business.

14:13Otherwise, it's not interesting to do it. Like, you'd be discontinued. Speaking of big companies, some of the foundational models are obviously getting really big, like OpenAI. and there are others following. There seems to be this kind of debate that's taking place in AI right now that there will either be sort of one model, one or a few models to kind of rule them all, or there's going to be this sort of long tail of models since it's becoming much, much easier over time to build models, to train models. How do you see this market playing out? Is there going to be this vibrant ecosystem of open source models or is it going to be a monopoly or a duopoly of open AI and maybe one or two other players?

14:52I think there will be a set of quote-unquote winning models, but I don't think anybody is winning in this space. It's just going to be commoditized, I think, fairly quickly. You'll have specialized products for different parts of the market and so on, but the language models itself, like if you look at the open source stuff, if you actually have tried training them yourselves, it's shockingly simple. it's kind of like annoyingly easy in hindsight. As a society, we've been stumbling in the dark and somebody flipped the switch and suddenly like it's all, oh, it's right over there. But has OpenAI already run away with it, right?

15:35They've got hundreds of millions of users now using this thing. Yes, I can spin up my own model tomorrow, but how do I get customers? People are conflating the product and the technology, right? The technology is there. If you want to build your own LLM, knock yourselves out. It's not even like, do it, right? You know, you'll have something not bad pretty quickly. What they're doing is building what seems to be a consumer business now. And they've basically created a product. They create a good product. ChatGPT. Yeah. The users use ChatGPT. They don't use the model, right? Right. So for me, it's about the product.

16:13It's not about the language model. The language model is the marketing asset. so i know i might sound jaded but it's it's true no no there's like a lot going on it's not like what about the api is that do you consider that the product part of the developer experience yeah it's a part of the product yeah okay uh i obviously had a lot of questions but i know there are people in this audience that have a bunch of questions we have about 10 minutes for questions from the audience i think we have some some mics running around if you want to ask a question, raise your hand. We'll bring a mic to you.

16:46Please stand up, say your name, say where you work, and yeah, we'll get some questions going. I've worked at Ramp doing AI there. I've got a question for you. On the startup versus incumbent thing, Amazon and Google have released recent pretty good vector databases. Amazon launched PG Vector and so there are our products. They've open search. Google launched Vertex AI, and we are an open search user after evaluating some of the options. We are an Amazon OpenSearch user after evaluating some of the options. In the long run, around the same question about model dominance versus vector database dominance, where do you see Pinecone evolving in the market versus the incumbent options from the cloud providers that startups and enterprises already use?

17:27It goes back to one of the very first things that I said is that you have to think from first the principles about your business. One of the reasons why Pinecone is a new kind of database because it is a new kind of database. The data layout is different. The query path is different. The write path is different. If you want to do that at scale, correctly, fast, cheap, and so on, you need to build a new kind of database. If the easy path was going to be to bolt this on Postgres and everything would work perfect, then Pinecone wouldn't have a reason to exist. And I wouldn't have started to begin with because the right thing would have been to bolt it on Postgres.

18:10By the way, some of the vector search libraries inside AWS other offers, I helped build when I was there. It's not the right thing to do. It's great if you're like a heavy Postgres user and you have some vectors. Great, you have that in the product you're already using. And by the way, all databases are going to have vectors as a type because they have to. because now it's something that people use, right? But if you have a large use case, you need something that's specially built to do that correctly at scale, cost-efficiently, conveniently, and so on, right? Whether it's also possible to do it in other places, sure.

18:51We're going to move on to the next question. I am Abbas, founder of Inova View building, back-office automation using generative AI. My question is somewhat similar. There was a research paper a couple of months ago which talked about expanding transformer memory to a million tokens. I'm sure we'll expand to 10 millions. Where do you see the role of vector databases when LLMs are capable of millions of tokens themselves, context memory? I just don't think this is the right mechanism at all. It's like saying, along with every Google query, I can send a bunch of web pages to search over. along with every Google query, I'll send the whole web with it.

19:37It's like, okay, how is that going to work? So you'll say, okay, instead of sending one page, I'll send 10 ,000 pages. Okay, but it's still like a trillion web pages out there. Yeah, the context window is great. It's a great tool, right? But it's not going to contain all of the knowledge you want to access. Even deciding what you want to send requires a vector database to select from maybe millions of documents that you have. Cool. I think we have a question here. So kind of actually you set up my question, which is I've heard mention of a post-transformer world in a few years in which perhaps an architecture that won't heat up as much compute will replace transformers.

20:21And I wonder, in a kind of open imaginative way, how might pine cone intersect with this world? So, you know, post-transformer might sound like as if I'm against transformers or something. I'll just say that I think... I'm not saying that... Yeah, I think they can be made a lot more efficient in the sense that today we bake into these large language models a lot of knowledge. We require them to actually memorize a lot of information. that's not very efficient. And so I think that if you can distill the knowledge and storage part of it from the compute and reasoning and language part of it, you would end up with models that are significantly smaller and more efficient and not any less accurate.

21:16Or if you want to think about it in a different way, if you kept the models the same size, they'll be a heck of a lot smarter. My name's Zach Mandel. I'm one of the founders of Push AI. And my question was less technical. So kind of along the idea of the commodification of models and products, I'm curious if there's any go-to-market learnings or insights that you learned along the way. I feel like you have to start with this big vision. And then it's like, what are the steps we need to take to get there? So what have you learned in your journey? I can tell you what we went through. I don't know if this applies to your business.

21:55I mean, I think different companies take different paths. Okay. I can tell you that we had to say no to like what customers were asking us to do pretty much all the time. We still do. Okay. We had to figure out, okay, what is a sustainable, hard differentiator for technology? And let's focus on only that. and it's very, very tempting in the early goings to get pulled by customers to do stuff that is adjacent but not exactly your core competency. And so, you know, you talk about commoditization. Yes, a lot of the stuff around you is going to get commoditized. Again, if you thought about your business correctly and you know what's the thing that you're going to be the best in the world at, that's not going to get commoditized.

22:52You're going to be the best in the world in doing that. Try not to spend time on the stuff that gets commoditized. Integrate with it. Give it to somebody else. Find partners, whatever. Let other people make money that you maybe could have made. But at least you get to keep your focus. And I see that all the time nowadays, especially with AI companies. Because, again, you are experts. and so your customers are going to ask you to do stuff that is obvious to you, and you know it's easy, but they have no idea, right? And so it's very tempting to say, yeah, I'll do that for you, but it's not really my thing.

23:31The more you do that, the less you get to focus on your core product. I think we have time for one more question before the demos. It's more of a feature request than a question, but love that. One of the problems that we have when we were trying to use Pinecone is getting the data from our database, thinking of Pinecone consistently. Getting it into Pinecone? So let's say we have a giant column of data and we wanted to vectorize, embed it and put it in a vector database and keep it in sync with that column. So as a column, rows get deleted and updated and added. Where is it? Where is it now? Where is it?

24:10It's in Postgres. Okay.

24:15It seems like it would be in your interest to add a solution to keep Pinecone in sync with existing data source. Right now, I've talked to a lot of people who are also using vector databases, and it seems like the best solution is just use a while true loop and watch for changes, and then embed it and put it in Pinecone. But it doesn't seem like a great way to do it. So, curious what you've seen. First of all, great feature request. Gareth is next to you as the PM at Pinecone. He'll take notes. I think it depends on what you mean in sync. Keeping any two databases in sync, even keeping one database in sync is hard.

25:03Let alone two. But if your definition of in sync is a little bit loose, like in the sense that they catch up on like what whatever, like on a horizon of minutes, that that shouldn't be very hard and we can work on it. I mean, we should, yeah. Feature request written down and accepted and we'll make it more obvious how to do that. That was a great question slash feature request to end on. Thank you to all the questions. And let's have a huge round of applause for Ida Liberty, CEO of TimeCon. Thank you so much. Thank you so much for listening to Generative Now. If you liked what you heard, please do us a favor and rate and review the podcast on Spotify and Apple Podcasts.

25:49That actually really does help us. And if you'd like to learn more, you can follow me at Magnano on all the socials, or you can follow Lightspeed at LightspeedVP on all the socials. Generative Now is produced by Lightspeed in partnership with Pod People. I am Michael Magnano. We will be back next week with another awesome conversation. Take care.