#219 Avthar Sewrathan: How to Build Smarter AI Applications with PostgreSQL

0:00I think what has really helped Postgres succeed is the reliability and robustness. I think having been around for 30 years, all the problems about the system have been known, and most of them have been fixed. And I think another defining characteristic of the Postgres ecosystem and Postgres database is its extensibility. So it has this notion of extensions where, you know, even if you're not part of the core Postgres team or a certain functionality that's not part of core database functionality, independent developers and companies can build these things called extensions, which essentially hook into the main Postgres planner and Postgres database.

0:37And that allows you to do all kinds of things beyond just kind of transactional workloads. What does the future hold for business? Ask nine experts and get 10 answers. Bull market, bear market, rates rising or falling, inflation going up or down. Can somebody please invent a crystal ball? Until then, over 40 ,000 enterprises have future-proofed their business with NetSuite by Oracle, the number one cloud ERP, bringing accounting, financial management, inventory, HR into one fluid platform. With one unified business management suite, there's one source of truth, giving you the visibility and control you need to make quick decisions.

1:24With real-time insights and forecasting, you're peering into the future with actionable data. If I were a larger organization, this is the product I'd use. Whether your company is earning millions or even hundreds of millions, NetSuite helps you respond to immediate challenges and seize your biggest opportunities. Speaking of opportunities, download the CFO's Guide to AI and machine learning at netsuite.com slash ionai. That's netsuite, N-E-T-S-U-I-E dot com slash ionai, E-Y-O-N-A-I all run together to get the CFO's guide to AI and machine learning. The guide is free to you at netsuite.com slash ionai.

2:20Great. So, Avatar, it's great to have you. Can you start by introducing yourself to listeners and tell us a little bit about Timescale, and then we'll get into some of Timescale's AI offerings? Fantastic. Thanks so much, Craig, for having me on. My name is Aftar. I'm the head of AI at Timescale. Timescale is really a Postgres database company. So the company started off in the time series space. It actually started before we got into databases. It was an IoT platform. And then in building that IoT platform, we noticed how difficult it is to build applications that require like time series and real-time analytics.

3:03And we couldn't find a suitable database. And so the team at the time decided, you know, why don't we just build our own? And that's where Timescale and TimescaleDB came from. TimescaleDB really extended Postgres, which is a well-known, well-loved relational database platform for time series and analytics without having developers use a separate database. And so that's really where the company came from and where we got our start back in 2015 or so. But since then, the company has really evolved outside time series to really extending Postgres for all these different use cases beyond relational data.

3:44And most recently in the team that I work on, we focus on AI and vector data and helping developers build these AI systems that every company seems to be working on these days, things like Rack, things like Semantic Search, and helping them build those just using Postgres as their vector database and as their database to power those applications. So that's a bit about the company. To summarize, we're really a Postgres database company. We have a lot of open source projects as well. And we also have a managed cloud, which is our primary commercial offering. I think open source is really at the heart of the company.

4:22And that's actually a thread through a lot of the work that we're doing in AI as well. But yeah, that's an overview of timescale and how we got into the AI space. Yeah. And just for listeners, Postgres was an open source project that came out of Berkeley, I think is that right? They had a project called Ingress, and then Postgres is the follow-on to Ingress. Can you just give listeners who are not database people a little thumbnail on Postgres SQL? Yeah, Postgres, it's a database that's been around for more than 30 years at this point. And it was seen as, I think, kind of the defining characteristics is it's open source.

5:16So it was seen as a great alternative to kind of more proprietary systems of those days. And I think what has really helped Postgres succeed is the reliability and robustness. I think having been around for 30 years, all the problems about the system have been known and most of them have been fixed. And I think another defining characteristic of the Postgres ecosystem and Postgres database is extensibility. So it has this notion of extensions where even if you're not part of the core Postgres team or certain functionality that's not part of core database functionality, independent developers and companies can build these things called extensions, which essentially hook into the main Postgres planner and Postgres database.

6:06and that allows you to do all kinds of things beyond just kind of transactional workloads that we're used to for SQL databases. There's extensions for geospatial data called PostGIS. There's extensions, as I mentioned, the ones that we've built at Timescale for time series analytics. There's PG vector for vector data handling. There's even extensions for document data handling. There's extensions for kind of everything under the sun. And so that's kind of one of the defining characteristics because that combination of this really solid open source core plus the different extensions in the ecosystem, that makes it attractive for a lot of developers out there and makes it a really compelling choice.

6:49I think in the past two years in the Stack Overflow survey, Postgres was the most loved and most used database among professional developers. And so that's a testament to kind of how popular it is and just what a great piece of technology it is. Yeah. But the extensions that Timescale has built are proprietary. Is that right? Or are they open source? Yeah. So it's actually not proprietary. So let me just take you through the list. So the extension that we started off with around time series, that's called TimescaleDB. That's actually, it has, there's two versions of it. One is open source under the Apache 2 license.

7:32And so that's completely open source. And there's another version that is under what's called the timescale license, which has some basically the complete feature set. And that is actually free to use. And it's not technically open source, but it's free to use for everyone, except if you're going to offer it as a service. So that's just some protections. You know, we see certain cloud providers. There's been a lot of debate in the open source community about, you know, what is the correct way to monetize. And so for that particular extension, we've chosen to have this really freely, it's quite a permissive license for the everyday developer.

8:12But unless you're kind of like a big cloud provider, that's really only when you're going to run into these kinds of constraints. And then for the AI extensions that we built, we built PG Vector Scale, which helps speed up and extend Postgres for these kinds of high-performance vector workloads, think like Boolean scale search. And then PGAI, which really is bringing AI workloads to the database. Those two are actually open source under the Postgres license. And so that's completely free to use. People can use it on any database of their choice. and it's an extremely permissive license. And so embracing open source is really at the core of our company.

8:53And we think that, you know, that's a defining characteristic of the Postgres ecosystem as well as like what's made us successful thus far. Yeah, although I have to ask, how then do you guys make money if everything is open source or free? Yeah, that's a good question. We get that all the time. The way that we do it, I think, is the primary way, as I mentioned, before we have a managed product. It's called Timescale Cloud. It's basically a database as a service or a database platform as a service. And so development teams who would rather not take care of things like managing a database, dealing with backups, dealing with replicas, and essentially all the hassle of what it means to do database maintenance and build an application in the modern era.

9:41We provide that product where we take care of a lot of that and taking that off developers' plates. So that's the primary way. And we think this balance of having open source products where any developer, no matter the budget, no matter your constraints, you can actually benefit from the software that we put out with also having a really opinionated, really compelling cloud and managed experience. That's worked well for us so far. We have thousands of customers across both kind of really innovative startups as well as folks in industry and like Fortune 500. And so that combination has really worked well for us and something we're going to be sticking with going forward.

10:22Yeah. And the cloud offering, does it allow people to build an application? And then if they want to migrate it to their own cloud or their own on-prem or whatever it is, they can do it. I mean, does it serve as kind of a sandbox as well? Yeah. So I think most of the time what we see is that developers are looking for a production, something to solve their use case where their application is moving into production. where then you need to take into account of things like high availability, you need to do read replicas, you need to have your backup and you need to have a backup strategy in place.

11:10You need to deal with like automatic failover and stuff like that. That's really where the cloud product shines. And I think just to be transparent, it's all just Postgres. And so what we pride ourselves on is to say, hey, if you're worried about vendor lock-in, you're not going to face that because it's just Postgres. You can take it anywhere. You can self-host it. You can move it to another cloud provider. But what we find in practice is that the kind of ease of use functionality and the functionality for production applications that I've talked about is compelling enough combined with, you know, we obviously have support and a really kind of really strong solutions architecture front that many customers are, And some customers might say, hey, we're just going to use this temporarily and they end up staying because it's just easier for them and it's less things for them to worry about.

12:04Yeah. Before we talk about the AI products and particularly the newest offering, can you take us back to the IoT time series issue that you were solving in the beginning? So you've got an IoT advice and it's producing a stream of data. And I'm imagining the issue is how do you get that into a structured database? Yeah, so I actually wasn't at the company at the time, but I'll do my best to kind of recount the story as I know it. So basically what was happening is that the team at the time was building, I think it was called IOBeam. It's basically an IoT platform for managing a whole bunch of sensors and devices.

12:54And so this is actually a use case we see a lot today where, let's say you have a smart factory. This is very common in manufacturing. Or if you have like a fleet of EVs for like logistics or in the automotive space. And you want to know essentially the latest about what's going on across your entire fleet of sensors. You want to encounter, you know, the latest status of it. you want to see if there's updates, if you want to see if there's problems. This is kind of like the platform that the team was building at the time. And in building that, they needed a way to scalably store and query time series data.

13:33And there's a couple of defining characteristics of this time series and analytics data. I think the first one is that it's extremely frequent in the sense that these sensors send readings every second, sometimes multiple times a second. And so this data piles up a lot. And so you're dealing with a large volume of data when you're ingesting. And then also when you're querying, you often are querying a lot of data, but you have these very specific query patterns where you either want to know, okay, for a bunch of devices in my fleet, just give me everything in the past five minutes, just as like a current kind of like status updates, these kind of like real-time updates, or for a few, like let's say you're looking at a specific location or you're looking at a specific factory ID, for the segment of devices, I want to go deep and I want to explore, you know, the monthly performance because I'm doing kind of historical analysis for a report or something like that.

14:33And finding a database that could keep up with those requirements of like you really need perform an ingest, you really need fast time-based queries. There's also these aggregations that happen where you want to know, okay, what is my monthly average utilization? What is kind of the seven-day average or the seven-day maximum of some particular metric? It turns out that that was really difficult to service on the existing time series databases out there. And so what the team did was ended up extending Postgres, which they had existing expertise with, existing familiarity with, for time series. And this is a story we hear a lot from a lot of current timescale customers where they start off with a certain specialized database, they realize like either the performance characteristics are not good, or it's just a pain to handle and maintain.

15:27and they look for the product that's familiar to them that their team might already have existing expertise on, which is Postgres. And funnily enough, we're seeing kind of a similar story play out in the AI space where developers are, you know, starting off with these different vector databases, but as they move into production, they're looking to consolidate, they're looking to go to something that's familiar with them. And that's where Postgres once again has a role to play. Yeah. And on time series, I immediately think of the financial markets. Can you, does timescale handle that kind of volume and speed?

16:17Do you have clients in the financial stock trading business? business. Yes. So in time series, finance and most recently crypto, which is just really a different version of traditional finance, is a huge industry for us. As I mentioned, in other in the IoT space, there's like manufacturing, there's also automotive, electric vehicles, logistics. But for finance, I think like that's, you know, we power some of the biggest crypto exchanges out there. I'm just trying to remember what I can actually disclose publicly, but there's a lot of, I think, you know, time series is so synonymous with finance.

17:02And so that's really where we see a lot of usage and especially for being part of data infrastructure that can support applications being built. That is a huge industry for us. Oh, okay. So then AI comes along, generative AI, and you have extended your offerings to include that. my understanding is uh i'm not going to be able to remember all the all the names of the products but there's postgres vectorize which in effect uh vectorizes uh data in the database uh so that people using rag for example don't have to have a separate vector database it can all be done within that database and then uh uh you you i'll let you go through them but the the one that kind of fascinated me i i think it's pgai uh where you actually have uh from what i understand an llm built into the database.

18:22Is that right? So that you can, when you query the database, it can produce a natural language response directly. But anyway, why don't you back up and start talking about when AI became a focus and then give us a chronology of the products that came out of that. And then the most recent one, I think it was just announced. And I'd have to look at my notes, but I'll let you explain it. Fantastic. Yeah, I think you hit on some of the big ideas. Maybe I can give a bit more kind of structure to what you introduced. So I think to set the stage, what we saw is that when, I think it all started with ChatGPT coming out in late 2022.

19:19The world was really taken by storm with what was possible with LLMs. And I don't need to make this argument. I think it's kind of, if you're listening to this podcast, you kind of understand the potential of AI and, you know, all the different cool things they can do. What people immediately wanted to do back then was build what I would call these chat with your data applications. It's basically, you know, having a chatty bt style interface, but with your own data. Now we know this as RAG or retrieval augmented generation. Now the question is, if you're trying to build these RAG or even semantic search, or most recently AI agents is kind of the main thing that folks are experimenting with.

20:00You need a place to store these knowledge bases that you want to augment the LLM's kind of training data against. Some implementations involve using search APIs to augment the LLM. In other cases, where you're dealing with private documents, if you want to really ground the LLM in certain documents for it to respond with knowledge of, using a knowledge base is the main approach there. That brought this question of, okay, I know what I want to do. I need to have a knowledge base for my RAG application, how can I easily build this? And so this is where we face a similar, you know, once again, I think it's kind of like a rhyming of history here where developers were looking at what choices of data infrastructure they have to build these applications.

20:50And they ended up with either to use specialized vector databases, which are really built for vector search, but they come with the additional headache of having to essentially add them into your existing infrastructure, which is fine in the beginning if you're just like playing around. But when you're building a production application, you have to deal with all the things that I talked about earlier of like maintenance and management and backups and high availability and stuff like that. And that was really difficult. And then on the other hand, you had databases like Postgres who were adding on vector support via extensions or via kind of first class support in databases like MongoDB and MySQL most recently.

21:35And what actually played out is that it turns out that Postgres was very popular. There's an extension that's open source called PG Vector that really took off in popularity. And we initially thought that, hey, if we just support PG Vector, this is great. This should solve all our problems. It turns out that that wasn't the case. PG Vector is great by itself, and it provides a really robust foundation for vector handling and vector search in Postgres. But what we started to hear from the community and as different developers started exploring what's possible with these AI applications, a common theme of scale being an issue came up again and again.

22:13And basically the problem was that, hey, as you scale past a certain number of vectors in your knowledge base, you get really slow queries. And to put it kind of in easily understandable terms, what this basically means is that as you add more documents or as you add more resources that you want the AI to talk about, you end up with this performance bottleneck, which is not great because ideally you don't want to have a limit of like, hey, I can only have, you know, 1000 documents or something like that. If you're building a, let's say, an application that services like a multinational company, you're going to hit that limit pretty quickly.

22:47And so what we decided to do was to try and solve this scale problem. And that's where we built this extension called PG Vector Scale, which takes some of the latest research in approximate nearest neighbor search, which is kind of the academic literature around vector search and vector databases and implement it in Postgres. And so what we did is we released this PG vector scale extension earlier this year in June. And what it does is it brings some really advanced indexing to Postgres and also some advanced compression methods. It's called statistical binary quantization. And that's been a huge, like really well received by the community.

23:27And basically what that does is work hand in hand with PG vector in order to give Postgres these really high performance, high scale capabilities and removing that kind of performance ceiling that was holding developers back from like building these applications. Yeah. And just, I misspoke earlier, PG Vector is a third party extension. It's not a time scales extension. Is that right? Correct. Yeah. It's an open source extension. It's not something that we've built at our company. Right. But PG Vector Scale is your extension that works together with PG vector. Yeah. And PG vector, does that vectorize the database?

24:14Because you have a tool for the vectorization, right? Yes. Yeah. So this is actually gets into the second set of problems, which is, you know, once you've solved vector search, which is about the actual like, hey, find me similar documents or find me kind of the nearest neighbors in a graph, you get into all the other issues around building an AI application, which gets to actually really tough engineering challenges. And so that's where the rest of the PGAI project comes in, which is a PGAI extension and the newest product that we just released last week, which is called PGAI Vectorizer. So the problem that these two tools solve is the problem of managing embeddings.

25:00So all these vector databases are simply a component in a stack or in an architecture where they expect vectors to be written in and then they can do search, then they output vectors which are then used in your application however you want. Now, what we saw is that developers were having a really tough time, you know, once you go beyond a POC to not just create embeddings, but to keep them in sync with underlying data. Now imagine an easy way to see why this is important is to imagine a, again, you have a company chatbot that is going to power and it's going to help your employees get the latest information about products that the company offers.

25:40And let's say you make a big update to the product where a sudden restriction has been removed or the product feature functions in a new way. You're going to want to change your documents in the knowledge base, and you're going to want your vectors to be updated as a result. But what happens with most vector databases today is that the vector database only handles kind of the vector embedding storage, and that it's on the development team and on the engineering team to actually go and update the documents and separately update the embeddings in the vector database. So there's all this synchronization, all this kind of engineering that needs to happen in order to keep your vector embeddings in sync.

26:27And that's really what PGAI Vectorizer does. It both creates embeddings, but it also ensures that as underlying data changes, whether it's additions, deletes, or even changes to the actual source data, your vectors will automatically be updated and it will reflect the latest state of the source data so that it prevents stale data in your applications. And this is not just important to keep your applications up to date, but it actually enables a whole bunch of easier experimentation and testing. And if you think about like in an AI application, the way to make it better is by testing and trying out different, whether it's new models, whether it's new kind of chunking formats, etc.

27:14because that's how you find kind of what works specifically for your application. And so what PGAI Vectorizer does is it makes it easy to create copies of your data with different embedding models, with different formatting, with different chunking specifications, with just a single SQL query. So what used to take engineering teams to say, hey, we want to test this model. Let me take a week or two to go and like set this up. Let me make sure we run all our documents in pipeline. Let me make sure that we, you know, all the updates that we made in order to get like a fair test. Let me reflect that.

27:48Now it's just a single SQL query and the system will take care of creating these documents and creating these vector embeddings for testing and for experimentation. So the whole suite, essentially, if I had to put these, all these things together, is making Postgres a better system and a better database for AI applications. and you have the performance and search component, which is PG vector and PG vector scale. You have the embedding management and embedding creation, which is PGAI vectorizer. And then as you mentioned, the PGAI extension has some of these functionality where you can actually call LLMs from the database.

28:28And that's useful. It might not be useful in every single application, but if you want to do things like in database transformation of data, where let's say you have, as your data is coming in, you want to do moderation or you want to do summarization and you don't want to do it in your application code and then write it into the database, you can actually do that within the database. And we interact with third-party models like OpenAI, like Coher, like Anthropic, and also open source models via OLAMA in order to make that happen. So this is really a cohesive set of tools for developers that are building AI applications so that they can use the Postgres that they know and love.

29:06Maybe they deploy it to their companies without having to spend time learning a new system and also spend time like maintaining and figuring out how to make these multiple databases work and play nice with each other. Yeah, I have a couple of questions here. One, I just want to say that I tend, because I talk to so many people about similar things, I forget that a lot of the listeners are not experts. So anyone listening, vectorization or a vector database is converting data into vectors, which are strings of numbers that are more easily searched. Is that a fair enough way to say that? Yeah, I think that hits on definitely the big ideas.

30:02I think one way that I found it easy to understand is what a vector is, it's essentially a compressed representation of some sort of data. So you might have text data, you might have image data, you might have a combination of text and images. and what it is is that it takes, let's say, a sentence or it takes a document and it compresses it into exactly what you said, a list of numbers, an area of numbers. And essentially what that is, it's a representation in, I guess, it's called high dimensional space. But when you have a lot of these vectors, let's say you have a thousand documents and you vectorize all of them, you can begin to see similarities and you can begin to see connections between them that it may be difficult to find by just looking at the actual source or the vectors itself.