Inside Google’s Database Infrastructure for the AI Era

26:21And because of these atomic clocks, we are able to keep that delta very low, which makes Panner run like a champ inside Google. Now, the question was, what do we do when we go to other environments? A few things I think that helped. One, the fact that we talked about true time and other things have meant various other systems have done a better job at providing better time primitives. And so the state of the world in 2025, 2026 is very different from the state of the world 15 years back, in no small part because of what we did with Spanner and what we helped teach the world. And so there are better time primitives that are there.

27:01Still not as good as our hardware clocks in our data centers, which we still think are the best, but they are probably actually not meaningfully that worse than Spanner, than our world in 2012. I talked to you about ads. 2008, 2009 was when we had our, you know, come to Jesus moment, right? 2012 is when Spanner went production inside Google for the ad system. And the world of what you can get with time outside Google is certainly not as good as what we have in our data centers. But it's ballpark how it was in our data centers 15 years back. So the world outside is kind of like the world inside Google in 2012.

27:44And Spanner was built for that time. So we built something called Software Through Time that we have available that you can run in other cloud providers and other legacy cloud providers and data centers. And with Software Through Time and with Spanner, we're able to provide still a very quality experience. There will be aspects of it which will not be as good as if you run inside our infrastructure. But we believe that for many, many use cases, it will do just fine. That's the secret sauce. That makes a lot of sense. So basically what I'm hearing is, so especially with TrueTime, the level of accuracy needed to scale at Google levels.

28:24You need that level of accuracy. but when you're at lower levels of scale, which that's not undercutting what some of these major customers probably need, but it's also not Google scale. It's not Google scale. And in many situations, it depends on the actual workload. You can accept a little bit of that extra. Maybe it's a slight different latency implication or something like that. In many, many situations, it's just fine. And by the way, I think other interesting things have happened. There have been customers who come and talk to us and said, hey, what if we install the same kind of atomic clocks that you have in your data centers?

29:01What if we do it in our data centers? And my answer is, let's talk. I'm very happy to support that. I think it's one of those things. In many ways, how data centers are built today are because of the work my colleagues have done over the last 20, 25 years. And so each of those lessons from commodity infrastructure, from things like atomic clocks, I think they will make their way to the wider ecosystem. And as a labs researcher, leave aside my role as a Google employee, I think that's a good thing for the industry at large. Yeah, that's fascinating. If you can share, the idea of that single binary fascinates me, first of all, because I love the idea of a single binary.

29:39I don't need to install anything else. It just runs. What is that written in? Is it some sort of, is it C++? A lot of C++ code, yes. A lot of it is primarily C++ code. All right. Cool. All right. Good old-fashioned C++. Plus, maybe AI will help us write it all in Rust or something. I've been talking a lot about Rust lately with some clients. So graph algorithms natively just shipped in Spanner as well. And I'm having a little bit of a hard time wrapping my mind around, why would I need that in a transactional database? So can you talk about a real use case and where this clicked for a customer and why this is really valuable?

30:19It's a great question. It really gets to the heart of why running these graph algorithms natively in an operational database like Spanner is so amazing, right? It's a real game changer. First, what we did with graphs let you natively model the data as graphs without moving the data out. You could have your data in a relational schema. You can build an overlay on top of it. You have a DDL command that says create property graph, and you don't actually move the data. You just start querying it like a graph. It's amazing. It's really, really, really powerful. It feels like a graph view. Essentially, that's right.

30:55Now, you can also come and say, I don't want the relational view. I just want a pure graph interface. We support that too. What actually happens when you create a graph view? Effectively, in a way, you're modeling relationships that are there. And then once you're viewing it with the graph prism, maybe you ask questions differently. You ask questions around traversing the system. By the way, it's still declarative. The database dogma is not lost. You still ask, find me all the nodes, which are two hops or three hops or four hops. And so where this becomes powerful is in any place where there are lots and lots of these relationships.

31:27And there are many classic use cases, of course. Just talking about graphs, and we'll talk about algorithms in a second. Some of them, infrastructure, right? You have ports and network switches and things like that, and you want to connect them. But a lot of use cases often come in the security world. You might have users and relationships between them and accounts, and you want to look at the trust graph, if you will. Customer 360 is something that is very big. That's a very big graph use case. You have a lot of information about customers. You have business questions that you're trying to ask, but customers are similar in many ways.

32:02Maybe they share certain purchasing profiles. Maybe they have interconnections in social media that you're able to find out. Another very important case is around fraud detection, where you were trying to understand relationships between users and accounts and things like that. But tipping aside, I think the biggest interesting thing is you can ask more complex questions on the data in your transactional database without running a messy ETL pipeline. Look, we can all run pipelines to move data and ask questions somewhere else. But do you want to? Do you need to? In large part, what we're trying to do with Spanner with the multi-model approach is to be able to ask more complex questions in your data in situ without having to move it around.

32:47And graphs are all about relationships. So the first version of what we did with graphs was made it easy, like I said, to define these graph views as you put it or a graph overlay and start asking questions, make these things perform like a beast. But then what we started seeing was again, a little bit of inspiration from what we're doing inside Google. So inside Google, we've been building these graph algorithms for decades for solving many complex problems in the web space. But now we found they're all actually very transferable to problems that customers face when they think about graphs. So a classic question, financial fraud detection, right?

33:23It really clicks for many customers. Sophisticated fraud like money laundering or synthetic identity fraud often involves networks of colluding accounts. And the problem is that these fraudsters create these complex webs of accounts and transactions to try to obscure their activities. And they're always moving. It's an arms race, right? It's a cat and mouse game. As your systems become more and more complex, they're changing things, they're doing different things, which, by the way, is one of the challenges. If I have to keep moving the data of my operational system to another system and run it, scans are moving very, very fast.

33:55So analyzing individual transactions in isolation often misses the bigger picture of coordinated fraud. So how Spanner Graph helps is that banks and payment processors can store customers and transactions in Spanner. They can model this graph. This data is a graph, like I said, as an overlay. There are accounts are nodes and transactions are edges. And you have very interesting community detection algorithms, like weakly connected components, modularity clustering that you can now run natively in Spanner because I have this big monstrous graph. I want to keep finding out what are those sets of related people, right?

34:33What are these communities? What are the clustering things that you're trying to do? And once you detect these suspicious clusters, some of them may be suspicious, some of them may be not, you can now you want to zero in on who are the most influential people within that suspected cluster now you can squint at this problem and say what is this like actually it's just like page rank in fact it is page rank you can run page rank on the nodes within that cluster and so although page rank was originally for web pages it's actually really very very good at identifying who are the most influential nodes or the central nodes in the network so not only can you figure out here's the colluding set, you can now zero in on your actual problem space.

35:14So I think this is kind of a great, great example of how you could actually run these things. And it's really valuable to be able to do this in situ without having to move this to some other system. So for example, I think we have a customer, DanaPay, it's an Indonesian digital payment solution. And so they're using Spanner Graph for their anti-money laundering use case. And by running these algorithms directly within Spanner, they can do a whole bunch of things. They can detect suspicious networks much faster, again, without having to go through an ETL cycle. They can identify key players within those networks.

35:49They can potentially flag or block transactions. And this is where the real time comes in. Sure, you can go run this in an analytic system. But if I can flag and block the transaction, I can prevent financial losses and really, you know, avoiding these unnecessary operational overhead. So maybe in general, the average transactional database doesn't need page rank, but what you really need is fast, actionable intelligence on relationships within your fresh operational database. And native graphs and Spanner deliver exactly that. I can see the value now. It's It's a proximity thing. It's built in right into the data.

36:27So that makes a lot of sense. And I would imagine even some places not at scale that can build from the ground up can take advantage of that and not have to worry about ETL and things like that. Right, right. And yeah, it just makes it easier. I think SoundCloud, for instance, they were running graph algorithms and batch more for years. I mean, a lot of times people may have been thinking, well, that's the only way to do it. And I think one of the things I told you earlier about how impressed I am, how inspiring it is to work with enterprise IT and developers who are so smart. I think as you provide more things, we get surprised.

37:01They'll find things that we didn't even anticipate. I think it's fascinating. Something that I'm seeing, like a pattern coming up is like, you guys dog food this amazing technology with this amazing scale problem that Google has. You productize it. And then it's out in the marketplace. And there's also more creative things that happen. And it's this, I don't know, it seems like a nice circle. Indeed. So let's zoom out a little bit. For a long time, the first hard decision in any project was which database? How do I structure my data? So now we have LLMs, which are happy to pretty much gobble up any data that you might want to throw at it, structured or unstructured.

37:42And we have agents writing their own queries. And so what actually has changed in the last few years about how you should structure data? And is schema design still relevant? Is it not relevant? I think that that's a little bit of a rhetorical question. But I think what still remains important about schema design? By the way, it just occurred to me, there's, I think, a famous Dilbert cartoon strip, which says the first thing you do is you design the database. I don't even know what I'm doing. I'm going to design the database. So I thought about that. I think I remember that. Yes. Because it's a very interesting question you're asking.

38:16And I think, you know, just to be very candid, I don't think we know everything that is going to change. It's changing very fast with AI. Having said that, there is an enormous amount of applications, bespoke applications of the self-applications that I don't think will disappear overnight. These applications still exist. Now, I think there's an urgent need for them to be more easily used in authentic applications. So even before you think about schema change and what needs to happen, I think you have perhaps a more urgent pressing problem, which is how do you activate and light up the existing database for these systems?

38:54And so in our vision of the agentic data cloud, we think there are three important pieces. One is the AI native infrastructure, and I alluded to you before. Things like graph, by the way, are very relevant. Vector search, full-text search, All of these things are very important, as you imagine, the world of structured and unstructured data coming in together. So that's the first. I think the second thing we also talked about, we think in this world of agentic AI, a lot of gaps, friction needs to be eliminated. And that's a big part around maybe a borderless world. You have these lake houses, which are spanning clouds.

39:31You have things like Spanner Omni. We have AlloyDB Omni for Postgres applications. So I think that's kind of the second thing. And the third thing, perhaps starting to get very relevant to what you're asking, is all about trust. And trust really means in order to light up these databases, you need to have the context. You need to know what's in these systems, what's in the data. And you talked about schema design. Yes, it used to be, you know, you have to design schemas. But it turns out for agentic applications, just the schema itself is actually not even enough. Forget getting to how a schema changes.

40:04Back in the day when I first worked on DB2, almost 30 years back, I was wondering, why does the SQL standard let you say, add a comment? I don't know if you know, when you say an object create table, you can add a comment to it. It seemed like a... Extraneous. Theoretical thing or something. Yeah. Now that kind of stuff is actually interesting because it now tells you what could be used to record, what is the meaning of something. But more realistically, I don't think we're going to depend on a comment in a schema. You're going to try to gather information about the schema. So while the structure of the schema is an interesting question we should talk about, I think perhaps the more interesting question is where's the context around the schema?

40:44I'll give you a very simple example. If I have a table with a billing address and a shipping address, it's two columns. If one of them is null, there is some implicit knowledge that these two are the same. That's not there in the schema. You need to extract that information. That's the critical context. So for LLMs to activate your operational databases, and I think before you even think about new greenfield applications, this is the pressing need to light up those databases. You need to have the metadata on, more than the metadata, the metadata and the context about your data in order to have agents go ask questions.

41:22And it's not just about things like that. If I have a column called city and I have an agent which is going to use an LLM to generate a SQL query to query the database, somehow something needs to tell the agent that the values in the city column, are they in lowercase? Are they three-letter airport codes? So again, this information about the data is super important. Now, I think so one stream is, yes, schema and context. Schema is important. What's more important is context, right? I think that's the first piece. I think there's a second piece, which is also interesting, which is, is there a world where you have citizen developers just generating apps?

42:03And I was reading some stat about how the number of applications on number of repositories in GitHub is exploding. We have a partnership with Google DeepMind with AI Studio, where you can go and say, when I'm building an application in AI Studio, you can say, I want this to be a persistent application. And you can choose either Cloud SQL to store the data in relational format or Firestore to store the data in document format. More practically, you ask the AI, the AI will decide. We are just actors on the stage, I guess. So in things like that, there's an interesting question. How is the AI going to define schemas?

42:39Well, the truth is AI is going to define schemas the way humans have been designing them because that's what streamed these AIs. So I actually think even when you look at that, AI models will tend to pick this kind of schema structure that made sense. Now, I think it's very valid to ask, is that the right one in the world of AI? Someone is telling me about a situation where they realize they don't care about abstractions of code as much as they used to, because they're like, yeah, sure, I'll just have AI go copy it and make changes. I don't know. I mean, surely that's a path towards more chaos.

43:13I'm not entirely sold that the general rules for defining and structuring schemas is going to be fundamentally different if AI is generating the code or humans are generating the code. But I think there's a third piece. The third piece, in my view, is there is a world here with new kinds of applications where the question of what is an application changes. Today, an application has what I like to call four walls. It has a boundary, an API surface area. I think we're going to see a new world where applications are ephemeral workflows. The application as such doesn't exist. Maybe today, one morning, a knowledge worker comes into work and says, you know what?

43:49I have system A and I have system B. I want to create some more complex purchase order that combines them. Well, what would they have done in the past? Well, maybe they had done a contract with a large system integrator, paid a lot of money. We have got some results after six months, whatever it is. But now in this new world, they're just going to ask the AI to go do something. Now, where is that information that connects all these things stored. It has to be stored somewhere. So you need to store these trajectories on the data, on these agent interactions somewhere. But systems also need to be more flexible.

44:23For instance, if you travel internationally, you're probably well-advised to call your bank and put a fraud alert so that your credit card keeps working. Well, what if you decide to chuck your job or go and spend a week in each country in Southeast Asia and come back after three months? And so in the AI world, you might call your bank and say, I want a fraud alert. I want one week in Vietnam, next week in Laos, the next week in Cambodia. Well, the bank system is probably going to choke because people who designed the system didn't think about the world where you have setting 20 different fraud alerts.

44:54But I think in a new world, people are going to expect that. Expectations are different. So I think, you know, one of the interesting questions is, should the schemas itself be more forward-looking, to be more flexible? I think maybe this is really the spirit of your question. Sorry, I came to it in a slightly long-winded fashion. No, it's actually not what I was angling at, but I think you're getting at something much more interesting. So please keep going. But then there's also this interesting question that when we look at think about agents on data, we're all being very conservative right now.

45:24We are trying to say, I want to be doing primary reads because everyone's generally scared, right? No one wants to have the Bobby tables thing going and deleting records from a database. And we've seen the horror stories already. Bad things happen if you don't think about security and trust really well. So in general, I think writes are going to be a complicated, challenging space because you're now going to try to think about how agentic applications or these ephemeral applications are going to do writes. How are they going to do schema changes? It's not fully clear to me what you need. For instance, even writes.

45:58Maybe here's a proposal that when you're trying to make a change and modification to the database, you don't actually make the change. You make a proposal for a change. It's a proposed change, and it doesn't actually make it right there. It may happen through some set of steps. So I think the entire way in which data has changed and schema has changed, I think it's all subject to revision. Applications as we know it, I think will change. But I think the existing applications are not going to disappear overnight. Business logic, which applications may be reviled, but business logic lives in applications.

46:33And so we have to activate them. Yeah, the data has to, you said the key thing, the data has to stay somewhere. And I think I've talked to a lot of colleagues and other people, and I tend to agree with this. And maybe I'm the one that's behind, but I'm certainly willing to let agents do a lot of my application code. But where I kind of draw the line is at the data model. That's right. Okay. You could take a cut at it, but I'm going to review the heck out of that. And I'm going to make sure it makes sense to me and that I can pressure test it based on my own experience and where I think things are going to go and creative thinking and things like that.

47:10So it sounds like it's very much in line with that. All right. We already said this. I think we're both Postgres nerds, which is great. It seems like every AI startup stack is Postgres plus PG vector. And so Google has alloy DB, which is basically like a souped up Postgres. What breaks when people try to do serious vector search on vanilla Postgres? And what did you have to change under the hood on alloy DB to fix it? There's, I think, a few different parts to this. One, PG vector is awesome, as you put it, right? I think it makes it very easy to use. And in fact, you know, it's a whole side thing about the Postgres ecosystem and extensions, which let the experimentation go.

47:50PG vector took off. And so for many, many use cases, PG vector with the HNSW algorithm works really well. Now, at Google, we've been working on vector search for, I guess, over 15 years now. And it's at the heart of a lot of things that we do like YouTube and search and so on. And so we have this amazing technology called SCAM. And it scales to over 10 billion vectors delivers very much more performance in terms of latency than just HNSW. And one of the things we did was we used Scan, which is a tree-based algorithm, and made that available inside AlloyDB. And we tried to make it look and feel like PGVector.

48:38We also offered the Venue Lab PGVector so that the applications shouldn't have to change. And applications that work against PG vector will also work against scan. So that's the key part. Now, there's a lot of challenging work in making that happen inside Postgres. And also various things that we've done on the side. How do you go build these indexes? How do you go maintain them and update them? And there are many, many reasons how we run AlloyDB with scan to really get faster performance. So in our tests, we can get 6x faster vector queries than HNSW, PG vector, and standard Postgres. We use 4x less memory, which anyone who's run a database knows it's kind of always a good thing because if you use less memory, you can keep more for your buffer cache, which makes your performance better.

49:28But we also support things like distributed index builds. And so in general, for many, many use cases, for the most demanding use cases, we have a big advantage with scan. We also have support HNSW and have taken some of our improvements in Scan and also made it available in HNSW. And the reason we've done that is... You've contributed it back to the libraries, or is that a product? I don't know what we've contributed back, but we certainly have made it. We are very happy to contribute this to HNSW. I don't want to say something that's inaccurate. I'm not entirely sure if you've contributed or not.

50:06Totally fair. The interesting observation from us here is Vector search feels a little bit like what we'd call an ensemble problem. There are different domains where there are different systems that are better. There will be probably some kinds of systems where some kind of regimes where HNSW is fine and maybe it's a simpler approach. But what we tend to see is in the most demanding workloads, when you have high scale, you really need something like scan. Well, we see value in both. And so we want to make sure both really work well. But I think there's a whole set of things that is more than just vector search.

50:42I think a lot of people were very focused on, hey, I want to get vector search. As you said, I have the simple stack I get going. What we realized is what gets more interesting when people go beyond vector search? Rarely do I see people asking queries which are only a vector search system. They are more typically asking queries which involve multiple predicates. It might be vector search and another predicate on my regular table, just a relation predicate. Classic example, I have a catalog and I have price. Price is a value that is in my table and I want to ask a question which combines the two.

51:22We call this filtered vector search. And in filtered vector search, my query may have two predicates, a semantic search predicate, which is on the vector index, and a predicate on price, as an example. Now, how do you know which one to probe first? This is a classic problem in dealing with vector search. If my problem space is simple and I only have to go probe a vector, then sure, I go check the vector index and get the answers. But in practice, I think what we see is this challenge that people want to ask more complicated questions. If you go probe the vector index first, maybe you get 20 results, and then you apply the rest of your other predicates and none of them may be satisfied with the other predicates because I say price less than$50, I get nothing, right?

52:08And then I go probe my vector index again and say, well, get me the next 20. And then I go check those next 20 and I still don't satisfy. I have to keep going down this path, right? So based on the selectivity, maybe I'm better off first probing the non-vector index and finding the set of results and then applying my nearest neighbor algorithm. without even going and probing the vector index. But again, I don't know that. Suppose I probe the non-vector index, suppose I do pre-filtering, what I just described was post-filtering. But now suppose I instead do pre-filtering and I probe the non-vector index, maybe I get a million records and then I can't go and run nearest neighbor on a million records.

52:45I have to go, that would be a bad idea. That's why you built the vector search. And so no one else really managed to address this problem. In AlloyDB, we built this really amazing filtered vector search system where we are able to probe multiple indexes on the fly and figure out which is the right order in which to probe things. Interesting. How does it do that? I think this is an interesting point. One of the key techniques here is when you have multiple indexes and you have a query and they're all in the same system, we can actually probe both of them at the same time. And we try to figure out quickly which is the right strategy and change it on the fly.

53:22So it's a divide and conquer. Yeah, it's an adaptive technique where we are trying a couple of things. It's not quite divide and conquer. You're trying a couple of things and trying to figure out which is a better fit. And we are adapting the right approach based on what your actual situation is. Okay. Right. It's an interesting piece, which is worth thinking about, because you described one stack which startups use. There's another complete stack, which is people say, oh, I'll go run a custom separate vector index, not in post-res. And that's also a path that people go. If you are doing a pure play vector search on just a corpus of text data, whatever that isn't changing, maybe that's a reasonable approach.

54:02But I think, again, a few milliseconds after people come up with this, queries become more complicated. As I said, developers are insanely inventive and creative people. They will find ways to ask more complicated questions right away. Or AI will do it for you, right? Users are very creative, too. They do, right? So people will want to ask questions that straddle these worlds. It comes back to what I said. What has changed in a big way? People don't want to just look at unstructured data. People want to look at structured and unstructured data together. This all comes together, right? So when you're asking this, I actually think it's a really poor idea to go build a completely separate vector system because then you don't have a choice.

54:42You have to stitch these systems together at the application level, right? And that's incredibly painful because who's going to do this, pre-filtering or post-filtering? And unfortunately, the pure play vector systems don't have a good solution here. And so when you're having multiple indexes under the control of a single system, you can do these kinds of very inventive, creative techniques that deliver tremendous value to customers. Very cool. Okay. And in addition to all of that, one of the additional things that we're doing is we're also combining high-speed vector retrieval with full-text search on operational data.

55:16So that gets hybrid search is now getting interesting. So your search is not just vector and non-relational, but also full text. And all of these things are really coming together. Wow. What a world. It's a long way from select where. That's true. So let's move on to MCP servers. So you all just released, I think it's called Toolbox for Databases. MCP Toolbox for Databases. MCP Toolboxes. Yeah. And I thought that was amazing. it's open source, basically MCP server on top of your databases. So you don't have to roll it yourself, which is fantastic. If I'm a dev, I'm going to take the thing that Google created and assume that they did it pretty well.

55:56And you've got those managed MCP servers for Alloy, Cloud SQL, Spanner, et cetera, et cetera. So for, you know, I think most people, most audience members here are engineers on the application level. What are the possibilities here? Like what are your What are teams at Google doing day-to-day with these? So, you know, just to be clear, my teams at Google are building databases and building infrastructure. So they're not necessarily actually using as much. But that is an area where we are. And this is for our user interfaces. So we're doing some really interesting work on something we call GenUI or Generative UI.

56:34And here, if you look at Google Cloud's managed databases, we're a humble application too. We store data in Spanner, the version of Spanner that we use inside. You use your Google Cloud, you create instances and databases, and that information is stored in the metadata system. It used to be that any time you built a new feature, you had to build a lot of complicated UI to help make that feature be available. One of the things that we are now doing is generating UI on the fly. and the way we're able to do that is by going and connecting to the MCP servers because each of these, if you think about the UI, it's a humble application that used the underlying API servers.

57:21And now what we figured out to do is based on what users are asking, we're able to generate UI on the fly by connecting to these MCP servers. So this is an example of something that my teams are doing, right? And we think it's really exciting because it's not just about saving time, it's about creating these artistically crafted user experiences based on what someone wants. We can deliver something that is really awesome for them. So I think that is really interesting. Now, from an actual customer's perspective, I think there's, as you said, there's both you can use toolbox, you can use our managed MCP services.

57:57I think you'll see different kinds of use cases. The best thing about our managed MCP services is that it's got enterprise-grade, it's got governance, it's got all the bells and whistles. And so, you know, a lot of customers will absolutely use it, will gravitate to it. And I think it's great. I think it's also true that there's a set of developers who are on the bleeding edge. They will be needing more. And that's why we started building MCP Toolbox, to really learn what developers need. And they need a whole bunch of things. For example, they need custom authentication mechanisms. They may be running in their own on-premises environment.

58:35They're not using Google Cloud. And so our problem statement when we set out to build MCP Toolbox was, how do you make it easy for developers to connect to databases irrespective of where they are? As I said, part of the agentic data cloud mission is openness. And so we want to solve for people no matter where they are. They can be on their data center and another cloud. We want to make it really easy to, as I said before, activate operational databases in agentic applications. Now, when they are in the Google Cloud environment, we also want it to be a very easy environment. And that's where you have managed MCP services, right?

59:11And so things like the GenUI application that my team has built is a great example of using those managed MCP services. But it could be various other things. I think classic examples on the control planes. I'm a SWE who is relying on a bunch of AI agents and these AI agents need to go create databases, run deploy applications. For all of that, this managed MCP is perfect because it just directly connects to the control plane and so on and so forth. But there are also things like custom tools that we have where custom tools that people often are using in the world of MCP, where they're not just issuing execute SQL, where they may be doing something like search for a flight.

59:57And that might be a custom tool someone might write. And that tool ends up going and runs a specific query, select star from flight stable, things like that. And so for scenarios like that, MCP toolbox, I think is great. I think what you'll see in general is each of these pieces of functionality that we are learning by working with customers in MCP toolbox. If it's valuable, it'll also make it to manage MCP. And so one way of thinking about this is open source. We want to be where people are doing things. We want to learn from our customers, from developers. And based on what we are learning, that influences the roadmap of managed MCP, what are valuable.

1:00:39Very cool. The Gen UI stuff, is that something that's also, is that internal to Google? Is that something that is widely available? I think we have some version of it for observability in our consoles. We already have it available. I don't think it's called that. I might have talked out of turn and used an internal project code name. I don't think we give it a name. It's just part of the user experience in our database console. Yeah. Certain kinds of things we are having. The reason I ask is it's a very interesting. So, you know, we had that conversation earlier about metadata and the context of data, right?

1:01:15Like the MCP server can help expose that and make sense out of it. And to be able to kind of, I guess, on the fly, create a UI is a really, really interesting thing. Powerful. Maybe that could be a follow-up or something like that. But if those two things paired, the average application developer would have quite a pair there to say, like, inspect the data and show something based on what we found. Correct. I think our own observation was, if you looked at our UI infrastructure, we had a bunch of widgets that did various kinds of things. And so what we were pairing up was the managed MCP services for the underlying systems and the ability packaging up those UI widgets also in some kind of MCP tool.

1:02:07Now you have an AI model that's able to say, I want to combine these things and put them together. That's an example. But you're absolutely right. What we did, any customer can do exactly that kind of work. I think the little bit of the trickiness always comes back to, well, what is read? What is write? How ready are you to go and let agents go right on to the operational database? Just to correct myself, I searched there. I had heard of GenUI before and I wasn't sure. So that is an architectural pattern. If you look it up, like I would encourage audience members to look it up. At least I think it's just an audit pattern.

1:02:43Yes. And there's different implementations of it. There's an ecosystem. It's fascinating. And I think paired with that MCP toolbox, it's something very interesting. So, all right. Okay. So an LLM with a live connection to your production database, it's a great horror story waiting to happen. How do you, with all this knowledge of database, how do you think about how to control the blast radius for something like this? We just led into this about talking about being careful with rights. What do you think is non-negotiable before anybody puts an agent towards prod? and like not obvious stuff like lease privilege and things like that, but the stuff that really kind of keeps you up at night and that you're thinking about and that you want people to know to avoid.

1:03:27Maybe some anonymized horror stories from clients or customers or something like that. I'll talk about two or three different things. One of them I think is a particularly interesting challenge when you're trying to, if you think what you want to do is have an LLM or an agent generate a SQL query on the fly and go against your operational database. As a classic, hey, wouldn't that be an obvious win? Here's an interesting challenge. If you're connecting to an analytic system, your persona is maybe a business analyst or a data scientist. The analytic systems, forget AI today. If a business analyst is connecting to your analytic system, they're connecting with their credentials, and there is access control typically built into the analytic system itself.

1:04:16These business analysts also typically have wider access to data. The nature of these kinds of BI queries are aggregate queries. It's a very different world for operational databases. In fact, it didn't used to be that different in the early 90s when you had client server applications, when you had clients which would take clients that would connect to the database. But with the web, with three-tier architecture, maybe late 90s, that changed fundamentally. So for an operational database, the end user's credentials are not actually part of the operational database schema. Databases have users and roles and groups and privileges.

1:04:54But the end user's information is not there. And I think one reason is because with the web, the number of end users went from hundreds to thousands, millions, hundreds of millions. So you can't really manage the data that way. And so with the web, with three-tier applications, you have the application connecting to the database as a service principle and figuring out what an individual logged-in user can actually access using where clauses in SQL queries that are scattered in the application. That's the state of the world. If you want to have, let's say, an end user, and by the way, this could be an internal user or it could be an external user.

1:05:35Let's say you are in a car rental company. You have, I don't know, 2 ,000 branch offices locations and each of them has a manager who can ask questions about the system. Now, obviously security is important. They're allowed when they ask questions maybe about cars in their store, not about something else. Or maybe they can or maybe they can't. You can't just let the agent go ask questions on behalf of any user when that agent effectively has a service principal access to all of the data. So you have to solve this problem. There's no solution, no way out. We have a solution. We've built something called Parameterized Secure Views.

1:06:16And this builds on some really awesome Postgres coolness. And now we're bringing it to all our database portfolio. Postgres has something called Secure Views. So you can define a secure view and that has a very interesting property called a security barrier. And with a security barrier, It makes sure that you cannot engage in an information theoretical attack to extract information. The basic idea is, when you have this kind of an attack, is to say, write a query, which will crash the system. You write a query with a divide by zero with a where clause. And if you don't have a security barrier, the Postgres optimizer will be too smart, it'll push this predicate deep into the system.

1:06:55And if you don't find anything or you find something, you may end up not getting a result or it'll crash the system or you'll get some error. And that error will tell you lack of information is something that's valuable for you, right? It'll actually tell you that. And so postgres's security barriers prevent that from happening because it will make sure no predicates can be pushed inside the table. Now, a parameterized secure view takes us to the next level where you can apply a parameter. Now, the kind of parameter that you might apply in the where clause here is essentially the same check for authentication, the authorization, authorized access, which I mentioned is there in three-tier web applications, in where clauses and SQL queries scattered to the application.

1:07:40But we take that essentially from the application logic and you define that in your DDL for this parameterized secure views. So now you have your PSVs defined for your schema and when the agent calls, the agent is part of some orchestration system, there's a logged-in user, the orchestration ends up calling the LLM and getting the SQL query. And you can attack the LLM, you can convince the LLM, spoof it, you can generate whatever query you want. But when that query gets sent, it only has access, it's a special service principle for the agent, it only has access to the PSVs, to the parameterized security.

1:08:17So it doesn't have access to the underlying tables. That is easy to put in. And we take that logged-in user's credential and have it as a side channel. It's bound to the underlying parameter. So the query can have whatever it wants, but the view is on the fly, only restricted to the user's credential. So it's a very important piece, critical piece of infrastructure. Without it, there's no way to get the kind of security you need for agentic applications. So that, I don't know if we call it role-based access, but that user-based access is stored alongside the data or managed alongside the data, I guess, maybe is the right.

1:08:58Effectively, what we're saying is don't change. You don't need to put roles inside the database. Okay. But you can take what was written as a query in the application and put it into the DDL. Now you can let the LLM do whatever the heck it wants. I see. We can still guarantee that you don't access. One of the papers I read when I started my PhD program in Berkeley was a classic paper from 1975 called the End-to-End Argument in System Design. It's a fascinating paper. It basically says something like, if you have any kind of distributed system, and whether you want to do encryption or security, you need to solve this at the edges of the system, at the ends.

1:09:38You cannot solve it in the pipes. And that's a really interesting design primitive. And that's really the same primitive. The end-to-end argument is you need to have security right at the level of the data. Because we are living in a non-deterministic world where agents and other things can do anything. So I think this is one of the most important pieces in this whole topic of, hey, LLM with access to production databases. There's another interesting topic here, which is isolation. It's very similar to the set of problems with analytic systems, analytic queries on your transactional database. when you go run your analytics, if you run batch scans on your transactional database, you can impact your transactional database, which is one of the reasons why people offload sometimes data from the transactional database.

1:10:22And so we've done a lot of work over the years, done isolation and figured out how to solve this problem. I was talking about horror stories. I was talking to a customer in a bank about these things like PSVs. And the customer said, what are you talking about? we just spent years making sure batch scans can't run against your operational databases. Do you want me to run agents now on it? Have you gone out of your mind, right? It's a totally valid question, right? You have agents and agent swamps that are asking questions. But I think the same lessons apply. Just like we learned how to isolate transactional databases, we need to be able to isolate agentic applications.

1:11:01I think some of the things can be worse because it's not easy to control agents. Agents can run bad queries. Agents can run malicious queries. So I think the lessons that we've built, we've taken from building transactional analytical systems actually apply to this world. And then other things, I think we need to have the right observability. You need to have the observability that tell you even little things. Can you look at your Postgres database and say, give me a list of queries that came from an agent versus from the application. You need to build those kinds of guardrails around this. Yeah.

1:11:34It's almost like a little bit of real-time threat detection for what's happening to your data. Exactly. That's fascinating, the comparison against how, because I mean, I've never even really, you know, I come from an era where I always just had transactions and it wasn't always that way, was it? Right? Like, and you had to learn that. Wow. That's fascinating. Okay. Let's close it out. I got two closers for you and I try to make the closers kind of something that folks can reflect on their own roles. So you have said in the past that Google engineers can use AI as heavily as they want, but whoever commits to code owns it.

1:12:09You said that something like three quarters of new code is now AI generated at Google. How is that actually playing out on your teams? And I think for your database teams, like what is the database engineer of, let's say, a year from now, two years from now, spend their day doing? What do you think? Look, honestly, I don't know what will happen three months from now, but... Me either. I was hoping you could tell me. Certainly, I think things have changed enough that if people had priors from six months back, they should at least revisit what is possible and what is not. So a lot more is possible.

1:12:43I think you have to be careful with numbers because they vary so much based on kind of what you're talking about. My gut sense is if you're talking about something like the distributed consensus protocol that packs those technology that's the heart of Spanner. My gut is that that's actually probably still quite hard for an AI agent to directly go and change in a big way. And we'd be very, very careful about it. But when we say AI is, code is generated, it means so many different things from refactoring code all the way to maybe code commit where in the extreme, so on the one side, it's just simple refactoring of code.

1:13:22And the other extreme is I have a whole piece of PR with a self-contained unit of code where the person who built it did not even enter an IDE, right? Did not enter Emacs or VI. So that is, I think, one extreme. The other extreme is I'm in the IDE, I'm just doing routine refactoring. And there's a whole continuum of steps between these. And so I think what you'll see right now, today is a lot of variation, that there'll be areas where I think you'll see a lot more success and areas where you see less success. What we work with our teams, I think this is probably useful for most people, because of a lot of variance.

1:14:04I think it's very important to build the right harnesses and skills for each work group, each whatever group of engineers working on some coherent whole. And so like everything in the world of AI and agents, I think the key is evals. You need to have a really good objective way of knowing, is this thing mostly working or is it not working? And so I think it's very important to continuously measure and figure out what the reality is. And another thing I think that is particularly important is the space for experimentation. I think it's really valuable to have, pick a project, pick something which you go and say, I'm going to focus on pushing the boundaries of what I can get with AI.

1:14:50And when you hit the obstacles and the roadblocks, which you will, I think the engineer's imperative and the right imperative normally would be, I hit a bottleneck, I'm just going to drop down to the way I normally do it, go solve the problem because that's what we're asking, right? Solve the business problem. But I think we need to have some set of efforts where we say, when you hit the bottleneck, fix the bottleneck, don't go around it. And accept that it'll take longer to deliver. So I think these are some of the things that we do, which I think are probably more useful in general. All right.

1:15:25Last one. What is the one piece of advice you want our audience to take away from today? I know I'm going to be thinking about data level authorization thing, but what do you want folks to walk away with today? You know, we've spent the last 25 years learning how to build deterministic, scalable systems. Being deterministic, being predictable is such a big part of how we've learned to build systems. We take it as an article of faith. I think we are living in a fundamentally different era. And in this era, we need, as systems people, as thinkers, as implementers, as practitioners, we need to be ready to deal with the world where data is messy, where it's not deterministic anymore.

1:16:11And in some sense, I think most people who are engaged in this exercise have to go through stages of grief, bargaining all the way to acceptance. And they're somewhere in that stage where first people start fighting about how do I make it deterministic? How do I make this AI thing deterministic? I think that's not a successful path. That's not the right path. And I think most people hit it and they hit it fast. Yeah. Correct. I think we have to embrace the chaos. And that means putting ourselves in this kind of weird spot, right? It's not how we've trained ourselves as an industry for, we're talking probably two generations of software engineers who've learned how to build things in this way.

1:16:56And so we have to accept that we are in a world of chaos and live with that chaos. And as part of that, I think evals. Just what I said a few minutes before, how we operate in software engineering applies to the kinds of systems we build. And I want to tie this back to where you started. You talked about what has changed. I think the fundamentally big thing that's changed for my area of databases is that we're dealing with structured and unstructured data together. that we are talking about. We're trafficking not in exact results, but the most relevant results. And so being able to have evals to understand the quality and the business outcomes, I think is a very big part of how you can embrace the chaos and actually make sense of it.

1:17:44Yeah, that's something that I'm learning too, is there's no longer a right or wrong answer. There's a spectrum of relevancy. I think that that's, yeah. That's right. And we're all somewhere in that. We're all just floating in the chaos of relevancy. This was fascinating. I really enjoyed this conversation. It fulfilled all of my database nerd dreams. So thank you for being here. Thank you for the opportunity. I always love geeking out.

1:18:24Thank you.