Foundation Models for Structured Data

0:00Predictive modeling is a core element in modern systems, and powers capabilities such as fraud detection, loan approvals, and recommendation systems. These systems typically operate on structured, relational data stored in enterprise databases, with rows, columns, and interlinked tables. While computer vision and natural language processing have undergone a neural network revolution, the tabular data layer underpinning predictive modeling still largely relies on manual feature engineering and task-specific models. Relational deep learning proposes a new approach. It treats databases as graphs and applies transformer-style attention mechanisms directly over structured relational data.

0:42Researchers are now building foundation models for tabular data that aim to generalize across predictive tasks without painstaking feature engineering. Jura Leskowitz is a professor of computer science at Stanford University, and he previously served as chief scientist at Pinterest and was an investigator at the Chan Zuckerberg Biohub. Most recently, he co-founded the machine learning startup Kumo AI. In this episode, you rejoin Sean Falconer to discuss the limitations of traditional predictive modeling, why structured enterprise data requires its own modality-specific neural architectures, how graph transformers generalize attention to relational databases, and more.

1:26This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.

1:45Yuri, welcome to the show. Thanks for having me. Great to be here. I'm really excited to get into, I think there's a variety of topics we can dive into today, but maybe before we get there, just kind of grind on the audience a little bit. Like, who are you? What do you do? What's sort of your background? Yeah, I am primarily a professor at the computer science department AI lab at Stanford, been there 15 years. My research focus on AI, especially AI over structured tabular relational type data. I work a lot with graph data. And in my career, the models we've developed are today used at Facebook, at Meta, at YouTube, and at a number of different places.

2:26In my career, I was also a chief scientist at Pinterest for six years, grew Pinterest from 150 employees post IPO and was basically building large scale AI machine learning platforms there. What drew you back to academics after having experience at a place like Pinterest? I think Stanford is the most amazing place. It's where the future happens. And what is also amazing about Stanford is that it has this allow us to kind of flow between industry and academia and really understand what are the biggest problems out there that are worth solving. We go to the industry, we come back with the ideas, the students at Stanford are amazing.

3:03And I would really say kind of the future happens at Stanford. And that's the most exciting part to me. Yeah, I spent some time at Stanford myself as a student and then was drawn out to industry and never found my way back. But I guess now that there's so much going on in the space of artificial intelligence, especially with companies, you have your open AIs, your anthropics of the world, and every large cloud provider doing amazing work in the space. How do you think in terms of where research is going to make significant contributions versus where maybe the private sector of companies are going to make significant contributions?

3:39That's a great question. I would say research academia is different, right? Like we cannot compete on scale. We cannot compete on pushing products to customers and things like that. That's why we have startups. That's why we have industry. And whenever we make some new research breakthrough and we want the world to see that, we spin off a company, we spin off a startup and scale it up there. But then at the same time, there is huge value in academia and research and education because the risk profile for us is very different, right? Like we can truly explore. We can truly fail. We don't have performance reviews.

4:13We don't have all that that is in the industry. So we can always kind of ask about what are the paths not walked yet? What are the interesting new directions that maybe the industry is too conservative to take? Can we show the path there? And there's been examples of this throughout history where basically academia found a new path or researchers found a new path where the industry was just like plowing forward full scale. And right now it's similar in, I would say, in the field of AI. Of course, the frontier labs are making humongous progress, scaling up these models and so on. But there is so much unexplored.

4:45There is so much more to do. And that's what we are focusing in on our research. Yeah, I mean, I think a big part of that would be there's no sort of commercial obligation in academics. You could chase a problem that may or may not ever have some sort of commercial application just because it's an important problem, or at least to the individual to explore and maybe mean something in the long run to how we think about ourselves, our own intelligence, some other type of scientific endeavor. Exactly. And even with that, right, I would say that we are very careful what kind of questions we ask. And we always ask ourselves, if we solve this problem, who's going to care about it?

5:20Who can benefit from the solution? So being connected to the real world is a very important part of the way we think of the research we are doing. I want to get into this a little bit and first talk a little bit about predictive modeling, which I think is something that has a long, rich history in machine learning, artificial intelligence. There's fairly simplistic ways of doing some form of predictive modeling. And then there's very sophisticated approaches as well. Can you give a little bit of background in terms of sort of the history? What are we talking about when we say predictive modeling?

5:53Why does that matter? And what is the history of the discipline? Yeah, that's a very interesting question. And I think as we go through, it will become clear why this is interesting. But predictive modeling has been around forever. And it's all about forecasting risk estimation. It's about filling in some information that doesn't exist yet. And where does this matter? This truly matters for, let's call it quantitative decision-making. Right. If I go ask for a loan, there is a predictive model that estimates the probability that I'm going to pay back that loan. If I'm in a hospital, the hospital wants to estimate what's the risk that if I get discharged that I get readmitted.

6:31Right. If I am, let's say, dealing with customers, I want to estimate what is the lifetime value of the customer. I want to estimate how likely this customer is going to churn. I want to estimate what next product or what next show item to recommend to the customer. If I'm a, let's say, financial institution, I want to estimate what's the likelihood that this particular transaction is fraudulent. When a, let's say, user logs in, I need to estimate how likely is this a stolen identity? Somebody else is logging in and things like that. These are all predictive type problems where based on the historical patterns, based on the data, we want to estimate something that we don't know yet.

7:10we want to forecast something and so on. And this has been around for a very long time and people have been building machine learning, statistics, data science, have been building these predictive models for the last 20, 30 years. And the point is that every percentage improvement in accuracy of these models means humongous business impact, right? Even 1-2 % improvement can have humongous business impact. How would I think about this in terms of with forecasting, say it's like financial forecasting, I'm taking a bunch of history and I figure out what is the function that scribes that history. And then I'm projecting that out to see where could this, I don't know, trend in the stock market go or something like that.

7:51How is it when you think of something like classification? So I want to go and take a bunch of history as my training set, and I'm going to use that to train some sort of classifier to figure out whether an email is spam or not. Is that predictive modeling or is that a different type of classification of how we would think about that AI model? This is all what I would say falls under predictive modeling. Both examples, time series forecasting, any kind of classification, churn modeling, as you gave the example, where the idea is that based on some historic patterns, you are trying to forecast, is this person going to cancel subscription in the next month?

8:28And because we don't have the information what is happening next month, we have to forecast. If we take a specific example, like a recommendation systems, I think Amazon's been pretty famous for using recommendation systems for products for a long time. How do those systems typically work? Yeah. So the reason we started opening this is because this field has remained practically unchanged for the last 20, 30 years, right? It's all based on this idea that you bring in, let's say, a data point, a unit of something that is described with a set of characteristics or a set of features, and then you are making that prediction, right?

9:05So, for example, for a recommender system, the idea would be that you have a description of the user, which would be maybe how long ago did the user register? When was the last time the user logged in? What were the last seven products the user visited? And what categories are these products from? And so on. And you have this kind of profile of the user. And then you would say, okay, I also have to now build a profile of the product. and now I need to learn some function that takes the profile of the user, profile of the product and tells me how likely is, let's say, the user to purchase that product.

9:36And then if I find top 10 products that the user is most likely to purchase, I show those to the user and my sales go up 20, 30, 50%, right? That's how this is generally done. And now majority of the work goes in building that scoring function that takes the user profile and the item profile and gives me the prediction. And if you say, how accurate is this prediction going to be? You have two aspects to it. One is how accurately do I build the user and the product profile? Let's call it this way. And then how powerful that predictive model on top is. And the point is that this is like super painful, super slow, and super manual to do, right?

10:15Like you need to hire a team of data scientists. They need to build these profiles. This is called feature engineering. They come up with some historical summaries of the user activity, put those as into the user profile, they do something similar with the product, and then they create these training data sets to build the models on top. And the models on top can be this kind of decision tree, XGBoost, CADBoost type things that work kind of well in practice and are completely respectable, as well as to more sophisticated neural network approaches and so on. But the bottom line is that you need about two full-time people to build a single model, right?

10:52Like if you say how expensive this is, it's like I need two employees to support one model. So if I now want to have 10 models in production that are making these decisions on the fly, I need that number times two number of people to support that. And if I build like my e-commerce recommendation system, I also need fraud detection. I can't just pick up my recommendation system model and apply it to fraud detection. I got to go train like essentially a new model. I got to do feature engineering just for the fraud detection. probably use maybe even a different type of model to train and test against, and then probably operationalize that model with a different set of people.

11:28Exactly. Exactly. And I would say now, what is the exciting thing? Why are we talking about the past? I think the exciting thing here is that this entire area hasn't seen real progress in the last 20, 30 years. And if you think about what has happened in the broader AI ecosystem is that we went fully neural network. What I mean by that is on the, let's say, In computer vision, we used to do some edge detection, some feature detection, and then build a classifier on top to say what's in the image. We don't do that anymore, right? Today, a neural network just learns directly from the pixels of the image.

12:01The same thing, I would say, happened on the, let's call it, natural language processing area, right? Where we used to do all kinds of parsing and feature extraction from sentences to try to say something about what the text is saying. And today the attention mechanism just attends over the tokens of the text and kind of the AI, the reasoning is born, right? So I think the exciting thing here is that machine learning hasn't gone through this neural network revolution. And that's the exciting new thing here is that there is the neural network revolution for machine learning ready to happen that completely changes how we are building these models, how accurate they are, how much feature engineering it takes, and things like that.

12:41Mm-hmm. So why is this kind of hard in practice? And why couldn't we just take, we've had this revolution around things like large language models that understand text very well, and now they understand images and audio files and even video. Can we not take those and just apply them to this problem? I think the argument is the following. I would say, if you ask what kind of data are we using when we are making this predictive modeling, predictive problems, it's structured relational data, right? So this is data that is stored in tables that are interlinked with primary foreign key relations. This is usually stored in a database in some structured form, right?

13:19And this is the most useful data that enterprises have because this is kind of the ground truth of the enterprise. All the events, all the activity, it's all stored there, right? So now what I'm saying is if you say for images, to process images, we have special neural network to process images. To process text, We have special neural networks. And to process this tabular data, we need special neural networks. It's a different data modality. Image has its own set of networks that are different architectures trained in certain ways. Text has a set of networks trained in specific ways and so on.

13:53And the tabular data also needs its own set of neural network architectures and its own way to train this that can directly kind of attend over this structured tabular data. So I think it's a different data modality. That's why it needs a different approach. Right. I mean, I think with something like text, where there's probably billions, if not trillions of examples of how sentences come together, and I can take a document from one place and a document from another place, and learning one of those documents can probably help me infer something about the other document. I think with the way I think about this problem around tables, rows, columns in a database is, is there that much I can, from a pattern standpoint, like learn from one table versus another table?

14:38Like those patterns seem like they could be fundamentally different in terms of how the person has modeled the data. So it makes a lot of sense that this is sort of a different class of problem, but how do you go about, I guess, like attacking that problem? And does this make sense in terms of the patterns from one table, not necessarily allowing you to infer something about the patterns from another table? That's a great question. I think the way to think of this is as you have this private data, right? You have patterns, properties in it that are unique to you. I think another point to make is also like you cannot just textify a table and give it to a large language model, right?

15:12Large language models are amazing at what I would say qualitative human-like reasoning, but they are not really good with numbers if you want to say it very simply. And if you think about what are we storing in tables, we store quantitative data. So we need to do quantitative reasoning, not qualitative reasoning over huge amounts of data. Another point I think that is important here is to say that no enterprise has data in a single table, right? You have data spread across multiple tables. You usually would have your customer catalog. You would have your product catalog. You would have a set of transaction records.

15:46You would have your website browsing click data. You would have your supplier data. You would have your returns data. And all these tables are interconnected. And the only way to learn over this is to learn over this collection of tables as they are interconnected with each other. And maybe to say more, right, in the past, the way we deal with this, we would say, oh, let's take the user table. Let's take the transaction table. Let's join them. And then somehow summarize the number of purchases, the number of transactions you had in the last time period. But there is kind of an infinite way of creating these summaries.

16:22I can count, I can sum over some time period, over shorter time period, in the mornings, in the evenings. I can add the prices, I can look at product categories, right? So the number of ways you can summarize the data is kind of blows up. And the problem with machine learning is that we kind of predetermine how to summarize the data before we start building the model. So the way to make these things better is to generalize the attention mechanism to attend over the raw events in the database and learn how to summarize them to give you that prediction. And that's the key differentiator. You don't need to be joining tables anymore, but let the attention mechanism.

17:01Very similarly, as in a language model, it attends over the previous words to say, OK, what's truly the meaning of this word? But here we are saying, if we are making a prediction, if we are filling in, let's say, some cell in a table, let's attend over the other rows in the same table, other columns, other tables far out, and figure out how to bring all that information so that we can make the accurate prediction. So it's really about generalizing the notion of attention mechanism to this structured, multi-tabular data. And if we have that, what does that unlock from an application standpoint?

17:33Today, I feel like a lot of people are trying to apply large language models to databases for the purposes of being able to do intelligent natural language to SQL conversion. Does this yield a better version of that? That's a great point, right? So text to SQL is amazingly useful. But if you think about SQL, SQL is summarizing what has happened last month, what has happened last week. So you're kind of aggregating some past and maybe creating a dashboard to understand historical trends. And then maybe you can use those historical trends to do some kind of qualitative decision making about what to do tomorrow.

18:11But if you think about predicting transaction fraud, deciding which customers to send an offer to, and so on, for that you need predictive modeling, right? So text to SQL won't get you anywhere with that. You may use SQL to generate historical patterns and then build a model on top. But as we talked about that, that's super brittle, manual, and takes a lot of time. So the approach we invented in my research group at Stanford, and then we founded a startup around it called Kuma.ai, is this notion of relational deep learning, where we basically take the transformer architecture and generalize it so that it can attend over this structured relational enterprise data.

18:55And the key to this approach is to think of the data as a graph, to think of your enterprise's data as a set of connections between the entities in your database, right? To think of the tables, how they are interconnected as a graph, and then generalize the attention mechanism to be able to attend over this relational, structured information. Agents are getting smarter every day. But even the smartest agents get stuck without the right context and the right tools. That's where Notion comes in. With the recent launch of custom agents, Notion became the collaborative AI workspace where teams and agents work side by side.

19:31And now, their new developer platform is turning that workspace into infrastructure developers can build on. Most agent platforms are single player, making you stand up your own infrastructure just to start. Notion's developer platform flips both. You get primitives to sync any data source in, give your custom agents tools that plain MCP can't deliver, and orchestrate agents like Claude or Codex alongside your team. The CLI authenticates in one line, and workers run on Notion's runtime, so there's nothing to provision. Because the workspace your team lives in is the same thing you build on, permissions and governance come standard.

20:07Write your code. Deploy. Done. Learn more about Notion's developer platform today at notion.com slash SED. That's all lowercase letters. Notion.com slash SED to try Notion's developer platform today. And when you use our link, you're supporting our show. Notion.com slash SED. If you're running Postgres in production, you've probably felt the moment analytical queries start fighting your transactional workload. Most teams end up adding a second database and all the pipeline complexity that comes with it. Tiger Data, creators of TimescaleDB, takes a different approach. We extend Postgres with hybrid, row, and columnar storage, so one table handles both writes and analytical scans.

20:47Native compression cuts storage costs up to 95%. Continuous aggregates keep dashboards live without bash jobs. And it scales to petabytes without you re-architecting. Companies like Cloudflare, Octave Energy, Schneider, Axpo, and Floco run production workloads on Tiger Data today. No stale data. No second system to operate, just Postgres. Managed for you, ready for the workload you're building toward. Try it free at tigerdata.com. You know Fidelity is a financial services leader. But did you know that Inside Fidelity is a community of technologists working together to shape the future of finance and tech?

21:21Fidelity is always investing in tomorrow, from emerging tech to cutting-edge tools that will transform what comes next. Their technologists are encouraged to keep learning so they can expand their skill sets, explore new ground, and stay ahead of this rapidly evolving industry. And right now, Fidelity is hiring technologists to join their team. Fidelity technologists get the best of both worlds, startup energy that's grounded in the stability of a financial institution. That means support, resources, and amazing benefits. Bring your skills to a culture where you're empowered to dream big and build the tech that drives an organization and makes a real impact on people's lives.

22:01Find out more at tech.fidelitycareers.com. That's tech.fidelitycareers.com. Fidelity is an equal opportunity employer. So, I mean, thinking of a database as a graph, I think people in the data modeling world have been using those concepts for a long time. Like, how is this different? Like, if you think in sort of designing schemas and things, people used to build like entity relationship diagrams where essentially you have your tables as a node and then you have relationships defined in terms of edges across foreign keys and so forth. Is this something fundamentally different or is it using a similar concept as essentially the basis for doing this deep learning?

22:42Yeah, it's a great point. Any database is a graph, as you've said. And these concepts have been around for a very long time. But I feel like nobody has put one plus one together in a sense. We've been working on this graph-based machine learning, graph transformers and things like that for a long time. but it was mostly applied to social networks. And people kind of that community didn't realize that actually the database is any database is a graph. And I think the database community was so stuck in feature engineering and running historical SQL queries that they did not think about, oh, how can we take these AI tools and apply them to the database, right?

23:19So what I'm saying in some sense is supernatural, right? We knew for a very long time, the database is a graph, nothing changes, but what in some sense changes is that now the field of graph machine learning is not this obscure, it only applies to social networks type thing, but it applies to any database. And the benefit that comes with it is that is the same neural network revolution that we have in computer vision, that we have in text, natural language understanding, videos, and so on, now to another data modality, which is the structured tabular data modality. With the same set of benefits and the same set of amazing outcomes that we have already seen play out in other data modalities.

24:00And does this only sort of work against like a traditional database? Or could you potentially generalize this to other tabular forms of data like spreadsheets? Yeah, I mean, where the data sits, it kind of doesn't matter that much. It can be in a database, it can be in Salesforce, it can be in Databricks, can be in Snowflake, can be in spreadsheets, can be in JSON. as long as it's structured, semi-structured, and it may include images and it may include text and include columns and categorical values and all kinds of geographic information, all kinds works nicely in this framework. How does training work and how is it different than training with traditional transformer models?