Relational Foundation Models for Enterprise Data with Jure Leskovec - #768

29:41Jure Leskovec:And then there needs to be a way for me to instruct the pre-trained foundation model what kind of prediction I want. right i want to say predict me the sum of purchase prices over the next one month for this particular customer and that maybe is like how much i'm predicting how much the customer is going to spend or i'm saying predict me you know transaction dot is fraud equals true for transaction id this much okay so this would be like predict me whether the transaction is fraudulent for this particular transaction id right so i have a way to specify my predictive task and now what the system does the system now goes into the database it extracts a set of labeled in context examples that then get passed through a pre-trained neural network to make a prediction okay so now when i say a set of labeled in context examples this means that you can take the task for the example of fraud i've got historical fraud that's already been labeled and i've got some new transactions coming in that don't have that label attached that i'm trying to to predict for example okay so let's do fraud fraud might be easier yes so the way this would work right if i say predicting the probability of fraud the system would go into your into your database and extract previous transactions for which we know whether they are fraudulent or not for each of those transactions we would then extract kind of the subgraph of entities around it okay so now what the relational foundation model gets on the input is a set of historical subgraphs of previous transactions and their fraud labels, plus the new transaction that is unlabeled.

31:35Jure Leskovec:We don't know it's fraud. And then this is passed through the relational foundation model architecture forward to kind of label the unlabeled graph, right? Like the unlabeled transaction and the graph around it. I'm not sure now that fraud is a good example because it's kind of, I can see how that could work. Like you've collected the graph around these known points and you're asking a model to infer relationships that might lead to this one individual label. And so maybe, I think maybe, and this may be where you're going, like I think a regression type of a problem would strike me as more challenging than a classification problem.

32:26Jure Leskovec:Yeah, I think the key here is, right, what are maybe the key components here is first is that you have a language where you specify the task. We can go generate almost like a mini labeled training data set, these in-context examples. and then you have a pre-trained model that is able to take these in-context examples, these subgraphs that have some certain columns and tables and so on, is able to encode them in a domain-agnostic way. And then the neural network is able to essentially build a predictive model in its brain in a forward path to give you accurate prediction. Right, right, right. Right.

33:09So it's not necessarily about like some universal understanding of numbers or what have you. It's about being able to identify the right relationships between numbers that it hasn't seen before, query the right, you know, examples and create the right universe and then formulate that as the right, I guess, like inference request or something.

33:34Jure Leskovec:exactly so there i would say two aspects to this one is how can you take data and encode it in a domain agnostic way right because we can take any database any set of uh any set of columns the model needs to be able to encode that in a in a universal way and now that it's been encoded then the second step is to perform in context learn so it means that the modeling its brain needs to be able to build a model, right? There's no training. There's no backpropagation. There's no gradients. It's just a single forward pass in which the neural network kind of in itself builds the model in a sense that gives us the accurate prediction, right?

34:19Jure Leskovec:So no training is necessary. No hyperparameter optimization is necessary. No feature engineering is necessary. All you need is a raw database and a way to specify the task. Does the model require some type of memory structure, blackboard or something in order to, you know, do a scratch work to come up with a representation or is this all like thought traces or something like that? No, no, no. This is not an agent. This is a single forward pass of a transformer-like neural network, right? So this is purely inside the neural network. There is no agent. There is no memory. There is no scratch pad.

35:02Jure Leskovec:There is no, let me do this, let me do that. The answer is truly a single forward pass of a neural network. There is no loop, nothing like that. So you get the answer in, I know, 0.2 seconds, half a second, whatever the time be. It's really a single forward pass of a pre-trained, frozen neural network. There is no language model here, right? This is kind of technology that's parallel or complementary to language models, right? You cannot textify a database and then go to ChatGPT and say, hey, what do you think? How likely is this transaction to be fraudulent? You get horrible results, right? So this is, yeah, frozen pre-trained architecture that allows you to do that.

35:51I feel like I've gone the full cycle from that's an outlandish claim to, oh, yeah, I can see how that will work. I don't know. It's still kind of crazy that it works.

35:59Jure Leskovec:No, it's interesting, right? And when we test this on data sets that are locked away and hidden and the model has never been trained on and on tasks that we haven't even thought about, we see a gain over best supervised models out there. If you would go and say, I'll hire a data scientist, they'll spend several weeks building the model, tuning the model, the latest neural networks, whatever, it's still a couple of percentage points worse. And then if you fine-tune the, let's say, the foundation model on more data for the specific task, then you get to this superhuman accuracy performance that, you know, present manual or semi-manual or agentic solutions are just not able to attain.

36:49That is the RFM2, Kumo RFM2, the relational foundation model. You also recently published that iClear relational graph transformer. Is the one based on the other or are they independent lines of research?

37:08Jure Leskovec:What I would say is at Stanford, we are pushing forward in the open new architectural improvements, understandings and as much as we can as academics we release everything open source we talk about everything and then of course what happens inside the company is that some of these innovations that we put out also also kind of diffuse inside I would say that internally the architecture we are using is a bit different it's composed of two different parts. The first part is basically it's the encoding or the attention mechanism over this set of tables. And then the second part is this in-context learning type machine.

38:02Jure Leskovec:There are two papers that are relevant here. One is the relational graph transformer that we mainly use for supervised fine-tuning type tasks. But then another paper we also published at iClear it's called relational transformer. And that one actually allows for in-context learning. So that one does attention all the way at the individual cell levels of a database. And essentially you have three types of attention. You have attention over a given column. So if you are interested in a cell, by attending over other cells in that same column, it kind of gives you a sense of a distribution, right?

38:42Jure Leskovec:Then we attend over the cells in a row. And that kind of then gives you a sense of what's the information in that row. And then we also have a graph-based attention mechanism that allows you to say, oh, this is a user and these are all their transactions. And then each transaction is a row and each row has columns this way. So this means that we can be attending over millions, tens of millions of cells. And the beautiful thing is that our attention mechanism, because of the graph, has much more structure. So the attention mechanism is never quadratic. And this means we can compute much more effectively.

39:21Jure Leskovec:And to do good reasoning, you really need humongous context sizes, right? Even the largest LLMs today, I know, go to a million tokens. For us, a million tokens is small. So I was going to ask, are there data requirements or shapes or use cases that this works well for or conversely doesn't work well for? It sounds like part of that is size, like you need a lot of data in order for this to work. Is that fair? i would actually uh maybe push back on that a bit actually because the model is pre-trained it can do amazing things where you have very little data because it's you know like training models from scratch yeah requires a lot of data but once the model is pre-trained it kind of knows what functions kind of appear in nature so it means that you can give it few examples and it's going to give you very accurate predictions more accurate predictions than something some you know supervised model that you have to trade.

40:33I think I picked that idea up based on you saying that the context that you work with is typically large. Is that saying that when you have a lot of data available, you can use it, but you don't necessarily need it?

40:48Jure Leskovec:Exactly. And then once, if you have a lot of data, you can either increase the context size and by increasing the context size, you get more accurate predictions. Or if you are saying, oh i'm doing fraud you can just fine tune your model for fraud in a sense that you don't even have to do in context learning because you know your data you know the task you just tune the model for that single task and then the model can be smaller much more efficient to run and also more accurate because it doesn't have to you know almost like re-learn the task every single time because you give it the in context examples what we see works best is some mixture of pre-training and in-context examples because the way you're choosing context examples can actually depend on what the target entity is, right?

41:36Jure Leskovec:So in a sense, you'd say, oh, if I'm, I know, predicting fraud for, I don't know, for me, then you could say, oh, let me put some other Stanford professors in my in-context examples. Let me put some other Bay Area folks in here because, you know, that's kind of the, I don't know, the peer group or the most useful examples from which you can learn to make accurate predictions about, I don't know, me being a fraudster. I'm thinking about the line between kind of the model and the system. The system is what is constructing the in-context examples, and the model is just that forward pass. And you need both, right?

42:10Jure Leskovec:I think is important, right? Because somebody has to generate these in-context examples. You won't generate them manually, right? And is that part also learned or is that, you know, kind of a formulaic graph traversal or something else? Somewhere in between, you can do it as a form, kind of just as a graph traversal and a bit of kind of time travel, right, to generate the forward-looking labels. But of course, how you do that and what in-context examples you generate makes all the difference. So there's a lot that goes into that to get top performance. And so speaking of performance, you talked a little bit about some of the challenges with collecting benchmarks, but how do you find performance relative to those benchmarks and also, you know, more importantly in the real world?

43:11Yeah.

43:12Jure Leskovec:So I can say, right, like we have a white paper on KumoRFM2 that people can read with a bunch of different benchmarks. What we see is that the foundation model by itself improves state-of-the-art over all supervised models ever published on this benchmark, right? So the baseline is very high. It's like just build the best model you can and see how high you can get. So the foundation modeling improves that, I think, for about 5 % relative to the accuracy. And then if you further tune the model, meaning if you would fine tune it and do some grain and base updates, then the performance goes to 12 % over the state of the art.

43:59Jure Leskovec:And those are quite sizable gains, especially if you think about putting this in production in recommender systems or fraud detection where, you know, every single digit performance in increasing accuracy can mean millions, tens of millions in business impact. maybe the second thing i would say is where we see these methods also shine is with noisy and incomplete data cold start problems because of the relationships because of the relational structure the model is able to much better kind of hone in and be much more robust to the data missingness data corruption and things like that so we've also done quite a lot of analysis around understanding and like how this performs in real world data, sparse data, small amounts of data, noise, incompleteness, irrelevant columns and things like that.

44:59And when you mention cold start, like that suggests, hey, I want to start identifying fraudulent transactions, but I have no labels. I just have a bunch of data. Can you tell me where I should start looking? Like, Does it work for that kind of problem?

45:15Jure Leskovec:Yeah, maybe I should quantify what cold start means. Usually cold start would mean when a new user shows, a new product shows up, right? So you still need to have some historical labels. I'm not, you still need some historical labels, but usually, you know, prediction is easy once you have a lot of data about a given user or a lot of data about a given product. But when the product is fresh or when the user is fresh, you are data poor. That's what technically is called cold start problems. So I still need historical labels, but to make reliable predictions, I don't need much data. I think I saw somewhere that the system is deployed at places like DoorDash and others.

45:58Can you talk a little bit about the process for deploying it?

46:04Jure Leskovec:Yeah, great question. So the system, the platform, we can deploy it in many different ways. We can, you know, run it as a SaaS, basically as a compute platform. We can deploy it in people's private public clouds, like inside, we call it virtual private cloud. So all the data stays with the customer. There's a bunch, there's, I would say, a bunch of different deployments depending on what organizations like and prefer. and then yeah in terms of let's say deployments or use cases right at uh at doordash it's uh restaurant recommendations and the notification system which user gets what notification at what time of day and things like that and we've seen you know uh revenue impacting hundreds of millions of dollars um another another great client we work with is reddit so the advertising models on Reddit are built on top of or are built with Kuma.

47:05And it was nearly a double digit

47:10Jure Leskovec:increase in ad click-through rates. So basically the revenue, yeah, it's like unbelievable, right? Usually an entire team, you know, like increases maybe 1 % that accuracy year over year, right? Because click-through rate is... And this is your original point about like, you know, domain expertise and like manual features. Like you would imagine that they've been working on this for a long time and they've kind of squeezed a lot of the juice out of that lemon. But, you know, here comes the machine. Exactly, exactly. And it's actually interesting. And I mean, you know, we have a great collaboration and great relationship with the Reddit team and they're amazingly sophisticated.

47:48Jure Leskovec:And of course, they build their own super optimized feature engineered pipeline. And then the way we do it there actually is that we said, okay, let's take your data represent it as a graph and let's create embeddings for users subreddits ads and things like that so now these embeddings actually get appended to their to their own features right and even with that there was there was a huge increase in the click-through rate because this signal that the neural network learn was kind of complementary to what the human feature engineering already had. So actually the model that is in production is combined from the neural network embeddings by graph embeddings by Puma, as well as the manual feature engineering.

48:37Jure Leskovec:So that's been a great collaboration. So it's add the recommendation, click-through rate prediction, if you want to think of it that way. You know, or have you looked at like if their manual features really make a difference? Like, Is that a feel-good thing, like you left them in there because they had them? Or do they provide lift that's been measured? Good question. I don't think we tried turning those off yet. But it's a very interesting question. Sometimes you still want to have those features in there, not maybe for the model accuracy, but because you have so many business rules, you know, like these advertising systems, they are not just pure optimization plays.

49:26Jure Leskovec:There is so many kind of other business rules that need to trigger for the ad to be actually shown to the user. So you kind of need sometimes those signals to be able to trigger business rules anyway. And I wonder in this case and with those hand-engineered features and more broadly with RFM, you know, what kind of explainability story there might be. That's another reason why people like XGBoost is that those trees are fairly interpretable and that's been a challenge with, you know, transformer-based networks. Yeah, that's a great point. Actually, I would say we do explainability really well.

50:11Jure Leskovec:And it's even more models, I would say, are even more explainable than these tree-based models because in tree-based model all you can get is you get a ranked list of features right so you can only explain predictions by the with the features you you engineered what we can do is we can do this and those features might be you know wrong understandings of the data or incomplete yeah exactly so what we do is we do the we can do because the model is fully differentiable we can basically run the model backwards and we see what tables, what columns, what cells the model is attending over. And we get this structure-based explanation.

50:47Jure Leskovec:But then we use a large language model to say, here's where the attention is. This is what the columns are. And this is what the semantics is. And then we generate a text-based explanation that is like super readable, super friendly. Like a text-based explanation of a saliency map of the data or something like that? Think of it maybe that way, right? But the LLM is kind of enriching it with all the human world background knowledge and so on. So it gets very actionable. So you were talking about use cases. You mentioned we were talking about the recommendations. One last one, which is around fraud.

51:24Jure Leskovec:So we've seen great results with fraud. Here we've been partnering with an amazing team at Coinbase. So we have these models running in production at Coinbase on the entire Bitcoin blockchain network. Right. So also we can scale to, you know, the size of the entire edit, to the size of the entire Bitcoin or Coinbase, you know, right. So these methods really scale. But then, you know, with some clients like, let's say Databricks or actually Snowflake, they are using us to run their sales models, you know, predicting what the customer is going to buy next, which customer is going to convert into a paying customer.

52:05Jure Leskovec:and this allows them to optimize their sales team, right? And if you think about sales team data, that data is smaller because the sales teams are, you know, hundreds, maybe, you know, thousand people or so. So you can do well on small data as well. You know, there are aspects of it that sound free lunchy. Like, how am I paying for my lunch? Like, what's the, you know, is it, yeah, maybe I'll leave it there for you to answer. Yeah, I mean, at the end, what is being paid for is compute. What we are really doing is, right, like machine learning, if you think about it, is really a CPU compute based, right?

52:48Jure Leskovec:Majority of the compute that happens, except maybe the final neural network training, happens on the CPU. What we are doing is taking that workload from the CPU to the GPU. so now the amount of gpu compute is is is larger because it's computed over the raw data not over the summaries generated on the cpu um and and that's uh that's i would say what the uh what the what the cost what the cost is uh in the end of course these models and um um don't you know are not in trillions of parameters they are billion parameter type ones they can be quite small So they're actually quite efficient and cheap to run because, you know, the reason we are making predictions is to make decisions, right?

53:39Jure Leskovec:Our commander system is making decisions what to show to every user. So we are like the speed of those decisions is that, you know, tens of thousands, hundreds of thousands, millions of times per second. So performance cost really, really matters. Yeah, I think I was also trying to get at limitations. and if you had someone come to you with a problem that, you know, was in fact like multi-tabular relational, you know, what might be, you know, some reasons why, you know, you ultimately, you know, tell them that it's probably not a good fit? The way I would say is we know if we use our technology, we'll be at least on par or better to what is already there, right?

54:27Jure Leskovec:now where we see bottlenecks usually we see bottlenecks in actually getting the value out like connecting those predictions to some decision making downstream business process so that the value can be reliably measured that's been i think the biggest bottleneck right in a sense that models are are built developed they work great but then engineering teams need to hook them up to actually surface those predictions or make decisions based on those predictions. Another, right, like maybe that's one use case. Another use case is sometimes like where we shine or where the technology shines in these predictive, well-defined, predictive type problems that can be mathematically well formulated and optimized.

55:16Jure Leskovec:If it's more about, hey, we want to understand the patterns, We want to understand what is happening in the past. That is much more, you know, this kind of traditional data analytics-y type things or some pattern detection type thing that our platform and what we discussed is maybe not a good use case. So that's, I would say, is, you know, some examples. You need to know what you are predicting. You need to be able to formulate that. I need to be able to measure accuracy, and then we can optimize. And you're not going to solve the traditional data science problem, like in organizations, if you've got a model, how do you use it?

56:03That's still going to exist.

56:06Jure Leskovec:Exactly. There is still the problem of how now that we have the model, how are we pushing that to, I wouldn't say to production. That's easy. The question is, how do we connect it with the downstream app or the downstream system so that actually somebody is acting on these predictions, right? And where we also see, I would say, a lot of traction recently is in agentic workloads, right? Because agents need to make decisions to take actions. And now you can make decisions based on this, you know, LLM-based common sense. But for anything more, like the best way to make decisions is to estimate or predict their downstream effect.

56:51So now I'm envisioning this model sitting behind a tool interface that an agent can call to, you know, when it needs to make a prediction about the data.

57:01Jure Leskovec:Exactly, exactly. right and even if you think about let's say a customer support agent you call me in i need to estimate what's your lifetime value how likely are you to churn i will respond differently what's the best offer for me to give you how i need to actually ask a counterfactual question if i make you this offer how will that make you happier and and and these are all predictive problems i cannot just hallucinate them or you know ask chat gpt it will do something reasonable right like something something I would say common-sensy, but that's far away from optimal. So these predictions, this reasoning over this structured relational data that captures the patterns, behavior of, let's say, customers inside the organization is crucial to make accurate decisions, right?

57:49Jure Leskovec:And as we are deploying these agents, we cannot be building now separate models for each of these and pre-anticipate the questions, right? the beauty of the foundation model is that you can ask any question and one thing i want to say here it's like just to show how big the problem is right like if you think about a organization like maybe like uh you know think of let's say sap right uh sap has i think i know 70 100 000 customers each of their customers has structured data because it's a it's an organization every one of them um uh changes the schema a bit so everyone has their own data has their own schema and every one of those wants to do a churn prediction, wants to do churn prediction.

58:33Jure Leskovec:But every one of them has a bit different definition of what churn is. So now can you hire 70 ,000 data scientists that are going to build per client churn model with the client's data and the client specification what the churn means? You can't. A foundation model, an agent can just ask, predict me probability of churn under this definition under under this data and you get the answer half a second later that raises a question for me it around uh like post-training the model or fine-tuning the model does it is there any value to um like intermediate it fine tuning. I think what I'm envisioning is like you partner with SAP, SAP has, you know, hundreds of these modules.

59:27There's like a supply chain module and there's a, you know, churn module and some other thing. Like, does it make sense to tune on, you know, the use case, separate from the individual customer's data or does the foundation, the breadth of the foundation model already capture all of the information at that use case level of abstraction and really you're only improving it if you're looking at a specific customer's data?

1:00:01Jure Leskovec:That's a great question. Actually, that's something we are deeply looking into right now. I would say there's several reasons why to post-train. One reason you would want to post-train, even in the in-context learning scenario, is to better learn the distribution of the underlying data, to better understand the distribution of the underlying data, so that prediction, like the data gets better encoded, and prediction later will be more accurate. So that's one reason you would want to, let's say, train, even in a task-agnostic way, over the underlying data to better capture distributions, to better learn priors.

1:00:41Jure Leskovec:another reason why you would want to fine-tune is for cost cost reasons because if i fine-tune for a specific task i don't need to do icl right because now i don't now my context is is much smaller i don't need to bring in the label data i just bring in the entity i want to predict on now the the attention is smaller it's faster it's uh it's uh cheaper cheaper to run if i have large amounts of data, the model can learn a lot from it. So I would say there is a spectrum, there is a continuum of what you can do and the benefits of it is kind of different depending on where on this continuum are you seeing.

1:01:25So what's next?

1:01:27Jure Leskovec:Yeah, what's next? We are very excited about agents, both basically surfacing these two agents as tools The second thing is right now, right, like the coding agents are out there. But what we see is that coding agents require a proper abstraction and a proper infrastructure to be able to be effective. Right. And for example, if you could say, hey, why don't I just, you know, give this modeling task to CloudCode and CloudCode will build the model for me. So, you know, what's the big deal? And when we do that, what we see, we've run this internally, right, is that these models write thousands of lines of code, but there are these like super subtle data science mistakes.

1:02:16Jure Leskovec:mistakes so for example we've done this uh together uh together uh with expedia um and you know when when it was a account level fraud and mistakes for example the agent make was that when it created features for that given account it created it aggregated the transactions till midnight not till the current time right so it said oh today is i don't know uh april 30th so we'll use the data up to midnight of april 30th not actually saying hey it's actually 10 a.m 10 10 a.m on april 30th we can only use data up to here right so that's information leakage was a little mistake in there another mistake it made was that you know we did it at a transaction level instead of the account level and these are like these subtle mistakes that really you need the human in there.

1:03:11Jure Leskovec:But if you give it this more like higher level Kumo-like API, then it's able to do the same work in about 50 lines of code. No mistakes. The task in this case is to do what? Like I thought the task that you were describing was to code up something like what Kumo is trying to do the task is build me a account level fraud fraud detection model over this data and so what you're what you're proposing is like as opposed to trying to the agent trying to code it up from scratch you create some kind of skill or something like that that teaches it how to use kumo to get the same information yeah or what i'm saying is agents you know they they can go they can autonomously maybe make two steps but not hundred steps so now when i ask it for a task i can say hey here's pytorch go build me the model that that's you know takes thousand lines of code to build a model with python i could say here's hgboost build me a model that takes about engineer features and so on that's that takes about 500 lines of code and and you know uh or i can say using the kuma api go build me the model that only takes 50 lines of code and the the the now if you think of this analogy of steps you know 50 lines of code is maybe like two steps 500 uh lines uh lines of code is uh is 20 steps right and a lot you can get quite lost navigating i don't know 20 steps in this right i think the observation is is more general i'm like the observation is that for agents to be effective they need apis that are agent or agentic friendly okay awesome well yuri thank you so much for jumping on and catching us up to what you and Kuma are up to.

1:05:26Super cool stuff.

1:05:27Jure Leskovec:Yeah, thank you so much for the conversation and very insightful questions. Awesome. Thanks so much.

1:05:50Thank you.