Why the Next AI Breakthrough May Come from Physics with Max Welling - #774

0:00Max, it's so great to be on the line with you again. It's been a while. It's great to be back, Sam. I'm looking forward to our discussion. I am as well. Our audience can look up our conversations from, I think, 2019 and 2020, where we covered what you were working on at the time and I think still echoes into your work today. Geometric neural networks and gauge equivariance neural networks and the like. But I'd love to have you kind of catch us up on what you've been up to since. It's been quite a while. Yeah, actually, the aquivariance theme has definitely continued. So, in fact, I found out that aquivariance was used very fruitfully in chemistry and material science.

0:49So in chemistry and material science, people train neural network models to predict the forces on atoms because people want to evolve atoms forward in time in order to compute their properties, which is called molecular dynamics. And typically you need to use quantum mechanics to compute these forces because a large contribution comes from the electrons and electrons are very light. And so you need to treat them with quantum mechanics. But, you know, if you get 10 electrons or more, in the case it becomes completely unfeasible to solve the so-called Schrodinger equation. So people have come up with approximations like density functional theory known as DFT.

1:34And the inventors of that got the Nobel Prize for that. But what now people do is they train surrogates. So they provide data using this expensive approximation to quantum mechanics. And then they use neural networks to shortcut the computation. So to predict the outcome of that computation, but at a much more accelerated pace. So in other words, three orders or four orders of magnitude acceleration, more efficiency relative to these quantum mechanical approximations. And in those models, because the world is three-dimensional symmetric, so if I rotate a molecule, all the forces will rotate with it.

2:17And so we could now put the same ideas that we use for images, we could put them in these molecules, these models that predict the forces, and we could use equivariance. And so that's why I kind of, also because my background is in science, I did my PhD in theoretical physics, I thought, okay, this is a perfect unification of my old sort of passion and my new passion. I can put it together, and that's when I started to be interested in AI for science. So that led pretty directly to the founding of Cusp AI. Actually, first I spent two years at Microsoft Research as a VP because they were building their AI for Science lab in Amsterdam.

2:59And so I helped that along. But after two years, I wanted to start a startup. I already did a startup a while ago, but this was actually the startup that got acquired by Qualcomm. And then I spent some time at Qualcomm. and I really like startups, the dynamical environment and the impact you can make. And I wanted to do it the Silicon Valley way together with my co-founder, Chad Edwards. And so we started Cospi in 2024. Talk a little bit about the progress that you've made since then. What is the kind of shape of the company today? It's been a huge ride, actually. It's a roller coaster. So we started, I think, about two years ago.

3:42So May, spring 24. Yeah, we started with a good initial investment of about 30 million from which we could hire an excellent team. So the team has grown to about 50 people right now across different geographies. geographies. So there's a headquarter both in Amsterdam and in Cambridge. Actually, the headquarter officially is in Cambridge, but the two initial labs were Amsterdam and Cambridge because Chad is from Cambridge, you know, I'm from Amsterdam. We now also have labs in London and Berlin. And we're also expanding into Asia and North America. Got it. And no surprise, your list of advisors is a bit of a who's who with Jeff Hinton and Jan LeCun at the top of the list?

4:35Yes. Yeah, the advisors are actually fantastic. So we have Jeff Hinton and Jan LeCun. We added to that also Martin Van Dambrink and Lord Brown. So Lord Brown is the former CEO of BP. And Martin Van Dambrink is the former president and CTO of ASML. They're both retired. and they like to spend their time with new startups and help them along. And then there's Verity Harding. She's working for the UK government and also DeepMind, or maybe formerly at DeepMind. And then Kristen Person, who has sort of initiated the materials project, and she's also advising on that side. Got it. So let's dig into the technology and the approach that you're taking to kind of apply your, this original set of work that you developed to materials.

5:38How did you get started with that effort? So I think the opportunity is, so to me, there's a deep fascination with the fact that there isn't sheer infinite amount of possibilities in which you can put together atoms. And the universe has only figured out so many of them because, you know, they form naturally, I guess, in the universe. But there's many more that you can design yourself with all sorts of exotic properties. When we started this company, both Chad and I were kind of concerned about the climate. We still are. And so we felt there's a strong need to accelerate the energy transition to more sustainable energy sources, as well as trying to take the carbon dioxide that's in the atmosphere out.

6:27So not many people know that by the time, but 2015, of course, we really like to be completely carbon neutral. But after that, there is still 50 to 100 years where we have to take out every year about half of what we currently put in. So that's 20 gigaton a year. That's an enormous amount. I think the size of the Lake of Geneva filled with sort of liquid carbon dioxide, a gigantic amount. And we don't have the technology for that because it's actually very hard to take it out because it's so dilute in the atmosphere. It's very expensive, too. And the two factors which are more expensive are energy and the material, the sorbit material that you use in order to take the carbon dioxide out of the atmosphere.

7:15And so we've been working, the first project we've been doing was improving the materials that take out this carbon dioxide from the atmosphere. And these are called metal organic frameworks. Actually, this is the material that this year won the Nobel Prize in Chemistry. And yeah, so we built a platform, and I can go into much more detail, but we built a platform that designs this material with the use of AI. So you can sort of think of it as a search engine, but it's not searching over existing documents. It searches over known and unknown materials. So it actually completely designs them from scratch if there's no suitable ones already available.

7:54And there's many components. It's a Gentix. So there is an agent sitting at the core of it who orchestrates a long computation. And it searches through existing databases. It generates entirely new molecules. and it also evaluates all these molecules with all sorts of tools. And in this generation and evaluation, you know, equivariance and all sorts of methods that we have developed over the years in my academic lab play a very important role. To be clear, you mentioned a molecule that won the Nobel Prize. Were you involved in the discovery of that particular molecule? No. No, I wish. Those were chemists who did that, who discovered it.

8:43I think Professor Kitagawa, Professor Yagi, and Professor Propson. I think those are the three. I'm not quite sure of the last one. But this is a class of molecules where there is a metal complex, a metal node with all sorts of atoms along it at the vertices of a graph. And then there is so-called linkers, which are also organic complexes, which are connecting these vertices on the graph. And they're extremely porous. So they have a very large holes in the middle with an enormous surface area. And so if you blow sort of air atmosphere through it, the molecules in the air, which is water, nitrogen and carbon dioxide.

9:30The carbon dioxide is only a small fraction of that. they tend to stick to the sides. And so then what you need to make is a molecule. You need to design a molecule where actually only the carbon dioxide sticks inside these pores and the rest goes through. So that's a design question. And also when it's full, you want to shake it or heat it to get it out so that you can actually reuse that particular material. What we added to this is a way to fine tune or design a molecule for a particular purpose. So people have actually made maybe around 100 ,000 of these molecules in labs and verified their structure.

10:18And so what we can do is we can now basically come up with an entirely new molecule for a very specific task and then make it in the lab and then use that for that. for that particular task. I should say, CUSP is not only working on MOPS. In fact, this was just the first set of molecules that we worked with. We have, hence, expanded to semiconductors. We're doing a lot of work on perovskites, which is materials for solar panels, improved solar panels. We look at semiconductors for new chip materials. We look at battery materials, fuel cells. We also look at removing PFAS from water. And the current set of molecules we use for that is, again, metal organic frameworks.

11:12Do you partner with other companies that have an interest in these particular molecules? or are you out exploring? And then if you find something, you will find partners and maybe license to them. What's the thinking around the business model? So we like to work with partners because there's a very broad class of materials and every class has its own super experts that focus on those particular areas. And they're either in academia or they're in companies. And of course, we also like to partner on the actual synthesis of these materials. So we partner with academic labs, but also with the labs inside of these companies.

11:50And so we build an ecosystem or a network where our engine can actually help in all of these different material classes, design the materials, and then we work with those companies to actually make it. So that's a partnership model. But there's also internal projects that we run. So for instance, the project on metal organic frameworks for carbon capture we ran self-funded internally. And then we also have another project now in the semiconductor side where we run it. And so if we discover something fantastic with self-funded, so we got the IP and then we can find customers for that IP. I think the most important thing is that we discover something that gives people certain confidence that we can do this.

12:41We actually really own this process and we know how to do it. And so then the cost that we work with customers to design materials for their specific needs. You talked a little bit about kind of the generative or agentic nature of this scanning scientific literature and, you know, using that to identify potential molecules. But, you know, we've also talked about, you know, some of the geometric implications of your work. You alluded to, you know, potentially the use of simulation. Can you talk a little bit more about the kind of end-to-end process of identifying these molecules? Yeah, happy to.

13:20So I guess there's a sequence of things that happens, right? So the first thing is, like in a search engine, you actually type a request. So you basically say, I want a material, and these are all the properties that it should have, and these are the things it should not have, these are the properties it should not have. And so you give this as a query, and then you could also tell if you have prior knowledge it's about how you want this particular search to happen, you could sort of tell the system, maybe use these tools and maybe sequence it in this way. So you can also give it some instructions.

13:55And then it goes through a process of steps. So the first step is it will look through its database. So we have a very large database of materials, which we have all ingested into this database. Lots of it is sort of exclusive licenses from the big publishing houses. and then they will start to look through all of this literature whether something exists out there that has these properties or which is close to having these properties. And so if it doesn't, then it will have to go into a new phase. So you can hold a conversation with this agent and talk about it. So it's already quite useful. But typically then the next step is that you go to a generative model.

14:37So in this case, it's the same generative model that generates images or video. And so, but in this case, it will generate molecules for you. And so you tell it, I want molecules with these very specific properties, the conditioning statement, it's called. And then it will start to generate these molecules, often hundreds of thousands of them, because it's quite cheap in the computer. And then comes the next phase, which is out of these generated molecules, we now have to sieve out the ones which look very promising. And this can be a very expensive step. In some sense, you build a multi-scale digital twin of the process that you really want these molecules to operate in.

15:25And so the first step is basically you relax the molecule to its ground state to make sure that it's the best energy state. Then you do a bunch of checks, like is it charged? If I shake it, will it fall apart? How big are the pores inside, if that's important? You know, all sorts of things that are easy to compute fast to throw away, you know, a whole bunch of things that do not look promising. And so then. So filtering it down. It's definitely a filtering step. Yes. And then go to the next step. So we have a pipeline that fine tunes or distills machine learning force fields for that particular material.

16:07So this is a process by which we take all the data there is about that material. We have a foundation model that's trained on a much wider range of data. And then we distill this force field into this. It's a very efficient force field for this particular class of problem. And we use that in an MD loop, typically the molecular dynamics loop, to simulate the molecule as it wiggles around and moves around, from which you can often compute very key properties. of that particular molecule those properties then often go into a partial differential equation at a higher scale and or in a process that actually models the device in which you want this up this material to operate and so that's again more expensive and so again you want to do this with fewer and fewer candidate materials and then at the very end you go to you know to an experiment right you go so now now actually do the experiment now that's more expensive and even slower um and that's you should do this only with order 10 materials at most um in the old way and so then uh so and then you get a candidate what we are currently the current i would say revolution that's happening in this space is self-driving labs where the amount of experiments you can do is much much faster so you could do maybe 100 experiments a day um and then the game is is more like the agent figures out what the settings of the experiment should be.

17:37The experiments are done, the data comes back in. And then you have the data from the experiment and the data from your computations. You combine them to set the next stage for the experiments. And so the experimental loop is much, much faster for those. And that's a very interesting development that we are now integrating our platform with. Got it. And so how many molecule classes and individual molecules have you kind of gone, you know, all the way through this cycle with? Yeah, we are engaged in a few of those, but, you know, the question is a little bit, what do you mean by all the way? The different materials are at different stages of maturity.

18:22So one of them, we went all the way to actually doing the lab experiments. Another way on semiconductors is on its way. And probably in a few months, we'll start doing the experiments. I meant all the way to lab experiments. And I was curious if you have enough data to say that your hit rate with this process is higher versus lower, or if there's anything you can say qualitatively or quantitatively about the candidates that you produce relative to, you know, the traditional approach? Yeah, so I definitely think that, you know, there's definite evidence that these things are much more efficient.

19:06So some of our scientists, they have said things like, we can do now in a few days what took a PhD before, but that's more in the digital domain. These are actually scientists that are more simulation-based work. We have two projects going which have experimental pieces to it, but we have a contract with a big national lab in Asia, which I cannot quite say the details of yet, where we have many more of these experiments planned out. So basically, the setup is that we bring some customers and we can do experiments in that lab, but they're lab scientists and they can also bring their partners and they can use our platform.

19:54And then we can collect data that way. And I think, oh, and then there is one other one that we're currently doing in Amsterdam on perovskites, which is running right now. So that's also a lab engagement. There is one. Perovskites is what? Sorry, yeah, proskise is a material class that's a semiconductor crystal structure that you would put on silicon typically on top of the normal solar cells. And that can help you filter out or basically convert a much larger amount of the energy in visible light to energy. Right. So we also have something with catalysis with DTU, which is the Danish Technical University.

20:39so that's on currently running on catalysis and we have a project that's about to start them off so there's quite a few engagements with labs that are either running or are starting to run but they haven't finished completely and does your lab or does CUSP publish are you active are you still active in kind of academic publishing around you know materials science now? So myself, I have one day at the university, and of course, I still work in that day with students, and there we publish everything. I'm very interested in all sorts of things which have to do with AI for science, so definitely yes.

21:21But also, COSP actually publishes. So we have interns that work with our scientists where we publish the results. So we, for instance, recently had results on machine learning force fields with uncertainty prediction. property predictors. And the most important thing, I think, that we've recently released open source and published a blog post about is our new molecular dynamics framework called COPS, confusingly. So that's universal particle simulator. And so the reason why we did this is actually together with NVIDIA is that, you know, Now, with this new development of these machine learning force fields that I talked about, these neural networks that replace quantum mechanical calculations for the forces, you cannot run them very efficiently in the current sort of MD simulators, the simulators that evolve a material or a molecule forward in time.

22:27because you need to run these neural networks on GPUs. And typically, these simulators, they don't run on GPUs, they're more on CPUs. And you want to run things in parallel. And so what was built by our team is a framework where you can compile these force fields into JAX, which is Python-based. and then it will actually very efficiently run these MD simulators and you can run them in parallel as well so that you use your GPUs. The utilization of your GPUs is high and so we released this open source a few days ago actually or a week ago at iClear and yeah we got a lot of excited responses to that.

23:15Are the representations of the molecules Like, are there standard formats for representing these things that someone working in the space would already have? And then your machinery just works on those existing ones? Representations, you can think of that as maybe a foundation model for chemistry. So what you do is, and this is also work that we did together with Meta. So what you do in that case is you train yourself one of these force fields. So you take a molecular structure. So that is basically the position of the atoms and their class, like if it's oxygen or hydrogen. And of course, you want this to be equivariant because rotations and translations don't matter.

24:04And then you map this into a latent space where they get represented by some code that is meaningful. And if you do a machine learning force field, you will then actually predict the forces and the energies from that. But you can stop at that intermediate level, and then you have a representation for that particular material. And you can train this on a very wide range of materials and chemistry. And from that representation, you can build property predictors. You can use it to condition your generative models. There's all sorts of uses for that foundation model. but it's also the starting point for distilling, let's say, models for specific material classes.

24:48And to be clear, are these foundation models trained per molecule or per class or are they very broad? Yeah, so that's a great question. So a foundation model, almost by definition, is trained on a very broad class of materials. The best data set for that is the materials project data set and Omole from Meta. So they have generated a very large number of DFT calculations, these quantum mechanical calculations, to create that data set. And those calculations are being used to train these force fields that I talked about, or those representations, those foundation models. And then if you say, but I'm actually interested in this particular material, What you can then do is start from that very broad representation, this foundation model, and then fine-tuned model for that particular material class.

25:44That way it is specialized for that material class, but that way it's also very fast. Because you need these things to do very fast. Because in a molecular dynamic simulation, you have to call them many, many times. One little step in an MD simulator is a femtosecond. It's a tiny step. And so you have to call them many times to make any progress. And you mentioned distillation earlier. Is that where distillation comes in? You're trying to get to a smaller model that's more focused on the molecules that you care about? Absolutely. That's what distillation is called. You distill the big model into a small model.

26:19In this case, the use of fine-tuning, how analogous is it to the process of fine-tuning in LLM, you know, reinforcement fine-tuning with traces, you know, language-based traces? Like, what is your data set that you're fine-tuning and your process that you're fine-tuning with here? So, actually, it's quite related to an LLM. In fact, the models we use are very related to an LLM. So you can basically think of the sequence, the information that you have. You can sequentialize it. And then from there, you can actually map it into this latent space. So there's either, typically you can do like a graph neural network, which actually looks at the three-dimensional structure, or you can use it as an LLM, like a sequential version.

27:13Like often molecules are represented as a, you know, as a string called a smile string. So then it becomes a one-dimensional string and you can use that to represent the molecule as well. So you can use both LLM style models as well as graph neural network style models for this. So your base model might be similar to an LLM, but you've got a very unique tokenizer in the case of representing molecules. It goes even further because you can basically also look at combinations of language and graphs and molecules. So you can sort of have, you can learn from, you know, let's say the literature where there's a lot of text.

27:54And then you can, every time a molecule is mentioned, you can then actually use the molecular representation, which is then the little graph neural network that represents. So that becomes a token. You turn this into a bunch of tokens using your graph neural network. And so you can really start to combine these things, actually. What do you see the work that you're doing at CUSP going in the future? Like what are the kind of near and midterm and longer term things that you're most excited about with that work? Okay, so we're constantly expanding the tools that we need for the different material classes.

28:32So we know how to train these tools, but every new material class has to be retrained. We're also going up the stack. So we're going into more coarse-grained representations, like larger scale digital twins, all the way up to modeling the actual reactor device in which this material is sitting. The effort that's most important for us right now is connecting the platform to self-driving labs. So to really generate large amounts of data from the self-driving lab, that's important. And I think ultimately it's very important to go through the entire process of predicting the molecule, making it in the lab, scaling the material, and then actually putting it in an actual device and getting a customer excited about that particular material and willing to pay money for it.

29:30So this basically means that this last part of scaling is something that still needs to be done. But getting there as quickly as possible is, I think, absolutely key. And so I'm really looking forward to discovering a material that is unique. It could not have been done by AI. and that makes it into an actual semiconductor device or something like this or a new solar cell. And we can say, you know, AI actually helped, you know, was a very important piece of the design of this particular material. I think that's a very exciting proof point that will help the entire industry forward. You also have a book that you're working on that has a really interesting title.

30:20It's talking about generative AI and stochastic thermodynamics. what is that what's the connection between generative AI and thermodynamics and tell us a little bit about the direction you're taking with the book right so maybe first a bit of a history on this so I started this almost two years ago now even before Cusp so I had a bit of a lull between you know when I stopped working for Microsoft and the startup Cusp I started so it was about six months between that and I started me and my wife were on vacation in Italy and sort of I cannot do absolutely nothing and so in the mornings I'm just going to write a book I had my cappuccino in the sun and I would start a book so wonderful, the best vacation you can have and in the afternoon we would hike through the mountains and enjoy life and so and then I was teaching a course in a town called Muizenberg, which is a Dutch word, but probably people say Muizenberg, but it's close to Cape Town in South Africa.

31:34There is the African Institute for Mathematical Sciences. And so I had the pleasure of spending a couple of weeks there teaching from these kinds of ideas that I was writing a book about. And then I recruited two students to help me out actually finish it because starting is easy, finishing is hard. And you need some help. And so it took two more years to actually finish it. It's a huge amount of work. But what is the book about? So the book is about actually the mathematics that describes modern generative AI, including probabilistic models, including diffusion models and many other things. That mathematics turns out to be equivalent to the mathematics that describes modern non-equilibrium statistical mechanics or thermodynamics.