Periodic Labs: A chatbot walks into a laboratory

18:10Dogus Cubuk:So this was back in 2010. I felt like we should see how machine learning can do physics. And back then, there was a lot of pushback because machine learning is probabilistic. It's biased. It's not always calibrated. And physicists, especially Platonist physicists, I guess physicists have different philosophical leanings. But the Platonist ones tend to think there's one reality, one theory, the beautiful ones, of course. So there was a lot of pushback at the time. But in 2012, when the ImageNet moment happened, and then deep learning got more and more prominence, by the time I finished my PhD, machine learning was everywhere.

18:44Dogus Cubuk:And actually, physics departments had to put machine learning somewhere in the grant proposal, especially in condensed matter physics, to even get a grant. So huge change in like five years. And during that time, Google Brain had this residency program. And I think it was like partially an idea by Ilya. It's a brilliant idea. They want to bring in people who have some machine learning background, some interest, but expertise in other fields to learn deep learning and contribute to deep learning at Google Brain. So that's how I joined. I was part of the residency program. And Liam also joined in a similar way.

19:14Dogus Cubuk:He'll explain himself. And yeah, so we were both doing deep learning research. I was at the gym trying to flip a tire. It was difficult. So I picked him as the strongest person I could find. Yeah, you saw the arms. This guy's going to help me. And he succeeded. We managed. and so I was continuing to do like regular deep learning research so you might remember this was an exciting time I would just publish at NeurIPS, iClear, ICML like deep learning was advancing with these conference publications yes there was a lot of fun to that and then I was still doing my kind of passion from before which is simulating solid state physics and then at some point Liam and our other friends Barrett, Luke have left and then started ChatGPT and I ended up staying and like really focusing on physics.

19:55Dogus Cubuk:So by the time, like before we left, before I left, I was running this large team of chemistry and material scientists at Google DeepMind. And we were doing a bunch of simulations and machine learning. Liam and I often connected and talked about physics and other things. And we felt like the technology had reached a place where we could really impact the market, the industry. So we left our respective roles to start the company.

20:17Liam Fedus:Amazing. Okay. What is your origin story. Yeah, origin. So I grew up in Maine and also had like a really early love for physics. And so in college, I was actually a bit between physics, chemistry, economics, and I was part of a dark matter detection program. And one of the key pieces for that was reconstructing the particle tracks, which became like a physics model. But it's really a physics informed machine learning model. And so I started doing machine learning in physics in a very early stage of still an undergrad. And then again, in graduate school, reconstructing particles, this time at CERN.

20:56Liam Fedus:Through this, we were pushing the amount of computation we're putting into this and, you know, some of the algorithms for machine learning. But after a while, it became clear, if I really wanted to push machine learning, I should be doing it in industry. And so I was actually at Google, part of the Google Accelerated Sciences team and then later Google Brain. So that's where Doge and I were meeting. I was working on a mix of some things in the science program, but then switched over to generative models, reinforcement learning, and really was just in this scaling mentality. It wasn't necessarily applying to physics, but just saw how impactful language models were becoming and was building some of the architecture necessary to scale those up.

21:38Liam Fedus:So this is using sparsity. So switch transformers, mixture of experts. This became an important architecture for serving today as models. Yes. And continued to scale that up, did some of the first trillion parameter models. But at some point, it became very evident that there is a huge possible impact for this in the broader world. Yes. In a product. And that's what led me and a few others to open AI. We felt it was going to be really a great time to start productionizing some of these things. And so at the time when we arrived, there was GPT 3.5 out as an API. GPT 4 was internally available, and we had some period to develop it.

22:21Liam Fedus:So we're working on that. And then at some point, we realized, let's just do a quick sprint using GPT 3.5. And it was a two-week sprint, and then we turned this into ChatGPT. And did that product do well? Yeah, it was okay. It exceeded some of our expectations. We had internal polls as to how many people use it, and people were guessing numbers like 10 ,000, 50 ,000. Yeah, like a high bar was like a million. No way. It was genuinely a low-key research preview. Your mental model must have then sort of like totally shifted, right? I mean, I guess that happened to literally all of us, but you were...

22:54Liam Fedus:Well, I mean, I think the context before that too is Meta had just released Galactica and had to withdraw it within a few days of release. I didn't remember. That was only a couple of weeks before. And there had been no successful chatbot up until that point. So the prior that this is going to be a valuable technology was quite low. It was further surprising because we had GPT-4 internally and it wasn't this hugely viral product within the company. So, you know, putting out a weaker model to the public, it was really just part of this iterative deployment. And yeah, obviously it worked better than our predictions.

23:29Liam Fedus:What was the period after the release like for you? Was that just like... Just totally frenetic. Yeah, chaos at all times. Yeah, I mean, like servers couldn't keep up and you're just trying to like rapidly improve it. You know, the early model couldn't do math, couldn't write very well, had many deep flaws, very different than the things we have today. Yes. But just a series of getting feedback, improving the task distribution, improving the data quality, just all these kind of boring details is what led to the system as of today. You know, very few people ever will have an experience like that, I suspect.

24:04Liam Fedus:But I imagine it serves you in very good stead as a founder today. What do you think it taught you? What did it give you? question everything. I mean, so it's really interesting. So people would come in years after the initial creation and it's very easy to anchor on a random decision made a year, year and a half ago, not necessarily like an optimal decision, but you sort of take that as sort of the assumptions that you're building upon. But I can recall back to, no, that was just one conversation, like one IC made that feel free to change these things. And as we were sort of building that up and eventually I was running the full post training team that sits between research and product, you know, we made incredible progress in the digital world, but Doge and I were connecting.

24:52Liam Fedus:We felt that the impact of these systems in the physical world was always going to be constrained unless we actually connected it to the real world. We just felt, again, reading textbooks, reading papers, deriving questions from that is a great foundational technology, but insufficient. And I think ultimately the main objective for artificial intelligence that we're really excited about is new scientific discoveries, new technology, something like really tangible you can hold in your hand. And so that was sort of the motivation leading up to Periodic. You know, you both have worked with some of the very best builders and researchers in this space.

25:29Liam Fedus:Why did you pick each other? Like, well, you know, what was it about Doge that you were like, you know, this is the guy I want to spend the next, you know, chapter of my career with? I mean, just absolute brilliant thinker. There's no one better in the world to do this problem. The Doge has been like at this intersection for 15 years more. Yeah. 15 years. Yeah. Right. And yeah, I mean, we've known each other for a decade. And so I think it was just obvious. There's no one better in the world to do it with. What about you, Doj? Why Liam?

26:02Dogus Cubuk:I mean, many reasons. So one of them is, you know, I feel like with a co-founder, the most important thing is trust because you're trusting your own career, but also the people you bring to the company. And, you know, there are some people in our company who I've worked with for many years. Liam has worked with many years and I wouldn't do it with someone who I couldn't trust fully. Another one is Liam, like he's been extremely successful on the LM side, but he's always been very passionate about physics. Whenever we talked about just randomly catching up, he actually has his ideas about quantum physics, quantum computing.

26:33Dogus Cubuk:So I think it's really important for the person to really care about these things. And I could tell that Liam really cares about physics. And when there's so many exciting things he could do with the LLM side, he would do this. And then, of course, the other part is we needed deep expertise in RL and LLMs. And again, Liam has done RL research and kind of contributed to Frontier for so long. So it was a very obvious, easy choice.

26:58Liam Fedus:You mentioned bringing people into the company. I think one of the things that, I mean, you can see it from the outside, but also talking to some of your investors in preparation is the talent density you guys have created is extremely impressive, especially for a company this young and still in the very early innings of what you're up to. how have you convinced some of the the very very best folks in in this industry who are being competed over um ferociously to to pick this as their mission i think mission yeah yeah like quite simply it's like people see just the ambition of what we're trying to achieve and the the possible impact of this like something that can actually design the physical world um something that understands physics chemistry is even more foundational than a system that understands language.

27:48Liam Fedus:And I think that really appeals to people. I think it really appeals to people too, in terms of how much leverage, how much of a diff on the world can you have in this position? And so while you might be a very successful contributor to a coding system, the coding system, there's always this counterfactual you can run, which is like, well, if I'm not the 500th contributor on this coding agent, what's the delta? And I think people are looking at that like, well, OK, I think my skills could be better used elsewhere and I can have a larger impact on the world. So I think that appeals to a lot of people, as well as some of the scientists, just seeing how rapidly AI is advancing in the digital era.

28:29Liam Fedus:So math, theoretical physics, et cetera, and wondering what could this mean for physical sciences? And I think that that's a huge appeal. I imagine it doesn't hurt that, you know, when you look at the best researchers in AI, so many of them have physics backgrounds. Like it really seems to have gotten a lot of that talent. So I imagine for a lot of these folks that you're talking to, it's sort of a chance for them to bridge their childhood love of what they were doing and, you know, their first passions with, you know, maybe where they've built some of their career. I don't know if that's how you've seen it.

29:00Dogus Cubuk:And definitely, like, I think there are two aspects to this. One, as you said, a lot of these researchers really deeply care about science. And this may be like a personal interesting. But on the other hand, they really want to have AI impact society in a positive way. And I think science is like such a good direction for AI because there's no end to it. For example, we expect that if we are successful with periodic, it will actually increase the number of scientists in the world. Whereas for certain roles, you know, AI, if it replaces the employee, it's kind of like a zero-sum game between AI.

29:34Dogus Cubuk:But for science, it's the opposite. Like the more we can show that science can impact the world, especially salt state physics, salt state chemistry, the more people will go into that field. And there's no end to it, as you know, like there's no end to science, there's no end to engineering, like we can figure out more science, we can visit more planets.

29:50Liam Fedus:Maybe let's get really granular to understand how this all comes together to the extent that you're, you know, able to share it. What does like one experimental loop look like with periodic? Like what are the different machines as part of that, the different processes as part of that?

30:05Dogus Cubuk:Yeah, so let's say we're doing powder synthesis, and let's say we're trying to make a superconductor. Yes. So the first decision is, you know, what precursors you want to mix, because the thing you're trying to make probably doesn't exist on your shelf yet. So we have to first get the ingredients, and the ingredients can usually be like these binary powders, like binary means two kinds of atoms are in it. It can be like the elemental precursor, which is the single element. But yeah, basically you figure out for the thing you're trying to make what precursors to combine. Is that you figuring it out or is it the AI figuring it out?

30:37Dogus Cubuk:Great question. So traditionally, it would be the human figuring it out. For us, the AI can actually really speed up this process. And especially, you know, down the line, if you want to run thousands of experiments a day, we can't have a human do it one by one.

30:49Liam Fedus:So when it's speeding it up now, it's maybe saying, here are 30 good options for you? Is that sort of like narrowing the scope of what you might look at and introducing new ideas?

30:58Dogus Cubuk:That can happen, or it can actually say, okay, for this goal you have, here is what you should mix. And we can try that. Okay, so you mix the precursors. Now you need to react them. Because at room temperature, these materials won't react with each other, so they'll just stay as the precursor. So there are different options, but a very common standard option is what we talked about, which is you put it in a furnace with some temperature profile. And Liam has often brought up that the temperature profile looks like a learning rate decay profile, right? Like in machine learning.

31:26Liam Fedus:And sorry, I'm just so interested in each of these steps. Putting it into the furnace. Is a human doing that? Is a robot doing that? Is that like already automated?

31:34Dogus Cubuk:That has already been automated by others before we started periodic. So there's already been attempts at getting a robot or a robot arm to take the mixed precursor, put it in that furnace. And we have both. Like we can either do that or we can also have a human do it. But again, as you want to scale these up to thousands a day, then you definitely need robots to do it.

31:52Liam Fedus:Yes. Okay. So it's in the furnace. Yeah. Then what happens?

31:55Dogus Cubuk:The very traditional path is you wait a certain amount of time you predetermined or the AI has predetermined, and then you take it out. And you hope that during that time duration and during the temperature profile, the reaction you hope happens happened, and the reactions you don't hope happens didn't happen. So now you have it, you cool it down a bit to not burn your hand or something. And at some point, you put it in a characterization instrument. For inorganic crystals, the method we use the most is an XRD, X-ray diffraction. It's the same X-rays you get at the dentist. It just happens that X-ray wavelength is of a comparable length scale as the distance between atoms.

32:34Dogus Cubuk:So then if you diffract X-rays through a solid, you can see what the atoms are doing and how they're positioned against each other. And then that tells you if your experiment succeeded or not. And that is one part, actually, we find that AI can play a big role because once you get the X-ray diffraction pattern, it's not always easy for a human to tell what you made. But the AI can help you with that and then tell you, okay, this succeeded or it didn't succeed and send it back to a new experiment. And then if it succeeded, maybe you want to see if it's a superconductor. So then you put it in another instrument that can measure the diamagnetic response, which tells you if it's a superconductor at a given temperature.

33:09Liam Fedus:Yeah, I think the XRD is a really interesting example too, because the materials don't come out labeled. Yes. Well, here is what you've done. Exactly, that's right. Right. So AI is allowing us to basically do the labeling and figure out what we're doing to see what we're doing at a much faster pace. Anyone could go into a lab and mix a bunch of powders and say they ran a lot of experiments. However, if you're not seeing what you're doing, there's no error correction loop. There's no intentionality to it. And so those are some of the first AI systems we had to build. And then that allows us to scale up the throughput of the lab significantly.

33:44Liam Fedus:Are you gathering sort of, I don't know, lots of ancillary data along the way? Is that useful? Is that not useful? I don't even really know what that would be. But I imagine a mechanized version of this process permits that in a way that a purely human version does not. I think like better telemetry is one aspect, but whether it's robots or humans, I think stitching the data together is one of the most important pieces. Interesting. So this is what I was alluding to at the beginning, where it said where most AI has been trained on the final artifacts of science. It's like the final paper, the final recount in a textbook.

34:21Liam Fedus:That isn't often how the scientific process unfolded, sort of a retelling of the story. and what we do is we're very careful in collecting what were sort of the intentions what was the hypothesis what were the computational predictions what was the execution through all of the machinery and this data that fully stitched together this full lineage is very very unique that's a something that we've been spending a lot of time putting together and then a system training is learning not just what the final outcome was but what was the process what were the judgments made? What were the reruns done as new evidence came in?

34:57Liam Fedus:Basically, what was the process of doing science? In a way, maybe this is a bad analogy. You'll know better than I do. But it feels like it's sort of almost chain of thought for science where you're sort of starting to see. Yeah. Yeah. Interacting with the real world. Yeah. Yes, exactly. As you've now started to run some of these experiments, what do you notice that having that trace actually gives you? Is it like, I don't know, better suggestions on the next experiment that you would have ordinarily got? Is it, you know, the next whole set of experiments is somehow leveled up versus what it would be if you started cold?

35:31Yeah.

35:31Dogus Cubuk:I mean, one thing is just like being able to pay attention to detail when there's so much detail. Like we've seen that the LLM can actually detect major mistakes and debug it. Like one thing that happened in our lab is, turns out the sample, you know, samples are put in this like circular sample holder. And turns out just we made a mistake and they were shifted by two. So now all the samples are mislabeled. You know, we have no idea what happened. And the LLM was able to just like go through it and then realize, oh, you just like shifted counterclockwise by two. So if you do the inverse rotation, now all of them are labeled correctly.

36:10Dogus Cubuk:Like something like that a human could do, but it just takes so much attention to detail and so much time. Now imagine doing these experiments thousands at a time. The other thing that's really, I think LLMs are really good at, which I'm really excited by, is they can be good at things that usually require different expertise and different PhDs. Like for example, our LLM can do simulations at the level of a PhD in simulations, but it can also do experimental data analysis at the level of a PhD. And that's very unique, right? Because most people either get a PhD in simulation or a PhD in experiment.

36:43Dogus Cubuk:and some things fall through the cracks. So the fact that we have a platform now that is an expert on all of these different modalities and can reason at the same time, so something that happened for us is the XRD result comes up and it's not what we tried to make. And it's also not a material that's in a database so we can't tell what we made. But then the LLM actually ran new simulations and then figured out what it must have been. And then we were able to validate it with the next experiment. Because I often say the people who should run simulations and the people who know how to run simulations are usually disjoint sets of people.

37:14Dogus Cubuk:And I think the LLMs will bring these together. And it's a fractal, right? So I'm just talking about one interface between simulation experts and non-simulation experts, but it's also true for like chemists and physicists, inorganic chemists and organic chemists. Like the expertise of humankind is kind of like a fractal and there are all these interfaces in the fractal. And an LLM that can speak the language of all these different groups, I think is like gonna be a qualitative change.

37:39Liam Fedus:I think another interesting thing to add, too, is as we build up this database of all the things run, all of the decisions, you can start constructing new types of machine learning tasks where you can envision snapshotting the state of the world. And you're like, OK, well, given the state of the world computationally, experimentally, what did the scientists do next? And then, like, what was the outcome of that? This is a very rich source of training and reinforcement learning. And like one analog that's been talked about before is let's train an LLM up to general relativity and then see if the system can like produce such a thing.

38:14Liam Fedus:And we're kind of doing that, you know, in our scale, in our own experimental data, our own computational data. And so this is like a set of data that really confers like a huge advantage where that just doesn't exist anywhere. You mentioned this idea of sort of a snapshot in time, which I think is such an interesting idea to maybe go a level deeper with that. Like what might be a snapshot periodic would take and what would you get out of looking at that? So let's just say hypothetically you had 100 ,000 experimental data points. What you can do is you can say, what was the state of experimental evidence at some date?

38:52Liam Fedus:So you can just literally say, these were our computational predictions. These were the experiments we'd run up to that point. Here's some notes and thoughts on what we were thinking. And from that point, that becomes your new environment state. and you'd say, now, given this basket of data, what did the scientists do next? And given what they did, like, what was the outcome of that? What were the new empirical measurements? This becomes a incredibly interesting source of reinforcement learning type tasks and data that is incredibly hard to construct outside of periodic. And this is really interesting because now you can build this synthesis superintelligence to begin to predict what are the outcomes based on all of this data.

39:36Liam Fedus:And it's easier for us to do this type of work because as we accrue more evidence and more data that doesn't exist in public literature, we know our model isn't contaminated. And there's a risk when your pre-trained model already knows the answer and you do reinforcement learning derived against an answer that was already in the pre-training corpus, that the system can basically fake some work, get to the right answer, and you reinforce those strategies. Strategies like that aren't going to generalize and aren't going to be particularly interesting. So we're able to circumvent that by having our basket of data that we know does not exist in literature.

40:11Liam Fedus:When I think about what you're doing, there's obviously so many different pieces to pull together. What does it not make sense for you to do? Do you need to develop every piece of hardware yourself? Or, you know, can you use someone else's robots? Will you, you know, create your own models ultimately? Or can you sort of just fine tune and do these different things as part of the process? Like, what do you feel like you need to uniquely own?

40:39Dogus Cubuk:You know, for the hardware things, like if there's an off-the-shelf component we can just buy and use, we're very happy to do that. You know, we don't have to reinvent the wheel. So we usually do that. Turns out for certain scientific instruments, the throughput isn't there or the quality isn't there. So for those, we actually invest in hardware engineering expertise to improve the throughput, et cetera.

41:00Liam Fedus:Like what's an example of something that just is not going to hit it at your scale that you have to sort of reinvent in a way?

41:06Dogus Cubuk:I mean, the one that already kind of matches our needs is XRD. So there's already so much work into high throughput XRD that we have these instruments that we can put, you know, 40 samples at the same time, and we'll do X-ray diffraction, all of them at the same time. But for things that are more materials property oriented and it's like slower measurement takes longer, you might get like one sample per hour, that could be a big blocker for us. So then we invest into like new IP, new engineering to make it work. For simulations as well. So for simulations, for certain things, there's already a really good code base, so we don't have to improve it.

41:44Dogus Cubuk:But for certain things, it's just not there yet or it's not scaled enough. And then we invest in the software engineering expertise to make it happen. And on the LM side as well, which, yeah.

41:54Liam Fedus:Yeah, I mean, I think right now we spend very little time trying to improve software engineering, right? So we can use these systems to build up our code bases, help with these types of analyses. And we are absolutely beneficiaries of just how good Codex and other things have become. What we're targeting now are areas where the frontier is just not sufficient or we can do it vastly cheaper. This is where we're focusing our efforts. We deeply believe, and I think I've seen very good evidence, that when you actually start training against this data, you can make the reasoning more efficient. You can kind of add in new types of capabilities and go beyond what inference time thinking can do.

42:36Liam Fedus:Compressing this knowledge into the weights, compressing these strategies into the weight is just fundamentally valuable. And that's why all the frontier labs continue to do this. And we didn't stop at GPT-4 and then just do inference time from there. compression really needs to happen and you get different types of patterns that emerge. So for example, think about tool use. There's declarative use of tools where you're like, okay, this is how you use the tools. These are the specs. These are the inputs. These are the outputs you should expect. But there's a difference of that versus the procedural, the intuitive knowledge of like, oh, I should use this tool in this situation.

43:15Liam Fedus:Yes. One of my friends had used the example of, you know, you could like read a manual on like how to play a saxophone versus learning how to play the saxophone. So really want to build that into the weights. How interesting. Don't duplicate things where the frontier is already sufficiently capable or cheap enough. We've had a few instances where because we're pushing the throughput of the lab so much, there are some analyses that just would have been prohibitively expensive. At one point, one of our engineers was coming to us and said to do this analysis, our max throughput of our second lab, it would cost roughly$30 million a year.

43:51Liam Fedus:And so that was a large chunk. So we were looking at how do we actually compress this into smaller models, make it vastly more efficient. so you take it from like 30 million dollars down to like you know like a sub-million dollar cost and that allows you to reinvest that capital elsewhere into other machines or training or simulations so this is the type of work that we've been doing we realized that we could do conduct very high throughput experiments and they're actually fairly low latency in a lot of areas of science there can be lower signal to noise uh they can take days weeks sometimes months this is a huge motivation for like some of our starting points.

44:29Liam Fedus:Wow, interesting. So that allows us to build up this huge quantity of data and also maintain the quality, maintain the diversity. And that becomes this really incredible asset. Yeah, the diversity piece was something I was interested in, maybe because I don't have enough understanding of superconductors. But how do you make sure you are doing these different modalities, like getting the data that is sufficiently diverse and not sort of overly specializing?

44:51Dogus Cubuk:Yeah, I'm sure there's definitely a trade-off, right? There's an obvious trade-off to consider, which is like, if you make it infinite, as diverse as possible, now these experiments don't learn from each other. You haven't really gone deep enough in any modality. Yes. Okay, if you make it really narrow, now there's a big problem because if that particular modality doesn't lead to a new superconductor, you're hosed, no matter how well you do, right? So we're trying to kind of walk this trade-off correctly. And, you know, one of our guiding principles is actually pretty technical. We want to make sure the different modalities we consider benefit each other.

45:23Dogus Cubuk:Because there are certain scientific experiments that are different attempts, like different synthesis methods. But actually getting good at one also helps you get good at the other and vice versa. So then we can build our LLM training, simulations, kind of our hardware in such a way that they benefit from both of them. And that's been really good. I think that's how we kind of build our experimental diversity while also really synergistically benefiting from each experiment.

45:50Liam Fedus:Yeah, to use like ML parlance, we want to have positive generalization. Does that stem from your intuition and sort of knowledge or is that also AI assisted? Like where does the human have to play in there?

46:04Dogus Cubuk:So originally it was mostly human intuition. But we have seen indications of LLM actually contributing to this already. So sometimes they will say, oh, you've done this experiment, this synthesis method. But, you know, if you were to add this, now this is a new modality, but it could help. So it's not like a massive hop in the synthesis methodology space, but it's a small difference. And it will usually actually have a pretty good understanding of the literature. So like what kind of synthesis attempts have been done for this chemistry? And then it can also guide us that maybe you should try this next.

46:36Liam Fedus:What is the right way to monetize this sort of product? Like, does it become the case that you should develop, you know, these discoveries yourself and start selling them? Does it make sense that you're doing more contract work? What's the right model to start and what is the right model at maturity look like? Maybe an interesting analog would be what we're seeing in software engineering. So in software engineering, when we first produce ChatGPT in these early models, we didn't promise full-fledged output of like software. It's like, okay, we're going to produce a Kubernetes replacement for you, and we're going to monetize that directly.

47:11Liam Fedus:It was a co-pilot to accelerate a software engineer and get them to solutions more quickly. Now, fast forward a few years, we're starting to talk about monetization of outcomes. So, sort of a new concept in AI. And I think a similar thing is going to unroll here. So, the tools we've been building up in our own discovery loops are hugely valuable to many different folks in industry. And we're basically able to create custom systems for partners, for customers, to help them get to their solutions much more quickly. But in parallel, we also are kind of talking about these outcome-based pricing, where then we retain the IP and the upside of the final output of our systems.

47:57Liam Fedus:Ah, there's also another interesting piece. Like, so for example, if the software has in its weights, the recipe to a room temp superconductor, you don't want to monetize it through, say, a software deal. You want to retain that IP. Yeah. So those are sort of the tradeoffs that we have to balance. Are there analogies from other industries that you end up referring to? You know, I imagine that some part of the current scientific apparatus is useful to learn from. But I could also imagine different forms of manufacturing have taken this much further, whether that's, you know, creating a computer or an iPhone or, you know, a car or whatever it might be.

48:34Dogus Cubuk:I mean, so one thing that's something very relevant to us is the drug discovery field. You know, there was a time where if you discovered a drug, you couldn't make a lot of money from its IP. But that changed. Like today, you can discover a drug, maybe go through some clinical trials, and then get paid billions of dollars just for the IPO that. So the field kind of changed, and their capabilities changed for that market to exist. I think materials is kind of at that point now. Like you can make a lot of money by selling materials. But materials IP doesn't really capture the value it deserves. But it could go through this transition similar to drug discovery.

49:08Dogus Cubuk:and we feel like as predictions get better, IP will have more value. So that's why we're really trying to close the loop all the way from the discovery to synthesis and scale up of synthesis. So that's one of the analogies, I think.

49:24Liam Fedus:Before we press the record button here, Doj, you were mentioning the sort of difference between LLM performance on mathematics versus science. Yeah, maybe you can tell us a little bit about that.

49:35Dogus Cubuk:Yeah, so this is something we're very passionate about. is something that we kind of like predicted and kind of one of the reasons we started the company, which is that, you know, LLMs, if you talk to them, they're incredibly intelligent, right? You're blown away. And on certain things, their performance is insane, like coding, math, theoretical computer science, amazing. But then you haven't seen that kind of impact on the physical world. Why is that? And, you know, we feel like one of the reasons is with math, theoretical computer science, and coding, the reward is instantly verifiable. Like you write code, you can emit it to a unit test.

50:08Dogus Cubuk:You have a mathematical theorem, you can validate it. And another reason is all the context is available to the LLM. Like if you're writing code, all the code the LLM can see. If you're doing math, all the axioms and all the corollaries you can see. The physics is, it will never be like that, right? Like for example, when I do an experiment, there are more than an Avogadro's number of atoms in there. There's no computer that can store all the positions of atoms. So we have to come to terms that in real life, LLMs will never have full context. But to be honest, outside of math, almost everything that's interesting is of this type, where the full context will never be available to you.

50:44Dogus Cubuk:Like, think about politics, finance, literature. Like, what makes life interesting is when you don't have all the context, but you have to make decision uncertainty. Physics is very much like that. So if LLMs will actually play a big role in the real world in the physical space, it has to figure out how to deal with missing context, not instantly verifiable rewards, some noise. And we feel like that is a very exciting future to be part of. So that's why we're working on this.

51:12Liam Fedus:I love that. That's so interesting. And it's like the strategies that are optimal under math, the strategies under optimal under software development may not be the ones with the sort of decision making under uncertainty. And so that's the vision we're pushing forward here. To what extent is periodic's fate sort of tethered to that of the foundation models? Like if we, and this is clearly not going to be the case, but if we didn't get another model from this point forward, like how far do you think you could push things? I mean, I think from the software engineering side, we could literally stop today and find every existing Nobel prize, at least from the physicals.

51:49Liam Fedus:I mean, a lot of the physical sciences are relying on simple analyses, it's not competitive programming. So I think from that perspective, it's very good. So I mean, we are huge beneficiaries of that and I hope they continue to improve rapidly and I expect they will. But I view that as like not a blocker for us anymore.

52:08Dogus Cubuk:Yeah, absolutely. I mean, there's so much to do on experimental analysis, simulation scale up, data analysis, and the current models and our models are already good enough. Let's say like AI is very popular today, but let's say tomorrow is not for some reason. There are certain things that happen in physics that will never go back. So one of them is force fields. Have you heard of this term before? No. So, you know, atoms are, like especially the atoms that we deal with are governed under simple quantum mechanics. And you can do quantum mechanical calculation to calculate the forces on each atom, or you can have an empirical model that tells you how much force will be on each atom.

52:42Dogus Cubuk:And it's called a force field. It's a very old topic. We use it all the time to model materials, molecules, et cetera. Force fields change so much because of GNNs, like graph neural networks, that I don't think it will ever go back. Because now we can do very general across the periodic table models. In the past, you could only do like three to five elements. It's a tough day. So I feel like machine learning has made such an impact today that it will never go back. And as Liam said, today's capability is already enough to really advance science and engineering. But of course, we suspect that they'll actually keep getting better and then we'll keep benefiting from it.

53:20Liam Fedus:You know, clearly, I think we all agree that these models are just going to keep getting better. So planning with that in mind, what changes for you? What do you expect to get unlocked that isn't unlocked for you now? Some things that we're really excited about is just really advanced software engineering. We do a lot of very advanced simulations and we're starting to do a lot more work where we construct more advanced simulations that we can control every like bit of code and these are things that would have taken teams you know a year or two yes now it's like an engineer's weekend time wow and so i think this is something that we're extremely excited about and especially an area where the ai system is able to write software write simulations that are maximally consistent with like experimental evidence.

54:09Liam Fedus:It still feels like what you're trying to do feels very differentiated in this market. Like, you know, this is at the moment max mimicry in many cases, right? You know, someone comes up with one good idea and there's going to be sort of five other folks and that's probably healthy in many respects. How long before you think that starts to happen to you and who enters the fray? Like, do the big foundation model companies start to say, actually, this is like invaluable data and we need to be spinning this up ourselves? Where does that competition come from in your view? I would suspect in the next couple of years, a system that can literally engineer matter is going to be of key importance to everybody.

54:49Liam Fedus:And I think that's sort of the motivation for moving as quickly as possible, building out a huge network of labs, of infrastructure, building simulations and AI systems that can control and orchestrate this. That's our belief. I'm curious here.

55:04Dogus Cubuk:I mean, I do think that we're pretty unique in the team we gathered and the approach we're taking. But taking a step back, there are now actually a lot of people working on this. Like almost every week now, I hear a company in the AI for science space, which is great. Again, if you really believe that science is endless, we really want just more and more smart people, dedicated people to work on this stuff, which is great. But yeah, I mean, looking back, I mean, maybe one thing is people are seeing how much LLMs can change the digital world. and they kind of wonder how much it could change the physical world.

55:36Dogus Cubuk:So it's not a surprise to us that a lot of people are getting into this field.

55:39Liam Fedus:You two are co-CEOs, right? Yeah. How do you sort of divide and conquer? What are the swim lanes that you naturally take? So very naturally, based on our expertise, I spend a ton of time on the AI LLM systems. I think that's been a really core piece of my thinking on a day-to-day basis, as well as thinking about like, okay, how do we monetize these AI systems in the real world? Yeah, there's sort of a natural division there. Yes. Yeah. I always like to wrap up with sort of some more abstract questions. Be curious what both of you would say to this, given what you do. If you had unlimited resources and no operational constraints, what is an experiment you'd really like to run?

56:20Dogus Cubuk:I would love to do reaction colorimetry. Like there's an experiment you can do that tells you the formation enthalpy of a crystal. So what that means is how much energy would it take to take this crystal and remove it, like remove the bonds and remove the connections. The reason that's really nice is that's a single number you can get, like formation enthalpy at some temperature, that you can directly compare against simulation. It's usually hard to find an exact comparable between simulation and experiment. It's usually convoluted, narrative-based. But here it's direct. But it's just like a very complicated, expensive experiment.

56:55Dogus Cubuk:But we had infinite money. I would love to do that, like thousands of them. So there's some data from NIST, you know, the Standards Institute in the U.S., and they've done these experiments for hundreds of materials, and it's like invaluable for cell state chemistry field. So if you had infinite resources, I'd love to just really scale that up and get so much formation enthalpy data that we can compare with simulation and machine learning.

57:20Liam Fedus:Amazing. What about you, Liam? I would love to see if we had access to a full fab, how quickly you could ramp up, say, like a new new generation of chips. So, like, how quickly could you overcome the materials engineering problems of like of logic, of memory, of processing? And you say, OK, here's the new generation. How can we take the yield that would have got taken like many, many months to ramp up and accelerate that significantly? So I think those would be incredibly interesting experiments. Oh, fascinating. Okay, last one. If you could recommend a book to everyone on Earth, what would you want to recommend to people?

57:59Liam Fedus:I would say probably a popular one in Silicon Valley, but The Beginning of Infinity by David Deutsch. That is a popular choice to this question. I still have not read it. I don't know what my mental resistance is to it. I would recommend it. Yeah, I think it's very much like a spirit of optimism. You know, it kind of goes to, there's this point about like the unboundedness of science. So yeah, I really enjoyed that one. Yeah, I think I read part of it at some point. Yeah. Anyway.

58:27Dogus Cubuk:I really like this book called Subtle is the Lord. It's a biography of Albert Einstein by one of his friends who's also an exceptional physicist, Abraham Pace. And the reason this is unique is, you know, most biographies are written by people who are not an expert in the topic that they're writing the biography of. and there's usually like a mismatch in terms of like what they understand and what they should say this book is incredible because it goes through Einstein's life but with the very precise technical detail about what Einstein did so it's like a incredible friendship book because they're friends and like this guy wrote this incredible book for his friend but it's also an incredible physics book so if you're interested in friendship or physics it's

59:06Liam Fedus:that's a great combo I love that amazing Liam Doge thank you so much this was a real pleasure thank you so much for having us super fun here. That's it. Thank you for listening to this episode of The Generalist Podcast. Please subscribe on Apple Podcasts, Spotify, or your preferred podcast app. Ratings and reviews help others discover these discussions. So if you enjoyed the conversation, I'd be grateful if you could take a moment to leave one. For all past episodes and more, visit us at thegeneralist.substack.com. See you next time as we continue to explore the future. Thank you.