🔬ESM: The Bitter Lesson is Coming for Proteins - Alex Rives, BioHub

27:47Yeah. Yeah. I mean, I think it's a really interesting question, kind of to what degree can you interconvert these models? I don't think kind of that's fully understood yet, but I think that's a very kind of promising direction to think about doing that and what are the right ways to do that. So ESMC is also approaching programmable biology, but I would say in a very different way. It's approaching it from this kind of world modeling perspective, where the idea is basically you have a predictive model and, you know, you're going to search the world model to find protein molecules that satisfy kind of whatever design criteria that you have.

28:25So we've been able to use this to actually now go and design mini protein binders. But I think sort of most excitingly, we've been able to use this to actually design antibodies, SCFVs. And we're seeing really, I think, exciting success rates in a small number of files now. Yeah. So can you explain what those SCFVCs are? Yeah, yeah. So an SCFV is basically, it's a single chain antibody. So it's a kind of therapeutic modality that basically has, so an antibody has a heavy chain and a light chain. And then it basically has a pair of, you know, one heavy chain, one light chain, another heavy chain, and one light chain that come together to recognize a target.

29:09So there's different variations of these kind of modalities that are used therapeutically. And so what's interesting about the SCFE is it has one heavy chain and one light chain. So it is able to kind of form these very complex binding interfaces where, you know, you can kind of have two different subunits coming together to engage a target. These are kind of important therapeutic modality, something like, I think, a quarter of new drugs are antibodies. So it's really, I think, you know, one of the critical modalities for medicine. And basically what we're able to see is that, you know, you can search ESMC and you can actually find antibodies that are reaching the level of affinity there, I should say, are really at the level of affinity that is needed for therapeutic function and activity.

30:01The protein design space has kind of exploded in the last five years. You know, everyone is doing protein design. You know, many people are excited about protein design. And my kind of high level naive understanding of the field is that things like mini binders are quite doable. People have done that, you know, quite routinely successfully, you know, in smaller. By the time you get to like nanobodies and SCFEs, they're a little bit harder to design. And then antibodies are still actually quite out of reach oftentimes. times. One of the common reasons, you know, for this is if you're in the alpha fold paradigm, you don't have MSAs, right?

30:40The evolutionary pressure for antibodies is actually the opposite in many ways of what the evolutionary pressure is for everything else. They go for diversity rather than trying to evolve along a very constrained path. So I'm curious, did you try larger structures? And is that something that you've seen success on? Or is this something that you still think for some reason it might be hard to do? We can actually take the SCFVs and reformat them as antibodies. So I think that would be kind of the quickest approach to do that. We've not tried full IgGs. I don't see any reason why that wouldn't work.

31:18Actually, it's something we haven't yet. You know, we've decided we're basically kind of releasing this now because we feel like it's kind of reached a point where, you know, we're seeing, I think, a really significant against step above kind of what's been possible in the past. And so we just, we wanted to get it out there, but, you know, I think there's a lot more progress that's possible. So we're, you know, we have a lot of collaborations to kind of look at some of the other applications here. You know, the thing about it, right, is it's a general model. So I think to me, that's the most exciting thing about it is just, you know, a general model for protein sequence, structure, and function, you can search it and, you know, therapeutic design basically emerges from that search.

31:56Yeah. I mean, to me, you mentioned that you're not using MSA's multistimus alignments, which was one of the, or maybe the critical insight that allowed AlphaFold to work really well. And the fact that you didn't need that in order to make it work basically as well as alpha fold three is really exciting to me because that means that your thesis of let's cover the space of possible proteins as well as we can and see what the emergent behaviors are. So that if this is an emergent behavior that we're kind of able to replicate what happens with multi-sequence alignment, when we have used multi-sequence alignment, what are the other things that maybe we don't have data for, but that we are able to also do in an emergent way?

32:43I would say, actually, we're doing significantly better on antibodies. So I think that's one of the things that's really cool. That's one of the theses that we had is antibodies are not going to benefit from evolutionary information probably in the same way that kind of predicting the structural topology of a molecule will. So I think you kind of see that now where the representation space is containing something that's really interesting about antibodies here. I want to talk about, because you mentioned something very interesting to me, which was talking about virtual cell and how this maybe interfaces or did this work here.

33:19I'm really interested to know, were you able to find other things in your mechanistic interpretability? What were some interesting things that weren't just validating biology, but there's a pattern that was unexpected? Did you find anything like that? It's complicated. So, because we have to now actually go and validate some of these things, right? Sure, yeah. I think what we saw are like interesting connections, right? So, you know, what we can see, for example, is that kind of distantly evolutionarily related gene editing systems clustered together in this space in ways that are consistent with and kind of reflect our knowledge of the origin of those gene editing systems.

34:01So that's really exciting. but kind of the thing is, right, there's a number of proteins that are in that map that are kind of brought together in different ways where we just, we don't know what they are right now. We don't know what they do. So one hypothesis there is, well, these are kind of novel gene editing systems. I think in this atlas, you know, there's going to be some really interesting basis for scientific discovery there. And if you think about kind of how people go out and look for new gene editing systems. For example, they're typically mining the large genetic sequence databases, and they're looking for kind of different sequence patterns or structural patterns that are linked to that.

34:38Actually, the first version of the ESM atlas was used by Feng Zhang's group to find a new gene editing system. So I think there's just a lot of biology out there that we don't understand that's waiting to be discovered and kind of being able to connect the dots between proteins so that we can go from, you know, what it is that we know today to kind of make those inferences about the unknown. So, that's what I'm excited about. And I think, you know, there are proteins for so many applications that nature has probably invented. You know, you think about the thermostable polymerase, which enables PCR, which came from a bacteria living in a thermal hot pool.

35:18You know, you have, there may be the solution to climate change, you know, somewhere in protein biology. There are probably all kinds of building blocks for completely green chemistry infrastructure out there. There's probably new medicines and therapies, you know, but the question is, how do you find those? And so I think, you know, kind of being able to connect the dots is really one way to start to be able to open up that space of protein biology to discovery. I'm curious, you've, one of the advancements of ESMC is an improvement in multimer, so basically protein-protein interactions. That structure predict.

35:54The ability to predict the way two proteins interact, I think you now claim to do better than anyone else, right? If I'm correct, I'm wrong? Yeah, I mean, I think we're state of the art for open models. Okay. One thing which I know some people would find very useful for a virtual cell is just an entire mapping of every single pair of proteins inside the human transcriptome. Have you thought about doing this in terms of like kind of a beginning to a virtual cell, like create that map. So I think something like that would be really valuable. I think, you know, fast. So the other thing about ESM Fold 2 is a really fast model because it doesn't require the multiple sequence alignment.

36:32So, you know, you can do inference kind of, you know, directly from the sequence. It takes seconds. You know, you can get an atomic resolution prediction. So, yeah, that's, I think, one really interesting application. at biohub i mean the other thing that we're thinking about is can we actually experimentally resolve this and so one of the things that we are we are building is cryo electron tomography and and we're really building systems that can greatly increase the contrast when you're looking at you know at the cell at the atomic level and so i i think one thing that i i hope to see is actually structurally, empirically resolved interactome at some point in the future.

37:16And I think there are some pretty big technical hurdles and technologies that have to be developed to overcome that. But I think that's something that's going to be possible. So we can use computational methods to start to get the proxy of that. And I think, you know, that's going to be really powerful. But I think a lot of the future of structured prediction is going to turn into structure determination, actually, you know, really bringing together these kind of tools that we have for modeling proteins and bringing them together with experimental data so that we can start to, you know, develop this picture that's, you know, informed by empirical biology, by what we can observe.

37:53So, is that the vision here, if I'm understanding correctly, is that you have maybe lab in the loop kind of thing where you have an agent that is talking to your, you know, C7 and whatever and then it predicts a property that you're interested in it sequences the the the genome or it creates the genome it creates the protein from the genome it then observes it with some version of this microscope what did you call the microscope again that's a cryo electron tomography okay okay and then you do whatever experiments or you observe it and then you use this as a lab in the loop to, oh, okay, this folds this way, therefore I want to check.

38:36The next one that I want to check is actually a different one and use the active learning system. Is that sort of the vision that you're articulating here? Well, I think there are going to be a few fundamental principles for the next era of biology. I think it's, yeah, it's such an interesting time right now because I think we're really, we're at the beginning of a new scientific paradigm is really just the beginning of it. And so what is, you know, what is defining in that paradigm, right? And so I think there are a few principles. You know, one is scaled data generation. I think that's going to be really critical.

39:12The second is computational, you know, predictive digital representations of biology. And we can kind of talk about, you know, you can think of ESM as being, you know, first generation, AlphaFold as being a first generation of those kinds of approaches. And so you can kind of start to think about what does that look like as we can model more and more biological complexity in that way. And then you have the principle of feedback and you have the principle of, you know, we have intelligence now that's scalable and so can be applied to every unit of a biological problem. What would it mean for all of that to come together?

39:47So I think we're going to have increasingly capable and accurate digital representations of molecules, genomes, cells, ultimately physiology. That's where you want to get. We're going to have to go up that complexity scale, the levels of biological complexity that requires traversing a data barrier. There's, I think, data that does not exist that needs to be generated to achieve that level of predictive fidelity. And then we're going to have reasoning. And I think, you know, what that will mean is that we can reason over thousands, millions, hundreds of millions of scientific hypotheses in parallel digitally using predictive oracles, which can, you know, actually predict the outcome of an experiment.

40:43So the scale that we can ask questions and the kinds of questions that we can ask will just fundamentally change through that. Feedback is going to be critical. You know, the models are going to need to, there's going to be sort of a scaling dimension of this, which is, you know, building the data to have those accurate representations. and then a feedback dimension where the models can learn from biology, can reason digitally, can reduce that to a small number of experimental hypotheses, examine the outcome of each of those experiments, update their understanding, and build knowledge in that way.

41:20So I think that's what it's going to look like. And we kind of have to build each of those components. What Biohub is really trying to do is to kind of bring together the experimental and the technology layer that will actually allow us to have these AI models interact with the biology and do experiments. And I think it's, you know, it's Amdl's law, you know, we see incredible, incredible advance in areas where we can get feedback computationally. So in closed domains, but of course, you know, experimental biology is completely open-ended. And so the feedback principle there is going to be very different.

42:00But, you know, something, there's going to be something like RLVR, you know, with experiments where we can, you know, have models that are just really building knowledge and learning from that knowledge and being able to develop more and more accurate representations. charts. You're the head of science science at Biohub. Maybe fun fact for those who don't know, the science section of Leighton Space was basically launched after or in response to Mark Zuckerberg and Priscilla Chan on this podcast about six months ago. It's actually very exciting to have you here and kind of come full circle. And Mark laid out quite an ambitious vision for what Biohub wants to accomplish.

42:39And I think you just laid up a very natural successor to that. I think you had just joined at like, you were like, there are two weeks. I joined, yeah, yeah, at the very end of October and watched at the beginning of November. Yeah, yeah. One thing I'm curious about is, in your eyes, you know, where is Biohub now? Like, what do you want to, you know, accomplish? What are your goals, big picture goals for listeners who haven't watched that episode with, you know, Mark and Priscilla? We recommend, of course, that they go watch it. Link in the description. And then have you learned anything even in just the short time of six months you've been here?

43:15And like, has the vision evolved? And, you know, where do you see this going? I think we can, you know, how does ESMC fit into this? How does, you know, the virtual biology initiative that you recently announced fit into this? And then I think there's like several other things that you're working on that we haven't even touched on. Yeah, well, I'm learning things every single day. But the way I think about it, we're building a scientific institution for this new paradigm. And, you know, to do that, you know, it's an institution that's going to be powered by frontier experimental biology, frontier technology for measurement, for observation.

43:55And it's going to be powered by frontier artificial intelligence. And this is all open source, right? It's a philanthropy. So our goal is to accelerate science. Our mission is to cure or prevent disease. And so to do that, you know, our belief is that there's a fundamental gap in our understanding. And we need to accelerate science to traverse that gap. And so we're really thinking about every layer of biological understanding that goes from the most basic, you know, level, like the, you know, the atoms of a protein in a cell, all the way to systems of cells in physiology and disease. And how can we create models that can capture that complexity, can allow us to understand that complexity?

44:41And I think, you know, if you think, what is the cure to disease look like, right? It's not a pill, right? It's not a medicine in the conventional sense. You know, it's going to have to be a system that is capable of modeling and understanding the underlying physiology of disease in a way that's differentiated for every single human being, for every single different genome. and it's going to have to be able to link events all the way at the molecular scale to the manifestation of disease in physiology. So it's an incredibly complex, incredibly hard problem. And for us, you know, we're trying to ladder up those layers of complexity and we're trying to build kind of the foundational tools that scientists can use to, you know, answer the fundamental questions there.

45:33And so we're creating atomic level imaging. We're creating light sheet microscopy that allows us to observe, you know, how all the cells move and develop in a developing organism. We're creating spatially and temporally resolved maps of inflammation. We're creating, you know, cellular programming and immune cell reprogramming to be able to actually design completely programmable therapies. And, you know, we're creating these digital representations at each of these layers so that we can accelerate the science, simulate what's happening, you know, make biological matter and make proteins and cells and genomes programmable.

46:14And I think all of that has to come together. And I think if you have the focus and you build, you know, the biology and the computational layers together so that they're tightly integrated, You know, that's how we're going to make the fastest progress. For the last 10 years, I think we've been one of the big champions of open science. You know, we're an organization that does, we both fund and we build. And in our funding, we've always supported open science. And in our building, you know, we've always done open science. So that's something that's going to continue. It's just really fundamental.

46:50We're not a drug development company. We're not trying to generate therapies. We're trying to build the technology that moves science forward. So I think Mark had this concept about if you provide the right tools, then the entire scientific community can leverage them. Yeah, so obviously you believe strongly in protein language modeling as a tool. What is the next most important tool for advancing a general improvement in our ability to tackle human disease? Yeah, so I think the next level of complexity that we have to address is the complexity of the cell. And, I mean, this is going to be tremendously hard.

47:29There are billions of proteins. I know that you say it's tremendously hard. Yeah. If you come and say it's going to be easy peasy, I think we would just... Well, I think it's a worthy challenge. But, yeah, I mean, it requires technology that doesn't exist today. It requires new, you know, modeling approaches and probably architectures and ideas and machine learning that probably don't yet exist. So, there's, I think, deep and fundamental problems to solve. But again, I think, you know, you take it step by step. And so, you know, we kind of started the molecular layer and we know that that is really fundamental and we can begin to link that to, you know, observables and cellular biology.

48:07I'm really curious because this has been the question that's been on my mind for a long time about we have virtual cell models, we have molecular scale models. And there's been a few, I've seen a few papers about trying to link them. But what are you guys doing? Because it sounds like this is becoming top of mind for you. So maybe to make the analogy with protein biology, you know, what I think makes our digital representations of proteins powerful and useful is that they generalize. They're able to make predictions for proteins that are entirely unlike the proteins in their training data. You know, they're able to generalize so that you can design, you know, fundamentally new folds, new binding interfaces, new structures.

48:50So there's this degree of, yeah, what we call generalization or generality. In short, they can predict the outcome of an experiment that we haven't already made, that they haven't already been trained on. So for digital representations to be valuable, you know, they've got to be able to be used to answer a new question. So I think that's the critical thing. So we're not there with cells. I think with, you know, kind of the current generation of models that are being called virtual cells, they are good representations of the underlying data, but, you know, they have a very limited ability to predict what will happen when you make a novel intervention in a novel, unobserved context.

49:35But to be able to answer the fundamental scientific questions about cellular biology, we need a model that can do that. So kind of, you know, our thinking about this starts with that idea. What's it going to take to get there? Going back to the protein-protein interaction, the human interactome of the, if you had that, just, you know, predicting static structures. Static structures are, in some sense, not enough for a lot of understanding biology. dynamics are probably for most people a much more useful tool to have you can start with static it can give you some insight but it's very rarely the full answer so you have you know a model which is capable of predicting a lot of different proteins we probably have almost all we have many of these resolved in the pvd some of them we don't given the dynamics and interactions are being are more important how do you bridge that gap is to me that seems like maybe one of the the key steps in going from like a really microscopic model of things to going to something which is closer to a virtual cell you actually have to be able to model local interactions of local proteins or RNA or DNA or lipids or you know whatever else is floating on the cell is that sort of like a goal that you would try to bridge or maybe I'm misunderstanding is there like another way you would imagine bridging these two?

51:03I mean, one day it will probably be possible to have a computer that can kind of simulate the cell from first principles, but we're very far from that, right? I think that's far beyond reach of current computational technology. I mean, kind of even just simulating the physics of the folding of a single protein molecule. Basically, we could do it for a fast folding, a few fast folding proteins. We can't even really do it for a while. Yeah, yeah. Yeah, so there's kind of this dual view of biology, this dual complementary view of biology. One view is kind of that kind of first principles reduction, you know, where kind of all of biology is explainable in, you know, more basic terms, in basic physical, chemical, biochemical terms.

51:52And I think historically, there's a long line of research that's really sought to understand biological phenomena and simulate biological phenomena in that way. And I mean, historically, you know, the field had believed that the solution to the protein folding problem or the protein structure prediction problem would come from, you know, this kind of first principle simulation. It really kind of came out of nowhere that, you know, this could be solved using essentially pattern recognition or, you know, this type of machine learning approach. So I think historically it has been productive to understand biology through information theory, through information.

52:35And, you know, you can think about the cell as a computer, as an information processing machine. It's in informational terms, there are, you know, these very basic principles that link the information encoded in the genome to the genes that are transcribed to the phenotypes of the cell that will result. And so, you know, if we could model and understand the cell at the level of its underlying programs, you know, that sort of gives, I think, the right abstraction. What do I mean by the right abstraction? Well, I think, I mean, the abstraction that is possible today, because we're in the era of information theory at scale.

53:15You know, we're in the, Claude Shannon, you know, had this idea of kind of the ideal predictor of the next character. And he had this really beautiful paper where he tried to compute the entropy of the English language and imagine basically, you know, taking an infinite context and then, you know, what is the entropy of the next character? and at that time, I mean, it was unimaginable to, I'd say it took a great leap of imagination to imagine that ideal predictor, but today we get closer and closer to being able to build that, and we can do that for text, and so, you know, what would that predictor be for biology, and that's kind of the idea of ESM.

53:55It would learn kind of the underlying structure of all biological phenomena. So, if you think about that from the standpoint of the cell, if we can collect enough outputs of cellular biology that we can observe to reveal the underlying programs, patterns, and structure, you know, then we could create kind of the information theoretic description of the cell. And I think that would be sufficient to understanding disease. This reminds me of a lot of the work that happens in signaling pathways right now, right, where you You have protein in a cascade of different protein-protein interactions that eventually cause a phenotypic change in the cell in some way.

54:33How do you translate that into something that can be sort of scaled into, or maybe it's something else, but how do you, for example, that? Yeah, going back to the better lesson. Yeah, going back. Let's just get back to the better lesson. We need data. I mean, I think, you know, why have these advances in protein biology been possible? They've been possible because of, you know, decades, I mean, for protein structure, half a century of work to experimentally determine the structure of proteins and, you know, and the effort across, you know, the scientific world to, you know, sequence genomes and metagenomes.

55:10And so that's created, you know, this data set that, you know, you can really, you know, train a scale on and really learn these deeper principles. And so those two different data sets are actually in many ways quite different. Like PDB, a bunch of very painstakingly constructed protein structures, many of which were an individual PhD thesis. And then maybe the follow similar ones came later, which might have been, you know, 10 of them for a PhD thesis. Then this, people, I estimate it's like$13 billion to create the PDB, some like very large number. The reason people created the PDB was because each individual protein was independently useful.

55:51Like people didn't create it with the sake of we're solving protein structure. They saw that like, oh, this protein we believe is involved in this D's pathway. Let's understand this protein so we can target it and so on. Of course, there's some caveats here, but at a high level, a lot of this genomic data was, especially for humans or viruses or bacteria, you know, was sequenced for a very specific reason as well. Right. Right. Well, it's great that these are useful after the fact, but I wonder if now going forward, especially since, you know, with the virtual biology initiative, Biohub's virtual biology initiative is like half a billion dollars, I think.

56:27and I'm sure there will be more large initiatives coming from Biohub in the future, you have the chance to be very specific, deliberate, and now collect data for the sake of solving a problem with ML rather than depending on a data set which was curated, created for some other purpose. So given that new opportunity, how do you do things differently? How do you think about data collection to enable science broadly when you have the option of doing basically anything from first principles? A little bit of context. We announced a few weeks ago the virtual biology initiative. We basically said, you know, we're going to invest$400 million internally in data creation and development of technology to scale data generation to be able to increase the number of modalities that we can measure simultaneously.

57:21We also announced that we're going to commit$100 million to catalyzing efforts outside of Biohub to generate data. And so, you know, we think that's, you know, a fraction of what's actually needed to do this, right? But, you know, the hope basically is that by making this initial commitment of giving starting funds to some of the groups that are really thinking about this, you know, working to build different core areas of the data that's going to be needed, that, you know, that's going to be a catalyst that's going to galvanize other groups to come in and contribute to this. So that's what we really hope to see.

57:59You know, the idea is that, you know, this is a broad-based effort. So it's not just us. So, you know, I can say kind of what my perspective is on what data needs to be generated here or what can be generated. But, you know, we also want to approach this really collaboratively with the scientific community. And so part of this is also kind of hearing from scientists what they want. So from my view, there are a few key principles here. The first is speed. So it took decades to build the data for proteins, and we can't wait decades. We need to figure out how to do this in a couple of years. And you look at the rate that general AI is developing, and it's just the limitation in biology.

58:42We're going to be fundamentally limited by experimental science and data. So we really need to work to address that gap as quickly as possible. So I think that's one key thing is looking at what are the technologies that we can scale up today to begin to give this picture of the information architecture of the cell. So there's speed. And then there's also the idea of generalization. So kind of going back to what I was saying before, we want models that can serve as oracles for the biology. They can predict an experiment that you haven't done. And so how are we going to be able to do that? We're going to need to look at a multitude of different interventions in a multitude of different contexts.

59:25And so it's kind of similar to the principle of training a language model on the internet or training a protein language model across all of evolutionary diversity. What does that look like for cellular biology? And so we have to scale interventional biology. So that looks like things like perturbation biology, perturb-seek, measurements where we can look at combined transcription, imaging, other layers of the cellular information hierarchy. And there are a number of groups, our teams are working on this, there are a number of groups across the scientific world that are working on problems like this that are ready, I think, to scale.

1:00:03The second is spatial biology. I think that's going to be really important. And so that's going to help us to really understand the cell in context. And I think understanding the cell in isolation is really not what we need. It's not the goal, right? The cell is part of an incredibly complex system in the body. And to be able to understand disease, we have to understand how cells interact, the systems that they form, the circuits that they form. So we need to see that. So spatial biology, I think, is undergoing rapid progress and is an area that's really ready to scale up. That's kind of what can scale now, I think.

1:00:40And Biohub has actually, over the last 10 years, really, I think, made kind of pioneering funding commitments in those areas. And so we've funded efforts like the Human Cell Atlas, and we've built Tabula Sapiens, which looked at large cell atlases. And we've built Cell by Gene, which is kind of a database of single-cell transcriptomics. And so we're really looking to kind of build on that. And, you know, I don't know how many cells there are in kind of the largest efforts. We're probably around like a billion cells or something like that today. So that we've got to go, you know, multiple orders of magnitude from that.

1:01:20So, you know, that involves scaling the technologies that we have now, but it also involves, you know, a new, the next generation of technology. So we're also funding and supporting efforts in that area. And there, you know, we really want to look more at cross-modality. You know, can you simultaneously see the phenotype, observe the transcriptional layer, understand what's happening proteomically, link that to the genome? You know, we'd like to, in the epigenetic state, you know, we'd like to be able to see all of that. And so really pushing technology to be developed faster that can reveal more of those connections and more of that biology and do that in a more scalable way.

1:01:59It's interesting because when I hear most of those ideas, there are oftentimes the things that people already think about in terms of scaling biology. What is the next technology that is going to allow for, like enabling data collection technology, going to be back to the theme of bitter lesson for biology? You don't have just scaling laws on compute and parameters, but now the scaling law is probably in data collection in some meaningful sense. where are the next big opportunities there? Like in, so you're talking about developing new technology as sort of like as part of this initiative. Yeah, so I mean, I think it was basically the same things that I'm saying.

1:02:39Scaling what we have now, kind of being able to expand the number of interventions that we can look at, expand the number of parameters that we can measure. So really kind of more and more multidimensional measurement and, you know, drive down the cost and all of that. So better gene sequencing, kind of better ways of encapsulating cells and being able to measure what's happening, not just the transcriptome, but other layers simultaneously. There's an interesting Pareto frontier there about if you have a fixed budget, how much time do you spend on improving your assay versus how much do you spend on actually scaling it?

1:03:14You know, where do you went out there? Yeah, we have to do both of those things, right? Because I think with current technology, you know, we can definitely kind of get data 10x to 100x where it is today with like relatively reasonable investments, you know, but then to get another 10x or more in there, that's going to require a lot more technology development. But the other really big principle is going to be feedback. And so I think that's going to be really critical. And I think you can see that as a layer of technology development that's going to need to occur. And I think there's a lot of great things happening right now, kind of automation, flexible robotics.

1:03:50It's going to accelerate where that can go. And the experimental design as well. So we typically ask our guests, what is a bottleneck that you would remove that would sort of unlock things? But we just spent a long time talking about that. Yeah, I think I answered that question. But I want to ask it, but I'm going to give it a spin, which is like, so maybe a little bit outside of your domain, like so language modeling or supply chain, something that is a bottleneck that is maybe non-obvious and not directly something that you are working on, but that maybe has an impact on the work of biology or Biohub in particular.

1:04:31I mean, it's a hard question to answer because there's just so many bottlenecks. I mean, the one that I always think about is compute, but I think that's a pretty obvious one. It's the bottleneck for all of AI in many ways right now, but, you know, especially because we're training these large scale models, you know, we're always focused on compute. And I think we're kind of limited both by the data and compute. I think we're in a position where I think we have, you know, incredible compute resources for a team working in biology. But I think like all teams working in AI right now, really the limit is just how much compute power.

1:05:07So if you could 100x your compute, you think that ESMC would be way better? I mean, it would definitely be way better. We also need to scale data. So both of those things would have to happen in tandem. Have you basically exhausted what's available right now? I don't think so, no. Okay. No, I don't think so. Large data sets out there, or you could relatively... Well, there's more compute and more parameters. So we trained ESMC up to 6 billion parameters. Oh, but I'm saying in terms of data available, like, have you exhausted most of what's publicly available in terms of like? No, no, not yet. And, you know, the Atlas that we just built actually has more sequences and structures than ESMC was trained on.

1:05:52So definitely have a little room to go. So like, is that an order of magnitude jump or twice as much? Like, how does that work? Yeah, I mean, I think ESMC is trained on, you know, say, order of a billion sequences. So there's definitely probably order 100 billion sequences. This is, a lot of them are largely redundant. 100 billion? Yeah. Oh, okay. To get that billion, you whittled down from 6 billion, 6.8 billion, right? So of those 100 billion, if you were to similarly cluster and find unique ones, what do you think, where do you think it would land? The sequences aren't actually redundant, right?

1:06:29It really depends on what you mean by redundancy because there's, I think, a tremendous amount that you can learn from small genetic variations, right? Because these are really revealing of, you know, kind of the very basic determinants of protein structure and function at a very fine level. So I think that, you know, as we think about protein space, you know, having a vast diversity of sequences across a wide range of protein families is really critical for the emergence of this kind of structure prediction capability. Because I think kind of large diversity is what trains the model to understand, to develop a representation of structure.

1:07:09But I actually think that to develop a representation of function, it's these very small variations that are important. And so I do think that there is probably a lot more. You know, it's like the models haven't yet been trained at that level of kind of just like really deep understanding of these very small but critical patterns in sequence. I mean, a single mutation is enough to destroy the function of a protein. So you could conceivably actually take all 6.8 billion of those, retrain, everything's the same. Yeah, you could train on more than that. I mean, that's kind of clustered down. Maybe the question is how far?

1:07:47When do you hit the law of diminishing returns here? I mean, it sounds like you have plans for an ESM4 or an ESMD or whatever you want to call it. We're always developing. Yeah, but I'm just wondering, at some point, is this actually something that you could exhaust? People talk about exhausting the pre-training data. Yeah, I mean, at some point. Yeah, yeah. At some point. Yeah, but it's actually something you could conceivably imagine doing in the next few years. Or even if you don't exhaust it, you hit a lot of diminishing returns for, you know, the applications that you're trying to predict here where maybe your resources are better spent somewhere else.

1:08:26I mean, it's basically, it's an empirical question, right? Yeah. It's truly an empirical question. And so we just don't know, you know. I mean, with ESM2, we weren't sure because there were some diminishing returns. With ESM2, you know, now there aren't, right? So, you're going to kind of look at that, extrapolate from the scaling law there. And, you know, there is enough data to train that next model. And the other question that we usually ask is any call to action or what do you want people to go take action on? And if the listeners want to get involved, get hired, build things, what would you ask people to do?

1:09:09Well, we just announced, or I should say we are going at the time that this podcast comes out, we will have announced ESMC and this world model for protein biology. It's going to be open source. It's going to be MIT licensed. And we want people to use it. We want this to be a tool that can unlock science. We're excited to collaborate. We have a team that works on that, and we want to hear from people and understand what we can build that can help to accelerate their science. Yeah, we might have a demo slash paper club of some sort on this channel, so stay tuned. Yeah, stay tuned for that. We'll invite you and your team, whoever can make it.

1:09:53It will feature this paper once it's in final preprint form and spend some time on it for an hour on the Latent Space Paper Club. Yeah. Thanks for chatting with us. Awesome. Yeah. Great to meet you guys.