#146 Viren Jain: How Google's AI is Pioneering Brain Mapping Research

0:00More importantly, we had computers that could finally do something useful with respect to this goal. You know, they could first of all store the images and look at them on a monitor. And then over the past 20 years, you know, we've built increasingly sophisticated machine learning systems that can analyze the images and trace the neurons and find the synapses. Look, I mean, ultimately the goal is to understand how the brain works. That's why people fund this type of work. That's really the first order of business. And, you know, more specifically, the reason people want to understand how the brain works is to improve human health.

0:31Hi, I'm Craig Smith, and this is Eye on AI. The structure and function of the brain has long inspired artificial intelligence research, while AI has helped neuroscientists understand the brain. In this episode, we glimpse into the painstaking efforts to reverse engineer thought by mapping the brain's neural pathways, a process called connectomics. Viren Jain, who leads Google's Connectomics team, discusses the history of the science from early microscopic investigations to modern large-scale electron microscopy data sets aided by AI. Viren shares his perspectives on how Connectomics could inform our understanding of learning and intelligence and talks about Google's participation in a national institute's Health initiative to map 10 to 15 cubic millimeters of mouse brain over the next five years.

1:37I hope you find the conversation as fascinating as I did. Hi, this episode is sponsored by Salonis, the global leader in process mining. AI has landed and enterprises are adapting, giving customers slick experiences and the technology to deliver. The road feels long, but you're closer than you think. You see, your business processes run through many systems creating data at every step. Solonis reconstructs this data to generate process intelligence, a common business language. With process intelligence, AI knows how your business flows across every department, every system, and every process. With AI solutions powered by Celonis, enterprises get faster, more accurate insights, a new level of automation, and a step change in productivity, performance, and customer satisfaction.

2:36Process intelligence is the missing piece in the AI-enabled tech stack. Search Celonis, C-E-L-O-N-I-S, to find out more. So yeah, I'm Varane. I'm a research scientist at Google. I lead the connectomics team there. You know, in terms of my background, you know, I got a PhD at MIT working sort of at the intersection of computer science and neuroscience with Sebastian Sung. And, you know, when I first started graduate school, you know, we were working on sort of more theoretical ideas, you know, sort of how would you mathematically think about how brains work. But we sort of realized that we didn't really have the data we needed to think about that or to advance that in quite the way we wanted.

3:25So then we turned to this other enterprise of really just trying to actually map out brains in detail and sort of from the bottom up build the data that we needed. So after graduate school, I went to a place called HHMI Janelius. The Howard Hughes Medical Institute has an institute devoted to these kinds of topics. And I ran a lab there for a few years. and then 2013 I moved to Google where I started this connectomics team and for the past 10 years have been working on and developing the effort associated with that here at Google Research. I mean there's been a lot of work done on single cell and there's one particular worm that they've pretty much mapped.

4:10Can you give us that history before we get to Jeff Lichtman and Absolutely. Absolutely. Nice. Yeah. Yeah. So, I mean, you know, the history of the field was, you know, that goes back in some days, you know, even to the earliest days of neuroscience where Cajal was, you know, you know, about 150 years ago, looking through microscopes and staining neurons in different organisms. And, you know, what he noticed, and, you know, this is really a key insight in some ways, is that neurons have different shapes. And, you know, these shapes are interesting because they support the general organizational principle of biological brains, which is that you have these units called neurons, and they, you know, form connections with other neurons.

4:56And, you know, the hypothesis was that there was a lot of information, a lot of, you know, important functional information encoded in, you know, how the neurons actually talk to each other, you know, which neurons form connections with each other ones and so on. And so this idea kind of, you know, started to take root in neuroscience a long time ago. And then in the 70s, Sidney Brenner, who was a scientist at the MRC in London at the Laboratory for microbiology, he decided to pursue what was at that time a very novel idea, which was, what if we just mapped out all of the connections in a small organism?

5:39And he chose this organism called C. elegans, which is a roundworm with 302 neurons, had already become popular in genomics research, and was beginning to become more popular for other forms of biological research. And so they took one of these worms, they sliced it into a thousand sections. They sort of imaged each of those sections with an electron microscope. And then interestingly, they printed them out. So they took all these sections, these basically literal films, they printed them out onto different sheets of paper. And then for about 10 years or maybe five to 10 years, they sat around tracing the neurons from one section to another to figure out where all the wires are going and who they're talking to.

6:26And from that, they were able to reverse engineer this so-called connectome describing the chemical synaptic connections within that worm brain. So that effort has actually been quite influential. People who study worm brains use that information all the time to guide their experiments. But it was such a pain, right, you know, 10 years essentially for 302 neurons that people, you know, became discouraged from trying to repeat the feat, you know, for anything larger than a worm brain. Until maybe, you know, about 20 years ago, around 2004, people kind of revived this idea and thought, you know, now, you know, we can do two things.

7:11First of all, we can automate the process of sectioning and imaging tissue in this way. We can build automated platforms that will slice up tissue and image them. But more importantly, we had computers that could finally do something useful with respect to this goal. They could, first of all, store the images. We could look at them on a monitor. And then over the past 20 years, we've built increasingly sophisticated machine learning systems that can analyze the images and trace the neurons and find the synapses. So over the past 20 years, this field has really progressed much more rapidly. There's been a lot of progress on mapping the entire fly brain, which is some work that our group has participated in.

8:02there's been a small amount of work done on mouse and human brains so you mentioned jeff flickman earlier um you know we worked with his lab to um reconstruct a cubic millimeter of of human brain tissue which was you know a very interesting project on the one hand you know one of the largest data sets ever collected in biology you know 1400 terabytes of data but on the other hand it was one one millionth so that's you know one over one with six zeros in front of it uh one millionth of an entire human brain um so uh you know the the technical challenges in scaling this up uh are really quite daunting um and you know what the field is converged on is the idea that after the fly the next big milestone could be the mouse brain which is about a thousand times larger than a fly but still a thousand times larger sorry smaller than a human brain uh and is obviously you know a nervous system and and general organism which is you know studied commonly in biology and uh for the roundworm that involved this machine is it the same machine that jeff lickman uses And was that then the same method for the fly brain that you're taking these?

9:21I don't know what is the thickness of the slices of the brain. So, you know, conceptually, the approaches are all similar in that, you know, what you're doing is, you know, using some combination of cutting tissue at very thin scales. So, you know, anywhere between 10 to 50 nanometers, you know, it's kind of like human hair thickness or smaller. And then you're imaging the tissue using some kind of election microscope. Now, the details can get quite different, though. And just to give you a flavor of this without, you know, getting too technical, you know, there's broadly speaking two different methods in the field, something called serial section approaches and another one called block face approaches.

10:08and in the serial section approach so this is really like the deli slicer idea where you take a brain you know you slice it up into independent sections and then you image each of those sections independently and you know you have to be very careful because those slices are very thin and delicate but the advantage is you know you know once you once you have the individual sections you can you can image them with whatever kind of microscope electron microscope you want the other approach in the field which was you know used to do some of the work which i mentioned is the block face approach and you know just real quick the idea is you have like a block of tissue you bounce electrons off the surface and then you you ablate you know you either shave off or literally vaporize the top layer so you can expose some more tissue to be imaged and you keep alternating those two steps anyway the point is there's you know it's a real technical challenge to image a piece of brain in 3D at nanometer resolution.

11:07And so there's been a lot of ingenuity put into, you know, how can you do this process reliably at scale with higher throughput? And that continues to be a big debate in the field, you know, just like in genomics, where, you know, there was a 30-year, you know, progression of technology that continually made sequencing genomes cheaper and more efficient. You know, in many ways, we're trying to do the same thing here to try and invent, you know, the raw data acquisition techniques, which could image brains at scale and more cheaply than in prior projects yeah and and in once you have this the the images of the slices stain uh you were saying these guys i don't know if you meant literally printed out sheets of paper and we're tracking neurons from page to page that's all done with artificial intelligence now, isn't it?

11:59That's right. So, yeah, certainly we're not printing anything out anymore, thank God. You know, that would be a lot of paper for a fly brain even. And so computers are used in many respects. First of all, just to store the images and organize them and recapitulate them into a three-dimensional object. that's certainly one task that needs to be performed. Second is, as I mentioned, this process of tracing neurons. So the way to think about this is you probably have heard of server closets, right? Where you have a stack of computers and in the back there's a bunch of wires connecting them. Imagine taking a black and white photo of a server closet and then trying to ask, okay, what's the wiring diagram of how the computers connect?

12:48right the only way to do that is to you know find a wire and visually trace it until you end up at some other machine right there's nothing um they're not color-coded you know there's nothing you know there's not some uh number on the wire which tells you where it starts and where it goes to and so the way to think about the brain when we image it with electron microscopy is like it's like a three-dimensional server closet and so if we want to figure out the wiring diagram we have literally trace out where one wire begins and where it ends. And that's quite a difficult task if you think about each of these wires can sometimes be like tens or hundreds of nanometers thin and they span millimeters to centimeters of tissue and they're all jumbled together in like this giant mess of spaghetti.

13:34So doing that process accurately and at scale has been non-trivial to automate, but is something which a number of us have put a lot of effort into designing machine learning algorithms for. That said, the algorithms are not perfect. So the final stage of all of this is actually to have humans go through and fix all the mistakes or fix all the things that computers cannot do. And that remains sort of like maybe the biggest bottleneck in this whole process is that we still need thousands to tens of hundreds of thousands of hours of human effort to finish a given nervous system. And one of the big practical and research challenges in the field for us is to continuously drive that number down.

14:19So the next time we do this, we need only 10 % of the human effort that we did last time and so on and so forth. Yeah. And just on that point, I was just talking to somebody about reinforcement learning and with AI feedback and having multiple AI agents sort of debating with each other to converge on a final answer, you know, to try and combat hallucinations and that sort of thing. Are there strategies like that where if there are errors in the map in the connectome that's been traced through machine learning, that you could have adversarial algorithms that eventually figure out, you know, what the accurate path is?

15:18There have been attempts to sort of combine the predictions from, you know, some diversity of algorithms to improve the overall performance of this tracing process. You know, it has been difficult to do that in a really sort of meaningful way. Just it tends to be the case for this task that if one algorithm is wrong, many of the others will be wrong also. But there have been attempts to do that. For now, I would say the push has really been, you know, once we've had some amount of human effort applied to the hard problems or the hard cases, you know, can we go back and retrain, you know, the initial algorithm with all that additional data to have it be more accurate in the next instance.

16:02Yeah. So with the mouse brain, how far, again, what percentage of the overall brain have you completed? And how long did that take and all the metrics surrounding that? So the mouse brain project, that's really, you know, that's just getting going. So that's, you know, the announcement last week was really, you know, a suite of projects that the NIH has funded to really scale up mouse brain connectomics. You know, what's been done there so far has also been at, let's say, the cubic millimeter scale, which, you know, if you think about a whole mouse brain as being 500 cubic millimeters, that's one five hundredth of a mouse brain that's sort of been achieved so far.

16:49And the goal of these new projects that the NIH is funding, of which we're part of one of them, is maybe to do 15 cubic millimeters or so. So scale up maybe 15x and get, let's say, a few percent of the whole mouse brain at the end of the day. That said, that's sort of the goal for the next five years is to hit that milestone and see which technologies are scaling best and are producing high quality data. If that's successful, it's possible the government will fund a second five year effort, which will really try and do the whole thing. So to pick a particular technology, which seems to be going well, and then put in the money and resources to do the complete map of the mouse brain based on that.

17:39And what do you learn from the connectome? That's one question. Because this is looking purely at the physical structure, not at the biochemistry or any of that. And then, well, why don't you answer that first? Yeah. Well, there's a real sense in which we're still learning what we learn from the connectome. But we've had, you know, I would say a number of interesting examples so far. So first of all, in C. elegans, where we've had the connectome for a long time, you know, that data structure, so to speak, having that information in C. elegans, you know, has been transformative for studying that nervous system.

18:30So people could, for example, go in and ask, okay, what are the neurons that are connected to the motor output of the system or the sensory input? Or what are the ones which, you know, are just sort of in the middle of all the processing? And, you know, if you want to then study those particular aspects of the nervous system, you know where to begin. Or if you want to ask questions like, how does the information flow, you know, accomplish some objective, some behavior that the nervous system has? that connectome will give you a number of different clues and hypotheses, which you can then use to decide your experimental strategy.

19:05Similarly, you know, I would say the fly brain, which is a much more recent story, you know, the connectome has really been a big success there, I would claim, in the sense that, you know, almost anybody who works on studying the fly nervous system, of which there might be more people than you imagine, you know, basically now uses, again, that data structure in all kinds of ways to guide their experimental strategy. So, you know, you can sort of trace the information flow throughout the brain and say, okay, well, it comes into these receptors, and then it goes to this part of the brain that processes, let's say, smell information, and then it goes to the part of the brain that stores memories, and then it goes to a part of the brain that decides what action the fly should make.

19:49And, you know, we've really gotten to the point in the fly brain where for some specific behaviors, you know, you can literally trace out the synaptic steps that are involved in, you know, computing some piece of information or making a decision and, you know, start to really build a detailed model and hypothesis for how the brain works. So I would say, you know, in the worm and the fly, there's now, you know, very good support for the idea that having the connectome can be extremely valuable for, you know, understanding how the system works. I think as we scale up to, you know, a mouse and one day maybe a human brain, things will get a little bit more confusing.

20:30Right. So, I mean, the difference, you know, one of the main differences between a worm and a fly and us is that, you know, much of our brains are there to achieve, you know, learning and memory and experience driven changes in behavior. Right. You know, a worm is largely born, you know, with the behaviors it's destined to accomplish. a fly, you know, mostly, although it does do a little bit of learning, you know, what's interesting about, you know, certainly human brains is, you know, we do a ton of learning, right? You know, much of our, you know, day-to-day behaviors are driven by things we learned after we were bored.

21:06So in that sense, you know, how interpretable the connectome will be in, you know, some of these larger systems and what form these explanations will take of neural circuits, It's a little bit less clear, but in some sense, you know, we're excited to find out. and are there any uh sort of principles from now that you've got the uh the roundworm i'm i'm not going to try and pronounce the name of the roundworm and the fly uh that you that there you can see principles that that apply to each maybe in the in the shape of neurons carrying certain kinds of information and that sort of thing. Yeah, absolutely.

21:54I mean, so, you know, one of the most, I mean, one of the most striking things in both, I would say, the worm and the fly is how closely related we found structure and function to be, where the particular shapes of the neurons and, you know, the connectivity that we observe just in the connectome really does seem to, excuse me, you know, have a close relationship with the function of those neural circuits that we observe when we go in and measure their activity and how they influence behavior and so on. So there are parts of mammalian nervous systems in which we see a similarly strong relationship between structure and function.

22:37So for example, in our retina, which is, you know, the part of the brain that's at the back of the eye and first processes light, you know, we see, again, a very strong relationship between sort of the shapes of neurons and their synaptic connectivity and what their functional role is. So, you know, I think that is one principle which really has quite a lot of power and, you know, has been, you know, proven to be true in some of these smaller organisms. Now, you know, I think as we move to some of the higher processing center, so to speak, in mammalian systems like the cortex, you know, there it seems a little bit more like a computer to a first approximation, where, you know, you have a bunch of, you know, sort of generic parts that get wired together in all different ways.

23:25And the relationship between structure and function there may not be quite as obvious. And so I think, you know, So those will be interesting cases in which the analysis that we do of the connective might have to be much more subtle and complicated in order to figure out what's going on. Yeah. So you can see the shape and function in these lower level brains, or is that also applied to the lower brain in the human? But you can see a connection between shape and function. Is there some sort of a commonality between this kind of shape in the roundworm is analogous to this kind of shape in the fly and they're performing similar functions?

24:25I mean, can, are there any, uh, is there any, uh, deduction that you can make from that? Yeah. So I would say the general concept which you're getting at is, you know, what we call cell type, right? So can we cluster, you know, all of the neurons, um, that we see in one brain or cross brains into categories, right. And say, this is one type of neuron versus another type of neuron. Um, and based on doing that, can we then infer, you know, other properties of them, like what their functional role will be in a circuit and so on. And this is a very, you know, this is a very active area of investigation for neuroscientists.

25:01So this isn't just about connectomes. So there are people who are, you know, measuring gene expression to see, you know, which genetic programs are active in different brain cells and using that to categorize cell types and so on. But in general, yes, I mean, the answer is definitely that, you know, there seem to be, some discrete set of types of cells, which have different genetic specification and different shapes and different physiology to go along with that and different roles in the nervous system. I think going from the worm to the fly, that's a pretty big evolutionary jump, actually. And it's not so obvious that the types we find in the fly are really that closely related to the types that we find in the worm.

25:50The worm is kind of an interesting case, right? Because they've been around a long time, hundreds of millions of years, at least. And they're small, they're 300 neurons. So they've just been hyper-optimized by evolution to do what they do efficiently. And it's almost kind of this boutique analog circuit where every part in the system is a little bit different and playing multiple roles at the same time and so on. And this is very different than if you go to, let's say, a human brain, right, which has 100 billion neurons. There's no possible way the genome could encode details of 100 billion different neurons.

26:32So what you have is a much more modular organization, right? You have a small number of cell types, maybe thousands, which get repeated into different motifs and organized in different ways. And so, you know, the sort of scheme of the nervous system there is very different. So the fly is then kind of like an intermediate, where, you know, there are definitely some very specific cell types. You know, it's maybe not all genetically encoded, but quite a lot of it is. We see a little bit of learning and memory. So we're kind of getting a taste of both styles of computation there. But, you know, coming back to your question, the identification and characterization of cell types is, you know, one of the most sort of pressing tasks in neuroscience, because, you know, that's really what will let us simplify in some ways our models of these systems.

27:26You know, if we have to think of each neuron as sort of its own special thing, you know, it's going to be impossible to really, you know, develop a description that's interpretable. The other thing is, from what I understand, again, I'm just a journalist, neurons, the connections of neurons in the brain can change over time. And even the shapes can change over time, if I'm not mistaken. So when you're slicing a brain, you're sort of capturing a moment in time. How do you track the changes in neural patterns? That's right. So, I mean, definitely with connectomic methods, we're looking at a static point in time for that particular organism.

28:22The method is highly fatal, I would say. Your brain ends up as a series of slices. So, you know, there's no more activity or changes. So it depends on the organism that you're studying, right? You know, for, again, for worms or flies, there, you know, there certainly are dynamic elements there, but they, you know, appear to be maybe less critical to the overall story as compared to human brain, where, again, you know, You can learn entirely new behaviors within a short time span and obviously encode new memories and so on. So I think studying static connectomes will let us make progress on quite a lot of aspects of the nervous system.

29:11Even in human brains, it's not that things are getting completely reorganized or even mostly reorganized from one day to the next. In fact, most of the structure is very likely to be quite stable over long periods of time because we have to support all of this stable physiology and all the stable behaviors that we have over many decades. So I think the connectome will be extremely useful in analyzing those types of issues in neuroscience. I think specifically for learning and memory, it's a bit more challenging for these methods. But certainly, you know, it's not impossible. People have, you know, been looking at ways to, you know, let's say you take two different flies that have had two different sets of experience and you look at differences in their brain.

29:56You know, can you pinpoint that, you know, using methods from connectomics? But but I should say, you know, I mean, neuroscience has has many different experimental tools available to it. You know, there are people who just image the activity in the brain or just look at specific connections over time. And really, it will be the combination of all of these different measurements and experimental tools which will be required to explain really the most complex issues in the brain, like how do we encode a memory or how do we learn things. Yeah. When you're doing this, you're taking cross sections at some point of neurons.

30:38You're obviously not able to, you know, slice so that you have one layer of whole neurons and another layer of whole neurons. How much can you see into the internal structure of neurons and axons and dendrites and all of that? Yeah, that's an excellent question, actually. And, you know, quite a bit. So, you know, what you have with an electron microscope is really a very high resolution, unbiased instrument for probing the structure of biological tissue. So while, you know, with connectomics, people typically talk about, OK, well, what's the you know, what are the list of connections between cells?

31:23But really, you have much more detail that you're getting out of this data set. So first of all, you have the precise shape of each cell, which can often be very interesting. And then, as you mentioned, within each cell, we can often see, you know, all of the organelles. So, you know, all the mitochondria, the nucleus, you know, microtubules, you know, all kinds of different internal structures which support the physiology of the cell. And an increasingly interesting, you know, goal in our field is to understand how are those details related to, you know, what the brain is doing overall. So if a neuron has one function versus another, does that imply that its internal organization, the kinds of organelles it has and where they're situated is different?