#197: AI, Fusion, and National Security with Lawrence Livermore's Brian Spears

26:29FAST is aimed at taking what we understand about the AI world and scaling it up to U.S. nation scale, being able to do things that are transformational for strategic deterrent, for bioresilience, for decision-making in the intelligence community. What we need is the capability to build frontier AI models on demand and at will aimed at national security missions, and we need to be the first to do it. It's absolutely critical. There's a global race on taking that compute technology that we built here in the U.S. and doing transformational things with it. There are adversaries in the world, especially China, who have declared that they will dominate in the AI space by 2030.

27:07The National Security Commission on AI took a hard look at this around 2021, 2022. And they said in this global scale, there were six structural features that they looked at. And most players in the world cannot compete with the US, save China. China's ahead in half the features. The US is ahead in half the features. Who wins is a toss up. So the Department of Energy has stepped up and said, we know how to do computing at enormous scale. We understand what the national security needs are for the country. We're in a unique position to bring together the compute capability and that awareness of those missions and put it together and do something productive.

27:45So FAST is the answer to that. It's a way that we do four things for the country at huge scale. One is take huge amounts of data and do something transformational from advanced light sources, accelerators, fusion facilities, you name it, the world's best experiments. Combine that with simulation data. So data is one. The second is a simulation and computing capability. We have the largest computers in the entire world. Data plus those computers is a really nice environment to do a third thing, which is build out transformational AI models. If you do all of those three things, data, compute, and models, you can make tools for AI that then transform the fourth piece, which is critical applications.

28:26Those are the national missionaries that I'm talking about. So the fast effort is this very ambitious whole of nation effort led by DOE to go transform what we're doing in AI. It will be pointed at things like we started the conversation with at fusion. If we want to go get that for taking care of the stockpile, we've got it now. We can push up the capabilities to expand in new ranges. We could think about inertial fusion energy. We could think about challenge problems that were unthinkable even five years ago. There's a concept that we think of that we give the name the one month medicine. how would you go from seeing an emergent biology threat to the definition of a therapy for it, to the production of that medication, to knowing it's safe in humans on a timescale as short as one month?

29:10We can't do that right now, but we can see a path. And to do that, it takes the precision compute that we have. It takes the AI capability and the joining of those two things. And you got to go to experimental facilities. If I can imagine a molecule that might be helpful, I also have to take an automated chemistry system and make that goop. see whether or not it is just goop or whether it's a useful therapeutic. If it's not, then I got to go back into that loop and do it kind of like the higher repetition rate laser. Maybe not every at 10 hertz, every 10 seconds. But can I make a molecule every hour, every couple of hours and not take a team of chemists and scientists a much, much longer time to make the first go and then rewind it and have a much longer loop?

29:56Two questions. One, on the models, somebody asked, Andrew Ross Sorkin, one of the talks panels here asked, I think, the CIA director, whether the U.S. is building models, whether it has its own proprietary secret model, kind of a Manhattan project of AI. And he said no, because the expertise exists in the private sector and that the government funds a private sector, the private sector develops these powerful models, and then the government essentially buys them back. But what you're talking about with this compute technology, for example, at your lab, are you building models yourselves? We are.

30:56So, yeah, let me be really clear about what that question was precisely. The question was, are you building large language models that are proprietary to the U.S. government in secret that nobody knows about? And the answer was no. I will say from the DOE perspective, we are entirely uninterested in competing with OpenAI or Anthropic or anybody else making those kinds of large language models. We're in fact legally prohibited from doing so. We don't have any interest in doing that. We are making U.S. proprietary models for all the things that are not language, for not learning the grammar of English, but learning the grammar of chemistry, for learning the grammar of physics and building out maybe even transformer style architectures, which are what's behind GPT and other things, but for building out molecules for the generation of a new alloy for advanced manufacturing, our new protein for an antibody, our new small molecule for a drug for cancer, all things that we're actually doing.

31:55Those models are not secret in the sense that the question sort of intimated at, but we're doing it at nation scale. and we will and we are doing it with data that only the U.S. has access to. And it is a transformational capability. It's something that no other country on the planet can do. Inside the Department of Energy, we have a workforce of about 60 ,000 highly trained scientific staff. We have the largest physical science database in the world. We have the most capable scientific machines, both experimental and computing, for producing more data. And our fast effort is designed exactly to take all of that data, all of our expertise, all of our compute and build out models that only the U.S.

32:36can have for U.S. advantage. So it's not quite as spooky as the question, is the U.S. doing something secret in the background? We are quite openly going out to seize global advantage and do something transformational for the U.S., especially aimed at satisfying our national security needs. Yeah. And those models, then, would they be made available to the private sector or are they for use in national security and therefore? It's quite possible. Some of the most exciting work that we've done has been in public-private partnership with, say, private partners like AstraZeneca for developing antibody therapies.

33:12We are interested in sharing those models. We are not interested in sharing them if we think that something bad could be done with them. So we will be very careful and highly responsible about what AI safety and security looks like. We're going to respect everything in the executive order, and we're going to go out and do this super carefully. So in the biology space, it's very easy to imagine a model that interrupts biological pathways so that you don't propagate disease. So I want to stop a COVID-19 infection. Great. The same kind of tool could be used to interrupt biological pathways that are critical for staying alive.

33:47I'm going to have to think twice about releasing a model that could be used for both purposes. Those sort of dual use ideas are at the forefront of what we're considering doing. And that extends to all kinds of models. That's not unique to DOE. When we think about building out these AI models, we have to think about what people with good intent are going to do with them. We have to think about what people with ill intent are going to do with them. It's an open research question to decide how to protect those models from doing things that are harmful while enabling them to do things that are beneficial.

34:19The Department of Energy is in a unique position to see what is harmful in scientific technologies, to understand what's beneficial, to have expertise and understanding what AI and high-performance computing can do, and to start pushing the frontier of the ways that we can release the beneficial things without unleashing the harmful things. It's not easy. So in our safety practices, we do three things. We assess the risks to see what we think the options are. Can we do something safely or is there a vulnerability? Second, we put in what are known to be best practices. So we protect models. We share them when it makes sense.

34:57We don't share them. We may share data. We may not. Then the third thing that we do is do frontier research and understanding what is the next best thing we can do in the future. And we drop those into the models. And then we do that safely in the future. This is an iterative cycle that we just have to run over and over again. Biology is a place where you have to think about it immediately because the access that people can have for doing good and for doing ill with those things. But we have to think about it for the nuclear stockpile. We have to think about it for energy production, for manufacturing, you name it.

35:28Yeah. And you were talking about transformer-based models, not necessarily language models. I mean, not trained on human or natural language. But, you know, DNA is a language and there are certainly other languages in quantum physics and in terms of, you know, codes that can be configured in different ways. So is DNA research one of the areas that this is being applied? Yep. And are these models, I know that in layman terms, people get hung up on parameter size because that's the measure that... You can count it. Yeah. Are they larger than the existing language models or will they be larger than existing language models?

36:31They are not yet larger than the biggest language models. So the biggest language models are trillion parameter class models. So think about GPT-4, which is a 1.x something parameter model. And the similar largest models are coming out at the sort of trillion parameter scale. The scientific models that are aiming to do similar things are far smaller than that right now. The challenge is to build out architectures that are useful. They might be transformer based. Before doing that, you have to sort of create the grammar, the language that goes into them. So molecules like DNA, DNA is a very long molecule, even smaller molecules.

37:11They're a really nice test bed for doing that. They're a little different than language. Language is super linear. So we have words that go from left to right in English across the page, and we know how to read them in sequence. A molecule exists in 3D space, and there are ways to project that into something that looks like language, a string called smile strings. but you lose the 3D nature of the chemistry. And that's really important for the physics and chemistry of how atoms interact in molecules. So before you build out these enormous models like you do for language, you should start with smaller molecules and then put a lot of effort on the 3D representations of that.

37:46So we have a project called FLASC, which is a semi-tortured acronym that I won't recall, but the FLASC project at Livermore, led by a colleague, Brian Van Essen, is building out frontier level models for understanding what representations of molecules look like. Can we build out these transformer style or other architecture models that understand the chemistry grammar and then can do predictions with them? But you don't only want to predict, will this molecule hang together? But you want to predict, how do I synthesize it? What are the chemical precursors that I need? What are the synthesis pathways that I need to activate?

38:19Is this a safe molecule to interact with? If it's not, can I make small tweaks that make it accomplish my goal, but are also safe so that I don't have to wear specialized personal protection equipment when I handle it, or if it's a medication, it can be put into animals or it can be put into people. So these models need to come a long way to get up to the kinds of scales where you see the sort of large language model, GPT kind of behavior. But we understand what the capabilities are. So we're pushing very hard right now at the small scale to get the ingredients together, just like we did with language models.

38:52The attention is all you need paper and the transformer architecture was not built at a time where we were doing trillion parameter scale stuff. We just understood that that model was transformational. And when you go to scale, you get all of these amazing things that come with it. We would like to repeat that, start with the chemistry case, do it at small scale, build out the language underneath it. We have all that data at DOE, build those models at small scale and then go to large scale. That FAST program that I talked about, this nation scale effort, is part of the build it at small scale and then scale it up to the enormous nation scale to give the U.S.

39:27competitive advantage that you shouldn't be able to get anywhere else. Yeah. On the 3D understanding the shape of molecules or the 3D structure of molecules, Google just came out with AlphaFold3. And we were talking yesterday about some of the work that you guys have done on protein modeling or molecule modeling. Are you beyond AlphaFold 3 or do you use AlphaFold 3 when something like that appears? Or is there some complementary relationship between your research and that research? And is that research, your research, public in the way that AlphaFold 3 is somewhat public? Right. So we do use other people's protein folding models.

40:24Our group that works on a project called Guide, which is a partnership with DOE and DOD, uses a model from Facebook AI Research owned by Meta. there's an open model out there that's really fantastic. We build on top of that, or really underneath that, the underlying scientific data from high precision, physical and chemistry simulations from what we call density functional theory or molecular dynamics theory about the way that molecules stick together to inform what proteins generated by that model look like in terms of their capabilities, their safety, a variety of other things downstream. And we join that into a predictive capability for models that we want to look at in the future.

41:06We don't use AlphaFold or AlphaFold 3. The models that we do have now had some nice features that made them very usable. We will look at large models like that in the future. We also, as we just talked about, are growing some of our own in-house models that we think have options based on sort of transformer-style architectures to do things that are like what AlphaFold and others are doing, but that are kind of specialized from the missions that we need to go after. To the open question, if we think that there's not a really negative dual-use opportunity, and we're really confident that there are good benefits to be had without ill, then we very often share these.

41:50We will publish our information about what we're doing, but sometimes we will stop short of sharing all of our code if we think we just can't be confident that someone's not going to do something really dangerous with it. Right. Yeah. But we there's very little sense that we're going to there is no sense that we're going to keep this proprietary just to keep it proprietary. The only reason we will keep it is if we think that there could be harm done with it. Our instinct as a federally funded research and development center is to push this out for public good. We share with private industry as much as we can.

42:21We have a very long history in DOE of developing technologies and giving them away to the private scale, to the private sector. to scale up and build out for US advantage. So if we can, we will give that to people. Yeah. I just had a conversation about the National AI Research Resource. Resource. NAR. Are you guys participating in that? Is your compute infrastructure part of that? Yeah, absolutely. So through DOE, a good fraction of the computational cycles that are available in this first round of NAR, the winners of which were just announced, come from DOE, DOE computational capabilities at Oak Ridge and other places inside the DOE national laboratories.

43:06So we have a collaborative research with NSF and with Nair. Nair is one piece of the puzzle for building out AI for the nation. That gives academic faculty and graduate students access to slices of compute time where they can learn what it looks like to operate at large scale. They can start to build out science and AI workflows and train a workforce and build out their research in ways they couldn't otherwise. What it's not is what FAST aims to be. FAST is putting out lighthouse or guide star problems that say this is what huge transformational science looks like. Interdisciplinary problems at very large scale.

43:43If you are trained to operate at the smaller scale of the NSF NER funding, those skills can be brought to bear in interdisciplinary teams to do larger, more emergent things than you can imagine operating at that other scale. They're absolutely complementary efforts. Nair is fantastic and not enough. FAST is transformational, but needs the workforce transformation that comes from Nair and the things that academics can do with that time. And together, we build out students, professors, national level researchers that can come into the national laboratories and we can do transformational things together.

44:16So they're very complementary. Yeah. Do you have trouble in recruitment? I was at NeurIPS in December, and people were saying that, I don't know if it's true, but people who certainly know more than I do, were saying that OpenAI is now offering a million dollars starting salary for top-level PhDs. I can't imagine the U.S. government can afford that. Or maybe they can. I don't know. Well, I don't make a million dollars. So that'll be my first answer. It's a tough competition. So there are, you can work in the AI space. You can go to our private partners in the world and you can make more money.

45:04That's not debatable. You will do that. We do have to compete with them. We do still bring in a pretty steady stream of hires. We do that because of the compelling missions that we have. let's say you walk into our laboratory. What have I done for the past two decades of my life? I've used the largest computers on the planet to operate the largest laser in the world, using the most sophisticated AI algorithms together to provide for the first time in human history, more energy out of a fusion implosion than what we put into it with the laser. That sounds like science fiction. You can't do that anywhere else on the planet.

45:41You would have a hard time paying me enough money to leave that to go do some of the other things, not begrudging our energy partners, all the very cool things that they do. What we have is the compelling mission to work on. We have to use that to our advantage, but it's not easy. Take my particular case, Lawrence Livermore National Laboratory is located in the Bay Area. We're a 40 minute drive from the heart of Silicon Valley. Do we lose people to our industry partners? Absolutely. Is it hard to recruit people? Absolutely. But we're working on it. Our FAST program is part of showing the world what we do for good at scale for US advantage and making a home for people who want to do that kind of transformational stuff.

46:23We cannot pay them what industry pays them, but we can offer a different kind of compensation. And we just have to be competitive in that space. We will seek out through DOE ways to pay competitive salaries. There are things that we're thinking about to make us look more attractive On a dollar-for-dollar basis, I don't think we'll ever win. It's got to be monetary compensation and something else. I will say that we've had people leave our laboratory, go work at places like Facebook, and then come back. It's a pretty attractive place to work. Yeah, well, I remember the National Security Commission on AI was talking about these sort of, I can't remember what they were calling them, but national service programs where people from industry could go into a national lab, work for six months with some agreement with their, with Meta or whoever their employer is, and then go back because they get a lot of money.

47:24of knowledge. It kind of benefits both sides. Are those programs active yet? There are pockets throughout the government where they work. They work pretty well in the cyberspace. They call them reserve forces, like you would think of the Army Reserve or the Air Force Reserve for the military fighting forces. The idea is exactly that. Instead of working for your day job and then spending a weekend working for your Army or Air Force Reserve training, you would work at Facebook or NVIDIA and then spend some time working at a national laboratory, for example. That's one of the most forward-leaning things that you can think of.

47:59I thought it was pretty prescient of NSCAI to suggest that, and I'm a huge fan of the idea. At scale, that doesn't exist yet. I'd like it to. Yeah. How do you feel about the competition? I mean, everyone's obsessed with China. I spent a lot of my life in China. Are they doing similar things? Do they have the same kind of national compute infrastructure that the national labs have in the United States or the DOE has? Can you talk about that? Yeah. Yeah, absolutely. There are lots of comparisons we should make between the US and China, some of which look favorable for the US, some of which look favorable for China.

48:47In terms of their ambitions, as I said earlier, their ambitions are the same as ours. We will try to lead. There's a concept that I call escape velocity. My firm belief is that the nation that first establishes a clear advantage in AI for designing their science and technology and national security capability will not be caught. They will establish escape velocity. They'll be out ahead. They'll be able to accelerate harder. And you cannot catch up to someone who does that. So in my view, the United States needs to be first. It's a neck and neck race right now. So there's no clear winner. China has said that by 2030, they will be the clear forefront winner.

49:26So we should think about everything that we're doing at nation scale against that backdrop. Let's look at compute capability as a proxy for that conversation. The things that we can, what we can say, if you look at high performance computing, is that China used to be a significant laggard in the number of high-performance computing cycles that they could put together. Now they have more than half of the computational cycles that you can rank in the top 500 supercomputers in the world. They've stopped, in fact, participating openly sometimes in that conversation, but we know that they have the ambition and the will and maybe have already accomplished having more cycles available to them in high-performance computing than what we have.

50:07What do you mean by cycles? Cycles, sorry, that's the ability to do numerical operations. So each computer chip can do some kind of multiplication of numbers, some number of those multiplications every second, every cycle of the clock. You can do those multiplications. They can do more math than we can. We will also say in the U.S. that we do much higher quality math. So there are things that we can do at high precision that our systems are capable of that in general the Chinese systems are not capable of. So it's not totally fair. They're a little bigger. They're a little clunkier. Ours are not quite as fast, but they're much more precise in their capability.

50:47Ours also, because of the DOE investment in computing, tend to be far more energy efficient than the general approach in China. So right now we're having a national conversation about data centers and the amount of energy that they pull down. We're reaching a place where data centers are going to be starved by their capability to be powered. Some of our hyperscaler friends are looking for ways to co-locate as many computers, as many GPUs as they can without bringing the grid down in the state that they operate in. Being able to produce energy efficient GPUs like DOE has helped our industry partners to generate is going to be an advantage in being able to power those with an electric grid.

51:27So all of this comes together in picture that asks how much compute can you have? How much energy does it require? Does your grid infrastructure support being able to put that together? Can you do it in a way that's not climate destroying? Is your energy clean in a way that you can produce it? Or is it not? This all gets wrapped up into a notion of sort of techno-economic competition between the two countries. It's in vogue to talk about China as an adversary in that sense, but it's very real. They're the only competitor out there that operates at that scale. And they've clearly stated they have the ambition to win that race because right now in the compute world, if you look at something like OpenAI has done, they are turning power into GDP, into domestic product.

52:11What you need to drive that engine is more power into the grid, more efficient computers for turning that power more effectively into economic product. Don't turn that off. Win that race. that's what we're trying to do in the US. This is not lost on China. Yeah, yeah. Is there something that you've been talking about here that I haven't touched on? No, actually, you've guided us through pretty much the whole experience of what's going on through the really wonderful SCSB Expo that we're at and the Ash Carter Exchange that we're co-located next to. It's a good chance to recap. One thing that I want people to understand is that if we had our preference in the world, the Department of Energy would be probably called the Department of Science.

52:56We do more physical science funding in the U.S. than any other agency. We operate at huge scale. We have as our responsibility laser focus, no pun intended, on national security missions from the nuclear stockpile to bioresilience to making very critical decisions for the intelligence community. And we do it with very sophisticated scientific underpinning on both compute and experiment. and now with a really leading capability in AI. And we're moving further. This fast effort is to take that capability and build out a transformational AI mechanism for the United States to make sure that we get out in front, that we have escape velocity, that we lead for national advantage and that we're not at a disadvantage on that global scale.

53:40So I appreciate the conversation that you led me through. We've touched on the science and we've touched on the compute. And what I hope people who are listening to this take away is that the U.S. can do a spectacular transformative thing in this moment. We need only have the will to do it. And I think we're very close. Quantum. Is Lawrence Livermore working or the National Labs? What are they doing on quantum? Are you a quantum optimist? And I would imagine that the compute power that you've been talking about would be important in the simulation power for solving some of the issues? Yeah, I'm a quantum enthusiast.

54:22In my view, quantum computing is still a physics experiment that's very close to becoming a computational capability. We have researchers who are working on both the physical side of building out quantum computing systems and on the algorithm side of developing algorithms for quantum computers to do the kinds of scientific things we're interested in that we've done with traditional computing. Christy Beck leads our Center for Quantum Computing, which was just launched at Livermore openly. In fact, last week, I think. We've done the research for a long time, but we've got an investment in making this a real path forward in computing.

54:59So you should check her and her team's work out. But I think it will be transformational. It's not a today solution, though. Today, it's about GPUs, lots of GPUs, how to power them, how to take algorithms that we understand today and make them work. Quantum computing offers the ability to potentially transform that for some types of computing, not for all. So we need to learn what that admixture looks like, the balance between traditional computing, von Neumann architectures, things like that, and quantum computing and other ways to move data to compute and vice versa. So I'm not a pessimist about quantum computing.

55:35It's just a future technology that's really exciting, if not yet today's solution. But we have absolutely got to push on that. It is another frontier. Do not be behind on frontiers. It would be a mistake. Hi, I wanted to jump in and give a shout out to our sponsor, SysAid. SysAid's vision is to lead organizations on a transformative journey toward AI-driven organizational processes and services, infusing intelligence and ease in the workday with SysAid co-pilot. SysAid orchestrates service management across the organization using generative AI that taps into specialized data accumulated from thousands of customers and millions of users.

56:22With zero setup required, SysAid's conversational AI manages employees' requests, assists with queries, and accelerates the resolution of issues. IT pros and service management leaders become pioneers, enabling productivity to thrive. Now employees can do what they're meant to do, and organizations are free to fulfill their purpose. Give SysAid a try.