#184 André van Schaik: Building An Artificial Brain With 100 Billion Neurons

0:00We've specced this system to effectively have the same number of neurons as a human brain, with the same number of average inputs per neuron as the human brain, running that at the same number of spikes processed per second as the human brain. Hi, I'm Craig Smith, and this is Eye on AI. In this episode, I speak with Andrei van Schaik, Director of the International Center for Neuromorphic Computing at Western Sydney University in Australia. We talked about DeepSouth, a brain-scale neuromorphic computing system with the ability to simulate up to 100 billion neurons in real time. Banschek explains how DeepSouth uses spiking neurons and synapses to process information more efficiently than traditional AI models.

0:52Deep South aims to help researchers better understand brain computation and potentially unlock novel AI architectures. I hope you find the conversation as incredible as I did. AI might be the most important new computer technology ever. It's storming every industry and literally billions of dollars are being invested. So buckle up. The problem is that AI needs a lot of speed and processing power. So how do you compete without costs spiraling out of control? It's time to upgrade to the next generation of the cloud, Oracle Cloud Infrastructure, or OCI. OCI is a single platform for your infrastructure, database, application development, and AI needs.

1:41OCI has four to eight times the bandwidth of other clouds, offers one consistent price instead of variable regional pricing, And of course, nobody does data better than Oracle. So now you can train your AI models at twice the speed and less than half the cost of other clouds. If you want to do more and spend less, like Uber, 8x8, and Databricks Mosaic, take a free test drive of OCI at oracle.com slash IonAI. That's E-Y-E-O-N-A-I, all run together. My name is André von Scheich. I'm the director of the International Center for Neuromorphic Systems at Western Sydney University. I've been in neuromorphic engineering almost since the beginning, since the early 90s, very early 90s, so it's getting on to 35 years now.

2:37And let's begin by explaining what neuromorphic computing is. I mean, I know that it's a different architecture that's meant to mimic the brain. I know that the neurons in, or the artificial neurons in a neuromorphic computer fire on action potentials and spike in the way that biological neurons do. But can you sort of break down, before we talk about Deep South, the computer that you guys are building, how the basic differences between von Neumann architecture and neuromorphic computing architecture, and specifically maybe start with action potentials and spikes as opposed to, you know, weights and what happens in an artificial neural network.

3:39To get back to neuromorphic computing or actually neuromorphic engineering to start with that, we're taking inspiration from neural systems in biology and try and implement that in technology. that means we also need to understand neural technology and biology to take that inspiration from it and there are sort of two main branches to neuromorphic engineering one is neuromorphic sensing taking inspiration of the senses that we have and how biology senses information around in the world and then neuromorphic computing is largely around how do we then process that information from such sensors like brains do in humans but also in dragonflies and mosquito.

4:23Neural computation then uses neurons. Neurons in biology indeed send action potentials to each other. They connect their axons onto dendrites of other neurons with synapses and these synapses typically have some strengths and that strengths can be modified through learning rules. So that is similar to weights in neural networks, but the spikes are the means of communication and they lead then to pulses on the postsynaptic neuron of a certain weight depending on the weight of the synapse. In this architecture of a brain, all neurons operate in parallel, they don't operate on a common clock, so it's asynchronous communication, it's pulse-based communication, and that the memory that these systems have is effectively in the synaptic strength, at least that's what we currently understand from how our brain works.

5:27And that's quite different from a von Neumann machine, where memory is a separate thing to the computer, and you're forever shuffling things back and forth from memory to the compute block and then write the result back to memory and then do the next operation in a sequential fashion. Yeah and also on the clock in von Neumann architecture and with artificial neural networks there is input information and there's calculation at the neuron level and then the result or the output of that neuron is passed on to another neuron according to the clock when the transistor resets, right? There has been work on building spiking neural networks.

6:19I know IBM has this TrueNorth chip. So can you talk about how a chip like TrueNorth works differently than a CPU or GPU processor? The chips like True North, and we've called this system Deep South slightly in reference to True North because we're down under and we're very far south in the world. And yeah, True North was sort of the first large scale neuromorphic system that was doing spiking neurons. This was part of a US DARPA funded project called Synapse that IBM did. that started a good decade ago now that that they did that that has been followed up by a few others that have been doing that most recent years it's been intel doing their low e platform which is similar in goals it's a different design i don't know the actual ins and outs of true north or of low e in terms of what actual architecture they they have on it but they are all aiming at emulating neurons that communicate with action potentials for spiking neural networks and implement those and run those.

7:38Deep South is a system like that too. We're emulating how a brain works on the electrical level by how it communicates with pulses and how it does communication with that. Some of these systems that have been designed, they have attempted to be fully asynchronous, just like the brain is, and not do it on a common clock. And that is quite difficult, actually, to get to work properly. So most of these systems, day it will still have a clock deep south for example it's built out of fpgas and field programmable gate arrays that's reconfigurable hardware that operates on a clock and so our system is not asynchronous in in that sense and and the point of the the clock in a current architecture is the calculation is done all at the same time across the transistors or across the chips.

8:47Is that right? It is. And the other point of the clock for us is that so deep south has almost 100 FPGA boards in it. We can do up to almost 100 billion neurons that we can simulate with that in real time. But that doesn't mean that every 100 billion neuron has its separate implementation in the hardware. In order to get to these numbers, we have to use the fact that silicon is much faster than biology. So we have one block compute, say, 128 neurons, one after the other in a sequential fashion. And again, that clock is needed to schedule that, Yeah. But the point of spiking, asynchronous spiking neurons in the brain is that they only spike when the action potential is reached, when they have enough stimulus to send a signal.

9:45And in an artificial neural network, they all update at the same time according to the clock. Am I misunderstanding that? No, that is correct. And that's a good point. So we do make use of that sparsity of signals that you see in biology where you only communicate when you have a spike. So when a neuron has passed a threshold, obviously the neuron itself is still updating even if it hasn't spiked based on its inputs until it reaches a threshold though, other neurons wouldn't know about that. And that gets used in this system too. So while the neuron internal updates are done on the clock, only those neurons that spike will actually communicate with the other neurons in the network.

10:31Right. And in an artificial neural network, regardless of the input, the strength or relevance of the input to an artificial neuron, it will communicate onward. is it's just a matter of how strong a signal it sends. Yeah, in a artificial neural network like current deep neural networks and convolutional neural networks, all the values from all the neurons are passed on to the next layer, right, and processed by the weights, including zeros. And is that why artificial neural networks consume so much energy and why neuromorphic computing consumes so little energy? Yes, partly at least. It is not making use of the sparsity, so all the operations have to be done.

11:20Another part that is expensive is this constant shuffling of data from memory to the compute and then back to memory. With the FPGAs, we can't totally avoid that either. But we have in our system quite a lot of local memory, high bandwidth memory integrated with the FPGA nearby, which reduces some of the energy costs of moving that data. Because again, as I mentioned, one block will calculate multiple neurons one after the other in a scheme that's called time multiplexing. And in order to do that, you need to store the internal state of each neuron back in local memory when you're updating the next one and then writing that result back to local memory as well.

12:09So we still have some of that, But we try and keep it as much local and distributed across all the FPGAs, and each of the FPGA blocks will have its own memory. Yeah. Can you just give us a thumbnail on what an FPGA is? Sure. So an FPGA is a field programmable gate array. It's a digital platform that can be programmed a bit like a CPU. But by programming it, you're actually setting hardware switches and basically changing how one block on the hardware is routed to the next block and how things are connected to do certain operations. So you're really configuring the hardware and you can do that multiple times, which is why this is also called reconfigurable hardware.

12:57And so once you've programmed it, you actually have a hardware circuit that sits there that does the calculation that you have programmed. Whereas on a CPU, when you program it, the hardware has stayed the same. You've got your CPU that's doing the calculations, and it is looking at what instructions it now needs to do from instruction memory and computing these instructions one after the other, but with exactly the same hardware that hasn't been changed no matter what program you're sticking. And that reprogramming, is that dynamic depending on the sensory input that the FPGAs are receiving? Or is that something that's physically reprogrammed by a technician?

13:45So that's something that's physically done. And so we try not to change the configuration of the hardware for the various models that DeepSouth might simulate. So the user will specify an architecture of their spiking neural network that they will want to run on it. It runs on the same hardware. We don't have to reprogram the hardware for that. We've configured it already to simulate the neurons and the synapses and the connections, and those things are available. And users would simply have a software description of their network architecture that they want, and that then gets sent to the memory of the devices.

14:31And that can run on that hardware without having to reprogram the hardware. Where the reconfigurability does come in is if we want to add other features to it, new learning rules, different neuron models, things like that. We can still add those after we've built this machine. And that's a big difference going back to, say, TrueNorth or Intel OEE. Those are chips that are made, custom made. and what's on them is what's on them and you can't add things to that without doing a whole chip design and fabrication and that's expensive and slow if you look at intel low e he to go from low e he won to low e he too that came out two years ago i think there was a six-year cycle and so doing updates and adding a learning rule to that that's not quick whereas for us with this hardware we can do that yeah that's fascinating the networks uh among uh neurons uh can can reconfigure themselves uh dynamically right i mean that's they're not fixed yeah and we tend to simulate that by changing the weights of connections so weights can go all the way to zero which effectively means the connection has disappeared and can grow and connections can start as well.

15:55Typically, I've seen networks defined where the connectivity is defined by the user. It might be a probabilistic definition in saying from this group of neurons, that group of neurons, there's a 20 % probability of a connection from one neuron to the other or something like that, rather than saying, oh yeah, this one goes exactly to this one, this one goes exactly to this one. But that does tend to be static in most simulations at the moment, and that is because adding this flexibility of reconfiguring all the connectivity indeed makes the simulations even harder. It is something we can support on this machine, though.

16:41Where are you with DeepSouth? Is it finished and in the testing phase? Are you still building? We're building it. And actually physically over the last few days, we've had the hardware put in the racks. That's still ongoing and it's being connected. It should be connected up in the next few weeks, all fully connected, and then we can run some tests. We're now looking to launch it in the beginning of June. Part of that delay is I'm going to travel for the next seven weeks, and I want to be there when we launch the machine. Yeah, and what will you launch it on? What sort of computation are you going to ask it to do?

17:35Well, it's called balanced excitation inhibition networks. It's a sort of traditional setup in spiking neural networks where it's based on what we see in cortex, in the human brain, where we have about 80 % excitatory neurons, so neurons that excite other neurons, and 20 % inhibitory neurons, neurons that inhibit or reduce the excitation of other neurons. Those are connected in networks that keep the excitation and inhibition balanced. That's why they're called balanced excitation inhibition networks and keep the activity level of the brain within a reasonable amount. If you just had excitatory neurons, then the excitation could easily run away and you end up with everything spiking all the time.

18:24And obviously, if neurons were just inhibiting each other, nothing would ever happen. So it's that balance there. That in itself is not a useful computation, but it is an easy demonstration for us to show that everything works, that all the neurons are computing as they should, that we're able to achieve the balanced excitation and inhibition with a very large population, that we can run that in real time, that all the populations can communicate to each other as needed and so on. So it's a good functionality test, but it's not doing anything intelligent, basically. Given that this is based on the biological structures of the brain and that artificial neural networks that run on von Neumann architecture are mimicking the neural activity of the brain, Do you expect then to run neural network architectures on the neuromorphic computer on Deep South?

19:33We can support running artificial neural network type architectures on the machine. It is not the initial plan to do that, but we specifically want this to run spiking neural networks. The reason for this is that artificial neural networks, the convolutional neural networks, underpin current machine learning and AI. They run very well on GPUs, graphical processing units. And we can do that now. Pretty large networks, it still takes a fair amount to train them. But then once they're trained, we can execute them in reasonable amount of time or pretty quickly. And they run well on that. And that was effectively a revolution or an evolution, because the GPU was more of an evolution than a revolution, I think, that really helped machine learning and artificial neural networks to get to the power that they needed and get to the size that they needed to be able to do the things that they can do now.

20:40I'm sort of hoping that with Deep South, which is aiming to be able to run spiking neural networks efficiently and large networks efficiently, that we can do the same thing for spiking neural networks. because spiking neural networks don't run well on GPUs or on CPUs. They're very slow to simulate. And that means that researchers that have been trying to understand spiking neural networks at scale, they've always had to simulate small models, toy models, basically. And that's a problem because, like, if you look at, say, a neuron in cortex, You can have 10 ,000 inputs, one neuron from other neurons.

21:27And if you're only simulating a network with 1 ,000 neurons in it, because that's what you can simulate on the computer, how do you get 10 ,000 inputs? And you can go, well, I'll just do 100 inputs instead and make each input 100 times stronger so they get the same total input strength. But you can imagine that if you get spikes that are 100 times stronger, one spike in just means one spike out. And the dynamics of the computation gets totally different. So you can't play that trick very well. So we really hope that this will enable researchers worldwide, ultimately. In first instance, we have to get this up and running for us, but it will be available for other people to use, and they can create accounts on it, and it'll be a cloud service effectively.

22:11And the other important point about DeepSouth is that it's custom, sorry, it's non-custom hardware. It's commercial hardware. so you can just buy the components and build one yourself and use the configuration that we've designed for the FPGAs configure it the same and you have a copy of the machine and again that's not so easy when you use custom-made chips often there's only a limited volume of those chips around and only one machine is built and that's what people have then maybe get remote access to Here, FPGAs is a commercial business. These companies, they make them, they improve them, they sell millions of them, they will continue to do that.

22:56People can buy this stuff. Or do you need new algorithms then to run on this machine, on spiking neural networks, or are they transferable? The transformer algorithm, for example, it's done so much in generative AI. Will we be able to run that algorithm? on a neuromorphic computer on DeepSouth? So there are spiking versions of, say, a transformer architecture that have been created. So they can be trained and they can be made, and we could run them. The thing is taking a transformer from an artificial neural network and making it spiking tends to work almost as good as the non-spiking version.

23:44but there's not much point then in making this the spiking version other than to say see i've made it spike our brain doesn't exactly do the transformer implementations i believe i believe there are indeed other architectures that are going to really unlock the advantages of spiking neural networks and those are the ones that are yet to be discovered and i think we need to discover them by trying things at scale with these spiking neural networks. Just like the transformer architecture for artificial neural networks was only discovered once we could make really large artificial neural networks.

24:27Back in the 80s and 90s, when we were doing three-layer artificial neural networks, there was no transformer architecture to be found because it just was not possible to make them. Yeah. Although the power consumption of the big transformer models is so enormous, they would be a lot cheaper to run in a spiking neural network on a neuromorphic computer, wouldn't they? That's the hope, yeah. So this system that we're building, it's not extremely low power. and it's about 40 kilowatts maximum power the system which is about the building air conditioning of a medium-sized building or something like that it's not a power that you see for data centers where it's megawatts it's not portable power either it's somewhere in between but that can be improved once we know what we want to do with it then if you really want to go to low power then you will have to go and build some custom chips to really get to the low power end.

25:36But my view on that is we still don't quite know what architectures we need, what type of features we need from the neurons, what type of learning rules we're going to need exactly. So let's explore that on this machine that is flexible first. And once we've figured that out, then we can start making some chips that make it really low power. And who are you talking to in the research community? You know, I started this podcast after meeting Jeff Hinton, and he's been on a few times. And he's talked about spiking neural networks, and, you know, his ambition is not so much to create AGI, but to understand how the brain works.

26:21And the last time I spoke to him, he was working on forward-to-forward networks. Is that what he called them? He was looking for an algorithm that could work in the brain because he didn't believe that backpropagation could work in the brain. Are you talking to people like Jeff that could use this computer to explore some of their ideas? Not yet, at least not yet broadly. We have, as part of the application for funding that we did to get this system built, we have a team of about, off the top of my head, 13 researchers, largely in Australia because this is Australian funding, but not exclusively.

27:17So one of the people we have as part of the team is Emre Neftsi. He's in Aachen in Jürich where there's a big supercomputer center. and he is one of the researchers that started the surrogate gradient approach that is used to train, to do backpropagation with spiking neural networks because the issue with spiking neural networks and error backpropagation is you, in artificial neural networks, you pass the error back through the gradients of the activation function. But in the spiking neural network, it's a discontinuous activation function because it either spikes or not. So you don't have a gradient that you can pass things through.

28:08So you have to come up with surrogate gradients in order to do that. And so that's one of the approaches. But I would agree with Jeff that I don't think the brain is doing backpropagation that way. and there's other architectures and learning mechanisms with local learning that need to be used in order to train such networks. Can you talk a little bit about the number of connections in Deep South? We've specced this system to effectively do the same number or have the same number of neurons as a human brain with the same number of average inputs per neuron as the human brain, running that at the same number of spikes processed per second as the human brain.

29:03And that's the number of synaptic operations per second, so that's a spike arriving at a synapse activating the neuron, the post-synaptic neuron, is in the order of 220 trillion per second. and that is what this machine can do. Obviously, they're very simplified neuron models. They don't have the full 3D structure that biological neurons have. They're digital neurons because we still store the states in memory. The precision of that is limited to 8 bits only because we can't have the precision too high or we would need to store too much in memory, for instance. So it's a simplified human brain scale spiking neural network simulator.

29:57But it is the first time we achieve this, I think, in the world. Yeah, that's really remarkable. And what did you say, 40 kilowatts? Is that what you said? Yeah, 40 kilowatts power usage. As opposed to, what, 20 watts in the brain? As opposed to 20 watts in the brain, yeah. But compared to, I don't know what GPT-4 is consuming. Yeah, I don't know either, but you hear the stories of it's the power consumption of New York City for a certain extent of time just to train a network like that. And is this an open source architecture? You were saying that people can then build these wherever they want.

30:47at a reasonable cost yes that's the plan um so obviously this year we will launch it then we will develop stuff for it internally in the lab and make sure everything works and so on for the remainder of this year then it will be opened up to the researchers we're that are part of the team and we're working with directly but that are remote at other universities around the world and then we'll grow that community and open it up to other people. And at that point in time too, our design will be made open source available so that people can copy the machine and they don't have to do it. So for us, it's three data center racks.

31:32It's 24 server computers with four FPGA boards in them. And that's an expensive machine, but you could scale it down to just use one FPGA and obviously much smaller neural networks or two or eight. Or if somebody is interested and has a lot of money to burn on this, they can make a superhuman brain basically with it, right? Or at least superhuman brain scale, I should say, because I don't know whether it's a full brain yet. Yeah, at current prices for FPGA boards, what would it cost for somebody to build? I mean, obviously, they're benefiting from all your time and effort. But to replicate this, what's the ballpark for building one of these machines now, once it's open source?

32:28Probably around 2 million US in terms of hardware. so it's not cheap but it's not way out there either yeah I mean it's within reach of a lot of companies and the the algorithmic the software research how far along is that how far does it need to go for your computer to be useful or doing some amazing things. Yeah, I would like to say, oh, we're ready to go there and all that is good. But actually, I think we have a long way to go there. And that is like my first goal with this machine is to better understand how the brain does computation with its electrical pulses. And we know some of the principles, but we haven't been able to study this at any large scale yet simply because we haven't been able to simulate it.

33:35And you can look at real brains, but there you can't observe everything you might want to observe, plus you don't have full control over what is going on either. So there's a lot to be discovered still around what works and what doesn't work. And once we have that, then we can start applying that to tasks, basically make AI. That's how I think of machine learning, right? You're learning to do tasks. And once we know some of the principles, then we can start working on doing tasks with the machine. But we first still have to discover the principles that work really well. And that's going to be a big research effort.

34:20And this is mimicking, when you say brain scale, It's the neocortex, or does it include the thalamus and the hippocampus and all the other structures in a human brain? So the numbers I used before and used to spec this system for, that includes all the neurons in the brain, including cerebellum, which actually has more neurons in it than cortex, for instance. Now, what the user ends up doing with these neurons depends on what model they define. So the model can be defined to specifically have thalamus or superior colliculus or cochlear nucleus if you want to do audio processing or all those parts in it as well.

35:13Or cerebellum, say you want to do some motor processing. Because it is a cloud system effectively still in server wrap, it's not ideally suited to do robotic motor processing and control loops because the latency would become an issue. What is your background in neuroscience? What was your educational background preceding this project? So my background was in integrated circuit design, my education. and first job. But in neuromorphic engineering, almost from the beginning of my final year undergraduate thesis was around making microelectronics circuits for artificial neural networks. And that was in 1990.