#307 Steven Brightfield: How Neuromorphic Computing Cuts Inference Power by 10x

0:00So neuromorphics is really how our biology does computations in our brain. So we don't have circuits, we have neurons, biological neurons, and they're highly connected to other neurons, and they communicate to each other with pulses. We might call them spikes. So you see a spike of signal go from one neuron to the next, and if that spike is strong enough with other spikes coming into that same neuron, that triggers that neuron to spike downstream. So it's a sequence of each neuron is getting spikes inputs and deciding whether it spikes the output. So it's a kind of simple concept, but you put a billion of those together and connect them all together and you have really what the brain is.

0:43We're augmenting our business model so our customers can see the value of our IP before they make the risky decision to go and integrate it into a chip development that might cost them tens of millions of dollars. Build the future of multi-agent software with Agency. That's A-G-N-T-C-Y. Now an open source Linux Foundation project, Agency is building the Internet of Agents, a collaborative layer where AI agents can discover, connect, and work across any framework. All the pieces engineers need to deploy multi-agent systems now belong to everyone who builds on agency, including robust identity and access management that ensures every agent is authenticated and trusted before interacting.

1:39Agency also provides open, standardized tools for agent discovery, seamless protocols for agent-to-agent communication, and modular components for scalable workflows. collaborate with developers from cisco dell technologies google cloud oracle red hat and more than 75 other supporting companies to build next generation ai infrastructure together agency is dropping code specs and services no strings attached visit agency.org to contribute that's A-G-N-T-C-Y dot O-R-G. Hi, I'm Steve Brightfield. I'm the Chief Marketing Officer of BrainChip, and my background has been in the semiconductor business for three decades plus now.

2:38I was trained as an electrical engineer, Midwest, Purdue, and I went off and was focused by the first two-thirds of my career on designing digital signal processors into semiconductors and solving use case problems from consumer to military use cases with digital signal processors. I worked at a lot of large, some of the other companies, but my longest stint was at Qualcomm. And at Qualcomm, I worked to launch their first smartphone chip. And it's always interesting, I was in product management at the time, is that you never know what product is going to be successful in the future. And there was a lot of debate and arguments about why would we want to build a smartphone chip?

3:27Because nobody's going to want to watch videos on their phone. And, you know, my phone makes good calls right now. So we fought with the management and we launched that product. And of course, you know, everyone's got one in their pocket today. And I think if I look forward to brain chip, everybody was going to have a brain chip in their pocket or their lapel or their hat in the future. And they're thinking right now, why would I want that? Right. So it's it's it's been an exciting journey for me seeing, you know, products that I worked on that used to consume a large, you know, refrigerator, then go to a box, then go to a board, then go to a chip and then go to a little tiny piece of IP that's inside of a chip.

4:09That was my journey when I started working on GPS back at the very beginning when it was literally the size of a printer was your GPS receiver. And then we worked on making it smaller and smaller. Now it's in practically everything. And you would never imagine that. Right. And I think that's what's going to happen with AI. You know, we see AI today as these big data centers, supercomputers with nuclear power plants next to them. But really what's happening in the industry is AI is dropping into all these everyday products that we have today. Your Fitbit, your smartwatch, your mobile phone, every consumer product you're touching is going to have a little bit in it.

4:49And I think that's what I'm excited about Brainship because Brainship is focused on those small consumer devices that put AI in there to help the individual that's wearing it. not these big, you know, AI data centers that are gobbling people's information up and using them to, you know, sell advertising. Let's put it that way. Brain chip neuromorphic from the start. And let's define neuromorphic for listeners who aren't familiar with it. Sure. Well, let's talk about what neuromorphics is first, and then it'll be easier to explain brain chip because neuromorphics is really a study of the brain.

5:26It really is. And it means a lot to different things to people. And it's kind of good to baseline that. So neuromorphics is really how our biology does computations in our brain. So we don't have circuits, we have neurons, biological neurons, and they're highly connected to other neurons. And they communicate to each other with pulses. We might call them spikes. So So you see a spike of signal go from one neuron to the next. And if that spike is strong enough with other spikes coming into that same neuron, that triggers that neuron to spike downstream. So it's a sequence of each neuron is getting spikes inputs and deciding whether it spikes the output.

6:12So it's a kind of simple concept, but you put a billion of those together and connect them all together and you have really what the brain is. And it's really it's been an evolution of millions of years to come up with this architecture. Right. And the key metric of it, it's super efficient. And it had to be because if you look at evolution, evolution was about survival of the fittest. So the animal that could think better was survived better. The challenge was he had to have low enough power so he could cool it off. Right. And he could power with by eating. and so forth. So, you know, I think, you know, your brain is about a 25 watt light bulb.

6:56If you think about that, that's the power of computing, right? So how do you kind of take those advantages that the brain has and put it into the chip business, right? And that's where brain chip really came from. It was designed by a guy, Peter Manumet, one of our founders, who was the brain of the company, and he was studying the biology of the brain and the spiking networks. And then we had a Neil Mandar. He is the chip guy who worked in Silicon Valley here for many years building modems and other chips. And together they basically form brain chip to emulate or inspired by the brain's architecture, they created a digital architecture of a computer architecture.

7:39And what we do is we take the spikes and we convert those into digital signals, really. And can I just, for people that don't know, the neural networks that dominate AI today, when one layer or one node of the network passes information to the next node, that information goes through a computation and is passed on regardless of how important it is to the output. But in neuromorphic, each node has to sort of gather inputs till it reaches a threshold before it passes information along. Is that a fair? I couldn't have said it better, Craig. You understand this. I'm glad that we've been talking about this.

8:34So, yeah, I think people, you know, when people think about AI, they think about data centers. And then the next thing they think about is NVIDIA. And if you got to look back how NVIDIA got here today, they didn't create this to build AI. They had a graphics processor that they had to compute every pixel. So they couldn't do that with a conventional processor. So they designed this graphics processor that you could have lots of processors in parallel doing the pixel computations. Now, what happened was Jensen Wang learned that the scientific community was using his GPUs to do scientific computations.

9:14They were using it to do physics problems, do math problems. And those math problems were matrix multiplies, could be paralyzed very well on the GPU. So he decided to invest heavily in making him better at doing linear algebra or matrix multiplication math. And I think it was prescient of him to find out that when in, you know, the 2012 to 2014, when the great inventions of deep learning happened, they had used the same matrix multiplies. So he had a ready-made platform for them. The problem is, is that a matrix multiply, like you said, every data in the matrix has to be multiplied by every single data in the matrix by the way they're architected.

10:01And the problem is, is that if half of the values are zeros, it doesn't matter. You're multiplying zero times something and you're going to, you know, you're going to get zero, but they do it anyway. Because it's just a brute force way you do algebra, right?

10:18And to create brain chip, we just could not follow that path. So what we did was we create more of like a data flow processor where the data flows in. it charges up these neurons and if they fire, then it goes to the next level and computes it. But if it doesn't fire, the next 10 layers of that computations, they're not computed at all. So that's essentially the secret sauce of neuromorphics versus the traditional AI. And I like to call it, you know, traditionally, I is brute force math. And neuromorphics is brain inspired, elegant computing, right? It's very elegant how it works, right? And it took millions of years of evolution to get there.

11:05Yeah, I had on the program maybe a year or two ago a guy from Australia at Western – geez, I can't remember the name of the university. They're doing a brain-scale neuromorphic computer that will have as many connections as the brain. who knows what good it'll be but it'll uh it'll be an interesting exercise for first studying the brain or first studying neuromorphic computing but but what brain chip is doing because the intel has a neuromorphic chip and i think intel does ibm certainly does north star right Right. And but those are largely research. Products, isn't that right? Or are they putting them into commercial products?

12:02Intel has what's called a low high chip and that low high chip is neuromorphic computing chip. And IBM has True North. It's also neuromorphic computing chip. And both of those guys had big research organizations and they kind of funded that. Right. Brain chip actually didn't have a product or a chip for the for the first half of its existence. It was, you know, it was incubated in 20 2009 and I think 2011. He started, you know, opened his garage up and started earnestly building on it. Peter Vandenberg. And it wasn't until 2014 that he brought on Anil, the chip guy. And it still took him three to five years of research before they said, OK, we got something.

12:42we want to make a product. And then the last six years have been productizing that architecture and getting it designed into some very interesting use cases. I'm sorry, that's the Akita processor. That's correct. Akita is Greek for spiking. So that's spiking network that we talked about, right? So the difference between what Intel and IBM are doing is, is that since they kind of had that research, they never committed to a product with it. But they did manufacture some chips. And a lot of researchers have done a lot of work on those with very interesting and promising results. And I think you'll see that even IBM is scaling up and building a very large neuromorphic computer with their chips.

13:27But it's a government, it's another government research program. The primary difference between brain chip and the Intel and the IBM solutions was they were analog. So they truly tried to match the analog waveforms of the brain, whereas the brain chip made a digital equivalent of the analog waveform. So now you could easily manufacture a computer, digital computer chip using the approach. The chips that you, the analog chips that are made today for neuromorphics, they're notorious for, you know, you have to have them biased and temperature stabilized. And there's all the problems with analog, which is the reason we don't have a lot of analog computers today, are the problems that they're faced with with their neuromorphic chips.

14:16So one of the major innovations is changing it from an analog spike to a digital event. So we call it event-based computing in a way to differentiate it from a pure play analog neuromorphic computing technique. Yeah. And you can, by putting these on the edge, you can move inference to the edge. Is that right? Are they? Absolutely. So most of the training is still done brute force with the matrix multiplications, right? So that's where NVIDIA is selling a lot of these big brute force boxes to train on massive amounts of data. But once you train the model, now you can put that in a tiny device on the edge and do what you said was inferencing, which is actually, you know, trying to infer, you know, what you're looking at or what you're hearing or what you're, you know, sensing.

15:09Right. So that inferencing is done constantly on the edge and it makes a lot of sense to do the computations at the edge. So you have to send the data from the edge all the way to the cloud, compute it and then send it all the way back. And one is latency. It's a round trip. Two is cost. It costs money to go run on that server and bring it back. And the third is privacy. That data goes someplace and you don't know where it goes. And I think we've seen a lot of cases where large companies doing this, they're not respecting the data, the privacy rights. And this is still, I think, one of the most interesting surveys I saw about AI was 70 % of the people think AI is good and it's going to help them do things better and easier.

15:58And then the same population of 72 % said they're fearful for their privacy of their data using the AI. So it's like a double-edged sword. You know it's going to help you, but you also know it's exposing you at the same time. When you compute at the edge and you're not sending your data out, it kind of helps solve the other side of that problem. So, I mean, you do see the whole industry moving to the edge. It just takes longer because when you have it at a data center, you don't have to plan anything. It's like an ocean of computing. But if you have a bathtub of computing at the edge, you got to make sure you can fit your problem into it, right?

16:34So it's fitting the inference problem into the amount of compute you have. And you were saying that someday everybody will have a brain chip in their pocket. It's computing. The data that it's computing on is from sensors at the edge, right? It could be vision or audio or heat or pressure or anything. Is that right? What are some of the use cases that you guys see? Yeah, we can support any sensor data. So anywhere from vibrations to microphones to cameras to radar, LIDAR, ultrasound, even chemicals. We actually had a demo where we could do smell detection with our AI, and we could detect what kind of beer it was by tasting the beer and looking at the chemical sampling of it.

17:31So what really happens in the when we say the edge, sometimes we say on device. So it's right on the device that you have in your hand. It's not going off the device. The other term that's popular now is called physical AI. That means that this is AI that's interacting with the physical world. It's not in some tower someplace with a bunch of data computing away. It's actually grabbing data continuously from a sensor and computing it right there on the fly. And we call that streaming data. And when you have data streaming in, it's going to demand you do something with it because you have only two choices.

18:10You either compute it or you store it. Or I guess three. Or you transmit it to someplace else to compute it and store it. So it's a lot cheaper just to compute it right there at the edge. Are there things that you can't do, computations that are just too heavy? That's correct. Yes, that's correct. We designed our architecture, so really focusing on the detection and the classification using the neuromorphic principles. If you go to LLMs and these other algorithms that are getting really popular in generative AI, they're not necessarily a great target for that because they have a different computational profile, right?

18:56And they actually deploy those matrix multiplies effectively. We do have technology at Brainship that we think is a different technology, which is a different neural network architecture than transformers. Transformers is the foundation of large language models and generative AI today. But we've adapted what's called state space models, which is actually an innovation of transformers that's more computationally efficient. Yeah. And it's kind of the leading edge right now. A lot of people are shifting to state space because it has better memory and recall and all of those things. It's got less computations, too.

19:43Yeah. One of the things that I think people need to understand the edge is that just because you have a microphone there or a camera attached to your computer, right, sometimes nothing's happening, right? There's no audio or there's audio, but it's just noise. There's not anybody speaking or anything. And a camera, the camera could be staring at a blank white screen waiting for something to fly through it. But, you know, the level of activity in the camera is quite, you know, a lot lower than the amount of data coming out of a camera. So this, how much information is in the data stream is what we exploit with neuromorphic computing.

20:27The analogy I always like to give is if you're staring at a blank screen, your computer isn't working overtime calculating every pixel. Your brain would overheat, right? But when things move in that screen, you can detect it and process it and classify it instantaneously, right? Now, cameras today were designed like television sets. Every 30th of a second, they put a new picture up and they fake your brain out into thinking it's moving, right? So when we do inferencing, people put cameras hooked up to AI and they did the same thing. Every 30th of a second, they'd give a whole frame of data and they'd say, go compute it all.

21:07And if nothing happened in that 30th of a second, it would still compute every pixel as if it was gold and then come back and say, well, we didn't see anything. Neuromorphics is saying, compute the changes in the scene. So if you see a change, you generate a spike. So that blank screen is going to consume almost no power in the AI processing because there's no spikes going into the processor. As soon as spikes come in, it lights up the network and it starts generating spikes and pulses. And how those propagate through the network is how you recognize what that is being seen in that scene. Same thing in a microphone.

21:47Last time we spoke, you gave a very good explanation or example of doorbell cameras and how they just have to process this continual stream of data, even if there is no movement outside. Is that one of the use cases? That is, actually. You know, originally when those cameras came out, they would stream the data continuously from the Wi-Fi over your Internet connection to Amazon or somebody. And then it realized we're not making any money at this. It's costing us more to compute it than it is that we can charge the customer. Right. And the customer, they didn't want to pay one hundred dollars a month to stream the data just in case somebody walked by.

22:38So they've got intelligent. They're still using like this brute force technique, but they have a little detection that says, wait until we see something move, then we'll start crunching on it. But even then, you can miss things and you're still looking every 30th of a second. So it isn't like the brain where the brain can instantaneously detect something rather than waiting for that next frame to show up. So what we're seeing is they've made a lot of innovations to do a lot of local processing in a blink camera and everything. But the problem is this, even with all those optimizations, you got to go change the battery of that thing every month or so.

23:18And if it's hung up on a wall and it needs a ladder, it doesn't get changed or it costs you to do that. So you get another 10x reduction in the power with neuromorphics to do it where it's always on. and that can translate to 10 times longer battery life. Instead of every month, every year you change the battery. Now that's a game changer from the consumer, right? Just think if your smartwatch lasted a week instead of nine and a half hours, right? At the end of the day, my smartwatches ran out of energy before I did. And I'm like, hold on. Yeah. So extending the... Yeah? I was just going to say, I have an Oura ring and I've got a like, you know, one of those rings that reads whatever.

24:06But it keeps running out of battery. I don't imagine they were using neuromorphic. No, they're not. But I can't really disclose. But those are exactly the targets that Brainship has in a wearable. So wearables is a huge focus for us because if you wear something, you don't want to take it off all the time. You don't want to have to charge it all the time. but you want it to work all the time. And if you look at an Oura Ring now, it's got a microprocessor in it that it's constantly calculating all the time just in case something happens, right? And that just sucks the power. If you have something neuromorphic in there, it'll wake up when some data happens, right?

24:47And it's going to reduce the computational power. So we think wearables It extends into medical, industrial, defense, as well as all these consumer products where you have eyeglasses that'll have computations in them. Your earbuds will now become medically certified hearing aids just off the shelf with AI in them. The ring that you've got now, instead of having to send all that data to the cloud or process it, it'll process it on the ring. and it might light up a LED that says, hey, you need to drink hydration or you need to exercise or you need to do this without having an$800 smartphone in your pocket and an account to a cloud service that you pay$20 a month for it just so that it can grab the data off your ring and give you results, right?

25:38So those are great products and they built them quickly because they could leverage all that infrastructure, But the consumer would be great if I could use this without some of the limitations it has. Yeah. How are, I mean, you know, autonomous vehicles is another obvious application. Are they using Neuromorphic now? How are they processing LIDAR and image data? They can't be sending it to the cloud, obviously. No, it's computing. In fact, I think early in the days of autonomy, you open the trunk of your car, it was a supercomputer hidden in there, right? So it kind of significantly took away your baggage and so much power.

26:27You know, I think in the early days with cameras driving the vehicles, they were collecting a terabyte of data a day per car. And it was so much data that they basically had to have these huge magnetic tapes that they would pull out of the back of the trunks every day. and loaded to the data center for training. Now, one of the great advantages of neuromorphics is that it's dynamic. It's not waiting for every 30th of a second picture. It can, 1 ,000th of a second, it can detect a change in the signal, right? That means you can do lower latency detection, which is really critical in autonomy, right?

27:09And you can identify an object in less than a millisecond rather than 16.6 milliseconds, which is the dwell time you're waiting for that camera frame to show up. The other is it can be lower power. Are you guys selling the IP or are you designing chips? We're doing both. And part of the reason is that people need to see the proof points of the neuromorphics. If we had just IP, it'd be too hard for them to go, how do I solve my problem with this and try it out before I commit to it in my product, right? So we just announced this week we're going into volume production on a neuromorphic chip from BrainChip, and we're entering the volume semiconductor business.

28:01Not so much because we're changing business models. We're augmenting our business models. So our customers can see the value of our IP before they make the risky decision to go and integrate it into a chip development that might cost them tens of millions of dollars. So, for example, we have a customer right now using that chip in a wearable device where they have it in smart glasses and they can detect from your brain waves activities, whether it be migraine. In this case, they're detecting epileptic seizures. before they happen. So it's a very therapeutic kind of product, but it's really enabled because you got to do that computations all the time with the person.

28:46You can't sample the data, send it to the data center, and then come back and say, oh, you might have already had an event, right? You want to detect before the heart attack happens, not after it happens. I just want to confirm that you had what you think you had, right? And on the other side of spectrum we found people in the in the communications business and the defense and radios where they're looking for signals and neuromorphics is perfect for that because it can find these the signals in a lot of noise and it can pull them out with very efficient power we had a customer swapping out an nvidia jetson chip and putting our chips in there and he got like a tenfold improvement in his power efficiency.

29:34And for him, that was the key metric that he had to accomplish because he had a mobile platform. So when you, let me round that back to your question, autonomous, autonomous vehicles. It's not autonomous vehicles. It's everything that's autonomous, whether it be a robot, a car, a ship, a plane, a drone, anything that's moving, any machine that's moving constitutes a collision risk with a human. So how do you do that? you have to really have this solved and and who's uh who's uh manufacturing the chips for you we're getting a manufacturer here in the united states by global foundries uh they have a a foundry up in uh upstate new york it's a 22 nanometer chip yeah yeah i think that was the ibm's facility up there at a time, right?