Deploying Edge and Embedded AI Systems with Heather Gorr - #655

0:09All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington, and today I'm joined by Heather Gore. Heather is the Principal Matlab Product Marketing Manager at MathWorks. Heather, welcome to the podcast. Thank you so much, Sam. Happy to be here. I'm looking forward to our conversation. We will be digging into your experiences deploying machine learning and deep learning models to hardware devices, and in particular, some of the main use cases and considerations that users need to be aware of when they're working on embedded AI use cases. You come to machine learning by way of physics?

0:46Yes. Yes, I do. I think there are quite a few of us that do that. I studied physics for my PhD and everything. And during that process, I did some machine learning. I mean, this was 10 years ago before it was cool. But, you know, I was looking at different fluid concentrations, if you could see the different concentrations based on a microscope image. So had a lot of the image processing, image analysis, of course, fluid dynamics, physics things. But, you know, that was the application that maybe you could actually take a picture of a droplet of fluids or blood or tears or something and, you know, immediately have some ideas of what diagnoses could happen.

1:28So that really sparked my interest in machine learning. Of course, all the mathematics and physical things kind of go hand in hand. But that's also what led me to MathWorks because, of course, at the time, everyone was like, ooh, machine learning, you do that. Come join us. But really, so many users of MATLAB, Simulink, the population is quite large of engineers, scientists, folks that are doing similar things to what I was doing, kind of combining these excellent techniques and their engineering applications. So that's basically what I've worked on for the last 10 years since joining MathWorks and very in forums.

2:06Now I really focus on kind of this, I'll say, kind of educational aspect where kind of really advising folks on, again, how to incorporate all these cool, interesting machine learning things that they're hearing about into these devices and engineering applications where it's not necessarily that straightforward. So that's what I've been focusing on and hopefully lends itself to this conversation. Awesome. I've used MATLAB as an undergrad and even in grad school doing engineering oriented things. I think digital signal processing is one of the applications that I remember. And I think the company still does a lot of that.

2:45But it's been a while. Like, how are folks using machine learning and AI in the context of MATLAB nowadays? And I guess one of the things I'm most curious about is like, is it an alternative to Python? Are they used in concert with one another? Are we going to start already, Sam? We're going to have this like Python, MATLAB, R, C++, like I love this. No, that's a great question. The tool wars are omnipresent. Yes, but of course, in reality, and especially with edge applications like this, and just realistic system applications, people use all sorts of languages. And of course, you need to use them together.

3:20So I'm glad that you actually brought up some of your background being in controls signal processing, because that's exactly what we're seeing now where people are super psyched about reinforcement learning because it is a controls application. And so it's really kind of the AI research community and the engineering community are starting to converge now where people are really ready to incorporate some of these outstanding models that are going to do some great stuff to some of those traditional controls applications. And again, thinking about Python, let's step back a moment and think about like an application like this, where maybe your machine learning model or your deep learning model is a PyTorch TensorFlow state of the art thing.

4:04But you also need to enter the frequency domain to get some of those features and understand your signals and ultimately do stuff in the controls area and eventually put it onto a device. So that's really what we see a lot. Of course, there's still that fundamental research where I think the neural network toolbox was 1992. So it's not new. There's been people working on this since a long time, right? Maybe it doesn't feel like that long ago, but that was a long time ago. And some of the early applications for the neural network toolbox was not only research and thinking about your transfer functions and your mathematics, but there was a block that would put it into a simulink model for exactly what you were seeing, where you really need to incorporate these things into control systems and capture data from all these sensors and things.

5:00And that really lends itself to the MATLAB environment. And again, you can use whatever model you like, whatever's the coolest, latest, greatest thing. It's nice to be able to just incorporate quickly and then ultimately take it all, put it into C or HDL or whatever your device needs at the end of the day. Over the years doing the podcast, we've talked a bit about kind of this pendulum that swings between using traditional physics-based models for things like control systems to statistical machine learning models. And as you know, recently it's felt like kind of a one-way flow of kind of machine learning biting off more and more of the traditional physics-based things.

5:47Do you see it as kind of this one way progression or are you seeing interesting ways that folks are kind of incorporating both types of some modeling into their systems? Yeah, that's a great point. I'm really glad you brought that up because at first someone might say, oh, geez, why would you do all this physics stuff? I mean, of course, it's been a lot of time getting a PhD in it. Hopefully it matters still. But again, you know, there's data that you could use. There are predictions you could make. Why bother with that physical system? Right. But there's a great example of thinking it was Baker Hughes, but it was exactly this kind of situation where they're trying to do predictive maintenance on pumps.

6:26And you don't want to break a pump. I mean, anyone listening is going to know that you need a nice spread of data or else you're going to have to do some mathematical gymnastics for your imbalance. But most days, the pump's going OK. It's working. So you don't want to break it. It's very expensive just to get that data. So we see a lot of that where you have a physical model that you can just break. You can add leaks and change the parameters across the board to understand the depth of the problem, basically, without actually doing anything. So we see a lot of that where it's almost, I think of it as supplementing data for exactly training the models and doing, you know, I'm sure we'll get into this eventually, but the verification, validation, making sure those edge cases and those imbalanced data sets are going to be operating properly.

7:19So you can use the physical systems and the physical models to sort of supplement that data. And I hesitate, I don't know if it's still in fashion, but digital twin is something that we also see, you know, again, where they have an entire simulation side by side. You know, sometimes we even see this in a dashboard where they're looking at it, they can flip over to another tab, look at the actual monitoring situation, and then look at the digital twin, tweak something, compare and do analysis that way. So there's definitely a great combination of using those physical systems and haven't even touched on some of the things in like automotive, aerospace, those big kind of systems engineering situations where you really need just a little extra physical modeling, a little extra system modeling.

8:08And AI might just be one component of that. Can you speak to some additional use cases where you're deploying machine learning models to hardware devices? I love this one from Mercedes-Benz where they were working on sensors. Of course, a car has a lot going on, right? There's a lot of sensors, a ton of data and a ton of information, analysis, modeling and research, right? So they were doing the sensor simulation. They also had a Python deep learning model. I believe it was a PyTorch model. They even went through the quantization in Python. Again, forgive me, I won't name the name of the package, so I'm not wrong.

8:48But they actually did all of that in Python. But then when it came to putting it onto the hardware, they needed that extra step of ensuring that it was going to work, right? They used fixed point designer in MATLAB. But the idea is that it's quite an actually easy tool. We assume that people are doing their research, they're prototyping their artistic stuff in MATLAB. MATLAB, and then they're taking that next step to make sure this is going to work on the device. And so that's exactly what fixed point designer, some of the other tools in MATLAB and Simulink will help you do, highlight some of those things that might not work and go back.

9:25And so that's exactly what Mercedes-Benz did. And ultimately, you know, it's now working on a device. So there are quite a few in the automotive area, especially because again, there's so many sensors, there's a lot going on and a lot of value and not predictive analysis with all sorts of data. But, you know, again, there are many, many great examples of this. And I tend to lean on that Baker Hughes example because some of the challenges are very challenging because they're streaming its sensor data. There's a lot going on. And again, same with that Mercedes-Benz situation because they wanted to do their simulations in MATLAB.

10:06They wanted to do their data analysis, their signal processing in MATLAB. And then, of course, use some cool, interesting research that they found with PyTorch, with some quantization algorithm that they liked, and then ultimately brought it back into MATLAB to make sure these things are going to check out, that that floating point math that they come to know and love is going to actually work on whatever device they're using. So I'm imagining when you're working on the types of use cases that you're describing automotive and you've got large teams working on this. So it's not an independent solo data scientist trying to get this model onto a piece of hardware.

10:47It's embedded systems engineers and other folks. Can you talk a little bit about, I guess, the ways that these teams work together? Like who are the main subject matter experts, personas that are coming together to get a model into embedded device for a real application? That's a great question. It's good to kind of step back and think about everything that's going on and everyone involved in the system. Right. So you have people that are working on the sensors that are like hardware experts. And, you know, maybe in the old days, it was like hardware versus software. You know, no one gets along, but they ultimately have to test and have to work together with the software team to make sure everything's going to jive.

11:31Right. It's a really wonderful, interesting world, especially for AI. As we hear a lot more about explainability and more useful models that can help with, again, explainability. It's so important in this area because not only do you have to pass along to the next person. Right. So you have maybe the data scientist is coming up with their model and they're going to have to pass it off to a hardware engineer or someone who's going to implement this on a cloud and a dashboard or JavaScript thing. Right. They may or may not know much about, of course, wonderful techniques have been created and explored in this very podcast on how you can talk about this.

12:13But there's an extra level whenever it comes to these types of systems, right? Where think of automotive and aerospace, there are actually certification bodies, like you just can't put an algorithm, even F equals MA, that requires a whole thing, right? So, of course, latest and greatest in AI, this takes a long time. And you have lawyers and certification experts and legal teams and safety people to convince that, again, are not invested like we are in this world where we can say, oh, yeah, it's fine. Just look at the Shapley values. And like, trust me, they need a bit more than that. And so it kind of just lends itself to more communication.

12:53You know, I think that's something we take for granted when we think about explainability. but the fact that you have this gigantic team of every skill level. Thinking about a lot of semiconductor manufacturers, the people on the floor still need to know what's going on. The techs need to know what's going on with these models, these anomaly detection things that they're putting through the machines. Everyone needs to know at some level. So again, there's no fancy math that I can tell you other than the fancy math that has already been discussed on this podcast for explainability. But it's that communication, that documenting your experimentation, you know, really being a scientist about it and making sure everyone understands, even the user.

13:37And if, unfortunately, something were to go wrong, you need to make sure you justify exactly what happened and exactly the decisions and the research that you were doing to help everyone understand. Got it. All right. So let's talk about the process end to end and really try to understand the main considerations that folks on these teams need to contend with as they are deploying these models into these hardware systems. Does it start with the end? You've got this big model that you train for GPUs in the cloud and you just take that and you want to shove that at the last minute into a hardware system.

14:14Is that the way it usually works or you have to start sooner? Yeah, you have to just buy a little bit better hardware device. They're getting better and better. They have GPUs and everything now. I think that you're right, though, in assuming that you have to start with the end before you begin, right? And so that also lends itself to talking about the teams that are involved, right? So I'm data scientist, machine learning person. I do understand the rest of the stuff, but I'm more the model person, excited about the tweaking and the math and all that. But a project I was working on, again, exactly with that Baker Hughes model, it's a streaming data set, right?

14:48So any given point, there's only one to two seconds of data even capable of being on this device. So I start with my usual stuff, right? All kinds of fancy math to do my pre-processing and collect all my sensors together and some fancy synchronizations and downsampling and do and some big LSTM and everything. And then I talk to the next person in line and they're like, we need two seconds of data max. So we can't possibly interpret any more than that. We can't use or inference on anything like that. So whenever you're training the model, you can't use 10 seconds of data, five seconds of data, no matter what fancy signal processing you're doing.

15:29So that's the other thing. I love data stuff. And especially with signals, I'm like, yay, let's do this fancy thing and this cool thing. But you really need to think ahead of what's going to be happening on the device. Sometimes, depending on the way the system's, like the architecture is working, you know, sometimes the data processing could happen in the cloud or on a different device. And like there are different ways to set it up. But ultimately, that's the thing that you need to think about. And it might even require like whiteboards and like talking to humans and, you know, saying, hey, what do we need here in the end?

16:03That's going to help me start with a model. So again, maybe many people... It sounds like you're saying that to get a model onto a piece of hardware, you have to start all the way back at the data processing, data preparation. Absolutely. And what is coming in, how it's connected, what's the latency, how much time you have before you have a prediction. That's exactly it. Again, I think that many of us take it for granted. We're just faced with this big pile of data on wherever, my laptop in many cases sometimes. But that's not the case whenever you're on an edge device, whenever you're streaming data, they need nanosecond resolution for the computations, right?

16:40So you can't just start going in with all your floating point math, I guess. You need to kind of think ahead and back out and keep it as simple as possible, as fast. You know, again, really thinking about the requirements for the rest of the system. And sorry to offend people in the audience, but I think many of us in the research environments and we've got our data, we're just going to go. You know, we do all kinds of fancy data things and fancy feature selection things. But ultimately, that stuff needs to also happen in the end result. You know, this model is predicting every fraction of a nanosecond.

17:17And like you may or may not be able to do your fancy data stuff. Yeah, you also have to just let go. Sometimes it's just F equals MA. You can't always AI. Practically speaking, it's clearly that impacts the kind of algorithms that you would use. Are there also things that you might do within that data processing phase? Like do you pre-compute things or I'm aware it's going to be very use case dependent, But I'm wondering what the tools you would have as someone working on the data side in anticipation of getting a model into a hardware device. What are some of the things that you're thinking about or tools that you're applying to try to give yourself more?

17:58I guess you're trying to get more kind of leverage as a modeler down the line. Right, exactly. And that's one of the ways that MATLAB really comes into the picture because there's so many users that are kind of faced exactly with the situation with like, it's a sensor, okay, it has like, like missing data, all sorts of crazy stuff. You have to make the decision on how to get rid of that missing data, how to deal with it, all those things. And again, it's in that streaming fashion. So in the prototyping phase, there are apps, things like Live Editor Tasks, you can even send along to your friends to try out different algorithms.

18:35And that's one of the best ways to think of, again, I tend to look at the fanciest one possible, like some crazy, awesome regression algorithm, but maybe a nearest neighbor would do just as well, considering the limitations that I have, right? And then again, one of the things that you were just mentioning was the sort of nature of retaining some information. You know, maybe you have like filter information. You don't need to calculate that over and over and over again. You can have it cached even very, there are nice ways, of course, in MATLAB and many languages to have a nice binary file, a nice thing that you can just quickly access, cache tables sort of thing and carry on.

19:16So that's another great technique. We also see that a lot when people are trying to update models. That's not a thing. You don't train models on devices like that. Maybe a Raspberry Pi occasionally. But it's one of those things you can keep and cache as much information as possible, retain that in the model, you know, in the flow and apply, you know, do very small tweaks if you need to. But it's mostly changing your approach and thinking about instead of taking it offline, completely retraining a model. Remember, you're on a device, it's possible to cache a little bit and then update some parameters.

19:54You alluded earlier to simulation is, to what degree is simulation part of the data process when you're modeling or when you're building for these kinds of systems? Can I go back to that Baker Hughes model? You know, again, I think it's very common when you have these edge cases where most of the time stuff's okay. You don't want to break it, right? So if you can simulate it, if you can get enough data to train a reasonable model without breaking things, you know, hopefully you have, well, maybe not hopefully, but likely you have one or two cases of the bad things, right? And so you can use that to inform.

20:34Also, many times these are physical systems, like you mentioned before. So it's far easier to simulate very closely and very reasonably. So I think it's quite common, especially when it comes to the verification validation stage. You know, again, we're not trying to go crash a bunch of cars and crash a bunch of planes or something. We want that to be a simulation, a very, very realistic simulation. And so that comes into play a lot whenever you're talking about verifying the VNV, so to speak, the verification validation phase just before you deploy. A lot of that comes into play. Got it. So you've thought ahead about your inference requirements and your device capabilities.

21:17You've maybe adjusted the way that you approach data prep and some of the features that you're creating, maybe pre-computing some things or doing some simulations. You've got your data together. What changes from a model and modeling perspective when you're trying to deploy to a device? I think it kind of draws on a lot of the things that we were talking about. You know, you think ahead with your device. What are the limitations? And most of the time, obviously not floating point is not going to be a thing, right? So their integer calculations, it's just different. But that's also something that you can think ahead with.

21:53So for example, so many people are faced with these challenges that are MATLAB and Simulink users. There are a handful of apps that will actually help you take your model, put it through a quantization. You know, you can do some adjustments. It'll tell you potential layers, potential calculations that may or may not work on your device. It'll help you kind of with that process. But again, that's also something to think ahead with. And even thinking further ahead, how is this thing going to be shared and how do I have to justify it? So as a machine learning researcher, just starting the problem, I'm probably not going to go with the most complicated hammer because I know at the end of the day, I'm going to need to justify things.

22:34I'm going to make sure this explainability is valid and explainable to lawyers, to everyone downstream, to users even. So, again, all these things you need to think ahead. Generally, a simpler model is going to be easier to deploy, right? less ridiculous math. Not to say that you can't do it. People do it all the time. But it's just something to think about whenever instead of just like I do, go in and try all the most exciting thing that you just read about. But it's just not life. You need to think ahead and make sure these calculations are reasonable. But like I mentioned, there are great tools for that.

23:14I mentioned there a number of Python libraries will help you with quantization. Of course, a slew of MathWorks tools, depending on what type of device you're putting it on, just will help even give you sort of these templates for testing in the next environment. So it just makes life a lot easier whenever you're really trying to focus on your part of the problem. It helps you if you aren't that great at thinking ahead and thinking, oh, geez, that layer has floating point math, that's not going to work. Utilize the resources you have at your expense to help alert to these kinds of things. FPGA, GPU, HDL, all these things are different.

23:52So there are quite a few nice tools out there to help you just, again, make sure that model, the mathematical minutiae, as I call it, is going to work properly on that device with the precision requirements. So one big thing to think about is quantization and the precision requirements. You also mentioned explainability and how to ensure that you're able to articulate the kind of your model, the operational characteristics of your model. You know, thinking about these kinds of use cases we're talking about, you know, you mentioned you don't want to crash planes and cars to verify and validate.

24:33You also don't want to crash these things when your model's deployed. Like how does a modeler in these environments or someone working one of these large companies and one of these mission critical use cases, like how are they thinking about robustness for their models? Right. And that's a super important question. I think so many people will be happy that you asked that. There's this notion of model in the loop, software in the loop, processor in the loop, hardware in the loop. There's a process that folks will tend to go through in this verification validation space where, again, let's say even we're doing something fairly simple in a vehicle.

25:13Right. So you have some kind of model, some kind of cool thing in Simulink, and then you want to make sure it's going to work in the next phase. But you don't want to put all of the things together. Right. You want to go step by step. Right. So that's one of the notions that people will go through. Again, this is pretty standard with ISO certification processes. Some of the, you know, again, these applications that have had these certification bodies in place for, I don't know, 50 years, maybe. So these tend to be pretty accepted, but now we're trying to introduce AI into them, right? So just an extra thing to think about whenever model in the loop, you can use Simulink, you have your block diagram, you have the whole system, you have your AI in there.

25:57And of course, the verification stuff that you normally do with machine learning. But then the next phase is to take it out of that model and start putting into the software, right? So commonly, this will be C, C++, like something device-y, right? So you take it out of MATLAB, Python, Java, whatever you're doing, and you get it into that design space. So then you start testing in that software in the loop phase. Then you go even further and do it on the processor or the FPGA. So that's the PIL or FIL phase. And again, just taking step by step, making sure these calculations, you know, you're modeling them.

26:35But is that actually going to work? did I, like you said, crash the, not the car, but the device because I put too much data in the flow, right? And then finally, it's the hardware in the loop testing simulation process where it's actually on a device in the rest of the system, right? So that piece of whatever Mercedes-Benz example that I was talking about, that piece that they were talking about now needs to be tested within the rest of the conjunction of the system on that hardware device. So it sounds boring, maybe, but it's actually takes forever. It does. And there are people that literally, that's their entire job all day as they're just they're running simulations, they have some devices plugged in, it's actually kind of cool, especially when it's automotive.