YOLOv9: Computer vision is alive and well

0:05Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at Fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents. So you can launch your app near your users. Learn more at Fly.io.

0:42Welcome to another fully connected episode of the Practical AI podcast. In these fully connected episodes, Chris and I keep you fully connected with everything that's happening in the AI and machine learning world. We'll take some time to dig into the latest news articles and releases from the AI community and hopefully share some learning resources that'll help you level up your machine learning game. My name is Daniel Whitenack. I am the founder and CEO at Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

1:24Doing great, Daniel. How's it going? It's going great. I'm spending a few weeks in the UK, which is a lot of fun and have got enough sleep to not be jet lagged quite as much. So that's encouraging. Okay. So we have a transatlantic podcast going here today. Exactly. Worldwide. That's right. Across the pond. Practical AI worldwide. 21st century incorporated. I don't know. We need rebranding. Exactly. Yeah. Yeah. Well, Chris, one of the things I was going through, I don't know how often people are flying these days, but one of the things that stood out to me as I took my flight across the pond was now when you board at least some flights, you don't even give them your ticket, right?

2:14You just go up and there's a little, I guess you would call it a kiosk, a little edge device that takes a picture of your face and matches it, I assume, with what was your scanned passport, which you scanned at the time of check -in and you board your plane, of course. And it was really, really fast as well. And the same thing happened, you know, crossing into the border, into the UK. As long as you have a certain passport, you just go up to the little machine and scan your passport and then it takes your picture. And I'm assuming I could do a little bit of research. I'm assuming what's happening under the hood is that it's matching your actual facial features up with the image on your passport and computing some score of shadiness or something like that or risk associated with you not being the person in the end.

3:10But I was amazed at how fast it was. And I'm assuming I could be wrong. I'm assuming maybe some of that's running at the edge, not reliant on an internet connection to do that facial recognition. I'm not sure if you know or if you've had also this experience, Chris. I don't know what they're using algorithmically, but I definitely partake of the technology. It's an area that I forgo privacy and always buy my way into expeditious processing. So, yes, I'm curious. Well, I don't know in that case if you have a choice. Maybe there is an opt out situation or something. I'm not sure. But it's pretty cool that some of this technology is being applied at the edge and in a very seemingly efficient way such that you could use it on a mass scale like that.

3:59Or I don't know if you'd consider that a mass scale, but it's definitely in use for many. You know, there's a huge flood of people going through those stalls and the computation happens very quickly. and reliably enough to make a judgment. In the midst of all the hype around generative AI, one of the things that stood out to me over this last news cycle, Chris, was the release of YOLO V9. So we're on the ninth iteration of this YOLO model. Did you happen to see any of the videos of YOLO 9 in action, Chris? I haven't seen the YOLO 9 one, But I'm kind of stunned. You know, when you think about it, YOLO has been around a long time.

4:42I was occurring to me because we actually had some conversations about YOLO back in the very first days of this podcast, which has been, you know, closing in on six years now. So it's V9 is a long time coming. And we haven't really gone back and touched such models in quite a while. It's we're long overdue. Yeah, yeah. So as everyone is freaking out and enjoying the hype over large language models and other generative types of models, Sora and all the things coming out in the background somewhere, there's these amazing computer vision people that are just really cranking and innovating actually at the architecture level of neural networks in really interesting ways.

5:26So it might be good to set a little bit of background for this. Chris, you mentioned we've been kind of talking about YOLO for some time. So if people just search for YOLO, Y-O-L-O, object detection, you'll see a huge set of articles and GitHub and everything about YOLO. YOLO actually kind of made a splash because it processed entire images in a single pass for object detection and bounding box detection. So if you think about if you've ever seen one of those videos of like a street with a bunch of people walking around and cars and dogs and shops and scooters and whatever. With their boxes around them.

6:11Yeah. And they have their boxes around them and they're labeled person or whatever. that's likely YOLO. So what happens is that single image in a YOLO model goes into the model and then out comes the bounding boxes and the actual classification of those bounding boxes, which is interesting because previous models, previous to YOLO, I'm still sure some models do this in a multi-stage way, which is more computationally expensive. So they actually take multiple passes through a model or multiple models to compute both the bounding boxes and the classes. Yeah, I remember way back when we were first starting and I was at a different employer.

6:54I was at Honeywell leading AI there at the time. I remember just as YOLO2 came out, we were using that for a couple of projects that we were working on way back in the day. But that seems, I mean, that's like before dinosaurs were in the earth by AI standards. But yeah, way back. Yeah. And I think even we had a podcast episode maybe about fast R-CNN or whatever it's called, the fast version of R-CNN. Good memory. Yeah. Good memory. That one's cool. I mean, that one, I think how that one worked was you pass your image in and then it detects the bounding boxes of objects. And then in a second pass, it then classifies each kind of subsection of the image as its class, which also is very effective, but it's less efficient computationally than the YOLO kind of single pass thing.

7:50And as you mentioned, there've been multiple versions of this. So between YOLO and now version two, version three, all the way up to version nine, kind of each version of these in some ways has, and not just in a kind of more train with more data way, they've actually made kind of very significant discoveries and improvements in neural network architecture, training methodologies, this sort of thing that has led it to be kind of the go to solution for at least real time. object detection in images, which is why you see all these videos of the bounding boxes around people and such. They've at least gotten the visual bit a little bit nicer than they used to, where you had the big clunky boxes overlaying everything.

8:38Correct. Yeah. Well, the V9 version of the project, which dropped, at least if the date on the archive article link is right, That would have been the 21st of February of 2024 as we're recording this. So not that long ago, but it was developed by an open source team and kind of built on top of a code base from Ultralytics YOLO version 5. And it's released, I believe, under the GPL 3 license is the code that they released. But it seems like what they focused on with YOLO v9 was continued focus on efficiency to where you can do real-time object detection, meaning as the frames of a video are coming in, you can process those in real time with the model.

9:34So efficiency is really key in these types of applications. And then they focused on one of the fundamental challenges of deep learning models of these deep neural network models, which is called the information bottleneck principle, which happens because especially as you kind of propagate, if you think about a neural network, what it is is a big data transformation, right? You take a bunch of matrix data in the front end, maybe representative of an image, and that gets processed through successive layers of processing. And then out the other end comes maybe these indication of classes or other things.

10:16And the information bottleneck principle talks about the errors or the lack of information or the loss of information that you lose as you process an input through the successive layers of the feed forward process of that neural network, which in some ways can be addressed by having bigger networks and more data. Maybe you're less prone to these informational problems, but it's more of a problem when you're dealing with these very efficient, lightweight networks like the YOLO networks, because you have less layers to deal with and you don't want to lose any information that might be relevant to the classification of the outputs.

11:01I notice within YOLO's nine docs, they talk about also reversible functions as well. Does that feed into, no pun intended. Does that feed into the ability to not lose data by reversing that feed forward through a function backward? How do you see that utility? Yeah, so the interesting way that they dealt with this or kind of addressed this, at least in this version of the model, is something that they're calling Programmable Gradient Information, or PGI. and the PGI portion of their research and advancement relies on a couple of things. But one of the main things is this focus on, again, improving the informational efficiency of the network.

11:52And one of the ways that they've done this is with what they call an auxiliary reversible branch. And this gets to these reversible functions that you mentioned. So the concept of a reversible function for those that maybe that's new to them means that the function and the inverse of the function can transform data without the loss of information. And so again, there's that loss of information piece there. And so it's a little bit hard to describe this on the podcast without having a whiteboard or a visual. But if you think about this PGI functionality that they've added into the network, it's kind of like they're bolting on this auxiliary reversible branch, which helps deal with this information loss as gradients are calculated during the training process.

12:42And so during the training process, this reversible branch helps not lose that gradient information as during the forward pass and during the calculation of the updates of the weights of the model. And that helps it be very efficient during the training process, but it's called auxiliary, which is key because you can actually unbolt it and take it off for inference, which means I think part of the problem in the past with these reversible branches and efforts at this were helped with the information loss, but it also decreases the efficiency in terms of computational efficiency of the model during inference.

13:23I'm going to throw a question at you and I realize this is not your thing, but just in case, is you're using a reversible function in that programmable gradient information process that you're talking about. and in a normal feed forward network, you know, you're maintaining the weights as they're going through and change those are, are changed. And are you reversing functions to maintain that back in the same space to where you're actually maintaining a new weight and you're keeping that, that gradient information for maybe future feed forward passes, or do you have any sense of what the purpose of that is?

13:57Yeah, I think that, uh, so definitely we'll link some of the papers and the explanations in the show notes. So feel free to look at that for accurate information and let us know if we get it wrong. But yeah, I think that the idea is that, and the reason why this is especially useful in the training side of what they're trying to do, and it's kind of unbolted during the inference side, is that during the training time, it's really crucial that as you're calculating the updates to your weights, you can do that in a very informationally accurate, precise manner, especially for these lightweight networks, which have fewer parameters to train.

14:44And so maintaining that information, especially as you're calculating updates based on the gradients is really important. Gotcha.

15:08This is a Changelog Newsbreak. Shipping quality software in hostile environments. Luca Klederic writes, quote, I once had the opportunity to work for a startup that had fallen from tech debt into tech bankruptcy. Bankruptcy, Michael, is nature's do-over. It's a fresh start. It's a clean slate. like the witness protection program exactly not at all although we managed to get it back on the right track it made me rethink the concept of tech debt and how we ship software especially in hostile environments end quote he goes on to tell this true story in great detail which is horrifying yet echoes so many of our experiences here's just one of the many horror scenes luca describes quote there is also a handcrafted build server a jenkins box hosted in the office but no record of how it's provisioned or configured.

16:00If something were to happen to it, the way you build software would just be lost. Each job on it is subtly different. Even for the same tech, you have an Android source code that you build three instances out of, but each of them builds in a different way. End quote. This is a solid essay replete with warnings and a plea at the end to ditch the tech debt concept altogether. You just heard one of our five top stories from Monday's Changelog News. Subscribe to the podcast to get all of the week's top stories and pop your email address in at changelog.com slash news to also receive our free companion email with even more developer news worth your attention.

16:42Once again, that's changelog.com slash news.

16:51We talked a little bit about YOLO version 9's programmable gradient information. I had to remind myself, PGI, programmable gradient information. The other piece of the architecture, and I think this is just really interesting. You've sort of got all of this going on on the LLM side where things are getting very interesting ways to fine tune and preference tune and all these families of models. On the computer vision side, man, they're really, really thinking deeply about the architectures going into these models, which have made them so, so efficient. The other thing that kind of is a combination of things that have come in the past that they're utilizing in this YOLO v9 is a generalized Elon architecture.

17:41So this is kind of a progression of a couple of things that have been in YOLO models in previous generations, but they've combined them in kind of a unique way. This stands for Generalized Efficient Layer Aggregation Network or G-Elon. And this combines a couple of things from previous generations of YOLO and from things like CSPNet. This has to do with how features are aggregated and gradients are aggregated through the model in a very efficient way. Again, leading to a very parameter efficient model, meaning a smaller set of parameters in YOLO V9 will have similar performance to maybe models with many more parameters.

18:31So this leads to the efficiency overall. It's pretty interesting. They talk about being able to adapt to a much wider range of applications without sacrificing speed or accuracy. is that a form of fine-tuning the model or something that they're doing ahead of time that you're then fine-tuning on top of that? At least how I read some of that flexibility in was, yes, there's kind of a parameter efficient, this is a parameter efficient setup for fine-tuning maybe to a variety of types of scenarios or even training a new model from scratch in an entirely new domain and doing that very efficient. And some of the things that I've seen, people have already quantized this model using things like OpenVINO, which is very popular for these kind of edge vision cases, and running this very efficient, so real-time object detection on even desktop or laptop CPUs.

19:35So the new architecture developments are both geared towards, yeah, that efficiency, but also squeezing every ounce of performance out of parameter efficient models, both in terms of training and flexibility across different use cases. Yeah, I think there's great applications for this on the edge where you're not in one of the giant clouds with essentially, if you're willing to pay for it, infinite compute available to you. whether it be training or for inference, either way. So the fact that this can run on just about anything, I mean, back in the early days, we could do YOLO v2 on smaller equipment, but it didn't run smoothly.

20:12You'd have points where it would overwhelm the computational cycle. And so it's nice seeing something like this has come this far. It's quite an open source library. Yeah, and there's a link that we'll add into the show notes, which includes a notebook for running YOLO v9 in a collab notebook, even like I say, on CPUs. So in terms of the efficiency, one of the things that I saw was YOLO v9 operates with 42 % fewer parameters and 21 % less computational demand than YOLO v7, yet it achieves comparable accuracy. So, you know, it was already fairly accurate, right and kind of an industry standard but now with much fewer parameters and i think that that is definitely a trend that we've been seeing not only in computer vision but in other cases where you see things like olama or other things that llama cpp that are allowing you to run large language models on a variety of hardware including just on your on your local laptop and you know quantization type of libraries like bits and bytes and optimum and big DL and these libraries that allow you to run maybe 7 billion parameter large language models or other generative AI models, but in lower precisions so that you can run them on a variety of hardware, optimize them for a variety of hardware.

21:49We also had Neural Magic on the show a little while back now, who has a set of libraries for optimizing models to run on CPUs. And yeah, so there's a lot of kind of precision and quantization that can happen even on top of the use of these parameter efficient models. One of the interesting things also that I saw this last new cycle, which at least in the circles that I run in with large language models people were talking about a lot, which is this release from Microsoft or a paper from Microsoft that I think is titled something like the era of one bit LLMs, which is interesting because, you know, a lot of people have talked about going from maybe float 32 to float 16 and eight and four bit precision, that sort of thing.

22:41And this kind of brings in this idea of one bit LLMs with this architecture bit net. And so I found it interesting that we got both YOLO v9, but now comes on the LLM side, this one bit architecture. And it seems like a similar thing is happening. I don't know if you remember back when we were talking about our CNN and some of the larger computer vision models, we've seen the progression to more and more parameter efficiency and flexibility across deployment scenarios. And now we're seeing that maybe in a more rapid way with LLMs and this, you know, one bit LLM, but also all the other quantization and that sort of stuff that we've seen on the generative side.

23:29Do you have any sense from an application standpoint, like where you might go with these one bit LLMs? Like what are some of the use cases that come to mind for you? Yeah, I think it's interesting. So this one bit LLM that was released, they talk about it having similar performance to a model of the same parameter size, but more computational efficiency, because of course, these parameters are bits are actually not just zero and one. We can talk about that here in a second, but more computational efficiency. So I think that this is really interesting for cases where you do want to run maybe an LLM on an edge device in a scenario like think about disaster relief and you have a device out in the field that's giving help to first responders or something, giving them information or processing information from training documents or something.

24:25and you're using an LLM to provide answers. It's likely very spotty internet connection in that case. And so having something that could run on device in a variety of scenarios would be quite relevant. Or maybe, so one scenario would be lack of connectivity. I think another scenario would be very latency sensitive scenarios where you want a response very quickly. you don't want to have to rely on network overhead or things going out of a network that you're operating in for security reasons, that sort of thing might be a good use of these. Yep. That sounds interesting. They have a term in here that I'm curious about, referring to BitNet.

25:10They talk about it being a 1.58 bit LLM and Hugging Face in their paper notes that all large language models are 1.58. Do you have any comment about that, what that means. The reality is if you, I think they talk about this in the paper, if you go down to a truly one bit LLM, each weight of your model is either zero or one, right? Then yeah, you would expect to lose a lot of information that might be important. And so they make a slight compromise in here. Maybe it's unfair to call it a compromise. They make an astute conversion from bytes, in other words, zero to one or bits, to what they call ternaries.

25:58So these are basically triplets of or three bits together. So you have, for example, weight could be minus one, zero, one or something like that. So you've got three numbers that represent certain information, and that's where they kind of get this 1.58 bit. Gotcha. So this is also why it's kind of they release this new type of architecture that processes these ternary bits or ternaries, these combinations of three bits. And that's presented in the Microsoft paper. But yeah, I think this is only the kind of latest. I think we'll see, my prediction would be that we'll see many more things like this where people are trying to be parameter and compute efficient with large language models.

26:52We've seen models getting more and more efficient and more compact over time. And as we're looking at so many smaller, very capable models being used out on edge devices, do you envision something like this where they're really targeting efficiency in terms of being able to do that in something like small electronics? Or is that a little bit overly ambitious for where this might take us in a reasonably foreseeable future? Yeah, it's actually a good question because one of the things that we saw also, I don't know if it was this week, but recently at least, was Qualcomm's announcement and release of a huge number of, I forget how many, a whole bunch of models on what they're calling the Qualcomm AI Hub for models that run on device on their Snapdragon processors and other things at the edge on small devices.

27:49So these wouldn't be like the small devices of like a microcontroller or something like that. There's still a good bit of power in these processors. But it is super interesting that Qualcomm has made the effort to make these types of models, whether that be object detection or large language models or other things available in optimized forms to run on very small devices. And I think it's a trend that we'll keep seeing.

28:33It seems somehow like in computer vision, it took maybe, what, five years we've been doing, five or six years we've been doing this podcast. And over that time, we've seen computer vision models shrink down and down and become faster and more parameter efficient. It almost seems like that's happening much faster on the large language model side and generative model size. It's like shrunk from five years to one year where a lot of that's coming out for on-device usage. When we and the rest of the changelog team are looking at what content to bring onto the show, So there are various guests and there are all sorts of topics and advancements going out.

29:17It's become quite challenging to narrow it down to just what we can cover in these shows. And largely that's because of what Daniel was just saying, that tremendous acceleration in the advancement of this technology is very hard to keep up with and report on, especially trying to figure out what folks are most in need of hearing or being pointed to. So on any given week, which of the dozens of things that are happening do you want to do? And I would say for those out there listening, like in this episode, we've talked a lot about parameter efficient models and whether it be the Qualcomm AI models or the one bit LLMs or YOLO and running these on device and at the edge, it might be natural to think, oh, the news cycle is totally switched to local models, running all the models locally, and that'll solve all the problems.

30:10And I think the reality is in the future, it's going to be kind of both and, right? You're not going to serve, let's say that you integrate a model into some social media application or whatever mobile application, or you're serving a web app and it's got some AI integration or something like that. It's very unlikely, I think, that you're going to want to serve up millions and millions of requests using only local models. And in the same way, if you've got an enterprise batch use case, right, and you want to process 1.5 million documents through a large language model, you likely don't want that running on your Mac M2 or something like that.

30:58Like that's not the deployment strategy for that scenario. But yet you will see a lot of models running at the edge or locally. And I think the reality is that we'll go into kind of a both and sort of scenario where, yes, a lot of things you'll be able to run locally. But the same as like, I mean, you can run a lot of software locally, but it doesn't mean that you're also not running software in the cloud. You know, AI is just a new layer in your kind of software stack. So we're going to run it locally. Yeah. And we're going to run it in the cloud. That's exactly right. That was where I was going to go.