906: How Prof. Jason Corso Solved Computer Vision’s Data Problem

0:02Jon Krohn:This is episode number 906 with Dr. Jason Corso, professor at the University of Michigan and co-founder of Voxel51.

0:19Jon Krohn:Welcome back to the Super Data Science Podcast. We've got an exceptional guest for you today. It's Dr. Jason Corso. He's professor of robotics, electrical engineering, and computer science at the prestigious University of Michigan, with over 20 years of research experience spanning video understanding, robotics, and AI. He's published over 150 academic papers in that time that together have been cited over 20 ,000 times. In addition to his academic work, he's also co-founder and chief science officer at Voxel 51, a leading platform for visual AI development. His work bridges academic innovation and obviously real-world impact, earning him major honors that I don't have time to list, but there's a ton of them.

0:59Jon Krohn:Today's episode skews a bit more towards hands-on practitioners like data scientists and ML engineers, particularly anyone tackling computer vision problems. That said, Jason is a charismatic and exceptional communicator, so perhaps any listener to this podcast will enjoy today's episode. In it, Jason details how his research spin-out Voxel51 is solving the biggest bottleneck in computer vision, the surprising way autonomous vehicles learn to handle accidents they've never seen, and why the secret to better AI models isn't better algorithms, it's something else that's hiding in plain sight. All right, let's jump right into our conversation.

1:33Jon Krohn:Jason, welcome to the Super Data Science Podcast. Where are you calling in from today? Great to be here, John. I'm calling in from Western New York in Buffalo. Nice. It is a town that, it's the airport that I frequent most in the world. I actually, I love that airport. It is tiny, efficient. It's kind of just the right size. It has everything you need, but you can get in and out of there very quickly. And so I frequently am going between New York and Toronto, and Buffalo is my airport of choice. Lots of interesting... There's a famous art museum in Buffalo, isn't there? There is. It recently had a renovation.

2:12It's called the AKG now. The AKG, that's right. It used to be called the Albright-Knox Gallery. and it was actually designed by a local architect by the name of E.B. Green. In fact, when I used to live here some 15, 20 years ago, I owned one of his houses as well. And it was such a cool little house on the west side. Buffalo's a great old city for old architecture and stuff like that.

2:35Jon Krohn:For sure. It's something that probably people don't expect if they've never been there, but it's stunning. You can kind of, from the highway, you can kind of look, there's a lot of great vantage points of the architecture in the city as you drive around it. And it's beautiful. Absolutely. Yeah. Good people, good views. And we got the Bills as well. So there you go. Yeah, the Buffalo Bills for sure. That was my football team growing up as well. In Toronto, we'd get American broadcasts over the lake, over Lake Ontario. And so my local football team growing up was the Bills as well. uh so i i lived through the was it four or five years that the bills made the super bowl in the 90s and never won it was four years although it predates my my buccalonian status but um it still is painful for the region i think although these days uh bills kind of the city kind of rallies behind the whole region kind of rallies behind the bills uh but anyway yeah for sure they're a strong club um so yeah so despite living in buffalo you are actually a university of michigan professor.

3:40Jon Krohn:You're a professor of robotics over there, as well as a professor of electrical engineering and computer science. Tell us a bit, Jason or Professor Corso, about the work that you do over there at Michigan. Right on. So please call me Jason. So yeah, so I live this dual life. I also have a family. My family's here. But yeah, I'm a 20-some year veteran of generally computer vision. The angle I take in computer vision is what I call like physically grounded cognitive systems, right? So I'm interested in problems since my dissertation, right? My dissertation was called techniques for vision-based human computer interaction, right?

4:20And so we had cameras watching humans and the humans would do things and then that would, you know, create interaction scenarios. In fact, we built this thing called the 4D touchpad. I think it was at like a workshop at CVPR in maybe like 2003 or something like that. And it used like gesture tracking and in some sense created multi-touch prior to there being like iPads and so on. But that's the type of work that I've done over the years. My research group has focused on areas like video captioning. You know, we have one of the first, if not the first paper at CVPR 2013 on video captioning. We apply that nowadays to guidance of humans doing activities, you know, like I'm cooking a dish in my kitchen or whatever, right?

5:05Like I'm about to reach for the salt, but the recipe is for sugar and my AI can tell me, you know, don't use salt, use sugar. Or more socially relevant, perhaps, is an exciting project we have right now, which is applying the same type of AI agent, agentic guidance type scenarios in rural healthcare scenarios, right? Like a major problem in the U.S. and really worldwide is just a shortage of trained physicians. So our project is trying to enable the upskilling of like RNs or physician assistants or nurse practitioners who can go out into rural America in say a mobile clinic or whatnot and do anything from like a cardiac ultrasound to like deep vein thrombosis in the lower limbs with AI kind of guiding them through every step of the process.

5:56So it's an exciting area. I really enjoy computer vision. I enjoy the boundaries of computer vision and the fact that humans are looking to be or we are trying to build systems that work alongside humans to upscale them or to basically create a better world in some sense. Maybe that's pie in the sky, but that's a real driver, right? We do things by humans, for humans, and of humans, right? So that's been a 20-year driver.

6:26Jon Krohn:Yeah, and it sounds like cutting-edge research. It sounds like it would be very impactful. It doesn't sound pie in the sky either. It sounds like a real tangible way to be making the world a better place with AI, which is perhaps the thing that we love most on this show above all. Absolutely. You mentioned something there, CVPR. For our listeners who don't know what that means, it's the Conference on Computer Vision and Pattern Recognition. And you can correct me if I'm wrong on this, Jason, but I think it's hands down the biggest, most important academic conference on computer vision in the world.

6:58It's definitely one of the two. Yeah. So there have been two historically that are among the top. So CBPR is one of them. The other one is called ICCB, International Conference on Computer Vision. Usually these generally there are two conferences per year. And so CVPR happens every year. ICCB happens every other year. And then the third one is ECCB, European Conference on Computer Vision, and that alternates with ICCB.

7:21Jon Krohn:But generally, those are the two key conferences in computer vision. There are many others that are amazing, especially now as the field has been exploding. In some sense, I wish there were even more conferences that were a little smaller, right? Because, I mean, when I was a grad student, CVPR had something like 500 papers max, maybe 1 ,000 attendees. actually probably even had less than 500 papers, you know, 20 years ago. Nowadays, I think there's like 2 ,500 papers on average every year, 10 ,000 plus attendees. It's great to see the growth, but it's also hard to know where to go, how to focus, like what you're going to learn at the conference and so on.

7:59Jon Krohn:For sure. Interesting to hear about that growth over time. And so with all of this experience, with this 20 years of experience that you have in computer vision, you identified about a decade ago some key problems that could be solved in this space with a technological solution. And you founded, you co-founded a company called Voxel 51. You were CEO of that company for, it looks like about seven years. And then for the past few years, you've been chief science officer. Tell us about Voxel 51, how it got started, how it came out of your university research? Absolutely. Yeah. So, so, you know, I, as a, as an individual, I identify kind of like as a creator.

8:40So I've always had like one hand on the keyboard while I'm lecturing or whatever, or, or, or meeting with someone because I just love to build things. And over time, you know, in, in the, in the research lab, we, we began to notice that data was playing a key role alongside algorithmic or model work. You know, usually when you, when you're an academic and you're writing a paper, generally that paper is going to be about a new model or a new algorithm. And although, you know, there had been some early works in data sets like Caltech 256 or like in an image net and so on, like the number of data papers was significantly dwarfed by the number of model papers.

9:20And it remains true today, right? It's just the general mind, the mindset. However, we began to notice that like as model capabilities began to improve, for a given problem, say object detection, even more concretely, like pedestrian avoidance for autonomous vehicles, just as an example, we began to notice that you can kind of pull a model architecture off the shelf, one of maybe half a dozen or so. And the performance you got out of the system you ended up training was more of a function of the data set you used to train that model than it was which of the six model architectures you chose. uh and you know we we basically began to build this conviction around data is at least as important if not more important than the model architecture you choose and ultimately voxel 51 grew out of that observation or that vision right like this this notion that wait people need data and it's not like we were the first to think this or the only to think this right but like our initial one of our initial mantras was better data better models and we like truly believe in that um but And importantly, as a creator or a builder, like there just was not enough tooling around how one works with data, how one analyzes data.

10:32Right. Like as a grad student, I had this data set where I actually went into the cafeteria early one morning and took some photos of the layout. I was trying to do basically like semantic mapping of the environment. And I think my data set had 100 images, maybe even a little less than that. So like, you know, I could look and study the impact of any algorithmic modification on every single sample of that data set when I when I was doing this, whatever it was, 20 some years ago. Fast forward to even just 10 years after that, like 10 years ago, when ImageNet came around, a million data samples are actually the full ImageNet is 20, 20 some million samples.

11:09Right. Nowadays, you have five billion samples per data set, even in some open source data sets like the Lion 5B you may have heard of or like I know the Florence, the Florence 2 data set or Florence 2 model was trained on a five billion sample data set. It's just impossible. It was it was becoming impossible to like basically put your eyeballs on enough of the data samples to build an intuition over what when you work over with your model, how it's going to be impacted by the data and vice versa and so on. So ultimately, Voxel 51 is a company that tries to speed up the work you do with your data and the work you do with your models by providing the right dev tool, in some sense, for visual AI.

11:49We are an open source tool. We released the open source tool in August of 20. It's been a while ago now that I'm forgetting. It's either 2019 or 2020. Don't quote me on which one of those it is. We have about 3 million installs of that or more. And, you know, we've always tried to have the IC, like the ideal user, ideal customer of that tool is really like heavy technical data scientist or computer vision scientist. And so it's super flexible and you can write plugins for it or extensions for it for the front end and the back end. And yeah, anyway, great. Great. So that's a quick overview of where we were and sort of why we got started.

12:33I'm not sure if you have any questions about that.

12:35Jon Krohn:No, it's a great story on the origin. You identify a pain point through your expertise, and then you're able to create a product to solve that pain point. And you've already had a huge amount of success with 3 million downloads and the GitHub repo. So we'll talk about, I realize the open source is a little bit different, but it also gives some perspective on how important solutions like this are. when the GitHub repo has 10 ,000 stars. Absolutely. Yep. And have you ever heard of, so there's, we actually recently in an episode, in episode 901, which came out a few weeks ago, we had someone on the show named Lilith Batlia, who she runs workshops at ICML and iClear, two other big machine learning conferences, which obviously you know, Jason, but just for our audience.

13:28Jon Krohn:um and uh so she ran working groups or she runs working groups at those uh on data-centric machine learning uh so it's a dmlr data-centric machine learning research is kind of like the acronym that's used there have you come across that that acronym before it sounds similar to what you're describing uh absolutely i mean data-centric ml can mean a lot of things uh but even at voxel we used to use that in our outbound community-driven marketing material as well. I mean, I say it can mean a lot of things, not to put it down, but it is like it's critical to recognize that when we think of building technological software systems, we think of writing code.

14:15And then we hope that we can debug a software system like the tool we're using right now to record this, right? But when you think of machine learning, there's code and then there's the data that goes into the code in some sense, like it gets kind of transformed into data weights or coefficients or something like that. But these two things are inseparable. And as the evolution from what some folks have called software 1.0, just code, the software 2.0, which is essentially just a different type of code. It's just humans can't really write it. We write other code to train it from data. um you know so like data data has been used to train up machine learning systems for decades now and but i think there's just an when we when we collectively as a as a research community or a user community think of data-centric machine learning i think what what i was saying earlier where we tend to emphasize that oh wait it's not just that there's data in this code and the code is more important it's that you really cannot separate or divorce the two things right like the data and the structure of the model structure, even like the ops underneath it, these are, these are wed together in a way that is critical to understand all facets.

15:27If you really want to build a successful ML system or AI system, you really need the right tooling around analyzing the data, analyzing the models, analyzing the ops. And they, they, they do need to work in concert so that you have a good sense of what's going on.

15:42Jon Krohn:It seems really obvious when we say it out loud like this, but it is amazing how much attention in terms of new releases, go on to some exciting new model. But in something like this, in computer vision, where you think about, it's probably easy for users to imagine something like the machine vision problem of an autonomous vehicle, where you have sensors on a vehicle that are driving through streets and you wanted to make safe driving decisions, obviously as close to 100 % of the time as possible. In that kind of scenario, it makes it kind of easy to imagine how the model, if it has some percentage improvement over some other model, that's great, but the model isn't going to be valuable at all if you don't have data covering the whole gamut of situations that that autonomous vehicle is going to run into.

16:34Jon Krohn:If you only train the vehicle on situations where there's no car accidents, there's no possible world where that AI system could know how to handle seeing an accident happen right in front of it. And so hopefully, that's a very simplistic example that I just gave, but hopefully that it kind of allows us to visualize pretty easily the critical importance of data in having any AI model work effectively in the real world, which computer vision systems, I think, basically always are operating in the real world. Yep. So I mean, I think you hit it on the head, really right like like and i and i think it's a great example especially because um you know i mean i have an 18 year old recently taught her how to drive and so on i'm wondering like how many miles does she have to drive before she's going to see even like a near miss or even just a situation where like a kid runs out into the street right um so i think the number is something like um it's in the tens of millions of miles driven for every accident that's recorded by like the US government or NHTSA.

17:44And so like actually finding the hardest part about this world of building highly successful, like 99.999, whatever percent accurate systems is getting the data, then getting the data labeled and then training the model and figuring out what are the failure modes? What are the success cases? What are my failure modes? And where do I need to add more data and begin this process, right? So actually something I'm really excited about at Voxel 51 right now is this new direction we've taken our product. I think annotation companies, the problem of taking raw data and labeling, put boxes or labels or classes or whatever on your data samples to train the machine learning algorithm on it.

18:28Those are probably the first wave of companies in computer vision, at least in modern computer vision, machine learning based computer vision. um but voxel 51 never identified as an annotation company we're always in some sense we explicitly decided strategically we are we are not an annotation company we actually don't even support you can do a lot of things in 51 including load varieties of different labels into the software tool and visualize them and so on but until this coming summer you will not have been able to edit them in our tool like we were so against it almost like we had almost like a one-button mouse challenge that Apple had over the years, right?

19:05And, you know, nowadays, like that annotation problem is so, has been so central, but I think it's transforming based on these two, these decades of progress. And so with performant foundation models, now we have this tool, this new product line called verified auto labeling, which can take this raw media, automatically generate labels on it via foundation models. And obviously, there's going to be a spectrum of performance, right? For certain classes that the foundation models have seen a lot, like pedestrians or bicycles or other vehicles, it's going to work pretty well. For other scenarios like teddy bears or certain types of hats or coffee mugs, whatever, maybe it'll work less.

19:53But the critical aspect And what's, I think, really exciting from our perspective is the V in the verification part of that, right? Well, our workflow is, you take your raw media, you apply foundation models, and we have like a battery of them you can apply against it. And then we have our custom ML that will rank the outputs from those foundation models so that you can, in batch, have high confidence that you're like automatically going to accept something like 70 % of them. And already, that's a huge amount of money that you're saving in time and so on. And then for the remaining 30%, we can still rank them again.

20:32So you can have your human labels or human QA people only spending time on the challenging scenarios, right? The scenarios like the ones you're pointing out, like the corner cases, the hard cases that we really need humans to look at and humans to verify and so on.

20:46Jon Krohn:Very cool. Cool. So this verified auto labeling, this new initiative, you said summer, so we're talking Northern Hemisphere summer for our international listeners. That's kind of like around the time that this episode is coming out is when this verified auto labeling starts to be something that people can use in Voxel 51? Yeah, actually, it recently went into alpha with some customers. So it's already in alpha, and we are improving it. And it will be in every release over the coming two months, probably, it will get more functionality and more users to adopt it. Fantastic. And it sounds like this is then solving what is the biggest bottleneck in computer vision.

21:24Jon Krohn:And you're doing that using intelligent techniques so that it makes it way more time efficient and cost effective orders of magnitude relative to having humans be annotating the data. Absolutely, yeah. I mean, the tagline that I, not approved by marketing, but that I like these days is like, curation is the new annotation right like i mean annotation 1.0 if we want to use that analogy was basically i don't really know how to filter my data so i'm just going to send it all to humans to label and i'm gonna have to pay for all that and it's time consuming as well and then i'm going to get it back and give it to my machine learning engineers and maybe one one percent maybe ten percent of that's useful you know it's hard to say you don't really know it's kind of like walking in a dark hallway without any light switch on, right?

22:14Which door are you going to try? I think nowadays we're probably in like the one, you know, annotation 1.5 era where it's obvious to apply a foundation model, even for something like pre-filtering, just so you can rank your data. So you're going to send it to humans to label. And I think the like verified auto-labeling is farther along the line in saying, wait a second, sure, pre-filter your data, only apply certain things. but we're also saying, wait, you also, you still don't have to have humans annotate everything, right? You can like, you can rank them, filter them and so on. So you can just accept the automatic labels out of the box.

22:51And where do I think we're actually going? Like what is actually annotation 2.0? And that this is not what we're releasing this summer, but it's likely coming in the future, right? If I have my way, you know, is this notion that instead of the humans asking the foundation models, what they should label or what the labels are or, or what have you, It's more agentic where there's a problem statement given, behemoth amount of unlabeled data. And then the models are able to actually ask the humans questions just when it's necessary. And it's more driven by the AI agent, if you will. So even fewer, less human involvement is needed.

23:30Jon Krohn:That is awesome, Jason. So it sounds like Voxel 51 has figured out how to leverage the latest technology in terms of what we can do with automation. to allow people to get the highest quality data for building high performance computer vision models at a fraction of the effort and the cost. That's cool. Absolutely. Nail on the head right there. Yep. Nice. So yeah, fantastic. I mean, this is a Friday episode, so it's not amongst our longer episodes. And so, you know, we that was a short but very rich episode with you, Professor Corso, Jason. We could certainly I mean, we could easily have had an episode that was like, you know, a Joe Rogan style three or four hours with you, I'm sure, on computer vision if we had to, because you have such rich, such a rich understanding of the space.

24:28Jon Krohn:Before I let my guests go, I always ask them for a book recommendation. What do you have for us? Cool. So I am an avid reader. And I was just saying, well, if you want to do a thing in my head, if you want to do a three or four hour episode one time, maybe when we're both stuck in the Buffalo airport and it's no storm, we can record that. But I have a reader here. I guess one of the best books I've read in the last few months is a book called Quit by Annie Duke. and it really puts the, you know, like I'm a hard worker, I'm a grinder, right? So like I'm always one to like really just want to see a project through and it really puts that type of grit, which every founder needs in some sense and every professor really needs these days as well, up against this notion that like make sure you're being smart about how you spend your time and how you're planning, pre-planning when you might want to deep switch or quit an angle and go in a different angle.

25:25I think for really any adult, I think the lessons learned that she writes in this book are fantastic.

25:36Jon Krohn:I like that a lot. Yeah, it's tricky for those of us, probably a lot of our listeners. If the way that you choose to spend your free time is listening to a technical podcast about data science and AI, you're probably somebody who has a lot of grit and is really pushing their career. And yeah, this kind of thing, it becomes important, especially, you know, as opportunities accumulate, as you as you focus and have more grit, more and more opportunities come up and you can't keep doing everything. You know, it's a it's a tricky thing. You know, even just things like I used to be pre pandemic, I used to be able to be inbox zero and respond to anything that should be responded to.

26:16Jon Krohn:And, you know, this thing's that's a simple, silly example. I mean, it sounds like this quick thing is kind of more about big strategic decisions, but just figuring out what you have to let go so that you can make space for even bigger things. Absolutely. I mean, there's no in terms of like the strategy and the tactics of decision making. There's no thing too small to think about, frankly. So like the notion of email inbox or inbox zero or whatever is highly relevant. I think the way I would put it is one of one of our investors uses has used the term indigestion. Right. Like if you're so successful, you're going to get indigestion over just trying to do too much.