Roboflow Simplifies Computer Vision for Developers and the Enterprise - Ep. 248

0:10Hello, and welcome to the NVIDIA AI Podcast. I'm your host, Noah Kravitz. 90 % of the information transmitted to human brains is visual. So while advances related to large language models and other language processing technology have pushed the frontier of AI forward in a hurry over the past few years, visual information is integral for AI to act with the physical world, which is where computer vision comes in. RoboFlow empowers developers of all skill sets and experience levels to build their own computer vision applications. The company's platform addresses the universal pain points developers face when building CV models, data management to deployment.

0:47RoboFlow is currently used by over 16 ,000 organizations and half the Fortune 100, totaling over 1 million developers. And they're a member of NVIDIA's Inception program for startups. RoboFlow co-founder and CEO Joseph Nelson is with us today to talk about his company's mission to transform industries by democratizing computer vision. So let's jump into it. Joseph, welcome, and thank you for joining the NVIDIA AI podcast. Thanks so much for having me. I'm excited to talk CV. There's nothing against language, love language, but there's been a lot of language stuff lately, which is great, but I'm excited to hear about RoboFlow.

1:25So let's jump into it. Maybe you can start by talking a little bit more about your mission and what democratizing computer vision means and making the world programmable. At the highest level, as you just described, the vast majority of information that humans process happens to be visual information. In fact, I mean, humans, we had our sense of sight before we even created language. It's how we understand the world. It's how we understand the things around us. It's how we engage with the world. And because of that, I think there's this massive untapped potential to have technology and systems have visual understanding to a similar way that humans do all across, really, we say the universe.

2:04So when we say our North Star is to make the world programmable, what we really mean is that like any scene and anything will have software that understands it. And when you have software that understands something, you can improve that system. You can make it be more efficient. You can make it be more entertaining. You can make it be more engaging. I mean, at Roboflow, that's like, we've seen folks build things from understanding cell populations under a microscope, all the way to discovering new galaxies through a telescope. And everything in between is where vision and video and understanding comes into play.

2:32So if this AI revolution is to reach its full potential, it needs to make contact with the real world. And it turns out the real world is one that is very visually rich and needs to be understood. So we build the tools, the platform and the community to accelerate that transition. So maybe you can tell us a little bit about the platform and kind of within that, how your mission in Northstar kind of shaped the way you develop products and build out user experiences. And I should mention a great shout out from Jensen in the CES keynote earlier this year for you guys. And you raised a series B late last year.

3:06So I want to congratulate you on that as well before I forget. I appreciate it. Yeah, it's a good start. I mean, as you said, a million developers, but there's 70 million developers out there. there's billions that will benefit from having visual understanding. I mean, in fact, in that Jensen shout out, just like maybe the sentence or two before he described some of the visual partners that are fortunate to work with the NVIDIA team, he described that the way NVIDIA sees it is that, you know, global GDP is$100 trillion. And he describes that visual understanding is like$50 trillion of that opportunity.

3:35So basically half of all global GDP is predicated on these operationally intensive, visual centric autonomy based sorts of use cases. And so the level of impact that visual understanding will have in the world is just a fraction of what it will look like as we progress. Now, in terms of like how we think about doing that, so it's really about empowering the builders and giving the certainty and capability to the enterprise. So, for example, anyone that's building a system for visual understanding often needs to have some form of visual input, camera, video, something like this. You need to have a model because the model going to help you act, understand, and react to whatever maybe actual insight you want to understand.

4:15And then you want to run that model somewhere. You want to deploy it. And commonly, you even want to chain together models or have a system that triggers some alerts or some results based on information that it's understanding. So Roboflow provides the building blocks, the platform, and the solutions so that over a million developers and half the Fortune 100 have what they need to deploy these tools to production. And you're doing it, you know, kind of mentioned in the intro, you're trying to make the platform available for folks who are, you know, deep into this, have been doing CV and working with machine learning for a while.

4:48And then also folks who might be new to this, they can get up and running and work with CV, build that into their toolkit. Yeah, the emphasis has always been kind of on someone that wants to be a builder. That definition is expanding with the capabilities of code generation, prompting to apps. we've always kind of been bent on this idea that like those that like want to create use and distribute software what's funny is that like when we very first launched some of our first products ml teams initially were kind of like oh i don't know this seems pretty pedestrian i know exactly what to do and fast forward now and it's like whoa a platform that's fully featured that has immediate access to the latest models to use on my data in the contexts of where i couldn't even anticipate so it's been kind of you know as the as the platform's become more feature rich we've being able to certainly enable a broader swath of both maybe domain experts of a capability.

5:37But I think broadly speaking, the places that we see the rapid, most impactful adoption, in some ways, is actually bringing vision to others that otherwise may not have had it. Like what used to be maybe like a multi-quarter PhD thesis level investment, now can be something that a team spins up in an afternoon. And that really has this kind of demand begets demand paradigm. I mean, for example, like one of our customers, they produce electric vehicles. And when you produce an EV, you know, inside their general assembly facility, there's all sorts of things that you need to make sure you do correctly as you produce that vehicle.

6:09From the worker safety who are doing the work to the machines that are outputting, say, when you do what's called stamping, where you take a piece of steel or aluminum and you press it into the shape of the outline of the vehicle. And you get up potential tears or fissures or when you like actually assemble the batteries out of the correct number of screws. Basically, every part of building a car is about visually validating that the thing has been built correctly so that when a customer drives it, they can do so without any cause for pause. And just a little bit of computer vision goes a really long way in enabling that company and many others to very quickly accelerate their goals.

6:44In fact, this company had the goal of producing 1 ,000 vehicles three years ago, and they barely did it. They did 1 ,012 in that year. And then they scaled up to 25 ,000 and now 50 ,000. And a lot of that is on the backs of having things that they know they're building correctly. And so we're really fortunate to be a part of enabling things like this. So it's kind of like you can think about it of a system that doesn't have a sense of visual understanding. Adding even just a little bit of visual context totally rewrites the way by which you manufacture a car. And that same revolution is going to take place for lots of operationally intensive processes, but any kind of place that you interact with the world each day.

7:18So kind of along those lines, what, to use the phrase untapped opportunities, you see out there? What's, you know, the low-hanging fruit or maybe the high-hanging fruit, but you're just excited about when it comes to, you know, developing and deploying computer vision applications? And, you know, we've been talking about it, but obviously talk about RoboFlow's work, not just supporting, but helping developers kind of unlock, helping builders, you know, unlock what's next. The amazing thing is actually the expanse of the creativity of developers and engineers. It's like if you give someone a new capability, you almost can't anticipate all the ways by which they'll bring that capability to bear.

7:53So, for example, I mean, we have folks that hobbyists that'll make things like the number of folks that make things that like measure the size of fish, because I think they're trying to prove to their friends that they caught the biggest fish. And then you separately have like government agencies that have wanted to validate the size of salmon during migration patterns. And so this primitive of like understanding size of fish both has what seems to be fun and very serious implications. Or folks that, I don't know, like a friend of mine recently was like, hey, I wonder how many cars out in San Francisco are actually Waymos versus like other sorts of cars.

8:22And what does that look like? What does that track over time? And so they had a pretty simple like Raspberry Pi camera parked it on the windowsill. And in an afternoon, now they have a thing that's counting, tracking and keeping tabulation on how many self-driving cars are making their way on the road, at least sampled in front of their house each day. Right. No, I don't want to call anybody out, but that's not the same person who had the video of all the Waymos in the parking lot in the middle of the night in San Francisco going in circles. No, OK. It wasn't that guy. Yeah. But I mean, like the use cases are expansive because I don't know, like the way we got into this, right, is like we were making AR apps, actually.

8:57And computer vision was critical to the augmented reality, understanding the system around us. right and we've since had folks that make you know board game understanding type technology dnd dice counters telling you the best first move you should play in katan and so basically like you have this like creative population of folks or this one guy during the pandemic you know he's really bored locked inside and he uh he thought maybe his cat needed to get some more exercise and he created this system that like attached a laser pointer to a robotic arm and with a little bit of vision he made it so the robotic arm consistently points the laser pointer 10 feet away from the cat that's like jumping around the living room and makes this whole youtube tutorial But then like the thing that's really interesting, right, is that like, you know, a technology has arrived when like a hacker can just like build something in a single setting or maybe like in a weekend.

9:40You know that the what used to be this far difficult to access capability is now broadly accessible. And that feels like a lot of like the similar sort of enterprise use cases. Like we kind of have a joke at Rebel Flow that like one person's hobbyist projects is another person's entire business. So the low hanging fruit, frankly, is like everywhere around us. Like any sort of visual feed is untapped, the sort of images that someone might be collecting and gathering. I mean, the similar things that give rise to machine learning certainly apply to vision, where the amount of visual inputs doubling year on year and petabytes of visual information to be extracted.

10:14So it's kind of like if you think about it, you can do it. That makes me think of an episode we did recently with a surgeon who founded a surgical data collective. and they were using just these, I say stacks, they weren't actually videotapes, I'm sure, but all of this unwatched, unused footage from surgeries to train a model, to help train surgeons how to do their jobs better. But it made me want to ask, are the inputs that could, the visual inputs that can go into RoboFlow, it doesn't have to be a live camera stream. You can also use, you know, archive footage from images. That's correct. Yep.

10:47Yep. So someone may have like a backlog of a bunch of videos. I mean, for example, we actually had a professional baseball team where they had a history of a bunch of their videos of pitching sessions. And they wanted to run through and do a key point model to identify various poses and time of release of the pitch. And how does that impact someone's performance over time? And so they had all these videos from the past that they wanted to run through. And then pretty soon they started to do this for like minor leagues where you might not have scouts omnipresent and you certainly don't have broadcasts.

11:16Right, right, right. You just have like this like kind of low quality footage on like cameras from various places and being able to produce like sports analytics out of this information that's just locked up otherwise. And this unstructured visual capture is now available for, in this case, building a better baseball team. Yeah, that's amazing. Building community, you know, is something that is both vital to a lot of companies, a lot of tech companies and developer platforms and such. But it also can be a really hard thing to do, especially to build, you know, an organic, genuine, robust community.

11:46serving enterprise clients, let alone across this seemingly endless sort of swath of industries and use cases and such, you know, also pretty resource intensive. So how do you balance both? How is RoboFlow approaching, you know, building that community that you're talking about just now with serving, you know, these, I'm sure, demanding in a good way, but, you know, demanding enterprise clients? I think the two actually go hand in hand more than many would anticipate. When you build a community and you build a large set of people that are interested in creating and using a given platform, you actually give a company leverage.

12:24Basically, like the number of people that are building, creating and sharing examples of Roboflow from a very early day made us seem probably much bigger than maybe we were or are. And so that gives a lot of trust to enterprises. Like, you know, you want to use something that has gone through its paces, been battle tested, something that might be like an industry standard. And you don't become an industry standard by only limiting your technology to a very small swath of people. You enable anyone to kind of build, learn the paradigm and create. Now, you're right that both take a different type of thoughtfulness to being able to execute on.

12:59So in the context of like community building and making products for developers, a lot of that I think stems from, you know, as an engineer, there's products that I like using in the ways that I like to use those products. And I want to enable others to be able to have a similar experience of the products that we make. So it's things like providing value before asking for value, having a very generous free chair, having the ability to highlight the top use cases. I mean, we have a whole like research plan where if someone's doing stuff on a.edu domain, then they have increased access to GPUs. RoboFlo has actually given away over a million dollars of compute and GPU usage for open source computer vision projects.

13:33And, you know, we actually have this, it's kind of a funny stat, but 2.1 research papers are published every day citing RoboFlo. And those are things like people that are doing all sorts of things. That's a super cool stat, I think. Yeah. I mean, it just gives you the context to like that is someone's maybe six or 12 month thesis that they've spent trying to inject and to be able to empower folks to realize what's possible. And it's really kind of the fulfillment of like our mission at its core. Like the impact of visual understanding is bigger than any one company. And anything that we can do to allow the world to see, expose and deploy that is important.

14:09Now, on the enterprise side, what we were really talking about is building a successful kind of go-to-market motion and making money to invest further in our mission. And enterprises, as you alluded, are very resource intensive in terms of being able to service those needs successfully. Even there, though, you actually get leverage by seeing the sorts of problems, seeing the sorts of fundamental building blocks, and then productizing those building blocks. You know, there have been companies that have come before Roboflow who have done a great job of be very hands on with enterprise customers and productizing those capabilities.

14:39Companies like Pivotal or Palantir or these large companies that have gone from, hey, let's kind of do like a bespoke way of making something possible and deploying it more broadly. Now, we're not fully, you know, like for like with those businesses. I more give that as an example to show as someone that is building tooling and capabilities, worst case is you're giving the enterprise substantially more leverage. And certainly best case is there's actually a symbiotic relationship between enterprises being able to discover how to use the technology, be able to find guides from the community, be able to find models they want to start from.

15:11I mean, RoboFull Universe, which is the open source collection of data sets and models, is the largest collection of computer vision projects on the web. There's about 500 million user labeled and shared images and over 200 ,000 pre-trained models. And that's used for the community just as much as enterprise, right? Like when you say enterprise, like enterprise is people. And so there's people inside those companies that are creating and building some of those capabilities. Now, operationalizing and ensuring that we deliver the service quality, it's just the types of teams you build and the way that you prioritize companies to be successful.

15:39But we're really fortunate that, you know, we're not writing the playbook here. There's been a lot of companies that, you know, Mongo or Elastic or Twilio or lots of post IPO businesses that have shown the pathway to both building really high quality products that developers and builders love to use and ensuring that they're enterprise ready and meeting the needs of high scale, high complexity, high value use cases. So you use the word complexities. And, you know, one of the things that I hear all the time from, and I'm sure you more than me, from people who are trying to build anything is sort of how do you balance, you know, creativity and coming up with ways to solve problems.

16:17And particularly if you get into kind of a unique situation and you need to find a creative answer with things, with not letting things get too complex. And, you know, in something like computer vision, I'm sure the technical complexities can spin up in a hurry. What's been your approach and what successes, how have you found success in balancing that? Complexity for a product like RoboFlow is always a balance. You want to offer users the capability and the advanced settings and the ability to make things their own, while also part of the core value proposition is simplifying. And so you often think, oh man, those two things must be at odds.

16:53How do you simplify something, but also serve complexity? And in fact, they're not, especially for products like Roboflow, where it is for builders, you make it very easy to extend. You make it very interoperable, meaning there's open APIs and open SDKs, where if there's a certain part of the tool chain that you want to integrate with, or there's a certain specific enterprise system where you want to read or write data to, that's all supported on day one. And so if you try to kind of boil the ocean of being everything in the platform all at once on day one, then you can find yourself in a spot where you may not be able to service the needs of your customers well.

17:28In fact, it's a bit more step by step. And that's where, you know, the devil's in the details of execution of which steps you pick first, which sort of problems you best nail for your customers. But philosophically, it's really important to us that, for example, when someone is building a workflow, which is, you know, the combination of a model and some inputs and outputs, you might ingest maybe like an RTSP stream from like a live feed of a camera. Then you might have like a first model that's, let's say the problem that we're solving is we're an inventory company and we're concerned about worker safety.

17:57You might have a first model that's just like constantly watching all frames to see if there's a presence of a person. A very lightweight model kind of running the edge. And then maybe a second model of when there's a person, you ask a large vision language model, large VLM, hey, is there any risks here? Is there anything to consider? Should we like look more closely at this? And then after the VLM, you might have another specific model that's going to do validation of the type of danger that is interesting or maybe the specific area within your store. Maybe you're going to connect to another database that exists.

18:27And then like based on that, you're going to write some results somewhere. And maybe you're going to write that result to have insights of how frequently there was a cause for concern within the process that you're monitoring, just as much as maybe you're flagging an alert. And you might be sending a text or an email or writing to an enterprise system like SAP to keep track. And at each step of that juncture, any one of those nodes, since it's built for us on an open source platform, which we call inference, you can actually mix and match, write your own custom Python, write an API one way or another.

18:58And so let's imagine like a future where someone wanted the ability to write to a system that we didn't support yet, like first party. You're actually not out of luck. As long as that system accepts a post request, you're fine. And so you have the ability to extend the system. Yeah, yeah. And so it's this sort of paradigm of like interoperability and making it easy to use alongside other tools. And it gets back to your point around servicing builders just as much as the enterprise. I actually think those things are really closely interlinked because you provide the flexibility and choice and ability to make something mine and build a competency inside the company of what it is I wanted to create and deploy.

19:34Right. The way you frame that makes a lot of sense and makes that link very clear. We're speaking with Joseph Nelson. Joseph is the co-founder and CEO of RoboFlow. and as he's been detailing, Roboflow provides a platform for builders to use computer vision in what they're building. Joseph, you know, I, in the intro sort of alluded to all the advances and buzz around large language models and that kind of thing over the past couple of years. And I meant to ask, Roboflow was founded in 2020? Roboflow Inc. was incorporated in 2020. That's right. Got it. And so anyway, kind of fast forwarding to, you know, more recently, the past, I don't know, six months, year, whatever it's been, a lot of buzz around agents, the idea of agentic AI.

20:17And then, you know, there was buzz, I guess, the word multimodal was being flung around kind of more frequently, at least in circles I run in for a while. And then it sort of dropped off just as people, you know, there were the consumer models that the Claude's and ChatGPTs and Gemini's and what have you of the world just started incorporating visual capabilities, both to, you know, ingest and understand and then to create visual output, voice models, you know, now getting into short video clips, all that kind of stuff. What's your take on the role of multimodal AI integration when it comes to advancing CV?

20:52You know, how is RoboFlow kind of positioned to support this? So multimodality allows an AI system to have even more context than from a single modality, right? So if one of our customers is monitoring an industrial process, and let's say they're looking for potentially a leak, maybe in an oil and gas facility, that leak can manifest itself as, yes, you see a product that's dripping out and you didn't expect it to. It also could manifest itself as you heard a noise, or maybe there's something about the time dimension of the video that you're watching as another modality beyond just the individual images.

21:28And those additional dimensions of data allow the system that you're building to have more intelligence. And so that's why you see kind of like all these modalities crashing together. And what it does is it enables our customers to have even more context. The way we've thought about that is we've actually been built on and using multimodality as early as 2021. So in 2021, there was a model that came out from OpenAI called CLIP, Contrastive Language Image per Training, which introduced this idea of training on 400 million image text pairs. Can we just associate some words of text with some images?

22:03What this really unlocked for our customers was the ability to do semantic search. Like I could just describe a concept and then I can get back the images from a video frame or from a given image. that'd be interesting for me for the purposes of building out my model. Increasingly, we've been excited by increases of models that have more multimodal capabilities on day one. That comes with its own form of challenges, though. The data preparation, the evaluation systems, the incorporation of those systems into the other parts of the pipeline that you're building. And so where there's opportunity to have even more intelligence, there's also challenge to incorporating that intelligence, adapting it to your context, passing it to other sorts of systems.

22:42And so RoboFlow, in being deep believers in multimodal capabilities very early on, have continued to make it so that users can capture, use, and process other modalities of data. So for example, we support the ability for folks to use vision language models, BLMs, in the context of problems they're working, which is typically like an image text pair. So if you're using, you know, QuenVL 2.5, which came out last week, or Florence 2 from Microsoft, which came out maybe about six months ago, or Polygema 2 from Google, these are all multimodal models that have very rich text understandings and have visual understandings, which makes them very good at, for example, document understanding.