Living Optics CEO Robin Wang on Democratizing Hyperspectral Imaging - Ep. 219

0:10Hello, and welcome to the NVIDIA AI podcast. I'm your host, Noah Kravitz. We're coming to you from GTC24 at the San Jose Convention Center, and we're here to talk about hyperspectral data. What does hyperspectral mean? What is hyperspectral imaging, and why should it be more accessible to more people? Well, I've got just the person to answer these questions and more. Here to help us understand all things hyperspectral is Robin Wang. Robin is co-founder and CEO at Living Optics, whose recently released spectral imaging camera is bringing advanced imaging to the mass market, including a host of new applications across multiple industries.

0:48Robin's here, so let's get right into it. Thank you so much for taking time out of your GCC week to join the podcast. Welcome. Thank you, Nor. Great to be here. How's your trip been so far? First things first. It has been utter carnage, but hopefully in the best way possible. Everyone's coming out of the woodwork, and we're still here, and it's still alive. I kind of want to end the recording right now, because what more is there to say? But I'm going to ask you about living optics and about what hyperspectral imaging is. So let's get into that. Right. So living optics, hyperspectral imaging. Where to start?

1:19So normal human eyes, normal cameras, your iPhone camera, your webcam, they see in three colors. That is a bit limiting. Right. Hyperspectral camera, like ours, ours one specifically sees in 96 colors, which means you can see tiny little hidden details that are floating around in light, but not detected by us with normal cameras. So things like blood oxygenation, plant health, plant stress, these things are completely visible to us. We can just see them. This is somewhat of a trivial question, given the nature of the whole conversation. But when you say my eyes can see in three colors, your camera can see in 96.

1:53The other 93, are these actually like novel colors, so to speak? It's not just blends of red, green, and blue? It's kind of hard to visualize because it's literally invisible. So out there in the world, like... But let's do it on the radio, you know? Yes, of course. So I think a good way to think of how three colors is limiting, but also the blend of three colors is think of a computer screen or a camera. They only have red, green, and blue receptors or LEDs. And every color you see out there, all the kind of salmon, mauve, cyan, they're all blends of these three colors. But what we're missing is the finer details between these colors.

2:35So a good example is blood oxygenation. This in theory is in, or it's not in theory, is in the visible range. But the spectral features or the color features are so fine that human eyes, no matter how hard you try, cannot detect them. If you train a computer vision model with a trillion images, you still can't see them. So these are the fine little features that you need hyperspectral imaging to see. Okay. That you just can't do that. But unfortunately, although it will make the world a better place, can't see what's a normal red, green, and blue camera. Right. Got it. I like to pull things off of people's websites and then ask them about them and make them speak to them.

3:12Your website says, at Living Optics, we are dedicated to a future where hyperspectral imaging is not just a tool for the few, but an asset for the many. I remember telling the marketing people to write something similar to that. Yeah. Well done. That was all good. Good marketing people. Yeah. So not obviously knowing too much about spectral imaging myself. when I was poking around, I was kind of surprised to see how many different use cases there are already across so many different kinds of industries. My question is, how are you going about making that technology more accessible to more people?

3:41But maybe before that, some of these use cases, where is hyperspectral imaging being used and kind of why is it so important? Right. So there's a few questions there. Where's it being used? I warned you ahead of time. Before we started, I told Robert, I claimed to ask one question, and it's really got 12 parts and I apologize. No worries. So where's it being used? Why are we different? And what's been, yeah, anyway, those are the two, I'll answer first because I forgot the third one, sorry. So where's it being used? So hyperspectral has been around for, say, 40, 50 years. So think about Newton's prism, the Pink Floyd, wait.

4:16Yeah, those guys, yeah, it's been around. So it's been around, starting with military and satellite imaging. The first hyperspectral imager was actually built in eastern Germany. It cost 200 million bucks. And they had to make a special machine tool to make it. So it started off seeing as a defense tool, seeing camouflage, seeing movement. You can see further underwater, for example. I should stop there before I get shot. And then slowly it moved into mineral prospecting. Where do I dig up things? So a lot of ores you can't see with a normal human eye or a normal camera. but you can see the chemical signatures inside with hyperspectral imaging.

4:59They're like, ooh, let's go dig up some good stuff here. And slowly over the years, this technology became a bit cheaper. And by a bit cheaper, I mean it went from a million bucks to about$100 ,000, which meant people, big national labs, started using it. So that's where I came from, like a big national lab background. So people started using it for plant health, like medical imaging. but the problem was these hyperspectral images were still 100 grand wasn't a million bucks anymore and they were slow and they were really hard to use they were big chunky bricks with no prettiness or industrial design and they scan things line by line so like we said like sorry like i said there's 96 channels or colors in this image that we take so the hyperspectral the 100 grand guys were taking these colors slice by slice it was like building up a ream of paper one sheet of paper at a time so to get one hyperspectral image that was a few minutes sure and then if you're moving your target moving then yeah get lost right yeah you're done yeah there's nothing there so what we've done here is like we've cut that 100 grand price down we've taken zero off and we've also my finance direction is not gonna like that but anyway and then we've also made a video rate so rather than waiting two minutes for one image we can go 60 frames a second wow okay use it like a normal camera yeah there's just better because you see more colors you see better detail you can do more so there's so many advantages now that we're in this age of computer vision is that one of the biggest problems is gathering your joint data set right say if you want to see something really niche or specific in your field say one of our customers wanted to do see the difference between different apple varieties, no matter how many images they gathered, they couldn't see difference between a Rayburn or a Golden Delicious.

6:57Turns out you only need 10 hyperspectral images of each apple variety to tell the difference to basically 90-something percent accuracy. The other weird thing is, I shouldn't say this out loud since Jensen said how big are bigger in service, is that we trained that model in about 0.3 seconds. We'll keep that between us, but that's fantastic. So I think I know the answer to this, but I'm going to ask anyway. When you talk about taking a hyperspectral image and then seeing, what is that actually like for the end user? Is it data represented in a chart, on a graph? Is it a long text string? Or are we somehow magically making me able to see the other 93 colors I couldn't see previously?

7:38Oh, this is a really spicy good topic is that you can't visualize this because it's literally invisible to the human eye. We realized that partially through the computing, it's been around a few years, we can still call it as a startup, but anyway, is that you needed to give people the tools to work with hyperspectral data. If you just try to dump hyperspectral on someone, they're going to look at you and say, what the hell do I do with this, Robin? You're hurting me. So we came up with the SDK. So that's completely open source. I think one of the goals of this company is that we don't want hyperspectral imaging to be behind closed doors, big national labs.

8:15is that we want you to access it. We want this to be everywhere. So a few ways to massage this 96-channel data. So think about it as a book. But humanizing can only see in three colors. So one way is to analyze this data to, say, bring out one of our applications is chlorophyll content. You calculate an overlay and you paint it on top of your image. It's like, hey, here's where your plant is super healthy. Here's where a plant disease is. You cut that off. otherwise you'll lose your whole harvest right or spray here because there's fungus here yeah so you overlay that on top it's a big red it's like think about a highlighter yeah we just highlight the scene and then that's the part of the user experience we really want to make easy is that you don't need a phd in spectroscopy to use our camera right so i'll answer your question really roundabout way i hope you got what you need no it's a good it was a good answer i like it um And the highlighter analogy, you're painting on top of the image, right?

9:15A 2D image with very valuable additional data sort of annotated on top. Exactly. You've distilled 96 colors or channels into a highlight or a Sharpie, whatever you like. Yeah. So you mentioned a few, taking care of plant health and that kind of thing, chlorophyll. What are some of the other uses, both sort of industrial and maybe some of the outliers, some of the weird, wacky uses of hyperspectral that you've seen. This is the weird part. I can't tell you too much, and you'll see why. It's that the military found us before we spun out the company. They had their own mega-spectral camera, and they saw you when you were invisible?

9:56I think they saw us as something that's an upgrade. It's faster, it's cheaper. You don't have to. Parliament One can spank you when you break it, because this is faster, cheaper. better yeah and over the medical side we've just started just today it got confirmed so i can talk about it working with the bill and melinda gates foundation for anemia detection and for blood oxygenation and for because bill gates and i we had dinner once together and we agreed that this has got to be everywhere yeah yeah and they said and i think i just discovered really weird things along the way because like i was a phd student i was a shut in the lab i was doing math I didn't know the world out there.

10:40It's like 1.8 billion people have anemia. There's no good way to see it. And they just locked onto this. Like, hey, come see this. And then other things were, I think we mentioned plant health. There's a lot of industrial inspection. Wafer imaging, color matching. Turns out when you want to quality assure, you want to see if something got print correctly or your clothes got dyed correctly, you hire someone, a human, that is qualified to see color correctly is really subjective. And then, yeah, this is a solution. This is a literal number you can get out. This is blue. This is navy. You can quantify that rather than just let someone tell you, this is blue.

11:23And then, of course, cannabis farms come out of the woodwork, fish farms. All these industries I never really knew about. More will come to mind. I think, yeah, one thing I will say is that most of the customers, most of the inquiries are unexpected because sometimes when I take a step back, which I don't do very often, is this is just a camera, but better. And cameras are used everywhere and cameras have problems everywhere. So people have been coming out of the woodwork and saying, hey, what you got? Can you see my thing? My answer a lot of times, I don't bloody know. So when people use the camera for the first time, how do they react?

12:04It's like a kid in a toy store. They're like, oh my God. Yeah, I remember there was a really surprising application. A civil engineer came to us and said, hey, we need help use the camera because we need help in identifying cracks and bridges. And the serious cracks, I'm paraphrasing here, of course, have algae or lichen growing in them. And it's just a thin, thin, like a microscopic layer that's not visible to the human eye. I remember she said, can you do that? I said, I have no idea, but let's try this algorithm we built for vertical farms. So we went to the car park and picked up a few rocks, and we put it on the table.

12:40And it turns out one of them had lichen on it, and it lit up like a Christmas tree. And then recently, we took it to a bridge, and we pointed it out to part of a bridge, and we see the lines where the cracks were forming and the lichen was growing. And that was just something that just came out of the blue. I remember when they messaged us, emailed us, and said, hey, can we talk? I almost said no. So as civil engineers, what have we got to do with you? So I think I've learned that lesson. I got to take some of these calls. I'm speaking with Robin Wang. Robin is the co-founder and CEO at Living Optics, whose new spectral imaging camera is, as we've been talking about, as Robin's been talking about, faster, cheaper, more accessible than spectral cameras of the past.

13:24Let's talk about the camera itself for a second. And as has been kind of a theme of this conversation so far, I'm imagining you'll say, well, I can't really talk about that, but I will anyway. So that's what I'm aiming for. Is your camera, aside from being faster and cheaper, and I'm guessing smaller than cameras of the past, is it built differently? Are there technological innovations? Are there things that you develop from the ground up that are patented or should be patented going on in the camera itself? or is there something else happening in the software stack and the algorithms or is just simply a matter of, well, nobody's updated the hyperspectral camera in a long time and technology and optics have advanced so much we were able to do it.

14:06I'm going to give you a terrible answer and that is all of the above. Yeah. And I'll go into the detail now. So how about we start from the beginning of the beginning is that this was actually my PhD project. It was a side project. I started off as a fresh-faced, stupid theorist. I'd never... Where? In Oxford University. Okay. Yeah, so I'm originally from New Zealand, which is why I sound like this. And yeah, I came out of an applied math and a pure math degree and went straight into the world of physics. Okay. My supervisor, being a diehard experimentalist, said, go and solve this problem. which was every time we turned on the laser, I used to work in inertial confinement fusion, either cost$70 ,000 or cost 2 million bucks to fire a shot.

14:53So we needed to get as much information out of that shot as possible. So he said, hey, Robin, I shouldn't try to do a British accent. Hey, Robin, go build a camera that sees at 100 trillion frames a second. That's a world record. That camera didn't exist, of course, because that's a world record. So we built one. So I haven't published a result yet because, you know, startups are busy. And yeah, but we got a video where light moves at about three microns per second per frame. That video was 100 trillion frames per second. But turns out only about six people cared for a video of 100 trillion frames a second.

15:32So I applied for a patent. So, hey, well, no, I didn't apply for the patent. My supervisor made me. And then out of the world comes a bunch of VCs and say, hey, what else can we use this technology for the technology was to take ingest a 2d image and spit back out a 3d cube so the 100 trillion frames a second is we did a one shot we ingested all the super fast events onto a 2d image they were reconstructed and did a video that's useless so we said okay wait we can reconstruct spectra too so the main funky part so okay we can reconstruct spectra and the rest follows like oh jesus yes lots of applications right right let's form a company here we are but the funky part of how this technology works is in order to make hyperspectral imaging i realized that you can't just improve the hardware or the software you're getting blood out of stone people are smart people have gone to the end of that path you have to do both together that's where you get the orders of magnitude improvement right so we kind of went back to first principles we rewrote ray tracing software that was horrible if i knew how hard it was going to be i'd never do it again we redesigned the software and the hardware to work perfectly with each other and since i had limited resources at hand i couldn't buy expensive components i was scraping by so i inadvertently by accident stroke of luck borrowed a lot for the mobile phone industry how they design lenses so from the ground up this camera was to be manufactured affordably and cheaply not because i thought i was going to start a company because i was cheap right so i bought a lot of components from aliexpress and scraped together and by a lot of luck it worked so to summarize designing hardware and software together yeah in an affordable way borrowing a lot from the billions of dollars people have sunk into mobile phone manufacturing right so we have a beautiful wonderful manufacturing team in taiwan that we poached from la gun who designed the iphone lenses in young who designed okay i can't tell you that young optics who designed something that is coming out that is very exciting and we stole some from the ipad team at foxconn gotcha and the goal by stealing those people was to scale this product right this thing was to be affordable scalable possible mass-manufactural consistent soon you heard it here first self-calibrating because that's one of the biggest problems in all instruments and so currently it needs to be calibrated before it can it takes about three seconds but we're trying to make that zero second nice good that's why i love doing this podcast because you'll get that question it's like well no it's just it takes a couple seconds it's too long we're getting we're done yeah it's this is user experience the more things the more barriers more friction we can take away from the extra recipe, the more it can change the world.

18:26And that's what very British about us, well, I'm not British, is trying to aim high and slather this over the globe. Right. Aside from the apparent trauma of designing your own ray tracing software, what was particularly difficult, surprising? Maybe you look back and say like, that was one of those out of nowhere aha moments during the development of the camera the hardware the software together that you kind of look back on that really stands out i think the hardest part so i'm going to completely hijack the question it was the people is that the technology always happens if you put enough smart people in the room together that will happen whether you like it or not right but i think one thing i realized is that how much was relied on the people is that if you want the best manufacturers, you can't go to the best optics manufacturers in the world and say, hey, I've got the best technology.

19:24You have to know them. You have to tell them the story. You have to get the right people to sign up for the best products to be made for you. I think that's something that I never really realized is that the best technology doesn't win. The best combination of supply chain people, user experience, that wins. And I think I learned that the hard way. But I think after those lessons are learned, you can hopefully see the fruits of it as, hey, here's a camera. It bloody well works. There's a friendly SDK. It helps you do what you do rather than fights you along the way. So we're talking at GTC. NVIDIA has been around for a while.

20:04They're around before this current AI explosion. I think when we were talking before, you used the phrase AI wazoo that's happening everywhere. How does living optics fit in? There's robots, there's SDKs, there's new hardware, there's pipelines for enterprise applications, all this stuff happening all under the AI umbrella. Hyperspectral imaging, living optics, your camera. How does that slot in? This is about going back to first principles, which is garbage in, garbage out. We've done computer vision to death. RGB, three colors. people are building billion parameter models people are training of that each model takes good big model it takes millions of dollars to train our argument is why don't you get better data to begin with why don't you get a richer more informative data set rather than just adding more layers to your neural network or better transformers lstms more data get better data use smaller models, not because you want to.

21:07No, not because use smaller models, because you can, because that beats state of the art in RGB. Use less data. Spend less engineer, expensive engineer time labeling. This is the argument. Get beta data. Get hyperspectral data. Whenever there's a vision problem to solve, don't use RGB. Just because evolution evolved us to see RGB doesn't mean machines you see in red, green, and blue. If you want to make a really informed decision, see beyond just red, green, and blue. See how it's virtually. I think I'm going to borrow that line and use it randomly with my kids. Thank you. Just keep saying that.

21:42Hopefully people just pick it up. You know what? You want to figure this out, see beyond our GP. I'm going to shift gears for a second. Living Optics is partnered part of the NVIDIA Inception program for startups. You can talk about your startup experience. You talk about it a little bit. You can talk about it in general if you want. But what is the Inception program like for folks out there listening who maybe have an idea, maybe they have a company already, and they're wondering, you know, yeah, I keep hearing about this Inception thing. What's it been like for you? I think inside all the startup program wazoo, NVIDIA Inception has been really bloody reasonable.

22:18They have not taken my time when they didn't need to. They didn't make me come to events. Well, I wanted to come to this one. They didn't make me come. So that's why I'm here. And then they've just tried to be actively helpful rather than trying to push their own agenda. So can't recommend it enough. Send them an email. Might be good to lose. Right. Well said. So what's next? Hyperspectral imaging, living optics. Is the camera shipping? Yeah, it's shipping out. It's on the shelf, I think. So somebody listening right now could go to the website and order one? Yeah, probably. If you're fast, it probably comes in three days.

22:53But if you are between the manufacturing cycles, maximum three weeks. So this is another part we're trying to change. Yeah, this guy is mass-manufactural. The yield is high through the wazoo. I should have made the optical design more risky. The yield is too high. Yeah, you don't really hear that. Could have done more. Yeah, it could have made it smaller, quicker. So yeah, go ahead and buy it. It comes in the whole dev kit. We on purpose made it come with support. We want to help you. Right. We want to make it work for you. You want them everywhere. You want spectral data everywhere, everybody using it.

23:30And what's next? Yeah. Smaller, better, faster, cheaper. We're selling a development kit, the products. Look at us. It's going to come next year. Is there more that can be captured and extracted from hyperspectral imaging? I don't know what a good, I'm not a photographer per se, I don't know what a good analogy is, but if we've been living in a world of three-color images for so long without knowing there are actually 96 colors, just can't see most of them. Is there something beyond that? There is, and it will come, but I will not tell you what it is. Oh, I could not have asked for a better answer.

24:11That's fantastic. Robin, for listeners who want to get a camera, who want to learn more about the company, who want to see a picture of the camera first, because we're doing a podcast, they can't see it, or for folks who just want to learn more about hyperspectral imaging, what you can do with it, how does it actually work in more detail, all that kind of stuff. Should they go to the website or should they go to find out more? Go to livingoptics.com. On the top right thing, there's a shopping bag button. Click on that. And I'm sure this is not the first time you've heard this, but do you do voice acting?

24:45Do you read bedtime stories somewhere people can go? And granted, I've been at the show podcasting for two days straight here, so, you know. But your voice is so soothing. Thank you. This is just me talking for so long. This is my hoarse voice that happened after six hours of shouting at people straight. Well, if you ever do voiceover tutorials for the camera, you know, shout for six hours and then record. Because we're talking about the future, that it's hard to visualize, literally invisible. And yet I feel so calm and soothed. Thank you. Yeah, it's fantastic. Robin Wang, thank you so much for taking the time to come on the podcast.

25:20I need to go find a way to experience hyperspectral imaging myself and see what 96 channels of data look like. You've inspired me. Come to the booth now. I'll shout at you. Fantastic. All right. We're going to sign off. We're going to go to the booth. Enjoy the rest of your time at GCC. Best of luck with everything. I'm very excited to chart the progress of Loving Optics and see where the next generation of imaging takes us. Thank you so much.

25:53Thank you.

26:26The End