Dotlumen CEO Cornel Amariei on Assistive Technology for the Visually Impaired - Ep. 217

0:10Hello, and welcome to the NVIDIA AI podcast. I'm your host, Noah Kravitz. We're at GTC 2024 in San Jose, California, and I'm joined now by Cornell Amore, CEO of DotLumen, a Romanian startup whose first product is glasses that empower the blind. I'll let Cornell explain it, but this is really an incredible looking use of technology that I'm personally really excited to learn more about. So without further ado, Cornell, thank you so much for taking the time to join the NVIDIA AI podcast. Thank you so much for having us here. really, really exciting to be here. I mean, we're from Romania. We're from like 7 ,000 miles away, but we came here to show what we are building.

0:50We're building some amazing things. So let me actually tell you a bit of what we do. We build glasses. Get into it. Yeah. We build glasses to empower the blind to live a better life. What do we mean by that? I'll start with the problem. You have 300 plus million people with visual impairment, but they actually use the same two solutions for their mobility for thousands of years. The white cane, so the stick for the blind, and the guide dog, the service dog for the blind. And these are great solutions, but they have a few problems, specifically the service dog. To train a service dog, it's more than$70 ,000.

1:20It's actually in some cases reaching even$200 ,000 to train a single guide dog for the blind. That's number one. But number two, and the biggest problem is that it's so much responsibility to take care of a guide dog. And because of that, in the entire world, to 300 million visually impaired, we only have 28 ,000 guide dogs. So really, really incomparable. The US actually has the best statistic, a million people blind and only 10 ,000 guide dogs. And that's actually the best ratio in the world. But that's where we come in. We said, hey, we come from autonomous driving. We know how to make cars drive themselves.

1:55What if we can make like a self-driving car, but for the visually impaired on the pedestrian side? And this is what we built with the Lumen Glasses. So basically as an analogy, a guide dog works by pulling your hand, avoiding you from obstacles, keeping you safe on the sidewalk, etc. Right, you've got the dog's leash, your harness in your hand, the dog walks ahead of you. Yeah, it pulls you in everything. And the glasses, the lumen glasses, the dot lumen glasses, they do exactly the same, but they don't pull your hand. We actually use haptics to pull your head. So you actually feel how something is, it's pulling your head towards the direction you have to go, avoiding you from obstacles, keeping you on the sidewalk and everything.

2:32But it's actually a self-driving car. Rather than driving wheels, we drive people with visual impairment. It's the most advanced assistive technology in the world. Over 300 blind people have been testing it. We are releasing it beginning of next year, starting with Europe. But then in the second half of next year, we're coming here to help people in the United States. It's the work of over 50 colleagues. We founded this company three and a half years ago. In our country, where we come from, Romania, we were the first deep tech startup. And the European Union is actually one of our investors. So really, really, really, really amazing.

3:07We're so proud to bring, you know, European technology, specifically Romanian technology, all over the world. But maybe to answer one thing, I founded this company because I come from a family of people with disabilities. I'm the only person without a disability in my family. And after a career in the automotive field, where I've seen that we can actually use autonomous driving for something else, I built this company in order to help people. There's so much in there. You covered it all. We're done. No, we're sitting, obviously it's an audio podcast, but we're sitting and you have a pair of what I assume is a prototype.

3:39Yeah, this is a prototype. This is one of the latest generations. This is actually not a working one. Those are the boot. So we do demonstrations at the boot. This is one of the mock-ups which I have with me. But it's basically all the components are inside, just missing a few wires in order to start it. So for folks listening who haven't seen pictures, to me it looks kind of like a VR headset. Yeah, it is. It is actually an XR headset by all definitions. Right, right, right. It's an XR headset, so it doesn't have a, it doesn't, it doesn't, it's an augmented reality headset, but it augments with haptics.

4:09Right. It doesn't go on your eyes, so it sits above your eyes on the forehead. On the forehead, okay. And that's where the haptics are and everything. But, you know, it has speakers, you can talk to it, it has voice vocal recognition, it has everything of VR headset, except the screens. Right. So maybe a way to get into this is to, because you started to talk a little bit about it. And how does it, what's the experience like when someone puts the headset on? Walk us through that. Sure. So the first time you put a headset on, it actually starts in a tutorial mode. So it begins talking to you so you can begin training with it.

4:41So the first thing, it will teach you roughly where the buttons are, what are the parts of the system. So we have six cameras in front. We have the batteries and the computer in the back and everything. That's the first thing which it does. But then it goes into describing how the haptics works. and you actually feel the haptic feedback and there's actually some tests and some trainings which you do with the device and the device does them automatically on you. So it asks you to train, to turn your head. Sorry, for people who might not know the word haptic, it's similar to if you have a phone and your phone buzzes.

5:10It's a little motor that makes a vibration. There are a lot of motors which vibrate on your forehead representing direction. Are they all on the forehead? Yes, they're all on the forehead. Okay. So it actually teaches you how to use them. So for example, if you feel the haptics, the vibration is going to the right. You have to turn your head to the right. And you do all of these training sessions so you can quickly get accustomed to using it. And, you know, a full training is like half an hour. But here at the booth, we do trainings in a minute and a half. Sure, yeah. And it's still good enough for people to walk.

5:38So we had at least 40, 50 people in the last two days walking blindfolded. So we blindfold people. Yeah. We put a device on. I said, okay, handle it. And you can go and enjoy GTC without seeing a thing. I mean, if you can navigate the GTC trade show floor, you know. And with, you know, two minutes of training. Yeah. It's that intuitive. But then, you know, when we test with blind people who are used to walk without any kind of visual stimuli, it's so incredible to see them, like, walk quickly, run with the device. We have so many videos when we do that with blind individuals, you know, running on the streets or on busy conferences and everything.

6:13Right. Absolutely amazing. And even, you know, when we see people, for example, we had recent experiences across the world where people who don't know English use the device. And even if the tutorials right now are in English and they're going to be translated in 70 languages, even without actually listening to the tutorials, they still get it and they still navigate with it very, very well. And that's really, really, really incredible and amazing for us. So you put it on, you listen to the tutorials, you feel how the vibrations direct you. And then once you pass some of the tests, which the device does on you to make sure that you properly understand the feedback, You can just start the guiding mode, which basically does exactly what the guide dog does.

6:53And you will begin navigating. So you will go, we'll keep you away from obstacles if you are on the street and there's like an intersection with the name of the streets. It will help you go on crosswalks. It will tell you if you have to go like on stairs up or down or on curbs or all of that. So it becomes a self-driving car, driving your head basically, rather than driving wheels. So what can it sense? You mentioned there are six cameras. Are there microphones? There are also microphones. So, of course, we listen to you, so we can talk. You can give the vocal commands and everything. But, no, to start a bit more technical, the first thing which we do, we understand where the ground is.

7:31And, you know, in the robotics world, that's something which was solved. But in the robotics world, you know, the distance between the ground and the camera is usually fixed on a person. You have so many degrees of freedom. You can move your neck. You can move your head. You can, you know, lay down, stand up and all of that. So we, first of all, we understand where the ground is, which is tremendously complicated. And after we do that, we understand the ground around you and understand obstacles which are above the ground or below the ground. That's the first level of understanding, like geometrical understanding, we call it.

8:01Below the ground meaning if I'm walking on a sidewalk, right? Potholes, potholes, or even worse, I mean, like droppings or anything like that, which is a bit, I mean, if you're going to the mountains or anything like that. Right, right. So that's like geometrical understanding. We also understand how you move. So we use all of these cameras to track how much you move. And with a very, very good precision, I mean, inches of precision to understand how much you move and which way you're going and everything. So all of that happening real time, that's the first level of understanding. However, it's not necessarily enough.

8:31Just because something is flat, it doesn't mean you can walk on it. And that's where the robotics world stops. They assume that if something is flat, because in a warehouse, if it's flat, you can drive. You know, a lake, geometrically, is perfectly flat. You cannot walk into a lake. And I'm telling you that if you bring some robotics from the factories or even some delivery robotics and you put them in front of a lake, they will drive into it. You're just going to walk right in. They will walk right in there. So that's where we begin using AI. And we use a ton of AI in order to understand what surfaces are safe.

9:02We call them walkable, unwalkable, or non-walkable. And there's a special category which we call conditionably walkable. For example, a crossing that's a conditionably walkable surface because it's conditioned by, well, first of all, being sure there's nothing on it, and then also conditioned by the color of the semaphore, of the traffic light and everything. So we use a ton of AI, custom-built models, custom-built architectures on our own data, hand-labeled by us. We have our own labeling team. Data from over 20 countries, over 1 million images are being used to train the lumen glasses. And these are images varying from sandstorms, snowstorms, general storms, rain, all of that in over 20 countries, I think over 100 cities at this point, and not only cities.

9:45And we train the AI on all of that data. I think every week we have a new set of models. And a ton of AI models are running in parallel to make sure that we understand the world. So, first of all, surfaces, where it's safe to walk, where not. But also what objects we can interact with. Are you indoor? Are you outdoor? What kind of a scene are you in? So all of that voice recognition also to recognize the voice of the user, all of that is running inside the device. We do not need internet connectivity. Even if we wanted to, because of latency, we couldn't like do cloud processing. You know, you wouldn't trust a self-driving car, which is connected to the internet.

10:21And that's all happening in the cloud. You wouldn't trust that. The same for us. We couldn't trust that. So we build everything in the device. And that's basically how we perfectly understand the world. Okay, we understood the world. What do we do next? we begin doing path planning. Where can I take you through the world? So over 100 times a second, we compute a safely walkable path, actually even more than one, where we can guide blind individuals. And we represent that path and actually we keep you on that path using haptic feedback. So basically, you know, taking your head and spinning it and turning it towards keeping you on that path, which we recompute 100 times a second.

10:58So self-driving car for your head, all happening on something which you can comfortably wear on your head. So many questions. How much does it weigh? Well, I'll have to give you the answer in metric units. Okay. The final version will be around 850 grams. Okay. The prototypes which we're showing here were a bit above one kilogram. Right. So about 1 ,000 grams. But these are like two-year-old prototypes, which we still use a lot. But the production version, which now we have in the final refinements of it, is under one kilo. So it's basically around 850 grams. They are VR headsets, which are heavier.

11:32Yes. And they still have cables, which are able to connect to the computer. This is all, that's everything. One unit, batteries inside, processors inside, everything. What's the battery life like? It's basically a full-day battery life. Now, what do we mean by full day? We mean two hours of continuous walking. So when is the last time you walked more than two hours? The answer is here at GTC. I was going to say for you, yeah. But for that example, for those cases, you take any USB-C battery pack, you plug it in, and it extends the battery life. Okay, sure. We wanted to have something which you can just put on your head, doesn't need anything else, and you can take you in 80 % of the use cases.

12:07And for those last 20%, take a battery pack with you and it extends it. I remember at the beginning of it, when we started it, there was no computer which can handle that amount of computing on something which you can easily wear on your head. So we actually had two units. We had the headset and we also had something we called the hip set, which was something which you put on your belt, like your hips. and we had that and it was like 3 ,000 grams and everything and the number one request was can you get rid of that thing which was on the hip pack yeah and then after we did that and everybody was wow then a few of them asked but can't you put it on the hip so it's lighter on the head and I was like we just removed that so now it's one unit doing it all but it's lighter than some I just recently tested a VR headset which is heavier than what we built and I'm like wow we can build all of that into something which is lighter than some VR headsets.

13:01Yeah, that's remarkable. And there's all of that computing inside. I want to ask you about the computing that's going on inside. But before that, you mentioned that the headset computes walkable paths and it can look at not only the surfaces, but sort of the condition. Forgive me, I forget the word you used. What about other objects? Like, can it detect a person or, you know, the door of a building or these types of things? It's actually, and we also have the patents for this and everything, it's actually much more than that. So, for example, I want to go to somewhere new. I've never been to. The device has never been to there.

13:38And that's something you cannot even do with the guide dog. I can't tell the guide dog, take me to the closest Starbucks because the guide dog wouldn't know where's the closest Starbucks. But we do because we have maps and everything. I don't know. There was a Boston Dynamics robot dog walking around earlier that might know. I don't know. I'm not that sure, but okay. Yeah, okay. Let's see. But there's something which we can help Boston Dynamics with, and I will tell you right away with what we can help them. So first thing is you can actually, for example, go on Google Maps on your smartphone.

14:05Blind people, they still use Google Maps. You can search where you want to go. You can press share and share it to the device. And then the device will begin taking you there. But for example, I take you up to very close to the Starbucks, but I want to get in. So we detect where the door is. We take you very close to the door, and then we represent you where the doorknob is with sound. So we have some patterns on that as well. So we tell you like directional sounds, we do some crazy things there. So we can actually guide your hand towards the handle. Right to the knob, yeah. Because when you say that you want to take somebody to the door, you actually want to take them to the door handle.

14:36Yes. Likewise, when I want to take you to the car, I don't want to take you to the car. I want to take you to the door handle of the door, which has a seat behind it, which is free. Right. So these are things which we take for granted as humans. Yeah. But from a computer science standpoint, it's massively complex. I can't even imagine. So we actually have the patterns for that and how to do it. and that's what the device does. So it actually takes you up to the door, helps you open the door and then it continues guiding you inside and everything. It detects persons, it detects tons and tons of things and for some of those objects, we know how to interact.

15:07So we know how to take you to an empty seat rather than a busy seat, a new seat. We know how to take you, for example, now we're integrating public transport. So if you want to go somewhere very, very far, it will actually take you to the bus station and then help you get on the bus and then help you get out of the bus and lead that. So we're now building it to be infinitely scalable. We're making jokes about booking flight tickets with it. So we actually had the CEO on the European level of one of the large credit card providers. He came to us one day and he tested the device and he was very, very impressed and everything.

15:44And then the first question is, how do we integrate payments into it? And he had some ideas about that. I was like, okay. Well, that's kind of, you know, and you spoke to it almost as soon as I thought about it, the navigation potential. But, yeah, I mean, you've got a very powerful computer, you know, running inside. And so the add-on possibilities. The thing which is, I think we get a lot perceived as a hardware company. But if you look at the company, less than 10 % of the company is working in hardware. Right. We're a software startup. Sure. We built spatial navigation AI. We built autonomous driving on the pedestrian side.

16:24This is what we know how to do. But we picked the most complicated challenge in the world, which is guiding people. You know, robots, they kind of answer to what they tell the wheels to do. And they are, you know, you know how a robot reacts, you know, the dimensions of the robots and everything. But a person is so much more complex because, you know, we have to track what they do with their hands. We have to track how they move. We have to track if they're wearing heels or they're not wearing heels. because that actually changes the performance of the system. And we have to continuously readapt.

16:53I mean, you put it on and it learns your height and everything. And then suddenly you walk with it to your shopping mall and you put high heels. And then we have to immediately recompute everything because you're suddenly higher up and everything. So we picked this challenge, which was the heaviest, hardest of all the mall. And now we build the best pedestrian navigation AI in the world. And we can reapply it to delivery robotics. We can reapply it. A lot of humanity robots are being discussed now. None of them can navigate in the outdoor world. We know how to make them navigate in the outdoor world.

17:23A lot of discussions about robotic guide dogs, which it's not a good idea because, you know, it's actually you take everything that is bad about a guide dog and you replicate it into a robot. The only good thing is you only charge it, but then you have to spend a hundred thousand on the robot and then you have to charge it and the range is not that good. Right, right, right. They don't walk a lot. Right. But they cannot navigate outdoors. They still need our software to do that. And we also have the patents for it. So we are the spatial navigation AI startup, which their first use case, their first product was in glasses aimed at helping the visually impaired because nobody else in the world was able to build that.

17:59And because it was a challenge so hard to solve that we knew we're going to create some amazing technologies. And this is what we created. I'm sure there's not just one answer, but what is the hardest part about getting to where you are in the development cycle? I don't even think it's a technical answer. Okay. But I'm going to give you like three different answers to this. The first one, we come from a country which is the least innovative country in the European Union. There were no deep tech startups before us. We started the first deep tech startup in a country where there wasn't any kind of investments in deep tech.

18:31There wasn't any kind of investments in anything related with hardware before us. And, you know, I can count like four or five VCs in the country. Right. So that was like how crazy you have to be to quit your job during the pandemic in the automotive world, in a very good position. Start a company which built something which very large companies have tried before and failed in a place of the world where there was never an investment in deep tech. Yeah. That's the first answer, like only creating this company. How'd you do it? Well, I should start from the motivation. I come from a family of people with disabilities.

19:06Right, you mentioned, yeah. the only person without a disability in my family. And that's where my motivation comes from. But at the same time, I worked in an automotive field where I dealt with autonomous driving and everything. And I say, hey, we kind of got to get one day to the technology which will make a car drive itself. What if we use that technology on the pedestrian side? What can we do with it? And, you know, disabilities in my family, together with that, that's how the idea came. So that was the first challenge, starting a company where we're in first deep tech startup doing this in our area.

19:37The second thing is when we started computing, there was no computing platform which could enable us to build this into a wearable. The computing power was not there yet. And we did a bet. Sorry, remind me again, when did you start? 2020. 2020. 14th of May, 2020. The first day out of curfew in our country. Oh, is that right? The first day out of curfew. Monumentous. We built our office. We moved into our office. And then one week later, a curfew came in back. Right. That was happening. Everybody was celebrating 14th of May, and we were celebrating by assembling our first office. Yep. When we started, there was no computing platform, which had enough power to do what we did, what we required.

20:18But we took a bet. We said, hey, I think it's going to come. In the following year, two years, I think it's going to come. And the bet paid off. It was true. It came roughly a year, year and a half after we started. Compressing all of the processing of a self-driving car into something which you can wear, reducing all of that and making it incredibly safe those were huge technical challenges the way we solved them we have the patents for it and we're still applying to some patents for it it's absolutely amazing at this point you can actually and we're going to test this you can actually shoot a bullet through it and it will still be safe enough to stop you so we have multiple redundant systems geometrically splayed across the front and the back of the system and I know the last problem you have is the guidance when there's a bullet coming next to your head but it's designed incredibly robust to always, always have the required feedback to put you in the safe state.

21:10And the safe state is to stop you. So if there's anything, the device will stop you. And it doesn't matter if the batteries die, if cameras are occluded, if they crack, if the device breaks into you, it doesn't matter. It will stop you. And we did test that, not with a bullet, but we're going to test it with a bullet and we're going to film it in slow motion. I have some good friends which have those incredible slow motion cameras. So that was the second challenge. at a time where there was no computing platform available, we decided to take everything, minimize it incredibly until a computing platform would be available.

21:40And the moment it was available, it all worked. That was the second one. The third biggest challenge, we designed something for people with a very, very tough disability, and we had to co-design it with them. And that was an immense. I mean, we did all the testing at the beginning during COVID. We had to create testing rooms which were sterile and large enough so we can actually test navigation and everything. So that was, I think, the third challenge. Working, building, testing during COVID with people with visual impairment to get their feedback, to understand what's working, what's not working, and to continuously do that scalably.

22:17And now over 300 blind people have tested already. Amazing. And so this version you said coming out in Europe, first half of 2025? Yeah, it's coming actually end of the year. End of this year. End of the year in Europe, and next year we're expanding internationally. And are you selling direct? Is it through healthcare providers? In Europe, there are specific funds for assistive technology. So technology which helps people with disability. They vary country by country, but in each case we target those. Our purpose for Europe, actually our purpose for the world, is for visually impaired people to get this either for free, either heavily subsidized.

22:56So that's our purpose. Now, it varies a bit around the world. So in the United States, we have identified over 79 different reimbursement programs. So it's a lot. And, you know, some are very few at federal level. A lot of them are state level. Some of them are local. Some of them are CSR. And it's a bit different than how we have it in Europe. So we're still mapping those. And this is why we kept the U.S. a bit further on. Because the moment we enter, we want to enter with the best amount of support, financial aid, in order to purchase this. What's the price point? Do you know yet? We know it.