Meet the robots and drones working in warehouses

0:00Starting a business comes with its share of ups and downs, which is why staying true to your vision is essential. a non-negotiable for Romeo and Milka Bregali, Capital One business customers and co-owners of Ra's plant-based restaurant in New York. Romeo and Milka took a leap of faith when starting their own restaurant, gutting an empty space and building it from the ground up. Every pipe, every wall, every detail. But building from scratch came with a heavy financial burden, which is when they turned to their Capital One business card. With the flexibility of the card's no preset spending limit, they were able to spend more and earn more rewards while bringing their vision to life.

0:36Today, Raz's success is proof that with passion and the right support, it's possible to make your dreams a reality. Learn more at CapitalOne.com slash business cards.

0:54As you probably know, this year is the Men's World Cup. But there's a lesser known tournament also afoot. The RoboCup. Oh, yeah. I think the referee decided that was a fair goal. So now you see the robots are moving back. So it was a goal for the blue team. Imagine autonomous bipedal robots, the size of a five-year-old, teamed up and duking it out on a small soccer field. There's a ref, a gaggle of fans, and the very human team members watching anxiously from the sidelines. The cool thing about the competition, the RoboCup competition, is that once the whistle starts, you know, you press the button, it's hands off, you can't do anything, right?

1:37So you got to watch them, and if they play well, great. If they don't, then so be it. Raffaello DeAndrea is a professor, engineer, artist, and yes, former robot soccer team lead at Cornell. RoboSoccer is one of the more novel projects he's worked on, but he's also using AI to transform manufacturing. He's co-founder of Verity, a company specializing in AI-powered drones. You'll find them at warehouses for some of the world's biggest companies. These drones take on the tedious work of logging inventory, doing it accurately, efficiently, and without the risk of human error. This is a very timely topic.

2:19Physical AI dominated the headlines at this year's CES, one of the biggest global tech conferences. NVIDIA unveiled new models that make it easier to train AI-powered robots, and we saw everything from surgical robots to humanoid robots on factory floors. I believe physical AI will be one of the areas where we see the biggest innovations, especially when it comes to manufacturing. There's so much to talk to Raf about, so let's get into it. I'm Rana El-Khalyubi, and this is Pioneers of AI, a podcast taking you behind the scenes of the AI revolution.

3:02Hi, Raf. Welcome to Pioneers of AI. It is so great to have you on the show. Thanks for having me. So you have such a fascinating background, and I want to double-click on the various things you've done in your career. You're Italian, Canadian, and Swiss. Correct. How did that impact your decision and your career choices, I guess? Yeah, I guess I think of myself as a citizen of the world. I was born in Italy. I moved to Canada when I was nine. I moved to the U.S. when I was in my early 20s to do my PhD. And then I was there for a while. And then I moved to Switzerland about 15 years ago. So, you know, I've lived in many places and I try to take the best out of every place that I stay.

3:41So you're also an artist and you've created numerous art installations that have also served like research projects. So can you tell us a little bit more about the blind juggling machine? We wanted to make a machine that could juggle a ball without knowing or feeling, seeing it, hearing it, touching it, where the ball was. Can we create a juggling machine? And it became an exercise in dynamics and chaos. And it was a lot of fun. These juggling machines have a simple design. There's a large metal plate which balances and then bounces a small red ball. Some of the machines propel the ball up and down.

4:17Simple kind of juggling. Others are more dynamic, swinging left to right on a pendulum, passing the red ball to itself back and forth. Basically, it can hit a ball like a ping pong racket with a ball. You hit it up and down, and by actually having the right shape of the paddle and exactly the right motion, you could create a stable limit cycle where this thing would stay on there indefinitely. And what was also neat is that we, you know, we discovered that you could make it chaotic and you could use this to transition between one periodic state to another periodic state with use of chaos in between.

4:57So using chaos as a, it ended up being chaos as a control mechanism for moving between different dynamical regimes. So it was a very fundamental research project, but it had a very visual incarnation where, you know, you see this thing juggling balls and then all of a sudden it goes chaotic and then it settles down to a completely different pattern and there's no feedback. You know, robotics, AI, however you want to call it, it always involves this feedback loop. But, you know, I wanted to challenge myself and say, what can you actually do without any feedback? Can you make something really impactful that doesn't use any feedback at all?

5:32And that's where the blind juggler came from. Yeah, that's a great segue into our robotics conversation. So you're also a founder, an entrepreneur. And before your current company, Verity, you started a company called Kiva Systems, which you then sold to Amazon. And that basically was the start of Amazon Robotics back in 2012. And you started a company in 2003. Correct. So you were super early with that company and with bringing robotics to the world. So I want you to take us back to that time. How was it like to build a robotics company? I mean, it was super exciting. I got to work with, you know, Mick and Pete, the other founders were, you know, spectacular people.

6:12You know, if you're going to do a startup company, make sure you pick the right people. That's the most important thing. But it was super exciting. We had just won, you know, this RoboCup championship four times as a professor. I was on sabbatical at MIT, hanging out with folks at the Media Lab and also in the aerospace department. And that's when I met Mick and he had this vision of using mobile robots in distribution facilities. And he was using our RoboCup videos to convince anybody that would listen to him that, hey, you could use robots in distribution facilities. So we hit it off. I quit my sabbatical.

6:43And along with Pete, we started Kiva and kind of never looked back. Hired many of my former RoboCup students and kind of progressed from there. Amazing. Can you tell us about some of the use cases that Kiva kind of went after? The use case was to bring inventory to people in distribution facilities. Before our system, the way that people would handle this, like at an Amazon facility, is people would actually walk around and fill an order, go back to their station and put in a box, pack it, and that would be the process. So in these distribution facilities, people would have to walk huge amounts to fulfill these orders.

7:20And the idea behind Kiva was simply to bring the goods to the people instead. and just we built a whole solution around it and it was successful technically and then economically it also made a lot of sense for our clients and which is what resulted in the Amazon acquisition. I think a lot of people don't realize that you know AI slash machine learning has been around for a while so I yeah I would love for you to just kind of share what kind of machine learning was implemented in these robots what kind of sensors did these robots have? Right well let's talk about the Kiva robots. So the Kiva robots had cameras that would be used for navigation.

7:59We also had some sensors that would be used for obstacle detection and obstacle avoidance. And that was pretty much the only sensors that we had. And in terms of actuation, it was, again, relatively straightforward. It was a two-wheel robot and a mechanism for picking things up and putting them down. Like a hand? No, not quite. It was like a corkscrew. The nickname was TAS for Tasmanian Devil. TAS looks like a large orange rectangular Roomba. It moves underneath warehouse pods that hold inventory. And using its arm, it lifts and secures these pods, moving them around the warehouse. Basically, the robot would go underneath the pod, and it would have the screw, and we would start turning the screw at the same time that we would spin the robot so that in inertial space, the screw didn't rotate, it just went up and down.

8:50So it's a very clever coordination of these two things. It would pick things up and it was very robust, very few moving parts. We only had three moving parts in the whole robot. And the magic was just creating all the algorithms around it to make it work all the time, never get lost, never hit anything. Did you keep the humans in the loop at all? We did. We would still expect people to be at the packing stations. It's just that they wouldn't have to walk around. They would be sitting there, they would look, turn around and look up and there would be a shelf in front of you with a light telling you exactly what to pick.

9:24You'd pick it, you'd put it in the order box. And by the time you looked around again, there would be a new mobile robot that brought a new shelf with a new light that would tell you exactly where to pick it. So we removed the extremely low skilled aspect of walking around a warehouse. You know, manipulation is something that we take for granted, but it's a very difficult thing for machines to do. Yeah. A lot of robotic companies today are trying to address that. Did you see this kind of new age of embodied AI coming? And I guess, how does the world look different today compared to when you started Kiva?

10:01Like if you were starting Kiva today, what would that look like? Oh, geez, that's a that's an interesting question. So, I mean, to answer your first question, you know, embodied AI, I mean, I guess that's what I've been doing for, I don't know how many years, just the name changes. But in terms of what's changed now, there's certainly a lot more capital available to do it. Kiva, the acquisition was$775 million. And I think the total amount that Kiva had raised was less than 40. Wow, amazing. Right? So now with 40, it's very difficult to do very much. I think the expectations are much higher now, number one.

10:43And number two, I've heard many people say this, Kiva started this robotics arms race. But frankly, there haven't been a lot of other successes. There's actually very few that actually are these big, big home runs. And it's because it's really hard. It's really hard to create, from a technical perspective, something that works all the time. And then the other one is that there's a market for it where, you know, it actually brings significant amount of value to what people do. The third consideration I would also add is cost, right? It sounds like you kind of constrained the problem enough so that these robots were cost effective.

11:20Say more about that. I think this is why, you know, there's always this debate between, you know, generalist robots versus tailor-made robots. And people always like to make the analogy that, hey, why can't robots just be like the laptop? You know, Bill Gates wrote this very famous article many, many years ago about how he felt robotics were at the same level as where, you know, software was and that the same thing would happen. And of course, it didn't. It's very fragmented. There's good reasons for it. What do you think the reasons are? When you're interacting with the physical world, you can't just abstract the world as, you know, stuff comes in and I do something and it goes out, which you can do with laptops.

11:58is very modular with compute. When you're interacting with the physical world, you have a whole bunch of other constraints that you have to deal with. If you can make something that doesn't cost very much, but you're spending all your time doing maintenance on it, it is a costly solution. So when I mean cost, I mean upfront, but also just to keep this thing operating out in the field. I think that if you do things that are tailor-made, you're going to have a tremendous advantage over something that's a generalist solution. In a minute, the role Gen.AI plays in the world of robotics and how it's accelerating innovation.

12:34Stay with us.

12:49if you've spent any time building ai products or leading technical teams you know this transformation doesn't fail because of ideas. It fails because teams can't move together. Enter Atlassian's teamwork collection. It has planning in Jira, documentation in Confluence, video updates in Loom, and now AI agents in Rovo, which connects the dots across your work so nothing gets lost. It's one AI-powered teamwork platform designed for how modern teams actually build. Learn more at atlassian.com slash teamchanger. That's A-T-L-A-S-S-I-A-N dot com slash teamchanger.

13:39Welcome back to Pioneers of AI. You can check out this episode on YouTube, where you can watch some of Raffaello DeAndrea's robots in action. What role does generative AI and kind of these foundation models play in the world of robotics? So it depends on what aspect of robotics you're doing. I can tell you a little bit about how it's becoming super important in what Verity does. Verity creates systems that allow our clients to operate their warehouses more efficiently. And the way they do that is they collect real-time data of the facilities and they use self-blind drones to do this. They're flying in the warehouse, collecting data in real time, uploading their findings, combining the data with the client's data, and then figuring out, you know, what is really happening in a client's warehouse.

14:27So visibility is super important. And what we're finding is that we collect all of this data. We can actually bring to bear visual language models to process this data and make inference. Safety, for example, is very important to our clients. They want to use our system to do safety inspections for them. You can use these models to actually do a lot of that work. So it's really making obsolete a lot of, you know, the workflow of here's a problem. I'm going to apply some machine vision techniques. So I'm going to hire a whole bunch of people and productize it. It's kind of bypassing that whole piece.

14:59Yeah. I would love to double click on this and like slow it down. I truly think this is very fascinating. I did my PhD basically in computer vision. Cool. And you're absolutely right that these vision models, these vision foundation models are changing all of that. So in the world of computer vision before generative AI, what does it look like? And then how does a vision language model change that? So, I mean, that's a very big, big question. But let's use the specific example that I gave, right? So right now we can take images and query these engines, you know, whether it's Claude or Gemini. It's the right types of prompts and you have to set up the flow properly.

15:38But you can ask questions like, you know, is there a fire extinguisher? And is it of the right type that's consistent with the way that this facility operates, right? So it can actually do that for you. Now, how would you have to do that traditionally? First, you'd have to figure out what are the different types of fire extinguishers that are allowed to be here. And you'd have to interface with someone. Someone would have to type that in. As opposed to now, you can just query the documentation that exists at a client's site, and it can actually figure that out for you, right? So that's already something that you can do.

16:09And then there's the part of, okay, now you got to write some code to find the fire extinguishers, right? So, okay, so people would then try to do shortcuts, like they're red, right? So let's look for red stuff, right? And all of a sudden now you could start making progress to look for things that are fire extinguishers, but you got to be a little bit brittle because if the lighting is kind of weird one day, then maybe that red doesn't look red anymore and then you're going to miss that fire extinguisher. or, and I mean, I'm just making something up, but it's specific to this example is what if all of a sudden now orange or green become acceptable?

16:41It's not going to happen, but you see what I'm saying. If something changes, now you got to change that, you know, that process that you've gone through. So we would do things that were, you know, we'd get clever people to create algorithms, take shortcuts, be clever. That's the whole, that was the whole point, right? Can you make the problem simpler, collapse it so that you can actually create something that works with high probability and doesn't take you years to develop. That's the difference is that right now you don't have to use that cleverness. You can just use these systems that have been trained on huge amounts of data to kind of bypass that step for you.

17:17Yeah. At my company, Affectiva, we service the automotive industry. And so, for example, one use case was we wanted to detect if there's a child seat in the car to then kind of, the safety application is to be able to flag if there's a child left behind, right? And so in the old world, we would train, we'd essentially actually have to collect data where there's a car seat, you know, a child seat in the car, and then we'd have to like change all the different angles and different car seat models. It was really expensive to train a model to do that, right? Yeah. But to your point now, you can literally upload a picture.

17:54Exactly. And it will just tell you if the car seat's in there or not. So it bypasses that whole process. It's very cool. All right. I want to get your perspective on humanoid robots. We are seeing companies like Tesla, figure AI, physical intelligence, you name it, enter that whole space. Yeah. Where do you think we're going with that application, I guess? The argument goes most of activity revolves around what people do, human labor. So if you create something that has a form factor of a human being, it's going to naturally fit into whatever work environment, which, of course, makes a lot of sense.

18:25So that's the plus. The market opportunity is super, super tremendous. So what are the challenges? There's many, but I would say if you look at the kind of two key ones is first, can you create something that is as capable as a human being to do tasks? And it doesn't have to be all tasks, but are there enough tasks that you can create a machine in a reasonable amount of time that it could replace what humans can do? And humans are amazing creatures. Like we talked about dexterity. We talk about, you know, vision and just our ability to interact with the world. I mean, it's amazing. But then the second one is, can you do that cost effectively?

19:02Can you create something that actually brings economic value because you can do it for low enough cost? And I think from a market perspective, that makes a lot of sense. From a technology perspective, I think the field is moving pretty rapidly. I think you could make the case that there's enough tasks that are low level enough that you could these things within five years could do. I think that's the case. I think where I'm struggling with is the economics. I'm struggling with creating something that just simply works, that doesn't cost a lot of money and doesn't require every six hours to do maintenance on it.

19:40Yeah, this is like being tongue in cheek a little bit, but I'm excited for a robot that can fold my laundry. But right now it takes forever because it is pretty complex if you think about it. Yeah. And it costs like, I don't know, at least$20 ,000. And I'm like, I'll just do it myself. That's right. I think that's the case of the, it's not low-hanging fruit. It's not a good first application for these machines. It might be 20 years from now. There's other things that are probably better matches, like, for example, being in a grocery store and stocking the apples. I think that that might actually make a lot of sense.

20:18That's a problem that's well within reach. Yeah. I'm an investor in a company called Chef Robotics, And it's not exactly humanoid, but it's a pretty versatile robotics arm that does food packaging. Yeah. So they can literally sense whether it's blueberries that needs to get picked up or spreading Nutella on a toast or whatever. Yeah, that's also a great, I think, application because let's say you create a restaurant theme around it. You can pick the items in your food that are easy for the machines to make. You can have a difficult version and an easy one and people like it just the same. So pick the easy one.

20:53Yeah. Fascinating. We had, I'm sure you know her, MIT professor Cynthia Brazil on the show. We talked a little bit about, does every robot need to be humanoid? And she had a very specific point of view around that. Her view was like, well, it depends on the use case. If the use case requires human-like capabilities, then sure. But if not, then don't do that. I'm curious, what's your point of view? I agree. And I think that there's two reasons why you want a humanoid. And one of them is we already talked about human environment is created for human beings. So it's likely if you create something that has a human shape that it will naturally find itself in that environment that can perform the task effectively.

21:31The other one is if there's an emotional component that needs to be affected, like, you know, of course, people talk about nursing homes and things like that. But it could also be a receptionist at a, you know, at a hotel or whatever else. By the way, that wouldn't work for me, but I'm just making a general statement. I'd rather just have a screen. But so I think I think that there's different reasons. But I think that it goes to what we said earlier. If there's a big enough market for a task and the task is relatively well structured, you're going to do better with a purpose built machine. Yeah, it's going to be cheaper.

22:06It's going to be cheaper. Yep. We also had Vinod Khosla on the show and he predicted that we would have a billion bipedal robots by 2040. Do you agree? I disagree because we are going to see impressive growth. And I think we are going to see that there's going to be hype and there's going to be, you know, the trough of disillusionment. And that takes a significant amount of time. And I think we're just at the early, early days. And I think we're not even close to the peak, but there is going to be a peak and there's going to be this trough of disillusionment, which will probably take us right around 2040.

22:41And I think the reason that we're going to have this drop is people are going to realize that it is hard to make these things economically viable. At that scale, at that scale, at a billion scale, at a scale of a million, at a scale of 100 ,000, different story, different use cases. It will make sense. When we come back, Raf will take us behind the scenes at his autonomous drone company, Verity.

23:24Meet Nicole Nicholas, Capital One business customer and co-owner of Ansett Uncles, a plant-based restaurant and community space in Brooklyn, New York, that got its start from a need for unity. The inspiration, it was born from the desire to create a space that felt like home, where we can connect community culture, good food, and come together with family and friends. That's how we birthed aunts and uncles. Nicole and her husband, Mike, were fulfilling their dream of bringing people together out of their home kitchen. But they soon learned that the demand for community was greater than they knew.

23:55It became overwhelming and we were like, we need home, but not in our actual home. We realized that there was also a need in our community for something bigger in our neighborhood. So we had to find a place. Moving from a home operation into a storefront was a huge next step, but Nicole and Mike were able to take it on with the help of Capital One Business. It's not for the weak. As a small business, finding resources is super important because that's the way you'll be able to manage and scale. We would have never done that without having Capital One to be able to help us along the way. The cashback rewards are very helpful.