Why humanoid robot demos still fail the generalization test

0:00Closing the loop of autonomy takes more than a great robot. Wybotic provides reliable wireless charging and fleet energy management for autonomous robots and other battery-powered systems. This patented technology gives robots more freedom to charge throughout their normal workflow, helping increase the uptime, ROI, and safety of your fleet. AMRs, drones, floor scrubbers, humanoids, and everything in between, Wybotic makes charging simple, seamless, and scalable. Learn more at Wybotic.com slash Robot Report.

0:41What's up, everybody? Welcome in episode 265 of the Robot Report podcast. I'm Steve Crow, executive editor of the Robot Report. Thank you, as always, for tuning into the show. Hope everybody's having a great week. Joining me today is Eugene Dimitri, Editorial Director of The Robot Report Gina, when was the last time you and I did this together? It's been a minute It's been a minute, although just for those of you in the audience who might not remember We actually were the two who started this That's right, baby Six years ago The OG's back together again Well, it's good to be here with you You're fresh off a visit to Berkshire Gray today How was that?

1:19It was great You know, I always love going out into the field and seeing the robots where they're made, where they're used. And so, big thanks to CEO Dave Sarator, who I interviewed recently, and the company for showing me around. I got to see their headquarters in Bedford, Massachusetts. And I got to see the systems that Structure Gray is working on. everything from ticking, from auto store, automated storage and retrieval systems, to truck and loading, to ticking. It was really cool. I feel like we haven't heard from them. Because they're private now, is that correct? Well, I'll get into that.

2:04What's interesting is when I talked to Dave maybe a month ago, I guilt tripped him. It's like, oh, you guys have been under the radar. And it's like, well, we've been hitting down. We've been working. We're not. They only recently hired a new PR team, and they're getting back out into the world. But to their credit, they've been doing the work in the meantime of refining their systems, getting real customers. And I saw versions of their systems that are definitely advanced based on the feedback they've had from real-world deployment. So I thought that was great. I got hints at things that they're working on, which I'll share with all of you out in the world as soon as I can.

2:48And yeah, they've been keeping a low profile, but I think that's going to change. And again, much thanks to them for giving me access. Yeah, that's awesome. I was wondering where they, when you said you were going down there today, I'm like, oh, they must be coming out with something. So that's fun. And then you were also at Locus, what, last week? Last week, yeah. It's been a busy couple months. So thanks to Rick Falk and the team at Locus Robotics for hosting me. They've got 8 billion plus six that they've assisted with. And it was great to see all of their Locus Origin and Vector robots being paired for the holiday rush.

3:29So they had hundreds of robots that they're shipping out actually daily now to help their customers flex capacity. And that was really interesting. and I got to see the new ripper that's going on to Locus Array, their mobile manipulator. Nice. Yeah, I've still yet to see that operate live. I saw it whatever trade show they introduced it at last year, but I've yet to see it operate, so that's a little jealous. That's awesome. Yeah, well, next time you'll have to come with me. Yeah, I know, right? And speaking of Rick Falk, the best in the business, he'll be at RoboBusiness here, one of our robotics shows, in a couple of weeks on a panel about robots as a service, and nobody better than that.

4:13So make sure if you're coming to RoboBusiness or if you want to get involved, still time to do that. But Rick will be there from Locus Robotics. But we have a great show for you today. We're going to play an interview later on with Janine Sinan Singh. She's the director of Gen.A.I. Research at Appen. You and Mike Oitzman did that interview. Gene, why don't you just preview that interview for us? Yeah, you know, we hear a lot about AI, but what's really interesting is to hear how it's actually being used. So we learned from Janine why high quality video and human judgment still matter in robotics training and also how synthetic data can help teams scale.

4:54we also touched on sort of the controversial topic of why a lot of these impressive demos you might have seen on YouTube or elsewhere don't necessarily prove that robots can generalize, right? A lot of these are really hard to prove that a robot is actually learning, is actually able to transfer knowledge and so we talked about that, we talked about AI agents safety and again the whole challenge of getting robots out of the lab into the real world. Yep. Big challenge. Major challenge. So we look forward to that. We'll also recap some of the more interesting stories of the week, which we do every show.

5:34But before we do that, just want to tell you about RoboBusiness, which is coming up quickly, Gene, 13 days away, which is crazy. October 20th and 21st out in Santa Clara, California. This is our 20th anniversary show. We're aiming for over 2 ,000 people, 100 exhibitors on the show floor, talking all things scaling, robotics deployments, scaling your robotics business, and everything in between. We have some amazing speakers on our stages over the course of the two days, kicking things off with the Future of Robotics panel that I'll be hosting with Rodney Brooks, Ken Goldberg, and Jonathan Hurst.

6:11That should be fun. Eugene, you're hosting a panel at the end of the first day on AI. It has Teradyne Robotics, Amazon Robotics, Collaborative Robotics, ABB, and Cambridge Consultants. And like I've said on this show a number of times, Gene, what we're trying to do with RoboBusiness is really focus on people who have deployed or are deploying robots at scale and how they're looking at or using these new technologies. So we'll be exploring AI from a number of different views, but not necessarily how these new frontier labs are using AI. It's how are established, very well-respected, powerful robotics companies, how are they using AI when deploying these things to help the users interface with them more to create these systems?

6:57So that should be a fun conversation. We'll also be doing on day two a State of Humanoids keynote with Agility Robotics, Aptronic, Persona AI, and Psionic. And then we'll be closing out the show with Morgan Pope. He's formerly of Disney. He was at the Robotics Summit, I don't know, three or four years ago, Gene, demoing the BDX Droid. He's now over at Familiar Machines and Magic, which is a new startup from Colin Engel. They're building some interesting robotic characters that they're looking to deploy in people's homes in the near future. So he's going to be closing out the show, talking about the keys to engineering robotic characters, which would be amazing as well.

7:34But head to robobusiness.com. You can see all the speakers, all the sponsors on the show floor. you can grab your tickets and again for our podcast listeners make sure you stay tuned to the end of the show where we'll share with you a special discount code that's exclusively for you guys who've been along for the ride here since again eugene and i launched the show a number of years ago so we want to say a special thanks help you guys save a few bucks on your robo business tickets but again we hope to see everybody out at robo business in santa clara california October 20th through the 21st for our 20th anniversary show.

8:09All right, on to the news of the week, and we have you covered with the latest news and information of the Global Robotics Network with our websites, The Robot Report and Automated Warehouse. We'll start with this one. Maybe the story of the week, Gene. Boston Dynamics has now found a permanent CEO. It hired Rohit Prasad. He takes over for Amanda McMaster, I believe, who was the interim CEO. And of course, all of this is a result of Robert Plater, who stepped down a number of months ago. But prior to joining Boston Dynamics, Prasad was a senior VP and head scientist for Alexa, for Amazon Alexa and the artificial general intelligence over at Amazon.

8:49So he's obviously been instrumental in building out Alexa from its earliest days into really turning it into a consumer AI service that's used in hundreds of millions of devices in households and other devices around the world. What was your sort of takeaway? You know, I think immediately comes to mind why hire an AI executive and not a career roboticist, right? Yeah, I think it's interesting that Lost and Dynamics took that direction. and it sort of shows the direction of the entire industry, I think, in that everybody understands that you have to have hardware and software for robots to be successful.

9:28But I think part of it is in Boston Dynamics' case, they've already proved the hardware side, right? They've already been demonstrating for years now that they have some of the most advanced humanoids on the planet that are incredibly agile, especially with Atlas 2, which is the electric model uh it's continuing to improve was it last week we just talked about a new gripper and uh so now in order to get from uh solid technology they have to productize that to commercialize it into something that can move at scale which obviously amazon has a lot of experience doing sure yeah i was a little skeptical of this at first right again not many robotics companies hire outside of robotics, I don't think.

10:16I mean, there's certainly, I mean, we talked about Locust, that's a good one, right? They've obviously, they hired and started with a lot of logistics folks, and that's worked out pretty well for them. But as I've thought about this more, I think it actually makes all the sense in the world, right? To your point, I still believe there's so many people that like to poo-poo Boston Dynamics and say that they've fallen behind in the humanoid race and all this stuff. I don't, I think that's farthest from the truth, right? They are so unbelievably great at engineering humanoid robots. It's insane. Right.

10:49And deploying these things. It's way too soon to tell, like when the robots are out in the world doing things, then we can talk about who's winning or losing. So to say, Oh, they've got this really highly engineered product and somebody else has a people product that can do a few dance moves. That's like, it's apples and oranges. they both have two arms and two legs and a head. But you want something that's capable. And I'm really interested in what's going to happen in something months or a year when all these trials that they're quietly conducting, okay, who's going to come out of that with real knowledge?

11:22Who's going to come out of that with something that they can actually put onto the market? And with Sundaei as a parent, that makes a big difference because clearly they want to enter the market as a manufacturer, as a provider of robots, It's not just as a user. Yeah, and they have, so Hyundai brings a lot of expertise, obviously, on the manufacturing side of things, right, along with a lot of other aspects of this, right? They have the hardware engineering already at Boston Dynamics. Like you said before, that's proven out. They have changed how they build these robots from a software standpoint instead of using a lot of the classical techniques they have in the last number of years, pivoted to all of this new AI stuff, right?

12:04I think that's obviously why he's here. So they have that whole feedback loop now. They have Hyundai. They have the meta plant up, running, ready to go. Do they have anybody there that understands how to build out the AI infrastructure for a product? I don't know if they do. Well, now they have arguably one of the best people in the world, you would think, to do this, right? Alexa is one of the most well-known and used and deployed AI products on the planet, and he helped build this. So I think the promise for me for humanoids has always been the ability to multitask. And I don't think any of them have proven they're able to do that yet.

12:44And I do think it's an intelligence problem at this point, especially for a company like Boston Dynamics that has such a hardware-capable product. I think in order to deliver on that promise, they need to deliver on the brain side of things, on the software, the AI side of things. And I think, again, look at Figure. They have all this stuff that they're doing on AI. Tesla, obviously, they have all this AI infrastructure. Agility Robotics is using a lot of AI. There's so many interesting parts of this story. Boston Dynamics, was it last year, announced a partnership with Google DeepMind where they're using some of the Gemini stuff on Atlas and some other robots at Boston Dynamics.

13:30People are thinking that Google is going to build its own humanoid. And to me, that makes a lot of sense as well. So what happens there? All these humanoid companies that are relying on Google DeepMind for certain things, is that a good thing? I would say it's probably not a good thing, right? I think we're going to see a shakeout. I think we're going to see whether it's licensing AI or partnerships between the big gen AI and advanced AI companies like Meta and Google and others. And then the humanoid developers who need that intelligence, like you said, to be multifunction, to be able to learn and adapt on the fly, which is the promise of artificial general intelligence.

14:14It's a promise of general purpose robotics and humanoids in particular. And so I don't really know how they're going to cross the threshold. It'll be interesting to see. I suspect we'll see some dance partners change over time. I suspect we'll see a lot more deals like this in the coming months because think of all the effort. It's taken 20 years to get Atlas to where it is, which is impressive. Why would Boston Dynamics start from scratch and try to build all the AI too, right? They have some AI of their own, to be sure, but I think it's going to be a lot of partnerships and especially for a lot of their rivals, the Teslas and the figures out there, I think they're also going to be figuring out, okay, where do we get data?

15:01Who do we partner with? In the much the same way that they've all partnered with potential end users, right? They need to complete that loop and I think it'll be really interesting to watch. Yeah, so we'll watch that. And it also is interesting. I know you wanted to talk about this, about the connection with the robotics and AI Institute. You know, that's all that was and it still is under the Hyundai umbrella. We reported that SoftBank is in the works to buy the robotics and AI Institute from Hyundai. But what connection? Why is this interesting to the story as well? Well, one, there's the question of what's the ongoing relationship going to be between the Robotics and AI Institute, which is now going to be part of SoftBank, and FOS Dynamics, which is Riff and Die, right?

15:48They're no longer under the same roof, necessarily. They may still have some informal relationships, but it is interesting that this technology might end up being licensed to other players, or the RAI's knowledge might be used by other customers. and so on the product side I think it's really interesting and going back to Berkshire Gray for a moment their CEO Dave Baratoy told me that not having to worry about raising constant rounds of venture capital has helped Berkshire Gray focus on its customer needs and getting into the market refocusing on building the right product for the right problems and I think that's a commendable approach because a lot of startups, they spend so much time hustling that the leadership has to take its eye off the ball of, are we getting off the market?

16:42Are we solving these problems? And I would say, compare it to what we saw with maybe robotics spinning out or RealSense spinning out from Intel, that there is an opportunity to be a little more independent with the RAI spin out, or rather, changing hands going to SoftBank. And SoftBank has made some pretty serious moves now in the robotics space in terms of looking at, okay, they're supporting, they're buying up companies that have a real customer focus rather than what they did years ago, which was more trying out a lot of different new technologies. Yeah, I'm not sure the REI has a customer focus at this point.

17:26No, no. Which I think is why I think all of this makes sense, right? So again, our AI has very talented AI folks and robotics folks, right? But they're mainly focusing on research projects. If you see some of the things that have come out from them, incredible stuff, but not necessarily being applied to robots that are being used in the real world by a large number of customers. So Boston Dynamics already had an AI partner of sorts with them and with Google DeepMind. But I think they obviously, Hyundai is pressuring them to deploy these things and get Atlas and get more robots out into the world.

18:05So they need to start having all of this stuff operating under the Boston Dynamics umbrella, I think. Right. So bring it all in-house and, you know, we'll find out. But hopefully their new CEO is the guy to help them do that. So, again, we'll try to get Rohit onto the podcast whenever he's available. I'm sure he's going to be quite busy for some time getting his feet wet over at Boston Dynamics. But Rohit Prasad is the new CEO of Boston Dynamics. The next interesting story that we wanted to talk about, did you see this, Gene? This is one of the craziest things. I saw this when it happened. There's this robot entertainment company, Rek, and they put on humanoid fights.

18:49It's usually two humanoids against each other that are teleoperated by humans right outside the ring. And it's fun. It's entertaining. We've tried we actually tried to get one for RoboBusiness. It just didn't work out. But recently they kind of took it to a next level where they put a human influencer in a cage against a 200 pound modified T-800S humanoid robot from Engine AI. And I don't know if you've ever seen these engine AI humanoids in person. I've seen them in person a couple of times. I saw it. The first time I saw it was at CES. And they're big. They're big. They're fast. They can move.

19:27Yeah, I would not. You can pay me to get into the ring with one of those or with motion lots. But when I saw that, the first thing that came to mind is that's a lawsuit waiting to happen. And then my other concern is, like, for the safety of the human. And so we know that California State Athletic Commission issued rec a cease and desist. Not that they explicitly ban even robot fights, but it's, you know, what do the bouts fall under? I mean, we've seen robot races, robots completing marathons. We've seen robots, you know, I play ping pong and pickleball against robots. You've probably done the same.

20:08That's one thing. But like contact sports like boxing. And the whole safety issue around humanoids already is an issue. But when I saw that, I'm like, oh, this could go wrong in so many ways. I mean, even if the robot is set in such a way that it's not going to activate it, what if it breaks? What if it falls on them? I mean, there's so many things that can go wrong. It's a great publicity stunt. But it brings, to me, and not all of you out there might agree with me, but to me it brings all the wrong attention to robotics. It only reinforces the narrative that robots are going to steal our jobs and that they're going to hurt us and that it's AI and robots at the end of the world.

20:51And I don't buy into that, but it makes it harder to make that argument when this is a video that everybody sees. I remember pulling you and I, this was at our robotics summit this past spring. You and I were walking around the show floor and I forget what what humanoid company it was, but we were standing pretty close to one of them. and it was large and it was doing some sort of dance routine and i remember pulling you back away from it and i was like man this thing just seeing it in person it was a lot taller a lot thicker a lot heavier than i thought and it's like man if this thing ever malfunctions it's crazy right so don't write me it's a bunch of human i mean that to me that's like really asking for trouble yeah so i i did some research on this i i don't i don't know i thought i had it in my notes here so i might not get the numbers exactly right but there is a stat about how much force this t800 robot could create right if it throws a kick i think it was 800 pounds of force is what it could could create right now i looked up like think about who are some of the best all-time boxers ever right so i just looked up mike tyson to try to compare the force that this humanoid could create compared to mike tyson back in his heyday and when he was in his heyday he was the best fighter on the planet and he created his his punches somewhere in between 1200 and 1400 pounds of force i believe it was so not you know this humanoid robot's not quite as powerful as mike tyson but not that far off and it's you know it only takes 12 pounds of force to break someone's collarbone or their nose yeah it's crazy i think so yeah even against a trained boxer who's bulked up and ready to do it um there's so many questions that we have just yeah glancing at this video that's like okay in this case people might accuse california's government of overreach but in this case i think it did the right thing because um as i'm publicly so that's one thing but it's putting people's necks on the line uh i'm not a fan yeah so it's again to your point it's not about whether or not you know humans versus robots are banned or not they're not um it's about combat sports they already fall under existing laws so the the state of california is saying that they did not get approval or a license for this combat sport match right and and that's ultimately but yeah i think it brings up all sorts of other debates and you can watch the video the videos out there you know you won't you would never you probably couldn't pay me enough to be in the well maybe maybe that's not true but i think the difference is the these are being tele-opt by humans outside the cage so they're not nearly as skilled i mentioned mike tyson they're not nearly as skilled as an actual fighter you know what i'm saying that they have half the feedback do they have do they know do they have a proper sense of how hard you're hitting their opponent do they have So, you know, as great as MoCap and Tele-Op have gotten, again, I still have some questions about that.

23:55You know, if we're going to regulate someone tele-operating a surgical robot, then we should do the same for sports. And you know what? I would not be surprised if somewhere down the line we have American Gladiator 2030 edition with humans and robots going at it. But hopefully by then we'll have worked out all the proper safety requirements. Yeah. In boxing, I'm not a boxing expert, but I know the boxing industry is very corrupt. Right. Promoters. And what was that guy's name? Don? Is it Don King? Was that his name? Anyway. But I think that's maybe what's going on here is they're just trying to make sure that any any interesting new sort of combat sports fall under their umbrella.

24:42They're aware of it. They make money from it. Right. But I've reached out to rec to try to learn more about this. The founder of rec has kind of poo pooed this on one of his social media channels. You can find that, which is interesting, which is also, I don't think, a smart idea on his part. But yeah, interesting nonetheless. Humans fighting humanoids. Look out for that. Final story. And this is a big one, Gene. So take us through this one. But Schneider Electric is looking to acquire PTC for just under$23 billion. Yeah. So Schneider Electric, which is better known for global energy company, right?

25:24And PTC, which is known for industrial design, engineering data management software. The deal is essentially$22.6 billion, which is a lot of money. It might be a little bit less than the company has worked in the past, but still a big, big deal. We talked with Matthew Sulzak of Interact Analysis, who said that this might not only help Schneider bridge the gap from QTC's design prowess to its own operational expertise, but it's been also positioned Schneider to compete with industrial automation and software giants like Siemens and DeSauce Systems. So there's a play going on here. And the other thing that's noteworthy is that QTC and Schneider Electric are Boston-based.

26:16Gissot has an office here at Waltham. There's a lot of activity going on, going back to that AI and robotics. But as everyone's trying to explore how to best combine these, one is the play. And in the case of Spider Electric, they want to have what they call the end-to-end digital thread through the product lifecycle from inception and design all the way through operations and maintenance. And I think that's an interesting play that they're doing. But when we see deals like that, they'll probably take years to really unfold and affect the industry. but I also think it's like telling that these companies, which in the past are like, they all could stand alone.

27:03They're all pretty big companies. Even they are feeling the heat from AI and they're like, okay, how do we partner up or how do we acquire in order to do these things? Yeah. Is this going to close this year? Did they say? They're still waiting on the customary regulatory approvals. And I think they already had shareholder approvals, but yeah. And something that's in the works, I mean, they wouldn't even announce it if they had not already done their due diligence. So it's more a matter of, by that point, I think the boards have already voted, but it's more a matter of like, okay, we know this is in the works.

27:34It may take another year to really fully integrate their products, but they are complimentary. And that's the other thing I think is interesting to note is that in the AI space, from the outside, it looks like, oh, it's all software. Well, there's lots of different things that they already have a customer base Or in the case of Schneider Electric, they have a global customer base. That's industrial. So if you're running a power plant, what robots or AI might be able to help you? Well, this is going to help keep you some scale to help that market. And I think that's going to be really interesting to see how that plays out.

28:11Yep. So we'll stay tuned on that. We'll keep everything else on our radar. And again, we have you covered with all the latest news and information about the Global Robotics Network with our websites, The Robot Report and Automated Warehouse. Make sure you follow us on social media channels. If you're not signed up for our weekly newsletters, make sure you do that. There's no better way to stay up to date than by doing that. All right. We're going to go on to the featured interview now. And again, we have Janine Sinan Singh on the podcast. She's the Gen AI Research Lead at Appen. Set this up for us again.

28:45So Janine helps model builders develop and evaluate AI systems. Her work spans expert data for large language models, multimodal AI, agentic AI, and robotics. So she's really looking at how human insight can support quality and real-world performance. So let's listen to what she has to say.

29:09All right, today we're joined by Janine Sinan Singh. She's the director of Gen AI Research at Appen. Janine, thanks for being here. Thanks, Mike, for having me. Yeah, so we're excited to talk to you. I always love talking about AI and how the world is so rapidly changing. And so to start, why don't you tell us a little bit about Appen and the role that the company plays in the AI ecosystem and your role there. Yeah, absolutely. Appen is, I love sharing this history with Appen, having been in the space for a few years. 30 years in celebrating its 30th birthday this year. So seeing all the changes, yep, it started out by some PhD linguists 30 years ago who were, you know, annoyed that they did not, you know, models didn't understand, models at the time didn't understand their Australian accents.

29:57So very big in the linguistic space. A lot of speech work, interesting work from high to low resource languages. And of course, in 2022, the big year that changed the world, have to pivot into more Gen AI work, supporting LLM development. Now it's agents, now it's robotics. And we're providing the human data, the expert data layer to train and evaluate models across the gambit from the large language models, tech space to multi-modality, as well as agents. And then again, moving into the robotics space as well. And then in terms of my role at Appen, I support the model builders, again, wherever they sit in that spectrum.

30:42So it's a fun space for me to touch a lot of different areas. And we design together with them as an extension of their research team in the data layer. So we support on designing custom evaluation suites, as well as collecting or determining what types of annotations are needed for training data that will help them move their models forward. Well, that's very interesting. And you just, you know, remember that our audience is roboticists, researchers, machine builders, folks that are, you know, designing and building, you know, the latest in robotics technology today. So I think it's very interesting how you guys are supporting that development.

31:20And that's what we want to dive into here. So let's jump right into it. And tell us about your take on the latest use of generative AI and some of the humanoid demo systems or robot demos that you've seen and how you see it from your perspective as to how things are evolving in the world of robotics today. Yeah, it's honestly fascinating to see. I mean, there's, of course, a lot of the toy demos that you tend to see a lot of, you know, the dancing robots waving high. And you ask yourself, you know, this is cool, but what's the point? But, you know, there's still that's like the step towards understanding physical movement, manipulation and whatnot.

32:01Balance is important when you think about humanoid robots. And it's really, I've been seeing a lot more lately, you know, in terms of research published and like demos being shown where the connection between the generative AI and LLM movement and robotics is becoming really clear at this visual reasoning understanding layer because I've seen some demos now. Now, granted, the latency is really slow, but it's incredible if you plug in like a Gemini 3.7 Flash or Opus 5 into a robotic system without any fine tuning and it's able to complete some basic tasks. It might take seven minutes for it to move an object from here to here, but you didn't have to train it on your own model with a bunch of millions of hours of demo videos and whatnot.

32:47So it's incredible to see that this kind of multimodal training that's been happening for large language models and generative AI is actually seems to be like a strong way to translate into thinking about approaching robotics from like a foundation that maybe you can advance faster off of. But then it's also the trade-off, you know, the roboticists who are actually training their own models are going to have that latency advantage, which is more critical than ever before when you talk about something interacting in the physical world. Yeah, we definitely want to dive into the multimodal aspect here in the conversation with you this morning.

33:24But before we do that, just staying at the high level again, what do you think some of these humanoid robots, again, from the output perspective, Gene and I and the rest of the editors talk about it all the time. I mean, it's like, why? Why did we do this? What's the demo actually showing us? What do you think some of these early robotic humanoid products are getting right or wrong with some of the demos that they're showing in terms of the advancement of the underlying AI, you know, capabilities and state of that world as you see it? Yep. Yep. I mean, it's definitely, it's still facing the same problems that large language models face where I don't know that it's truly demonstrating generalization yet.

34:04You know, it's like, I can fold laundry. How many hours of folding laundry videos did it take to get it to the state that it's at? And like, can it, you know, translate that action to something that is tangentially related and out of distribution? So I think that's, you know, the struggle in the space, everyone has their cool demo of the robot doing something very specific because obviously it's been trained to do that over, um, you know, very targeted, high quality demo videos around that exact subject. Um, and it's really, you know, I think it's still to be seen about generalizing, you know, you hear a few stories here and there about like a zero shot capability that came up, but, you know, I'm, I may be, uh, I don't want to say cynical person, but I questioned a lot.

34:48And I, you know, I'm like, great, but I'm like very curious, like how out of distribution that zero shot thing you just showed in your demo video actually is. Yeah. So it's great. It's okay to be skeptical and it's okay to be pragmatic about it all. But since you brought up the topic of video, let's just touch on that for a minute. We'll come back to it. I'm sure later in the conversation, you know, how is video being used today, you know, to train these robots? And is there more in the future, you know, on the edges as things are advancing from the use of video? Yeah. So, you know, from the Appen side, of course, and in the market, you see a massive demand this year that I think is an analog to the pre-training days of, you know, LLMs.

35:33The difference is that this type of information isn't just available on the Internet. Everybody knows this. There's a few open source data sets that are coming up here and there, but the type of video data that is needed for pre-training needs to be acquired at such massive scale. You can't just expect YouTube to deliver exactly what you need. The egocentric videos being the most important, the most valuable right now. And there's, of course, interest in getting those in different types of settings because there's been large data sets that can be acquired with, you know, human with cell phone mounted to the head, easy to scale.

36:14You get a lot of similar type of homework, type of duties, but there's also a strong interest in, and it's, you know, more challenging to collect, but it's definitely offers some different and important value in terms of in business settings. So like logistics and whatnot, and, you know, trying to go after what I, And, you know, we probably know Amazon has millions of hours and incredible experience collecting from their massive logistics and distribution warehouses. So I think there's some idea of like catching up to that through collecting these egocentric videos. Janine, speaking of catching up, I think we first covered Appen in 2019 or so.

36:56and at that time I think we were talking about annotation of scenes for autonomous driving and for hospital robots. Can you tell us a little bit about what's changed in the time since then? Because you mentioned obviously the models have advanced, the data collection methods have advanced. How has general AI research evolved and what should robotics developers be paying most attention to? So because of the underlying tools advancing so much, you know, there's still always this need for raw data collection, but there's so much more you can do with a smaller set of data, given the tools that are available.

37:38Because with generative AI, you can create better simulations that scale up a few examples that you found. So, you know, you find a few, you know, edge cases, out of distribution examples, and then you can generate some more videos off of that using, you know, generative AI, for instance. You can even use it to help with some level of like labeling and analyzing for quality data. There's still a need for human involvement for the long tail and also to identify and collect those edge cases. But it's amazing if you use all the tools available, you can get a lot more mileage out of less. And I think that's honestly driving the startup ecosystem in robotics.

38:20You know, you can spend less than$1 ,000 now to build your own robotic arm. And the same thing can be said in a sense for the data needed to put into training. You can, you know, utilize some stuff out there and then use generative AI in a way that you would have to, in the past, like 2019, actually hire humans to do all of this work at massive scale. When we're talking about the data, and it is interesting that we're able to train with smaller or more targeted data sets than before. Can you explain for a moment what makes robotics training data different than general LLM? I think everyone has played with ChatGPT and all of the text-based ones or the internet-based ones.

39:05But when we're talking about video, and we'll get into the multimodal in a second, but there's a lot more when we're dealing with the physical world. Exactly. Yeah. And there's that action layer. And I wouldn't even say intent layer. You know, you see a video of someone reaching for something, you don't know why. And the why is a really important thing to, in order to create a world model, which would be like the gold standard of generalization for robotics. if it actually understood the world and could, you know, generalize actions to behaviors and outcomes, which physical models, I think, in a way are a little easier to grasp because they are rule-based to some extent.

39:46But when you create like a future robotic task would be something that is like LLMs have gotten to the point where you ask it an open-ended thing and you actually want to see what direction it takes. We're nowhere close to that with robotics. You wouldn't want to, you know, have it try to put a square peg and a square hole any other way per se. But imagine if, you know, in the future when you can give it a more open ended situation to deal with, it would need to understand more than just the physical rules of how things behave when you interact with them. would have to understand intent and planning and all of those stages that we're, you know, seeing in the agentic world in the digital landscape.

40:30But there's so many other factors that come into play when you are operating in the physical plane. But it's, you know, a lot of the agentic thought patterns that are being explored now for like long horizon tasks, as well as, you know, planning and all the other steps in between have an analog in the physical world. But it is that and that makes it challenging from a training data perspective, because even for agents, it's a challenge to train them towards, you know, long horizon tasks, thinking over a longer period of time or taking actions and planning and then recovering from failures. those same types of challenges and agents are going to be hit when it comes to robotics.

41:15It's just we're at the early stages, like the chatbot level, where it's like input-output is where we're at now. And the data will catch up once that kind of baseline is established. But I would say that there's so much opportunity as the capabilities develop further in robotics. Well, and I think it's interesting that you mentioned obviously the cognition and action aspects. There's continuous stream and it's not simply, oh, we need to recognize an image or we need to observe. Again, it's different because you have physics involved, you have a dynamic environment in certain cases. but when we're talking about the data collection and annotation, you mentioned obviously more that's becoming more automated.

42:05How should robotics development teams think about labeling or evaluating images and video and LiDAR and tactile and all these other sensor streams? How should they be either weighting it or trying to collect it? Because right now, a lot is obviously being put on the video side of things. But, you know, we've spoken with a bunch of companies lately that are trying to collect multimodal data to better build that, not necessarily world model, but the steps towards the world model where they have a better comprehension of the environment. them. Exactly. Yeah. And I think for robotics, it's a multimodal is almost multi-sensory and that we're trying to, you know, it's the feedback loop that is a little bit more and is very sensitive to latency.

42:55You know, once you miss the feedback, it's gone in the physical landscape. So I think the, one of the big challenges I think for robotics more so than Gen AI is that time factor because you lose a lot of that signal that should be used as feedback if you're not, you know, it's not coming in fast enough and is not being processed fast enough. So that is a, you know, that's an area that I think is really interesting for robotics. You talked about the egocentric data collection and at the same time we just talked about automating some of that judgment of the multimodal data. but where is human judgment still important is it in collecting edge cases is it in looking at successful tasks or evaluating failures is it about validating the robot behavior because we think of the human largely at the egocentric data collection stage where they've got something strapped to their forehead or they're wearing gloves or there's a camera watching them conduct have to task like folding laundry or doing something else but uh where do we still need that human in the loop in order to build those robot skills and behaviors that are we're looking for yeah i would argue that a human in the loop is necessary like every one of those stages in the chain like you mentioned it's just that with the again with the tools we have available you can make uh get higher value more mileage out of those human in the loop interactions so for instance you have a bunch of robot demonstrations of robot completing tasks.

44:34At first, in order to train a VLM judge to tell you if it's performing the task well or not, you still need that human labeling because it doesn't have some sort of ground truth to base on. And the thing is that the VLMs in this specific example, for instance, can be very good at identifying certain temporal issues, but humans are much better at that longer horizon. Like let's say the start and end goal were accomplished, but along the way, it's taking forever, struggling, making mistakes. Like a human's going to detect that way faster than a judge model. So human in the loop for each of these phases is necessary because quality at the end of the day is the most important thing when it comes to the data being introduced.

45:26And you sacrifice quality to some extent when you go straight into a synthetic pipeline, you go straight into 100 % judge model making all the decisions. You have to start with a strong human basis. And then I've been finding in some of my other work that there's always value in having a sense of where the judge is not able to make good decisions and you need to plug a human in the loop in so that you can have that efficiency while also maintaining quality along the way. So from, you know, even from the data collection phase, which again, since we're creating data to be used, which arguably could be also simulated at some point, you need a human to do that collection, but then you can potentially simulate off of that.

46:13And then at the, you know, behavior labeling phase where, you know, what are you doing? What are you trying to accomplish? What are the steps along the way? What are the actions? very helpful to have a human outline all of that over, you know, a certain set of videos or other behaviors. And then again, expand out with the models and synthetic strategies we have available. And then, you know, down the line at each phase, there's an opportunity for a human in the loop throughout. And then on the macro level, of course, identifying where out of distribution is, edge cases, all of those rare events that you need to go out and collect, create, or observe.

46:53Only a human can do that. You mentioned the importance of data quality. What are some of the dangers of trying to take shortcuts? Because, again, when we talk about the different modes of sensing or perception, and video is well established, but perhaps people will say, oh, I don't need this or that angle or this or that. haften gain or whatever, what could that mean either for the work down the line or for even the robot behavior itself? Yes, the lower quality videos, I think people see the impact pretty quickly when it comes, especially if they have a reasonable evaluation type of harness set up or whatnot.

47:36And that's like the constant cycle of training and then evaluation that's important. So it is possible, though, that let's say you have a very low quality data and your evaluation set doesn't actually cover those situations, then then it might come into play later down the road. And that's also why it's really important to design your evaluations around many different behaviors just so that you can identify those types of issues early on. Because otherwise, you might end up potentially having some wild behavior show up much later in the game. And then tracing it down becomes much more painful.

48:12I think that's an interesting topic in the concept of how you're helping your clients to create quality data that's going to become usable. So, you know, what kinds of benchmarks have you sort of found? What sort of best practices have you found on that side of it that are useful for robotics across all the other use cases you've had for AI over the past 20 years? and where do those benchmarks maybe fail to capture real world complexity? Yeah, you can only measure what you set out to look for. And sometimes, you know, our best efforts you find out later, sometimes through random paths, random walk, you'll realize, oh, that was actually not great.

48:57And we didn't even think about that potential, you know, poor quality situation. But that's where experience comes into play. So, you know, Appin having done data collections of this sort for many years and scaling workforces to do this, understanding what the failure modes look like is immense value for determining what is high versus low quality. Because if that list isn't exhaustive, then it's very easy to miss things. But as you see the types of failures that happen when you have like a, let's say, egocentric video data collection, you know, there are obvious ones that everyone can think of.

49:33Like, you know, the camera angle is not the correct camera angle and the hands are not in view. Other things like that, that we can apply quality assurance processes that are automated on the video for like object detection. you know, the framing of the camera, for instance, that for the types of scale data collections that are happening to produce these egocentric videos, you have to have some sort of automated QA processes in place. It's impossible to review everything. But then there's also the human behavior side where, again, having some experience with having training and coordinating and understanding humans and their work because it comes down to them.

50:17You know, a bad collector is generally going to stay a bad collector and it's best to remove them early rather than let them continue. So again, on the human side of things, having processes around the actual people being involved and constantly monitoring and curating and keeping that kind of workforce in line with the requirements and quality procedures is the best preventative measure. But in terms of making sure that the quality is there, especially for this pre-training data where there's a little bit more flexibility around what the quality fuzzy line is. But as we get further along and past this massive bulk pre-training stage with robotics, the quality bar is just gonna keep going higher because then you will start to see you know, those failures coming through a lot faster as it tunes into specific actions and vision language action protocols.

51:19Yeah. I mean, we're still very much back at the age of mechanical Turks, what we had for image labeling back in the day. You know, is this a cat? Is this a dog? Is this a squirrel kind of thing, but for the operational data.

51:37And But, you know, simulation is becoming a part of the workflow as well. And so, you know, again, back from your experience, speaking from your experience from Appen, you know, to the audience who may be still beginning to do all of their harnessing here for how they're going to pull this together. How do you balance automated testing, simulation, as you just said, the human evaluation or field of validation? What are some of the best practices that you can share on that side of it? That is a great, tough question.

52:10In terms of best practices, I'm a big believer. I've been a startup founder in my past. It's about failing fast. So you want to, you know, trying things at a smaller scale, seeing what works, and then building off of that. And it's also a lot easier to have a lot more control over what goes in when it's a scale that's manageable to a human, and then figuring out what works based on more rapid evaluation and then training loops, keeping that loop tight in the beginning. And then as you see success off of certain things, then scaling up. But it's also important when it comes to anything computing related, which this is, to understand that scaling is another problem of its own.

52:56So you can't forever stay in this tight little feedback loop of like train small set, eval and scale up by just doing more loops, you have to actually scale up in terms of quantity at some point. So being targeted and seeing what happens when you do your simulation, because there's so many variables at play here. Some simulations are better than others. Choosing what to simulate against is another thing. But my approach from a fine-tuning perspective, I haven't trained a robot yet. That's still something I would like to do. But from fine-tuning and training large language models, it's helpful to understand what the kind of coverage gaps are in terms of data to behavior, and then using that simulation or synthetic aspect to scale up on examples that you need in an area that it's not performing as well.

53:55We talked about a lot of different flavors of AI with regard to robotics from general AI to embodied or physical AI to agentic AI. Can you talk a little bit about the relationship between the general AI and the agentic AI that you were talking about a moment ago? Yeah, absolutely. So, you know, the embodied physical or even humanoid robots, I think, are really strong analogs to what we're going after with agents in the digital landscape. I think robots are like the physical manifestation of an agent and an agent's the gen AI version in the digital landscape. So we want agents to be autonomous, which is the scary part and interesting part at the same time.

54:44So unlike the generative AI, very much input-output, prompt-response. There's a direct action trigger to the next thing. Where robots are now, it's a little bit similar. It's like you do this task and then you complete it. Whereas agents, you want them to, in the background, be doing a goal-oriented behavior. You want them to be going after solving something that doesn't have a clear endpoint per se. in the ideal situation. There's still a lot of agents, I think, are largely being used to accomplish some more specific endpoint, but people are moving towards that goal-directed behavior. And that's where these long horizon types of tasks are coming in, where it's constantly doing work for you in the background.

55:33So it's like a, you know, it's a true agent. It's operating on your behalf, accomplishing some goal-directed thing autonomously. So I think that a lot of what we are learning and will learn from agentic AI should translate well into the robotics landscape because there's planning. There is, you know, thinking while not being observed in a sense. you know, they say agents dream to, you know, so there is all of this, you know, human-like behavior happening or desire to be happening behind the scenes for agents using these underlying models. What does it take to train or evaluate an AI agent that can complete a multi-step task or a long horizon task, as you said, and not just generate a response?

56:24Because I think that's something that, as you mentioned, emergent behavior, or you want the robot to be able to figure things out on the fly on its own, what does it take to get an AI agent to become that capable? Yes, the evaluation problem is still an ongoing, tough question that we're dealing with, you know, because there's this big trade-off with evaluation to evaluate something in these automated ways. You need it to be a bit deterministic and you needed to have a defined outcome that you can actually measure against in a repeatable way. You know, this is science at the end of the day. And the true thing we want to evaluate with an agent is something that is far more subjective and changing a lot more.

57:10So there's this, you know, this challenge of assessing the system when it comes to evaluation for agents. So we see some level of human-in-the-loop involvement around that. And then, of course, there's just breaking down the system into components to a certain extent to evaluate those components and see how it's performing against certain evaluations that we can actually measure. Um, and the big problem with, uh, you know, with agent evaluation is that you need an immense amount of context for it to be operating in a realistic environment. Um, in the digital world, you want an agent to be looking at your entire email and all of your Google drive documents, not just what it's been handed in the, again, input output scenario.

58:04So that has, um, we, We've seen, you know, RL gyms, environments, and those types of things being available in order to evaluate and train. And I really think that the, you know, a lot of analogs, in a sense, to the robotic space with the, you know, context-driven information. But in a way, the physical world is maybe a little more confined than the type of context an agent has to deal with online right now. Yeah, and we started seeing these AI or robot gyms starting to pop up, and they're interesting. Do you think, in terms of that safety element or that checking aspect of it, is there a role for multi-agent setups, whether it's multiple instances of an agent, several different robots in an area, or multiple agents running on a single device?

59:04Is it too soon for that, or is that one way of setting up these checks and balances? Oh yeah, there's definitely multi-agent setups in many places right now. And there's a lot of interesting work happening or things happening there. again, might be setting up my own in the next periods because you have multiple agents who are able to check each other in a way that helps you have more confidence around the safety. So you have one agent that's tasked with, you know, fixing the PRs to your code base, but another agent is actually like the testing agent that's like, you can't get past this bar unless you pass my test.

59:44And you can imagine like the only way to really have confidence in these systems is to have kind of multiple players. It's almost like having your own little business under the hood where you have your checks and balances encapsulated in different agents that are tasked with one specific goal. So I think that it's almost a necessity to have a multi-agent setup as opposed to just entrusting one agent to do everything. I love the analogy you've been working through this whole conversation about agentic use in the digital landscape and now physical agents in the real world. And the evolution that we're seeing happen now so quickly in the growth of AI to the deployment in the physical world.

1:00:31And one of the reasons why I think I'm still skeptical about the deployment ramp that many of these humanoid company CEOs are talking about. I'm not going to name names, but you know who they are, about having robots on the factory floor and in our homes in the next three to five years. And we still know that that's quite a long reach. And so I'd love to, again, to come back to your perspective about this underlying technology of the AI constraint. So whether it's agentic or not, is agent a part of that future? and if it is of robots in the wild, right, in our homes doing one task or the next task, whatever that may be, you know, how are those still failing today, you know, in the planning tool use, the memory, you know, between tasks that AI has to have or the robot brain has to have to understand, I just did this and now this is the next thing logically that I should be doing after this task.

1:01:34So just sort of bringing back to, you know, where the failure modes are today, the walls that are in front of us still that we have to get over before we can let all these machines to go in our real world. Yeah, exactly. I think there's a little bit of crossover. I have a similar skepticism about the reality and the value, I guess, of having a robot in every home and what's the real value there. Whereas I see a lot more immediate value for like the digital agent just because there is so much information online that you have to deal with. And it's an inhuman amount of information, in fact. It's somewhere where technology would have an advantage, whereas the physical plane, you know, sure, it's nice if a robot folds my laundry, but I know I can do it faster, cheaper, like myself.

1:02:28So, you know, my opinion on the robotics is I'm more interested in where it can provide better economic value over, you know, augmenting the human or supporting the human in a way that improves our capacity for productivity. Whereas in the agentic landscape, the big problem and also the big promise if we solve it for agents doing well is just being able to operate over this large corpus of information at once. It's the long context problem. You know, everyone's trying to push the limits of how many tokens, an agent can operate over at a given time. And it starts to break down as you have longer conversations.

1:03:06And then there's the question of, can it really operate over all the data I want it to if it can't handle more than a million tokens at once in a conversation? So that's the biggest hamstring right now, I think, with agents moving forward. They're still able to do a lot of great work through search and retrieval and limiting the amount of tokens they consume as they complete a task, only acquiring the information they need. But imagine if it could actually operate over 10 million tokens at once, the type of work that could be possible, which I don't think is reasonably doable right now. And I think that's the biggest problem with allowing agents working over the long horizon and also contributing in the next order way that would be superhuman, I believe.

1:03:52Yeah, I'm glad you brought it up. Optimization seems to be another piece that's creeping now into the AI development process that, you know, for the longest time, AI was cheap. Everybody was, all of the model providers were giving it away. Now, suddenly they put the paywall in front of all that. And suddenly everybody's having to optimize how many, well, you were talking about tokens in a different word, but, you know, actually the money that we're spending to deploy all this stuff. So on that same perspective, Talk about that state of the underlying architecture and infrastructure that's actually running all these amazing things in the kind of conversations you're having with your clients as well about that world in front of us to optimize the cost to develop all this stuff.

1:04:41Exactly. And there's a move with clients now with the strength of open source models to you get lower latency and better cost profiles when you fine tune your own model for very specific tasks. So you're seeing, I'm seeing more of the orchestrating many small models as opposed to let's play OpenAR Anthropic, the like premium to just use them for everything. So it's an interesting landscape where, you know, I don't think there'll be one model builder to rule them all. Like I think it's going to be a mix of, you know, a mix of tools in the space. And there's so many available now that it's, you know, competitive and you can design around.

1:05:24And it's very true, especially in the enterprise, the attention more now to token optimization as opposed to token maxing of the past after all those fun stories of people blowing through budgets in like a day. when they don't put caps on usage. I personally have some other libraries I use in packages to limit tokens, to maximize how much I can get for, bang for my buck with the type of work I'm doing. And people are making more conscious decisions around model choice as a result. Well, and speaking of trying to save on the model building, how big a role does synthetic data play in this journey of AI, agentic AI, and robotics development?

1:06:10Because there's a challenge of validating that data against real-world performance. And so where does that fall in Appen's perspective? Yeah, there's value for synthetic data in many targeted cases, I would say. When you are at the frontier trying to push the limits of capability compared to humans, that becomes very human necessitated data. But if you are fine tuning a model for a very specific purpose, I, you know, synthetic data is a great use. And I think that's why it's becoming cheaper, easier for people to, you know, deploy their own models and train them and get them to do something very specific.

1:06:55You can use synthetic strategies to generate those data sets and eval sets. It's still not trivial work, But models are helpful for doing that type of thing with the right touch. But synthetic pipelines have a lot of value, but they also do involve a certain amount of, you know, skill and observation around the data that comes out the other end. Yeah, I think I could probably share a similar example of what just happened this last week with the humanoid games in China, right? Just the advancement we've seen from the mobility side on humanoids, the speeds that these machines are running at now.

1:07:31A good example where synthetic data optimized probably that gate management and optimization to push the machines in simulation right before they actually deploy those types of gate models to the actual robots in the real world. Good example of that, don't you think? Yeah, I 100 % agree. Well, we're getting close to the end here. I think one of the other questions we want to ask you before you go is on the deployment side. So once these systems are deployed in the physical world, as we just talked about, what are some of the best practices that you work with clients to collect information once these things are live in production realms?

1:08:14and now they've met some hurdles, but now we want to optimize over the long term or reduce other sort of aspects of the model deployment, whether that's reducing its size, its cost. As we just talked about, what are some of the things that you work with clients once things get deployed into production? Yeah, robotics is a unique case because it's production, but it's still testing, I guess. with that specific type of use. Outside the lab. Yeah, yeah, exactly. It's like, you know, it's putting a beef jerky in a cup on the street instead of in the lab setting. We'll see if it can handle the environment a little bit better.

1:08:56But yeah, so we see like the need sometimes for humans to help with. I think I mentioned this before. This is like the human in the loop part where it's doing a task many times and then you want a kind of reinforcement learning signal about like, was there an error? What was that error? in the action? Was it a success? Did it accomplish the task? And then, you know, scaling that up into more demonstrations. A lot of the, you know, pre-production simulation testing work happens, you know, kind of, I wouldn't say offline, but offline in terms of like, it's not literally moving the thing every time.

1:09:35There's other ways to, you know, test before it goes into the actual mechanics of the robot moving and those trajectories can be analyzed in at least a somewhat automated fashion. And then of course, you know, once it's been introduced into the, you know, robotics, figuring out what the failure modes look like is a very human task that we see needed. So it's like, okay, it did accomplish moving the bowl from one side to the other, but did it drop the bowl in the process? What was failure recovery like? Can it handle different environment than it was trained on? And then of course, the latency component, I think, is becoming more important.

1:10:15That's maybe something that like the human data field doesn't have as direct an impact on because that has to do with, you know, training and weights and, you know, the data that goes in and optimizing around probably simulation, but making sure that we're not compounding failure modes in the process is where we can add a lot of value for it with that human expert human touch. And speaking of that added value, Janine, obviously there's a lot of activity around AI and robotics right now. Why should startups be looking at app intercompanies like you and what can they gain from such collaboration because there's so many aspects to the AI landscape right now that people may not necessarily know where to start, but what can they gain from working with you?

1:11:06Yeah. So, you know, the obvious things amongst the data sets that we have available, taking pieces of that. And then, you know, working with kind of me and my team specifically, we're very happy to, you know, have conversations and brainstorm and see where we can plug it in a, you know, cost effective way, just because we have, you know, tools available and experience where we can have like a light touch human in the loop and then also provide some support around automated solutions that leverage that data set so that you're getting more, you know, getting more bang for your buck out of what you spend in terms of like expert human data.

1:11:49So there's like a couple places and, And, you know, we love talking to, you know, I love talking to founders about how we can best support them. And they sometimes bring some interesting problems that they've, you know, hidden the space that helps us, you know, expand our horizons as well. Well, Janine, it's been a great conversation. It's been great to learn more about Appen and the role you play in the ecosystem. Thanks for helping us connect the dots between Gen.AI, data quality, and the future of robotics and joining us on the show today. Yeah, thanks so much, Mike. Good to see you again, Gene.

1:12:23Lovely to be here.

1:12:29Thanks again to Janine for doing that and lending us her time. Really like that. She cautions Gene, a robot performing one impressive task doesn't necessarily mean that it can handle unexpected changes in its environment, which is something that we often talk about a lot, not only on the podcast, but also at our events as well. What were some of your takeaways from the conversation? You know, for all the talk about robotics having a cat-TPT moment where AI enables it to generalize behavior, there's still a need for humans in the loop, according to Janine. And I think that's an interesting point that she really hammers home.

1:13:07Human judgment is still essential for spotting subtle problems, rare failures. just as you need real world data to build your synthetic data or your simulations off of you also need that human judgment over what makes sense and so this is where a lot of people are looking at egocentric data which you know in the simplest form and someone's strapping a cell phone to their head and taking video of them doing a mundane action again and again and there are probably people already doing that right now but it's important to help train robots and she noted that one of the exceptions that happened is that people were turning off their video and it's like no no we want to see the mistakes we need to see how people handle that how people recover from exceptions and can train the robots and so there's that whole again that egocentric data at that human level of yeah humans do things instinctively right we might be popping up a cucumber or screwing together a device or something like that and we're teaching robots but we also have to teach robots that there's not just one way to handle it and humans are adaptable that's one of our strengths and so how does our judgment show that the way for robots to do things and we've all seen the cool demo videos of robots unloading dishwashers and folding laundry and preparing food.

1:14:40But in most cases, robots have been trained over maybe thousands, maybe millions of iterations in order to conduct just that task, be able to then apply that to something else. That's the hill that everyone's now trying to climb. Yeah, I think the chat GPT moment is still quite far away. We're getting closer. You know, people are figuring out the steps to solve it. And I will say that what Janina is doing at Aften is absolutely innovative in its approach, right? Okay, we're trying to have data, but it's got to be high-quality data. We're trying to have human-centric data and not relying just on simulation and making sure that there is that transferability.

1:15:20We don't even completely know how the human brain works, so trying to create artificial one that works just as well takes a fair amount of effort. And there's billions of dollars going into that right now. There sure is. There sure is. Well, great stuff there from Janine. So thank you. And thank you guys for doing that interview. That's going to do it for the show. But before we go again, we promised you a little discount code to help you guys save some money on robo business, which is quickly approaching Eugene. It takes place October 20th and 21st out in Santa Clara, California. Again, this is our 20th anniversary show.

1:15:56You and I have been involved. How many of these do you think we've been involved with over the years? Ten? Well, at least ten. I mean, not consecutively, not all of them. But, yeah, we've been involved in it for at least half of its lifespan. And in robot years, that's a long time. Yeah, it's a very long time. But it's going to be a great time. Again, you can check out the agenda. We're going to have over 2 ,000 people there, over 100 exhibitors on the show floor. We have a startup alley where we've given away space to 10 startups that are going to be coming out of stealth mode at the show. So if you want to, if you've been in the industry for a long time, come help them, mentor them, say hello.

1:16:34But it should be a great time. So, again, head to robobusiness.com if you want to get your tickets and enter code ROBO15. That's R-O-B-O-15. And you'll save 15 % on your full conference tickets. So, again, Robo15 at checkout for RoboBusiness will help you save 15%. And we'd love to see you there. Again, this is a labor of love for us. We've been doing this for a long time. The show gets better and better every year. We have some amazing people speaking at the event, people who are the best of the best when it comes to deploying robots in the real world and everything that comes with that, the good, the bad, the ugly, everything in between.

1:17:14I mean, there's no better event to learn about how to deploy these things in the real world, how to build your robotics business than at RoboBusiness. Again, takes place October 20th and 21st out in Santa Clara, California. Enter Robo15 at checkout. You'll save 15%. And again, that's going to do it for Episode 265. Thank you again to everybody for tuning into the show. Thanks to Janine Sinan Singh, Director of Gen AI Research at Appen, for joining us on the show today. Thanks again to Wybotic for sponsoring the show this month. New episodes of the Robot Report podcast drop each week. You can find us anywhere.

1:17:50You listen to your podcasts. Please subscribe. Leave us a rating. Give us a review. Gene, this was fun. We should do this more often, man. We should. Thanks to Mike Oitzman who produced this. He is on a much-needed vacation, but he set this up for us. It was edited by Tyler Kubisti. Our executive producer is Amanda Buhner. I'm Steve Crow. Thanks so much, folks, for tuning in. We'll talk to you again next week.