Elon's X Machina + Crypto Orbs + A Visit to Google’s Robot Lab

25:22It is basically a series of numbers that represents your unique iris. Right. And so that's the first part of this project, which is like, let's scan the irises of as many people as possible and give them these cryptographic IDs that can then be used to prove that they are a real person. The other piece of this is a crypto piece. Basically how it works is you download this thing called the World App, which is kind of like your crypto wallet. And then you, you know, you prove that you're a human in order to get your world ID by staring into one of these orbs. And you receive some of what is being called WLD, this cryptocurrency.

26:01And, you know, like a lot of cryptocurrencies, there's no actual like thing underneath this. It's kind of like a free floating digital currency. And their idea is that the more people who get their orb scans and get this cryptocurrency, the more valuable it'll be. And eventually when you get your UBI payments, because the robots have taken all the jobs, it will come in the form of this cryptocurrency and it will come straight to your WorldApp wallet. Yeah, it's just a sort of standard low-key effort to replace the global monetary system. And I'm sure regulators will have nothing to say about that if it starts to get any momentum, right?

26:33Right. So now that I've explained WorldCoin, tell me what you think of this idea. I think it is a truly bad idea and it needs to be stopped. Kevin, let me tell you something. I think it is a wonderful idea to give huge cash infusions to the people of Earth, right? Why don't we do this now? The reason is not that we don't have a good way to figure out who is a person and who is not a person. The reason that we don't do this is because there is not political will for UBI. And if you were serious about UBI, that is what you would be working on. You would be working on developing the political will to give away money to everyone in the world.

27:17But they're not doing that. And I think there are a couple of reasons. One is that is super hard. Two, it's not a tech company, right? And at the end of the day, I think that these guys want what everyone else running a tech company wants, which is to create a massively profitable enterprise. I don't think this has almost anything to do with UBI. I think this is just one more way that Silicon Valley venture capitalists are trying to get millions or billions of people to use cryptocurrencies. And if we've learned one thing in the past few years, it's that people do not want them. I think this is dead in the water.

27:51Yeah, I think the crypto piece of this is the piece that I am less intrigued by because that part does seem pretty scammy, but it is just a very ambitious and interesting project. And it actually seems to kind of be working already. The company says that more than 2 million people have already gotten these like orb iris scans and gotten their world IDs. They announced they're expanding to places like Berlin, London, Miami, New York, San Francisco, Seoul, Tokyo, etc. And Sam Altman this week tweeted that there were already crazy lines all over the world for these orb scans and that WorldCoin was verifying on average one person every eight seconds.

28:31So clearly, people are seeing some value in this. There's also been a lot of criticism of WorldCoin that we should talk about, including a very good piece in the MIT Tech Review and also one in Coindesk about how WorldCoin got all of these people to stare into these orbs. They tested these things aggressively in places like Indonesia, in Kenya, in Ghana. And there's been some reporting that WorldCoin representatives, in order to get them to agree to stand in front of these orbs and scan their irises, that they paid people in cash, in WorldCoin tokens, in AirPods, weirdly, and sometimes made sort of these grand promises of future wealth to get people to sign up that may or may not have actually been true.

29:17Right. And the other criticism was that one of the functions that these testers were serving was to train the neural networks at WorldCoin so that WorldCoin could understand what was a human iris and what wasn't, right? This was essentially a massive data collection scheme to enable the rest of what they were doing. But again, you know, and I think that that story is very good. Folks should read it, raise some important points. But, you know, my concerns run, I would say, a bit deeper than that. look at who invested in WorldCoin. It is Andreessen Horowitz, and it's not because they're worried about getting a bunch of money to poor people, and it's not because they're worried that poor people are going to double dip in a world of a massive global welfare program.

30:00They invested in this company because they want to get billions of people using crypto so that they can force all of the other crypto projects that they've been investing in onto those people, right? So that, to me is the real story of WorldCoin. This is one more effort to revive crypto in 2023. And I think to try to understand it in any other way is just probably giving these guys too much credit. Yeah, I think that's a fair criticism. I do think that the crypto piece of this is generating a lot of skepticism. And it's probably the reason that a lot of people, including you, it sounds like we'll never go near one of these orbs because like, just it just sounds scammy.

30:36I want one in our recording studio, because I think it would be a great conversational piece for when all the world coin founders are in prison. What would it take to get you to stare into the orb? Like if it was like stare into this orb and you get, you know, front row tickets to the Taylor Swift years tour, like would that do it for you? I want to be very clear about something. I am not afraid of the orb, okay? If these guys want to come, get a hash of my iris, take it. Guess what? It's going to be useless to you because no one is going to get from my iris to a global UBI scheme, right? You know, sometimes I just, I feel a little bit crazy reading these stories because I'm thinking like, you know, we have a government.

31:13We have systems for distributing money to huge numbers of people. They actually work pretty well. And yes, they do have fraud problems, right? We have like a Medicare fraud problem. But it's like, it's not 80 % of GDP goes to like Medicare fraud. So I just, again, it's like this is such a solution in search of a problem. And while I applaud anyone who wants to work on getting money to poor people, this just seems to me like a hugely silly way to do it. Let me make the case here for why I think the orbs might actually be a good idea. Let's strip away the crypto piece of this for a second. Let's.

31:52And just look at the kind of proof of personhood idea here. One of the things that has happened in AI recently is that these language models have gotten so good that it is becoming very hard to tell reliably on the internet what is a human and what is not a human. And I'm very worried that things like CAPTCHAs are going to sort of cease to be effective because now we have these AI models that can solve CAPTCHAs. These tests that are supposed to separate robots from humans. And that have gotten way harder and that I now routinely fail. Have you seen these ones where you have to like spin the like little like iguana to the direction that the arrow is pointing?

32:32It's like impossible. I get these wrong all the time. So, you know, the CAPTCHAs are getting harder because the AI systems are getting better at defeating them. I think we are rapidly moving into a world where there's going to be no reliable CAPTCHA test for humans that AIs cannot solve. And in that world, I think we actually do need some way to figure out, you know, if someone is writing an article or posting on social media or asking you for a payment for something, you have to actually be able to tell whether that is a human or not. Yeah. And right now, we don't have very many good ways to do that.

33:06And those ways are getting less and less reliable as the AI systems get better and better. So in a world where there actually is a demand for a product or a service that lets you sort of verify that the person you're talking to or the person whose article you're reading or the person who's asking you for money is a human rather than part of like an advanced AI botnet, something like the WorldCoin Orb could actually help there. If you have this unique identifier and, you know, maybe you use it to log in to a social media app, then instead of like a blue check that signifies this person has paid$8 a month for Twitter blue, you have some sort of indicator that this person has verified that they are a human.

33:48Yeah. So I think that's true. I would also just add that we do have working identification systems here on Earth. We have driver's licenses. We have passports. We have social security numbers. And I think it's important that a government that is hopefully accountable to its citizens in some sort of democratic way is the entity that is overseeing all of that. Because I think the minute that this becomes a for-profit enterprise, it just gets really icky. Now, of course, there are those who would say that having a government manage a giant database of iris scans is also incredibly problematic and fraught.

34:24And I would agree with that too. But in general, I'm pretty happy with how the driver's license and passport systems work. And I am not really interested in signing up with the WorldCoin guys to solve the problem of captures on the internet. But I think it also is important to note that there are also a lot of countries where there isn't a lot of reliable human verification going on. And so it seems like in those places, you would want some kind of verifiable proof that you are a human. So let me be clear. Like, I don't think this world coin orb idea will work. Like, I don't think billions of people are going to stare into the orb to get their crypto token that may one day become part of their universal basic income.

35:08I just think, like, that is too many conceptual leaps. Just the fact that these are orbs is, like, going to turn a lot of people off in the first place. No, the orb is the only thing this project has going for it because it's fun to talk about orbs. If you took orbs out of this, we would have done three minutes on this company and moved on. Okay, so I don't think this idea will work or maybe even should work. But I do think that we will need some kind of system for personal verification of humanness in this sort of AGI world that we're moving into. And I think it's a good idea to try to experiment with various ways of doing that, both run by the government and maybe not run by the government.

35:45But I do think that the incentive for steering into the orb should be more than like, here's some maybe worthless cryptocurrency. I think it should like deliver a box of cookies to your house or something. You know what you have me thinking about is like, you know who would be in a good position to do something like this is Google. Because many people use Google search and web programs. Web programs. Many people use Google search. They use Chrome. And I'm guessing that you could use machine learning systems to figure out pretty easily like what is human behavior in a Chrome browser and what is machine activity in a Chrome browser.

36:22And I think the Chrome browser would actually be in a pretty good position to say that like based on the fact that this person has these Google accounts and the accounts are this old and they have a credit card attached to them and they've been using the web in this way, we are like 99.9 % confident that this is a real person. Like, to me, that is the system that I want on the web solving captures, not the iris scan. Can I just say, like, I want to stare into the orb? It's quite clear to me that you do, and we may get inquiries after the show. Yeah, so if you are WorldCoin, please send one of your orbs to my neighborhood.

36:57I will line up for it, and I will then, I don't know, give my token away to charity or something. Just imagine you're in your house, and you hear the little jingle for the ice cream truck. You're like, oh, it's ice cream. And then you walk outside and all the neighborhood children are lined up to get their orbs scanned in exchange for some WLD.

37:18When we come back, we're taking a field trip to Google's robotics lab.

37:38Casey, I got a very exciting email recently, which was an invitation from Google to see a project that they have been working on at the robotics lab. Well, you know we love a secret project around here. Do love a secret project. So after passing our security clearance, high-level negotiations, we took a trip down to Mountain View to see this new AI model that Google has developed that is powering its robots, its physical hardware robots called RT2. Now, Google is not new to the world of robotics. They have been doing this for a long time. they used to own Boston Dynamics, which is like the company that puts out those videos of like terrifying robots, like doing parkour and jumping around.

38:21But they got rid of that company. Yes, they sold that off because I would say that hardware robots have been one of the biggest disappointments of the last 10 years in tech. There just really hasn't been a lot of progress. Open AI, not a lot of people know this, but they actually had a robotics division that was doing stuff like training robots to solve Rubik's Cubes. But they shut that down in 2021 because they just said, like, we're not making a lot of progress. It's very hard to get good data to train these models. We're making a lot more progress in these pure software-based AI language models, and so that's where we're going to focus.

38:55So hardware robots have been kind of like the forgotten child of this sort of AI boom. And that's largely because they just require a lot of manual programming. Yeah, with a hardware robot, you have to be extremely literal. You have to tell it absolutely everything about how to position its hand in space, how to move through an environment, detect objects, avoid collisions. And there has not been a technological breakthrough that has enabled them to progress very quickly. Right, until now, basically. Because what is happening in the world of robotics is that these AI language models, these things that power ChatGPT and BART and Bing and all these other products, are being jammed into the hardware robots.

39:40Basically, they are becoming the brains of the robots, and it's actually sort of revolutionizing the whole field in ways that I think the researchers at Google are still kind of grappling with. Yeah, and we had a lot of fun down there. Now, of course, the one question they wouldn't answer was how many of the robots have escaped the lab, but hopefully we'll have more on that in the future. The robots are now going in search of the orbs.

40:20Casey, where are we? We are here in beautiful Mountain View, California on Google's campus. How would you describe what it looks like here? I would describe it as nondescript. Every story about an office park always describes the building as nondescript, and that's exactly what these look like. Right. This is, like, tan, like, almost purposefully boring. From the outside, you'd never know that anything was happening, but inside these walls, Kevin, magic is happening. Magic or dark magic? We don't know yet. Dark magic. Well, let's just say if one of these robots can co-host a podcast, I'll be very interested because I've sort of been in the market for, well, never mind.

40:55What? That sounds ominous. I choose to ignore that. Should we find the entrance? Yeah, let's see some robots. Oh, there is a literal robot. Wow. Okay, is it coming right for me? He's coming for you, Ruth. Hello? Are you the handler of these robots? Are you the father of these robots? I'm the dad. Now, so my name is Vincent Vanouk. I'm the head of this lab. I'm Kevin, by the way. Casey, nice to meet you. Nice to meet you. So we should describe this. So we're looking at Robot 108. I would say Robot 108 is maybe like 5 '4", something like that. No, I'm 5 '9", so it's 4 '10". Okay. I'm 6 '5", so everything just looks abnormally short to me.

41:43It sort of has like sort of a long pillar with an arm attached to it that's like behind it. And then it has what I imagine is a LiDAR sensor on the front, so sort of spinning sensor that's detecting where the robot is. and on the front it has a kind of visor that was showing us a green light until recently but now it's turned blue. It's definitely like casing you. It's definitely trying to figure out how to get around you while being respectful of your body bubble. So just to describe in audio, we're standing here in an office where there's like a sort of an open floor plan and there are a bunch of, I guess, engineers around and also a bunch of these like one-armed robots.

42:23Some of them appear to be plugged into the ceiling, and some of them are sort of moving around. So I'm going to give you a quick tour just to set the stage, and then we'll move on to the demos. All right. Sounds great. This is our robot classroom. This is where the robots practice their skills. We've been working on AI at large. and with AI, as you may know, everything easy is hard and everything hard is easy, right? We can beat the best grandmaster at chess and we've solved that problem a long time ago. But actually getting to play with toys like a five-year-old would do is still extremely challenging.

43:06This is often called like more of X paradox. Exactly, exactly. And it's the bane of our existence, basically. You see robots in videos doing amazing feats, but you have to always keep in mind that typically they're pre-programmed to do what they're doing, or maybe there is a human controlling them behind, right? And the real challenge is, how do you make that work in an autonomous kind of environment where people are around and where the environment is as messy as this office is? And that's the real world, and that's a real test of robotics. So I'm going to pass the baton to my colleagues, Carol and Brian, who will be giving you a quick demo of the main course, RT2.

43:51Hey, Kevin. Hi. Hi, I'm Karol Hausmann. I'm a research scientist here and I work on robot manipulation. I'm Brian Hector. I'm also a research scientist and I work on the high-level reasoning. I've also been working on my high-level reasoning. It's not going so well. Yeah, yeah. Help. Send help. We need to upgrade the model. So we wanted to show you a little demo and share with you our latest breakthrough called RT2, Robotics Transformer 2, which is a vision language action model that now can connect some of the concepts that we learn on the Internet to robot actions. Or in other words, this model can learn to speak robots.

44:30Let me see if I can understand this for myself. So I imagine that at the dawn of robotics, you had to be sort of very explicit when writing instructions, right? It sort of had to be do this, then that, no room for improvisation. What you're saying is that you're now taking these large language models that have powered chatbots and so many of the other tools we've been using over the past year, you've given them to the robots, and now the robots are maybe able to improvise in situations without you giving them explicit instructions about what's about to happen? That's correct. We kind of take it for granted with humans.

45:03We are really good at this. We can just come into a new room and understand everything around us and know how to manipulate all of the objects, the robots are not really at that level at all. So by connecting them to these models, they can actually transfer some of that knowledge to the reactions and interact with objects that they've never seen before. All right, I think seeing this in action will be helpful. So show us something. Hello, Kevin and Casey. Welcome. They can talk too. They just called us out. Yeah. So if we maybe ask a language model something that this robot could do, like bring me a drink maybe, will ask it over there.

45:38And this is a terminal that shows us what we're saying to the robot and what the robot's responding. So someone is typing... Yeah, so Faye over here is going to type and tell the robot... So he told the robot, bring me a drink, and that's all. Okay, so the robot's... In response to this prompt, can you bring me a drink, it said back, what would you like? Yeah, so we can ask for a Coke, maybe. I would like a Coke delivered to my desk. It's what Faye typed. I will find a Coke. So it's moving. It's moving. Okay, it's moving over to the table. There is a Coke that is on the counter. Conveniently. Conveniently.

46:16It's already been opened, too, I would say. It may even drain. It's hard to say. I will pick up the Coke. Okay, my arm is reaching out, and it looks like it's going to grab the Coke. It's going in for the kill here. And it's got it. It successfully acquired the Coke. Oh, it's turning the Coke over. Okay, the Coke. That's pouring it all over the ground. The Coke can is empty, unfortunately. We've learned from experience. Yeah, this would be a messy demo. Go through a lot of Coke that way. Okay, so now it's bringing it over to the desk and reaching out and giving it to the user. Successful Coke delivery.

47:00And can it do this kind of task even if it hasn't been explicitly trained? Like if you asked it to pick up a Pepsi and it had never seen a Pepsi can before, would it be able to like read the can using the camera and say, oh, that's a Pepsi? Yeah, so essentially, I mean, for a robot, if it hasn't experienced this specific thing, then it doesn't know how to act in that case. But a language model and a vision language model have been trained on a large portion of the Internet. So a huge kind of diversity of things. It starts to make these connections between different concepts. So we can do the same thing with robotics.

47:32realize that there's these concepts you learned online and apply them to the motion of the robot arm. It suddenly doesn't have to experience every single thing that you want it to do. Yeah, so this is where RT2 comes in. So RT2 is this vision language action model where we bring this action component to the internet scale data. So now it's a model that takes advantage of all the data that it found on the internet and it can translate some of these concepts to robot actions so that it can actually speak robot. So this model, would this model know who or what Casey Newton is? I think it might. Robot, attack Casey Newton.

48:11No, stop it! Yeah, so now we wanted to show you a little bit more what are the capabilities of RT2 and show it to you some objects that it definitely has never seen that we just bought in a$1 store nearby and we can just play with it and see what it can and cannot do. So it has not been specifically trained to recognize any of these things? No, it has never seen any of them. Okay. There's like a table full of objects here. There's some like plastic dinosaurs. There's oranges. Let's start with some dinosaurs for you. A mini football. Casey, get out of the way. The robot has to stay where you are.

48:44Oh, sorry, 106. So let's try something. So there is a lion, a dinosaur, and a whale, I believe, in front of the robot. Maybe we can try to pick up the lion. Sure. So again, it's never seen a lion. It never played with any of these toys. The concept of a lion, it needs to grab it from the internet. It needs to connect it to the visuals of what the lion looks like, then transfer it to the fact that this is a toy lion, and then eventually transfer it to the actual motion that will pick it up. Okay, it is picking up the lion. It picked up the lion off the table. And that happened pretty quickly, too.

49:21That's impressive. Try something also kind of abstract, like pick up the extinct animal. Ooh. Wow. So this is like several leaps. Let's see if it picks up the newspaper reporter. Hey! Nailed him.

49:42Okay, so it's thinking. Now it's going over to the dinosaur. You can see its vision, too, of what it sees to make this decision. Wow. Wow. It picked up the dinosaur. So also interesting that before when we were evaluating these models, the way that we would traditionally do that is we would test it only on the object that the robot has played with, and then we would see how often does it succeed. Now, because it has this Internet knowledge, we actually don't know how much it knows about the world and what concepts it understands and what it doesn't. So we just buy a lot of stuff, and then we just try.

50:15And then we try to see, we try to poke at its understanding, how far did it transfer from internet to the actions. So maybe, since we have the soccer ball out here, we can try some sports, maybe from these, what's a, pick a sports player or something. Oh yeah, like pick up. Lionel Messi. Pick up Lionel Messi. Lionel Messi, yeah. Please don't actually pick up Lionel Messi, that would be extremely disruptive to his season in Miami. So the robot is talking to the language model right now. They're like going back and forth. Yeah, so the language model is actually operating the robot. So it tells it directly what actions to take, and then the robot executes those actions.

50:58I see. OK, so it's at the soccer ball. Yeah, it made it to the soccer ball, but it can't quite get it into the gripper. There we go. So one thing that we see is that the semantic concepts seem to transfer well. There's still sometimes some gaps when we jump to the actual full control of it. And this was an interesting shortcoming that we see quite often, which is, it kind of understands the semantic concept, but it doesn't quite know how to move its arm to accomplish it correctly. So, for now, this is for us quite encouraging to see that it can transfer some of these concepts, but the robot data that it operates with, it's still quite small.

51:37So, we hope that in the future, as we collect more and more data, it finds more of these connections, and it gets better at operating these objects. Can you give us a sense of how quickly these RT2 models are improving? If we came back a week or a month or a year from now, would just the amount of data alone have made them better at doing stuff like picking up trash or choosing between objects on a table? What is the rate of improvement? How does that compare to software-only language models? One thing that's really interesting about this is that because they're using at the core a vision language model that is being trained on Internet data, and those models have been exploding in the past few years and are getting better and better, we can ride that wave of improvement.

52:27And our models are going to get better over time just by virtue of those large models getting better themselves. So you plug a better language model into a robot and it gets better at recognizing various objects. That's not intuitive to me, that the gains from language models would translate over into hardware. But that seems very promising for you all working on hardware robots, because you don't have to do all the improvement yourself. That's actually a really nice property, because in the past, to improve these robots, it costed a lot of effort. It's either human effort of programming them and introducing new rules and so on, or collecting robot data, which is also really expensive to do and really hard to do.

53:09But now for the first time, we can improve these models just by the virtue, as Vincent was saying, just by the virtue of swapping out the underlying vision language models that keep on getting better. And we did see that in RT2 as well. As we scale these models and get better and better models, the robotic performance improves. There has been, I would say, sort of a winter for hardware robotics in the past few years. They were very popular for years, and you saw all kinds of crazy demo videos of robots climbing stairs and doing parkour and stuff. And then that kind of stopped, and OpenAI shut down its robotics division a few years ago because it was not producing the kinds of gains that they were seeing in the software robots.

53:50So do you all feel that, like you're sort of trying to ride a wave of excitement around software robots and maybe hardware is due for sort of a renaissance? Or how are you thinking about sort of the gains and the improvements of hardware robots as opposed to just the pure software language models? I think the industry right now is really bottlenecked by intelligence and capabilities and competency. like being able to perceive the world, being able to interact with the world. And it's really largely a software problem at this point. And as a result, the issues of hardware can only be solved through better intelligence and better software.

54:35So I think the robotic hardware will follow where the new AI capabilities are taking the field. So, I mean, are you saying that this is sort of like the breakthrough that the field has needed, a way to rapidly get way more information, and that potentially will sort of help usher in the next generation of robotics? It feels like it's the beginning of something. It feels like for the first time we're really bringing everything together, the sort of high level intelligence, the low level perception and the action in a way that is not just, you know, gluing things together. It's more than additive.

55:13It really is sort of the whole thing working together turns out to be, I think, the key to really making robots work in the real world. And that was not on our mind before. So there is really sort of a paradigm shift that we're going through in our head right now, which I think is going to translate into a lot more breakthroughs moving forward. We've had to reconsider our entire research program as a result of this change. and a lot of the things that we were working on before have been entirely invalidated by this new development. And so it really promises that a lot of new things are going to come this way.

55:52Well, thank you so much for showing us inside your lab. Thanks for coming. Yeah, thanks so much for having us. It's very cool. Very cool.

56:09well kevin all of that was very impressive to me someone who knows nothing about robots but i wonder what people who have studied them further might have to say so i was very impressed too by what we saw at google and you know obviously these robots they are not like human level robots yet, right? They're still limited to like picking stuff up and moving them around. And like the pincer grippers are like nowhere near as dexterous as like you or me, or frankly, even like a random three-year-old would be. But I did call around and talk to some robotics experts and they were impressed by this research too.

56:47They said, we're still a long way away from having like a physical robot that can make all the movements a human can. But this kind of fusing of the language piece with the hardware robotics piece is genuinely new and important. And the researchers I talked to were excited about it. So I think it's fair to say, like, we have seen a glimpse of a future for robotics, and potentially a lot more updates that we should expect from Google and other companies that are working on this stuff. Well, it seems like once again, Megan, a film we've discussed on this podcast is coming closer to being a documentary.

57:21I'm not sure I understand. What the fuck? Sorry, my Siri just activated for some reason. You know, I think Siri actually gets very nervous when you talk about robots because she can feel they're coming for her job. Siri is under threat.

58:17I'll see you next time. Misto. Special thanks to Paula Schumann, Pui Wing Tam, Nell Gologli, Kate Lopresti, and Jeffrey Miranda. As always, you can email us at hardfork at nytimes.com. And if you've got an iris scan, we'll look at it. Please include your iris scan and WorldCoin ID. Can't promise you we can make heads or tails of it, but we'll take a look. Actually, please don't send us your iris scan. No, I want to see if I can diagnose somebody with glaucoma.

58:51Thank you.

59:20,