ChatGPT Gets a Body with the Figure 01 Robot

0:00Today on the AI Breakdown, we're looking at Figure and OpenAI's new collaborative, interactive robot that is blowing people's minds. Before that on the brief, Google DeepMind releases a new generalist video game playing agent. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our Discord, our newsletter, and our YouTube. Welcome back to the AI Breakdown Brief, all the AI headline news you need in around five minutes. The AI agent future is coming on quickly, And Google DeepMind has made another advancement in that area.

0:36Yesterday they tweeted, Introducing SEMA, the first generalist AI agent to follow natural language instructions in a broad range of 3D virtual environments and video games. It can complete tasks similar to a human, and outperforms an agent trained in just one setting. We partnered with gaming studios to train SEMA, which stands for Scalable Instructable Multi-World Agent, on No Man's Sky, Teardown, Valheim, and others. These offer a wide range of distinct skills for it to learn, from flying a spaceship to crafting a helmet. CIMA needs only the images provided by the 3D environment and natural language instructions given by the user.

1:07With mouse and keyboard outputs, it is evaluated across 600 skills, spanning areas like navigation and object interaction such as turn left or chop down tree. We found CIMA agents trained on all of our domains significantly outperformed those trained on just one world. When it faced an unseen environment, it performed nearly as well as the specialized agent, highlighting its ability to generalize to new spaces. Unlike our previous work, CIMA isn't about achieving high game scores. It's about developing embodied AI agents that can translate abstract language into useful actions, and using video games as sandboxes offers a safe, accessible way of testing them.

1:40The SEMA research builds towards more general AI that can understand and safely carry out instructions in both virtual and physical settings. Such generalizable systems will make AI-powered technology more helpful and intuitive. So basically what you have here is an attempt to train a generalist AI agent and see how it compares, in the context of these virtual worlds in the form of games, to agents that are trained just specifically on one game. The broader goal of the research, as Google said, is about generalist agents that can navigate any virtual or physical world environment, not just those of video games.

2:08The Google DeepMind blog post has a lot more about how the process worked. For example, they write, We used four research environments, including a new environment we built with Unity called the Construction Lab, where agents need to build sculptures from building blocks which test their object manipulation and intuitive understanding of the physical world. By learning from different gaming worlds, CEMA captures how language ties in with gameplay behavior. Our first approach was to record pairs of human players across the games in our portfolio, with one player watching and instructing the other.

2:33We also had players play freely, then rewatch what they did and record instructions that would have led to their game actions. I think the big upshot and what Google DeepMind is excited about comes from the fact that, as they write, we show an agent trained on many games was better than an agent that learned how to play just one. In our evaluation, CEMA agents trained on a set of nine 3D games from our portfolio significantly outperformed all specialized agents trained solely on each individual one. What's more, an agent trained in all but one game performed nearly as well on that unseen game as an agent trained specifically on it on average.

3:03Importantly, this ability to function in brand new environments highlights Seema's ability to generalize beyond its training. Google D-Mind has long been a fan of using these sort of gaming environments to figure out advances in AI, and this seems to be no exception. Next up on the brief, what is looking sort of like a fairly disastrous interview by OpenAI CTO Mira Mirati with the Wall Street Journal. The piece was called OpenAI Made AI Videos for Us. These clips are good enough to freak us out. Now the article as written isn't damning. In fact, it's part of the coverage of Sora that reflects just how much people are blown away by it.

3:34As they wrote, when OpenAI began previewing videos made with the generative AI tool last month, the internet understandably lost its mind. Other AI video technology has produced choppy low-resolution clips. These look like something out of a nature documentary or big budget film. Sora brings new intensity to the now familiar AI feelings loop, amazement about the capability followed by fear for society. And if you'd just read the WSJ piece, you'd probably come away pretty impressed and fairly thoughtful. However, if you actually watched the interview, particularly the question where the interviewer asks what data Sora was trained on, and specifically if it was trained on data from YouTube, Facebook, or Instagram, you would have a very different impression.

4:11The Verge writes, when pressed on what data OpenAI used to train Sora, Maradi didn't get too specific and seemed to dodge the question. Quote, I'm not going to go into the details of the data that was used, but it was publicly available or licensed data. Maradi also said she isn't sure whether it uses videos from YouTube, Facebook, or Instagram. Ed Newton-Rex, the CEO of Fairly Trained and the person who left Stability AI over their copyright policies, writes, Disappointing to see OpenAI's CTO dodge questions about Sora's training data today. If a GenAI company dodges questions about their training data and uses the phrase publicly available, it seems fair to assume their model was trained on copyrighted work without permission.

4:45Even if one doesn't agree with that, I think a lot of people are stunned that she wasn't prepared for this question. That she didn't assume that this is going to be a question asked in basically every interview from here to forever. Given the fact that OpenAI is fighting multiple lawsuits on multiple fronts around copyrighted data, of course they're going to ask this question. You would have thought that everyone from the PR team to the executive team to the legal counsel would have figured out exactly what they were going to say to this answer long before the interview ever took place. Interestingly, given that we talked about the EUAI Act recently, Louisa Jurovsky points out that this is potentially more than just a PR problem.

5:19Louisa writes, The clip below could put OpenAI in trouble. Here's why. In case OpenAI Sora is classified as a high-risk AI system per the EUAI Act, they will have to comply with transparency obligations such as informing users about training, validation, and testing datasets used. In the clip below, Mira Mirati, OpenAI's CTO, cannot specify or exemplify the sources of data used to train Sora. If she were from the marketing department, this would be okay, but as a CTO, this is a core aspect of the technology that can lead to legal liability, and she should be able to answer it in a straightforward way.

5:47Still, I think while this might be the conversation inside of our enfranchised AI community, by and large, still what people are paying attention to when it comes to Sora are its incredible outputs. Lastly today, another update around election-related policies for AI. Mid-Journey has changed its policies and is now blocking images of Biden and Trump as the US election comes into focus. This was announced during an Office Hours event yesterday, where MidJourney CEO David Holtz said that the company was starting to block requests for images of the two candidates, which is a reversal of a decision that they had made back in February.

6:19When the associate press tried Trump and Biden shaking hands at the beach, they got a banned prompt detected warning, and after a second attempt got an abuse alert warning. Petapixel tried to get around it by saying the 45th president of the United States with the 46th president of the United States holding hands, but also got the banned prompt message. Said Holtz during the office hours, I don't really care much about political speech. That's not the purpose of MidJourney. It's not that interesting to me. That said, I also don't want to spend all my time trying to police political speech. So we're going to have to put our foot down on it a bit.

6:47Sounds to me like this is less about taking some deep principled stance and more about just not being distracted by what is going to be a very distracting conversation. Expect to see more of this type of decision happen as we get closer to November. For now, however, that is going to do it for today's AI Breakdown Brief. Up next, the main AI breakdown. Today's podcast is brought to you by Plum. If you're a startup building AI features for your customers, you're probably feeling the pain of hallucination, prompt testing, unstructured responses, subpar queries for embeddings, and of course, the mind-numbing process of general iteration and refinement when your engineers have to make every change by hand.

7:24That's where Plum comes in. Plum is a no-code AI app builder designed for product teams who care about quality and speed. What is taking you weeks to hand code today can be done confidently in hours. Check out useplumb.com, U-S-E-P-L-U-M-B dot com, or reach out to me for early access. Welcome back to the AI Breakdown. This week has been weirdly filled with buzz about exciting things coming, and everyone has been trying to kind of figure out which of the crazy things that we keep seeing are those things that people were teasing. Was it Cognition's demo of their AI coder tool Devin? or, as it increasingly seems, was it a demo of OpenAI's integration with the figure humanoid robot?

8:04We're going to talk a little bit about that today, and to kick things off, I highly suggest that you go actually watch this video. For those listening to the podcast, I will leave a link in the show notes. For those of you watching the video, I will just actually show it. It's about two minutes long, but it's worth watching in full. This was a video of the figure humanoid robot, the figure 01, engaging in real-time conversation with the human, not being teleprompted, and figuring out how to engage in fast, low-level, dexterous actions. But to understand the context for this, let's actually jump back a couple of weeks.

8:36Throughout the beginning of the year, there were rumors that Figure was raising a massive round, and that was confirmed at the end of February, when they announced a$675 million fundraise, which included participation from Jeff Bezos, NVIDIA, Microsoft, Amazon, and more. As part of that announcement, Figure also shared a new partnership with OpenAI. Popular Science Magazine, which now goes by PopSci, wrote that the partnership marked, quote, one of the most significant examples yet of an AI software company working to integrate its tools into physical robots. Going on, they continued, Figure founder and CEO Brett Adcock described the partnership as a huge milestone for robotics.

9:10Eventually, Adcock hopes the partnership with OpenAI will lead to a robot that can work side-by-side with humans completing tasks and holding a conversation. By working with OpenAI, creators of the world's most popular large language model, Adcock says Figure will be able to further improve the robot's semantic understanding, which should make it more useful in work scenarios. Now, overall, this AI robotics space is a huge one. Figure is competing with Tesla's Optimus humanoid robot and a number of other well-funded startups as well. 1x Technologies AS, which also received investment from OpenAI, recently raised$100 million.

9:41Agility Robotics is reportedly testing its robots in Amazon warehouses. And that's just the tip of the iceberg. This partnership isn't actually OpenAI's first flirtation with the integration of its software into robotics. For about a year between 2020 and 2021, OpenAI actually had a robotics research team. In July of 2021, however, they announced that they were shutting down that team and shifting their focus. An OpenAI spokesperson said, After advancing the state-of-the-art and reinforcement learning through our Rubik's Cube project and other initiatives, we decided not to pursue further robotics research and instead refocus the team on other projects.

10:14This is obviously a big shift. Given that OpenAI had started showing off robotics research in May 2017 and then again in October 2019, giving that research up was probably not an easy decision. Although given we know what happened next with ChatGPT, probably the right one. In any case, I think it puts more color on the fact that OpenAI decided to partner with Figure as a different approach to something that they had always cared about. Now this brings us back to the Figure01 slash OpenAI demo from yesterday. Figure founder Brett Adcock writes, OpenAI plus figure, conversations with humans on end-to-end neural networks.

10:46OpenAI is providing visual reasoning and language understanding. Figures neural networks are delivering fast, low-level dexterous robot actions. Discussing the video, Brett writes, the video is showing end-to-end neural networks. There is no teleoperation. This was filmed at 1x speed and shot continuously. As you can see from the video, there's been a dramatic speed-up of the robot. We are starting to approach human speed. Figures onboard cameras feed into a large vision language model trained by OpenAI. Figure's neural nets also take images in at 10 hertz through cameras on the robot. The neural net is then outputting 24 degrees of freedom actions at 200 hertz.

11:16In addition to building leading AI, Figure has also vertically integrated basically everything. We have hardcore engineers designing motors, firmware, thermals, electronics, middleware, OS, battery systems, actuator sensors, mechanical, and structures. Diving a little deeper on the technical side was Figure's Corey Lynch. He writes, We're now having full conversations with Figure 01 thanks to our partnership with OpenAI. A robot can describe its visual experience, plan future actions, reflect on its memory, explain its reasoning verbally. Explaining what's happening in the video, he writes, All behaviors are learned, not tele-operated, and run at normal speed.

11:47We feed images from the robot's cameras and transcribe text from speech captured by onboard microphones to a large multimodal model trained by OpenAI that understands both images and text. The model processes the entire history of the conversation, including past images, to come up with language responses, which are spoken back to the human via text-to-speech. The same model is responsible for deciding which learned closed-loop behavior to run on the robot to fulfill a given command, loading particular neural network weights onto the GPU and executing a policy. In just a few words then, Corey is explaining how complex this actually is.

12:17There are numerous processes of translation to get from human language and the visual input of the cameras on the robot into actual activity. Going on, Corey writes, Connecting Figure-01 to a large multimodal model gives it some interesting new capabilities. It can now describe its surroundings, use common-sense reasoning when making decisions. For example, the dishes on the table like that plate and cup are likely to go into the drying rack next. Translate ambiguous high-level requests like I'm hungry to some context-appropriate behavior like hand the person an apple. Finally, it can describe why it executed a particular action in plain English.

12:48For example, it was the only edible item I could provide you with from the table. Finally, Corey writes, let's talk about the learned low-level bimanual manipulation. All behaviors are driven by neural network visiomotor transformer policies mapping pixels directly to actions. These networks take in onboard images at 10 Hz and generate 24 DOF actions, wrist poses, and finger joint angles at 200 Hz. These actions serve as high-rate set points for the even higher-rate whole-body controller to track. There is a useful separation of concerns. Internet pre-trained models do common-sense reasoning over images and text to come up with a high-level plan, learned visiomotor policies execute the plan, performing fast, reactive behaviors that are hard to specify manually, like manipulating a deformable bag in any position.

13:26Meanwhile, a whole body controller ensures safe, stable dynamics. For example, maintaining balance. Of course, even beyond the demo itself, what's been interesting to watch is the response to the demo. Dan Fitzpatrick writes, This is a I-remember-where-I-was-when-I-first-saw-it moment for me. I've been researching the embodiment of AI a lot recently and talking about them in my work. But this new robot from Figure and OpenAI is on another level. Bilal Sidhu jokingly responded to Figure's tweet saying, Hey, Figure 1, I'm headed out for a bit. Please clean the house, do the dishes, and take the dog for a walk.

13:55Feel free to watch a little Terminator 2 until I get back, but don't get any wild ideas, m 'kay? Some people did try to call BS, claiming that the fact that the robot said uh and stuttered suggested that the whole thing was faked, to which Teslabot Journal pointed out, You're wrong. You can prompt it to converse like how people typically speak. Put that in ChatGPT's custom instructions so you can have that in every conversation. The demo probably used a variant of GPT-4V where the same tricks can be applied. Indeed, overall, the broad assumption that I'm seeing is that the demo is legitimate and that it really does mark a massive phase shift moment for AI and robotics.

14:27Courts even went so far as to argue that OpenAI's new robot is, quote, way ahead of Elon Musk's Optimus. Tesla's Optimus is taking baby steps while OpenAI's Figure 01 is doing burnouts on the track. I'm sure that sort of headline won't inflame Elon at all, given that he's already in lawsuits with OpenAI. In any case, this is going to be an area that continues to see massive and rapid advancement, and so one to watch very closely. For now, however, that is going to do it for today's AI Breakdown. Until next time, peace.