1. Podcasts
  2. Software Engineering Daily
  3. Physical AI at NVIDIA
Software Engineering Daily

20 Oct 2026 · 57m · 19 chapters

In short

Physical AI at NVIDIA—how “world models” move AI from digital text/coding into the physical world (robots, self-driving cars, factories, homes), including planning vs action, safety/guardrails, and the Cosmos model family plus Cosmos 3 Edge for on-device deployment.

Guests (backgrounds)

  • Amit Goel (NVIDIA): 15 years at NVIDIA; leads robotics ecosystem and edge AI product; oversees Jetson-family edge computers and NVIDIA’s physical AI/robotics work.
  • Matt Merrill (host): 20+ years scaling software teams; backend/cloud/distributed systems; architects products and leads engineers at Dept Agency.

Key claims

  • Physical AI models predict future physical states from multimodal sensor inputs and can plan and act.
  • World models output predicted future video/3D and can counterfactually evaluate multiple futures.
  • Robotics needs a “two-tower” design: a lower-frequency reasoning/planning tower plus a high-frequency action/control tower sharing latent context.
  • Training requires new data pipelines (video/action retrieval), heterogeneous compute (simulation + ray tracing GPUs), and separate deployment hardware.
  • Safety uses planning constraints, simulation evaluation of adverse/out-of-distribution cases, and external monitoring redundancy.

Notable examples

  • Self-driving cars (Waymo-like), Nest thermostat, traffic intersections.
  • NVIDIA Cosmos 3 Edge running a ~4B-parameter model in real time on Jetson-class “passport-size” computers.
  • Alpamayo self-driving work: predicting 10 futures and choosing the best.

Written by AI. May contain mistakes.

Chapters

1:48

Understanding Physical AI

Amit explains what physical AI means and how it differs from digital AI.

“My name is Matt Merrill, and I am here today with Amit Goel from NVIDIA, who is head of robotics and edge AI.”
3:04

The Role of World Models in Physical AI

Discussion on how world models function and their importance in robotics.

“Yeah, the big difference between LLMs or in general digital AI and physical AI is that you're working with atoms.”
5:27

The Complexity of Physical AI Models

Exploration of the data and processing requirements for physical AI models.

“Like a self-driving car or even a Nest thermostat is technically physical AI.”
9:00

Real-time Decision Making in Robotics

Amit outlines the workflow for a robot responding to sensory inputs in real-time.

“It just seems like a lot, like a video, potentially audio.”
14:00

Understanding Motor Controls in Robotics

Learn about the complexities of controlling motors in robotics and the need for continuous output generation.

“And that part is very complex because depending on the robot type, you could be controlling 30 to 40 motors and you need to generate actions of all of that synchronously.”
23:14

Hardware Considerations for Physical AI

Understand the hardware requirements and challenges faced by physical AI systems compared to traditional models.

“process it, store it in a way that now you can keep the training data coming in.”
26:29

Imitation Learning and Generalization

Learn about imitation learning and how robots generalize from diverse training environments.

“the horsepower even just to do that simulation is mind-boggling.”
28:00

Understanding Real-World Data in AI

Learn how reasoning models combine real-world data to perform tasks.

“the real world has an infinite distribution.”
30:15

Simulation and Data Collection for AI

Explore the importance of simulations and real-world data collection for training AI.

“to be able to react to different things.”
31:47

Guardrails for Physical AI Safety

Discover how guardrails and simulations ensure safety in physical AI.

“you can drive in that simulated version and test how your robot or your car is going to behave.”
Show all 19 chapters
34:15

Ethics and Open Models in AI

Delve into the ethical considerations and community roles in AI model safety.

“The planning brain is sort of like a VLM or an LLM where you can do the guardrails into that so that it does not plan something that is not acceptable.”
38:21

Cosmos 3 Edge and AI Planning

Learn about Cosmos 3 Edge models and their role in robotic planning.

“So you mentioned the open models, which is really exciting and also a little bit scary because one of the things where my catastrophizing brain goes is like, what happens if this tech gets into the wrong hands?”
42:00

Understanding the Cosmos Model

Explore the capabilities and applications of the Cosmos model in robotics.

“And for somebody who's not familiar with this, I don't really know what that, I think you said it's a model, but I don't really know what that means, Edge.”
44:44

Getting Started with Physical AI

Learn how to begin working with physical AI and robotics as a developer.

“But when you guys talk about Edge, it's like, nah, it's in a factory or it's.”
47:18

Navigating Robotic Hardware Options

Discover available robotic arm options and how to find compatible hardware.

“Is that something NVIDIA produces or is that something that other like third parties produce?”
49:56

Future of Robotics and Programming

Discuss the potential advancements in robotics and simplifying programming tasks.

“As you said, that humanoid or the big arm that they saw in an automotive plant that was carrying the entire car and turning it upside down.”
52:50

Bridging Software and Hardware Development

Learn the importance of collaboration between software and hardware developers in robotics.

“Multifunction robots that can be just taught by telling them what to do, showing them what to do.”
56:00

Exploring User-Friendly Robotics Concepts

The discussion focuses on how robotics is becoming more accessible and user-friendly.

“You got to really get in there with the concepts and now it's becoming a lot more user-friendly.”
56:29

Closing Remarks and Gratitude

The host expresses gratitude towards the guest and shares parting thoughts.

“And I really enjoyed being here and hope you have a good rest of your week.”
A man walking through an airport, asking his phone a question

Ask about this episode

Get the gist, explore an idea or ask about something said.

Claude

On Software Engineering Daily, what are the key points of this episode?

Searching your listening

Transcript

Automatic transcript. May contain errors.

0:00In recent years, the main impact of AI has been on the digital world, and most notably software engineering. However, its longer-term effect on the physical world may prove even more profound, reshaping how cars drive themselves, how factories run, and how robots move through homes and warehouses. World models are at the center of the movement to bring AI into the physical space. These models take in observations of the current state of the world and predict what comes next, which lets a robot plan its next move or car anticipate the road ahead. NVIDIA has been building a family of these models called Cosmos and sits at the frontier of physical AI and robotics.

0:42Amit Goel has been at NVIDIA for 15 years, and he is currently the head of robotics ecosystem and edge AI product. In this episode, Amit joins Matt Merrill to discuss what separates physical AI from digital AI, how world models plan and predict, the two-tower architecture that lets a robot reason and act at once, why training on video demanded a new data pipeline, and how any developer can start building in this space. Matt Merrill is a software engineering leader with over 20 years of experience building and scaling software teams across enterprise and product-focused organizations. His background is in backend development, cloud architecture, and distributed systems design.

1:26He currently architects and delivers software products and leads a team of engineers at Dept Agency. You can learn more about his work at code.theothermattm.com.

1:48Hello, everybody. My name is Matt Merrill, and I am here today with Amit Goel from NVIDIA, who is head of robotics and edge AI. And I'm really excited to be here to talk to you about physical AI. So why don't I turn it over to you, Amit, to introduce yourself? Hey, Matt. Great to be here. I'm Amit Goyal. I head up our robotics ecosystem at NVIDIA, which means my team gets to work with all the developers who are doing some amazing work with AI and robotics. And I also lead our product team that builds the computers for running the AI on the device on the edge. It's called the Jetson family of products.

2:31I've been at NVIDIA for 15 years. We're very excited to talk about physical AI today. All right. So I am super excited to talk about this because I think that a lot of folks listening here are probably at this point pretty familiar with the software side of Gen AI and LLMs. And what I kind of want to do today is just start from zero, right? Like, and that's actually, I'm not playing a role here. I purposely didn't dive into this deep so that I could kind of be fresh. So when NVIDIA or anybody says physical AI, what do you mean? What makes it different from LLMs and coding agents that most of our listeners are using every day?

3:09Yeah, the big difference between LLMs or in general digital AI and physical AI is that you're working with atoms. And what that means is you are dealing with sensors, you're looking at spatial information, 3D information, you're looking at videos, you're understanding dynamics of the world. So physical AI essentially is a model that can take as inputs the information of the world around you and predict what's going to happen in that world. So everybody is probably familiar right now with self-driving cars. If you live in San Francisco Bay Area, you'll see the Waymos driving around. So that's probably the closest way to think about physical AI in the real world, right?

4:02You have a machine that is taking the information from the cameras, the LIDARs, and the actuators in the car and making a decision on what it should do next based on what the driver wants out of the car, right? So it's taking the data from the observations from the physical world and determining actions that have to be executed in the physical world. And it's been around for a while. The simplest form is probably your Nest thermostat that's controlling the temperature of things in your house based on your activity. And even like a small traffic intersection that's observing, what's the activity that's happening and controlling that particular traffic light.

4:44So physical AI has been around for a while, but it's been very narrowly defined for one particular use case. So now what's happening with the really large models and general models is that you can have a single model that can understand all of this. And similar to what happens with LLM, right? You don't have a sentiment analysis model, you don't have a translation model, you don't have a summarization model, you have one big model that understands everything in the textual world and you can program it to do what you want it to do. We are getting to the point where we can build this general physical intelligence models that you can use for a lot of applications in the physical world.

5:27Got it. Yeah, that was one of my big questions. I never really thought about that. Like a self-driving car or even a Nest thermostat is technically physical AI. But when I was prepping for this, seeing some of the stuff that is currently happening is so much more advanced than that. And it makes sense that in the past year, the explosion and what these models can do enables a lot of this. So let's dive in and try to give folks like a mental model of how these things break down. And so one thing that I was reading about prepping for this was a concept of a world model. Can you tell folks about that and like how that fits in here and how it might drive one of these devices?

6:05Yeah, so the way to think about the world model is in the context of what are the inputs and what are the outputs, right? So for a world model, the inputs are observations of the current state of the world. It could be a sensor coming data from a motor of a car, of a robot. It could be a signal that is audio command that has been given as an input. It could be a textual prompt. And of course, it has to have the cameras and the visual representation of the world that's coming into it. The next question is, what does the world model predict, right? And inherently, the world model, because it's dealing with the physics of the world, is a physical AI model.

6:50It is multimodal because the real world has multiple modalities, right? We are hearing, we are listening, we are processing information. So it's multimodal, many inputs and also many outputs. the typical output of a world model is a video which given a prompt given a current state if the world model understands what happens in the world it can predict based on what you told me this is what's likely to happen in the case of a car you can say i'm going at this particular speed this is what i see is happening around me tell me what is likely going to be the state of the world in the next five seconds.

7:31Based on that, I can then control the car to what action it should take. Same analogy applies to a robot, right? You asked me to pick this thing and put it over there. This is what I see is my current state of everything around me. This is the object that I see. How should I move? Should I go left? Should I go right? Should I go backwards? Do I need to bend? It can predict what is the next few things it needs to do. So the world models can predict the world around you. It can do a 3D representation of the world. It can also understand, hey, I want to be there. This is my objective. How do I get there?

8:10What we think about in our head, it's called planning, right? So the world models can predict the plan to go from the current state to the desired state. So this is essentially the planning function of the world model. And the other thing is you can say, hey, this is my current state. I'm planning to take this action. Can you tell me what is going to happen if I do this? And that is super important for robotics, right? Like if you can say, I'm going to do this and tell me what's going to happen, you can actually decide whether that's the right thing to do or not. So rendering the entire world around you, given an action, predicting what the world is going to look like after you take that action, or given the current action and the desired state, predicting the action I should take to get to that desired state.

8:56These are all of the outputs of a world model today. That sounds like it could be, especially for something like a self-driving car, an absolutely enormous model or set of models. Yeah. It just seems like a lot, like a video, potentially audio. I don't know. Just a lot. I mean, I'm almost having a hard time wrapping my mind around how much stuff is going into that. No, you're right. I mean, the amount of data that gets fed into a world model is orders of magnitude more than what you would feed into a text LLM, right? Because just a video of 15 minutes is several gigabytes of data. So, of course, the amount of data that gets processed in a world model is a lot.

9:41But at the end of the day, I think the size of the model we see varies depending on what you want the model to do. You were asking about physical AI and difference between LLMs. And one of the things is that you have to run some of these models on the device itself, on that particular robot or on that particular car. You do not have the luxury to say, it's okay if it takes one minute to get back to it with an answer. I'm going to throw all the compute available at it. So you need to think about how you're going to make distilled version of these world models that can run on device. but for something like evaluation, which can happen offline, right?

10:22If you're training a model, you collected some amount of data, but you want to train what it is going to do if it was in a particular street of New York. You can just ask the world model to give it a particular image and say, can you create a world around me and I want to drive this thing in New York, right? That part is super heavy, but that part can also run offline. You can throw a lot of compute at it. So at NVIDIA, we have been working on the Cosmos models and a variety of those exist. And we have many different sizes of the model, starting from things that can run onto the edge all the way to really, we have nanos and super models.

11:02They can be used for creating data. They can be used for evaluating things offline. So you're a hundred percent right. These models are processing a lot of data. You anticipated my next thought comment, which was just the speed. These things can't wait. It's not like me waiting for Fable to analyze my next presentation or something. Yeah. Okay. You cannot just leave it around and come back and see what happened. Especially if you're deploying it on the device itself. Yeah. That's amazing. So I'd love to walk through, you know, you get these interview questions of, I remember back in the day, it used to be like, what happens when you press enter on Google?

11:40Right. So I kind of want to give you this for one of these situations. So we can pick a type of robot or whatever. What happens in 100 milliseconds between that thing capturing, let's say, an image or some sort of multiple sensor inputs and then a motor command going out of that thing to perform an action? Yeah. So typically, if you think about, let's say, a robot, the moment you are capturing the information from the robot's camera, typically you need more information than just the camera. You need to also understand what is the state of each of its actuators. You need to understand some of the other sensors like IMU, which is an inertial measurement unit that tells you the pose of the different parts of the robot.

12:29And once you capture the information, first you need to synchronize all of this. What that means is my camera is observing at this particular time. At the same time, what was the state of my motors? What was the state of the IMUs? and you need to fuse them together. The next thing you have to do is, if you're given a task, let's assume we're running a vision language action model. You need to tokenize this data, all of this data, the data from the current frame, but also the data from the past frames because your next action is dependent on not just what you're currently seeing, but also a little bit of history.

13:06So you have to take the data, you have to tokenize it, and then you feed it into this model, which is now going to do two things. First thing it has to do is it has to do a planning, which is the task of taking my current observation, my past observations, and my objective function. What should I do, right? If I was asked to pick up that particular cup, do I need to move left? Do I need to move right? Do I need to go up or down? How far do I need to reach? So this is the planning task. And today, this is something that is done by an autoregressive model that will predict the sequence of tasks that you need to do based on the objective function and current input.

13:50Now that's your reasoning and planning brain that kicks in. And the planner tells you, okay, based on the task that's given to you, this is the next step you need to do. Now comes in the second piece of the model, which is generating all the actions. And that part is very complex because depending on the robot type, you could be controlling 30 to 40 motors and you need to generate actions of all of that synchronously. You cannot serially generate that, right? You need to control the output. And one thing that's very unique about physical world is that a lot of things are continuous. You have to control the current and you cannot send it for like this much current.

14:32it has to be continuously generated. So the kind of models that you need is different than what you need to predict a discrete output. A sequence of text versus the current for 50 motors that you have to send it at every millisecond. So the frequency at which the output must be generated is very different for a robotic and a physical AI model. So you have to make these two models work together. A reasoning model that is planning can exist at a low frequency. The inputs of that goes to another model that actually generates the motor control commands. And that one is running really, really fast because it has to control all of that at high frequency.

15:18And once you have that output, typically that's happening in the brain. But typically at the end of that, it goes to a small computer that is attached to the motors, which does your PID control based on the input that it got from the model. I'm not familiar with PID. What's that? So PID is the closed loop control, that low frequency control loop. That is, you have a target position, you get the current position, and you control the current. So that is running right onto the motors. It does not need to know what's happening on the other 30 motors, right? Because its objective is to control that one motor.

15:55And that runs on a very small computer or a microcontroller. I'm thinking of this in terms of like a BIOS or something like that. Like just very simple. Yeah, it's just a very simple firmware that's running. It's just taking commands. I'm listening to this and I just can't help but think like this is the world's most insane state machine ever. That's just so much to keep track of. Yeah. And so you talked about this in a way that almost seems like there's two different models happening. There's a reasoning and a planning, but yet they're all sharing the same context, right? That's right. How does that work?

16:33How does it share that? Yeah, so that's the interesting part. There are many ways these models are being architected. And one of the ways that we are looking at this is having a shared latent space where the attention from one model and there's a cross-attention between one model and the other model. The embeddings are shared so that you have a really high bandwidth communication between these two brains. If you look at the latest Cosmos 3 family of models that NVIDIA released, we have these two heads, a reasoning head. And on top of that, we have the reasoning tower and the action tower on the output side.

17:15And they talk to each other. They have shared context in which they exchange information in the latent space. If you were running them distributed, you may want to take the output of one and decode it and send it to the other one. and again, Encore did, but that slows down the bandwidth, right? As I was telling you, for a robotic system, they need to operate really in a closed loop. So that's why we build Cosmos with that shared context so that both these can coexist and exchange information with very high bandwidth. I'm almost picturing this just like, I don't know, I'm trying to think of like a CPU analogy, but it's logical.

17:52It's almost like a logical division inside the same hardware. Is that correct or do I have that wrong? You're absolutely right. Think about like in a physical space, we have a shared memory, right? Where you have two different processes, you don't want to send the data over because it based your bandwidth. The same thing we're doing at the model level now, right? Where in the latent space, we can share the information and the context between these two models. So they can operate, although they're operating in different frequencies, they can share the same context. got it that's really interesting i was going to ask about that two tower design because that was something i came across and it didn't make sense to me but as you explained that reasoning step that does make more sense yeah and you know the best another way to think of it is like is an architect which is planning everything that you need to build and then there's an executioner which is given the plan is doing the execution they both need access to the same information but what they do with that information is slightly different.

18:54This episode of Software Engineering Daily is brought to you by VariaCode. If you're an engineering or IT leader, you know how painful this is. An open rec sits unfilled for three months. Your tech lead is burning out, covering the gaps, and your product roadmap gets pushed back again. VariaCode solves that. For over 10 years, they've been embedding senior Latin American engineers directly into U.S. engineering teams in under 10 days at up to 40 % less than local hiring. We're not talking about generic contractor pools. These are pre-vetted, high-quality engineers with real on-call and SLA experience.

19:28They work directly in your Slack, Jira, and daily stand-ups in your exact US time zone with zero friction. If you're ready to stop losing sprints to open recs, head over to variacode.com slash S-E-D right now and get a custom shortlist of senior candidates in 48 hours. That's V-A-R-I-A-C-O-D-E dot com slash S-E-D. Every visitor on your platform looks the same at first. Can you trust them or are they a threat? Fingerprint lets you answer this question right away. With one API call, no model to maintain, and implementation in minutes, Fingerprint recognizes bots, AI agents, VPNs, tampering, and over 100 more unique signals to give you durable risk scoring and threat detection in real time.

20:16Even as human and agentic activity blends together, you can identify suspicious traffic faster and build safer and more secure sites and apps. Over 6 ,000 companies rely on it every day. Join them now at Fingerprint.com. Think about your mobile app source code. Once it hits the app store, it's out in the wild. And without the right protection, decompiling is easy for malicious actors looking to steal your IP or tamper with your software. That's where GuardSquare comes in. GuardSquare provides the highest level of mobile app security for Android and iOS applications and SDKs. Their advanced tools integrate seamlessly into your CICD pipeline.

20:59We're talking polymorphic, multi-layered code hardening techniques and automated runtime application self-protection paired with mobile application security testing and real-time threat monitoring to deliver the highest level of mobile app security without compromise. Don't leave your hard work exposed. Secure your mobile applications today. Go to guardsquare.com to learn more. So one of the things here that's like, I mean, we talk in textual models about using GPUs because they're really good at this task, but I can't help but think like this requires just so much more horsepower, so to speak, than what those need.

21:39So if you can simplify it, what is the difference in the hardware that's powering these things? There are a few things that is different. As I mentioned, when you're thinking about the world model, the number of modalities that you're dealing with is quite different. What that does to your architecture is when you're training a model, when you're dealing with the text inputs, the bandwidth that you're processing for data itself is not that high. when you are working with the world models and you're training a world model, you need to retrieve the images. You need to retrieve the actions or the motor control commands.

22:20You need to retrieve the text annotation for all of that in parallel and bring it into your model. That creates pretty unique challenges from the hardware perspective as to what do you keep in the storage? How do you manage the data? we in fact had to go ahead and build out a completely different stack in order to process this data. Because GPU's resources are super scarce and precious. And a lot of this preprocessing typically was done on a CPU. And so you would have your model waiting for the data to do the training, but it's just not possible to do that with the existing pipelines that were built for text-based model training.

23:08So we created something called Curator, which was the ability to now take large amounts of videos, process it, store it in a way that now you can keep the training data coming in. So that's on the data side of things. On the hardware side of things, when it comes to training these models, as you know with LLMs also, there are multiple phases, right? There's pre-training, there's mid-training, and then there's post-training. You see a lot of post-training happens with using the reinforcement learning. You're using different tools. The same is true for world models. But when you're training a world model, let's say it's a robotics world model, how do you verify it?

23:50How do you do the reinforcement learning? For a coding model, reinforcement learning is you go run it, compile the code, verify it works or not. For a world model, its job is to simulate the word. you actually need to go to a physics simulator as a tool, right? And that physics simulator is best run on another kind of GPU, which has the ray tracing cores that can do the sensor simulation. So it becomes a heterogeneous computer architecture. On the one side, you need something that can process all of this data. Then, of course, you have your transformer model that needs to get trained. And then after that, you have to continuously call a simulator where you go and do the reinforcement learning and testing whether if it's working or not.

24:36So it becomes a pretty complex architecture. And inherently, it has become the heterogeneous computing problem. That's one other thing that developers who are used to LLMs get access to a whole bunch of computers, all latest Blackwell clusters. They can go and just train the models, test the models, evaluate it, serve it all on that cluster. But physical AI requires actually three computers. You need a computer for training the model. You need a computer where you do simulation, either to create data or to do reinforcement learning post-training. And then last piece of it, deployment. You cannot deploy it on the same infrastructure that you trained it on.

25:21You need to deploy it on a small computer that's sitting inside the brain of a robot. So you need to go do some more work on the model quantization, distillation, compression to go deploy it. So it requires a very different way to think about the compute hardware that you need if you're building a physical AI model. I'm joking here, but I'm imagining you being at like a cocktail party talking with a software developer about textual LLMs and going, that's cute. Yeah. I'm just kidding. Yeah, I think Jim Phan, who leads our research team for physical AI and humanoid robotics, he often jokes like, I'm always envious of my friends in the LLM world.

26:02Their problem is I don't have more GPUs. And my problem is like, okay, first of all, the data that I got, like half of the sensor data is missing. The synchronization is not there. and then I'd have to, I get the GPUs, I train it, but next I have to find a simulation environment to go test it, deploy it onto the real robot, the cables are breaking, you know, so it's a very different world, yeah. That is something, the simulation piece of it, never even, I mean, yeah, the horsepower even just to do that simulation is mind-boggling. Yeah. So switching topics a little bit, like when these things are deployed, they need to react to somewhat unpredictable things, right?

26:43Like they effectively need to learn. And I read a little bit about imitation learning and watching humans do tasks. And can you say a little bit more about how that happens in the, at least in current research and out in the real world and how that operates? Yeah. So two things that's being done right now to make these models more generalizable. First is to collect data in diverse environments. So this is, if you are supposed to assemble, let's pack a bag of shoes. Often the size of the bag of the shoes is going to be different. The size of the box is going to be different. So one way to do this is you have a lot of people collecting data in different environments, in different factories, bringing the data of doing the tasks in different environments and training the model so that it now has seen enough distribution so that the real world falls somewhere within that distribution.

27:44So that's why when you see imitation learning, that's what is happening. We are collecting the data, increasing the distribution in which it's been collected. But as it turns out, however much data you collect in the real world, the real world has an infinite distribution. It's continuous. So you can collect it in 10 different lights, but there's going to be something that's always going to be out of distribution. That's where the second part comes in, where you need to think about having a reasoning model paired with a model that has seen enough distribution. They go together, right? You cannot just have a reasoning model, but not have enough real-world data because the reasoning model can tell you this is closest to this particular real world distribution and then you can go act on that, right?

28:36So the reasoning model is learning from data in the wild. It's using all the text data that's available. It's using the video data that's collected across different domains, right? And learning from that. So combining this pre-training with data in the wild, textual video data, along with a real data, action data, which is people doing the task, robots doing the task, combining these two, the world models are now trying to solve this problem of how do I know enough that I'm physically capable of doing the task? I know how to pick up an object. I know how to slide something. I know how to bend over and perform a particular task.

29:22And how do I do it in different environment in a different context, right? So both of these things are being trained together. The first one, how do I do a certain particular task is being done combination of either imitation learning and reinforcement learning. And then how do I do this in any environment? This is where the data in the wild is playing a role. Okay, so the reasoning models, are these very task specific? The reasoning model, this is where the generalization is coming from, right? So you're trying to collect diversity of information. Your robot might be designed to work in a factory, but you're putting in all the data from homes, from outdoors, from indoors, because that's when you build a reasoning capability that can generalize and you can draw parallels.

30:10I see. Yeah, that makes sense to me, that you need to give it tons of different inputs to be able to react to different things. So, okay. Yeah. Because if you were just looking at factory, you would probably not see a cat or cow come in. And so, you know, when you're doing the reasoning model, you expose it to all elements. This is a little random digression. I was on Reddit the other day and somebody posted, I live near Boston. So it was on the Boston subreddit. Somebody posted a picture of this guy with his wacky looking backpack on. It was like a crazy backpack and people were making jokes like, oh, he's the rocketeer or whatever.

30:46And somebody was like, no, I think that they're gathering world data for AI models. I see you nodding your head that that's probably what that guy was doing. Yeah, one of the things that is happening is simulation is important, right? As I said, for physical AI, because that's where you can safely test your models. That is where you can do reinforcement learning faster than real-time speeds. But the challenge with simulation models is the cost of creating that simulated world is quite high. So what you would probably likely saw was somebody carrying a backpack with some cameras and LIDARs. And what they are doing is capturing the real world and passing it into a model that can create a 3D representation of that world.

31:36What we like to call real to sim. You take the real data, run it through a model and creates a simulated version of it. Now, once you have the simulated version of it, you can drive in that simulated version and test how your robot or your car is going to behave. You can use that and do augmentations. What if this building's colors were different? What if the lighting was different? What if the road was narrower? So you are closer to the actual distribution. And this data is often used for post-training. This is where, you know, I want my car to operate really well in this particular block. Okay, I'm going to collect the data and let my general model post train in this world.

32:22Now it becomes really good at that environment. That's really interesting. That's actually the opposite of what I might have thought. So thank you for clarifying. Cool. One thing that just came to mind here is, you know, like hallucinations and text models or, you know, you just kind of chuckle at them. But hallucinations in a physical space sound a little scary. I would imagine that there's better guardrails for things like that. And I'm curious how you handle that in the physical AI space. So one of the things that is important for physical AI is, as you mentioned, that if you're out of distribution, if you're hallucinating, the impact is pretty significant.

33:04So you have to do two things for that. either you can make your model really conservative and says, okay, this is out of distribution. I see something that I've not seen before, or this is not part of my learning curriculum. I'm not going to act on it. The other thing you can do is, well, I'm going to act on it. And in fact, the way I'm going to make sure that I know how to handle these out of distribution scenarios is, I'm going to actually give those and simulate in that environment. I'm going to evaluate my policies or my models in a lot of different environments, which some of them may not even be real, and then score them on their performance.

33:49So one of the ways to make sure that your robot policies are not hallucinating or if they are out of distribution, they're still safe, is to expose them to those adverse environments in simulation and see how they perform. That's one of the ways to kind of make sure your policies and the models that you're training are within the guardrails. The second thing also is, you remember we talked about a planning brain and an action brain, right? The planning brain is sort of like a VLM or an LLM where you can do the guardrails into that so that it does not plan something that is not acceptable. If it does not plan, then the action head cannot, it's operating on the plans that are coming out of it.

34:35How do you put the guardrails around your planning? It's very similar to how do you put guardrails around an LLM to not give you an answer or information that is not suitable. So you can constrain things at the planning stage. On the action stage, again, it is mostly learning in domain. You can go to simulation and test it. And the third thing that we are also thinking about is when it comes to safety and actions is you have to have some level of redundancy, some level of external element that is observing and verifying. So, because that kind of separates the concerns, right? The person that is acting and the person that is monitoring, if there's another AI that can observe what is happening and then have a way to control the robot or the car, that's another way to manage safety, right?

35:34So I think it's a combination of all of these things that are coming together to make sure that as we are building these models, as they go into the real world, we can make sure that we're not being extremely conservative and not being able to do half of the tasks. We don't want that, right? But we want them to be able to do that safely. So we are bringing this outside-in element of safety where you're monitoring, you're looking at it and making a decision. You make sure that your reasoning brain, planning brain has the guardrails and then extensively test in simulation. That's one thing I wanted to dig in.

36:13When you say it evaluates those plans, it makes sense in training that it does that. But I want to clarify, is it also doing this in the real world too? Like if it is unsure about its action, is it actually going to run simulations in real time to evaluate which one may be the best? That's the biggest value of the word models is. They can do counterfactuals. They can simulate, if you look at the work that we published recently with our self-driving car team called Alpamayo, it's a reasoning model. And within that, it can predict 10 different futures and choose one of them. Because it is a world model, it is not limited to only the right thing.

37:01that it's not just imitation learning on, you know, this is the right way to do it. It can explore 10 different directions, 10 different outcomes, score them, and act on just the safest and the one that gets you the most to your objective function. So that's the beauty of the world models. It is doing that at evaluation time. And it's important to have these models open for that reason as well, right? Because what is acceptable in one part of the world versus what's acceptable in another part of the world? Thinking about driving rules, right? What is an acceptable driving behavior in Boston is likely different from the acceptable driving behavior in the Bay Area.

37:46And if you go to a different country, of course, it changes. So taking these word models that can predict all of the different counterfactual scenarios, and then post-training them to choose, you know, what is acceptable is important. When I heard you talk about the security, I can't help but think like, this is security in layers, security in the planning stage, security in the execution stage, evaluating options. So that's really good. And that is not something that, well, I don't know. I was about to say that's not something you hear too much about in the LLMs, but that's probably not true.

38:19We're starting to do things like that as well. That's right. So you mentioned the open models, which is really exciting and also a little bit scary because one of the things where my catastrophizing brain goes is like, what happens if this tech gets into the wrong hands? right like and i know like you're not a soothsayer we you know we all at least you know our listeners have the best of intentions but like is there anything being done in the research community to help get it get ahead of this in terms of ethics legalities and just you know human safety in general yeah i think fundamentally to think about it if these models are open right as long As long as we have more people who have access to them with the right intention, they will outnumber the people who will have access to this with a bad intention.

39:10Hopefully that's the world we live in. So in that case, I think being open and accessible actually gives us more tools for everybody to build systems that can monitor them. As I said, the thing with safety is you have to assume that, what if it happens? How am I going to counteract? I think we saw what happened a couple of weeks ago with the agent getting attacked and hugging face. They were able to use an open model to actually go and triage that and fix it. I think that's why we believe that having a pretty strong, strong, open ecosystem is vital to make sure that there are checks and balances.

39:58It's vital for people to create models that are better aligned to their philosophy. And it's especially true for physical world. I was off kidding with somebody that me and my wife can't agree on how we load a dishwasher. right like that is no right way so we need an open model where you know i can she can reinforce train it to do it the way she likes it or i can do it so if it is a you know somebody decides for us how the dishwasher is going to be loaded there's no way that's going to work i mean do you have a dishwasher loading robot like is that are you doing that at home like because i'm totally jealous like is that a perk of working in a video no okay not yet but i'm looking forward to it i think it'll happen soon give me a prototype yeah that's really interesting and tell me if i'm completely off here but like i almost think of this in terms of like human morals right like you the deeper the moral is set in someone the harder it is to compromise it and i think what you're saying is that these models are open and they are deeply ingrained in ethical things it's very hard to unprogram that is that kind of what you're saying i think if these models are open you can introspect them and you can steer them.

41:18If they're not, then you can put monitors, you can put all kinds of things, but you're limited in what you can do. And then you also introduce a single point of failure as well. So I think diversity and choice and one model checking the other one is the best way for making sure that you have a better answer to the security problem. I can see what you mean by the transparency being huge because if there's a bad guy, you don't know what he's thinking. But if you can see what he's thinking, that's a little bit of a different story. It's really interesting. I want to talk a little bit about, so you released Cosmos 3 Edge.

42:02You mentioned this Cosmos model. And for somebody who's not familiar with this, I don't really know what that, I think you said it's a model, but I don't really know what that means, Edge. So can you talk a little bit about what that is? As I was telling you earlier, the world models can do many different functions. They can create, predict random worlds in which you can evaluate your policies. They can take real world data and augment it to change the environments. But some of these things are pretty compute intensive. But one of the functions of a world model is also, what we were talking about earlier, is that they can be the planning brain for your robot because they can predict a lot of different scenarios and pick one that is most appropriate that gets you to their objective function.

42:54That model needs to run on the robot, right? That part of the world model. This model is not really generating pretty videos, 10-second long videos. It doesn't have to do that. this model does not have to, you know, you can distill it down to certain particular domains that you care about, but it should be able to, you know, do the task of predicting the future and controlling the robot. So we took our Cosmos 3 model, which comes in different sizes, and we created a 4 billion parameter version of it. And the main purpose of the Cosmos 3 Edge is that it's a world model that can take the inputs as text or actions, or videos, and generate the next action that the robot needs to do.

43:43And by making it a full built-in parameter model, we can now deploy it onto the robot. So it can be used as the policy of the robot itself. And we optimize it. It can run real-time on a Jetson Tor, which is a computer which is the size of a passport that has 2 ,000 teraops of compute. Did you say the size of a passport? Yeah, it's the size of a passport. That's what is inside a lot of robots. If you look at the 1X robot, the Boston Dynamics, HumanEye, they are all powered by this small computer, but it has 2 ,000 teres of compute. It can run 10 billion, 4 billion parameter model on it. So that way it can do the reasoning.

44:30You were asking about counterfactual, it can do all of that on device. That is wild. That is really cool. So I can see why that's a big deal. Some mini model that can run on robot. On the robot. It's so funny because when I think Edge, I think of, you know, like, oh, CDN distributed geographically. But when you guys talk about Edge, it's like, nah, it's in a factory or it's. Yeah. Yeah. It's battery powered, you know, running on device, running all the time. Yeah. Yeah. And the power consumption of those things has to be reasonable, too. Not only is it small, but the power consumption. Wow. Yes.

45:06Yeah. That's crazy. okay so i'm a normal developer i want to start playing around with this next weekend how do i get my hands on one of these passport size computers they're only like 20 bucks right i'm just kidding how accessible is this like i would love to play with something like this like how does a normal dev get into this yeah i think it has become much easier in the last i would say 12 to 18 months. In the past, the hardware was very expensive because it was designed for roboticists. And one of the things we started working with the team at Hugging Face was to target and enable all the software developers who are used to writing programs to start programming robots.

45:53And to do that, you need to have a robot. You're building physical AI. Of course, you can do a lot in simulation. That's the first step. You want to get started with this. You use your existing computer, you download a simulator, and you download an existing model, connect it together, and see how it works. And this is all just running on your laptop? Yeah, you can run on a decent laptop today, which has a 4090 GPU. You can get a DGX Spark, run it on that. So you can run the simulator, you can train the models, test it. But the interesting thing about physical AI is that experience of touching a real thing and dealing with the connection and the motors and the variations there.

46:43So now it's like for a couple of hundred dollars, you can get a robotic arm that you can train, you can teleoperate it, collect data, format it, publish it to Hugging Face, train a policy, Convert it, deploy it onto the small Jetson and let the robot do its thing, you know. And all of that, I would say like for you to get started, if you have a good computer, a thousand dollars budget, take a couple of days and you can have your first physical AI model doing things. Wow. What are the arms? Is that something NVIDIA produces or is that something that other like third parties produce? It's just third party.

47:26There are lots of projects. If you're into 3D printing, you can just order the material, 3D print these things, get it together. That's really cool. Pretty big open source ecosystem. If you look at SO101 ARM, that's the most popular with the makers right now. You can 3D print it, lots of vendors do the kitting of all together and they ship it to you, but you still get the joy of connecting it and building it all together. So that's there. And then, you know, you can start from there and go to a humanoid and get one of these, you know, humanoids for$15 ,000,$10 ,000 these days that you can make it do whatever you want, dance or, you know, or clean your home.

48:07Wow. When somebody's like looking into the, like those, I don't know what we call them, like classes of product, like what would I Google for? Or what would I ask Claude for in terms of trying to find that? That's something that to me, if I were to step away, I'd be like, okay, I guess I could search like open source robotic arm. But I feel like there's probably something more specific that people could search. Yeah, I think if you're interested in using this new age of robotics, which is end-to-end models, right? That is taking pixels in, controls out. You want to look for models that are supported by, that support NVIDIA's like CRUDE or physical intelligence or, you know, these popular AI models, right?

48:56So those robot hardware that has been tested with these models. The way to search for it is look for different robot policies, AI models that exist, and say, I'm looking for a hardware that works with this particular model. You don't want to have to customize the model for a different hardware. Yeah, that's a bridge too far for somebody trying to get into it. Exactly. That's why the hugging face checkpoints are really good because they already have checkpoints for this SO101 arm. If you're looking at action-chunking transformer model, you are looking for a Groot model, you're looking for any of the popular AI models for robotics, you can find a checkpoint for specific hardwares.

49:41And that's one way to go find it because then you can just cobble it together and have your first rollout. And then you can say, okay, I want to go customize it. right yeah oh that sounds fun okay yeah cool i hope that inspires some people all right let's close it out i got a couple of open-ended what if questions so what sounds like science fiction right now that you actually think might come true in five years with this if you think about robots today what comes to your mind probably that boston dynamics thing that can like you know jump upstairs and jump up on platforms and things like that yeah yeah i think for most people when to think about robots, it's the embodiment that comes to mind.

50:24As you said, that humanoid or the big arm that they saw in an automotive plant that was carrying the entire car and turning it upside down. But I think in five years, we will see definitely robots that are more around us, more ubiquitous, that are multifunctional and can do many things. Robots today are purpose-built. You think about robot, you think about either that humanoid, or you think about a Roomba, or you think about that industrial robot. You don't think about a brain that can do 10 things for you. You think about the body, not the brain. I think in five years from now, I expect that we will have robotic embodiments that are going to be capable of doing multiple things.

51:15So when you think about robot, you will not think about the body, but you will think about, oh, that thing that can do all of these tasks for me. The embodiment is just, you know, an artifact. So that would change. You would not think about a robot for A, but you would think about a robot that can do a class of things for you. The other thing that's going to change is for the robotics people is the task of programming was very intimidating. It's like you need to be an expert. You need to be either a technician or a PhD to be able to program the robots. That is going to look very different. With these foundation models, you would be able to likely show a task.

51:59Do it one time. Do it with the robot two times. And teaching the robot with data is what is going to be magical. It seems impossible if you tell a robotics person today that, hey, programming a robot is just like, go show it. Hey, this is the task I want you to do. Here's a video. Can you go do it? That seems like magic today. But that's going to be how we're going to tell robots how to do a task. We're going to either demonstrate it, we're going to teleoperate it just one time, two times, and the robot will go do it. I mean, if we didn't have this inflection point with software-based LLMs in the past year, I probably would not believe you, but seeing what happens in the past year, I believe it.

52:43Yeah, it's crazy. So I think those are the two things that I think would happen. Yeah. Multifunction robots that can be just taught by telling them what to do, showing them what to do. Right. You make such a good point about, I mean, for the history of robots, it's always been the thing with the arms and the face and it talks. Yeah. It doesn't have to be like that, right? So, yeah, I can't wait until I have a disembodied arm robot that can do whatever I show it to do in my house, like load dishwashers. So, AI is reshaping a lot of software engineering jobs, right? I mean, it's a little scary and also enthralling at the same time.

53:24Some people are wondering where their careers are going to go. And, you know, maybe you've inspired some people to look into this. So, if I'm a senior engineer today, I have no robotics background, but I have a heck of a lot of software background. what would you start learning besides the basics that we just talked about playing around like what what's the best way to really get into this the best way if you want to you know get into robotics it's just starting i believe with simulation because that is where you know it still feels like you're working with software but you get to experience the physical challenges where you don't think about, you get exposed to gravity, you get exposed to friction, you get exposed to all the real world challenges that you will encounter when you get to the hardware.

54:20So I think that's the first thing, is get out, start working with the simulator, get exposed to those terminologies and the data that you're not used to when you're working with code and software. And then once you get comfortable with the simulation, then deploying it on hardware, as I mentioned, there is a lot of different hardware modalities that are available now where you can deploy the models that you've trained. You can get one of those and start deploying them. You have to get physical when you're working with the physical AI. So don't be intimidated by it, embrace it. And I think the more and more software developers and hardware developers work together, I think we'll have better systems.

55:11Today still, I feel that a lot of times the hardware developers who are building the mechanical team is a little bit operating differently, decoupled from the software team. But I think when they start understanding each other's languages, which is what would happen if, you know, as software developers get into physical AI, they need to understand, you know, all the mechanical constraints. That would be pretty exciting. And I think we'll have much more capable systems. Before we talked, I was wondering if you're going to say like, oh, you need to go get a PhD in math or something. I'm joking. But like, I think what I'm hearing is like, no, that's starting to change.

55:52Just like in software, when you wanted to mess around with, let's not even talk about LLMs, but machine learning, it used to be, you got to know a lot of math. You got to really get in there with the concepts and now it's becoming a lot more user-friendly. Sounds the same thing. Yeah, I think the open source ecosystem around robotics is pretty strong. You have everything that you need from tools to assets to models, it's all available. So you can really go ahead and get started. All right. Well, this was fantastic. Super, super interesting. I really thank you for being here and explaining this to us.

56:28This is great. And I really enjoyed being here and hope you have a good rest of your week. You too. Send me that dishwashing robot when you get it. Yeah, I will.

56:55Thank you.

From the publisher

In recent years, the main impact of AI has been on the digital world, and most notably software engineering. However, its longer-term effect on the physical world may prove even more profound, reshaping how cars drive themselves, how factories run, and how robots move through homes and warehouses.

World models are at the center of the movement to bring AI into the physical space. These models take in observations of the current state of the world and predict what comes next, which lets a robot plan its next move or a car anticipate the road ahead. NVIDIA has been building a family of these models called Cosmos, and sits at the frontier of physical AI and robotics.

Amit Goel has been at NVIDIA for fifteen years, and he is currently the Head of Robotics Ecosystem and Edge AI Product. In this episode, Amit joins Matt Merrill to discuss what separates physical AI from digital AI, how world models plan and predict, the two-tower architecture that lets a robot reason and act at once, why training on video demanded a new data pipeline, and how any developer can start building in this space.

Free Physical AI Courses: https://nvda.ws/4xqCfo9
Cosmos: https://nvda.ws/4zJ4iRj
GR00T Teleop + LeRobot: https://huggingface.co/blog/nvidia/nvidia-isaac-teleop-and-gr00t17-in-lerobot


Sponsorship inquiries:
[email protected]

The post Physical AI at NVIDIA appeared first on Software Engineering Daily.

More from Software Engineering Daily

All episodes