The Difference Between LLMs and Robotics Data | Lightwork

0:00Typically, that's why a lot of older generation of robotics companies have just struggled because you would have to make adjustments and tweaks for every new action, every new embodiment or tweak to the hardware. We've had a big summer of robotics breakthroughs and releases with robots that can now learn a task by watching instead of being programmed, robots that can drop into rooms they've never seen before, and a whole lot of funding going into physical AI. A lot in the past few years has moved towards this architecture of these visual language action models. So effectively there's these models that take in visual and also language to encode them into a model that then predicts the next action.

0:41It's hard to understand who is playing where and where the open space actually is. So today we're going to draw on one board across two axes and map who's building what. I think we're going to start to see a lot more of these companies, which is why I put this deploy codes in brackets. I'm here with Amber Yang, partner at Lightspeed, who spends her time across AI, the frontier of robotics, to break it all down. We've now seen an entire new market form around robotics data collection, akin to like a map plays across all the scale, but for robotics. Been a big summer for robotics. There's been a ton of big announcements.

1:20We've seen a lot of advancements in in-context learning at the model level, which we'll get into. And then when it comes to hardware, I feel like a lot of people are reconciling with the fact that we might have humanoids in our homes and our warehouses for the first time. But I'm so excited to have you on here to sort of help us make sense of this space. I think for a lot of people, when they think robotics, it's just a bunch of different companies. They're all, some of them are building bodies, some of them are building brains. And you helped us kind of lay out all of this. I think before we get into specific companies and all of these quadrants, it might make sense to just walk through or help me understand a little bit of what you're seeing at both the model and hardware layer.

2:02Maybe with the model layer first, what have been some of the big breakthroughs in the last couple of months? Yeah. I mean, what's been really interesting, just even in like level setting, the robotic space, a few years ago, right, even before what we currently call like the main architecture that a lot of these robotics foundation model companies use. It's an architecture called VLA, so these visual language action models. Most of the robotics field was actually based in imitation learning, which is a type of reinforcement learning. So reinforcement learning or RL that we call in software world now that's used to train a lot of the large language models was actually grounded in robotics.

2:40And the idea was like, can you basically train a robot with a reward model that gets rewarded when it takes the right action? And can robots sort of learn from human behavior and imitate that in like an action by action sort of motion, which is not very generalizable. And typically that's why a lot of our older generation of robotics companies have just struggled because you would have to make adjustments and tweaks for every new action or every new embodiment or tweak to the hardware. A lot in the past few years has moved towards this architecture of these visual language action models. So effectively, there's these models that take in visual and also language to encode them into a model that then predicts the next action.

3:27And we've seen more generalization across a number of companies like skilled, generalist, physical intelligence, etc. I think what is now becoming like an interesting conversation is the advent of world action models. So it's this idea that you can take sort of an action in one current state of the world, and then the model tries to predict the environment of the world at a future reference point, and then predicts all the actions that the robot should take in the current state to reach that next state in the world. And that is predicted to have more generalizability. But we're starting to see this idea of going from very specific action sequences for a robot to can a robot sort of just take in a context of its environment and sort of one shot predict what is the best action in any given moment.

4:20The best way to think about this is we have one brain that can fit into many bodies, right? Without me saying you need to learn this specific pancake flipping action or you need to learn the specific unloading, loading motion. Is there still value in these more specific models? It's a great question. So if you were to talk about the most purest of purest companies in that first regard, it would be physical intelligence. Their idea is we are researchers. We're going to train the most generalizable model that will generalize to any embodiment of hardware. I think where we're still at today is it is still very, very hard to deploy in the real world.

4:58Actually, one of Lightspeed's company is Dexterity from sort of a previous generation of robotics. They have been very, very focused from day one in deploying robots across manufacturing warehouses, et cetera. And I think there's still a lot of value with that because the real world is messy. Building customer relationships and deploying in those environments is very hard. So today there is still a lot of value in owning that customer relationship and being specific, but we'll see how the ecosystem plays out. And it sounds like in this vision towards one brain, there's been a lot of emphasis or breakthroughs in specifically something called in-context learning.

5:35What are you observing around that? Why is that such a breakthrough moment? So in-context learning is fascinating because maybe to ground a lot of this, what makes robotics and robotics foundation models very different from large language model world is that there isn't a corpus of internet data that can just be thrown at a model to be trained on. So with robotics companies, they've had to rely on a number of data collection techniques to really scale up their training. That's primarily taking the form of tele-operated data, which is basically humans that try to control a robot. And then that data is then used to sort of like train the model And then egocentric model, which is the human performing tasks.

6:21And we've seen a number of these companies now that basically give people cameras that they can stick on their head and they record themselves doing tasks. And then that can be used for training robots. What has actually been more popular recently that Skilled has actually used a lot of is simulations. So trying to use a lot of NVIDIA's ecosystem, actually, to take in not necessarily like real world data, but stuff that is like, you know, virtual data. And then to train those models and upscale. In-context learning is actually fascinating because in the real time of the robot interacting with the world, you're sort of observing when the robot does well versus badly, and you're updating your model in real time.

7:06And that's actually potentially the highest quality type of data because it's the robot specific embodiment, right? It's like the robot specifically taking in actions and then you get all of the sensor recordings around it that can be used to better improve the model instead of trying to take in some human type of data and map that into a robot, for example. That's super helpful. And I feel like we'll definitely get a little bit into the weeds with all this and how they all differentiate. You know, you mentioned synthetic data versus others. So that's like the model layer. Now, maybe moving to vertical with hardware.

7:41I think for the longest time, a lot of people have thought of hardware as this like highly commoditized thing. Help me understand any sort of developments that are happening on the hardware side. Yeah, I mean, hardware is actually a very interesting space right now because a lot of it actually feeds into regulation and also supply chain. There are a lot of robotics companies that are not as focused on the humanoid form of deployment, but maybe just one arm or like a picker and placer. So there's this company called Ultra that is working on, you know, robots in warehouses and in factories. That is not the humanoid embodiment.

8:18You know, the cost of hardware is rapidly going down in those sectors. And so actually deploying with customers becomes a lot more interesting as a business model. The interesting thing with humanoid robotics is that 40 to 60 percent of a humanoid's build is actually spent on actuators. Actuators today, we are almost basically completely relying on China for actuator supply chain. And there have recently been restrictions passed around using actuators from China. That might be turned around in the next administration, but because of, you know, increased tariffs on getting actuators and also all these actuators are built with rare earth magnets that we're also completely dependent on China's supply chain for.

9:01So it is hard to imagine today that the cost of building humanoids will come down rapidly. Maybe in the long tail, we can make that bet. But because it is closely tied to a lot of this supply chain of dependence on China, it is not necessarily a fair claim to say that humanoids as a hardware embodiment will become commoditized. Maybe moving to a specific quadrant, maybe I'll start with actually the most specific model, a specific embodiment because it feels like that's how we imagine a lot of robotics businesses today is that they were building their own models, systemic embodiment. Tell me a little bit more about the different companies playing in the space and why a company might specifically still focus on building their own models and their own hardware.

9:47Yeah. Yeah. So in this quadrant, I've sort of picked out Dexterity, TACDA, Reflex Robotics. For Dexterity, they're very focused on the warehouse and the manufacturing play. And that makes sense. They were started before skilled physical intelligence, generalists, et cetera. So they had to be very focused on specific customer deployments and worked really hard to build those relationships and collect very specific data. TACTA is more focused on specific dexterous manipulation type tasks in the warehouse and also factory environment. Reflex does sort of a similar thing. And the bet today is that that works in being very deployment focused and training on a specific embodiment.

10:30I think the question over time becomes like what happens to these companies as like if physical intelligence does build a generalized model where it doesn't matter if you have a specific model. And I think where that comes down to is you could see a lot of these companies ultimately they have the customer relationships. They've deployed over many years. They can swap out a model with a more generalized model and they don't have to have any allegiance or loyalty to their model, PI's model, generalist model, et cetera. And they can just be very focused on sort of the hardware deployments angle and working with the customer.

11:06That's where this sort of quadrant is today. So this is a company called Ultra, which is fascinating because their models are completely trained. They're basically fine-tuned off of physical intelligence's latest model. So they have not spent any of their own time training their own model. And they say that with like 20 to 30 hours of data that they can collect in a factory, they can get something that works pretty well. So give me an example of why I would go with Ultra versus, you know, doing the work of fine-tuning visible intelligence models. Yeah. So I think it is, today, it is still quite gnarly to fine-tune PIE's models.

11:45It is way better than it was a year ago, which is why Ultra exists. But I think today, a factory is not necessarily going to be able to take in their own types of recorded data, understand the quality of data that is needed to fine-tune on top of PIE's models. The other great thing with Ultra is that the robot is actually stationary, so it doesn't move. And that also constrains sort of the environment in which the robot needs to complete a task. And I think like with Pi's models today, if you want to imagine a robot that's also mobile and is doing all of these tasks, like any specific change can be like, it's not straightforward to fine tune.

12:30So who do you see as the customer here versus here? Yeah, it's a good question. So I think with Ultra, there's targeting a very different type of task than these companies today. They're mostly focused on picking and placing within warehouses. I think we're going to start to see a lot more of these companies, which is why I put this deploy codes in brackets. I think this is going to be something we see a lot more in the next year where there's these companies that are going to take Pi's model or they're going to take, you know, generalist model and they're not going to have like allegiance to them.

13:07And they'll sort of become like an open router for robotics. And they can, you know, they might have three or four different types of robot embodiments that may be a humanoid, maybe something on wheels, et cetera. And they'll specifically go to like a restaurant or a laundromat or somewhere else like a nursing home and say, hey, we're going to deploy specifically for your environment. But you won't, as the end customer, know what's under the hood. You might not even own the robot hardware specifically. I definitely want to get more into what developments you think you'll see in the next year or so.

13:47But maybe moving on to the flip side where it's like ultra-specific model, ultra-specific embodiment. We covered a little bit when we talked a lot about what was happening on the model layer and in context learning. But there's a lot of companies that are playing this space, this idea that you can have one brain that plugs into multiple bodies. How are you thinking about this ecosystem of players? Yeah. I mean, I think what's been really promising in the past few months is actually seeing how Skilt has actually been really focused on deployments since day one. So I think what makes this team really special is that not only do they have great researchers who are focused on building a general model, but they also really care about the deployments.

14:27And so they've signed large contracts now across manufacturing, et cetera, in many different use cases and have actually rolled out to the real world. And I think they just announced that they crossed$100 million in revenue like a few weeks ago, which is amazing. You see companies like Physical Intelligence that are super pure on like the research side of things. They're starting to get more into deployments and have released some demos recently, like live demos, not just demo vaporware, of their robots going into new Airbnbs and being able to generalize immediately, which is really promising.

15:05And then Reward AI, which just announced their launch, I think, last week at this point, they have the claim that they can one-shot. So they can take in sort of a video feed of a human doing an action and then immediately generalize that without any type of fine-tuning additional work with their model. So we're starting to see signs of general embodiment and general model. And it's great to start to see that a lot of these companies are actually starting to deploy too. And it's also a great sort of sign that companies like Ultra can exist because of Pi's models, for example. Yeah, it's really interesting seeing all these different layers actually like start to converge and overlap in certain ways.

15:50Yeah. Maybe the last quadrant, this is taking that, you know, this is taking a more generalist approach to the brain, but they have their own embodiment. And these are sort of like the humanoids or robots that we've seen doing those laundry demos. Tell us more about this approach. Yeah, I mean, I think Sunday Robotics, for example, their claim is they've developed this amazing sort of hand that is like filled with a bunch of sensors that is taking in live data of the robot performing actions. And that's actually very high quality data, right? Because you don't have to map it from, for example, like an ecocentric data collection where you have to map a human hand to a robot hand, for example.

16:32And so I think this claim over time can play out and work. All of these companies today are more focused on the humanoid consumer use case, which I think is going to take a lot longer to play out in the long tail. There's just a lot more noise in those environments and you also have to be way more concerned about safety. But I think those sort of deployments all make sense. I think 1x and Figure don't necessarily have the data mechanism that Sunday has that made Sunday a very compelling sort of pitch initially with the robot hand. But we're also starting to see, like, for example, Figure has customers and then I think OneX is trying to make strides in that area.

17:21So it's great to start to see all of them start to deploy as well. Yeah, this has been so helpful. I think maybe just zooming out now that we have a little bit context on where all these folks are playing. What is your sense of where, you know, things are a little bit commoditized, where things are a little bit overcapitalized, and where do you think there are opportunities? Yeah, it's a great question. So this quadrant is definitely the most capitalized for good reason. I mean, it takes a lot of capital to train a model from the ground up that is generalizable. I think what are areas that I would like to see more companies in?

18:01I actually think there's a lot of room for this deploy code angle that I was mentioning. I think specifically people who have worked at Skilled or Dexterity, for example, or another company that I'm a really big fan of called Dyna that is focused on deploying, you know, robots across a number of different environments and use cases can start these types of companies. I say that because deploying robots is a very non-trivial thing. So more of what? It's very, very difficult, noisy environments, building trust with customers. It is not as trivial as just like giving a customer a piece of hardware and saying like, go after it.

18:43It's a lot more operationally intensive. And it's not necessarily someone who comes from a researcher background who will be good at this. So I think that's a lot of room to play in. I honestly think a lot of the infrastructure and sort of tooling that we've seen in the LLM world will start to become interesting in robotics world. So we've now seen an entire new market form around robotics data collection. So akin to like a map plays across all the scale, but for robotics. And so they're like collecting either teleop or egocentric data and selling to all of these companies that are training their own models.

19:24I've now started to see like companies that are designing specific inference chips for robots. And the idea is, you know, for a robot, when you're trying to do inference in the real world, you need it to be low latency and you need it to be extremely computationally efficient in a way that LLMs didn't necessarily have. And so I've started to see both inference layer and also physical world chips being built out for robotics use cases specifically. I've now started to see conversations around how do you ensure like data quality and like a data pipeline for the RFM companies and also the robotics data collection companies?

20:05Like, can you sort of build like models that annotate data better? So I think, yeah, the tooling and infrastructure around RFMs will play out more and begin to serve all of these companies as the market becomes more mature. Yeah, that's super fascinating because we've talked about all these companies, but underneath that, there's just like this entire infra stack, whether it's like model routing or data labeling or, you know, data itself. All of that needs to be powered by this kind of core stack. And maybe we'll do a follow up whiteboard on how you're seeing all that play out. Yeah, I guess you mentioned, you know, deploy code vision.

20:43You mentioned that stack. Is there any space that you're particularly excited about or would love to meet founders building in? Yeah, it's a good question. I mean, another sort of category of things that I think will start to develop but haven't yet is stuff in the robotics observability space and then also in evals for robotics. There is no specific benchmark or eval set that companies point to and say, hey, like we have benchmarked our model against all of these these criteria. And because of that, we deem this safe or we deem this high quality or better than skilled model or pies model or whatever.

21:24How is it functioning today? What happens today is like all of these companies will will sort of release their demo. they'll release some sort of white paper on their model's architecture and they'll talk about, okay, for all the tasks that we tested on, like this is the percentage that, you know, the model and the robot were able to successfully achieve. It's hard to compare all the companies together because they're all working on their very specific data set and their tests. And so I think what we see in LLM world is now there's like a very established world of evals and benchmarking and all of the model companies are at a race to say, hey, we've outperformed OpenAI's last model on these benchmarks.

22:08Therefore, we are the dominant player. I think that will start to develop more in robotics world. And then that will also be very tightly coupled with deploying robots in the real world, because how do you as a customer trust or choose between all these models? And I think it might actually be even more important in robotics world because it's so closely tied to safety. And so I think there's a lot of parallels that'll start to happen here, even to like, we can learn a lot from like autonomous vehicle world, right? Like all of the sort of safety constraints and testing and observability and infrastructure that Waymo and Cruise had to build out in the early days, I think will start to build out in this ecosystem too.

22:51And then just as parting words, you meet with a lot of founders building across the stack. And And obviously, these are very different products, even within these quadrants, but even within the stack that we laid out. Any advice or common hurdles or road bumps that you've seen these founders interact with or advice for people who want to build in the space? Ooh, that's a great question. I think nowadays I see a lot of companies that try to pitch this quadrant, general embodiment, general model. typically the round is very frothy. I think the missing part or the thing that gets people caught up is like, how do you differentiate and compete against these established players?

23:36And it's hard because these established players haven't fully sort of defined what their niche is. And, you know, there's a lot of open questions about how these players will play out. And so I think, you know, being very specific about either an architectural improvement that you have or, Or, you know, maybe you have some very specific like in-house data collection mechanism and that makes your data pipeline way higher quality and you can speed run all of these companies that need to rely on these data vendors. Like there has to be something very differentiated in how you can sort of speed up and be more computationally efficient than a lot of these companies.

24:14So that would probably be my biggest. Super helpful. Thank you so much for taking time. This was so helpful, especially given all the robotics headlines of the summer. And I'm sure in a couple of weeks, we'll have you on again to explain even more for us. Yeah, this was a lot of fun. Thanks so much.