Pioneering physical AI with Archetype AI’s Ivan Poupyrev | E1951

0:00We also have conversations with people from printing press companies and they use really old printing presses. and they are touching sensors to all printing presses because they want to kind of like, you know, they're getting old and they need to, you know, adjust them for their availability. And the same problem. I have all the sensors around this printing press. Can you tell me when things are about to go wrong? Like predictive maintenance, recalibration? Because time is money for these guys. You know, every minute machine doesn't work. That's money. This Week in Startups is brought to you by Dot Tech Domains.

0:37Don't miss our Jam Session with JCal contest coming soon. To apply and get more details, go to jamwithjcal.tech. Brought to you by Dot Tech Domains. OpenPhone. Create business phone numbers for you and your team that work through an app on your smartphone or desktop. Twist listeners can get an extra 20 % off any plan for your first six months at openphone.com slash twist. And Vanta. Compliance and security shouldn't be a deal breaker for startups to win new business. Vanta makes it easy for companies to get a SOC to report fast. Twist listeners can get$1 ,000 off for a limited time at vanta.com slash twist.

1:23All right, everybody. Welcome back to This Week in Startups. Obviously, we're moving into an era of startups and employment and work and life that is going to be driven by absolutely mind-blowing experiences powered by artificial intelligence. Videos by Sora, MidJourney, we've seen all that. We're starting to see robots like Optimus and Figure, music generation, all of this stuff is incredible. And it generally uses text-based prompts. But what if AI could understand the real world in real time. Well, Archetype AI is here to bridge the physical world with AI. And they call it just that, physical AI.

2:04They've created Newton. It's a first-of-its-kind AI model that understands the physical world. Okay. The innovation allows integration of sensors with machine learning. So you can have sensors, pull this stuff in here. And today, we're lucky enough to have the CEO and founder Ivan Kuperev to explain what they're building and to show it to us. If you're not watching us you can go to youtube.com it's a new website that hosts videos uh you're gonna love it by the way there's a lot of videos up there like thousands of them and go to youtube.com and search for this week in startups you'll find the episode ivan welcome to this week in startups how are you doing good great to see great to be on your show yeah uh i i've seen some demos of what you're building right and it's really interesting so why don't we get started and we'll just show the audience what you've built how it works how you want to start do it like you want me to show the demo do we just talk a little bit more about the demo because i think it's like one of these things where once you see it you start to understand and you do such a good job of demoing it and explaining what's happening behind the scenes and the other demos i got to see online right okay all right so ivan show us how you take motion and and you find some meaning in here the idea of okay that is to build a foundation model which can understand physical world when you think about the physical world, you can think about sensing and sensor data.

3:26Because human naturally observe physical world throughout biological sensors. But when you go to machines and talk into the physical world environment and industries, they run on all kinds of sensor data, motion data, radars, spectrograms, and so forth. So let me show the video, which is very much inspired by our conversations with logistics companies, like one of our investors in Amazon and how they can track packages through the long logistics supply chain and know what's happening when the package moves. Because you kind of don't know. You send it somewhere, you have no idea what's happening with the package.

4:04So how can we get sensors to tell you what's happening to the package? So this is an example of the demo we built. So in this case, I'm going to pause here. In this case, you can see there is accelerometer in the box. and you can see all the sensor data coming up on the screen so you can see you have an accelerator box which looks like a you know playing cards a pack of playing cards and uh you have an accelerometer in your phone so you get the idea and then you see like essentially a wave signal of some type three waves a purple a green and a yellow here so you shake it it moves that's right so you can see the you know the box and she's shaking it moving and now she's putting the box inside of the package.

4:47What you do is, in our interface, you can ask Newton, pretty much tell me, tell me the transit status of your package, how it moves through this thing. She puts it inside of the package, and now the person does that transit status and turns it on. Now what's going to happen with Newton is that as she moves the package around, then Newton translates this complex sensor data into the very understandable message. the package is in motion. The package is still. So you don't need to go and understand what the sense of data means, but it's actually, you know, in simple language. And now you should change the prompt to reward package mishandling, so a dropping or shaking.

5:31And without changing the model, without reversing the model, without retraining the model, the model can understand that sense of data needs to be tracked for the package dropped. And you can see now it's analyzing and see if the package dropped. So this demo demonstrates that how you can, in real time, kind of steer the model to look after these particular events or what they call behaviors in the physical world, which demonstrate captured from sensor data, something you naturally cannot understand. Let's make sure you want more demo. And this is a difference between physical AI and classic LLMs because physical AI is not a chatbot.

6:11it's not something you are chatting it's something you're asking a prompt and then the model is looking for these behaviors, is trying to understand and report these behaviors to you based on what you ask it for and the output doesn't necessarily have to be textual because if you see, imagine a worker at a factory or a doctor in a hospital or anybody who is working in a physical environment they have to be focused on the physical world They have to be focused on the task at hand. So the textual representation is not the most natural for that kind of environment, right? So the model has to also produce outputs in other formats, in the visual.

6:55So let me show you. So this is a dash cam we're going to see now. So it's a dash cam recording what's happening in the world. It has some sensors. I don't know, LiDAR or just video. And then you're going to translate that into a language model, right? Right. But in this particular case, what you see here... Or a visual model, I should say. It's a unique model. That's not a language model. It's a visual model. It's the same model, Newton, which can translate either in a text representation, but the same model can render that in a very different representation. And later during the show, I can show you a diagram which shows how that's happening and why that sort of translation is possible.

7:35But in this example, on the left side of the screen you see the real video on the right side you can see the overlap, visual overlap that Newton creates in response to the prompt or response to the question you ask. Let's just show how it works. This particular case you ask a monitor for a car in front and you can see there's a car in front and the Newton highlights where is the car in front. Obviously when a car crosses the road you can see this case is stop highlighting this and when the car the car passes by it's going to continue highlighting the car in front and the interesting thing here is that you can change the focus you can say stop the car and look for pedestrians show me how this pedestrian and now you can see on the right side it's pedestrians who is being highlighted to give you know to direct your attention to them and for the same video so you're steering your model to do things which you need by text language.

8:33Here you're asking something to show me crowded areas, but the output right now is not the visual overlay, but on the right side you can see a heat map on the map based on GPS data which shows you crowded areas. So you can imagine a very simple use case where you have a fleet of vehicles, and that's actually a real use case we're discussing. When you have a fleet of vehicles, fleet of cars, which drives around the town to deliver goods or products, you would like to report all other cars, what's happening in the city so they don't get stuck somewhere because of the flooding or because of something, some other events.

9:10So dynamic update of the map based on the semantic understanding of the world delivered by Newton. And that's one of the many use cases when it comes from understanding the physical world. All right. You guys know I'm passionate about innovation and tech, and I love hearing from founders, I've got a crazy exciting opportunity for you to consider. I'm hosting something called Jam Sessions with Jcal. It's a contest. It's powered by my friends over at.techdomains. And over the summer, I'm going to have five founders get the chance to do a jam session with me right here on This Week in Startups. It's really simple.

9:44You tell me in this one-on-one session what you're struggling with as a founder. What are your challenges? What's your vision for your startup? Tell me about your product. Tell me about your customers. And we sit there and we jam out. I deeply listen to you. I ask you really deep, thoughtful questions. You give me deep, thoughtful answers, and we try to figure out how to grow your business. And then we publish it here on This Week in Startups so everybody gets to learn. It's really simple. There's only two rules here to get one of these five jam session slots. One, you got to have under$2 million in funding, so this is for new startups.

10:13And two, you just have to have a.tech domain name. So here's what you do. If you want to get more information, jamwithjcal.tech. Jamwithjcal.tech. That's that .tech domain name you keep hearing about. .tech domains and I are trying to find the most innovative founders. We have used the.tech domain for many things. And there's tons of people in the industry like rabbit.tech, you know, that really slick AI hardware device, aurora.tech, onex.tech. If you're using a.tech domain name for your startup's website, I want to hear about it. So apply for Jam Sessions with Jcal, jamwithjcal.tech. Don't wait.

10:45There's five slots. You've got a good chance of getting one if you apply now. Okay. So this model that you're building newton can take any uh sensor data it could be lidar it could be cameras it could be an accelerometer put it into the model and that let you ask questions to i don't know solve problems in the real world or understand the world better that's exactly correct yes okay so is it an open source model or you is it a closed model right now at this time is a closed source model we are not opening sourcing got it and so you're building this model and then you're hoping to get a bunch of training data and then solve problems for businesses and then allow them to the business model here is obviously to make this a hosted services like an amazon web services or something where people can give you sensor data and then query their sensor data and get some output so how are you training this you know because you show different sensors a camera sensor and then you showed an accelerometer i saw in another demo you gave you showed somebody touching a doorknob.

11:50Then I think you have other models. Tell me what sensors, what inputs you currently have coming into it. Right now, at this point, we're focusing on four kinds of sensor types. First of all, it's cameras, so there's people using cameras, obviously. Audio, time series data, and RF, which is pretty much radars. This is the kind of sensors we are focusing on right now. We're training our model for those sensors. The data is coming. The way the approach we're taking is that very early at the stage of the company, we did a pretty broad review of the market. We went out to literally hundreds of companies.

12:27We talked to them, asking what sensors they're using and what kind of things they want to do with sensors. That's how we selected this group of sensors initially. And then very early, we started engaging. We built design partner program. We started engaging with our company to build specific, understand how our model can be solving their specific use cases. So the training data comes from either from partners that are giving us data to be able to train our model for specific use cases. And with every use case, the model can learn more and more things. Or for some of our partners, we're collecting data ourselves in the physical world.

13:02Got it. For specific or less stuff. It would seem to me the number one use case here is self-driving, and it was in your demo there. This is, I think, what Elon's gotten to when he shifted hard coding to a language model. So is what you're doing essentially a broader version of that that's available to anybody who wants to use it? That's exactly correct. We're building architecture within the Newton's architecture and the way we're designing Newton. It designed a way so you can take any kind of sensor within those categories with a very small amount of modifications, sometimes out of the box, that can use those sensors to solve their problems.

13:39It's a very general purpose, universal model for everybody. Because when you go to the physical world and the physical world businesses, you can build bespoke solution for every single person or for every single business because it's so diverse and kind of messy, the physical world in general. Yeah. So it's universality of the model, which is extremely critical for, you know, for being successful in this field. Do you believe what Elon's done with FSD and making this model and what you're pursuing will solve self-driving? And if so, when do you think self-driving will get solved? Because in your model here, you're asking it, hey, tell me where it's congested.

14:19Tell me where there's a car. Tell me where there's people. Tell me where there's a cow and obstruction, et cetera. so you know one of the core questions is will we solve self-driving in all the edge cases by just watching humans drive and make mistakes and knowing it's a mistake or not so knowing what you know how close is tesla to having perfect driving yeah i or better than human because you must have used 12.4 12.3 and you're building something similar so just humanity in general yourselves tesla let's just broaden it out because you obviously don't work there. Yeah, well, I don't work for Tesla, and we don't really focus on self-driving.

14:59Self-driving is just one of the use cases. We're working with a few companies to help them with self-driving, but that's not one of our own. We're actually building a horizontal model across multiple modalities. We're working with a semiconductor company, working with automotive companies, obviously, but also consumer electronic companies and construction companies. We're trying to build a generic model. As always, it's very hard to predict with anything which happens in the future, like self-driving, when it's going to be solved. But I do believe that being able to understand contextual information beyond what's sensing from the direct sensing, but understand the context information, behavior of the complex system, behavior of the people around it, and using large language models, style reasoning about the world around you, would definitely bring full self-driving closer to solve all those complex use edge cases.

15:51So how much of what you're doing is predicated on having a large data set? There are some people who have cameras in entire cities. London, China are both known, or different cities in China are known for having massive surveillance systems for safety, et cetera. And so they have a massive amount of data. If you had access to that, man, you would understand a large portion of the world. Then you have satellite data, maps, GPS data. And then I guess people walking around with sensors on them or bicycles riding around with sensors on them. Obviously, a Tesla or the Waymo cars have massive sensor arrays.

16:26So what is your training data? You know, people look at all the language models using OpenCrawl or Reddit data or Twitter data or Quora data or Stack Overflow. There's all these pools of data and oil. What are the ones you're tapping into to understand the world? Right. So, as I mentioned, we're working with design partners. And depending on the use case we're trying to solve for them, we're tapping in their data that they provide to us. We're also using, obviously, a lot of open source data out there, all these data sets that are available. We're using them to kind of seed our model with the initial understanding of the world and kind of train the model on those.

17:05What we found out is that when you work with very specific customers, the specific customer has a very specific problem. You really have to work with those customers to get this data from them. And then you fine-tune the model on their specific use case. But we also were quite surprised that in many cases, those customers are quite open to let the model to train on this data. They can keep the data later, but once the model is trained, everybody benefits. So the approach is to do piece by piece. You're solving for one customer, and that empowers everybody else. Got it. And so factories are places where there's a lot at stake.

17:46There's a lot moving around. There are complex environments. There's robots. There's humans. So getting into factories and just understanding what's happening in a factory, is that one of the early use cases here? and then do you need to make more sensors for those factories? They already have the sensors in there, don't they? That's exactly correct. One of our investors is Hitachi. The Hitachi actually were interested in our company exactly for that reason because they're already storing a massive amount of data from the sensors. When we talk to them, they say, look, we have all this data, but understanding this data and figuring out how these multiple data streams can be analyzed to understand not what a particular sensor does, but how all together they can draw a holistic image of the factory, that's what we're looking for.

18:37And the use cases, there's just like endless there, completely infinite amount of the edge. Well, let's double click on that. You know, you have a factory building, I don't know, robotic arms, right? There are factories that have robotic arms building robotic arms, quite meta. Well, let's say you have a factory that builds robotics, and you get all the input for the last five years of everything that's occurred in that factory, then what would they ask and what would the benefit be once they have all that data in the language model? Because we showed very basic proof of concept demos here. But in the real world, what do you think they would then start asking it?

19:15What could they ask their factory that built, I don't know, cars or televisions? I can give you the real use case. Okay, please. example with the real customer we have. So we're having conversations right now with a very kind of large semiconductor company, right? So these machines, which making chips, you know, literally they're saying like, look, we have something like a plasma reactor for etching the silicon wafers, right? And some of these machines have up to 400 sensors inside of the machine. and 200 of them are critical which means if they're off the value which is supposed to be it cannot work and what they're saying is the problem there is that you take this machine and literally you move it or you shift it or change it by a meter and because the precisions and the tolerances are so high everything goes out of whack right away it means they all start getting false alarms the machine stops, yield drops, and then somebody has to come and reset the entire machine which can take days and a lot of money lost, right?

20:27Which is obviously being passed to end customer and eventually to us. So the question was can your model not just look for the threshold values of the data, but actually understand which data is correct and which data is not correct. So when you move it and it's slightly moved around, the model self-adjust to itself so this is one of the very specific use case where the factory is looking for solutions wow yeah that i mean that's incredible when you think about it these highly precise machines if monitored they have monitors and sensors already of course this is now the language models watching it it could tell you what to fix it could maybe even fix it in real time i don't know if this it can actually adjust those 400 you know nuanced or it can actually do the adjustments with the machine itself.

21:15I don't know how the machine is configured, but at least being able to monitor is going to save a lot of time and money. I was thinking of like a printing press to go old school, not that we print much anymore, but you watch those newspaper presses or magazine presses, it was a very similar situation if they were off just a little bit because you see how fast they move. Exactly. The whole thing is just, you're just throwing away a lot of off-printed newspapers. Yeah, exactly. It's a lot of loss, right? It's surprising, by the way, if you finally mentioned printing press, we also have conversations with people from printing press companies and they're using really old printing presses.

21:48And they're attaching sensors to old printing presses because they want to get old and they need to adjust for their availability. And the same problem. I have all the sensors around this printing press. Can you tell me when things are about to go wrong, like predictive maintenance, recalibration? Because time is money for these guys. You know, every minute machine doesn't work. That's money. Juggling multiple devices and apps to run your business is a mess. OpenPhone is here to make it simple by simplifying your business communications with one easy-to-use app. OpenPhone has rethought every detail of what a modern business phone should be.

22:30And here's the magic. It works through a beautiful, elegant app on your phone, or you can just use it on your desktop, making it super easy to get a business phone number for your entire team. And you know how brilliant OpenPhone is? My teams use it every single day. My sales team loves it. My ops team, they use it all day long. And here's the features that we love. You can create a shared phone number like customer support with multiple employees fielding all the calls and all the texts to that one number. At my investment firm launch, we pride ourselves on replying to every single call or email instantly.

23:03And Open Phone is the number one rated business phone on G2 for customer satisfaction. So here's your call to action. Super easy. Open Phone is already affordable. Starts at just 13 bucks a month. But Twist listeners get an extra 20 % off any plan for the first six months at openphone.com slash twist. And if you have existing numbers with other services, no problem. Open Phone is going to port them over easy peasy, lemon squeezy, no extra cost. Head over to openphone.com slash twist to start your free trial and get 20 % off. We talked about one type of vehicle, cars. I wonder battleships and airplanes, also pretty complex and with unlimited sensors.

23:40My God, the sensors in an airplane or a battleship. I mean, incredible. Has the military in space, you know, started to come out and say, hey, let's just take a look at all the sensors we have. I could imagine a SpaceX rocket or a giant airplane, a complex Boeing airplane, having all these sensors reporting in and then being able to ask questions. I wonder if you could avoid accidents or maybe come up with insights on how to make those products have less drag or, you know, be more efficient in some way. We haven't yet from the, anybody from aerospace, we have this conversation from aerospace industry or anybody, anybody else from that.

24:16But, you know, of course we open these conversations and love to talk to them. I mean, if you think about automotive industries as a kind of a proxy for this complex machinery, so there's a lot of interest from automotive industry. Because the car generates a gigantic amount of data. I've just read, maybe yesterday, an article that once AI cars bring to the cars, even not full self-driving, just like AI, incorporated cars, it's actually like 25 gigabyte data per hour going to be generated by the car. So how you process all that amount of data and how you can make sense of that, How humans can understand that data?

24:59So you need a sort of something in between which can help you to analyze this data. And that's what Newton is. Newton is looking at the physical world, all this data, and helps you to make sense of that world of physical data. That's kind of our… I was just wondering about environmental stuff. You've got the obvious factories that are packed with centers, but then we have the real world. And we're very concerned about the rainforest. We're concerned about oceans and temperatures and pressures and, you know, the amount of sunlight and et cetera, precipitation. And those sensors have also been deployed in many cases.

25:35And those systems, you know, are, I think, incredibly complex. Weather systems come to mind, you know, global warming and CO2 and all of those. have you have you started to think about how we might be able to use all the global sensors on the planet to maybe understand uh what's happening to to the ecology of the planet yeah i mean obviously like if you if you be yeah of course certainly certainly the um decarbonization and and um supporting kind of environmental you know environmental monitoring it's one of the most interesting directions we can take to. I'll give you another example which we discussed quite extensively with one of the partners in the process of conversation.

26:22You know, the gigantic windmills. These, those things, you know, that rotates offshore. So they have a very specific problem. The problem is that vibration of the gearbox is a prediction of failure. And you have basically this windmill farm of dozens of those windmills. And they all vibrate slightly differently. So trying to understand, is this normal vibration? Is it the ground vibration? Is it the wind vibration? What's making sense of this vibration would allow them to do either predictive maintenance or slightly adjust operation of those windmills to optimize their performance. So this is exactly, this is one of those problems which relates to what you're mentioning, how to control these gigantic infrastructures which we are building in the physical world and how to use the sensor data to actually predict the future of what's going to happen with those machines.

27:28And how do you think about the connection between robots and artificial intelligence? Obviously, we've got Figure and Optimus and a bunch of people are starting to look at this and there's lots of sensors in these robots Boston Dynamics uh obviously has been been doing this for a while so are those going to eventually be out there in the world mapping the entire planet earth to give us some more information than we currently have right because what could be unlocked if you had perfect insights into everything occurring in a city everything we have seen this with perfect mapping right with GPS has had a profound impact.

28:08We don't get lost as a species. Pretty hard to get lost these days. Pretty hard to be out of communication with satellites and, you know, SMS to mobile phones now, et cetera. So if we could, with these robots, you know, if there were a billion robots on the planet and you had all the sensor data, how does life change for humanity in your mind? And maybe you talk just generally about robotics and the impact here. Robotics is very interesting. Again, just like with the satellites and space companies, we haven't yet engaged with robotics companies. We're mostly focusing on our current focus is construction and automotive and factories and semiconductor factories, particularly all that stuff.