Large Action Models (LAMs) & Rabbits 🐇

0:05Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at Fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents. So you can launch your app near your users. Learn more at Fly.io.

0:42Welcome to another fully connected episode of the Practical AI podcast. In these fully connected episodes, we try to keep you up to date with everything that's happening in the AI and machine learning world and try to give you a few learning resources to level up your AI game. This is Daniel Wynack. I'm the founder and CEO of Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who's a tech strategist at Lockheed Martin. How are you doing, Chris? Doing very well, Daniel. It's enjoying the day. And by the way, since you've traveled to the Atlanta area tonight, we haven't gotten together, but you're just a few minutes away, actually.

1:24So welcome to Atlanta. Just got in. Yeah, we're within not maybe a short drive, depending on your view of what a short drive is. Anything under three hours is short in Atlanta. And I think you're like 45 minutes away from me right now. Yeah, so hopefully we'll get a chance to catch up tomorrow, which will be awesome because we rarely get to see each other in person. It's been an interesting couple of weeks for me. So for those that are listening from abroad, maybe, we had some major ice and snow type storms recently. And my great and embarrassing moment was I was walking back from the office and the freezing rain and I slipped and fell.

2:12and my laptop bag with laptop in it broke my fall, which is maybe good, but that also broke the laptop. So actually the laptop works, it's just the screen doesn't work. So maybe I'll be able to resolve that. But it's like a mini portable server there, isn't it? Yeah, exactly. You have enough monitors around, it's not that much of an issue. But yeah, I had to put Ubuntu on a burner laptop for the trip. So yeah, it's always a fun time. Speaking of personal devices, there's been a lot of interesting news and releases not of, well, I guess of models, but also of interesting actual hardware devices related to AI.

3:03recently. One of those is the Rabbit R1, which was announced and sort of launched to pre-orders with a lot of acclaim. Another one that I saw was the AI Pen, P-I-N, which is like a little, I don't know, my grandma would call it a brooch, maybe. Like a large pen you put on your jacket or something like that. I am wondering, Chris, as you see these devices, and I want to dig a lot more into some of the interesting research and models and data behind some of these things like rabbit. But just generally, what are your thoughts on this sort of trend of AI driven personal devices to help you with all of your personal things and plugged into all of your personal data and sort of AI attached to everything in your life?

3:57Well, I think it's coming. Maybe it's here, but I know that I am definitely torn. I mean, I love the idea of all this help along the way. There's so many, like I forget everything. I'm terrible if I don't write something down and then follow up on the list. I am not a naturally organized person. So my wife is, and my wife is always reminding me that I really struggle in this area. And usually she's not being very nice in the way that she says it. So it's all love, I'm sure. But yes. So part of me is like, wow, this is the way I could actually be all there, get all the things done. But the idea of just giving up all my data and just being, it's like so many others.

4:41It's not, that aspect is not appealing. So I guess I'm not leaping. How much different do you think this sort of thing is than everything we already give over with our smartphones? It's a good point you're making. I mean, we've had computing devices with us in our pocket or on our person 24-7 for what, at least the past 10 years, you know, for at least for those that adopted the iPhone or whatever when it came out. But yeah, so in terms of location, certainly account access and certain automations, what do you think makes it because obviously this is something on the mind of the makers of these devices, because I think both the AI Pen and the Rabbit R1 make some sort of explicit statements in their launch and in their website about privacy is really important to us.

5:39This is how we're doing things because we really care about this. So obviously, they anticipated some kind of additional reaction. But we all already have smartphones. I think most of us, if we are willing to admit it, we know that we're being tracked everywhere and all of our data goes everywhere. So I don't know. What is it about this AI element that you think either makes an actual difference in terms of the substance of what's happening with the data? or is it just a perception thing? It's probably a perception thing with me. I mean, because everything that you said, I agree with you're dead on.

6:16And we've been giving this data for years and we've gotten comfortable with it. And that's just something that we all kind of don't like about it, but we've been accepting it for years. And I guess it's the expectation that with these AI assistants that we've been hearing about for so long coming and we're starting to see things like the rabbit come into market and such, that there's probably a whole new level of kind of analysis of us and all the things and in a sense knowing you better than you do that is uncomfortable and probably will not be as uncomfortable in the years to come because we'll grow used to that as well but I have to admit right now it it's an emotional reaction it makes me a little bit leery yeah maybe it's prior to these sorts of devices, there was sort of the perception, at least, that yes, my data is going somewhere.

7:09Maybe there's a nefarious person behind this, but there's sort of a person behind this, like the data is going all to Facebook or Meta. And they're like, maybe they're even listening in on me and putting ads for mattresses in my feed or whatever the thing is, right? So that perception has been around for quite some time, regardless of whether Facebook is actually listening in or whatever, or it's another party like, you know, the NSA and the government's listening in. But I think all of those perceptions really relied on this idea that even if there's something bad happening that I don't want happening with my data.

7:50There's sort of a group of people back there doing something with it. And now there's this sort of idea of this agentic entity behind the scenes that's doing something with my data without human oversight. I think maybe that's, if there's anything sort of fundamentally different here, I think it's the level of automation and the sort of agentic nature of this, which does provide some sort of difference. Although there's always like, you know, if you're processing voice or something, there's voice analytics and you can put that to text. And then there were always NLP models in the background doing various things or whatever.

8:31So there's some level of automation that's already been there. But I agree. And I think, but you mentioned perception up front and I think that makes a big difference. I guess with, like you mentioned NSA, intelligence agencies, I think we all just assume that they're all listening to all the things all the time now. And that's one of those things that's completely beyond your control. And so there's almost no reason to worry about it, I suppose, unless you happen to be one of the people that an intelligence agency would care about, which I don't particularly think I am. So it just goes someplace and you just kind of shrug it off.

9:06There's a certain amount of what we've done these years with mobile where it's, uh, you're opting in. I think it's leveling up to, we're saying with some of these AI agents coming out, we know how much data about ourselves is going to be there. And so it's just escalating the opt-in up to a whole new level. So, um, hopefully we'll see what happens. I'm yeah, hope it works out well. We haven't really, uh, for the listeners maybe that are just listening to this and haven't actually maybe you're in parallel doing the search and looking at these devices but in case you're on your run or in your car we can describe a little bit so the i i described the ai pen thing a little bit the rabbit i thought was really really cool design it i don't know if there's any nerds out there that love this sort of uh synthesizer analog sequencer teenage engineering stuff that's out there but actually the sort of hardware design teenage engineering was involved in that in some way so it's like a little square thing the rabbit r1 it's got like one button you can push and speak a command it's got a little hard actual hardware wheel that you can spend to scroll and the screen is kind of just they show it as black most of the time but it pops up with the song you're playing on Spotify or some of the things you would expect to be happening on a touchscreen or that sort of thing.

10:31But the primary interface is thought to be, in my understanding, speech, not that you would be pulling up a keyboard on the thing and typing in a lot. That's kind of not the point. The point would be this sort of speech-driven, conversational and even call it an operating system conversational operating system to do certain actions or tasks which we'll talk a lot more about the kind of research behind that but that's kind of what the device is and and looks like it's interesting that you know going with the device route you know and the fact that they're selling the actual unit itself and you Over the years, we started on our computer.

11:14Well, we started on desktops and then went to laptops and then went to our phones. And the phones have evolved over time. And we've been talking about wearables and things like that over the years as they've evolved. But I think there's a little bit of a gamble in actually having it as a physical device because that's something else that they're presuming you're going to put at the center of your life. That versus being kind of the traditional phone app approach where you're using the thing that your customer already has in their hands. What are your thoughts about the physicalness of this offering?

11:46I think it's interesting. One of the points, if you watch the release or launch or promotion video for the Rabbit R1, he talks about sort of the app-driven nature of smartphones. And there's an app for everything. and there's so many apps now that navigating apps is kind of a task in and of itself and the Silicon Valley meme no one ever deletes an app right so you just accumulate more and more and more apps and they kind of build up on your phone and now you have to organize them into little groupings or whatever so I think the point being that it's nice that there's an app for everything, but the navigation and orchestration of those various apps is sometimes not seamless and burdensome.

12:36I'm even thinking about myself and kind of checking over here, you know, I got in the Uber. Oh, I forgot to switch over my payment on my Uber app. So now I've got to open my bank app, right? And then grab my virtual card number and copy that over. But then I've got to I go to my password management app to copy my password. There's all these sorts of interactions between various things that aren't seamless as you might think they would be. But it's easy for me to say in words conversationally, hey, I want to update the payment on my current Uber ride or whatever. Right. So the thought that that would be an easy thing to express conversationally is interesting.

13:22And then have that be accomplished in the background, if it actually works, is also quite interesting. I agree with that. And I can't help but wonder if you look back at the advent of the phone and the smartphone and, you know, the iPhone comes out and it really isn't really so much a phone anymore, but a little computer. And so we kind of the idea of the phone being the base device in your life has been something that's been with us now for over 15 years. And so one of the things I wonder is, could there be a trend where maybe the phone doesn't become if you think about it, you're texting, but a lot of your texting isn't really texting.

13:59It's messaging and apps. Maybe the phone is no longer the central device in your life going forward. And maybe you're actually having your primary thing. And so that would obviously play into Rabbit's approach where they're giving you another device. It packages everything together in that AI OS that they're talking about where conversationally it runs your life if you expose your life to it the way you are across many apps on the phone. But it's an opportunity potentially to take a left turn with the way we think about devices. And maybe the phone is no longer in the future, in the not so distant future, maybe the phone is no longer the centerpiece.

14:48If you're listening, you know that artificial intelligence is revolutionizing the way we produce information, changing society, culture, politics, the economy. But it's also created a world of AI-generated content, including deepfakes. So how can we tell what's real online? Read, Write, Own, Building the Next Era of the Internet, a new book from entrepreneur and investor Chris Dixon explores one possible solution to the Internet's authenticity problem, blockchains. From AI that tracks its source material to generative programs that compensate rather than cannibalize creators. ReadWriteOwn is a call to action for a more open, transparent, and democratic internet.

15:32One that opens the black box of AI, tracks the origins we see online, and much more. This is our chance to reimagine world-changing technologies to build the internet we want, not the one we inherited. Order your copy of ReadWriteOwned today or go to ReadWriteOwned.com to learn more.

16:11all right chris well there's a few things interacting in the background here in terms of the technology behind the rabbit device and I'm sure other similar types of devices that have come out. Actually, there's some of this sort of technology that we've talked a little bit about on the podcast before. I don't know if you remember, we had the episode with Ask UI, which they had this sort of multimodal model that I think a lot of their focus over time was on testing. A lot of people might test web applications or websites using something like Selenium or something like that that automates desktop activity or interactions with web applications and actually automates that for testing purposes or other purposes.

17:02And Ask UI had some of this technology a while back to kind of perform certain actions using AI on a user interface without sort of hard coding, like click on 100 pixels this way and 20 pixels down this way, right? So that I think has been going on for some time. This adds a sort of different element to it in that there's the voice interaction, but then they're really emphasizing the flexibility of this and the updating of it. So actually they emphasize like, I think some of the examples they gave is, you know, I have a certain configuration on my laptop or on my screen that I'm using with a browser with certain plugins that make it look a certain way.

17:50And everything sort of looks different for everybody. And it's all configured in their own sort of way. Even app wise, apps kind of are very personalized now. Right. Which makes it a challenge to say, click on this button at this place. It might not be at the same place for everybody. all the time. And of course, apps update and that sort of thing. So the solution that Rabbit has come out with to deal with this is what they're calling a large action model. And specifically, they're talking about this large action model being a neurosymbolic model. And I want to talk through a little bit of that.

18:30But before I do, I think we sort of have to back up and talk a little bit about AI models, large language models, chat GPT has been interacting with external things for some time now. I think there's confusion at least about how that happens and what the model is doing. So it might be good just to kind of set the stage for this in terms of how these models are interacting with external things. So the way that this looks, at least in the rabbit case, is you click the button and you say, oh, I want to change the payment card on my Uber, unclick, and stuff happens in the background and somehow the large action model interacts with Uber and maybe my bank app or whatever and actually makes the update, right?

19:20So the question is how this happens. Have you used any of the plugins or anything in chat GPT or the kind of search the web type of plugin to a chat interface or anything like that? Absolutely. I mean, that's what makes the, I mean, I think people tend to focus on the model itself, you know, I mean, that's where all the glory is and people say, ah, this model versus that. But so much of the power comes in the plugins themselves and or other ways in which they interact with the world. And so as we're trying to kind of pave our way into the future and figure out how we're going to use these and how they're going to impact our lives, whether it be the rabbit way or whether you're talking chat GPT with its plugins.

20:03That's the key. It's all those interactions. It's the touch points with the different things that you care about, which makes it worthwhile. So yes, absolutely. And I'm looking forward to doing it some more. Yeah. So there's a couple of things maybe that we can talk about. And actually some of them are even highlighted in recent things that happen that we may want to highlight also. One of those is if you think about a large language model like that used in chat GPT or, you know, neural chat, llama to whatever it is, you put text in and you get text out. We've talked about that a lot on the show.

20:40And so you put your prompt in and you get a completion. It's like fancy autocomplete and you get this completion out, right? Not that interesting. We've talked a little bit about rag on the show, which means I am programming some logic around my prompt such that when I get my user input, I'm searching some of my own data or some external data that I've stored in a vector database or in a set of embeddings to retrieve text that's semantically similar to my query and just pushing that into the prompt as a sort of grounding mechanism to sort of ground the answer in that external data. So you've got sort of basic autocomplete.

21:25You've got retrieval to insert external data via a vector database. You've got some multimodal inputs. And by multimodal models, I'm meaning things like lava. And actually this week, there was a great published on January 24. I saw it in the daily papers on hugging face. MMLLM's recent advances in multimodal large language models. So if you're wanting to know sort of the state of the art and what's going on in multimodal large language models, like I just mentioned, that's probably a much deeper dive that you can go into. So check out that and we'll link it in our show notes but these are models that would not only take a text prompt but might take a text prompt paired with an image right so you could put an image in and you say also have a text prompt that says is there a raccoon in this image right and then you know hopefully the reasoning happens and it says yes or no if there's isn't there always a raccoon in the image there's always a raccoon everywhere that's one element of this as a that would be a specialized model that allows you to integrate multiple modes of data.

22:42And there's similar ones out there for audio and text and other things. So again, summary, you've got text to text, autocomplete. You've got this retrieval mechanism to pull in some external text data into your text prompt. You've got specialized models that allow you to bring in an image and text. All of that's super interesting and I think it's connected to what Rabbit is doing. But there's actually more to what's going on with, let's say, when people perform actions on external systems or integrate external systems with these sorts of AI models. And this is what in the sort of Langchain world, if you've interacted with Langchain at all, they would call this maybe tools.

23:32And you even saw things in the past, like tool former and other models where the idea was, well, okay, I have, maybe it's the Google search API, right? Or SERP API or one of these search APIs, right? I know that I can take a JSON object, send it off to that API and get a search result, right? Okay. So now if I want to call that search API with an AI model, what I need to do is get the AI model to generate the right JSON structured output that I can then just programmatically, not with any sort of fancy AI logic, but programmatically take that JSON object and send it off to the API, get the response, and either plug that in in the sort of retrieval way that we talked about before, or just give it back to the user as the response that they wanted, Right.

24:28So this has been happening for quite a while. This is kind of like we saw one of these cool AI demos every week. Right. Where, oh, the AI is integrated with Kayak now to get me a rental car and the AI is integrated with, you know, this external system and all really cool. But at the heart of that was the idea that I would generate structured output that I could use in a regular computer programming way to call an API and then get a result back, which I would then use in my system. So that's kind of this tool idea, which is still not quite what Rabbit is doing. But I think that's something that people don't realize is happening behind the scenes in these tools.

25:14I think that's really popular in the enterprise, you know, and I'm, you know, in the enterprise with air quotes there, because that approach is, you know, in large organizations, they're going to other, you know, the cloud providers with their APIs, you know, Microsoft is, you know, has the relationship with open AI and they're wrapping that, you know, Google has their APIs and they're using RAG, you know, in that same way to try to integrate with systems instead of actually creating the models on their own. I would say that's a very, very popular approach right now in enterprise environments that are still more software driven and still trying to figure out how to use APIs for AI models.

25:55Yeah, and I can give you a concrete example of something we did with a customer at Prediction Guard, which is the Shopify API, right? So e-commerce customer, the Shopify API has this sort of Shopify, I think it's called Shopify QL query language. It's structured, right? And you can call the regular API via GraphQL, right? And so it's a very structured sort of way you can call this API to get sales information or order information or do certain tasks, right? And so you can create a natural language query and say, okay, well, don't try to give me natural language out, but give me ShopifyQL or give me something that I can plug into a GraphQL query, and then I'm going to go off and query the Shopify API and either perform some interaction or get some data, right?

26:49So this is very popular. This is how you sort of get AI on top of tools. What's interesting, I think, that Rabbit observes in what they're saying, and others have observed as well. I think, you know, you take the case like Ask UI, like we talked about before, and the observation is that not everything has this sort of nice structured way you can interact with it with an API. So think about pull out your phone, you've got all of these apps on your phone, some of them will have a nice API that's well defined. Some of them will have an API that me as a user, I know nothing about, right? There's maybe an API that exists there, but it's hard to use or not that well documented or maybe I don't have the right account to use it or something.

27:42There's all these interactions that I want to do on my accounts with my web apps, with my apps that have no defined structured API to execute all of those things. So then the question comes, and that's why I wanted to lead up to this, is because even if you can retrieve data to get grounded answers, even if you can integrate images, even if you can interact with APIs, all of that gets you pretty far, as we've seen. But ultimately, not everything is going to have a nice structured API, or it's not going to have an API that's updated, or has all the features that you want, or does all the things you want, right?

28:25So the question, I think the fundamental question that the Rabbit research team is thinking about is how do we then reformulate the problem in a flexible way to allow a user to trigger an AI system to perform arbitrary actions across an arbitrary number of applications or an application without knowing beforehand the structure of that application or its API. So I think that's the really interesting question. I agree with you completely. And there's so much complexity. They refer to it as human intentions expressed through actions on a computer. And that sounds really, really simple when you say it like that, but that's quite a challenge to make that work in an unstructured world.

29:16So I'm really curious. They have their research page, but I don't guess they've put out any papers that describe some of the research they've done yet, have they? Just in general terms, and that's where we get to the exciting world of large action models. Somehow that makes me think of like Arnold Schwarzenegger. Large action heroes. There you go. Exactly.

30:06you know when we started podcasting back in 2009 and online stores just the furthest thing from our minds now we have merch.changelog.com and you can go there right now and order some t-shirts and that's all powered by Shopify it's so easy all because Shopify is amazing Shopify is the global commerce platform that helps you sell at every stage of your business. From the launch your online shop stage to the first real life store stage, all the way to the did we just hit a million dollar stage? Shopify is there to help you grow. Whether you're selling security systems or marketing memory modules, Shopify helps you sell everywhere from their all-in-one e-commerce platform to their in-person POS system, wherever and whatever you're selling, Shopify has got you covered.

30:53Shopify helps you turn browsers into buyers with the internet's best converting checkout up to 36 % better compared to other leading commerce platforms and sell more with less effort. Thanks to Shopify magic, your AI powered all-star. You know, nothing gets me and Jared more excited than when our guests get that coupon code in their email when their show ships or to everyone out there who loves change law podcast and can go to merch.changelaw.com and get your favorite threads to support our podcasts. It is just the best thing ever from stickers to threads. All that is at merch.changelog.com. And did you know that Shopify powers 10 % of all e-commerce in the U.S.