Inside Instacart's AI-Powered Smart Shopping Cart | NVIDIA AI Podcast Ep. 302

18:58David McIntosh:You've talked in depth about the benefits for the shopper as well as the retailer. How are the store employees taking to the new systems? And are there benefits of the cart and all of the data flywheel stuff you have going on, as you described? How does that trickle down to improve the employee experience?

19:17NVIDIA AI Podcast Hosts:Yeah, absolutely. A couple of different ways. One thing I will say is that we found adoption of the product absolutely relies on employee support. employees have to be enthusiastic and supportive to help really drive adoption in the store. And so there's some things that sound simple, but are incredibly critical, like stackable charging. So if you think about it, the carts stack into each other exactly like traditional shopping carts, so they can fit modularly into the store.

19:43David McIntosh:And when you say stack, I'm thinking I push the cart, it goes across, the whole parking lot perfectly lines up in the next cart. Exactly.

19:51NVIDIA AI Podcast Hosts:And so you don't need to plug in each cart to charge. They charge it. And so that's huge because otherwise the staff would be burdened. They say, this is work for me. I have to plug in each cart individually. Right. So that element is extremely key. We now have the ability to deploy the carts outside. So you can have a stack charger, a set of carts outside in the cold winter weather. And so the stewards don't have to do anything differently in terms of their operations. The modularity element is incredibly critical to success. And just stepping back, when I talk about the complexity of the physical AI deployment of CAPER, it's these types of things I'm talking about.

20:30NVIDIA AI Podcast Hosts:Okay, let's say you deploy the carts outside. The Wi-Fi signal is probably pretty weak outside. What do you do about poor Wi-Fi at the start of a session when somebody is trying to log in with their phone number to get their loyalty reports? All those types of things, really getting the system to work at the frontier is very, very challenging.

20:47David McIntosh:I keep, when you're talking about these challenges, I keep going back to the mental image of being at the self-check. And somehow I put something on when I wasn't supposed to. And it says remove the item and I remove it, but I didn't do it right. Or something happens and you're in that loop. So with the complexities, everything you've described moving around the store and everything. That's right.

21:08NVIDIA AI Podcast Hosts:You know, what you described to that loop that you get into in traditional self-checkout. you've got typically an associate that's standing a couple feet away from you that can come over and help. With a smart cart, you're in aisle 10, there's nobody around you to help. And so it makes it even more important to get the basket accuracy systems right, to understand what customers are adding, what they're removing from the cart. And, you know, to maybe a final point on the trust of associates, this is why basket accuracy is so important because the weight system acts as sort of an x-ray, so to speak, for the cart contents that works in tandem with the visual signals coming in again, the sensor fusion system.

21:47NVIDIA AI Podcast Hosts:That's really important because ultimately associates want to make sure that as customers are leaving the store, that the total is accurate. And so building that depth of technology along with retailer facing tools where they can keep an eye on the carts as they go throughout the store and they know the system is highly accurate and performant. that's all part of how you make these deployments really successful at scale, even with the chaos of a store environment. Another thing I'll share with you that is pretty interesting is we announced a technology called StoreView, where Instacart shoppers that are shopping an order for another customer can actually scan the shelves of a store with their phone to build an understanding of what's on the shelf.

22:31NVIDIA AI Podcast Hosts:Yeah. CAPER also with the side facing cameras, as I described, can build an understanding of what's on the shelf. And so that understanding could then feed notifications to the store, to employees about things that are running out of stock. So then they don't have to be reactive. They don't have to wait for a customer to tell them, or they don't have to go and do these lengthy checks of the store. It comes to them proactively. And so that's another example of how this understanding of the store, of the shelf, can really improve the customer experience, improve it for users, improve it for retailers, improve it for associates.

23:06NVIDIA AI Podcast Hosts:And so that's really the ecosystem that we're building. Whenever we build features, we've got to think about the users, the retailers, the CPGs, the store associates. How does it all come together to deliver a win for each component of the ecosystem? Sure.

23:20David McIntosh:David, with all the data you're gathering and everything you're able to do with it, you know, as you said, from from Instacart's depth of years of serving billion plus orders to building store views and this kind of thing. One of the great things about AI, right, is that the more good data you feed in, the more it learns, the better it gets. How do you do system updates for the stores? How do you deliver these continual improvements without the store having to close down for an evening for sort of inventory type things?

23:54NVIDIA AI Podcast Hosts:That's a great question. And by the way, I think that's one of the benefits of digitizing the store. As you digitize the store, you make it measurable, which means you can start to optimize it like software. And again, going back to an observation I made at the beginning, one of the things that was astounding to me when I first joined Instacart was that most retailers don't really have a good sense of what's on the shelf.

24:13David McIntosh:Sure.

24:14NVIDIA AI Podcast Hosts:Right. And I inevitably asked, well, OK, don't they have a point of sale? Don't they have an inventory system? You can track all the sales when people leave. So what's the problem? Right. Not so fast. The problem you have is people are pulling things off the shelf and putting them in their basket. So the item might have been available a minute ago, but then it isn't available the next minute. You have what's called DSD vendors. The CPGs often will have their own employees. And you see that big truck next to the store. That employee will be willing things on the shelf totally independently from the employees in the store.

24:43NVIDIA AI Podcast Hosts:So it's an inherently chaotic environment.

24:45David McIntosh:They're always the ones I ask where something is, you know.

24:47NVIDIA AI Podcast Hosts:And they're always super nice about it. It's like, I don't actually care. Yeah, it's exactly that, right? And so what we're doing is we're building the best understanding of the store. You think about this caper cart with the slam location system, the side-facing cameras. We actually had a slide in our GTC presentation showing the 3D map of the store that we're constructing. So think about all the things you can build on top of that. One area we're going is we've got a suite called AI Solutions. So what we're doing is we are bringing AI to our retailer partners. things like assistance on their website.

25:24NVIDIA AI Podcast Hosts:And so for example, both Kroger and Sprouts recently announced that they're gonna launch Instacart's a powered card assistant on their websites. But also analytics tools. So for example, you could imagine a world in which we understand that something is missing from the shelf. And an AI agent in the background kicks off and starts to communicate with the merchant. and say, hey, this is the fifth time this week, this thing has been out at 3 p.m., but the delivery comes at 4 p.m. How could we either make the delivery come earlier or maybe we need to adjust the quantity, right? And so you can imagine that once you make the store measurable, right, once it's observable, you can start to optimize it.

26:11NVIDIA AI Podcast Hosts:And even take the caper carts as an example. It's a software system. It's sort of like getting an update to an app, right? or your phone. We ship regular updates to our retailer partners, so the software's getting better all the time. And then, this is probably a little bit in the weeds, but from a technical perspective, some of these models about the store might refresh as frequently as 15 minutes.

26:37David McIntosh:Okay.

26:38NVIDIA AI Podcast Hosts:And so they're really designed to get as close as we can up to the minute view of what's happening. Because again, going back to the example of out of stock, The item might have been on the shelf 15 minutes ago, but then when somebody comes to the shelf, either to grab it and put it in their own cart or to pick the order for a customer online, it may not be there. And so that real-time understanding of the store that we're building from the more than half a million Instacart shoppers that go into a store every day, and from these caper carts that go through the store and are continuously scanning the shelf with the side facing cameras, that's unlocking a foundation for us to build new agentic experiences on top that start to automate the optimization of the store to then ultimately deliver a better experience to users that makes the associate's job easier and ultimately drives value for the retailer.

27:32David McIntosh:Yeah, you said the A word, agentic. What's your approach to using agentic AI right now? You mentioned, you know, the one agent kicking off and doing something in the background, but is Instacart, are you deploying agentic systems as part of what you're doing in stores? Are you looking at agents for sort of different expertise kind of tasks and lines of thinking? What's your philosophy on using agents?

27:57NVIDIA AI Podcast Hosts:Yeah, I think our vision is, I think, consistent with NVIDIA's vision, which is we believe that there's going to be experts for different tasks, right? And so the way that we're approaching it is that we're building the foundation model for grocery. And so that foundation model is taking in a couple different sources. It's taking in the 1.6 billion lifetime plus online grocery delivery orders, the 2 billion item catalog that we have. And then it's taking all the in-store data. It's taking in this understanding of not just the shelves, but also the clickstream. Where is a user pausing on the store?

28:35Going back to your example, a kid taking something out of the cart.

28:38NVIDIA AI Podcast Hosts:What convinces you to take something out of your cart? What convinces you to put that thing in? Where are you in the store when you're adding something? that you normally don't add to the store, right? And so it's that triangulation of where you are, what you're doing, building the best understanding of what is the most personalized experience for you in-store and online that we're then feeding into this grocery foundational model that's building the best understanding of users and the store, which you then can stack on top of agentic applications, whether it's a cart assistant that lets you plan your trip to the store and then take that plan and sync it to the cart when you're in the store or shop it within store mode from an app, right?

29:19NVIDIA AI Podcast Hosts:Or it's building associate-facing tools, like the example I described, or even CPG-facing tools. So another example, if you look at a lot of the ShopRite WakeFront stores we're in, you will find bread from a particular vendor being, let's say, half a dozen places throughout the store. That bread company wants to understand where in the store is my bread being bought? Sure. Where are people actually taking it off the shelf? And so today what we can do is give that CPG an understanding of stack rank list. Here's the top six places people buy that bread in the store. Right. And so then you can imagine the next step in that process is to use all the location data, the heat map, the 3D representation of the store and expert agents on top to say, you know what, CPG, bread CPG, you should put a seventh location.

30:13NVIDIA AI Podcast Hosts:and here's where you should put it in the store.

30:15David McIntosh:You should put an eighth location.

30:16NVIDIA AI Podcast Hosts:By the way, and it'll weigh that against the cost of a person not putting those locations in another store because they're burdened by those additional two locations. Or it might say, look, you know, in this store, you only need four. Take away those two locations and put that in another store. And so, again, those are all the types of things that can start to be unlocked as you build that really rich understanding of not only the experience in-store, but then online and really marry the two data sources together.

30:42David McIntosh:Yeah, yeah.

30:43NVIDIA AI Podcast Hosts:That's the continuously learning system.

30:45David McIntosh:Right, right, right. What's something that's maybe an insight, you know, that's been uncovered through, I mean, through all of this work, but using AI with grocers, with retailers, an insight that surfaced that maybe surprised you? Yeah.

30:59NVIDIA AI Podcast Hosts:Well, I'd say the first thing that was surprising to me is often the consumer value props that resonate the strongest seem very simple, but are very difficult to execute against. So take the running total as an example. It's amazing that the running total is often the number one thing that users cite when they use the cart. But to actually deliver that, as I stepped through before, how do you build an understanding of the basket and make sure you know what's happening? You remove things, add things, the cart hits bumps throughout the store. Just delivering that is a very, very challenging technical problem.

31:34NVIDIA AI Podcast Hosts:but that output itself was very simple, is extremely impactful for users. I think the second thing that's been interesting to see is how people, how their behavior changes depending on these different contexts. So what we've seen, for example, with assistance in the cloud, people are planning a trip to the store, using it differently than they would a traditional online grocery experience. A traditional online, so online, This is probably your experience with Instacart, right? You're searching, adding, searching, adding. Here's the thing I bought before, right? You basically know what you want.

32:10NVIDIA AI Podcast Hosts:You're building a list. You're checking out. Versus an assistant online user might say, hey, got a family of five. I have this X dollar grocery budget. You know, my oldest child has this allergy, right? Second one doesn't like fish. The third one only eats fish. Can you make me a meal plan? with that budget for this family? And can you do it for the next two weeks? Right? Those are the types of behaviors and queries that you start to unlock with these new agentic workflows. And then by the way, the behavior gets even richer because you then can take what you've done online and bring it in store.

32:49NVIDIA AI Podcast Hosts:And so that card assistant I mentioned that Quigger and Sprouts announced they're rolling out, that will exist both online in the e-commerce website that we're powering for those retailers, and it will exist in Kaper, in the smart card, which will then start to remind you as you go through the store and actually start to shape your behavior. And what you do in-store will then start to improve the online experience and vice versa. And so in general, we look for places where we can really push the envelope on user behavior and add new value to users with these new capabilities. It's the same. Another example would be take the bag-as-you-go use case with Kaper.

33:30NVIDIA AI Podcast Hosts:often people will put their bags in the cart, reusable bags, and then because they can take the cart directly to their car, they'll unload directly in the car. Those people might have shopped with plastic bags before, store provided bags before, but because of this brand new experience where they don't have to actually take anything out at checkout and put it on the conveyor belt or take it out and re-scan it, because it goes into their bag once, they then start to shop with reusable bags. So there's a lot of things like that we've seen where because you use AI to make the experience more convenient, more personal for customers, it has second, third order effects in the way that they behave.

34:07David McIntosh:Yeah, no, that's really interesting. The example of make me a meal plan just makes me think back to like one of the first use cases that I remember seeing kind of anecdotally, I guess, when chat GPT first broke and everybody got on Gen AI was, you know, here's a picture of my fridge. What can I make for dinner? And so that kind of assistance really resonates. I mean, you're the expert, not me, but it resonates with me of that whole like, oh, we had chicken twice this week already. Like, what do I do kind of thing? And that can be so helpful in the moment for sure.

34:42NVIDIA AI Podcast Hosts:Yeah, and I think I've been talking more about the user facing benefits as well. One of the things that we announced at GTC was that migrating workflows from CPUs to GPUs with respect to ads had really big benefits. We reduced latency significantly. We actually increased click-through rate in the experiments that we ran. Oh, no kidding. And so everything here, again, we've been talking about is very visible to consumers, brand new use cases. But when we think about the impact that AI is having at the edge and at the cloud, it's really across the board, right? We're pushing the frontier with physical AI with these carts in the store, with shoppers scanning the shelf to understand what's happening at the edge, but also by bringing these systems online, there's also impact that you can drive to recommendation systems, ad systems, and so forth, right?

35:34NVIDIA AI Podcast Hosts:And we're really seeing those two pieces come together. That's what we mean when we say, look, we're collecting millions of sensor inputs daily now. That was one of the things we announced at GTC, and then combining with our online set of data.

35:48David McIntosh:So David, what's next?

35:50NVIDIA AI Podcast Hosts:Yeah. You know, when I think about it, It's really the vision I laid out for you at the beginning of the call. It's on a 10-year horizon. We think that customers won't have to choose between shopping in-store or online. There won't be a solid line between the physical store and the online store. It's going to be one single unified mode. What you do online will plug into in-store and vice versa.

36:16David McIntosh:And it just continues.

36:17NVIDIA AI Podcast Hosts:And it's going to be a continuously improving loop behind the whole system that ultimately makes the shopping experience more personalized, more seamless.

36:26David McIntosh:So what does it say about me that as you're describing this, I want to like slow down as I drive by the store, hit the button that opens my hatchback, the groceries to get pulled in. Like, you know, you're talking about that kind of there's no line between online and offline, right? I'm like, yeah.

36:42NVIDIA AI Podcast Hosts:Yeah. I mean, look, if you want to talk even longer horizon, I do think, you know, if you think about the data that we're collecting and the 3D maps, for example, of the store, the SLAM system, you could easily imagine that that starts to, now I'm talking distant future, that could start to unlock robotics workflows, right? Because again, you know location of things in the store, where they're at. You even know, by the way, how heavy is each item. Why? Because the cart is weighed every single item that you put in the cart. You've got multiple cameras looking at the item as it goes in. So you can imagine starting to create reconstructions of that item over time, you know, potentially in 3D, right?

37:22NVIDIA AI Podcast Hosts:And so you can imagine with that understanding, it could even start to unlock those types of use cases. But that, by the way, really goes back to the vision of Connected Store, which is how do you start to unify the experience? For some people, it's going to be delivery. It's going to be pickup. For some people, sometimes they're going to want to go into the store. They're going to grab a cart. They might even grab a portion of their order via pickup. It might be a hybrid experience where, you know, as you approach the store, you've already ordered. You say, look, I know I want to get, you know, this type of cereal, the second type of cereal, but I want to pick my own produce.

37:56NVIDIA AI Podcast Hosts:So you can even imagine maybe that picked order is actually waiting for you in the cart. The cart has your name on the front. You grab it from the line and then shop the produce yourself. And so that totally is consistent with the future we imagine, which is, again, that we're really breaking down that barrier between in-store and online. We're really turning it into one single unified personalized mode for customers.

38:21David McIntosh:It's exciting stuff. It's incredible. David, for listeners who want to learn more about Instacart, about the Caper Carts, about everything you've talked about in the future, best place to go online? Instacart website? Is there a technical blog, something specific about the carts, social media accounts to follow? Where would you direct them?

38:42NVIDIA AI Podcast Hosts:Yeah, we've got a lot of this information on the instacart.com website, both pages around our enterprise technology, also our blog where we posted the announcement with NVIDIA for GTC. And so for an audience that's looking for more of the technical details, we've got in that blog post a pretty lengthy description of what we're doing. And by the way, given the reaction we saw at GTC from the audience, We're planning to do a lot more of that, actually, in partnership with NVIDIA, because there was a broad appetite from the audience and really going deeper. But that blog post already has a fair amount of detail on really how we're pushing at the frontier of physical AI.

39:20NVIDIA AI Podcast Hosts:So that's one resource. And then I would say for a more consumer-heavy audience, we've got videos that you can search on YouTube, both videos we've shot up the cart, but also there's been a number of TV broadcasters that have covered CAPER and done their own segments.

39:37David McIntosh:Yeah, excellent. David, this has been great. Thank you so much for joining the pod. And, you know, as I said, I've always been a fan of the grocery store. I know where it is. I just like going in. So I'm looking forward to getting my hands on a paper cart myself. Awesome. Thanks for having us. Appreciate it.