Generating product imagery at Shopify

0:07Welcome to Practical AI. If you work with artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners for helping us bring you practical AI each and every week. FASI.com, fly.io, and typesense.org.

0:37Hello, it's your co-host Daniel here. Just wanted to let you know about an awesome event that's coming up this December of 2023. It's called the Advent of Gen AI. It's a hackathon that's being put on by Intel's liftoff program and Prediction Guard. And it's going to be a seven day journey into the world of generative AI. you're going to get access to some really cool hardware from Intel. You'll also get access to run prompts through the latest open access LLMs via prediction guard. And every day of the challenge, you'll get a new chance to show your generative AI skills and learn a bunch of cool stuff.

1:18So I encourage you to register. It's totally free. You can take part and learn all of the cool generative AI things that you're hearing about on this podcast. Find out more at adventofgenai.com. That's adventofgenai.com.

1:53Welcome to another episode of Practical AI. This is Daniel Whitenack. I am the founder at Prediction Guard, and I'm joined as always by Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris? Doing very well, Daniel. How's it going today? Oh, it's going great. You know, as our listeners know, or at least the ones that have listened to a lot of our shows. My wife owns a business, which is an e-commerce business. Quite a business. And next week, as we're recording this for those listening at a future date, next week is Thanksgiving here in the US, which means next Friday, basically, well, it's already sort of started, but next Friday to that following week, Black Friday, Cyber Monday is a huge retail and e-commerce extravaganza in our world.

2:45And so I'm really excited because leading up into that today, we've got the expert with us, Russ Mashmeyer from Shopify, who is project lead for spatial commerce and Magic Labs at Shopify. Welcome, Russ. Hey, thanks, guys. I'm super excited to be here. It's been cool to follow along with the podcast and just super stoked to chat with you guys today. Yeah, for sure. Well, I'm coming into this super excited because over the past, well, it's been 10 years, my wife's business, and at least nine of those years, they've been on Shopify, which means I have been in Shopify. I've dug into all the data behind I've worked with the Shopify API I've built chatbots on top of Shopify to like sign up wholesale customers I've like dug into the liquid code on the site so I'm like all about whatever I can learn from you today and hear about what's going on at Shopify I am super excited I didn't even know he was that much into it Russ you know Well, when you're the husband of an e-commerce entrepreneur and you're also a data scientist, occasionally favors are asked.

4:03I'm feeling very third wheel now. I just wanted you to know. Well, over that time, it's been cool to see how Shopify has added so many amazing features and is really powering a lot of huge brands, not only small brands, but larger brands. I'm sure you all are gearing up for a create. First off, I just have to ask, like what is the week leading up to Black Friday, Cyber Monday like at Shopify? You know, it's very busy as you can imagine. I was going to say what a loaded question. Hits the kickoff to the biggest shopping season of the year. Shopify powers just an enormous amount of that holiday shopping season.

4:43So you can imagine the teams internally are prepping for it. They are getting products like locked in place and just, you know, operating at their optimal, maximal performances, just to support the load that's coming in this upcoming weekend. But, you know, every year, we also launch this really cool live globe. That's a 3d visualization of all the live data and perched and like orders happening all around the globe in real time. So you see like, you know, orders streaming around this globe. And so this year, I've been also helping to lead some of those efforts. I'm really excited for that to get its annual debut this year.

5:20You might see some ideas we talk about today. Cool. Yeah, it's like the live view of Santa going around the world at light speed. Totally the Santa map for entrepreneurship. There's all sorts of interesting things that Shopify is doing, but specifically here we're talking about AI and what you're doing in regards to that. Maybe as we jump into that, Could you describe a little bit from a person that's embedded in this kind of e-commerce world and seeing what a lot of people are doing in various industries, various stores? How do you view the impact of AI on e-commerce specifically right now and kind of where it's headed?

6:07Like, where are we seeing the biggest impact in terms of AI right now in e-commerce? And I know we're going to be talking about some of the recent things you've done, but kind of across the board, how does it look to you and what are people thinking about? Yeah, I mean, Shopify was pretty early in kind of this new wave of AI capability to say like, hey, whoa, like this is a completely new class of possibility for the tools that we make for merchants and the shopping experiences that our platform provides on the other end to shoppers. And Shopify is really just kind of here to make commerce accessible and entrepreneurship accessible to everyone.

6:43And we're really excited about these tools as a way to kind of further democratize entrepreneurship. There are so many things you have to create and produce and ideas to develop and knowledge to gain about markets and positioning and strategy and branding. There are so many tasks that entrepreneurs have to learn and develop along the road to building a successful business. And LLMs, generative AI are all incredibly powerful tools to help accelerate that learning curve for new merchants and to help them kind of get up that curve faster and build better businesses. I'm curious, we all hear, you know, us consumers in the world, we hear about AI impacting retail and stuff.

7:26But for those of us who don't have as much of a view on it, can you kind of talk about how you've seen retail change in these years since AI is really the last few years with AI really kind of getting in everywhere. What are some of the things that might surprise people? You know, they kind of know there's AI there in the background, but they don't really know how it plays or what it is. You know, surprise us a little bit. Like, what's something I go, oh, wow, I didn't realize that. Well, you know, I mean, we started by just like adopting these tools in our engineering practice to begin with. We got some of the early previews of Copilot and started using that to help accelerate some of our development work early on.

8:03But really the place where we've seen it have the biggest impact in the near term is on tools for merchants. You know, like when we think about who our core customers are as Shopify, it's the merchants who we power with our platform and enable them to do really creative, amazing things, you know, at this scale that they never maybe thought was possible for them. Right. And AI is, again, like sort of a way to accelerate that work and give them more time back to, you know, instead of spending an hour and a half, like trying to craft the perfect product description, because you're not totally sure exactly what makes a good product description.

8:35You know, last year at our winter edition, we shipped a really simple tool where you just like enter in like a couple of raw details about your product and hit the magic button. And it just writes a well-crafted narrative product description that speaks to product benefits and all the great standard practices of writing a good product description. And you get that in seconds versus an hour of human toil. And so the place where we've seen AI really have the biggest impact early on is just in accelerating the work that merchants are already doing and allowing them for. And well, I guess it's e-commerce, but also like web content development.

9:10It's a very like multimodal thing, right? Like you've got these product descriptions. That's part of it. You've got product imagery. You've got website layout. layout, you've got potentially ads and integration with other platforms. Talk a little bit about within that space, because there's so many, as you mentioned, there's so many tasks to address within that space. As Shopify kind of looks at the merchant experience, how have you narrowed down on the particular problem sets that merchants really want to hand off versus those things? I know also from just being in it, marketing teams love to get in there and tweak things and be part of the process.

9:59But they also really don't want to do certain things too or things that are just kind of grunt work, essentially. That sounds like it's coming from experience right there. Yeah. Yeah. I mean, we have a word that we use at Shopify, toil, this idea of work that kind of has to be done but isn't desirable work to do. And so we look for toil that merchants do. And so we spend an enormous amount of time sitting down with merchants, talking with them about how they use our platform, what they want more out of our platform, what they wish they could be doing with their business, what they are doing with their business.

10:35And from that, we've learned a ton about, you know, what are the ways that merchants would like to spend their time? And then what are the ways that they just kind of have to, because that's the way the world works right now. And so I think the opportunity for us is to find those moments and to build tools, like particularly magic tools into those spaces that just sort of like make that go away. And when we do that, what we hope is that merchants will take that extra time that they have, that hour that they got back, not spending on that one product description or that one blog post or that one email headline.

11:07They're like, ah, should I use A or B? I don't know. And just give them a really easy tool to generate that content, make it really high quality, give them the control to adjust if needed, and then publish it really quickly. You already mentioned one of those, this sort of product description thing. Are there a couple other ones that you could kind of highlight just to give a sense of the breadth of how this technology is applicable in the space? So we've launched this suite of tools that we call Shopify Magic, right? It's our free suite of AI-enabled features across our whole Shopify admin. And these things crop up in a few different places.

11:44It can sort of help you take the power of your own data and make it work better for you. And we've applied that in places like email headline subject writing for marketing emails and things like that. We've leveraged it in the context of generating blog content. Obviously, product descriptions is another. And we're obviously really excited about some of the early work that we've also done in the image generation space. We recently released a Hugging Face space that I'm super excited to dig into more, I'm sure a little bit later. You know, when you think about a storefront and the kind of content merchants need to produce, it really falls generally into one of two categories.

12:22It's either text or it's images. We're really excited about both of these spaces and helping accelerate merchants there. I know Daniel has used the tools a lot, but if you had someone who was a novice and they were getting into business, and let's say they're starting it now. So they haven't been doing it. It's a new store. Chris is going to sell what, like socks to fund your animal charity. Raccoon socks. There you go. Raccoon socks. There you go. I like it. Christmassy raccoon socks. How's that? Will it keep the raccoons out of my trash cans? Will the socks get what they do? If you want them to, that's no problem.

12:59For those who are going, what just happened on the show? In the world away from technology and AI, I'm a wildlife rehabber. And I right now I have 20 raccoons at my house. So that's what that's all about. It's a full Christmas party. It's a full. Yeah. Oh, it's quite a Christmas party. You put 20 raccoons loose in a room. Oh boy. Yeah. Okay. So back to my, back to my new store that I just opened up. I'm excited. I don't have Daniel's depth of experience at this. What are all of the amazing things I'm either by myself or I don't have a lot of help. Everyone's tossed me to the wolves. I've come to Shopify because I know you have all these magical tools.

13:38Can you tell me a little bit about that experience from a merchant standpoint? Like on day one, what am I getting into? How should I think about it a little bit? And how do those AI tools directly impact what I want to start doing today? Yeah, totally. Well, I mean, I think from a merchant's perspective, if they were to log into the admin today, I don't think they'd be overwhelmed with the amount of AI tools all over the... I think today we really started with a focused approach that feels super seamless and integrated into just the activities that merchants are already doing. For example, product descriptions.

14:13Let's say you're a merchant, you've just started building your storefront, you're super excited to get that up, put a great face out on the web, and you're starting to build out your product catalog. You're starting to think about, how do I merchandise my products? How do I talk about them? You're a new merchant, you haven't really done this before, you don't know what the best practices are for product descriptions or you know if you want to create some seo content to kind of market your brand and product expertise in the space you can go into your product detail editing page for a product you want to add just drag and drop your images one of the really cool things and i'll say this because i'm also product lead for spatial commerce is you can also drag and drop 3d models into that image bin and it'll handle it beautifully so So there's some cool stuff going down the line with that.

14:59Raccoon 3D models. That's awesome. This is going to be an awesome site, Chris. Looking forward to it. So if you've got 3D models, drop them in there too. Those will display on your product detail page on your web storefront. And then when you get to that challenge of like, okay, now I've got to write a product description. Oh gosh, I haven't thought this through. I'm not really a copywriter. I went to business school and maybe I can write things, but I don't know what's a good product description. What does that even look like? I could go and spend, you know, an hour, two hours doing Google searches and combing through results and sort of like collating my own idea of what makes a good practice for product descriptions.

15:36Or I could just click on that like lovely little sparkle button after entering in like, oh, it's white. It's, you know, these dimensions. It's got these materials and just like, boom. And you've got this incredible, you know, text description of your product. It pulls from your product title. So like, you know, if you've mentioned that it's like this kind of product or this category, it gathers all that context initially, and then brings that to bear on the description that it writes, you'll have the ability to pick what tone you want that description to have. So we give the version some ability to kind of shape like, oh, do I want this to feel sophisticated?

16:12Do I want it to feel fun? Do I want it to feel like there's deep expertise behind this product description? And so I think those really simple tools just kind of placed seamlessly into the UI exactly where the merchant is kind of doing these activities today anyway, is really kind of the powerful first step that we want to take to introduce merchants to these new tools. And then we'll expand from there in some pretty powerful ways.

16:42What's up, friends? AI continues to be integrated into every facet of our lives. And that remains true because you can now index your database with AI. You can write more code, become that 10Xer you always wanted to be. And you can even draft a letter for a lease on an apartment or a new property. AI is everywhere. And it might be time for us to start questioning, is AI our friend or our worst enemy? And that's the focus of the three-part season opener of the award-winning podcast called TraceRoute Podcast. You can listen and follow the new season of Traceroute starting November 2nd on Apple, Spotify, or wherever you get your podcasts.

17:18And this show is all about the humanity and the hardware that shapes our digital world. In every episode of Traceroute, a team of technologists seeks to untangle the complex question, who shapes the internet? Seasons one and two gave us a crucial understanding of the inner workings of technology while revealing the human element behind tech. And season three tackles not just AI questions, but also how can we use technology to preserve the earth? Who influences the technology that gets made? And what happened to the flying cars we were promised? I think it's safe to say that the future of AI is both exciting and terrifying.

17:50So it's interesting to hear the perspectives of experts in the field. Listen and follow this new season of TraceRoute starting November 2nd on Apple, Spotify, or wherever you get your podcasts.

18:14I think how we initially started chatting back and forth was at least partially because seeing this hugging face space, which is really cool that you all put up and I know got a lot of attention partially in the merchant world, but also in the AI world around the community that's being built around open source AI tools on Hugging Face. And you had this space there that had to do with product photography. Before we go into the technology, the space, kind of how this works, could you kind of describe a little bit of the motivation behind this project? Because you mentioned product photography, but people might not, if they haven't been exposed to kind of e-commerce as much or worked on their own e-commerce store, they might not realize kind of what product photography means and some of the challenges around it.

19:11So could you kind of set up the motivation for this before we hop into the technical pieces? Yeah, absolutely. So merchants spend an enormous amount of time and money generating visual media that's like compelling, that gets people excited about their products, either the details in the design or the lifestyle that it might afford. you know, whoever buys it. And these images are really core to what drives a lot of commerce online, whether it's advertising, or whether it's building out an attractive storefront, like a web storefront, or whether it's appearing in various channels, you know, in different marketplaces as well.

19:49But not least of which is on the product detail page where somebody has landed a shopper and ostensibly they're interested in this product. And the job of those images in that context is to do the best job possible painting a picture of what that product looks like in somebody's life, as well as all of the details about the product. And early on, last year, when Stable Diffusion and other open source image models started to land, I got really excited about a future where you could imagine merchants just being incredibly more agile and cost efficient in how they create these images. And so we started digging in pretty quickly.

20:30We played with Dreambooth as soon as that was available and stable diffusion. And we started to see like, actually, could we train a Dreambooth model that could encapsulate the concept of a product and recreate it in high fidelity over and over and over again? That's like the dream, right? And we're getting closer and closer to that, but we weren't quite there yet. But some of those early explorations proved beneficial to understanding the space, understanding the technology, and thinking a little bit more deeply about some simpler ways that we might be able to bring this to market in the near term.

21:00When you think about the opportunity for image gen in commerce, I mean, it's massive, right? And the ability, the promise of being able to recreate your product in high fidelity in any scenario is kind of the dream. You know, you could imagine requesting any kind of lifestyle or product detail image and just in seconds getting that out the other end to use in your storefront or to use in blog content or to use in advertisements about your product. And that's incredibly powerful because commerce is always changing. Taste is always changing. Seasonality is a huge piece of commerce and thinking about how do you merchandise and market your products differently in the spring versus the fall and keeping up with the amount of imagery just required to drive that part of your business is really challenging.

21:46I think the reason that we got really excited is that we saw an opportunity to take the existing imagery that merchants had, either from past photo shoots or from humble like at home photography with their kind of mobile cameras set up on their kitchen counter or whatever they might have access to, and give them a tool that could not really change any pixel of the product itself, but otherwise completely reinvent the reality around that product. And so we started to work on, you've seen a lot of examples of this out in the market, but I think the key problem that we saw with a lot of these early examples of this, where you sort of, you do object segmentation to select the product and keep it, you know, sacred.

22:28You don't touch any of the pixels that you sort of guard and mask there, but then all the pixels in the background you reimagine with AI. And what we saw with most of these early tools, as I said, was that there was this real disjoint appearance between the product that got masked and safeguarded and the reality that was created around it. The camera angle, like it looked like, oh, well, this one was taken from above. But the original product image was from straight on. And there's no grounding shadows and there's no realistic reflections of the product in the environment. The pixels of the product and the pixels of the environment aren't speaking to each other.

23:02They don't know, one hand doesn't know what the other is doing. And so they can't knit those pixels, those moments of grounding around the product that really sell the illusion that it's part of this other reality. Those shadows, those ground reflections, seeing maybe some of the light of the scene hit the product object itself. And so we wanted to really tackle those grounding problems that we saw in a lot of these early examples. And so I'm happy to dig into all the technical details of kind of how we're into that. But that was really the opportunity that we saw was to begin to bring some of this magic to merchants really early before we're even yet to that perfect personalization of you upload a bunch of images of your product and now it produces them again perfectly at the other end.

23:45We can begin to bring really powerful tools to merchants in the space already, even with techniques like this. And just as a point of clarification, when you say grounding in this case, you're kind of talking about that visual context going across different as opposed to technical grounding with a model and such, just because we talk about both on the show. Yeah, no, it's a great clarification. Yes. And purely talking about sort of the visual aspects of the output image and making that product feel seeded in the new reality in some visual way. Before we hop into more of the details about how you actually accomplish this, I'm wondering how you see the kind of state of open source generative models in comparison to maybe some of the other platforms that are out there that do enable amazing things, but not in an open source way.

24:34It sounds like at least for your team, I don't know if it was kind of personally important to you and your team to kind of leverage some of this open technology or it was just like these things are openly available. They're licensed permissively for our use and they're enabling things that we couldn't do before. How do you view kind of the state of generative AI on the image side specifically? Because we've talked a lot in recent weeks about the text side of things and how maybe text generation models that are open compare in certain ways or other ways to close models. But I'm wondering from a team that's actually used this kind of image generation models that are open and licensed permissively, what was that experience like for you?

25:20It sounds like this grounding element was one thing that you had to deal with, but what was it like for you generally to kind of work through the details of getting the model, figuring out how to run it, figuring out how to scale it, maybe that sort of stuff? Could you kind of describe a little bit of that process? This is a really early field, so we're still figuring out what the tools need to look like and how to work efficiently. We were working on some of these early ideas in a very sort of falling over ourselves way in some notebooks, trying to collaborate and work together and just not sort of seeing the pace that we wanted to see in sort of our iteration speed.

25:57Our team is, you know, works really quickly. We work on kind of these three week sprints to just very rapidly prototype and understand a new technology space and develop some kind of potentially useful concept there. And so we needed a way to move faster. And Toby, the CEO at Shopify, is incredibly technically adept. He's an incredible developer in his own right and was really interested in some of the image gen work that we were doing in the early phases and suggested that we pick up this new tool called ComfyUI, which is an open source tool. So we're big fans of open source at Shopify. It's why we shared to Hugging Face, because we want to contribute back to that community.