128 | The strategy and use cases used by a company to implement AI successfully with Peter Gostev

0:00Hello and welcome to another live episode of the Leveraging Ad Podcast. We have a really special episode for you today. Special number one, because I'm in Florida and there's a hurricane around us. And it's still pretty far, but I really, really hope I'm praying for the hurricane gods to keep my power up and running so we can finish this live session. But the other reason why this episode is special is maybe the two questions that I get asked the most. The first one is, how do I get started with AI? But since a lot of companies are kind of already gotten started, the second question that is very close second is now we kind of getting started.

0:42We have a few use cases, mostly specific individuals are doing like their own thing, but we don't know how to go from that to a company wide implementation of the process of how to implement AI in a more structured way as a company. I'm happy to tell you that's exactly going to be the topic and the focus of our episode today. Our guest today, Peter Gostev, is the head of AI at Moonpig. Those of you who don't know Moonpig means you're not in the UK. If you're in the UK, you would know them. So Moonpig is a really large company. They're actually a publicly traded company in the UK. And they sell personalized gift cards and personalized gifts.

1:22They obviously sell them online, but they deliver them in the real world, which means they're a very interesting mix of an online business combined with a real world business that actually has inventory and shipping and supply chain and customer service and everything that comes with living in these two universes, which makes Peter the perfect person to have this conversation with. So we are going to be discussing both strategy, how do we actually think about these concepts of AI implementations from a company-wide perspective, which is from his seat, that's what he's paid to do. So hopefully he knows what he's doing.

2:00No, he does. And you'll see that in a few minutes. But then also we'll dive into a few specific use cases that are practical ways that they're using AI right now at Moonpig. And so I'm really excited to welcome Peter Gustav to the show. Peter, welcome to Leveraging AI.

2:21In the next few years, AI technology will change our world dramatically. Whether you are a business executive trying to catapult your business forward, or just somebody who refuses to be left behind and want to advance your career, this is the show for you. I'm your host, Isar Maitis, a serial entrepreneur and an AI enthusiast. You'll hear invaluable practical tips from innovative business leaders, AI practitioners, and some of the brightest AI minds in our world today on how you can leverage AI in ethical ways to advance your career and grow your business.

3:02Lin, thanks a lot for inviting me. I'm looking forward to the conversation. Yeah, same here. It's very rarely that I get to talk to somebody who's running AI in that size of company? Like I talked to a lot of AI experts who are either consultants like me or work in smaller businesses. You really have a unique perspective because you're working on a sizable company. And again, I said in the beginning, it's a unique company because you're both in the digital and real universes. Tell us a little bit about from, again, a 30 ,000 foot perspective. How did you even get started with the process? What were the things that were done from a strategic perspective in Moonpig to even set the ground to enable the stuff we're going to talk about later?

3:48Yeah, sure. So I joined the company about, I think it's about nine months ago now. So we've certainly done things before I joined as well. And I think the recognition was that we probably needed more concerted effort around it where we needed to, apart from just general, please go try things and experiment, we needed to have some center of gravity where those sort of experiments could get going. And the hard thing about Generative Fire specifically is that people don't really have intuition for what works and what doesn't work. And that's why I think center of gravity is quite useful and that we've got a bit more space to experiment, to try things out, build out patterns for kind of things that work.

4:32And that was really the idea behind the team. We've got a central team. It's a fairly small team where we really experiment, we try things out, and then we partner with different other parts of the business to really help them also deliver. So we've got mixed mode. In some areas, we build new projects directly. We test ideas out and implement them. And in some other cases where we've got already a strong engineering team with some problems they couldn't solve maybe or couldn't easily solve on their backlog, we would partner with them and help them deliver better. So there's a kind of a bit of a central unit where we experiment.

5:13But still, I think what's important is that I've seen a lot of teams, maybe in my previous roles, in large organizations, where because it's so hard to innovate in the existing teams, they quite often would take innovation outside of those teams. It does help for some time, but then you can't actually deliver anything. So I'm a big fan of maybe experimenting a little bit, but you have to really get it to the teams who would actually build it and own it. So that's really the key idea behind the model. Yeah, I want to ask a follow-up question. But first of all, I want to thank all the people who are joining us live, both on LinkedIn and on the Zoom call.

5:54So I really appreciate all the people that are here. Feel free to introduce yourself, say where you're from, share a little bit about your company, share your LinkedIn link. If you're not on the LinkedIn side, if you're there, obviously people can see who you are. And if you have any questions, please feel free to write them in the chat. I'm monitoring both what's happening on LinkedIn and on Zoom. And if you want to chime in and be a part of the conversation, please go ahead and do that. So my first follow-up question to what you said is, in a quick summary, you basically created this center of excellence for AI where people have both the time, because they don't have a full-time day job to do other stuff, as well as the resources, meaning some kind of a Zoom or a playground and access to licenses, and you're experimenting with different ideas.

6:41My first question is, who are the people? Which department they come from? How did you pick them? Because one of the first things I do when I come into businesses for consulting and the things that I teach in my courses is to build an AI committee, which is basically what you have created. Who are the people on the committee? How do you pick them? What would be your recommendations to how to start a process like this? The team that I have is a very engineering team. So when I look for people to join the team, the way I think about it is that they should be engineers who then learned to do AI. But to be honest, there's definitely more than one way of approaching this problem.

7:29And I've definitely seen some very strong people who don't have AI background. They come in from research perspective or product perspective, and they just learn about how to use it. And then they apply it very well for what they need to do. Then I think probably the most popular path is to go from data science kind of background, machine learning background, and get into generative AI. And I think this is an interesting path that I think probably it is very popular because it's the closest in terms of the kind of the technology because it is machine learning, it is AI. But it's interesting that the mode of operating, I think, is quite different.

8:07That what we want to do with Genesify a lot of the time is, to be honest, just use the API and do the easy thing, not overcomplicated. And I think what data scientists, data engineers, or rather the machine learning engineers are used to doing is really to solve very difficult problems, prepare the data, train the model, and so on. And not that I'm against that skill set. I think it's actually really critical that we still not use the people who are very skilled at doing the hard thing. We just make them use the API. So I think it's actually important that we give those people the tasks that they can actually do really well.

8:46But I can also imagine a lot of people moving into the AI engineer role and actually not being as happy with that role because they actually just build software and use the API. But I think what's really important is that I think we have not really reached any level of maturity using the APIs. And certainly not at Moonpik yet. I think we've got a hundred things that we could do. And I could imagine it's probably the same at most other organizations. organizations. And I think what we should focus on is just basically use the API and find good ways of using the API to just in a very reductionist sense.

9:25Yeah, great answer. I'll add my two cents and then I have a follow-up questions when you said you have a hundred things that you could possibly do. My two cents to add is that even if you don't have technical people, there's a lot of stuff that you can do without having any technical skills. Like in most of the companies that I work with, we just either use the chat platforms as is or the automation tools that are built on top of them, custom GPTs or Cloud projects or Gemini Gems or whichever platform they want to use, or that in combination with external automation tools like Zapier and Make combined with different capabilities from the AI still allows you to do magical things.

10:08And the thing that I always recommend to companies in that committee is always to have people from each department participate in the committee, because then you get the inputs of what everybody needs, as well as you get different kinds of brainstorming, because it's different kinds of people. Finance people think differently than marketers or think differently than customer support people. And so you get the brainstorming that is a little better. And the thing that I think that is as critical is you get champions in each and every one of the departments to actually use the thing that the committee puts together.

10:39So beyond the techies, and if you have technical people in your company, yes, you want a couple of them in there because they can do a lot more stuff than you can do without those scales, but it's not a necessity and adding stuff on top of that is actually very helpful. So now to my follow-up question, that was a very long comment. I apologize. But you said a hundred things and yes, there's probably a very long list of things. So I have two follow-up questions one is how did you come up with the list of things that you can do and then what was the process to prioritize what are the things you actually want to do and actually maybe just to build on your comment in terms of the opportunities i would say for us we break it up into three buckets one is actually just a little bit like what you said in terms of using the tools and then we've got everyone from i know hr to comms to crm everyone using basically gpt pretty much we built our custom gpts in the areas where it makes sense and but we still experiment with new ideas and train people up and so on so that's definitely one bucket second bucket opportunities where we where we need to do some engineering but they're quite the things that we're pretty sure will work we just need to put some engineering behind it and then we just make it happen and then the third bucket is the more exploratory innovative bucket where we're not even sure it's going to work and for us quite often it is maybe image models where we want to see what is quite a visual business for us at Moonpeg so working with image models is quite interesting, but it's quite tricky as well.

12:21But not only that as well, maybe audio, video, all of these things come out and we want to experiment. So these are three buckets. In terms of how we come up with opportunities, I would say that there is a more kind of corporate answer that we've speaking to stakeholders and collecting ideas, evaluating ideas, prioritizing ideas in the kind of formal way. And that happens a bit. I would say what's probably pragmatically what's the most important thing for me is whether it is deliverable. And in the first instance, is it a thing that people care about? That's a given. But the second part, is it deliverable?

13:03And that has two parts to it. One is, does it technically work? And sometimes it just wouldn't work. and sometimes the issues where it's a, for example, something that we need to bring seven data sources together. And you know what? It's just not going to happen. We've got better things to do. So we just don't maybe not prioritize that one. But the other one, other category, which I think is really important is, does the stakeholder care? Would they actually put time behind this? And would they work with you? And would they partner with you, resolve issues, provide you extra documentation and feedback and so on?

13:41And I think that's probably the biggest thing because we can build the best theoretical prioritization model to calculate the value, calculate the uplift and so on. And then if the stakeholder says, you know what, I'm busy for the next 80 months, come back later, and you can't get past that point, then that's the biggest issue. In practice, I would say, apart from all the things that we all know that we should do, I would say the most important one is, can you actually work with the stakeholder to deliver it? No, I think that's a huge point. It goes back to the people you want involved in the process, right?

14:15In the committee, if you have the people who are the stakeholders, or at least their representatives, who can say, yes, this is a great idea. But like you said, I'm busy between now and March. So talk to me in April, then it's not a good idea to go down that path, even if on paper and the Excel file, the bottom right corner shows it's going to be a good ROI and so on. So I agree with you 100%. Something that in general is very obvious from everything that you're saying, and I agree with that from my personal experience working with multiple companies, is at the end of the day, the hardest part is the people part.

14:48The technology implementation is in most cases doable, sometimes easier, sometimes a little more complex, but it's doable. It's the people part that is usually the hardest. And you touched a little bit earlier when you said about training the people on how to build GPTs and how to do that. How do you do that? How do you go into what kind of training is delivered to employees? At what frequency? By what people? On what platforms? Do you evaluate them on that? What's the actual overall training process on a company-wide deployment of AI? I'm not going to claim that I've cracked it for sure, but we are trying.

15:28It's certainly not a soft problem to a point. I think it is about the people. And I think we did a bit of a survey across our chat GPT users. And the biggest point of feedback was, I don't have enough time to try things out. And I don't have enough time to dig into it and invest time into it. And I think that will be true because I think maybe if you're listening to this and thinking to yourself, or maybe that's easy, I use it all the time. The big difference towards probably 90 % of people is that it's not easy. They don't really know all the details, all the nuances about how it works. And sometimes when I speak to people who haven't used it before, don't really know about it much, you realize how much tacit knowledge you built up just using these tools every day about what kind of things work, what kind of things don't work, how much data you can put into it, what kind of data is usable and so on.

16:24So that's basically, that is a hard problem. So I don't think there's like a perfect answer. But I can tell you what we are doing at MoonPix. So we've got about nearly 200 users of chat GPG. which is a bonus is a lot i must say i didn't expect that many people to be interested and this is not really we did not make anyone sign up this was pretty much one slack message a number of months ago people just obviously teams it gets spread word of mouth and so on but i deliberately did not want to market it too much because i didn't want people just to sign up because it's cool and they don't want to use it so that that was the first stage and then we had a just kind of optional drop-in sessions for people to learn about how to use it that was part of it then also we had team sessions so i would try and pick out some teams who maybe expressed interest or for some reason i knew maybe they they want a bit more help and i would have team sessions with them and in those team sessions we would build a custom gpt together just go through a little bit of a training about how that would work.

17:31And then we also have sometimes big sessions with maybe the whole department. And we recently did one for our senior leadership where we also got them to build a custom GPT, which is quite cool. And I think it's probably the more senior you get, the harder it is to find time to just sit down and actually do the thing that you've been reading about. But I would say the overarching lesson out of all of this is that you must do the thing. just you have to try it it's just there's no the thing to realize is that there is no possible way how without trying you could develop the intuition for what works what doesn't work like what the precise intuition and if you don't do that then how could you know what what use cases are good or not and it's not that someone is i don't know not talented enough or something like that.

18:23It's not humanly possible for you to just emerge with that intuition. And even now, I'm still struggling to find a way to pass that intuition to people who haven't tried it much, and it's quite difficult. So I'm hoping I'll get better at that. But one other way that we do it is, what I find is that we can have all the training sessions, and then people still go back to their day job, and they don't really then necessarily find time to do the thing for four hours so sometimes i would still pick out a few people where we can't do the use case and we'll just sit down for half a day and just get it over the line and they're just that a bit more effort makes all the difference but yeah i could probably do that 30 more times and still have more ideas but i ideally what i'd like to do is to build more of a network where it's people more and more people do that we don't quite have that yet so there's definitely more that we can do in that space I want to summarize some of the critical things you said, because you touched on a lot of really important points.

19:25And again, I do this. I've been teaching AI courses to businesses, like private courses where companies hire me to do the things that you just said. Since April of last year, I taught hundreds of companies. And so I want to touch on some of the points that are very critical. The first and most important point is you have to block time on people's calendars to do this. And when I say block time is not give them a task on Slack, is actually have a meeting when they're going to show up and their department is going to show up and their boss is going to show up and they're going to be there for an hour, two hours, half a day, two days.

19:57That is a condensed time where you focus on learning how to use these AI tools. There is nothing that gets you more benefit than doing that, because as you said, people have day jobs. And even though people know this is important and they want to do this and they know the company is saying strategically, we've got to move in that direction, there's always more urgent stuff that happens. So this is the number one most important thing. The number two is what you said, that people have to experiment not with the concepts, but with the actual use cases that they're going to use. So the most effective thing that you said and that I do with companies is these are like mini hackathons.

20:34Here's the use case we're going to work on. Let's work on it for an hour, two hours, a day. It doesn't matter. But everybody sits in that room and working in a group or individuals, and we're going to solve this problem together using AI tools. And that does two things. First of all, it forces people to get their hands dirty. And like you're saying, the concepts are awesome. And you can listen to this podcast, which is fantastic, and follow people on YouTube and on LinkedIn and on TikTok. Wherever it is you follow people is great because it's going to give you a lot of ideas. but eventually you have your data, your company, your limitations, your co-partners, your processes, your licenses, your unique solution is going to be different than most of the stuff that you see by other people.

21:18These other people give you great ideas and starting points and cheats to jump through a few hoops, but you will have to figure it out on your own. And the only way to do this is to actually try out. And if you're going to try out for five minutes between meetings, it's not going to work. And then you're going to say, this is all bullshit and you're going to leave it. And that's going to be a big loss for you and your company. And you have to do these two things like free time for somebody who knows with a bunch of people who doesn't know to sit together after some initial training. So they have the basic concepts in place.

21:48What are the tools? What's prompting? What data you should not put in there? Like all these things needs to be there, but then dive into the actual hands-on, let's build this thing that solves this problem and let's invest the amount of time that's required because then people get the understanding of how it works, they understand what doesn't work. And then they also have something that actually does work in the end. And maybe it doesn't work perfect, but it works 70 % and they have enough data and knowledge and scale and excitement to go the other 30 % and then they're going to start using it.

22:16So great points. I want to move into the practical tactical side of things because I think we covered a lot on the strategy side. What are some of the use cases that MoonPig are currently using AI for in an impactful way that it's actually generating great results and people are excited about? Yeah. So one of the best categories that I would recommend to any business and certainly applies to us is basically looking at your unstructured data and seeing what you can do with that and the what i mean by that is any customer interactions any customer transcripts chat transcripts for example you could analyze them and say was this a good conversation or bad conversation what were the entities mentioned in this conversation was the did the agent do a good job of handling this and i know if they had a bad sentiment what were they talking about what which specific bit they're related on.

23:17Was this like a bug on a website, for example? And there are many things like that that you could just extract from the conversation. And I think most businesses, the way it worked for us otherwise is that maybe the agent raises it with their manager and then the manager raises with the developer team and hopefully get closed off, but it probably wouldn't happen. Then one of my favorite things is normally you get an MPS score or whatever feedback method you're using and maybe you get one percent of your conversations to get an MPS score whatever it might be you know I think I don't know what it is for us but it's like low digit numbers and but now you could get put it through LLMs and basically get the model to estimate what the MPS would have been for that conversation and it's not something we've deployed yet but through just light experimentation it seems to work really well so the kind of calibration seems pretty good so imagine now suddenly you go from maybe getting one percent coverage to 100 of coverage and by the way you can also do it fairly instantly okay maybe pragmatically next day now you can have a dashboard and you can have different correlations about why do you get that kind of i don't know why did you get that score you can dig into that a lot more.

24:40Maybe people talking about that specific problem. And so in terms of what we've deployed, we've done bits of it. The reason why we haven't done it instantly is actually data pipelines. And just we do have some data that we're pulling through, but not all of it. And the data pipeline is complicated. And anyway, real things like that that make it hard to actually deploy things in real life. So we're definitely going to push on that more. But in terms of high level experimentation that works really well. And in terms of... I want to pause you just for one second, because you touched on a few very important points.

25:17One of the maybe most magical things that these LLMs can do is qualitative data analysis at scale, which is something that was not possible, period, stop, before that. Because the only way to analyze large unstructured data like transcription of calls was to have people, lots and lots of people to go and listen to the calls, take notes, try to calibrate it in a way that they're all giving it the same level of score and whatever, which is impossible. If you have lots of data coming in and you have 50 people analyzing it somewhere in India where the math would make sense somehow from a cost perspective, how do you train those 50 people to give exactly the same scores?

26:00It's just very hard to do. And the only way to do this was to hire like a McKinsey who has this army of people to do a one-time project for you. Like it was impossible to do this ongoing. When I was running my travel company, I was running a hundred million dollar travel company. We had a call center to do customer service, to do outbound sales, to do inbound sales, like all of that. And what we had to do is there were people, full-time job people would listen to calls randomly because they can't listen to all the calls. And when there was something interesting happening, they would pick it up and we'll make it into a training session.

26:31And that's exactly your 1%, right? And now you could do this for 100 % of customer communication. It doesn't matter whether communication is verbal over the phone or on a chat or through email or through a third party tool like G2 software, if you're a software company, or it doesn't matter what the data source is you can take all the communication you have with customers as well as prospects and learn from it at scale. I want to ask an interesting follow-up question because you said you're already experimenting with this and I'm experimenting with this. So I just want to, I want to make it a little more tactical.

27:12How are you, even if you haven't deployed it and you just started playing with it, how are you practically planning to do this? Meaning, are you sending the data to LLMs and then putting it back into some kind of a database? Are you using a third-party tool like Intercom or something like that to do some of the work? What's the practicality of what you said? The likely pattern for us is that we'll get data out of a tool that we're using for customer service, then putting it probably in Snowflake, so just via ETL process. And in there, we've got choices with snowflake they've actually got some inbuilt lms now so you can do the analysis directly there so we we just got access we're experimenting with that another option was just to pull data output it through lms via whatever open air whatever ai tool you're using and then there's there is a question of how we exactly make that surface that to the end users one pattern we've got is to have a slugbot and basically just make that visible or just have passive updates via slugbot for now so it's not going to be like a chat to a data kind of use case but you just get updates and people subscribe to the channel and so on and something i want to explore as well whether we can make it more available as well to the managers directly for example in customer service centers so they it'll be more powerful if they've got access rather than people in some other departments have access.

28:42So there are probably a few things that we can do there. But yeah, we need to sort out a few of those steps in terms of pulling data in a way that makes sense. Things like that are hard. Yeah, it's a funny thing. AI doesn't really make that easy. Like you still need to do the data pipelines and so on and visualization and deployment. So there's one bit of it is a lot easier and then or it at least became from not possible at all to possible. but other things are pretty hard as well. Yeah, two cents to that and then I'll let you continue because I paused you in the middle of a sentence. This is a really important point for people but two things that I want to add quickly.

29:23One, you can test it at small scale on your own without any developers very quickly just using tools like Make or Zapier or NA10 that can grab any transcription that shows up on whatever platform you're using to transcribe it. And by the way, if you're not recording your interactions with your clients, please start recording them. Even if you're not going to use them now, at least six months from now, you'll have historical information to work with. So start recording everything. But you can literally grab the transcription of the recording from existing tools that either Zoom itself or Teams or phone recorders or whatever it is that you're using and run it through an automation tool with a predefined prompt through a chat GPT or a cloud or whatever and get whatever summary you want.