70 | Data Scraping to Solid Decisions: Mastering AI for Unmatched Market Insight with Dennis Tröger

0:00Welcome to a live episode of Leveraging AI, the podcast that shares practical ethical ways to leverage AI to improve your efficiency, grow your business, and advance your career. This is Isar Maitis, your host, and I've got a really amazing show for you today. If you've been following me, this show, or even my previous podcast for a while, I'm a huge believer in making data-driven business decisions. Now, this is awesome, but there's three big problems with making data-driven business decisions at scale. Problem number one is you need the data. You somehow need to find a way to collect relevant data that will allow you to make these decisions.

0:40That's problem number one. Now, by the way, that's why most of the companies who do this at scale are easily large companies who have a data team or they hire a consultant to help them get the data for a one-time thing, but it's very hard to do ongoing. Problem number two is, okay, let's assume you have the data and you found a way to collect it, how do you analyze it? And analyzing quantitative data is relatively easy. There's multiple pieces of software from spreadsheets to more or less any platform you have that will allow you to build dashboards and see quantitative data. But qualitative data is very hard to analyze because you have to either read or listen or whatever to a lot of information and then categorize it and then analyze the stuff based on its categorization.

1:27And it's a lot of work. And so that's problem number two. Problem number three is, let's say you've done all of that. Now you need to get to a point where you can actually create actionable insights that people in your business can actually take because otherwise there's no point in doing this. So these are inherent problems to large scale, continuous data-driven decision-making. The good news is that AI can help us address all these problems with very little investment and do this for any single company. So any company today has no excuse why not to do that unless you don't know how to do this.

2:05And that's why I'm really excited about today's show, because this is exactly what we're going to learn how to do. So our guest today, Dennis Stroger, is an AI automation expert, and his specialty is how to go and collect data through data scraping processes from relevant sources, how to put the data in a way that you can analyze it, and then how to analyze the data to make actual business decisions that can change the trajectory of your business. So this is obviously extremely valuable to any business out there. And hence, I'm really excited to welcome Dennis to the show. Dennis, welcome to Leveraging AI.

2:44In the next few years, AI technology will change our world dramatically. Whether you are a business executive trying to catapult your business forward, or just somebody who refuses to be left behind and want to advance your career, this is the show for you. I'm your host, Isar Maitis, a serial entrepreneur and an AI enthusiast. You'll hear invaluable practical tips from innovative business leaders, AI practitioners, and some of the brightest AI minds in our world today on how you can leverage AI in ethical ways to advance your career and grow your business.

3:25Perfect. Awesome. Quick note before we get started. Today's episode is brought to you by the AI Business Transformation course. It's a course we have been teaching since April of last year. A lot of people have been through that course. It's mainly for business leaders, but anybody in business can benefit from it. It's a really transformational course. It's four sessions of two hours each, so four weeks. And every person that has taken it has completely changed their business around using different AI tools. And the course is available through our website. So if you're interested to know more about that course, you can find it over there.

3:58The course was completely sold out between January and end of March, but we're opening another cohort in the beginning of April and there's still seats on that one. So anybody who wants to really learn how to use AI in the business, please check the details. Obviously, you don't have to sign up if you're not interested. But now let's dive into how to make data-driven decisions with AI. Dennis, what's the first step? How do we actually get started? Perfect question. First of all, I'm going to share my screen. And since not everybody's going to watch it, I try to explain everything, what I'm saying as good as I can.

4:33So you can even follow through if you're not seeing my screen at this moment. You should see right now in my robot from my side and I started to call it how to scrape and analyze data from Amazon. So the first question we have to answer is the data we want to get easy to get. Are we allowed to use it? This is a question we should ask in the beginning. If it's our own data, this question doesn't arise because we are harvesting it from our customers and so on. But if we get the data from somewhere else, we always have to ask, are we allowed to get this data? This is why for this particular example, I've taken Amazon reviews as an example for this today's webinar, because this data is publicly available without logging in to Amazon.

5:20And that this makes it, this allows me to use it without being scared that something is going to happen. So the first step we have to see is, or ask the question, what do we want to do with the data in the end? And first of all, what I'm going to show today is we will start with a simple scraping of Amazon reviews. So we will have a look on one Amazon product out there. And we're going to scrape the reviews from it, and then try to make business decisions on what we learn from this product and the reviews we get. Because this is what you already said and mentioned, if we make data driven decisions, it's much better.

6:00And this is the first time that we can do sentiment analysis and can do quantitative analysis pretty easily by asking AI. And this is what we're going to do. So the first step is we will scrape the data. We will put it into an easy to read format. And I will choose for this, because a lot of people are using it, Google Sheets, because it's a very easy tool to use. And most of businesses node already. And then at the end, we will do analysis of the, okay, sorry for my English spelling, not the perfect one. So this is what we're going to do today. And when it comes to scraping data from Amazon, I want to show you two paths you can choose to get this information in this particular case.

6:47First of all, there is a platform and it's called Appify. I hope you can see it. Epify is a platform where you can choose in a store different tools you want to use and different scrapers you want to make use of to get information out of different services. For example, what I just simply did, I take Amazon reviews and what I get is a lot of different scrapers who all are pretty much the same. And sometimes it's just a question, how reliable are they or how much do they cost and do they offer big packages for what you want to do. For our little example today, I've decided for the Amazon Review Scraper from Jungle Bee here.

7:33It has 1 ,800 active user. That's actually quite good. And when I'm entering this Review Scraper, what I can see here now is, you can forget most of the information you're seeing here. Most relevant for us is you can do here a trial period of 14 days. I already subscribed to it. And what I have to do now is I can add here in this search field, which product on Amazon I want to script. Basically the URL of the product. Exactly. This is exactly the point. You just go to Amazon, you're searching for any product you want to search for. You're just going here to the upper URL bar. You copy this, you would copy.

8:17And all you have to do is you put it here, pair pasting inside, and that's pretty much it. This is like the easiest way how you can get information right from Amazon. And all you have to do is here to save and start. And now what's going to happen is this tool has now started in the background. It will start in the background. And in this case here, in a few seconds, we will see that the virtual machine in the background will do its work. and this might take, depending on how much you want to scrape, maybe one minute to two minutes and then the results will be there. I actually did this already.

8:54You can see now that it started, that we are now in queue and that the process starts here. And when I go here to runs, then I can already see here how many results have already been scraped and how much it costs me. And the beauty of Epify is that you have$5 of usage per day. So I don't have to pay for it, even though it says it costs 5 cents now, roughly. I want to pause you just for one second, just for people who are like, oh, but I don't sell products on Amazon and I don't care. The idea here is the concept, right? So you can scrape data from other sources. If you sell software, you can go to places like G2 and scrape that and see customer reviews for yourself and your competition.

9:37if you are in whatever selling stuff on other platforms and you can go Shopify and see what's happening there. Or if you just want to scrape data from your competition, you can probably do that as well. So the idea here is not the particular fact that we are scraping data from Amazon. The idea here that there are tools out there that for free or almost free will get you a huge amount of information in a format that now you can digest. And we got to talk about that in step two to analyze whatever you want to analyze, either your own products, your competition, what's happening in the market, what's happening in the economy.

10:16It doesn't really matter. If you find, if there's a reliable source of data, there are tools out there. And Appify probably has scrapers for most of those, where you can go and grab data for anything you want and bring it in the format that you'd be able to work with. Well said. And thanks for lifting this concept to a higher level because what you said is actually pretty much true. The Amazon reviews is just an example everybody can grasp and it's pretty tangible. But as you said, the important thing in business is that most platforms you're visiting daily, which are open to the public, They can be scraped.

10:55And as you said, there are so many scrapers out there which allow to scrape G2. You mentioned, for example, we have a G2 product scaper. Then there is product hunt. I'm pretty sure when I'm looking for this, we have Etsy scrapers and so much and so forth. And they pretty much all work with the same mythology in the background. And if you really can't find something here, then there's always another tool you can use. And I won't show it because it's a little bit more technical and I don't want to deep dive too much. But there's another tool. It's called Rapid API. It's pretty much a little bit more for developers so that you can do so-called API calls.

11:39API calls is when you are having interfaces between machines and you can do very large amounts of requests with coding. This is just, if you don't want to have a look here, that's pretty much fine. I just want to show it because it's something I'm using very often. Often for our example here today, we will stay with Epify because this is something everybody can use. And now after a few seconds, I see here that the Amazon review scraper already run through. And I see here that I have 100 results because that's the Mexican one I have now took here because that's what I needed. And now the beauty is, I just can say in what format do I need it?

12:21Do I need it in CSV or in Excel? That's like the common thing we would do. And if I now download, then a few seconds later, I have now the data here on my MacBook. You can't see it. I know that I'm just opening the file. So you see now that I have, what is the country code is empty. I don't know why, but it's what we have right now. We see if the person is verified, what is the position of the comment? We see the view URL. We get a lot of information of one of those comments. And what we will do now is we go into the webinar sheet here on Google Sheets that I already prepared. And Izo, please tell me if I'm zooming in too much.

13:02No, you're perfect. So again, quick recap. We used Appify, one of the tools in Appify, and there's hundreds, if not thousands, to scrape the data. We downloaded the data as a CSV. Now we're uploading that CSV into Google Sheets so we can start analyzing the data. So it's perfect. So in this case here, I'm going to use the same sheet, say import data. And what we have now is exactly what I showed you before in numbers from Mac. And I get now that I have a lot of rows which could be interesting but aren't that interesting for me. So I'm going to delete the obvious empty ones to get a little bit less the product.

13:45ASN I, that's an internal number of Amazon. I'm going to delete it because we know it's the review ID is not interesting for us images. We don't need. So I'm going to remove them as well. The title we will leave the URL category variants and what we have left over after a little bit of looking, what kind of data we want to have. we see that we are left and usa is here repetitive so we don't want to take this as well and you see now that we are left with a few categories of content and we can start to work with it i will remove this one as well so now we have all the data all those results and all those comments and all the titles in one google sheet after a few seconds now we talked a lot between but This is something only took a few seconds.

14:35And it depends, of course, on the number of requests you do. But in this case, I will delete the is verified as well. The position we don't need and the rating score. We are now left with our results will be biased. Because when I see now we have only got the five stars reviews, we know that most of those comments will be positive because otherwise you don't have a five star review. But sorry for those buyers. Something I haven't thought about before when I scraped the data. It's all good. For those of you who are not looking at the screen, we're looking at now only four different columns, the date, the rating, so star rating, the review description, so the detailed review and the review title that appears on top.

15:16So that's all we have. And because we scraped only 100 reviews, then they're the ones at the top of Amazon and they're mostly five and four star reviews. So most of them are obviously going to be positive reviews. That's what Dennis meant as far as the bias. I assume if we picked up a different product or if we scraped more than 100, we would have gotten a bigger variety. Yeah, that's well said. In this particular case, I want to show you something that really can help us because now we have the data and now the most important part begins. I have the data now. Now I have to ask myself, what do I want to learn from that data?

15:54What I've seen when I've done my AI hackathons in the past, what people did. I'm just doing an example. They went here, marked a lot of comments, switched to chat GPT, and just asked, tell me what we can learn here. So I just copied all of those comments, and now I will drop them into chat GPT. And what we will get out of it is like pretty much nothing, because when we are unclear of what we want, AI can't help us to determine a good result. And to make this even a little bit more clear, for this particular case, what I will do is Google Sheets allows us to have something what is called an AppScript.

16:42AppScript is something where you can define your own functions in Google, for example. And I can show how easy it is to get ChatGPT to write a function like this. And so I will go to chat GPT here and I will say, please write a app script function that shows a popup and says, hey, it will do this. This JetGPT will give us all that's needed to copy and paste this. And I have now here a function that's called show popup. And when I copy this and post this into my file, I can save it. And now when I run it, the popup will show here. So this is nice, but nothing special, nothing fancy is happening here.

17:36But I just want to show the theory behind it. And what we are going to do now is what is also possible to do is we can write functions. For example, if I had five numbers or two numbers here, four, five, and three, I could say sum up those in Google Sheets. And then I get a result back here like an eight. And we can write custom functions. Why is this interesting for us? I'm going to show you. What we can do with Apps Script is that we can call the API of JetGPT in the background. For our example, and because it's a little bit cheaper, and for you, it doesn't make any difference, I will now copy and paste my code, and this is fully generated by JetGPT.

18:26I haven't done anything here by myself. I've now created a function called call Mistral, what is an alternative tool to chat GPT. And as you can see here, in this particular case, this function can be used to give it something to work with. So what I can do now is say call Mistral, who was the second president of the United States. I should save it. I haven't saved it, and if it isn't saved, then it won't run. In a few seconds, the second president of the United States was John Adams. He served as president from 1797 to 1801. So we get an answer from Mistral in this case for something that we asked it here.

19:22And now what we can do is we can have several columns up here and say, for example, does this comment include information how to improve our product? Simply answer with yes or no. Only one word. It could be that it sometimes answers with more. But what we can do now is we can say that we want to call the Mistral API. We will use what is set in this first cell up here. And we add simply the comment here at the beginning. And what's happening now is Mistral is going to analyze it. And we see now that Mistral thinks that in this review is something how to improve our product. So we can have a closer look now on if it doesn't work well.

20:29And we see now, yes, this comment includes information on how to improve the product. It mentions specific concerns such as limited internal storage and Joy-Con drift. I don't know what Joy-Con drift is, but if it's your product, you hopefully know what the people are writing about. So what I can do now is I can simply, maybe from Google Sheets, drop this down. And now I do a lot of API calls. And hopefully, I know I made a mistake because I haven't said that the first cell needs to be always the same. So I want a quick recap for people who are just listening to what we're doing. So we took the data from the scraper, dropped it into a CSV.

21:11and what we've done now is we've taken a app script which you can create on your own and would you be able to share it so we can put it in show notes the actual script yes yeah okay so dennis will share with us we'll put it in the show notes so all you have to do is in excel is go and create a new script which is in the drop down menu on the top paste that script and then you can basically communicate with a any api of a large language model which means you can now create talk to the large language model from within the data ranges of this particular set of data. So what you can do, and that's what we're doing.

21:55So in the top of the columns, we're asking a question, which is basically the prompt that we're sending to the large language model, and then copying that down to look at every single row separately. So every single row has a different review of a product or a service or a software, again, depending what data you brought in. So now what you're getting is you're getting the ability to analyze, ask questions about qualitative data, written text of all this information across as many rows as you want. and literally every single one of them just goes and calls in the backend to the API to actually ask the large language model.

22:40In this particular case, Mistral, but you can do the same thing with Gemini. You can do the same thing with Claude. You can do the same thing with OpenAI, ChatGPT. All you have to do is go and grab, and there's a multiple YouTube videos that will show you how to go and grab the API key, but all you have to do is sign up. It's nothing, it sounds really scary if you haven't done this. It's as simple as following a one-minute YouTube video, and you'll see how to get your API key. You put it in that script, and that's it. And you're up and running. And you can even do that with several of them, right?

23:12You can ask the same question, to Chachupiti and to Mistral, and see what's happening in each and every one. But that's what we're doing. So we can now ask questions about hundreds or thousands of rows of data with one drag down of that one cell that basically calls the API. and you will ask the same question for every single row and will give us a very specific result for each and every one of the pieces of data that we brought in. You have summed it up very good. Wow, that was on point. Listen, I love this process. I use it a lot myself. So I find it extremely powerful. And what you said, I just want to underscore it.

23:53All you see here is no rocket science at all. Everything what you've seen here, you can easily find. And if you're a businessman or woman, you can ask a colleague or someone who is a little bit fond to do stuff like that. And he or she will pretty much easily can redo this. This is nothing special, nothing fancy. And you will find thousands of videos explaining it very good and very easily. The most important thing you have to take with you is that as you can see here in this particular case, and for this particular questions, we have a lot of yeses here. We have one yes to yes. And we have one question, one comment that actually says no, but otherwise, oh, we have here yes or no.

24:33So you see that if we take care of this, we could now create a filter in our Google Sheet. And for example, only let us show which of those actually were really yes or no. And then we don't have to scrape through 10 of thousands of product comments. We'd only have to scrape through a few. To explain why this is important, so what Dennis is saying is you can filter. So you've done your first question should be a very high level question to let you know which of these pieces of data are relevant to what you're actually trying to find out. Now, since you scraped, you're scraping 100 % of it. It might not all include the stuff you're looking for.

25:14So in this particular case, he asked one question. But again, going back to our examples, let's say you're taking, I'll take a different example that you can use without scraping. You can take all your recorded sales calls, transcribe them with any tool that transcribes, bring the data in here, and now you can do the same thing. In which calls are there specific objections raised by the client or the prospect? And it's going to tell you yes or no. And then you only care about the ones that actually had objections if you're trying to analyze what are the biggest objections you're getting. So now, instead of running the next question on every single row, which doesn't cost a lot of money, it's like fractions of cents for every single one of those API calls.

25:57But if you have 10 ,000 of them, it starts adding up. And being able to filter before you do the next step based on the first question is very helpful because now, instead of running this next question on 10 ,000 comments, you're going to run it on 600, which is obviously going to save you a lot of money and time. So how do we do the filter? That's the next question. That's a very good question. We have now the option, we can do two things. As you said, we could either say we do another question, which is very high level, because there might be a lot of questions we want to have answered on a high level thing.

26:33And what we could do now is we could write a simple, we can do two things. There is a little bit, I'd say, the easy way would be we just ask JGPT to do the following, we'd copy this output and that in this output, there's already a yes inside. And for some, there's only a no. And what we can do is we go to JGPT and tell us, please write a Google Sheet formula that checks if a text contains the word yes. And what we do now get is that it simply says here, in this case, we should use the search function used by is number function. And now what we can do is pretty easily here, we can copy and paste this here.

27:30And in this case, I get back a yes. So instead of the yes, what I could also have done is that I now start here to do another call of Mistral. And now I can say, listen, does this comment include something about the product itself? Is that a good question? I don't know, but let's take this for the sake. itself or yes or no. No, if the review only says something about other things. I don't know if this is a very good question, but I will use it right now. So now, because there's a yes in this, in the answer, we get, yes, this code includes something about the product itself. In this case, the Joy-Con drift, I think, has something to do with the controllers that can be used.

28:27And if I now do the same thing, I simply run the same prompt now for all of them, then I get no answers only for those reviews which have yes before. So I can save a lot of data and I can pinpoint certain things I want to learn deeper and deeper. And this I can do, as you said, in the sales call, for example, I wouldn't start with, you could say, maybe the person has, we can ask, is there objection in the sales call? Yes. What are, please name now all the objections in the sales call. And then suddenly we can go deeper and deeper and deeper. And then you could ask, okay, please now show me what were all the answers that the salespeople had for this objection around this.