30 | Unleash the Power of Your Business Data: An Actionable Guide To Training Models Leveraging Company Data. With Mayo Oshin, Founder & CEO Siennai Analytics, a private chatbot development company

31:04Yeah, it's called a system prompt. System prompt, perfect. Another follow-up question. So there's a big difference, I think, in my eyes between creating a tool for internal usage and opening the tool to clients. Definitely means of risk. If it's an internal tool, it could do stupid things. You still have somebody from the company looking at it and saying, this doesn't make any sense. Maybe I should go and do some additional research versus the same answer goes to the end user. Are there any best practices on what are the steps of deployment? So, okay, now I have done all this process. I have done all the stuff we talked about.

31:45I've created them all. I've created the user interface. I'm ready to go. What are the steps you recommend to customers, to clients to do in order to make sure that they're a enjoying the best benefits possible, but also reducing the risk of shooting themselves in the foot by deploying this to the world before it's fully tested. Yeah. So there's a, we spend a pretty good portion in authentication, right? Because if you just have the live URL, then anyone can go there and basically use your application. So the first question for it is typically, who do you want to have access to this information?

32:24And then we have a preset email. So it's usually provided by the company of the individuals. And you can see when they've logged in, when they've logged out, the questions they've asked, which is, again, is something very important for ongoing evaluation and testing, because you can see the questions the user has asked and the responses that came back from the model. So authentication is a big part of this. Another thing too is moderation. So if you have users asking inappropriate questions, they get flagged, right? And eventually you can blacklist them, right? So you can actually block them from asking questions.

33:04Another measure is known as rate limited. You don't want a situation where someone just keeps spamming questions, either themselves or using some sort of a bot, which will cost you a lot of money and cause your system to crash, right? So again, there's rate limit mechanisms in place. So those are the three typical things that are done so that from a security point of view, at least the authentication is in place and you don't have randomers who can just use the application. Interesting. Okay. So my next question is, how do you measure the success of these things? Because I go back to what you started with.

33:48I'm like, okay, I defined a goal. That goal hopefully has like a success criteria, right? I want to improve my customer service efficiency by 20%. That means that I can either spend 20 % less money answering the same number of customer service calls, or I can do 20 % more with the same amount of resources that I have. Doesn't matter. How do you go back and measure what's the action? And again, I go back to this problem that I just mentioned. There's five other moving things that are happening at the same time, right? I've hired additional more people. We've upgraded our ticketing system. Is there a good way to measure the actual benefit and impact on the business of deploying such tools that are quote unquote embedded into the actual solution so I can track what it's actually doing from a business performance perspective?

34:43So this is like also part of the prelim process would involve the client providing what we'll call an evaluation data set. So what this is essentially is what is a typical question being asked and what is a satisfactory answer for the end user. And so what the goal is here effectively is if we can replicate what a human being would respond like with 80 % or more accuracy, then you can deduce from that the amount of support you can cut down so we start with the data set provided by the client's evaluation data set and a huge part of the process is how do we keep optimizing this model to get closer and closer to what the client considers to be a good result so in the case with the training company they already knew the material inside out.

35:45They already knew what would constitute the typical questions that the users would ask and the answers they were expecting. And they knew the cost of having a coach or some other support person manually provide those answers. And so when we hit those goals, for them, it was like, oh, wow, this is essentially automated what a human being would do. So there has to be a benchmark in the evaluation process or in the prelim process from which the client or the company can look at and say, if we can replicate these answers, then we can automate the costs and effectively raise our business. Interesting.

36:31So basically you set the benchmark up front to the level that it needs to perform at, knowing that if you hit that level, that's the level of savings or growth opportunity that you're generating on the other end. Yeah, because again, what is AI and what is all the fear and excitement about it is automation of human manual tasks. So whether you have a sales team doing something over and over again, whether you've got junior legal petitioners drafting agreements over and over again, or in the courses or any other case. The point is, if a machine can do it with high accuracy and consistently, then you don't need those, you don't need as many employees or you don't need to pay as much.

37:23And you can do the maths on the machine because I think opening eye charges, depending on the model, but you're looking at anywhere from approximately$0.04 per, I think it's a thousand tokens, which is 750 words. And there's other mechanisms to save costs. We talk about cash in and stuff like that. So yeah, it must, because otherwise that means that the client didn't set their goals. Their goals weren't clear. Part of the setting the goals must be what does success look like? Awesome. I love that. Since you started talking about money and more the business aspect of this, roughly how long does a process like this take?

38:05And obviously it depends on the amount of data you have, but let's say to a small to mid-sized company who has, let's take a very simple use case that we talked about before, which is customer service. How long should a process like this take from starting to work on this to we have a model that we've tested internally that is mature enough that is providing the business result and we can actually start deploying it. So how long does it take? And roughly, how much does it cost to set up and how much does it cost to operate moving forward? Yeah. So if a client already has an idea of the data source they want to work with.

38:44So let's say the customer services, they have tons of website pages and maybe they have some other PDFs for internal docs. Yeah. And the ticketing system that they've used with all the open cases and closed cases and what was the process, like all of those things. So if they have that and then they want to use open AI, just keep it simple. Yeah. 46 weeks. Oh, okay. So it's not a huge project. no it's still yeah i'm speaking for my i don't know what other no i'm saying this what you gave is a great answer i was wondering if this is a month two months six months a year and a half so a month to a month and a half is a very reasonable time frame yeah because the main job for me is get get it because a lot of people just want to experiment with this stuff they don't really know what they want so my job is to provide clarity to ensure the data is processed properly.

39:42That's a lot of what I spend time on. Once all that preparatory stuff is done, we know what we need to do and it's a lot more straightforward. A lot of the issues come from clients not really knowing why. They're just doing it out of FOMO or they just want to keep up with competitors. They don't really know why they're doing it. Yeah. So. I have another question related to that. There are more and more no code, do it yourself kind of tools that do this process, right? Where I can go in and upload data to it or give it access to something and then get a chatbot that works out of the box. What are the biggest differences, things that I need to consider, good and bad, whether I want to do and use one of those tools or whether I want to hire a company like yours to do a more robust process?

40:33Or maybe what are the use cases or cut points where, okay, if you want to do this, you should go with this. If you want to go below the line, you should do that. Yeah, I think it's the same as any other industry or before AI, right? There's always going to be out-of-the-box solutions that you can use. But at the end of the day, if you want that personalized, high-accuracy solution, you have to go custom. There's no way around it. And I I know I've got friends who are running some of the most popular no-code solutions. It's impressive what they've done at scale, but on an individual level, you're just never going to have that.

41:16It's like food as well, right? You go to a restaurant where a chef makes food particularly for you versus you go to some mainstream chain, right? So what do you want as a company? And for me, for people who are new to the space, I think it's important they have a first experience that's mind-blowing, right? If they go and use some out-of-the-box solution or something that's just mediocre, then it proves their doubts. Because a lot of people are coming into this saying, okay, look, it's all hype, it's there. And so for me, I'd rather their first impression be something custom that kind of blows their mind.

41:58And then they get excited and they tell the stakeholders like, look, we need to bring this in. But if it's just some generic solution, which one is going to be cheaper for you. So I'll talk about the pros. It's going to be cheaper for you. They will, you can get up and running immediately. I'll say that's another benefit because it's a SaaS solution. Outside of that, the cons are, as I've said, whereas with the custom solution is definitely going to be a lot more pricey and it's going to take time because you need to be involved in terms of get everything up and running so basically what i hear you say that the out-of-the-box solutions are good if you want to maybe experiment knowing that the level of you know what i'll rephrase that i think if you're willing to accept a lower level of accuracy in the out, then it's a good enough solution, whether you just want it for experimentation or you want it for an actual solution.

42:58If you're in an environment or if your use case, if it works 70 % of the time, it's good enough, then perfect. Then you got a quick and dirty solution that you can do in-house without going through bigger project. But in cases where you want higher level of accuracy, which is probably most of the things in business where clients are involved, you probably want a more custom solution. Would that be a good way to classify the two solutions? Yeah, I would not even say it's up to 70%. I'm not saying this because I'm pro my stuff. It's closer to 50%. And also your options are limited. They focus more on simple stuff like PDFs, maybe what else?

43:42A website, stuff like that. It's more straightforward. Once you start wanting certain requirements. Maybe you've got some other types of your database or you've got some other types of quirks. You don't want certain names to show up, certain emails to show up. Maybe you want it to be in a particular design, for example. Maybe you want to, and a big one for a lot of clients is they want to be able to see the chat, the conversations of users and have control, basically. If you want more control, then I think that's the way to go. It's yours, it's your IP, you can do what you want. Last business question of this, we said it's about four to six weeks.

44:25How much are we talking about budget-wise that a company needs to be able to prep in order to say, okay, let's take this first step and see where it goes? Yeah, so on the lower end, you're looking at five to 10 ,000. That's just a basic plan. You can just imagine it as a lot of the stuff. It's similar to ChatGPT, but it's trained on typically PDFs and stuff that's from a data person point of view, it's not too crazy. But then you do get your own application effectively, authentication, all that kind of stuff. 10 to 20 is where you're starting to look at, you're more interested in a combination of two things.

45:10You're working with things like databases, APIs, things that are just a bit more complex that require more security measures. You're wanting to place a lot more emphasis on accuracy. So the basic one is just more, okay, building the solution for you. It's done, it's good. But then if you want to go from good to better, then you're looking at, okay, we need some sort of way to improve this. And then 20 up is when, yeah, now you're talking full on, basically full scale, working on ongoing evaluation, making sure that all the security boxes are ticked, making sure there's just a ton of testing. I'll say 20 plus is just usually enterprises, a lot of testing, a lot of security checks, a lot of trying to get as perfect as possible, I guess is the way to say.

46:04So obviously each of them require different resources from a developer point of view, which is why you have those different tiers. So I would caution anyone listening to not, I've noticed that some people try to shop the market and find lower and lower prices and all these kind of go offshore and do all these things. But then you risk having something that's not secure. You don't know what they've put in there. So be careful with that. I think it's very simple, right? You got to think about what is the level of risk that this can put you in, right? If this is an internal tool that's going to be an assisting something, okay, you can take some bigger risks and then maybe invest a little less upfront.

46:54If this is going to be basically allowing your customers to query your company in order to impact large business processes, whether it's sales, whether it's marketing, whether it's customer service. The future of your business depends on this thing working perfectly. And hence, you just got to define how much budget you're willing to put into this. And at the end of the day, I think if this thing works, if you applied it to the right problem, the ROI should be very easy to prove. And so it's invested$100 ,000 in this, but it's going to save you$1.5 million in manpower in year one, or it's going to make you$10 more million on top line in year one.

47:40It's a no brainer. It's a very easy investment to make. And so I think it goes back, and I love that we went full circle, it goes back to picking the right goal in where investing in building the right solution will yield a high enough ROI to make this a very easy business decision. Yeah, that's exactly it. Because again, I will say, as I said before, who are you replacing? Okay, let's say in the case of a private equity firm, you managed to replace two financial analysts. How much would that have cost you? We were talking about at least$200 ,000. So$200 ,000 versus the price range we're talking about, I think, again, like you said, it's investment.

48:28Awesome. Mayor, this was great. I think we covered a lot of stuff and we managed to do this without going too technical. So I think it's still a great conversation for any business person who's considering this and now can just go and understand the process and the steps and the pros and cons. And we talked about really a lot of stuff. I really appreciate you taking the time and sharing your information. if people want to follow you, learn from you, work with you, what are the best ways for them to connect with you? Yeah. So my website is enyaianalytics.com. So that's S-I-E-N-A-I analytics.com.

49:06That's the consulting website. I'm on Twitter as well. So at M-A-Y-O-W-A-O-S-H-I-N. So I'm pretty active on Twitter. I share the latest things I come across. I also recently launched a newsletter for leaders interested in building AI chatbots, which I think is our scene as well. Those are the three main ways. I'm also on LinkedIn as well, if you want to say hi. So yeah, I'll be happy to even provide just general strategy. If that's something that you're interested in for your business, I also offer consulting for that as well. So just AI strategy and how to set that up as well. It's a very exciting time.

49:42I know it can be overwhelming to know where to get started. So that's what I try and do. Simplify this stuff and take the noise away. Awesome. Thank you so much. This was really valuable. I appreciate your time and thank you for joining us. Awesome. Thank you. Great conversation with Mayo. And it definitely helps demystify some of the concepts and needs when it comes to training models and what you need to prepare or be ready to do if you want to go down that path. I definitely do not suggest that as step one for any business, you should start with low hanging fruits and gaining efficiencies by learning how to prompt better and using existing models that are out there across more or less every aspect of the business.

50:27but this is definitely a logical next step. And definitely if you're a bigger company and you have a lot of solid data across different aspects of the business, that is well-organized. And now to some exciting news from this week. As in previous weeks and specifically last week, there were a lot of really important big news. So I'll try to go through them quickly, but I will also add in the show notes, additional news that did not make the cut. So they're still big news, but they're not important enough for me to share through the recording of the podcast, but there will be links if you want to find out more.

50:58And we'll start with government regulations. Two senators, Bi-Parkisson, Blumenthal, and Holloway, are planning to introduce comprehensive AI regulatory framework. To do that, they've met with the leaders of Microsoft and NVIDIA and Elon Musk and Nadella and Altman and the biggest names in the AI world in order to come up with what the framework is going to include. The framework includes provisions for AI licensing and auditing and a new AI oversight office and definitions of company liability and privacy and civil rights protections and data transparency and safety standards and so on. The goal is obviously to reduce the risk that AI represents to the society while maintaining the innovation and the capabilities that we gain from it on both personal and business sides.

51:51It's definitely an important move forward and hopefully turning this into laws in the near future. There's obviously pros and cons in the fact that the leaders of the industry are the ones that are helping dictate the law because then it might become a self-fulfilling prophecy of what they want. I really hope that the lawmakers at the Hill take this a little deeper and maybe consult with additional people that are not from the industry in order to balance these people's views, but either way, I see this as a very positive move. Another regulatory action is the FTC is looking into potential antitrust problems in the AI space.

52:33They fear that incumbents may try to leverage their power against new generative AI companies, hence reducing competitiveness. This is obviously a serious risk because some of these big companies are gigantic and they control both software, hardware, engineers, and so on. So their ability to limit access to newcomers is significant. And the FTC I'm quoting, appears ready to intervene aggressively if they feel that anti-competitive actions being performed in or around generative AI space. This is already problematic if we are combining it with the previous piece of news, because just the fact that some of these large companies that has already violated multiple laws are now helping to write the new laws that will prevent from new companies doing the stuff that they did in order to train their models is already a problem.

53:29I see that as a positive move and I really hope that we'll be able to see a full ecosystem flourishing, including smaller startups, et cetera, that can compete with the bigger players, at least in niche specific markets or specific subjects. A few huge companies have released information about new AI capabilities in this past week. One of them is Salesforce. Salesforce Dreamforce conference happened this past week. And on September 12th, they introduced Einstein Copilot. Back in March, Salesforce introduced Einstein GPT, which was a smaller version of that, which allows users to use GPT capabilities to write and create content.

54:11but the idea behind Einstein Copilot is the ability to basically perform and query anything within Salesforce in natural language. As an example, a salesperson in a company can use this to research new accounts or newer customers, or a customer representative can look back and see what were results of similar cases of previous customers, or a product manager can create a storefront for a new product that he or she wants to launch. Really for almost any action within the Salesforce world, instead of the users having to ask questions from other people in the company on how to do that, or have to know all the set of clicks and menus that they have to go through in order to find the right buttons, send the right queries, et cetera, they'll be able to just ask for what they're looking for and generate whatever they need in a natural language while conversing with the Salesforce co-pilot.

55:10They also added what they call the Einstein trust layer with two goals. One is to reduce AI hallucinations and false responses, and also for data security. In addition, they've announced Einstein for Developers, which is a coding tool built exclusively for the Salesforce-specific coding languages, meaning developing new capabilities within Salesforce, will become significantly more efficient. My take on this is it's not surprising. Again, it continues a trend that we've seen from Salesforce and that we'll see from every other large software provider. Otherwise, they will lose market share to the competition that will offer these kinds of capabilities.

55:49And to prove my point, Ernest & Young, also known as EY, just announced that they have invested$1.4 billion in developing their own large language model and AI platform. They call it EY.AI EYQ, and it includes data management and use cases and framework analysis specifically for AI adoption. The goal of this is to position Ernest & Young as a global leader consulting company for large organizations who want to implement AI, which is basically any organization out there. This again, doesn't come as a surprise. If you look at their industry, other companies already announced this before. So peers like KPMG and Accenture and PwC and Deloitte all have made similar announcement in the past few months, all varying between one and$3 billion investment each.

56:40So they fall within the ballpark. And if we're already around the topic of large consulting companies, BCG, Boston Consulting Group just announced an alliance to deliver enterprise AI to clients with no other than Anthropic, the creator of Cloud2 large language model. So the goal is to give BCG's clients direct access to Anthropic's Cloud2 AI assistant, and BCG will be the advisory company that will help those clients implement these models in the most effective way. It's obviously a big win for Anthropik to be brought on board with BCG, one of the largest consulting companies in the world. But again, it falls in line with what we're seeing in the industry.

57:23I think we'll see more and more of these partnerships from the companies who generate the large, more successful models together with big companies who want to provide these capabilities and do not necessarily want to develop them from scratch, like we've seen in the previous example with Ernest & Young. The last two pieces of news relate to how effective is AI in different business components. The first of those pieces is research performed by professors at the Wharton School and the University of Pennsylvania. And they were trying to check who is better at generating innovative ideas, whether it's their MBA students or was it ChatGPT?

58:04and Chachupiti won by a landslide. And if you look at the statistics and read the article, they found the following. Chachupiti can generate ideas much faster and cheaper than students. On average, Chachupiti's ideas were a higher quality per survey. Chachupiti generated 800 ideas per hour compared to five by the humans. Now, this is obviously not a quantitative test by a qualitative test, But at the end of the day, when they picked the top 10 % ideas that were generated across human or Chachi PT, 87.5 % of the 10 % best ideas came from Chachi PT. And in addition, the cost per Chachi PT idea was 63 cents and the cost per human idea was $25 considering an average salary that they considered for these people.

59:01So ChatGPT won on speed, it won on quality, and it won on price. That's not very good news to anybody who considers themselves an innovation consultant or just somebody who is an innovator. The flip side of that, it will allow us as a society to innovate much faster, which overall is a great thing. And the last piece of news that has to do with productivity that AI brings is a research that was done by the Nielsen Norman Group. They did three different studies in multiple companies with the goal of evaluating the improvement in efficiency across different aspects of the business. Study number one was done around the topic of customer service.

59:45Study number two was done around writing routine business documents like sales proposals, et cetera. And the third study was around coding small software projects. The findings were not surprising knowing that it's going to be more efficient with AI versus without AI, but the numbers are pretty astonishing. On average, the productivity across these projects was 66 % higher than using AI versus not using AI. Also with a very big spread, customer service only gained 13.8 % improvement in answering inquiries per hour, which is what they were checking. Writing business documents, on the other hand, saw a 59 % increase in efficiency in creating documents per hour and writing code.

1:00:33So 126 % improvement when what they were measuring is how many tasks can a programmer complete per week. That more than doubled while using AI compared to not using AI. What does that tell us? Well, if you can improve the efficiency of things across your business by 50, 60 % or 100%, but let's take the average of 66%, one of two things can happen. Either you can find new clients that will allow you to use the extra capacity in order to grow the business, which would be amazing. The reality is that's not always the case. It's definitely not the case right now in a problematic economy. And it's definitely not continuously the case across all sectors, which means for the companies who do not have the flexibility to grow fast enough to make more use of the extra bandwidth that is generated, they will cut costs by letting people go.

1:01:36And anybody who says otherwise just does not understand how effective this technology is, meaning we will see significant job losses in the immediate future and definitely moving forward. By the way, I know I said that I've already covered the last piece of news, but it relates directly to this. Forrester's 2023 Generative AI job impact analysis anticipates that 2.4 million US jobs will be replaced by Generative AI by the year of 2030. I think this is totally underestimating it. I think the impact is going to be much bigger, and I think it's going to happen much sooner. But who am I to question Forrester's analysis?

1:02:15but they also anticipate that 11 million jobs will be influenced and will require retraining to work alongside AI. I 100 % agree with that. And again, I think that will happen way faster and at a much higher scale. The good news is if you're listening to this podcast, it means you're one of the people that actually cares, that understand this, that the world we know today in business is changing and changing fast. You understand you need to acquire new skills in order to learn how to work with AI in order to become more efficient, which means you have a much higher likelihood of staying ahead of the curve, keeping your job, or maybe even starting your own company or a better job within your company, because you will have those skills.

1:02:59That's it for this week. Have an amazing week. Explore AI, share what you find with the world, share it with me on LinkedIn, and have an amazing week. you