90 | How To Use AI to Chat With Your Data with Joanna Stoffregen

0:00Hello, everyone, and welcome to a live episode of Leveraging AI, the podcast that shares practical ethical ways to leverage AI, to improve efficiency, grow your business, and advance your career. This is Isar Maitis, your host. And maybe the question that I get asked the most from people who are a little more advanced and started playing with AI is, how can I chat with my data? Like I have all these data sources and I have data in my CRM and I have data in my emails and I have data in my ERP and I have data in my Google Analytics and et cetera, et cetera, et cetera. And I want to be able to chat with my data and ask holistic questions.

0:36And I don't know how to do that. And most people don't know how to do that. But the reality is beyond the fact that I get asked that a lot, it's a huge need and not being able to access your data in an effective way actually costs businesses a huge amount of money in overhead, mistakes, missing clients, losing clients and so on and so forth, just because they cannot get the right information accurately and in a timely manner, a problem that having a capability to chat with your data across everything solves. So how do you do that? So I'm really excited to tell you that our guest today, Joanna, is an expert on exactly that.

1:21Her agency helps businesses develop these kinds of solutions. But what we're going to talk about today, beyond the custom developed solution, there are other ways that you can develop on your own with tools that exist in the market. You can start very small, test things out, and then slowly upgrade to more and more advanced capabilities. This is obviously maybe the holy grail for businesses when it comes to using AI and machine learning is the ability to really know what's going on across everything that you do in your business, from finance to customer service to marketing to projections to emails to etc, etc.

2:02And hence, this is obviously a very important topic. So I'm really excited and honored to welcome Joanna to the show. Joanna, welcome to Leveraging AI. Oh, hello. Thanks for having me. Excited for today. Yeah, listen, you and I have been talking about this for a while, and the content that you're sharing on LinkedIn on this topic is actually pure gold. And I think most people hear the technical aspects of this. Oh, it's called drag. I'm like, okay, I don't know what that is. Okay. And once people start diving into the technical stuff, they get really scared. and so they stop even though it could actually be very simple.

2:44So I know you prepare like a full kind of like step-by-step process to take us to first of all why and then how and then other considerations like safety and security and costs. So I'm sure everybody who's going to join us is going to learn a lot. So let's get started. I will give you the stage. Thank you. So yeah, let's get started. I will now demystify RAC for businesses and hope you guys will get value out of it. Let me share my screen when you see it. Yes. And by the way, those of you who are listening to this as a podcast afterwards and you cannot see the screen, a few comments about that.

3:24First of all, join us on the next live. We do this every Thursday, almost every Thursday at noon Eastern time. So you can join us and then you can see the screen and even ask questions and participate. So that's number one. Number two is we're going to describe everything that's on the screen so you can understand if you're driving and so on. And number three, we have a YouTube channel under the Multiply brand. It's just instead of a Y, it's spelled with an A-I in the end. So if you want to watch this afterwards, once you're done walking your dog or doing the dishes or yoga, whatever it is that you're doing, listening to this podcast, you can later on go and watch it again on YouTube.

3:59But let's dive in. It's Davey and I will also try to make it more understandable in case someone listens to it in podcast afterwards. Okay, so we start basically with the billion dollar problem that RAC solves. And I want to introduce you basically to Julia. So Julia is a product marketing manager that works for SoftTech.com. It's a tech company based in Silicon Valley, has 300 employees. She's working remotely from Berlin. And in her day-to-day, she actually talks with a lot of teams to do her job, like design team, tech team, sales, marketing. And of course, uses also a lot of apps for communication, task management, like Gmail, Slack for team communication, Jira for product tickets.

4:54She has tasks in Asana, Figma. So basically all these familiar apps that we all know if we're working in tech or not. So today, basically, Julia, she has she was working on a promotional campaign to launch their new product feature. And she arguably needs basically a market research report that, you know, her colleagues prepared a month ago or so. So she remembers that actually someone sent it on Gmail. So she looks there, but she cannot find it. Then she thinks, okay, maybe in marketing Slack channel. So she searches there, doesn't find it either. Then she looks again and maybe in Google Drive, she said maybe it's in the marketing folder.

5:43Does not have success either there. Common scenario, it happens to me all the time, even if I'm not even working in such a big corporate. Right. So then she remembers that actually her colleague Max might actually know for sure because he's also involved in this project. So she decides to Slack him. So she sends him a message. She waits now 20 minutes. There's no answer. Then actually she remembers that Sam is a current in San Francisco. It's 3 a.m. there. And actually she will take a lot of time and she will get her answer. So frustrated, she basically decided to move to another task, but actually she lost a lot of valuable time.

6:24She's now behind on her project and she might even lose the deadline for the submission of her project. So very common scenario that I guess a lot of us working in tech and in big companies or like a smaller company's experience, right? And we have also the data to support that. not our research, but from the Data Institute Corporation, they found out that 2.5 hours per day, on average, is spent basically on searching information. So imagine like one out of five days of work, we basically spend or the average worker to search through different apps to find their information that they're looking.

7:07On average, we use like around 11 apps and it takes around eight searches before basically we find the right information. And that obviously has a lot of costs to it because time is money, right? They estimate that a company that employs 1 ,000 employees loses around 2.5 million per year on actually this problem that we are not able to find, to search and retrieve the right information. That has obviously impact in not being able to find the right information for your project, like Julia, or you might miss an important update because you did not get the information on time. and fail to use basically important updates of your project.

7:58So all of that basically also making wrong decisions because you don't find the data. It costs a lot of money for companies. And this is basically the billion dollar problem that Rack solves. And also companies like Glean help to solve. Glean is a$2.2 billion company. so hence they're working on a very good problem to solve and just basically to explain you a bit on high level what Glean does and not just Glean there's few companies that do the same they help employees like Julia essentially search through their company data and retrieve this relevant information so for instance we can see here there's this employee that he's looking for the status of a project and normally what he would do he will search slack he will search google docs to find this status he might search the hub repository to see the commits of the code but now with the solution of that also cleaning of like technologies rack solve is that he just asks a query and then the system provides him basically all the information in natural language in one place to see.

9:16So imagine you don't wait. It's directly, you can ask anything you want the system because it knows basically what your company knows. So this is also their promise. Know what your company knows. Yeah. I want to touch on one short thing from everything you said. Every single company is sitting on a gold mine of data that is just not accessible today. And even small company people, I know a lot of people who are listening don't run billion dollar companies. They run a$20 million company, a$50 million company, a$100 million company. And they're like, I don't have any unique data or any proprietary data.

9:56And that's not true because every email of every employee, every phone call that you have recorded for customer relationships, every proposal you've ever written, every data in your CRM or your ERP, all of that can be helpful in making future decisions and understanding what's currently happening in the business and so on. And it's right now requires a lot of time and effort to do the analysis across these platforms, just because it's built into silos. So yes, we do this every now and then. There's like a project manager who will spend once a month and once a quarter writing a review by collecting all that data.

10:35It's time consuming, it's not 100 % accurate, and it's not accessible in real time when you need to make decisions. Like it's once a quarter when the project manager did that review. And it, again, may or may not dive to information that other people need. And having a solution like this literally enables, think about having a Slack channel with your entire data, where you can literally ask any questions about anything and you don't care where it's stored and what is the source. And this is exactly what this thing does. And then every employee in the company, if you're the customer service person, And you can find out the answer to answer the call you're on right now.

11:11If you are the VP of marketing, you can understand stuff about your recent strategy that you deployed. And if you're the CEO, you can understand trends across everything that's happening in the company. What are things that are happening? What's not moving fast? Like literally any questions you want to ask, if you connect the right data sources, you can ask and get answers, accurate answers in seconds. So it's a game changer compared to everything that we know. Exactly. And also another very common use case for salespeople before actually they're, before they're having their call with the lead, they need to access a lot of information in their CRM, in Slack, in their product to really understand what is their company and the lead about.

11:56and it spent a lot of time and now basically systems like rack systems you could just write hey give me all this information about elite x and it just pops pops it up for you so super helpful yeah and also you don't need to be a hundred million company or whatever even i know companies with 10 20 people that have exactly the same problems super helpful for any company size so exactly now so what what is RAC so yeah so RAC stands for retrieval augmented generation and it's essentially a way of for large language models like chat GPT to be aware of data that they have been not trained on so for instance chat GPT is trained on all of this public data but it's not trained on your slack conversations on your basically company data on your CRM data on your Jira tickets, or basically this data that are proprietary for you or for your company.

12:57So it's essentially a way how we can basically make a model be aware of this data. And of course, we have this problem of nations and that they're not, models are not up to date in their training data. So it's also solves that as well. And to better understand that, I want actually to showcase an example and use a case study that we were working on. So basically, we developed for car mechanics of Mazda cars, basically mechanics that repair Mazda cars, a way how they can basically get repair instructions from the system. So their problem was that it will take very long for the mechanics to search through manual Mazda instructions to find out how to repair the particular problem of a car.

13:50And that had huge business impact on basically unhappy customers because it would take very long time until they would receive their repaired car. And often they would also, the mechanics ordered the wrong repair parts because they found the wrong information that they needed to repair. Yeah, so you can see here. So let's say there's this mechanic and ask how to fix a broken airbag of Mazda G4 car. You can see here, if you ask this question to chat GPT, it will give you basically just generic data, generic information. And then it will also prompt you basically, hey, ask a specialist. The system that we developed is a rack system.

14:35So it has all the Mazda manuals. So it's able to tell you exactly how to repair, let's say, a broken airbag for the particular Mazda car. So now, obviously, the mechanics, instead of searching manually and doing all these mistakes, they have a system that directly basically tells them how to repair easily. So this is basically what RAC is and how basically it is able to become aware of your data and an example to showcase the difference between a normal chat with GPT that might have general knowledge about car repairmen, but not specified knowledge about the particular problem that we are looking forward to solve here.

15:21Yeah, so I'll touch on two things that you said just as a quick clarification to everyone. One, I like to say about all these large language models that they follow the common say that they're the jack of all trades but the master of none. as you start deep diving into specific topics and really getting needing like expert advice they usually don't know the data if you're trying to get deeper and deeper on a specific professional topic they just don't have it and even if they have it they may not have it for your business your company or thing so this is one thing that it's coming to solve is you give it the data that you needed to know and then ask it to retrieve just from that data and as joanna said, the other thing that it solves is hallucinations.

16:05The level of accuracy, and it would still hallucinate probably to an extent with RAG, but it's going to be a significantly lower percentage and a much higher level of accuracy when it comes to providing you the right information if you are using RAG versus just using a open, like a market available large language models. so how is the magic done? Take us through the process How is the magic? Yes, so on a very high level non-technical level there is the user that asks a question in our case the mechanic asks how to fix a broken airbag of Mazda car so now the system has basically all this Mazda manuals in a vector database stored, right?

16:51And then the system will actually semantically search for in the database for the relevant chunks or the relevant basically documents for this question asked. So you can see here, maybe search three, four documents. There have like the information needed in specific. And then this is the retrieval part. Then the augmented part is where basically the user query is augmented with the retrieved information from the first step. And then the last step, which is the generation step, is where basically the system generates the answer, right? So remember this example from before where you ask a question and the answer might be in very different sources.

17:38So this is the different sources. And then as what just a user is actually the generated answer from these different sources as a answer to your question. So this is a high level. Yeah. No, it's a great explanation. I think what we're missing here for the common person is, I assume people are not technical, the combination of the words vector database give them diarrhea. And so how do you get the data from your CRM, from your emails, from your Slack channel, et cetera, into a vector database that then the system can retrieve from? Yeah, sure. So basically, first, like a vector database is storing this, how I would say, it's basically storing the semantic information that is relevant basically to the query.

18:36So just take it from the beginning, how you would go about it, you will have all this data, you would need basically to use to load them into the vector database and you would use load data loaders like llama index or lang chain and you will always need to basically pre-process that so you would need to have basically unified format usually in markdown then basically you would need to split that data into chunks, that these chunks, it will be easy basically to create embeddings. And then these embeddings, we basically store them into this vector database so the system can retrieve from. And the vector database, basically these embeddings are basically the semantic meaning of basically the...

19:29So what is the semantic meaning? let's say a king and a queen it will be on the same kind of the same level we'll have the same semantic meaning as opposed to sound that it will be basically not relevant to king a queen so this is basically the semantic yeah sorry that the vector of databases i want to make this a little more specific is the common person can he use tools so people who don't know lang chain and don't know how to do this from a technical perspective, are there easy ways, like something that a common business person can do or specific tools that they can use in order to get to that outcome without maybe as a first step, right?

20:15Let's do a test case, see that it's actually working, see that it's giving us some things, and then come and hire a company like yours to actually do the more complex, sophisticated solution. Yeah, sure. Exactly. There might be levels to that. So first, as I mentioned, we would need to collect and prepare the data that we have available. It could be internal data. It could be your PDFs, invoices, all of that. But it could be also external data. It might be basically websites, blogs, or basically YouTube videos, transcripts of it, and so on. And you could basically get started very easily with, first of all, if your use case it might be you want to have your company knowledge database.

21:01So basically use Rack for your company knowledge. Use out-of-the-box tools. Don't try to basically start this from scratch, develop this from scratch. Use tools like basically Glean or similar, but depends on your use case. For instance, if you have a couple of documents with your company policies, let's say, and you want essentially to train a chatbot, to have it in the chatbot for the users on your website to ask questions about your policies, about how is your refund or about products, you might just prove, validate the use case with a simple GPT. So we start basically very simply with simple GPT from OpenAI.

21:48so take the documents and just upload them into a gpt and validate the use case there is it basically does it answer correctly my questions is it actually useful and then you can also let's say you want to get buy-in from stakeholders that's the perfect way actually to showcase them hey look it works it answers questions that our customers answer us and we have now a person answering them. We put the first level tool. So before you go to level tool, those of you who don't know what GPTs are, so OpenAI under their platform within ChatGPT, by the way, until a few days ago, it was only in the paid version as of this Monday, it's available to everyone, which is a magical tool that they now made available on their free platform.

22:38It allows you to develop these mini custom automations that again, sounds very technical, but you do them literally by giving it instruction in regular words. I want you to allow the user to ask you questions about the information in the attached documents. Questions they may ask might be one, two, three, four. I would like you to start the conversation by telling the user, what do they want to know about HR questions? And that's what's going to start. It's going to say, hey, what do you want to know about HR questions? And then the person will type. And if you've uploaded all the HR documentation into that GPT, which again is a simple drag and drop.

23:19There is no, all the stuff that Joanna talked about before happens in the backend magically. Like you don't have to set up the data, structure the data, convert the data. Like all of that, literally all you have to do is drag and drop the HR documents, your employee handbook, your guidelines, these kinds of things, drag and drop them into the GPT and that's it. and people will be able to chat with that data on a very high level of accuracy. Again, right now, it costs exactly zero dollars to do. And to figure out how to create the GPT, after a little bit of training will take you minutes, including the training will take you an hour, and you have something that you can chat with the data.

23:58The biggest problem maybe of GPTs is the amount of data you can upload to them. So the HR example is a great example because they don't have huge huge amounts of data, but you're limited with the amount of data you can upload into these GPTs. But as I mentioned, it's one of the most amazing capabilities we had in businesses ever. Definitely now that it's completely free, not that I think that$20 a month should make a difference, but now you don't even have that excuse. So it's definitely worth checking out and testing it. And as Joanna said, it's a great way to do a test case and see what results you're getting before you go to level two.

24:33So now let's talk about what is level two. Yeah. And before we go to level two, if you have still issues with data privacy, you don't want to share your HR data or whatever, you can actually generate synthetic data based on to model your real data and create a GPT to still basically validate the use case. So that's also creating synthetic data might be viable as well. Use, sorry, level number two. So in this basic level, you could use a drag and drop tool like Stack AI, for instance, that it's no code, low code, that you can create this RAC system or RAC chatbot by drag and drop. They do also pretty much everything for you as just more advanced version than a GPT, more capabilities there.

25:21And of course, if data privacy is a concern, you could actually cost locally, like a tool like Verba. Maybe we can show it. I don't know if I... Do I share now something? Yeah. I see a dog on your screen. Oh, here we go. Ah, cool. They changed also the user interface. Yeah, great. So basically, yeah, this is the no code. Sorry, not no code. basically a way how if you have proprietary data let's say if you're a lawyer let's say and you want to chat with your cases but the cases are proprietary data private data you can use a tool like verba it's yeah so you upload again your documents here and you chat with your documents yeah now they just change the user interface crazy okay anyway um so yeah that's level two And of course, level three is where you build that from scratch.

26:22The solution that I basically showed you with the car mechanics, that it's basically built from scratch. And there you use basically frameworks like LankChain or Lama Index. Yeah, you would need an embedding model to do the embeddings, yeah, to store the embeddings in a basically vector database. can use different ones, Quadrant, Pinecone, Beaviate, one of the most famous ones. And we use also Streamlit for the prototyping. So you can actually create basically these prototypes as I showed you on the use case, the case study basically before. I want to summarize very quickly the three levels. So level one is use an existing model.

27:05You can use a GPT in OpenAI. You can even just use Claude and upload documents, but that will be like a one-time thing, Like every chat will have to upload the documents again. By the way, if you want to chat with a lot of data as a one-time thing, Gemini 1.5 Pro from Google, as of this week for developers, so if you sign up, has a 2 million token context window, and I'll experiment with that in a minute. The platform that's available, if you don't sign up for the new program, is still 1 million tokens. So 1 million tokens is 700 ,000 words, which is a huge amount of data. It's way bigger than any of the other tools that exist today.

27:42So the second best one you're going to get is Claude, and that has 200 ,000 tokens. And the cool thing about Gemini 1.5 Pro is that it's free. So if you go to their AI studio by Google, you can upload documents and chat with those documents. So if it's a one-time thing, if you're working on a proposal and you want to look at what's in the RFP or what's in the proposal so far, or if you're working on analyzing existing data and it's not an ongoing thing that you'll need to do again and again, using a model is actually not a bad idea. Like I said, Gemini 1.5 Pro has two benefits. One, you can load a huge amount of data and two, they're getting a very high level of accuracy in getting correct responses, which is north of 97%.

28:27And so this is obviously very helpful. So that's level number one. Level number two is using tools that were built to do exactly that. There are no-code tools that allow you to upload and connect various levels of information and then chat with that information. And then there's, again, some of them work in the cloud, meaning you actually give them the information and it gets loaded over there. And some of them, like Verba, works locally, which means you don't have to give your privacy and you don't have to think about who am I actually giving access to my data in order to be able to chat with it.

29:01And 11.3 is a custom-built solution that is built specifically for your needs based on your exact data, based on your use cases and so on. The differences is obviously the amount of data you can upload and how much is it going to be customized to your needs. But the flip side is how much time and money it's going to cost you to develop it. So that's the thing. And it's not a bad idea just to go through all three steps. Start with step one, make a quick test, see if it works. Go to step two. that works well but you can solve everything with these tools then go and have somebody like joanna build a custom solution for you to solve that problem yeah no i totally agree like a hundred percent if you have a tool or out of the box solution uh to do your job don't even bother to build stuff from scratch yeah there's a question there's a question is verba the open source version of data stacks i don't know if you know that answer.

30:01Question from Verba. Sorry. Is Verba an open source version or like an open source option for data stacks? I don't know. What do you mean from data stacks? Yeah, it's probably a different tool. Yeah, it's from Vaviate actually. It's from the vector database Vaviate. So they basically created this Verba tool. It's open source. You can basically try it but i'm not sorry for not being able to answer yeah it's all good okay so let's continue yeah so basically you're still able to see my screen right okay okay what to be aware of when you basically the rack solution go into production besides basically the technical issues and not being able to retrieve stuff which basically but there's like a very technical solutions to those.