Private, open source chat UIs

0:05Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at Fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents. So you can launch your app near your users. Learn more at Fly.io.

0:42Welcome to our next Practical AI webinar. This is our second webinar or live event that we've done. Chris, we've always done the podcast pre-recorded, and it's been fun in last time to talk about text-to-SQL live and now to have another chance to have a live webinar. Are you enjoying these? I am enjoying them. I think we need to do these more often. They're a lot of fun. Yeah, yeah. And today, so I'll kind of frame the conversation for today. But first, let me also welcome Danny from LiberChat. He's joined us to talk about the topic today. Thanks for joining, Danny. Yeah, of course. It's an honor to be here.

1:26Yeah. Well, thank you for everything you're doing on the LiberChat project and in the community. today as those that are in the webinar know and maybe if you're listening later we're going to be talking about crafting the next generation of ai chat interfaces and just to kind of frame the setup for this we've talked a lot of times and i still frequently even this week last week i hear about people saying oh my company doesn't let me use the chat gpt interface or even literally today, like three hours ago, I was in a meeting with a number of customers and they were asking the question was, how do I get a chat interface that allows me to switch between models and like try different things with different models?

2:17Like if I want to try Llama 3 or something like that, are there ways for me to do that? Are you still encountering that as well? literally every day. And I'm very sensitive to it. You may recall that I was getting after teachers online and a teacher pointed out that, hey, this is not our choice. It's the school system. So I've been super sensitive to this ever since then. And we have challenges. I'm hoping today can put a big dent in that one. Yeah, great. Well, I'm super happy that we have Danny with us because this is what Danny has devoted a huge amount of energy to with LibraChat, both in terms of providing an open source chat interface, also providing a chat interface that allows you to plug in different AI systems, whether that be OpenAI or many others, closed source and open source types of models and systems, and providing even functionality related to, I think even rag and and plugins and and other cool stuff so I'm gonna I think now pass it over to Danny and let him kind of share a little bit about you know the background of LiberChat what it is and how he views kind of the need for private or open chat interfaces and what they're trying to accomplish with LiberChat and how they see that fit into the industry and what's sort of needed.

3:45So over to you, Danny, looking forward to this. Yeah, absolutely. You know, part of the original idea was kind of inspired by a chat GBT leak. I don't know if you remember this, but there was someone whose messages were being seen by a different user. Yes. You know, he's from Poland or Russia and someone woke up one day and all their messages were in Polish. That surprised me, but it really planted the seed for what I wanted to see. and I thought that was just a basic thing to overlook and just started crafting from that impetus. And yeah, I think it's inherently completely private with the flexibility of having like remote stuff in there too, which I think is important.

4:29But yeah, I honestly started off as a learning experience. It was ChatGPT had just come out and rocked everyone's world. So I was like, wow, I really want to learn how this interface works. Like, what are they doing here? And I had just started learning about UIs in general and web development. So it really interested me. And something I saw was, you know, this was such a huge tool in my learning. You know, as I learned, this thing was being built out. But also, you know, there was that need right away because, you know, I posted it on GitHub. And the next day I had six stars and I was blown away.

5:08I was like, what, six stars? That's great. Right. Who's looking at this? Yeah, it was already, you know, getting picked up by search algorithms or GitHub's algorithms. And I was just like totally blown away by that and totally motivated. You know, a lot of people started commenting right away what they want to see. And I think having access to these tools sooner rather than later is going to be a huge thing, no matter what your team size is, what your business is. And obviously, you have to tread carefully with the privacy side. But I think I've built something battle tested at this point that thankfully it's as much as a contribution from, if not more so a contribution from the people using it would really help me along the way.

5:56So you mentioned kind of like the initial problem that you saw that kind of motivated you to go down this rabbit hole, which was kind of seeing others messages, which is definitely a piece of it in terms of how an application like this manages state and data and that sort of thing. What did you find like going down that rabbit hole? Is there a way that you can like categorize the main categories of things that people have come to find useful about having their own chat interface rather than kind of a one provided by a model data and privacy? but what are the kind of those main features or things that are on people's mind when they're thinking about having their own interface?

6:48I think for me too it's just owning your own data and data is like the new commodity it's so valuable even to these big AI companies they're constantly releasing their own interfaces that are cutting edge I might add but they're also looking to collect data. And I think a trend I want to see in tech, and especially from the open source world, is just owning your own data that stays between you and these large language models and your company. And you really have that luxury through this app. And so that's a big driver for me. And I think that's a big component for a lot of people. And yeah, and it's also as you're learning from these things, it's like, I think it's so valuable to kind of like categorize and piece through like, you know, certain conversations you've had in the past.

7:41And that's why like one of the main features that's been like a mainstay since the very beginning was being able to search your messages. And to this day, it's not a feature on ChatGPT or many different interfaces. So I think that's very interesting to see play out. But I also, So I know a lot of people, just that one simple feature gets them on board. I'm curious, as we talk about the applicability of this, for instance, large corporations, and I work for one of those large corporations which built its own interface some time back, and others will have well, but going forward, that takes a lot of maintenance, a lot of concern.

8:21If you were in front of a chief digital and AI officer, a CDAO for a large corporation, and it may have already created its own, what would be your pitch for saying, you know, come over to LibroChat because of X? How would you convince the Fortune 500 companies that are out there that this is the way to go rather than investing on their own compared to their whatever investment they've already made that sunk costs? Number one, it's completely open source. It's got a lot of contributions. There's nothing being hidden in terms of its interoperabilities. And also, number two, it's highly configurable with any kind of intranet network you might want to do.

9:04Or, you know, it can be completely like sealed and even work with large language models without needing to hit some kind of remote service. And it could be completely on local connections. It really just depends on the admins level of expertise and connecting all these things. But even just using the default Docker Compose, you can just spin up something that's only available to you and is configured in a way if you're using like insecure default variables and things like that, it'll warn you right away. So, yeah, I think those are the top three things I'll say to try to convince someone. Well, I think people are eager to see a chat interface.

9:45We've been talking about them. so we'll kind of let you go from here and show what you want to show and I'm sure we'll have some questions and thoughts as you're showing these things and yeah if you could talk through kind of the demo and and what's on your mind as you're thinking about the different features that you're showing all look good all looks good great so yeah I'm running this locally and I'm using olama i have it hosted on my computer so i'll just write hi there and it should usually takes a second and load up and olama for those that aren't familiar do you want to describe that just a second i guess it's hard to find a specific term for what it does but basically it helps you manage local large language models like it helps you pull down their latest build files and then it helps with the prompt wrapping process and serves them on an API.

10:42So it just makes them really accessible wherever you can run them. Cool. So it finally got to me. And now that it replied, you know, the next couple of replies should be a little quicker. But basically, you know, this interface is, it should look pretty familiar to a lot of people. unabashedly taking a lot of inspiration from ChatGPT. There's just a couple core things. Like I mentioned, the search messages. And if I search here, it's already picking this up. It's a previous conversation I had. I was testing some file there. And aside from that, going back to this, this is kind of segmented, but for people who have a need to set more custom parameters or even just setting instructions here, making sure that it generates what they want to see.

11:36So I'll say, make sure to write code in Markdown. And I'll say, write me a recursive Python function. Yeah, so it's doing the job. Whether or not it needed my instruction, it depends, but it had it. And so it kind of steered it right away to use Markdown, which gets rendered like this. It looks beautiful. And it even has like the copy code and the nice sort of like edit button, copy, all that stuff that one would expect. And really, it's pretty simple too. And I like that simplicity. I think I've seen a lot of interfaces kind of get lost with the technical side of it. And I'm sure that has an audience.

12:29And those are great interfaces for certain technical things. But something about this is just immediately accessible, I think. Of course, we mentioned that we could switch AI providers. And I've got, I heard someone recently call this, you know, gotta catch them all Pokemon of AI. But these are just a little showcase of like all the different ones we can use. So I personally like Grok just because of its speed. And it's blazing fast. This is running Llama 370B. And this was like also a switch. So in the interface that you're showing, you're having a conversation. It's also was a switch between Ollama and Grok in the same thread.

13:17Am I understanding that right? Yeah, correct. That's awesome. Yeah, so you can kind of, and does that message thread sort of history carry through to the different models, I guess? Yeah, so that's where the database comes in, just keeping track of the conversation, not just the back and forth, but also any changes you might make. So if I make changes here on the fly, that'll get recorded with the conversation state. Yeah, I love how you can stay in context of the problem that you're trying to solve and yet still optimize against different models in terms of what they're better and worse on the fly without it kind of taking over and becoming the primary concern.

14:02So very nice. Thanks. Yeah, and that brings up something really good user feedback I got maybe along the lines. there could be like a smart router that kind of knows or even you could pre-configure beforehand like which is the best AI for this sort of task and just kind of switches it for you. So maybe that's something down the line I'm still kind of drafting in my head. But of course like as these things evolve there's like we mentioned RAG. So a lot of people have the expectation for for files to work with these things. And of course this LibreChat supports that. And here I just dropped a CSV.

14:47I'll say, tell me about these sales. Yeah, and this was just mock data I made up. So it's about Gadget B, Gadget B, different sales data. So I was able to look at that and just kind of give some context about it. And of course I could switch the model just as before. So I switched to Cohere. and didn't give as good of a response as GPT-4, but it was able to see that and kind of work with it. And even that, like the file processing, that's all based off a local RAG solution. So it's using like a local vector database and a local server that's just dedicated to the files. And yeah, that's one of the things there.

15:31One of the things I'm particularly excited about is agents and agent workflows. Of course, we have OpenAI recently made a solution of their own with assistance. And I think people are still discovering the capabilities of this. But I think it's exciting for me, like not just working with like what AI companies have as cutting edge, but also as inspiration for the open source side, because I think they model this really well. And it's giving me ideas of, you know, how can we do this for with Olama? How can we structure something like this with the latest meta AI models? I have a test prompt here, and this is how I generated my sales from before.

16:16But I'll add something here.

16:20Finally, output the file you created. so this can pretty much not just write code but execute it within open ai sandbox so that's really great for things like data analysis and just generating mock data like this too yeah so it might take some time while that's running uh there's a question um did you test your local rag system against others like open ai so maybe some interest in kind of the performance of that rag system with a variety of models and locally versus kind of those built into closed systems? I did. I definitely, with OpenAI especially, because they have a rag system through assistance.

17:09And to be honest, right now it's not doing nothing too special. It's what they call naive rag. But I found even with naive rag, you have a really good prompt. And so I tested that prompt, several different iterations of it. You can really get something effective almost across the board like any LLM. And with OpenAI solution, it's really a black box. You can't even see the prompt that's being generated. I'm not sure how to like steer it better. Yeah, transparency is an issue in terms of, well, I guess it's nice when things go right and it's sort of automagical. But then when things aren't going right, you really wish you could understand a bit more.

17:52So where are you in terms of your multimodal chat story? How far along are you in terms of what you're trying to get to? One of my main goals right now is to offer even more access controls and configuration over the interface experience like admins want to create. So, for example, I understand that, you know, especially the first time someone logs in here, they might not know, oh, what are all these models? I mean, I recognize Google, I guess, or they would need a little more clues or they might not even think to click here for the model. But I really want to see an update. I'm actively working on this where there's just one dropdown and you kind of get a bit of more info on that, what you're selecting, what it's good at.

18:47And it's like, OK, this guy can search the Internet and I need that for this task. And also just being able to control like which users can access what, too, because that's a pretty big need, especially in the enterprise setting. So in terms of multimodality, like in the sense of of AI being able to work with different formats, I think down the pipeline, we'll see integrations with videos. But right now we're handling vision with images and that's pretty useful and a huge help for me. So we started exploring LiberChat at PredictionGuard because a bunch of our customers who are using PredictionGuard wanted a private chat interface because PredictionGuard itself is a platform that allows you to run large language models in a private secure environment with safeguards around them for like factuality and toxicity and prompt injections and a bunch of other things.

19:48And so our customers are all those kind of privacy focused, security conscious customers who are maybe running Prediction Guard either on their own infrastructure and want a private chat interface for the models that they're hosting with Prediction Guard, or they want an interface that's not a closed one for usage of our models. And so here, what you can see is we've taken LibraChat, which again, Danny mentioned is open source, and we've been able to take it into our kind of branding. And we have prediction guard here where you can, you know, set your API key and use prediction guard running on top of our platform.

20:32And because it's open source, because it's transparent, we are able to take this and also, you know, integrate our own sort of flair into this. So I know engineer from our team, Ed and Danny work together. So thanks for that, where we were able to integrate some of these checks for like toxicity. and integrate our various models into the mix. So still kind of like Danny was showing in terms of running here, I'm running with neural chat seven B. This is running in a privacy conserving setup in Intel's AI cloud on Gaudi two infrastructure. So it's a very unique setup that we've kind of optimized and we're able to connect to our own model, use this really slick interface, which is liber chat.

21:19It's just sort of branded a bit with our colors and logos and that sort of thing. But also we can integrate the unique features of our take on an AI system, right? So let's say I'm really concerned because I'm using an open model that doesn't have some of the guardrails around it like closed source models. I can go into the config here and turn on a toxicity filter to make sure that the model isn't cursing me out or giving me any sort of like stuff that I don't want to see. Right. And so here you can see we have a little toxicity score. Thankfully, it wasn't very toxic in this time around. So continuing similar to what Danny was showing, but again, our kind of own own take on that with our models and kind of the safeguards around that.

22:13One cool thing that we found really useful is that a lot of our customers, they want an interface like this, but they also want it authenticated. So they have their system set up. So we've integrated we're a G Suite company. So we've integrated Google login here and it's only our org that can log in. So the prediction guard org and now I'm authenticated. Here's my chat, like Danny mentioned, that is private and searchable. So, yeah, this has been a really amazing thing for us where we've been able to take and build on the great open source stuff that Danny has built at LiberChat and create something that works really well for our customers and for our setup.

23:01So before I leave and stop screen sharing, I saw that there was a question earlier on about translation with language models. A lot of what we've been showing is English, right? Some language models like OpenAI, they say that they'll do other languages, but sometimes that doesn't always work out. So we have a translate endpoint in our API. And so we've done a bit of this testing with large language models, translation, and kind of standard translation systems like Google Translate and Bing Translate and others, or even other models like NLLB, No Language is Left Behind from Meta. And in our translate endpoint, you can send a translation and then actually get the result along with a score.

23:50So we're using comet scoring, which is a way to score translations. And I think the question was, how well do large language models translate and are able to chat in different languages versus machine translating with a commercial translation system? So what we've seen in scoring both commercial translation systems and large language models is that some large language models, depending on the language, like if you're going into Hindi with open AI, you might get a good translation or one that is comparable to Google Translate a small amount of the time, like five to 10 percent. but mostly the commercial translation systems are generally better.

24:34And definitely as you go down the longer tail of languages, it gets sort of worse and worse, even in chat in like Mandarin. A lot of models don't do so good, even though that's kind of the next highest represented language in data sets out there. So yeah, it's definitely a mixed bag there. I don't know if Danny or Chris, if you have a comment on that, but before we go to other questions. I'm good. So some other questions on the LiberChat side. Are you building tools for LLM evaluation since you have all the comparison models out there? I think they're kind of imagining, oh, I can switch between models easily in this interface.

25:15How does that help me in an interactive way? It could help me evaluate the performance of different models. But there's probably a non-interactive version of that, I guess. Maybe toward your roadmap, you were describing later the idea of automatically switching to optimize. So this would be an incremental step in that direction. Yeah, absolutely. Back to the data ownership, I just think it's absolutely crucial to have built some kind of pipeline for evaluation as well, especially if you're really into fine tuning your own models. And really, it's crazy to think about, but even the data that, you know, we have just casually with these large language models.

25:58And if they're a very capable model, it's almost like a goldmine for the next model. And just having that, you know, at your own ownership, not just some cloud service. I definitely want to start it off simple, you know, being able to, you know, thumbs up, thumbs down. but then being able to integrate complex evaluation tools like you guys already have with the toxicity score or the translation rating, I think that's awesome. Cool. Could you talk a little bit, this isn't one of the questions, but I saw in your documentation discussion of plugins. Could you talk about that a little bit, what that means in the context of LiberChat?

26:34So they are inspired by just ChatGPT's use of plugins and really what the AI services now refer to as tools or functions. And really, it's just a way to be able to interact with some algorithm or API that's kind of programmed there already. And you're just kind of letting the model decide the inputs and outputs or the inputs and rather interpret the outputs. And I'm using it, obviously, in the plugin system where you can make requests to Dolly or Staple Diffusion for image generations. You can search archive papers, things like that. But really, what I came up with, the plugin system specifically, is almost a year old now, which is crazy.

27:25But I actually developed it before OpenAI had these functions in their API. So in the process, I learned kind of deeply how these LLMs were, you know, understanding certain tokens a little better for like formatting. And now we have like such a rich environment now for like getting, you know, only JSON responses or being able to use tools with Anthropic. So I've got a lot of things planned there where I want to see just that tool environment really grow. And also like for people who are building on top of LibreChat, I want to see better documentation and better developer experience in like adding those extra tools where, you know, this is a tool only my company can see and just being able to like plop it in real quick.

28:15I'm looking at your GitHub and notice 117 contributors listed there. As your community built up around this has evolved, going from sole developer in the beginning and now you have a group of people that are actively contributing at some level, how has that changed the project and changed how you're spending your time to fulfill the expectations of so many people and all the folks that they're serving in turn? Yeah, it's been amazing. I've learned so much in the process. I think I need to be conservative with my estimates on getting things done so I can address contributions and things like that.

28:55And that is definitely a thing I want to keep, if anything, even more, devote more time because some people are making really great things. I'll even shout out Marco, who is constantly contributing things. And there's things that he just gets through so much quicker than I do that I don't quite find time to review. But it's sitting there and it's great. It's already working. And I want to dig in the weeds a little bit. But also, I think that's really what's helped the project explode, that there's such an openness to what people want to see in it. You know, I just had someone today say that they were, it was their first open source contribution.

29:35And I just thought that was, you know, that's really cool to see just kind of people learning in the process. And I was there too. I was always kind of daunted of like contributing to anything. So just seeing people, you know, step in the water, I definitely want to foster that more. I'm curious as a follow-up to that, has there been a point where you've seen adoption occurring? And as part of that adoption, not that you have favorite children, so to speak, and I understand that you're super happy for every organization out there. But has there been a moment where you've seen some organization that, you know, you might be super familiar with or something adopt or know about, you know, and kind of went, holy mackerel, I can't believe that they're using my stuff.

30:23Has there been a moment like that for you? Oh, yeah, for sure. I caught wind of Mistral using the app just to prototype their chat interface. That's the only one I know for sure. but I've also you know there have been people within Microsoft who are kind of just helping people prototype you know their own interfaces and things like that and it's just like that to me is you know a step back and I'm just kind of blown away the big boys of this space definitely yeah in our system in the way that we've kind of customized LiberChat here we use this model-based factuality score which is actually factual consistency between reference text and text out of an LLM so you can do a factuality check between two different pieces of text to get a score that would show kind of factual consistency between the two which that's kind of the most relevant thing for most LLM use cases because many people are using RAG or they have internal company data that represents a source of truth so in libera chat here we've uh we're working on the integration with the rag piece which would be a cool integration there but um for now we just have this sort of factuality context so facts that shouldn't be violated right and the i could put uh something here and turn on the factuality check and then ask a question so the fact i put in was that the sky was green, I could ask, you know, what what color is the sky?