Adam Wenchel: How Arthur AI is Making LLMs Trustworthy

25:04And basically as a way to evaluate the LLM before integrating it, they use Arthur to do that. Is that right? Yeah, absolutely. So we kind of work across three major areas of the lifecycle, right? So free production validation, which is like what LLM should I use? If I'm using, most people are using RAG, Vector Data Store, Retrieval Automated Generation. And there's a lot of choice of which RAG database do I use, which Vector Store, what's my retrieval strategy, how do I chunk up the documents when I put them in there. There's all these kind of knobs and dials when you're building these systems.

25:34And so you need to understand as you're turning these knobs and dials and getting it all hooked up, am I making it better, am I making it worse, is it good enough, basic questions like that. And, you know, on the model side, do I need GPT-4? Can I get away with GPT-3.5, which is a lot cheaper and a lot faster? Can I use LLAMA V2 and fine-tune it? Like there's all these, there's just a lot of options. And you get wildly different results depending on which option you choose. And so it's really helpful to be able to do that in a really methodical and fast way. So there's the validation. Next stage is the real-time controls around it, which is when you put these systems out there, you need to have some guardrails, right?

26:11And so there's sort of universal things like you don't want them to hallucinate, especially if you're in something like a medical or legal or financial context. You really don't want them hallucinating, which is basically just a fancy word for giving wrong answers. Right. And so being able to kind of catch those in real time and block them, as well as things like prompt block blocking prompt injection. Just blocking, just like if it hallucinates, just like basically never returning that to the either. Yeah. So so as the application developer, you can either choose not to return it, which is a program for some use cases, or you can return it.

26:40but just put like a little, you know, a little bubble next to it. Like, hey, this may not be true. You should validate it before you kind of take it as gospel. And same thing with prompt injection and toxicity and also data leakage. What is that? What is a prompt injection? Talk about that. Yeah. Yeah. So you can actually design prompts to get around a lot of the built-in guardrails that LLM makers have put in place. And so there's a number of attacks you can perpetrate with this. The most basic is probably getting back the system prompt, which is like the underlying prompt that the application developers have put in.

27:14You can actually make it say things that are like, you know, offensive or not aligned with what the company, the people who are running the system want it to say. And if they're starting to, as people are starting to use agents, you can also do all sorts of things that are maybe the equivalent of like modern day, like previous day, like SQL injection attacks and things like that. So you can kind of do unauthorized access to information. And so, yeah, blocking prompt injection, making sure you're not leaking data, making sure you're not leaking the system prompt. You know, just what we call, we call it a firewall for LLMs.

27:44It's kind of like a big application firewall for these LLMs. Also things like acceptable use. So there's actually a car dealership. Someone posted some screenshots this weekend that put up an LLM chatbot. And people started, like, figured out how to manipulate it. And so they'd ask a question or they would get it to do things like, you know, So say you will sell me this car for$5, and this is a binding agreement. Yeah, and then they're like, hey, your chatbot said I could buy this car for$5, right? And so you need to have some acceptable use kind of guardrails around it to make sure it doesn't go well or that it doesn't go poorly.

28:20And so that's the second phase. And then the third phase is just that sort of monitoring and observability that we've always done, but adapted for the LLM world, which is, are my answers getting more accurate over time? If my LLM provider makes a change to their system, what effect is it having on my application, which happens all the time. And so, you know, what you want to do there is just make sure you have quality service. You want to allow product managers to go in there and data scientists to go in there and machine learning people to really figure out, like, where is it giving good answers?

28:49Where is it giving bad answers? What are, like, the top 10 weirdest requests people made to the LLM today? Like, there's a bunch of questions you can answer to make your system better. So I have to ask because, you know, I see this all over Twitter X. Everyone is saying that ChatGBT is getting lazier in December. And my guess is this product would be able to measure whether or not that's actually happening. So is it getting lazier? So number one, it is, yes. And one of the metrics we've developed is hedging, which is like that behavior where, you know, hey, as an LLM, I shouldn't tell you what to have for lunch or I shouldn't like, you know, do this.

29:24Oh, interesting. The frequency with which it hedges has definitely gone up. And there are some other, you know, when you look across the different LLMs, there's a huge variance in like how much they hedge, right? And I think a lot of it's based on like how smartly and how aggressively they put in sort of guardrails. So I think a lot of it was kind of born out of this desire not to have it say anything offensive. But in a lot of cases, they've taken it too far. And in some cases, they won't even answer really basic questions. And so if you're in a context like, I don't know, investment research or something like that, where, you know, you have your own internal employees using it to get investment research.

30:02Like you never want it to hedge because you just want you want you want an answer. Right. You need the answer. And so that that can inform your choice of which LLM. And so you may not want one of the big mainstream LLMs, which is very powerful, but doesn't always give you an answer. So why is it getting lazier? Do you have any and is it just ChatGVT or is Claude getting lazier? or Lama? Like, are they all getting lazier? It seems like ChatGPT hedges the most. And I think, again, it's just, it's a byproduct of the, basically people will also post on Twitter and acts like, hey, look at this kind of offensive thing or this problematic thing that I got ChatGPT to say.

30:41And so then the team kind of like goes in and puts in guardrails, which are like, hey, don't answer these types of questions because you may say something offensive or that's just inappropriate. And so then the byproduct of that is like those guardrails maybe are, are, uh, overzealous sometimes and, uh, lead to it then not even wanting to answer certain innocuous questions, um, which, which is frustrating as an application developer. Got it. So it almost sounds like maybe we've just reached a tipping point in terms of the guardrails being implemented such that it's, it's creating like a bit of a restrictive environment for the LLM.

31:16Um, it's not that it's actually getting lazier. Yeah, it's all, I mean, this is all, this is all really early technology. Yeah, of course. Yeah. We're everyone's learning as I'm sure open AI is too, right? I mean, Oh, big time, big time. Yeah. So I think it's a lot of like people are seeing in a very public way, just sort of like the, the throes of a very powerful, but very early technology. That's, you know, it's not, it's not hard to see the possibility, but, uh, it's also, there's just so many things that can go wrong right now, uh, given the early nature of it. Yeah, totally. Speaking of open AI, like, how do you think about, um, sort of the, the big kind of LLM wars.

31:52I mean, is open AI kind of walking away with the market? Is Claude still in there? Are we actually moving more towards an open source future? Like, how do you see this playing out? Yeah, I think that the future is going to be pretty diverse, actually. And I think that's good for everyone. Open AI is certainly a really powerful model and system, right? And definitely kudos to them. They've been really kind of attacking this problem in a bigger way than other people for a long time now and I think that you see that. But I also think that they're building for kind of the mainstream and there's a lot of like the open source models that we see already have huge advantages in areas where people are doing really cool things like taking a smaller like 7B model and fine tuning it for certain specific use cases and they can get wildly better performance.

32:41And so I think that there's, we're starting to see more of that. We're starting to see people begin to train their own models more often. And so, you know, that stuff's a little bit early. I think it'll be another year before you really start to see that in the real world a lot. But we're starting to see it now. And so I think that there's going to be a really nice mixture of commercial providers as well as open source offerings, everything from kind of large general purpose models like GPT-4 to these really kind of focused models like, you know, fine tuning a small LLM that just detects prompts for prompt injection, you know, things like that, that are very sort of purpose driven ones.

33:18And as a consequence, they don't have that horrible latency. They're much cheaper to operate. They've just got a big, big advantages that way. Yeah. And it seems like every week there's sort of a new model coming out that is claiming to be better than the last previous model, you know, coming out with their own benchmarks. Gemini, obviously a couple of weeks ago i mean how how should we um as customers of these models think about these benchmarks that are getting released every other week i mean are like are they accurate can anyone just kind of uh tweak their measurement in such a way that their benchmark always looks the best i mean there's certainly some of that that goes on that's always really what's what what's there's like a name for that effect is it obviously is it dunning kruger or different effect Like once you sort of like start optimizing for a particular metric, it becomes useless.

34:06And there's definitely some of that. I think, look, those kind of general purpose metrics are a good, you know, first cut at like understanding the performance of a model. But what we preach, and so we have an open source tool called Bench, which allows people to understand. What you want to do is understand not just how does it perform against some generic benchmark, but how does it perform against my exact kind of workload, right? So if I'm using it for customer service or for investment research, I want to take like, you know, 500 of my real prompts and responses and run it through the system and see how it how it does on that.

34:37Because I don't really care about these like generic benchmarks. And they're not right. They're oftentimes not that indicative of how it's going to perform on my application. Right. And so that's what we have, like basically a testing tool that allows you to test it with your with your prompts on your data. and it's allowed you to know, like, all right, this big headline about how this is like revolutionizing that this is like the latest and greatest state of the art LLM. Like, does that actually matter for me or not? Right. The generic benchmarks don't matter. It's my benchmarks and how this product is actually going to perform for me that I should care about.

35:09Exactly. With my vector data store and my data and like all these variables. Yeah, that makes so much sense. So, you know, my my understanding is that Arthur is mainly focused on LLMs, but obviously as we're talking about sort of the future of models, we're seeing all these different types of models, right? Image models, video models, models for generating music. How do you think about the opportunity for Arthur in, you know, other, other formats and other forms of media? Yeah. So we, I mean, you know, prior to the, the generative AI boom last year, we like, we support a computer, we support computer vision to this day.

35:39You know, we have really robust, um, support for computer vision, for NLP, um, for, for tabular data, all, um, time series, these recommender systems. And so for us, I think we see a ton of business value being created or about to be created through text, generative AI for text, so LLMs. I think that the business value creation part of these multimodal models is still earlier. I don't doubt that it'll happen. But I think one of the things we're looking closely at and talking to our customers about is like, you know, what kind of value are you going to get out of being able to create these videos on the fly or create images on the fly, like in a business context, like, is this going to help you automate some sort of back office process?

36:23Or, or is this going to help you like service your customers better? And so I think that it's still kind of early in those conversations, like people are still figuring that out. And so we're watching it closely. I think because we do have like, you know, good expertise in computer vision and some of these other areas, we'll be ready for it when it when it really starts to have an impact in the world um so we're watching it closely uh but it's still early days yeah i can see what you're saying like especially in sort of the b2b use cases which is where you know i think arthur plays most like the the value is still not super clear where it obviously very very much is with llms and language right you know i think we need to see how it plays out outside of the consumer use cases for some of these generative media models yeah talk a little bit about like you know we've talked about the products we've talked about what they do, but talk about the impact of the past year on your company and your team.

37:13What's that been like? Again, at the beginning of this conversation, we were talking about how you have been working in AI for literally a couple of decades. Your team has been working on AI at Arthur for five-ish years. But now in the past year, it's like the whole world wakes up and decides that this is the most important technology in the world. What's that been like for your your cadence, your culture, everything. Yeah, it's been a crazy year. So I think, number one, don't go into working in AI if you're not the kind of person who thrives off of constant learning and change and adaptation because it's just wild.

37:50Every few months, there's something new coming along that just completely changes your understanding. And I've been working in AI for 20 years, and it's just one of those fields where every day I learn more and more and feel like I know less and less because there's just so much new stuff happening. But it's been, you know, it's obviously been a huge boost for the team, for hiring, for like talent, like so many talented people now are really just, I think, just moved like in a really existential way by the power of this technology. And they're like, oh, my gosh, I want to go work at a company that's building, that's like helping kind of define the future with generative AI.

38:23And so that's been really exciting. you know it's funny I think when Chatsby was first released a little over a year ago the I think there was a part of me who has been you know from having been in the world I was just like alright this is like the latest like hype du jour and I was almost sort of like dismissive of it for the first couple months but then you know by the time we got into January and we were just spending time with our customers and they were like talking not just that they were excited about the technology but here's all the ways we're going to use it and here's how like all the areas that we've identified that this could really be a game changer I started to get past my jaded nature and really get excited about it.

38:57And that's, I think, when we really started to say, hey, we've got to do something. This isn't a matter of just supporting generative AI and our observability. We actually want to build out this validation tool because that's one of the biggest impediments we've heard from our customers to getting started with generative AI. And then the second one being these real-time controls. And so there's a real opportunity for us. I think anytime you have this sort of just massive change, it just it creates opportunity. Right. And there's like winners or losers created out of it. So we just really wanted to go back to first principles and not be beholden to kind of what we what we were already doing and really just think about what the what the market needed and what our customers needed to be successful.

39:38For sure. And how do you think those needs are going to evolve? I mean, as an example, I mean, could you see a lot of these LLMs, especially the big ones like OpenAI and Anthropic? Like, could you see them building evaluation tools, firewall tools, performance tools directly into their models and offerings? And what would that do to Arthur? We already see like little pieces of it for sure. Really? And I think it's not that different from when we started just AI observability and people were like, well, AWS is going to build that, right? And I think so. I think there's a couple of things. One, those kind of first-party offerings are tied to their stack.

40:16And even when they build it in a way they say it's not, really, you're only going to use it with their stack, right? So if OpenAI comes out of tools, you're only going to use OpenAI. What we find is our customers, especially large enterprises, they don't just have OpenAI. They're agnostic. They have all of them, right? And they want to be able to choose the best tool for the job. and so they want that kind of single pane of glass across all of them that gives them that real-time visibility into what's going on. And then the second thing is, if you're like Amazon and you're selling compute or Azure, you know, compute and storage, like it's a lot, as you get up the stack, sort of the imperative to do a great job becomes like, you know, less and less strong.

40:57It's less and less existential for you. And so what we find is like, you know, They may have basic versions of observability or basic versions of guardrails, but a lot of times it actually helps us because people will start down the path of using those basic built-in tools, and then they realize they're not sufficient, and then they start looking around and they find us. There's still opportunity for software companies, even though obviously those companies have created a lot of momentum and they've created some really powerful engines, but we still like where we're at. What about companies that are sort of doing their own model development?

41:33You know, sort of starting with OpenAI, then maybe graduating to being a little bit more model agnostic and then starting to build and train their own models. Is the need still there for something like Arthur? Or are they just building this evaluation directly into whatever they're doing? Oh, no, those are that's a huge part of it. I think both for the validation, because there's like, again, there's like the amount of when you're training your own model or fine tuning the number of knobs and dials and also just the potential that like you can actually make things way worse if you fine tune a model, if you don't know what you're doing.

42:04And so you need to be you need to like make sure that you're you have a really good bead on what affects the changes you're making are having. So it actually creates even more opportunity for us. And the other thing is like where some of the guardrails like toxicity is a good example. Right. I would say the major LLM providers have done a pretty good job of like getting rid of like the most harmful like that. You know, you can't get it to cuss up a storm the way you could like in the early days, things like that. But if you're building your own, it's like, are you really going to invest like a hundred person team in like, you know, in building in those kind of toxicity filters?

42:39Are you just going to buy something off the shelf for for a relatively small amount of money? You'd much rather just that's not the strategic part of what you're trying to do. And so it's better to kind of work with someone like us who can just bring that to you right away, like a best debris product. How does your product get more challenging to build and maintain as the capability of the model improves? You know, for example, we've seen information leak that, you know, GBT 4.5 may be dropping soon. GBT 5 will be here, you know, whatever, before we know it, I'm sure. What happens each time a new model drops?

43:13Like, what does your team have to do in that moment? Yeah, I think what we do, so we're kind of agnostic to the LLM, but where it does affect us is, as I mentioned, a lot of our evaluation routines do use LLMs. And so like hallucination detection, we have kind of a layered approach to techniques. But one of those techniques is actually using LLM to help in the kind of determination of where the hallucinations are and what they are. And so, you know, every time a new LLM comes out, we benchmark our own internal tools against it. Like, hey, and there's typically it's not like, is this better? Should we worse?

43:48It's typically a tradeoff, right? Like, oh, hey, we got a better score out of like GPT-10, but it's also costing$7 million a day to run. Like, maybe that's not what we need. Or it takes 30 seconds to come back with an answer. And that's like not acceptable. Right. And so it's really like every time one comes out, like we ourselves use our own tool bench to understand, like, what are the tradeoffs with this particular LLMS? Like, how does it perform? How fast or slow is it? How much does it cost us to run? like what is it worth it is it not which of our own internal kind of like routines is it is are those trade-offs good for which are they bad for things like that got it we've talked a bunch about hallucinations how exactly does firewall cut down on hallucinations i mean this might sound simple but like is it doing sort of real-time google searches to like validate claims like how like how does that actually work so there's like a whole taxonomy of hallucinations and wrong answers and so there's some that are intrinsic to the model the kind that we focus the most on because this is the most helpful in the world.

44:47So like 99 % of the LLM systems that go in the real world are these RAG retrieval automated generation systems where you're taking all of your proprietary knowledge and putting it in a vector data store. And then when I go to ask a question like, hey, what's this drought in Brazil going to do to energy stocks there? Something like that. It'll use like all this investment research that's been loaded into this vector store. And a vector store is a way of like, it's kind of a way of indexing knowledge by, in a way that like makes sense from a language perspective. And so like, you know, cat and kitten are actually like, even though they're spelled very differently, they don't like say anything.

45:26If you put them in a vector data store, they're actually like right next to each other because they're very close in meeting. Right. And so, so if I say like, what is this drought in Brazil going to do to energy production in that country? What it does is it doesn't just give that to the model and answer the question because the model won't have a lot of information. What it does is it goes to the vector data store and it pulls back pages and pages of relevant kind of research that maybe my analysts have done in the past that goes back decades, that's maybe seen similar trends. And it'll append that knowledge base to my question.

45:58So it'll say to the LLM, here's Adam's question and here's some information you should use to answer it. And then it generates the answer that way. And that way, you know, you can get a much richer answer about data that the model has not been trained on. And it tends to be much more accurate. Now, what we do is we take the response. There's a number of techniques, but one of the more common ones is we take that response, break it down into a series of claims. And for each of those claims, we validate that against the underlying knowledge that's available to make sure that each claim is well supported.

46:33Because oftentimes it's not. And some of our clients see some pretty high rates of hallucinations depending on what they're doing. And so being able to spot them is huge. It's a simple concept. There's a lot of tradecraft that goes into, you know, you can get like 80 % accurate pretty easily. But getting them to like the high 90s, like we're achieving, it takes some iteration. So the team's done some pretty good work there. So much of what Arthur is doing seems like it's geared towards keeping LLM safe inside of organizations. And so, you know, I have to ask about sort of regulation and maybe the climate in general.

47:08Obviously, it's a very, very hot topic these days. Where do you think we as a country are headed in terms of regulation? And what do you think is needed versus not needed? It seems like this has become a very, very sort of tribal discussion very quickly. Help us make sense of it and kind of cut through maybe somewhere in the middle to figure out what's the actual ground truth. Yeah. So, I mean, we could do a whole podcast just on this topic. It's a fascinating one. But it's, you know, generally speaking, you want the regulation to be light enough that you can maintain room for innovation and companies can innovate.

47:41But you want it to be heavy enough that people aren't getting hurt in the process. Right. Like whether it's like getting run over by a driverless, literally getting hurt, like run over by a driverless car or just being negatively impacted, like can't get a credit card or something like that. And so I think the U.S. has done a pretty good job balancing that. Compared to the EU, the EU has been much more pro-consumer, or I shouldn't say pro-consumer, but much more focused on consumer protection and less on innovation. The U.S. has probably balanced more towards innovation. But I think the executive order that came out last month was a good start.

48:14And in the executive order, it's a good framework. If you read it, a lot of the pieces are sort of like charging, like, all right, we need to do something in this area. know, we're going to give this agency, you know, six months to figure it out and report back. And so it kind of created the framework and a lot of the actual details are going to be filled out in the next year. And so we'll see how that all shakes out. I think, you know, from my perspective, the thing that needs to happen is there just needs to be clear accountability for the people who are deploying the models. And so I think one agency that's done a good job of this is the EEOC, the Equal Employment Opportunity Commission.

48:51If I purchase an off-the-shelf resume screening tool or candidate screening tool and I deploy it, regardless of whether I built that model, like if I'm a large enterprise that employs tens of thousands of people and it's shown that I use this tool that's discriminatory, I'm liable for it because I'm the one running it. I'm the one using it. So it doesn't matter whether I built it or not, the fact that I'm using it. And I think that kind of clear accountability for you're responsible for any sort of bad repercussions that come of this is really important. Yeah, for sure. What are you maybe as a consumer most excited about with respect to AI looking forward?

49:28Yeah, I mean, I've had a ton of fun with it. Just like even silly thing, like I use ChatGPT almost every day and just doing fun stuff, like helping my kids with their homework. My daughter's old enough now that she's starting to get into pre-calc and things like that, that I don't, chances off the top of my head, remember I have to go back and kind of refresh And so, you know, we were asking for to help her with a trigonometry problem the other day. And it didn't get it right, but it did get it actually broke it down into these really clear, concise steps. And the thought process was actually right spot on.

50:03The arithmetic was like slightly off, but that was easy to correct. And so I think it's been pretty awesome just to see that. Like, I don't know, it's just one of many examples of being able to kind of use it. Another fun one is, if you look at our website, we have a specific illustration style that we've kind of made our own. And we worked with an illustrator who, and this was all done sort of in an ethical way, but to have her produce a set of illustrations. And then we use different foundation models to actually create different ones in that same illustration style. So if one of our, you know, someone's writing a blog post and they want this abstract illustration that represents, I don't know, measuring hallucination rates or something like that, they can feed all this into this foundation model and actually get an illustration in that same style that is, but that's like completely custom.

50:55And so there's just so many fun applications of this as a consumer I'm doing. What about yourself? Do you have any favorite ones? I mean, I use ChatGPT every day as well for, you know, all sorts of use cases. And, you know, my background has a lot to do with media and creator tools. So I'm, you know, I'm very, very interested in in some of the the other gender models around video and imaging and things like that. I'm constantly always playing with that stuff. So maybe looking ahead, you know, I don't think any of us could have predicted what the past year has been like. But as we record this, you know, near the top of 2024, like what are your predictions for the year ahead?

51:35Yeah, I mean, to your point, I'm almost like, like I know whatever prediction I make will be like, it'll be wildly different, right? Because it's the nature. But I do think that, you know, we're starting to see the hype turn into reality a lot more. And so I think it's going to be what I think is there's going to be some really compelling applications of this technology that that hit the real world. And there's also going to be some high profile kind of like unintended effects of it that were people kind of bad things happen or funny things happen, depending on the scenario. And so I think that is definitely going to happen.

52:11I do think that the multimodal models like it's going to be very interesting, especially with an upcoming election. you know, what, what effect being able to just really quickly generate realistic videos and, and, and images will have on things. So yeah, that's also the, the, the coinciding of, of kind of a very contentious election, foreign interference and, and, and generative AI is going to be, it's going to be interesting to watch. For sure. For sure. I got to ask, is Arthur hiring and where can listeners find out more about how to apply if they're interested? Yeah, we are absolutely hiring.

52:48We're at, just go to arthur.ai, our website. Appreciate asking and, you know, hiring across the board for machine learning engineers and a number of like customer solutions type of engineers are definitely top of mind. And we're based in New York here, down in Soho. So, you know, we have opportunities for both sort of virtual people as well as people who like to be in the office, as a lot of our team does. So definitely check us out. Drop a line. Awesome. And anything else we should tell the listeners about Arthur or anything you want to make sure they're aware of? I think we covered it pretty well.

53:21You know, just it's an exciting place to be right now. If you're the kind of person who gets bored easily, if you're not constantly having to learn new things, then it's the right place to be. For sure. Awesome. Well, Adam, this has been so much fun and really appreciate all the time you've given us. Thanks for coming on the pod and hope we can do it again sometime. Yeah. Thanks for having me on, Michael. Thank you so much for listening to Generative Now. If you liked what you heard, please do us a favor and rate and review this podcast on Apple Podcasts and Spotify. It really, really does help.

53:54And if you'd like to learn more, you can follow Lightspeed at LightspeedVP on YouTube, Twitter, LinkedIn, or any other social platform. Generative Now is produced by Lightspeed in partnership with Pod People. I am Michael Magnano, and we will be back next week with another awesome conversation. See you then.