Evaluating AI Models: Arthur's Launch of Bench, an Open-Source Tool

0:00The holidays are here and that means it's the most wonderful time of the year to save with Rakuten. Use Rakuten to stack cashback at your favorite stores on top of holiday sales. That's savings on savings. With Racketing, you get cash back on gifts for everyone on your list. From toys for the kids, to kitchen gear for the person who loves to cook, to electronics for everyone. You can even save on something for yourself. Just shop the stores you love and cash back is automatically added to your account. And you can get paid with gift cards, PayPal or check. or eligible American Express card members can even choose to earn membership rewards points instead of cashback.

0:41It's truly a no-brainer. Join for free today and get a new member bonus after minimum qualifying purchases. Just go to Rakuten.com, download the app, or install the browser extension. That's R-A-K-U-T-E-N. Terms and conditions apply. What can 160 years of experience teach you about the future? When it comes to protecting what matters, Pacific Life provides life insurance, retirement income, and employee benefits for people and businesses building a more confident tomorrow. Strategies rooted in strength and backed by experience. Ask a financial professional how Pacific Life can help you today. Pacific Life Insurance Company, Omaha, Nebraska, and in New York.

1:24Pacific Life and Annuity, Phoenix, Arizona. Since the AI boom began, a lot of people have been asking the question, and that is what AI model is the best? Now, we know, of course, ChatGPT is the biggest one that first came out. But since then, you have Anthropic, who's come out with Claude, which is really good. You have Pi, which is creating inflection. You have a lot of other big and heavily funded AI models. And a lot of people want to know which the best one is. So today on the podcast, we are covering a startup called Arthur that has just unveiled Bench, which is an open source AI model evaluator.

1:55We're going to be diving into what they're building and why this is important. So the headline here is that this open source tool is specifically designed to evaluate and compare the performance of major LLMs, such as, you know, you have OpenEyes, ChatGPT 3.5, Turbo, and then you also have Meta's Lama 2, a bunch of different of these AIs, and you can actually compare them. So Adam Winchell is the CEO and co-founder of Arthurings. he said, quote, with Bench, we've designed an open source platform that lets teams dive deep into understanding the variations between different LLM providers, different prompting approaches, augmentation strategies, and custom training methods.

2:32So if we kind of dive inside Arthur's, you know, bench mechanism and what that looks like, I think the main advantage of Arthur Bench is that it enables enterprises to gauge the efficiency of various language models according to their individual use cases. So it offers metrics that allow a comparison between models based on accuracy, readability, hedging, and more. So for anyone that is familiar with LLMs, hedging can be a reoccurring problem. The issue essentially arises when an LLM gives unnecessary language that either alludes to its terms of service or programming limits. So for instance, phrases like, you know, as an AI language model, you know, that you get from ChatShapyT a lot.

3:15And this, you know, these don't typically relate to what a user actually wants. So if a customer, you know, like, for example, that's ChatGPT, whatever, we get over that. But like when an enterprise is integrating ChatGPT into their product is it's using it to help a customer with a specific task. They don't want it to say that, obviously, right? So Arthur's platform showcases a variety of initial criteria for comparing LLM performance. and given its open source nature, users can actually add criteria tailored to their unique requirements. Wenschel further clarified the platform's capabilities stating, enterprises can input the last 100 questions from their users, test them across different models, and Arthur Bench will point out the distinct variations for a manual review.

4:03The overarching aim is to streamline the decision-making process for businesses looking to integrate AI. I, for one, think this is an absolutely amazing tool. It's really cool. They've done this open source. So big tip of the hat to them in that regard. But honestly, this is amazing. There's so many different companies that are looking at integrating AI right now, and it's really hard to know which is best. This is a really quick way, right? You throw in the last 100 queries from your customers or like whatever the AI is gonna be used for. You throw in like 100 variations, and it's gonna give you all the different variations between the different AIs, and it's gonna help you narrow down very quickly which one gives the best responses that are actually helpful to your customers.

4:42So I think in essence, Arthur, you know, not only benchmarks, but also converts academic evaluations into tangible business outcomes. And I think the tool really employs a mix of statistical evaluations and LLM assessments to kind of juxtapose the response of various LLMs. So if you kind of look at the real world application of bench of arthur bench um i think you know you can think of you know like financial institutions um they have already been leveraging arthur bench to expedite the creation of investment hypotheses and analysis which is really interesting in the automotive sector manufacturers are using the tool to transform extensive equipment manuals into llms that can swiftly and accurately answer customer inquiries um drawing directly from those manuals and minimizing errors.

5:32So Axios headquarters, an independent media and publishing platform, you may have seen Axios, right? They got a journal, they write a lot. It is another ArthroBench user, our ArthroBench user on its product development front. So Oberoi, who is a data scientist at Axios HQ mentioned, quote, ArthurBench has allowed us to build a standardized evaluation framework for LLMs and present the results to our product team in a comprehensible manner. So I think emphasizing its you know kind of commitment to the open source community which i think is important right if you want to get people on board in the open source community you got to let them know you're committed and so kind of in that vein arthur is offering bench for absolute free right we've seen other open source projects i think particularly out of facebook right they released some of their lama models where they're like yeah it's open source but you can't use it for any like enterprise activity it's like it doesn't feel super open source when you kind of put stipulations on the usage of it, in my opinion.

6:30And so I think this is interesting. They're offering this for free and they are expecting that the best products emerge from open source initiatives. So the company actually envisions generating revenue through team dashboards, which is kind of interesting. And I mean, at the end of the day, you have to make money from a product. So I don't hold that against them in any way. I've even seen, I've seen quite a number of AI startups that launch with an open source play. And then, you know, they get to a point where they're like, hey, our open source library has over a million downloads and they kind of launch into like a premium tier where they take that their product to the next level, they support it or they add extra complimentary features.

7:09They're able to monetize that way. But I think this is a really great approach because it really, number one, it proves that there is market demand for what they're offering. And number two, it really puts in a lot of credibility and good word for their brand and their name out there. So when people are looking to kind of take something to the next level, they can see, hey, this company already created this great product, started this project, and now they're, you know, they're taking it further. And I mean, even I, for one, use a lot of different open source projects, or a handful in different things I'm doing.

7:41And, you know, they'll typically have a premium tier that beyond the free tier, if you're an enterprise using it for something, you got to pay extra licensing, which we're usually thrilled to, because we could use the open source version to get integrated and evaluate it. And then when we're scaling, There's some kind of nominal fee that kind of helps keep the lights on, which we're actually happy to pay because we know that by paying that, the software continues to be developed and worked on and improved, which is, you know, at the end of the day, what you would like to happen to your software suppliers.

8:12So I think in doing this, they really envision gathering, you know, revenue through their team dashboard, as I mentioned. And I think it'll be interesting to see how that monetization strategy actually goes. As far as some upcoming collaborative initiatives that they're doing, I think Arthur recently broke the news about a hackathon in partnership with AWS, Amazon Web Servers, and also Cohere, which is a massive AI company. And the goal is to motivate developers to craft new metrics for Arthur Bench, which I think is really interesting. So Wenschel drew parallels between AWS's bedrock environment tailored for choosing and rolling out different LLMs and Arthur Bench, and he kind of highlighted their shared philosophy.

8:52be. So earlier this year, the company also introduced Arthur Shield, which is an initiative aimed at monitoring LLMs for potential errors and anomalies. And I think it's really interesting. This company really is making some big strides and they're coming up with some great technology. So this is definitely one I'm very curious to follow in the future.