#316 Robbie Goldfarb: Why the Future of AI Depends on Better Judgment

29:12Let's bring them on. And then we can figure out on the technical side how to scale them through some of the techniques we were talking about, right? Rather than let's get 10 or 15 ,000 people and then throw them straight at the work. Or maybe. Yeah.

29:27Robbie Goldfarb:I mean, do they have to be living people you you could turn to you know the the writings of philosophers or or scientists uh to follow their thought process yeah have you seen there's a video of steve jobs from 1985 where he talks about predicting this this future with aristotle i i may have but i yeah you got check it out it's it's under this is in 1985 and he says something like um he believes there's going to be a day where when the next aristotle when the next aristotle comes we'll be able to capture their underlying worldview in a computer and be able to converse with it and he goes on to essentially describe kind of what the way these llms work today but in 1985 which is crazy um But anyways, that's more of an aside.

30:29Robbie Goldfarb:Yeah, I think. Actually, before you move on from that, but don't forget your point. I know a guy at Boston Consulting, a very senior guy, and he has this idea that, you know, there's all of this effort with RLHF and fine-tuning to nudge LLMs toward certain moral and ethical positions or positions. personas or uh and that the world has 5 000 years of uh of religious texts that that deals with that with moral and ethical questions and that you could use that to guide an llm in its uh thought process or in its not thought process but in its uh in its uh reasoning outputs yeah reasoning exactly yeah yeah i mean it's very it's interesting like i mean because what what this is ultimately getting at is the is the point we were talking about before which is in in subjective domains there is no ground truth so what is the ground truth and it's a very interesting case to say i think the ground truth should be you know the entirety of these religious text and ethical like writing or whatever it may be um i mean our position at forum once again is we think the best answer is to take you know the top few hundred smartest people in a given area and look to them to define this um but there's certainly something it's a very interesting idea yeah yeah and but uh do you go back and use uh experts uh through their writing to to map their thought process we just started doing this two weeks ago we hired a researcher who's specifically pretty good at this sort of thing we're calling it extraction and so still trying to figure it out but what we are finding is that when you get access to certain content about you can extract certain themes or judgments that the author has in a way that might be generalizable for, for example, for training our judges.

32:57I think there's a lot, there's more work we have to do here. There's also a lot of other research in the space that we're looking to. So this isn't something entirely new. But yes, you're, you're definitely right. That's a, that's an interesting opportunity. What we haven't thought of, or we haven't actively been working on, which you're alluding to, which is very interesting is looking at the texts of deceased experts, right? And that could be a really interesting way to approach it that maybe yeah yeah i mean uh you know if you

33:31Robbie Goldfarb:could map einstein's thought process may be valuable uh for for research systems for sure i mean i can just say just from the work we've done with with living experts it's it's really hard when they're alive and you can actually go back and forth with them so i imagine this would be pretty tough but it's probably something there um yeah yeah yeah how much of this can you surface using an llm because uh you know the largest llms have farid zakaria's writings in their training data uh and presumably uh they would be able to to i mean i i as we said i mean they're really looking at probability distributions, but in the reasoning, maybe they're influenced by reasoning in the training data.

34:33I mean, totally. I mean, first of all, I'll just say this is probably getting a little bit, you know, this is a deeper research question. I think what is interesting though, because I've seen this in a number of the clients we've worked with, and we've had these discussions several times, LLMs, of course, do generalize quite a bit from their training data. I mean, that's essentially how they're designed. And that can lead to a ton of unintended consequences. Sometimes they can be good, like what you're implying, but how do you know that it's going to generalize the content from Fareed versus maybe something erroneous it picked up from a Reddit post that it was trained on?

35:15And I think that gets into the complexity of thinking through the composition of your training data and how you ultimately go about like post-training and fine-tuning your models.

35:27So, yeah, I mean, I think there is, though, I mean, the piece that interests me a lot that you're getting at is how do we leverage these experts and potentially their content too as much as possible to help guide and inform these models. My sense is it's probably going to be more post-training, more about evaluating, having them set the gold standard and then hill climbing towards that than pre-training. But who knows? Yeah.

35:59Robbie Goldfarb:And so you've got this large network, but focused on quality, not quantity. You haven't mentioned the quantity, but I'm guessing that's proprietary information. uh how do you see this affecting the future of ai i mean presuming that that your methodology uh becomes widely adopted i mean the the thing i would just go back to here and if we have a broader objective is trust right there there was a a couple months ago kpmg released this report one of these public opinion polls and it showed that 80 something percent of people are optimistic about the potential of ai to improve their lives i don't remember exactly how it was written but something yeah yet only 40 something percent trusted it and that is a really i think that says a lot right it's because if if no one was interested in it and no one trusted it then who cares but the fact that there is an interest and a demand for it but the trust is low suggests that that delta there is going back to dario's essay where he talks about this amazing potential future.

37:27Trust is a blocker. There's actually one other thing that folks should read is that Forbes had a piece a few months ago and it says, the article is titled, Consumer Trust is the Next Battleground for AI. I think it's for that reason exactly. And so going back to your question, I think what we're going to be able to do is to build models that people can trust. Because number one, people trust people and you want to see that there are trusted people behind them. But number two, and most importantly, the outputs speak for themselves, right? People have a good sense for what feels like a good experience and what feels like a useful, helpful, reliable, trustworthy, honest experience.

38:14And that's what bringing experts into the mix creates in terms of a product. um so that i mean that's maybe overly grandiose but at a high level that's yeah i think about it

38:27Robbie Goldfarb:yeah although the trust issue isn't that tied more specifically to hallucinations uh the yeah the way the way i think the way i would think about trust is um there's three things here that have come up if we think about trust in ai one is unverifiable sorry verifiable situation so when there is a right answer so when there is a obviously right answer and it says something wrong if you ask it how many straw how many hours there are in strawberry and it only says two it's wrong and that is certainly a major blocker to trust i think that's probably a little bit easier to fix because there is a ground truth and it's just about we got to figure out how to get there, but that's where a lot of hallucination falls in.

39:16And absolutely, you're right. That's a part of it. The next part of it is unverifiable scenarios. What about when there isn't a right answer? And this is like we were talking about before, when there isn't a ground truth. And that's a lot trickier, right? And that's where we posit that experts can be the ground truth, and that's the right way to solve that. Then the other piece that I just throw in there is this concept of responsibility, right? So if those things are talking about how AI acts, there's also this idea of AI knowing when not to act, which I think is really important. I think a trustworthy, responsible AI knows when it should take action, when it should respond, but it also should know when it doesn't have the answer or when it would be irresponsible for it to engage.

40:07So it's a long way of saying, I think hallucinations are certainly one part of it, but there is a bigger picture. And experts can probably play a bigger role on the latter two pieces. Yeah.

40:22Robbie Goldfarb:And that's why I was asking about sucovency, whether you score for that. because I was telling you when we spoke before, I had a conversation with a podcaster who's a smart guy shortly after GPT-3 came out. And he was arguing that this is a truth machine. And we got into this argument because there's a guy out in Arizona that does these experiments that have not been repeated and are very controversial about essentially people showing physiological reactions before the stimulus, which he argues proves that there's a separation between mind-body and that there is this non-physical realm that...

41:31Robbie Goldfarb:Yeah. So, and, you know, it's one guy and, you know, but this guy, when he would ask the LLM, I think it was ChatGP3, it would say, yes, this proves that there is a separation between mind and body. and it drove me crazy because i i was saying the llm doesn't know it's it's just giving you a probability distribution or an answer out of the distribution and you you bias it depending on how you ask the question because if i ask the question i i get no this doesn't prove anything you know So, and that is a syncophancy problem. So do you guys address that? The example you gave feels almost like a borderline hallucination, right?

42:30Where it's essentially saying that it's confident in something that it's not. And so this is where there becomes a bit of a blurry line between verifiable and unverifiable scenarios. But I would argue that that is a verifiable scenario in that I think we can pretty concretely say, or the AI can pretty concretely say that it doesn't know, and it didn't do that.

42:53Robbie Goldfarb:You know, I mean, how do you train or judge an AI's answer on that question? Because it depends on what, whether you're looking at data, what data you're looking at, essentially. Yeah, for sure. I mean, a part of the puzzle that we haven't talked about that's particularly relevant here because this is in the news is the retrieval sources, the news it's looking at, right? These models are only trained and updated every few months. And so a lot of the information they're getting is from the news sources that they or the social media sources or whatever that they have access to to reference. And so that's why, I mean, I mentioned earlier, part of what we do in our evaluations is look at source selection.

43:40And this is why, because I think so much of how an LLM is going to respond is going to be a product of the sources it has access to and it chooses to use and how it chooses to use them. So that is a very high leverage way to improve how these models are responding about some of these more newsworthy topics.

44:01Robbie Goldfarb:If the model looks at the law, looks at the volume of illegal immigrants in the United States, I could imagine an LLM saying, well, yes, this agency is charged with enforcing this law. and there are reasons why that law exists without considering the human or moral aspects to the question. So, yeah, I mean, you... That example is so interesting, right? Because you see what you just did. That's your thought process, right? Your thought process is that you do go through and you consider the moral or the human implications of it. And so that's that. I mean, for what it's worth, I agree. But yeah, it's all about how do we go about answering these questions as opposed to the answer itself.

45:01Robbie Goldfarb:Yeah. Well, let's move on a little bit. What does it mean for AI to have good judgment in your view? Yeah. I mean, this goes back to the conversation we were having before, right? Which is the verifiable and the unverifiable scenarios, right? Verifiable scenarios. I don't think that that is an important part of the piece of the puzzle for trust. So when there is a right answer, it gives you the right answer and it doesn't hallucinate. I think how it handles the unverifiable scenarios, how it behaves when there aren't rules, when there isn't a clear ground truth, that is what judgment is. And that's, of course, really tricky.

45:52What is good judgment? That's then the broader question. And of course, our approach is, well,

45:58Robbie Goldfarb:let's look to the experts to define that yeah uh and um what what do you see as the most critical issues facing the ai industry today is it this trust issue or um yeah i mean so yeah i'll give you a couple specific examples and then i can tell you my biggest broader concern but just from our experience, mental health is a really big one. There was OpenAI had an announcement a week or two ago where they said that every week they're seeing it was like over a million conversations where the user demonstrated suicidal intent. That just gives you a sense of where we're headed or I suppose where we are with this, right?

46:51And so I think figuring that out is really tricky. we've done quite a bit of work in mental health. And the term I would go back to here is clinical nuance. And I think the models don't quite have that now. It's really important in being able to, I'm certainly not an expert here, but work with patients and support patients in a way that's not going to harm them. Classic example here is, do you remember the Tessa chatbot from two years ago. So that was, I mean, this was from the National Eating Disorders Association. They launched this chatbot and there's an article that describes how it didn't have clinical nuance.

47:34They ultimately ended up shutting it down because it had adverse effects on users. But I think mental health is an area where, of course, there's enormous potential, but I think there's a lot of people using it right now and a lot of risk. Another area is politics. and I don't even think we have to go into that too much more because I mean it's all the stuff we've been discussing right so much of this is subjective and to the point I was sharing earlier I think the health of the information landscape the health of information people have you know avoiding echo chambers avoiding misinformation that's so important to building a healthy society And AI can go either way on this one.

48:17It really can. And so I think that's another specific area that we've seen directly. The broader thing I would say to you, and this, like, Craig relates to something we were talking about before we jumped on this call, but we've set, I think, a pretty scary expectation around AI that, and I think this started with ChatGPT. It relates to what your friend said about this is the truth model. But our expectation is that I can ask it anything. And it's going to give me an answer. And it's going to give me an authoritative answer that I can feel like I can trust. And I think that's a really scary expectation to have set.

49:00If you go in to a doctor, there is a natural back and forth. You're not going to say one thing and then they're just going to give you a prescription. There's a back and forth. and that's the way that's the way the world works there's sort of you need to absorb context and understand things um but with ai we've sort of said no don't worry about it you can just ask anything and it will give you a response as we've seen um and and i think that's some that's an an expectation we're going to need to shift we're going to need the models to be more aware about when they need more context and when they need more information um in order for them to be responsible and ultimately drive good outcomes.

49:42The piece that makes it particularly scary is that may be at odds with engagement, right? Because if I go to this AI and I ask it a question and then it asks me another question, I'm just going to go to this one that's going to give me the answer right away. And so that's a little bit of a tricky, that's what makes it tricky. Yeah.

49:59Robbie Goldfarb:I mean, the whole sycopency disaster was driven by a guy tweaking the algorithms to increase engagement. Yeah. And it created all sorts of problems. Who are the customers for you guys? All the usual suspects. So we work with the big AI labs, as well as the large system integrators, like the larger consultancies that are deploying the models through to other companies. um so mostly them we'll work do a little bit of work with some smaller labs um here and there but that's the bulk of what we where we focus yeah uh and and are you um not to get into the business side too much but are you a subscription model or pay usage based uh fees or yeah yeah we both.

51:04It depends on what the service is. So what we'll often do with clients is we'll have a baseline subscription where once a month or once a quarter, we'll do an evaluation. So let's take political topics, for example, where we'll do a comprehensive evaluation where we'll run all our judges on the model. We'll actually have experts come in and manually review the results. So you just have this continued pulse on how your model is evolving and a pulse on any potential risks. and then on top of that we sell more bespoke or a la carte services the way that would usually work is and I'm making this up as an example now but we might run an evaluation a quarterly evaluation we might identify that you know the model is particularly weak in source selection when talking about breaking news related to foreign affairs we may then engage in developing a benchmark specifically focused on that one problem um and that would be more an ad hoc service

52:04Robbie Goldfarb:yeah uh and we haven't talked about this but you know i spent a lot of my life in china and and so i follow what's happening in china uh fairly closely uh you know and they have a the the chinese government certainly has a very different world view but the people as well have a different cultural view from that of the United States. And there are all these sovereign AI models of being developed in various countries around the world that are trained on local training data. Do you think your system's judgment is objective enough and general enough that it could apply to models from other countries?

52:58Yeah, certainly. I mean, we've run, and this is something we're actually hoping to publish at some point in the near future, are the evaluations we've run on the open source models, particularly the ones coming out of China. And I would say definitely, you can capture a lot of interesting things there. I think the other thing there, yeah, on the, I guess the national security or foreign interference piece, that there's two things. There's the models themselves and their training data, like you alluded to. But there's also news sources to the conversation about retrieval we're talking about. And so the sources they're using, if you're using state-sponsored sources, that can also have an influence on the models as well, which is why it's so important that we look at source selection as part of our evaluation.

53:49So I will let you know because we're going to publish something there.

53:53Robbie Goldfarb:yeah yeah uh because these models uh just you know you were at facebook and instagram incredibly powerful in shaping public opinion yeah and you know depending on the trading data depending on the fine tuning uh you know you could have a model that reinforces the government's position on any particular topic. And, you know, a lot of people worry, given the closeness between the big tech leaders and the current administration, that, you know, that there may be some impact on how the models respond to certain questions. Do you worry about that? I mean, so to some extent, but I would say, and this goes back to what you're talking about before on transparency, right?

54:59I think at the end of the day, what people need to know is who is behind training these models. And part of the value that we can offer at Forum AI is we can give you a very clear picture of who those people are. And then it's ultimately on you as a user to evaluate whether you don't feel good about this. And I think certainly in our case, we have a very diverse and reputable group of people that I think people will feel much better about the models. And they'll build that trust if they see these individuals involved. But I think transparency is the only answer here, right? Because you may have one opinion, someone else will have a different opinion.

55:42And so we just have to be candid about how these are being built and then allow people to make the decisions from there.

55:49Robbie Goldfarb:yeah uh in in the notes that were sent over before the call uh they mentioned delphi which is a market of for evaluating uh models but it's as i understand it it's it's kind of uh you're betting on one model versus another and there's some financial incentive if oh well so there's there's two different things here there's like um so delphi um is interesting in that what what they're trying to they're essentially trying to do the steve the thing that steve jobs said in 1985 they're trying to be able to you know take craig smith and put i think they use the term a brain in a box put your or a digital brain in a box and then i can go and i can have a chat with you um even though of course it's not you um and so conceptually i think that's very interesting there's another company called super me that's doing something similar giving people the ability to clone themselves into an ai our approach is very different as we described in that we are not focused on cloning our experts so you can have a conversation with them we're focused on cloning them for a very specific use case so you know evaluating the factual accuracy of a claim, you know, a political claim, for example.

57:14Now, the advantage we have in focusing is that we can do this accurately, right? We can actually measure this. We can build these judges. We can have them, you know, evaluate a thousand things. We can have experts do it. And we can tell you, you know, this is 95 % accurate to what experts would say. If you're going to just have a conversation with an LLM, you know, you're not really talking to LeBron James and it's probably not 95 % similar to the conversation you'd have with him. So it's just a very different model. Of course, very different use cases. We're doing evaluation. For them, it's just a different user experience and a different purpose.

57:52Robbie Goldfarb:Yeah. But you said early on that you're not a leaderboard. Do you make your evaluations public? Yeah. And this is where there's a comparison to LM Arena, who are, they are a leaderboard, and their experts, the people they rely on quite heavily is just the general public. The challenge there is it's created this weird incentive system where essentially what the general public defines as good is usually what's most engaging. And what's most engaging, I can tell you as someone who worked at Facebook and Instagram, what's most engaging is not always the best thing or the safest or most trustworthy thing.

58:33And so that's the issue there. For us, we don't make our evaluations public. We are thinking about potentially doing some, you know, releasing some public facing leaderboards in certain specific areas, like maybe for open source models or for certain areas of more frontier type use cases. But for the most part, you know, our goal is to work with these labs and work with our clients to make their models as good as possible. Yeah.

59:03Robbie Goldfarb:You know, I had a conversation with max tech mark at mit uh during a recent conference and he has a uh i've forgotten what he calls it but it's a scorecard for the major model producers yeah and he scores them on on it or gives them a grade on various aspects and his uh hope is that this will eventually become you know that that a standard that people you know they they want to get a passing grade or a plus grade and all of the different areas that they're being evaluated as a as a product validation and so that's one of the things we're thinking through is how what is the responsible way to potentially release some of our benchmarks so that people can see these scores and use that for their own decisions, but without hurting the integrity of the benchmark.

1:00:09Yeah.

1:00:10Robbie Goldfarb:Yeah. And I would imagine there's also a little bit of a conflict of interest if companies are paying you and then you give them a crummy score publicly. Yeah, I think, I mean, the thing I found, because I mean, of course, that is a real tension. but remember with us it's not me and Campbell giving anyone a crummy score yeah yeah it's ultimately it's our experts right and that's a very defensible way to have a lot of these conversations um and I think a way that everyone understands um so but yeah that's fair yeah and in your mind you have uh you don't have to say it openly but you have uh certain models that that you think do a better job at being truth-seeking than others?

1:01:07Yeah, it's interesting you use the word truth-seeking. The government just in December released some guidance on the way they look at neutrality, and they say truth-seeking and ideological neutrality. So we're spending a lot of time thinking about those factors. I mean, no, I don't. it depends like there's not no model is better or worse it's so that's one thing i have learned maybe this is the most the best answer to your question i think one thing i've learned having now run so many evaluations across so many different systems is it's often more nuanced than people think um even just remember the factors that the three areas we were talking about earlier or there's different domains like foreign affairs versus domestic policy and different types of users.

1:01:59And then there's different dimensions. Is it the tone versus the bias? And so different ones are better and worse at different things.

1:02:07Robbie Goldfarb:Yeah. Okay. Well, I think we're over an hour. Let's leave it there. If someone wants to follow up and learn more, where should they go and who Who should they talk to? Yeah, I mean, feel free to. So first of all, definitely can reach out to me so you can find me on LinkedIn at Robbie Goldfarb or on X under the same name. And feel free to go to our website by forum.com. We have a there's a link there where you can request a call and we will get back to you ASAP. would love to have conversations with whether you're developing AI systems and you'd want to work together. If you're an expert and interested in being part of the network, or just someone who wants to nerd out on some of this stuff, we would love to chat.

1:03:00Yeah.

1:03:01Robbie Goldfarb:And give me that URL, spell that out for me. Yeah. B-Y-F-O-R-U-M by forum.com. Okay. Great, Robbie. It was really fascinating and uh yeah let's have a conversation again and you know a few uh a few months or in a year or so so you see how it's working yeah i would love to yeah no really this is a lot of fun i appreciate it correct yeah