Brainfood Live On Air - Ep412 - How to Assess for Human Judgement

0:09Hello everyone and welcome to Brain Food Live on Air. It is episode 412 bringing it to you every Friday no fail. and today we've got a very exciting conversation for you and to be honest with you it's a bit of a pet kind of thing for me which is this idea that you know where we're seeing with artificial intelligence in my kind of perception is that AI is kind of starting to inhabit spaces progressively inhabit spaces that human beings then are forced to vacate. So I kind of feel like there's an invasive approach by AI. Like it's kind of moving into domains that we think that actually, you know, shouldn't this be stuff that we're doing?

1:01And we persuade ourselves, having vacated these spaces and occupied new higher ground, that we're safe in that higher ground. And we say things like, you know what, that's fine, leave that stuff to AI. We're in this new place now, which is totally defensible against artificial intelligence. But I don't think that we are entirely clear on why that is. And I think it would be very useful for us to be clearer on it. Because AI has a discrete factor, which is very difficult for us to avoid. And one of the areas that we've now gone toward is this idea of human judgment. You know, have you heard the phrase, folks, artificial intelligence, AI is going to deal with all the processing, but the humans will always be in charge of doing that human judgment bit.

1:49How many tech CEOs have you kind of heard say, oh, don't worry, we're going to, human judgment is the most important sort of characteristic for our future human hires. All sounds really good. However, it begs the question, what is human judgment, folks? And, you know, can we identify that? Is it domain specific? Is it something we can train? Is it something that we can get better at? Is it something we can assess for? And if so, how? How do we assess for human judgment, folks? We've had an earlier conversation on this particular topic, but I think it's worth extension today, particularly as we are now kind of rounding on this story of human judgment being the thing that we care about.

2:34But I'm not sure on all of the questions I've asked sort of just now about it. And I'm also not sure about whether we're confident about what the implications are if we do think that human judgment is the most important thing that we now hire for. Because it strikes me that we're going to have to basically reverse a a lot of the ideological positions that HR and TA have been basically defending and building a case for for the last 20 years or more. Anyway, end of monologue. That is basically what we're talking about today. So if you're interested in all of that chat, you're in the right place.

3:10But before we get on with any further in this, let me just confirm actually the audio and visual is okay. I should have asked that question. People watching this on Crowdcast, let me know whether the audio visual is fine. You should be able to see me in a somewhat grainy kind of image because this is my mum's kitchen rather than the proper studio. Let me know where the audio visual is fine. I should be broadcasting this on LinkedIn as well. Apologies last week I didn't do this. I have to hear why it didn't work. I'm just blaming technology over which I have no control, of course. Let me know where the audio visual is fine on LinkedIn.

3:44I don't think it is actually. I don't think I am on LinkedIn. So that's another reason why I should not schedule these streams well in advance because I think part of the problem might be that if I schedule them in a month ahead of time, generally speaking, it doesn't fire off for whatever reason, which is super annoying. Anyway, we're good on Crowdcast, which is good to know. The captions are on. So thank you, everyone, for being able to see all of this. Thank you, Ivan. Thank you, Jess. Thank you, Gavin. Thank you, Anastasia. Okay, good stuff. We're going to get on with the chat in a moment.

4:21But before we sort of carry on with it, I of course have to thank our sponsors for supporting Brave Food Live every week. They do it every week. Technology companies that care about recruitment and the future recruitment step in and say, you know what, these conversations have to continue. and today's sponsor is one of our favorites. It is, of course, Greenhouse, one of the leading ATSs on the market and have been for the last decade or more. They've got tons of competition, but they are responding with some fantastic innovations, which I think are probably more radical than we realize. So keep an eye on what's happening on Greenhouse if you're not already a user.

5:01And if you are, do dive into some of the new products and features that they're releasing. They've got a new thing coming out, as a matter of fact, which is AI voice interviewing. We're going to start seeing a lot more of this. Essentially, AI sort of ATS, should I say, starting to build really quite capable functions that would otherwise be taken by best of brief point solutions, whether they're schedulers, screeners, CV resume screeners, AI interviewers now, all coming into the same platform. So they are just out. So yeah, check them out at greenhouse.com. Rob is the only man who's live streaming.

5:44So well done, Rob. At least one person's got it right. I include myself in failing to do that. But that's all well and good. All right, good. I'm going to bring on our guests straight away because I think I've assembled a panel here, which is at an intellectual level that we've never yet seen before. It's really quite fantastic. And I don't want to take any more of the bandwidth because I really want to get people listening to our guests here because they're a fabulous bunch of people that have proven their capability more than a few times. Each one of these, I think, deserves time on screen, more than they've had so far.

6:29But let me just get them all on together. We can all join this conversation. So Richard and Joseph and Jess, you should have all received an invitation. I just need you to toggle the camera on and the mic on, and then I'll push you up onto screen. So let me know when you've done all of that, and then we're in good shape. Okay, there's Richard. He's first to go, so let's get him on screen. Hello, Richard. How are you? I'm good. How are you? that very well thank you and very nice shirt by the way i might be joining you in singapore in a couple of weeks you know i'm around so we'll go out for a drink just totally clocked i totally clocked at you there i'm in hong kong now i'm gonna go back to the uk then i'm gonna go back to apac and on the way back on the second time i might well drop up drop by and sing so um if i do that we'll i will do an event or something yeah definitely good stuff and there she is i tell you The world's not ready for Jess Von Bank, but they're going to get a bit of Jess Von Bank today.

7:34Jess, great to see you. Thanks for dropping into the call. I was afraid you were going to have an all-dude panel. Do you know what? That's okay. Sometimes that happens. Sometimes it happens. Sometimes that happens. That's fine. But great to have you on the call, Jess. I know it's early for you, so I appreciate you jumping on. I've got Joseph here as well. also East Coast, I do believe, very early for you guys as well. So thank you very much. Okay, before we say, let's do some intros properly. So Richard, why don't you quickly introduce yourself to the audience? Who are you? What it is you do? So I basically do leadership development nowadays.

8:14I'm an organisational theorist by education. But nowadays I do leadership development and all elements related to that. And as you can see, I'm doing this interview in my bedroom. That's totally okay. I think bedroom's okay. I think bathroom is probably a little bit dodgy. I probably would like that. That's it. This is where we need to get some sort of filter going on. But I think we're okay. Jess, brilliant to have you on the show. Can you quickly introduce yourself? Who are you? What it is you do? Well, I want a new title. I want to be an organizational theorist. That sounds so cool. I'm going to come up with a new title for myself.

8:55I am happy to be here. Nice to see everybody. And yes, I was a little bit of a jet setter this week. It is early-ish in Minneapolis, but I almost joined you from Las Vegas. So all good. All good. But I'm Jess Von Bank. I'm the co-founder of Now to Next, which is an AI transformation firm. We're helping organizations get more value out of their AI transformation initiatives by starting with people. So I love the topic of your conversation today. And I'm based in Minneapolis, but I just came from SAP Connect in Las Vegas. Lots of news coming out of those folks this week as well. Yeah, very exciting stuff as well.

9:35And it was actually your co-founder, Jason Averbrook, which inspired this topic, actually, because he was talking about human judgment. And then he bailed. He bailed. Well, I'm here so that he can be with the conference board in DC. So all good. You got an upgrade, I think, didn't you? We can agree, just from banks, absolutely an upgrade on Jason Aberbrook. We know that. Everyone knows this. Yes. Okay. We have Jason, sorry, Joseph Braithwaite here as well. Brilliant to get you on screen, Joseph. I know you've been brilliant at commentating and just adding value massively on lots of the conversations you've had online.

10:16So wonderful to have a proper one-to-one, sort of a live conversation with you, should I say. Can you quickly introduce yourself? Who are you? What it is you do? Yeah, so Joseph Braithwaite. I am a business consultant focused around leadership development, engagement, and obviously the HR side of the world in how it applies to moving business forward and creating real value. I started a company a while back and developed a tool that looks at the capabilities of an ask in the job description and aligns it with the capabilities that someone's defining within their resume. So taking a deeper level from the keyword searching and actually aligning.

11:03And I use your wonderful tool that you love, the spider diagram, to show the overlay of the two aspects. So, yeah, it's something I've just rolled out and looking to move that forward, too. Yeah, fantastic stuff, Joseph. Great to see sort of, I mean, I guess one of the features of the world today is that ideas can suddenly turn into applications as well now. And that's going to really be, I think, a very empowering sort of experience for all of us. It will be a huge shakeup for just tool use and what have you. So brilliant stuff. But let's talk about this topic, human judgment. I'm one of these that's a bit of a critic of, you know, the latest thing that happens in TA and HR a lot.

11:50I think the iteration before this human judgment thing rolled around was human in the loop. And we kind of just accepted human in the loop. We said, hey, that's fine. Don't worry about AI, folks. The human is in the loop. And then we sort of realized, what the heck does that mean, really? you know what is a human doing if the human is just accepting the AI recommendation like are they are they even in a position to understand how to veto something what do we do with the human veto like do we just accept the human veto without question and then we kind of moved on to this concept of human judgment and again we're not challenging or going deeper than the phrase to say you know what human judgment is fine you can use AI because human judgments is the most important thing that we care about, but how we examine what human judgment is and do all humans deserve the same equal judgment capability, for instance?

12:44Is it sufficient to have any human judging the outcome? Or do we need some sort of expertise in there, which suddenly becomes, okay, a capability conversation? And have we even applied any of that thinking around it? So, So, Richard, going to you first, what's your take? The first question is, what is human judgment? I know we've kind of knocked this around a little bit before, but I wanted to get your view on it because I thought, you know, you've got you put some thought into this before. How would you define human judgment? Is it even possible to do that in a way that makes sense to normal?

13:17As you know, I think it's domain specific. So we can judge reasonably well in domains that we have expertise in, but I would argue is multidimensional expertise. So I've read more than one perspective on the subject I'm judging on. I've read multiple perspectives and I can juggle them in my head and kind of say yes. but just because I'm good at judging in one domain doesn't mean I'm good at judging in another domain so I could maybe hire a candidate really nicely but I could maybe end up with a very terrible wife different domains different decisions so you know you need to be you need to really build up an area of expertise and judge it but it does have to be multi-dimensional if if in my field quite often you have leaders who have read one book about leadership and they think that's given them all the answers and they make lots of terrible decisions because they're just basing it on one book if you um but then if you read 10 leadership books it tells you 10 contradictory things and i think that's one of the real elements of judgment is is there are always contradictions in a judgment and part of what we do is kind of wrestling with these contradictions and taking a stance on I think this because and having enough knowledge behind it to stand by that stance.

14:43So that's where I would go with the answer to that question. All right, very interesting. So basically what we're saying is it has to be controversial, right? So there's certain things that don't require judgment. For instance, two plus two equals five. I'm not exercising equals four, should I say. I'm not exercising judgment by concluding it's four. It's simply an equation. There's no ambiguity there. But where there's ambiguity, AI is going to destroy the world. AI is not going to destroy the world. I've got to make a judgment on it because we don't actually know the facts. So maybe future facing has not been proven.

15:20There's still new information to acquire. We've just got to kind of make a guess. So judgment is not kind of related to guesswork it's not conclusive um and it's the ability to wrestle with lots of different ideas as you say contradictions and then i guess kind of weigh in your mind like which which which kind of uh argument um is stronger in in the balance of things in this context that you have and i take your point with regards to domain expertise as well you wouldn't want someone who doesn't know how to pick a horse to pick a horse. You're not someone who's been to the bookies to be able to do that.

15:58That seems to me the difference between random guesswork and actually making a judgment. There needs to be depth of knowledge there. Jess Von Bang, I love it when you cock an eyebrow. That tells me there's trouble coming. Trouble coming down the pipe from Jess Von Bang. So let's hear it. What's your thoughts on human judgment? How would you define it? How would you agree and disagree with what Richard just said that. Yeah. So a little bit of context, I have three teenage daughters, 17, 15, and 14. And I think about this a lot because I think AI is not necessarily... This might be... I don't think AI makes us smarter.

16:42I think it's really good at making us look smarter. And so I worry a lot about the distance from reality. And when it comes to judgment, I think it's, to Richard's point, it's what you apply when the rules run out. And models are really good at applying rules with consistency. And so when you don't have a good or perfect answer, which I think is also hard to judge these days, how do you then apply judgment? And I think it does rely on having lived in that world, in that environment. You're relying on expertise you might have gained over time. And so I do worry when it comes to young people like my kids or early career professionals, I worry about the distance from reality.

17:33If you've never done the reps, if you've never lived in that domain or had that experience, what are you calling upon? What are you hearkening to say when the rules run out? This is where I would go to apply good judgment. I think we're missing sort of the set of criteria that would say what good or bad looks like or how much risk we're willing to apply to a decision. I love that sort of Pandora's box. You're kind of sneaking open, Jess, because what do we do? as you say maybe judgment is only a thing that happens for people pre-ai post-ai right so we've kind of grown up in this world where we're not leaning heavily on ai everyone on the screen i think knows the world pre-ai but your kids growing up and kids younger than your youngest one they don't know they'll never know a world without this oracle telling them the truth and i just wonder whether that ends up eroding their instinct to exercise judgment because it's just telling me what's true, isn't it?

18:35I mean, it's literally very, very, you know, I can't argue against what it's come out with. It's much more convincing than anything I can come up with. I'm just going to accept it. Yeah. And it's so, I mean, the problem is that AI, they think we're hilarious, by the way, for talking about AI all the time. They don't know an environment, a digital environment that doesn't include it. It's just sort of there. And so also your ability to detect, like it's so pervasive. Your comment in the beginning, Hong, about, I think about in Minnesota, we have a ton of lakes and you can't go from one lake to another with your boat without having it inspected for invasive species.

19:16In their worlds and in their digital environments, it's omnipresent. So even their ability to detect bullshit becomes more important moving forward. Where did this come from? And is it real? And is it actually true? Like even that will require a lot of human judgment moving forward. It's very, very interesting. And you know what, we've got to, we have to create some sort of safe space that enables the Jeffs von Bank kids to come on. I want to hear from kids, you know, I want to, we need to talk, we need to learn from younger people about what they're thinking about the world, because they're actually got a perspective that people on webinars literally have a very distant away from apart from your parents uh like yourself jess that have got that kind of degree of intimacy obviously you're in-house but at the same time it's it's it's narrow in a sense it's your family and that's it so let's yeah i'd love to find a way where we can get teenagers on and then they can tell us how dumb we are um i'd love to have that okay really interesting joseph your thoughts on human judgment you've heard from jess and richard How do you interpret human judgment?

20:26Why would you agree and disagree with the two folks that have just spoken about it? I think on the aspect of judgment, we don't know whether it's good judgment or bad judgment until history tells us. So I don't know whether the decision I made was good or bad until I look back at it and go, yeah, that was the right decision to take that education or do this thing. And sometimes that's years or decades down the road. So when I look at AI and to Jess's point, we don't know necessarily what's bullshit and what isn't, but we have to have the insight and the knowledge in ourselves to be able to read that and understand it and go, well, I'm going to challenge that thinking or whatever we want to call what AI is giving us.

21:10We need to have the ability to challenge that. And the more educated you are, the more you've been in places, the more scars you have from getting things right and getting things wrong. That is where we gain the ability to go, I see what you've given me, ChatGPT, but I don't necessarily agree. And here's why I don't agree. And it'll always come back and say, yes, you're right. Here's the blah, blah, blah, whatever the rest of the response is. So it's always going to be patting me on the back and gaslighting me as far as it can take it. So our personal judgment needs to be, how do we take what we know and how do we apply it as we move things forward?

21:50And if you look at stuff like the kids today and AI, a lot of them don't know anything other than computers. They weren't around with DOS 3.0. They don't understand what a rotary phone is. But that's evolution. We evolved from those things into where we are today. And they will recognize AI of today will not be what they're dealing with in AI in 10 years or 20 years down the road, if it's still AI or whatever it ends up, you know, morphing into. So I think the judgment that we need to look at is, are we giving people the opportunity to gain the scars they need to be able to be informed enough to question what they're seeing?

22:31And that's what education is going through now. They're going through that whole challenge of kids using ChatGPT to write their assignments. Well, they're not getting the knowledge that it takes to think outside of the box to understand what's actually going on. And if we continue to just rely on AI and what it's giving us, AI is giving us an opinion, right? If it's not scientific fact, it's giving us an opinion of the person who trained it on what they believe that right answer should be, right? So now we're taking an opinion times it by 500 people who gave it the same opinion that it's learned.

23:05All of the bias is included, but it's still an opinion. Well, we haven't formed our own because we're relying on AI to give it to us, right? And I think that's where the challenge going forward is. And it's in a couple of your things, your post. We're not giving the new recruiters the cycles to gain those scars. We're not giving them the cycles to learn what their opinion, the direction their opinion should be taking them. And we're not giving them those cycles because we're not hiring them anywhere. We're using AI to do all of these things. Well, in 10 years, nobody's going to have the skill set and the knowledge to be able to make the opinion other than using the tools that are there.

23:46And that's actually a crisis I think many disciplines are aware of. And not software engineers are definitely aware of this, where they're starting to say senior folks there are starting to say, look, we've got basically a five year cliff, five to 10 year cliff before we get slapped with this. because we're under pressure to hire individual ICs, senior ICs that know what they're doing, that have pre and post experience with AI, no junior hiring, no point with this. And then everyone's thinking, okay, what happens when everyone ages out? Or what happens when these ICs decide, you know what, I'm going to do my own business.

24:20I'm done with this. We haven't trained a talent pool to go in and do basically the apprenticeships. and in fact it's hard to imagine people doing that like can you imagine um if you like john blastelika talks about this can you imagine an organization getting away with sub-optimizing efficiency in order to train their staff to do it the hard way i can't see that happening guys generally it's like oh uh you can use ai to be like 10 000 percent more effective but i don't want you to do that. I want you to use the old way. And then, you know, we're going to get read the benefit in 18 months when you get skilled.

25:01It's like no organization is going to say that. I think that's that comes down. I mean, a financial standpoint, you're talking about a fiduciary responsibility is to make sure you have the long term outlook of the company in place. Not training your people is a short term outlook to keep your efficiency super high. Today, A, you're going to have the impact in 10 years and five years and four years. You see the consulting firms who aren't training anybody. These guys don't know how to think intelligently about what they're seeing. They're not able to apply what they've learned or the concepts that are going to drive real innovation and help companies out.

25:38They're just applying the slop that you're getting from five different models and throwing them together and calling it done. Like there's a gap there. And I think, although I agree, the efficiency needs to be there. you need to have a stream that says we still need to train the people to do the right things. I agree with you in the sense, yes, this is great, Joseph. But I'm just a skeptic of the system because aren't CEOs like literally on quarterly KPIs? Like the entire KPI sort of mantra is to say, look, where's my productivity? Where's my efficiency gain? Where's my profit margin? AI can make all of this go northeast, you know?

26:19How are you going to fix that? The only way you fix it, if you're an owner manager of a business, just from bank. I'm dying. We have the wrong KPIs. If all you're chasing is efficiency and productivity, welcome to your demise. Like you're just counting the days until your organization is going to die because you will die a death by a thousand cuts that way. So Randy Bailey already mentioned Vivian Ming in the comments. She has a theory. She's done a small body of research. I think we are going to see more and more of this research. She calls it her cyborg theory. Humans doing human work alone, AI doing AI work alone, both produce what they produce.

26:59But humans in well-designed environments working alongside well-designed AI produce higher quality output when working together. And I think that's what organizations... Unfortunately, you need to find a way to play the long game. You need to find a way to chase quality in the long game, not just quarterly efficiency KPIs, because that's the only thing that's going to keep your company alive. So if you're breaking your own talent pipeline and if you're removing all of the reps, you're removing all of the judgment, you're creating zero future leaders for your organization and you like welcome to your own demise.

27:40There's no way to survive that way. I again I agree with Jess but again I'm pointing I feel like I feel like oh the street is tough on companies that need to show something this week this quarter but I just don't think it's a long game right I'm reminded of the like the cobbler you know the artisanal cobbler making the shoe I mean I'm sure there's folks there that literally resisted mass mass production said you know what That's not how you make a shoe. You make a shoe like this. And most of those companies went out of business. Like the only few just kind of fluked it that stayed as an artisanal business all the way through the industry.

28:21And today they're probably making shoes for like$18 ,000 a pair or whatever. And they're great. But the numbers of organizations that have survived doing that are absolutely infinitesimal. Everyone wears like$150 trainer now. A mass produced in a factory at the cost of like$2. to make. So I'm a skeptic that the will is going to be enough to stop the

28:48remorseless logic of the system that we're all locked into. Like, how do we, I mean, that's a bigger topic. I mean, we're talking about political sort of systems, economic systems, and maybe they all fade away. I had a really interesting, I listened to a very interesting talk in Sarajevo a week ago, and this lady, Mima, was talking about how the political, economic, social systems have all emerged during the industrial revolution that we're used to now. If we're now getting an intelligence revolution, maybe new systems will emerge that will be radically different. I think that probably is the case.

29:24But in the current system, we've got, I don't see a spinning out of it. Okay, let's move on to the next point, which is, we care about human judgment, we know it's about reps. And we want to, I guess, recruit for people who exercise good human judgment, don't we? Right? Don't we want that? We want it, right? So how do we do that, folks? Can we identify good human judgment? And what are the signals for this? Anybody? Well, I'll cycle back to something Jess said earlier on is that one of the challenges that we have in trying to recruit for good human judgment is that people using AI seem to be far more considered and articulate than they actually are.

30:13So what we're seeing in the broader leadership stuff, again, just to move into the domain I have some level of judgment about, we are seeing this kind of idea that leaders are getting a whole bunch of articulate that considered reports coming up at them at a speed they've never experienced before, really sophisticated. But the people who do it, who've written them, don't really understand what they've written because AI has done most of the carrying. And these leaders are looking at this going, well, OK, now I've got to read five times the amount that I used to read and all of it's got errors in.

30:52And all of this I'm accountable and responsible for. And I've got to get this out the door by Friday. And I do that by Friday. And then next Monday, I get exactly the same stuff coming in. And I would see your challenge in recruitment as exactly the same. Everything that's coming down your pipelines looks considered an articulate because everyone's using AI at both ends of the tunnel, to set up the job specs and then to answer the job specs. And then nowadays, even if you're getting AI interviews, you're not even getting people talking to each other. You've got exactly the same challenge that we have in the leadership space.

31:33And again, to sort of cycle back to what Joseph and Jess said, I mean, three or four years ago, it was only 11 percent of CHROs that said we've got a leadership bench. And that's getting and certainly in the work I was doing, I was working with companies who had a leadership bench. but even they are now kind of saying there's nobody coming coming up that we can trust to do this complexity of work because they don't have the judge we don't trust that they have the judgment because they haven't done the hard yards and and a final point kind of related but not related to that question but what you said we're talking about before um there's some really interesting research saying that ai increases productivity to task productivity by about 200 so instead of doing one task you can do 200 in the same amount of time but it increases the rework so the judgment work as to what's correct and what needs to be fixed before it goes out the door 860 percent um and in my industry to to to talk to your cynicism um we are now looking at organizations having to ramp up the development budget because they're saying we just do not have people who can cope with this level of complexity um so so lots of the big tech firms and the big um the big not the not the the big four but the kind of the tier one consultancies are kind of saying we just do we just have to develop people we have to properly develop people again so there's kind of light at the end of the tunnel that way but nobody really knows how to do it so i think it kind of talks to your to your questions we recognize that judgment needs to be recognized but we can't trust what we're hearing and seeing because we can't trust what's coming up the pipeline we don't actually know how to test um now there are potentials to design things using ai to test but we don't really know how to do that yet so we're all kind of walking to stand still at the moment so i think i think it's a billion dollar question because i think there are answers um whether organizations will i mean certainly in my industry i've had some of the most interesting contracts ever come through because they're kind of going shit we need people to be able to do this level of work uh whereas before i had hr saying oh can you teach someone to be authentic come on guys so where's the business value in that um now it's like okay get them to be able to do this level of work so there's there's really interesting stuff coming in in my element of what you're doing because they're recognizing that they have to have people with a complexity of thinking to do this level of work and to short to sort out the wheat from the chaff of what's coming down the pipeline so i i'm not sure i answered your question but i was trying to throw in a whole bunch of extra stuff to play with no you did it you did and you didn't um because you added lots more that we want to dig into in a minute rich but one of the things that you that struck home with me was that the the the volume of output that everyone is being able to generate now is basically disguising our ability to discern any sort of signal.

34:44And therefore, we can't actually detect judgment or good judgment via CV. I don't think anybody can do that on an application, right? Do we agree? Like we can't see whether someone's got good judgment from a document. We're going to have to interview the person, I guess. But that's obviously not scalable unless we start thinking about, okay, can AI interview for human judgment? Which I know it sounds completely paradoxical, but is that maybe the only solution in a very weird way? We get super sophisticated AI interviewers to interview human beings and they're able to come out with a judgment score.

35:28What are our thoughts on that? Joseph and Jess, any contributions to that idea? But I think I've got a bit of a challenge in and Richard raised it in the document that he shared with us earlier in in that. What are we actually looking at to decide whether someone is capable of doing the job or not? We've got Richard just said, everybody's asking where are the leaders? We've got nobody with the leadership skills coming through, but we're also getting the AI sameness in everybody's resume. Right. So so we've told everybody write your resume all the exact same way by using the tools. We cry about it, but we want it done because we want fast resumes and we turn over jobs in five days because we've got a thousand resumes.

36:16But out of those thousand resumes, how many are actually good? Right. How many actually define what the person can actually do as opposed to shoehorning in a bunch of keywords that are going to allow the ATS to say, you're wonderful. Here you go and rank you high enough that someone actually calls you. Right. So so I know Greenhouse is is your go to and and they're sponsoring this. But I think that's the big challenge with the ATS is they pre filter what the recruiter actually gets to see. Now the recruiter is looking at a pre filtered version of who has all of these capabilities and all of these skills and all of these different things that the company wants.

36:53But that pre filtered is already biased. It's already Harvard Business Review and multiple people have done the research to show that you're skipping out the people who don't fit within the box that the ATS has created. Right. So now you've got all of these skilled people sitting on the fence for months and years who have the skill to do the work. We're crying. We don't have it. Yet we're still using the same systems to say these people are not there. We need to find a better way. Well, the better way is there. We just need to stop looking and saying what the ATS gave us is the starting point.

37:26The starting point is looking at all the resumes and understanding what they are. ATS should go back to what it was, which is helping you keep track of your files, keep track of your job descriptions, keep track of your resumes that come in. And somebody with judgment needs to be able to look through those resumes and say, the ATS didn't find this when I think this person's a great candidate. And the tools are out there to help. You know, I love that, Joseph, but I know people are going to hate it because I'll summarize your very well argued case with my kind of crude, triggering language. You're basically saying, you know what, a human recruiter might just be able to just have a good feel about this great candidate, which I'm pro good feel.

38:11Absolutely 100 % pro good feel. But what else are you bringing to the table? What is human judgment? I think it's very related to gut feel, isn't it? Joseph, come on. Well, I think the gut feel comes with the eight to 10 second review and the eight to 10 second review that recruiters say they look at every resume they get and they give it eight to 10 seconds. Well, you're looking at logos, you're looking at titles, right? Who did they work for? What title did you have? And that's all you're looking for. And that's another bias towards the recruiter. What I'm suggesting is that there are out of the thousand resumes that you've got, and you're giving your shortlist to the hiring manager and saying, here's the best of what we've got.

38:52Is that true? Is this really the best of the resumes that you got? So the hiring manager is taking this pre-filtered list that the recruiter went through and said, yes, these look like they are capable of doing the job. Go back to Richard's point. What is the actual job they're doing? They got five different things from five different domains that they need done. And this recruiter, who's only worked in one domain, is trying to determine whether this person's actually capable of doing these other jobs. Right. So it's a mishmash of all of these things coming together. And now the hiring manager is going, well, if this is the best I've got, I have to make a decision from this group.

39:27But that's not the best you've got. Right. The way we've set the tools up, the human interaction setting the greenhouse tools up and the ATS tools up has created the internal biases. It's created the keywords. It's created all of these different things to help make their job easier. But the end result isn't better result. Do we ever go back six months down the road and say, we hired that person. Did we make the right call? Yes, they can do the job. But were they the right one out of the group of people we had? We have no way to manage that and look at that. But the recruiter gets to hand that off.

40:02Someone makes the decision in the call. I did my job. Like I gave you the list. You made the call. But you gave me a flawed list to start from. And I got a flawed list to start from, from the tools. Right? So there's a whole gap there that's creating a muddled environment that we're not getting the people we need to be able to out there do the job. We're not getting the cycles to give people the skills to do it. I get what you're saying, Joseph. So basically the pool that we're drinking from is muddy. And the ideal case is that we clean the pool before we drink from it. I get it. My own, I don't disagree.

40:38I just think that's too hard to do. And I think what we need to do is actually build a filter into our throats before we ingest it. So we have to accept we're drinking from a dirty pool, I guess. But I want to bring Jess in on this because we want to talk about how feasible it is to assess for human judgment in any way, but in a scalable way as the second question. And I've thought, you know what, firstly, I don't think it is possible, but I don't think it's scalable. I think it needs an interview. What are your thoughts? Yeah, to Debbie's point, how scale is it? So the irony of this conversation is we're talking about assessing for judgment.

41:18The assessing we're talking about, to Joseph's point just now, is a judgment. Like interviewing itself is judgment. There's no more judgy occupation that I know of than deciding whether or not to hire somebody, deciding whether or not they're good at something. That is the ultimate judgment call of all time, as every recruiter knows. People just said in the comments, quite right. Human judgment comes with bias, 100 % correct. I just don't think we have options. Yeah, back to what I said in the very beginning, judgment is what you apply when the rules run out. And the more room, so the irony of this conversation is the more room you leave for judgment, instinct, gut feel, the more opportunity you create for bias to be present.

42:08So you could say the best way to eliminate bias in a hiring and screening process is to apply more rules and rigor so that you can't dip into sort of human judgment territory where. But anybody who's sat in a recruiter seat knows that sometimes it really does come down to a gut call. If I'm assessing talent and you're all equal on paper and I've applied all the rules and rigor, you either have this or you don't have this. You have this degree or this certification. You've been in this environment. You've done the reps in this type of work or you haven't. So I've applied all the rules and rigor I have to be as bias free as I can in my assessment of you.

42:50But because I know who thrives in this role, or I know this hiring manager or this location, or I know that there's something I know that tells me this is a tough gig, or this hiring manager is... So there's something that I know or have that tips a candidate in the right direction or the wrong direction, because there's something I know. And maybe that's hard to put a pin in. And hopefully I'm relying on something like that and not just I'm reading into your name or where you graduated from or your profile pic. Like that's the biased part, right? Or applying no judgment at all is probably the worst thing.

43:33So I think what we have to get down to is where is if you've ever heard Ethan Mollick talk about dark factories, in other words, fully automated, fully robotic. There's some production piece of work or assembly line where there are no humans required in that environment. So why do we have lights on? Who needs to see anything? This is a fully robotic process, dark factories. What most organizations are getting to, or teams or whatever, are sort of the concept of twilight factories, you could say, or twilight workplaces where human in the loop, like most of this thing is automated or can be. But what is the layer?

44:14You could call it the twilight layer where humans need to see. We need some visibility into this process or this output. We need the ability to interrupt or inspect or interrogate whatever is happening here so we can validate and stand for the quality of the output. We can explain how the decision was arrived at. That twilight thinking is sort of the human judgment layer that we're talking about. yeah i love that jess um and maybe it always stays like nebulous um you know maybe it is always very difficult to define we talk about the x factor don't we in recruitment and we all kind of like you know recoil a little bit internally now when we say this but at the same time folks let's be honest when you were recruiting someone in your long history did you not identify a few people that were a little bit off spec, but you felt they had something.

45:09And they did. All of us can immediately think of a person we hired that was off spec, but for whatever reason, we thought, you know what, this person's got something. They're in the business and suddenly they're amazing. And it's that human intuition. I don't think we can – we've denigrated, I believe. We've denigrated it and we've mixed it in with bias and all this type of stuff, but we need to rehabilitate it in some way because I think it is the only defensible place for humans when AI is able to process formal data faster than us. This is informal, kind of implicit data, high context information that maybe even process in our subconscious, which I agree, potential risk of cognitive bias coming into play.

45:57But it may be something that we need to kind of bring back into our conversation rather and just push it into the taboo territory because I don't think that we can stay kicking this stuff into taboo while still talking about human judgment. Folks, in the panel, I want you to look at the poll because there's a poll on the right-hand side of the screen. And it's like, how do you currently assess human judgment? Folks who are watching this on Crowdcast, by the way, I've given you five options. I want you to vote on these options. How do you currently assess human judgment? Number one, interview. Number two, psychometric test.

46:33Number three, functional test. Number four, work trial. And number five, other, please comment. So you're voting other, please do comment. Folks who are watching this on LinkedIn, I do see people are watching this on LinkedIn. Glenn and Kathy, quite correct. Not all judgment is good judgment. But also play this game with us a little bit. How do you assess for human judgment currently in your business? Do you do it at all? Do you do it by interview? Do you do it by psychometric? Is there some mechanism that you've isolated and say, you know what, this human judgment, we care about it. We've got a specific test for it.

47:05Let me know. Okay, let's go around the table on this. So human judgment, how do we assess for it? Currently assess for it, we've got interview in the lead, psychometric test, two of those, functional test, one of those, work trial, one of those, which is similar to functional test, I guess, and please do that. I want to talk about psychometric testing. Psychometric testing, can we test for human judgment is there anything with testing psychometrically that is in any way a legitimate signal for human judgment any thoughts um yeah i can roll my eyes for hours at the the the lack of predictive validity and psychometric tests um that there are some interesting ones on the market but they are not suited for recruitment um because they have any any kind of self self reporting at psychometric level you can't trust further than you can throw it so so yeah i i would i would just discount it almost entirely definitely no how about you uh guys jess and joseph psychometric test that we is it is it just not viable for human judgment uh is human driven to contextual, for instance, or to domain specific, you know, that kind of stuff.

48:27Yeah. So I used to manage recruiting for Kroger and they had an assessment they used, not necessarily psychometric, but they, I mean, for people on the floor in their stores, they needed fast, fun, and friendly, literally your ability to think on your feet, to make quick judgment calls, to handle messy customer service issues. But you also needed to be just good with customers and actually nice. I want to talk to you. And there was a specific way they designed to test for that. But you can only test so far. You really need to see if it's true. Put them in an environment where somebody is complaining and see how they react in the first five seconds.

49:07That's the ultimate test. So for me, it's reps. It's live work experience. As much as You can see somebody in an environment. Aiden Griffith is in the audience. I hired Aiden after seeing him at Bentley University on a student panel. Somebody from the audience asked him, do you trust AI? And his answer was, I trust my judgment of AI. Hired. Hired. Literally, based on one sentence. There's a lot that went into what he meant by that. And I discovered that in further conversation with him. but but you can only see that when somebody is actually in an environment acting that way so you got to figure out to see how to see somebody act that way yeah really really good so work trial was one of the things but again non-scalable right there unless it's like a simulation you know maybe it's an ar vr you maybe you can do something like that who knows joseph your thoughts um on this Well, I think any type of test is twofold.

50:12It's contextual. So what is the context you've given me that I'm doing these things against? And that context could be different than what I'm used to, right? The nomenclature, the structure of the way you're asking these questions, that type of thing. The other piece is the occasional noise aspect. So how am I feeling today compared to what I'm feeling five days ago when I did this test? So all of those, the tests for me are all about what's the environment you're putting me into to do that test and what is the occasional noise that I'm being rated against or what's in my mind. So I'm not always going to have the best interview, but I will always have a pretty good understanding of what the question is and how I want to answer it.

51:04But my way of speaking it might be flawed. Right. So so you get into the biases and all the rest of the aspects of of what people are being. What do you what are you comparing me against? What is good to you as the recruiter? What is a good answer that you're that you're expecting? How does my answer help you understand whether I'm actually going to be able to do those things? So all of the testing aspect, I get why people want to run out and do it. but I'm more of the opinion. I just want to see you doing it, put you in an environment, see how you're doing it. Just go do it. Show me that you, that you actually know what's going on as opposed to all of these guardrails and these flags.

51:48And if he did this, if he rolled his eyes this way, if, you know, if his right hand twitched, all of those things are bad. Well, it's, it's noise. It's irrelevant. So the reality is, can you do the job? Can in the cruder aspect, can they actually make the customer happy? yeah right if they can make the customer happy and it didn't cost the company a ton of money then that's a good result that's the end result is what we should be focusing on not how not the thought process necessarily how the person got there but can they actually do the work i think this is this is wonderful joseph and jess um i think it's wonderful because i think it's going to mean tons of work for recruitment right um because you throw away any efficiency Like we firstly, we can't scale this out.

52:31We have to interview everyone. Secondly, we have to see them at work. So nothing, not anything we do with scale is not actually possible. It's all a figment of our imagination. We're just fantasizing about it. We've all heard the expression no-brainer, right? It's a no-brainer. Give that to AI. If it's no-brainer, give it to AI. If it requires a brain, keep it. That is the most simple rule I can apply to everything we're talking about. There we go. No-brainer. It's not a bad filter, you know. It's not a bad filter. What is like obviously something that people should do? That's a process you can automate.

53:12rules can be applied perfect no-brainer exactly exactly it's a good that's actually a very good framework can a rule be applied to it um and if so just like execute the rule um but if you can't apply the rule with any sense of good outcome um or the best or there's nuance like if it's so nuanced that it's hard to apply a rule please keep it people literal people humans literal people Okay, we've got five minutes to go here I want to say to the panel Hey, wonderful panel Oh, by the way, before we let everyone go I just want to say, you know what? Brain Food Live, of course, is a conversational show We have to make sure conversations of this type continue Regardless of how long we stay on air Typically, we only stay on air for about an hour or so But it would be a tragedy If we were unable to continue this conversation Even after we go off air please make sure you continue our chat and do so with the people that have basically participated in this.

54:13And the best way to do this, of course, is simply to grab your LinkedIn URL and chuck it in the chat stream now, and then connect with everyone else who's done the same. If you're watching this on any of the LinkedIns, and I think my LinkedIn is actually working, just grab your LinkedIn URL, just put it in the comment stream there, and just connect with everyone else who's done the same. They've all hung on long enough in this conversation for you to know that they care about this topic, just go and chat with them offline. Let's keep talking. It's very important we do that. Okay. What has been left unsaid that you would like to say on this topic?

54:48Let's go around the screen here. Dr. Richard Claydon. So, yeah, my first point is to kind of congratulate Jess on the no-brainer, because I think that's that's key there but it's really into what's been unsaid as well because you can strike if it's rule-based you can structure it into the interviewing process and to joseph's or to the selection process to joseph's point do we trust the rules that have been chosen or not but that that's a key element so you can put that into a so in the language i use it could be simple or complicated as long as there are rules you can you can design around it as soon as you move beyond where the rules make sense which is actually the interview process itself very often you're then in what I would call complex the domain of complexity of thinking here is where you're asking really interesting questions that Jess said the answer about AI there's a pattern there she's sense making something that's different and she's saying that's a pattern that's worth talking to further that's that's the key and there are in certainly in my domain in the leadership development debate there are interviews that do that they test your level of complexity of thinking as a leader playing the leadership role and it can be rubriced down to you know your your they've got god knows how many gradients about what what layer of leadership you're suitable for at the moment in terms of your complexity of thinking i think you could do something like that within the interview process in recruitment.

56:23So you work with whichever software, whichever AI you want to, to put the rules in place. And then you have a rubric for the interview. So I'm looking for this level of complexity in these answers in this domain. And I think you get somewhere really interesting really quickly. And certainly in the leadership development space, that technology is fabulous. you can grade people really really gradually you know what that's very very interesting Richard I'm just reminded of like a scandalous episode in recruitment recruitment specific episode I think it was about two years or so ago when some TA person kind of demanded the chat GPT history of the candidate and the theory behind it was simply oh we just wanted to actually see your thinking and i was thinking actually that's probably legitimate you know because um firstly they were scandalized because there's a massive invasion of privacy of course you can't use that information etc etc so correctly lambasted for it um but the idea of actually understanding how you interact with ai probably is a good signal if you could ever permissibly get access to it um of course you can't but the idea of doing it i think is very very sound okay very good uh richard Joseph Braithwaite, what's been left unsaid that you wish you could said or would have volunteered some information on?

57:50I think the concept, there's been a couple of points in the chat stream that what are the tools to help the actual candidate? Well, there's a few tools out there to help the candidate get over what's happening. But I think the biggest challenge we have in the industry right now isn't the difference between what a candidate is doing and what the job description is asking for. I think, you know, those have been baked in the sameness from AI is creating the noise that we need to find a way to filter out from the 5000 resumes down to the 500 to the 50. and those filters are where we're challenged right now is the ATS systems are not there to do that work yet and hopefully they will get there but I think we're starting off with a flawed short list that we're making decisions from and I think that's the biggest challenge that I have right now with what's going on.

58:45To Richard's point, the ability to break that down and the tests down and farther to understand, you know, what people can actually do and what level they're actually working on. You know, I've integrated that concept into the tool that I built to help to help drive that out. But I think that's the direction that things need to go is if we're going to ask the tool to do stuff, what is the framework that the tool needs to work within to make sure that we're able to say, okay, you followed this framework, here's the way things fell out. Here's how we're now going to look at this and the view I'm taking on that.

59:18And now I can make an opinion or a judgment against that. I'm actually encouraged, Joseph, because I do believe there is a technical support for this challenge. And it's great that you're building something in that direction. So what's emerged for me in this conversation is that, yes, this is fiendishly difficult. Previously, you've only had to rely on this, obviously, hugely flawed human judgments or to judge human judgment, right? Can we use tech support, technology to support it? I think we can probably get measurable improvements on it. So encouraging to see some work there. Jess from Bank, last word to you.

1:00:00What has been left unsaid that should have been said on this topic? Yeah, I mean, back to the beginning with when I think about my kids coming up through high school and entering college, if they pursue that, the world of work when that time comes. I worry about what we're doing to create, to protect the opportunity to learn and to actually learn judgment as we go. There's this trend, I guess, about eliminating schools from learning or eliminating phones from learning environments, no phones, bell to bell. I understand the distraction. They're trying to eliminate. They're also trying to eliminate cheating.

1:00:44Just learn, just listen to lecture. I think there's an opportunity we're missing to actually use the tools, to Aidan's point in the chat, and others have made this as well, to use the tools to support what we're trying to get at, which is we actually want you to learn. We actually want you to form an opinion, to have the ability to apply judgment. And there's only one way to get that. So we have to use the tools not to scapegoat, but to actually create that. So how am I supporting judgment? How am I creating discernment in the way we're using these tools? That's the big miss, or I should say the opportunity.

1:01:23No, really good. And again, I'm encouraged, Jess, because I think it is possible to cultivate better judgment, right? I mean, we weren't born being good judges. Imagine being, you know, a nine day old Jess, I guess, you know, what's not particularly good at judging anything, right. But over time, you've accrued information, you've accrued experience, and suddenly, there's certain topics you make very good judgment calls on. So that gives us some human encouragement. We can we can progress and get better. I guess we can also regress and get worse. But that tells me that judgment is actually a skill, which is good, which means we can do something with it and maybe we can do something with tech support with it as well.

1:02:06Okay, listen, we've got to leave it there. What a brilliant conversation. I knew it would be amazing when this panel was assembled for this topic. We're going to have to keep talking about it. I know it's a little bit beyond some of the practical how to source for XY candidate type stuff, but I don't think we can neglect the intellectual effort to talk through this because I'm a bit tired of us and TA just adopting the next catchphrase, which is often generated by the AI tech firms and then saying as if that's like good enough. It's like a little sweet they put in your mouth and saying, go away, kid.

1:02:41No, we need to figure out what they're asking us to do. So, okay, brilliant stuff. Richard Claydon, Joseph Braithwaite, Jess from Bank, thanks for joining the show. Brilliant to have you on. folks make sure you follow these geniuses your life will get better if you do that everyone thanks for watching it's been brilliant having your show we are back next week of course we're going to be talking actually we are talking about sourcing sourcing is it dead or alive folks I missed SOSU in Amsterdam because I'm in Hong Kong right now but it's one of the best sourcing events on the planet of course still going one of the big topics of the day in that event was of course what the heck is happening with technical sourcing this idea of being able to be a super skilled identifier of talent no one else can do, AI has emerged.

1:03:30Okay, how does that impact this particular sub-community in recruiting? That's on the show next week, next Friday, 2 p.m. BST. I think it's still BST, isn't it? And we'll be there. Okay, that's it. Thanks, everyone, for watching. We'll see you next time. Thanks, everyone. Thanks.