How AI is shaping the job market, with Hilke Schellmann

0:00Starting a business comes with its share of ups and downs, which is why staying true to your vision is essential. a non-negotiable for Romeo and Milka Bregali, Capital One business customers and co-owners of Ra's plant-based restaurant in New York. Romeo and Milka took a leap of faith when starting their own restaurant, gutting an empty space and building it from the ground up. Every pipe, every wall, every detail. But building from scratch came with a heavy financial burden, which is when they turned to their Capital One business card. With the flexibility of the card's no preset spending limit, they were able to spend more and earn more rewards while bringing their vision to life.

0:36Today, Raz's success is proof that with passion and the right support, it's possible to make your dreams a reality. Learn more at CapitalOne.com slash business cards. Pioneers of AI is made possible with support from Inflection AI. It's not just enterprise AI. It's your enterprise AI.

0:59So I caught myself a lift, a right chair, and I got in the back of a car. And just asked the driver, how are you doing? And in the history of me taking a Lyft, this has never happened, that a driver said, oh, I had a really weird day. And I was like, really? Why? And he was kind enough to tell me that just a couple hours before he had a job interview with a robot. Since 2017, I had no clue what he spoke about. And I was like, what? A job interview with a robot? So he had applied for a baggage handler position at a local airport and, you know, got a call. And this pre-recorded voice, which he called the robot, had asked him three questions.

1:41So I'd never heard of it. We chatted, you know, it's a pretty quick ride to Union Station and kind of forgot about it until a few months later. I went to the first FACT AI conference at NYU and remembered the job robot and I was hooked. I wanted to know more. So that started this whole eight year, nine year journey. That is not over yet. Hilke Shelman is an investigative journalist and author of the book, The Algorithm. She spent years reporting on artificial intelligence as it shows up in the workplace, and specifically, how AI is being used to hire workers.

2:23If you've been on the job hunt recently, you know it can be hard. A survey from Aratech found that nearly 70 % of people say that their current job search was way more challenging than their last. And it's true that the way companies hire is changing, from how they find candidates to how they evaluate resumes. AI is part of that change, as well as a growing factor in who gets promoted and even who gets fired. On this episode, we're going to dive into Hilka's findings around all of this, and my experiences too. I'm Rana El-Khalyubi, and this is Pioneers of AI, a podcast taking you behind the scenes of the AI revolution.

3:23Hilke Shellman is a journalist who tackles big systems and how they impact or fail people. She's covered the barriers for sexual assault victims reporting crimes, as well as the student debt crisis. Her book, The Algorithm, takes a holistic look at the effects of AI on people in the job market. I wanted to start with that first Lyft driver who gave her the idea. Hilke, thank you for joining us today. Thank you so much for having me. When this Lyft driver was sharing his story, was he kind of saying it with excitement or was it fear? Like, what was the emotion? Yeah, I think he was a little weirded out, to be honest.

4:04It definitely shook him up a little bit. I don't know if it was fearful. It was just really odd and really surprising for sure to him. You know, now I've talked to a lot of people who've done one-way video interviews, and I don't think they're necessarily fearful. I just think they just thought it was so weird, the experience that, you know, you would assume that if you have a job interview with somebody, somebody is on the other side and talking to you and they're like, there's no one on the other side. There's like, you know, you get like pre-recorded questions and then you record yourself answering.

4:34And they were like, yeah, I'm really excited about the job, but I'm talking to my camera on my computer. And that's kind of really weird to get excited about that. This moment where I met the Lyft driver, you know, was the beginning of writing this book. It technically was, but there were a couple of years in between where I was like working on stories for the Wall Street Journal on AI and technology. And I started to write more about AI and hiring for other outlets. And like over the course of looking into this industry and seeing how big AI is becoming in HR and hiring and in the world of work, I sort of felt like, wow, I think this really warrants a book.

5:07This is a real sea change. Like so many companies are using this kind of technology. And I didn't see a lot of people talking about it Because I think it's a little bit hidden in a way, because I think a lot of job seekers don't necessarily know that AI is being used in them, right? They upload their resume and their application to a job platform, be it LinkedIn or Indeed or ZipRecruiter. They don't understand that on the other side, there might be AI that is, quote unquote, looking at their resume and possibly rejecting them or putting them in the next round, right? It could also be human. But we know that like large job platforms, they all use AI.

5:40But I think it's not really clear to job seekers. So I thought there was a little bit of an information vacuum. And I think also that the technology was started with folks who were in retail, in fast food often, where employers have to hire a lot of people and have high turnover. And then slowly, we know that the technology was used to hire flight attendants, teachers. Is there a cultural change in the hiring process? I mean, one of them is these huge platforms like LinkedIn, where like hundreds of people can apply to a job, right? Is that a part of it? Yeah, I think that's probably the biggest driver, right?

6:13That with the dawn of job platforms, so easy to find job openings. Now it's all there. An algorithm helps you find the best ones. And then it's so easy to apply. It often takes seconds to upload your resume. So it democratized hiring for job seekers. But I think on the other side, you know, what companies say is like they get a deluge of applications and they feel they're drowning in applications. So, you know, Google says they get about 3 million applications a year. IBM says they get about 5 million resumes a year. Goldman Sachs said a couple of years ago that for their summer internship program alone, they got over 220 ,000 applications.

6:51So obviously, they're not enough humans to look through all of these resumes. And I should also say that like humans are very biased in hiring. So that might actually also not our optimal solution. Absolutely. Humans are really biased when it comes to hiring. And so I'll share just some numbers, right? Okay, white-sounding names and resumes get 50 % more callbacks for interviews than non-white-sounding names. Women applicants are 30 % less likely to receive a callback for an interview. And blind hiring increases the likelihood of hiring a woman by 25 % to 46%. And then this last one is really shocking to me.

7:27Basically, 48%, which is about half of the hiring managers, admit to being biased in their choice of a candidate. So I guess the question I'd love for us to explore together is, can AI do better or not really? I wish I could answer that question, but we don't have those longitudinal studies, you know, and I do think that would be a huge thing for humanity if any company could do this so we can actually tell, well, does this AI tool actually work? But also, is it better than humans? And I do think that in general, humans are not very good. And we've seen a lot of research in trying to get bias out of humans by training them.

8:04And that is also not very successful. So I actually do think that we need technological solutions to the problem. We just need to find the right technological solutions. Because I think what we've seen in the sort of the first generation, maybe, if you will, of AI tools used in hiring, that we've seen some misfires and some misapplications. And I think we really need to learn from that and build better tools and test those tools for discrimination and use the less discriminatory algorithms. So the ideal algorithm, or actually the ideal hiring manager, whether it's a human or a machine, I think, ought to really focus on the skills needed for the job and be blind to everything else, right?

8:44Blind to your gender, your ethnicity, your age. Can you talk about this concept of blind hiring and what does it actually mean? Yeah, I mean, the idea is really like, if we should hire someone for the job, what is the most important thing this person needs to do in the job? And we should really hire for that. So that's usually skills, capabilities, and not what hair color you have, where you're from, your gender. And I think what sort of happens when we often use AI, we take in a lot of information, right? So the software looks at a resume and often the software doesn't only look at the skills and the capabilities and maybe work history.

9:21Because, you know, that gives you some sense of people's capabilities as well. When I talked to employment lawyers who brought in as outside counsel to look into some of these tools, they found out that some of these tools, unfortunately, you know, did what AI did best, did a statistical analysis. And then one of the AI tools found out that the name Thomas was predictive of success. So obviously, the first name Thomas doesn't qualify you for any job, right? Sorry for all the Thomases. What probably happened is that a company gave the tool, you know, a resume of people who are successful in the job right now, maybe hundreds of people, and maybe there were a bunch of Thomases in the pile.

9:54So the AI found the statistical pattern and suddenly Thomas is a proxy for success. So we see this again and again, right? In another tool, it was Syria and Canada were predictors of success. And that could actually be discrimination based on national origin. Another tool used Africa and African-American. Another tool gave people the word baseball on their resume, more points. And if you had the word softball on your resume, fewer points. Obviously it has nothing to do with the job and that's probably gender discrimination, right? So I think that is the problem when we use AI and don't constrain it, when it looks at, in this case, at all the words on a resume.

10:32And we see some companies that mask pronouns and that mask names and addresses, and that's all great. But the problem is the bias can come in through proxies that seem neutral, that seem non-problematic, and then they happen to be problematic again. So I think that is a thing that is really, really hard. So we need to supervise these systems. And I think that's often lacking. So we know how AI is affecting the job application process and the role bias plays. But what does that process look like for a job seeker? And is AI helping or hurting their chances for getting that job? That's after the break.

11:14If you've spent any time building AI products or leading technical teams, you know this. Transformation doesn't fail because of ideas. it fails because teams can't move together. Enter Atlassian's teamwork collection. It has planning in Jira, documentation in Confluence, video updates in Loom, and now AI agents in Rovo, which connects the dots across your work so nothing gets lost. It's one AI-powered teamwork platform designed for how modern teams actually build. Learn more at Atlassian.com slash TeamChanger. that's a-t-l-a-s-s-i-a-n dot com slash team changer in your book you talk about sophie a software developer in her 20s and you share her employment journey apparently she was a star candidate with many qualities that employers seek but somehow her job application just never seemed to go anywhere what was happening there she had everything that I think if you were a software developer recruiter, wouldn't want.

12:23She had an undergrad and a master's degree in information science and software development and UX design. She had taught a girl's coding camp. So she was a teacher. She had a portfolio because she had done internships. She had everything you wanted. Also, she's a veteran and she's black and she's a woman, like all kinds of things that people in tech would want, right? We found her through her professor who put her forward and said like, hey, you know, she's a really interesting story that I'm really shocked by too. And I was like, wow, I thought the recruiters would like throw offers at you, right?

12:55Like she was part of like sort of, you know, women in tech groups on LinkedIn and other places. And she's like, yes, I'm sending 200, 300 applications and don't hear anything, which, you know, I had assumed for some jobs, but maybe not software development, because we always hear companies saying we don't have enough talent in tech. I found incidents of bias. And also we know from surveys of leadership as well that when they use AI tools, almost 90 % said that they know that their AI tools reject well-qualified candidates. So we kind of all know that it's not working super well or as well as we would hope it does.

13:32And I think there we really need to push into and build better tools. So what happened to Sophie? Did she get a job at the end? she did get a job after 146 applications. She did get a job. She was very happy about that. And the way she got hired is she did a little bit of a roundabout thing. So she would send in her application and then she would find out who's the recruiter and hit them up on LinkedIn and send them her resume with a message. And she was like, that's how I got the interviews. And that's how I at the end got a job offer. She and I did some AI tests and it turned out that her resume wasn't necessarily well picked up by some of the AI tools.

14:12So job seekers can check that online, how much your job description and your resume have an overlap. It's not foolproof because we don't exactly know what kind of AI a company necessarily uses. But if they use job description and resume and have an overlap, this can be a good indication. I want to dig into another example of how things can go wrong. So in your book, you talk about One particular example that totally hit home for me, job recruiting platforms may accidentally discriminate against women by amplifying these like really subtle behaviors, right? So specifically, men often apply to jobs, even if they only partly qualify for these jobs, whereas women wait until they check all the boxes, right?

14:54I've definitely done that where I look at, I was like, I only have like half of the qualifications and should I really apply? I'm not sure. And, you know, I think a lot of men are a little bit more confident in their abilities. You know, we see this a lot in like sort of the psychology literature. There's confidence versus competent. And in a lot of workplaces, if you appear very confident, that is seen as you are very competent, which may or may not be true. So I think we see this here too, right? And I think it really comes out at job platforms because they track what the job seekers do on the platform, like every click, what you do.

15:28And it was kind of interesting when I talked to John Gerson, who's the former vice president of product at LinkedIn, he said that AI isn't necessarily built to find the most qualified people. It is built to find the most qualified people likely to apply. Because somebody like me, who's been very happy at a job for seven years, I don't really apply to any job on LinkedIn. So if I was an AI, I wouldn't put me at the top either, even if I was the most qualified, because I'm very unlikely to apply. And you want to make a recruiter happy, and they want applicants, not just resumes of people who will never apply or don't want the job.

16:05How do you measure if somebody is likely to apply? It's usually with signals on the platform, right? Do you follow companies? Do you message back recruiters? So it turns out men are a little bit more aggressive in general, not all men, but you know, more than women message back recruiters. And so that's a signal to an algorithm you are likely to apply. So I think we see this like gender-based behavior that really most of us can absolutely not control. I mean, I can tell you, and you know, I've definitely changed my behavior. I actually have been messaging back recruiters now and started following companies just so I'm like, oh, it's kind of interesting.

16:40We also see now like LinkedIn and other companies have AI built that pushes a little bit against that. So for example, like women are also often, you know, more modest in putting on their skills. And that's one of the problems of a resume, right? Like it says, I'm looking for a software developer. So it maybe says that, you know, the programming language Python, but as a hiring manager, I don't know, like, are you a beginner? Are you a master developer? No one knows that, right? And so I think what also happens often is women are a little bit more modest. So maybe, you know, a man maybe takes, you know, a two-month class and puts Python on their resume, where women often wait two or three years or so till they have like a master level competency.

17:18So now we have algorithms that infer qualifications. So we see a little bit of a push with AI to level the playing field. But I think this gender-based behavior is really, really hard to overcome. What is the process of building these AI algorithms? And it all starts with data, right? So tell us more about that. So in hiring and at work, I think one of the real problems is that a lot of the data that we have is already biased, right? So you could think about like, you know, I want to build like an AI that promotes people, that finds people that have been promoted previously because, you know, John always puts forward Alex and, you know, we know that Alex isn't really that competent.

17:55I want some new voices. Like who are the hidden gems in my company? So I'm going to build an AI maybe based partially on performance reviews. Well, it turns out performance reviews that are usually done by humans, they're also biased. Women, people of color, people with disabilities are underrated in performance reviews, even though they have the same achievements in performance than, for example, white men. So if you already have biased data, if you build an algorithm and don't supervise it, but have an unsupervised system and don't test it for what we call disparate impact, that's a real problem.

18:29So we see this again and again, and there really isn't a lot of unbiased data. And bias mitigation takes a lot of time, takes a lot of work, and that is not always done. I mean, there's sort of best practices, but I think what we see in hiring and in work algorithms a lot is there is guidance from the government. Unfortunately, it is 45 years old. It's the uniform guidelines from 1978 that tell companies It's best if you look at different races against each other and gender, men versus women. But we don't often look at like the intersection of like white men versus black women, for example, where we know where the crux is, where the problem is.

19:05There's no real bias mitigation for people with disabilities. Like we don't even actually check for that. So there is a problem in these systems. Yeah. Back when I was running Affectiva, because we were building emotion recognition technology, We were very intentional about the data, but also about how we tested these algorithms, right? It is so important that you ask these tough questions of the algorithm. You try to poke holes at it. But you're right. Like, it takes a lot of time. It takes a lot of money. It slows down your product launches. And so you have to be really committed to that. When you were doing your research, how top of mind were some of these issues for the companies you interviewed?

19:43I think enough stories come out of bias in these algorithms that I think, you know, a lot of people are more aware of it. And I think a lot of people, especially companies that buy these tools, you know, are always encouraged to do pilot studies and like test the technology. Don't believe what a vendor tells you because, you know, they're selling the technology. And to be honest, a lot of them are venture capital backed. They have to bring a product to market very quickly, right? They might not have the time to actually do all of this testing. And, you know, I was also going to ask you, like, I know that HireVue used Effectiva and did the emotion recognition system for hiring.

20:17And, like, did you think that was a good application of their technology? HireVue is an AI and human resource company. They enable employers to conduct video interviews where the applicant initially interacts with a computer instead of a human interviewer. In fact, the Lyft driver that inspired Hilka's work to begin with could have encountered HireVue's technology in their job search. Yeah, you know, that's interesting. I'm glad you brought that up. So one of the applications we explored at Affectiva was this idea of can AI help de-bias the hiring process and also bring people's resumes to life, right?

20:59Like if you are applying to be a flight attendant for Southwest Airlines, right? And you're like really like empathetic and you have very high EQ. That's very hard to portray in a Word document. Oh, impossible. Impossible, right? How do you represent that? So I love the idea of a video interview. It does bring your story to life and it gives you an opportunity to really showcase who you are. So that was the impetus, right? The team at HireVue was really focused on leveraging technology to help recruiters sift through all these videos. The great thing is these algorithms are super blind to gender and ethnicity and age.

21:38It's really looking at your emotional reactions. But if there's a little bit of bias in any of these algorithms, you're right. It's going to be deployed exponentially and exacerbate a lot of biases that exist in society. So we were very thoughtful about that. We did end up pausing our partnership with them, but I still love the team. And one of the things that I also realized in this whole process is who's building these algorithms really matters, right? The diversity of the team around the table is important. So can you say a little bit about that? I do think that diversity in teams is like really, really, really, really, really important.

22:17One example is like, you know, in hiring, I played a lot of these games and a lot of video interviews. and you know one of the video games that I played that was supposed to find out you know my personality and capabilities you know sort of kind of like soft skills so to speak one of the things I had to do is I had to hit the space bar as fast as possible in a certain amount of time when I was doing this I was asking myself like what does I have to do with the job that's odd I've never had to hit the space bar as fast as possible right but then when I played the game with somebody who is quadriplegic he's like what is with people who have a motor disability and I was like oh yeah you're totally right.

22:52Like what would happen to them if they, you know, maybe they can't hit the spacebar as fast as possible. I have no idea how diverse the team was that built this algorithm, but it felt like they're probably missing a lot of these questions here. So I think diversity is really important. I have unfortunately also found out that even though we as humans can think of a lot of ways that algorithms can be biased, man, bias proxies come from anywhere. So what's a bias proxy? A bias proxy is, for example, something that indicates that you'll be successful. So for example, I talked to the former head of talent acquisition at Walmart.

23:28One of their core objectives is like we need to hire people who stay longer in the stores, right? They had found out that in a survey that if you have a friend or an acquaintance at a store, you stay longer. So that looks pretty neutral, right? So they were thinking maybe we should use this as one of the criteria. So I did a pilot testing and it turns out it's very predictive that if you have a friend or an acquaintance, but when they looked at the results and tested them for race and gender, it turned out that mostly Asian Americans had acquaintances and friends at the store and African Americans had not.

24:01So even something that looks like a neutral proxy for you as an indicator of success can actually be very biased because you would have discriminated against African-Americans in this way without ever intending it. And the law in the United States makes no difference. You can intend or not intend discrimination. And it doesn't matter. If there is discrimination, the federal investigators might investigate you and bring a case forward. We haven't seen a lot of investigations in the space because I think that the way AI works, it's a little bit obfuscated than maybe traditional assessments where you had like for a firefighting job, we knew you had to like, you know, take 200 pounds and carry them and move them from A to B.

24:44right now the problem is that maybe I'll play a game. Maybe I know there's an algorithmic assessment, but how do I know what is actually being assessed? I have no idea as a candidate. And also like, is there quote unquote harm being done or being discriminated? I have no idea. I get rejected or I get put in the next round. And you know, that's also obviously very ubiquitous as part of the hiring process, right? Like we get rejected all the time. So as a job candidate, I'm like, I don't know why. And you know, usually in a court of law, you have to show that you've been harmed. So just being rejected is really often not enough.