The Interview Process Is Broken. AI Can Fix It

0:00Everybody is feeling this because the amount of cheating going on is insane and the old way of hiring is completely broken. Welcome to the Talking AI Podcast, where we talk AI with both experts in the field and early adopters. I'm your host, Matt Page, and we're here to demystify AI for you so you can get some value from it. Let's talk some AI.

0:20Interviewing in the age of AI has gotten completely crazy. Candidates are using everything from ChatGBT to cheating bots to, I wouldn't be surprised if we got people using AI avatars to do interviews for them. I even saw somebody the other day using an AI avatar for a virtual court hearing, which is just mind-blowing. But what does all this mean? It means it's super hard to vet somebody for a role you're hiring for, but our guest today, Ken Schumacher, founder of Ropes AI, is building a way to solve that, but not really in the way you would think. Welcome to Talking AI, Ken. Yeah, super great to be here.

0:55Cheating, it's a negative thing, but there's a lot of positive and negative going on in the AI world, and yeah, excited to hop on and jam with me. Yeah, man, it's, I'm excited about this topic too. I want to start with an anecdote from the other day. There's several rabbit holes I want to jump through, but I was at a Starbucks. I have a communal table as you sit at, and there's like this group of high school students across the table. And I just overheard them. Every other thing out of their mouth was, oh, let's ask ChatGPT. What has ChatGPT said? And being the curious person I am, I was like, okay, I got to ask them, how often are you using ChatGPT to do your homework?

1:30And I'm curious, what percentage do you think they said? I'm going to go knowing what I would do when I was in high school. I'm going to go like 80, something crazy high. Oh, they were a hundred percent. Like it was not even like a hesitation. They're like, oh, all of it. Yeah. But like, I say that to, cause he led me, I did a video on it and got lots of comments and feedback, but it's like this path you have from teacher to student, to AI back to teacher and the student isn't doing much. All that to say, I think what y 'all are doing with ropes and I want to hear your philosophy on this is you're not trying to necessarily restrict AI in the process, which is it's logical that I think it's hard for people to wrap their mind around because in the job, you're going to be using AI.

2:13Like, why are you trying to stop it in the interview process? I'm curious, give me the, like your philosophy on this. I'm curious, like, what was that kind of moment you had that led you to say, okay, I need to create this thing because there's this gap in the market. Yeah, no, a thousand percent. And thanks for having me on. It'll be fun. Yeah. The homework thing is very timely. I just got back from GSV ASU. Our investors is kind of the tech conference in the world. I think there's thousands of folks. And this is on everybody's mind is, listen, like people are going to use these tools for better, for worse.

2:44They're not going to go away. Just as we've had the change course before, you know, it's happening again. It is what it is. I think that's a lot of my underlying thesis. It's like these tools are here to stay, whether any of us like it or not. And they're not just here to stay, but they're actually going to keep getting better and better. My founding story of Robes, again, I was on the interviewing side. I worked at a startup called Retool. I got to lead a lot of AI projects for them, which was a blast. Except for one hour, two hours a day, I had to hop in a phone booth and I was giving the same old sort of interview problem.

3:12And this was right when ChetGV had just first come out. GPT 3.5 was out. These tools could do some amount of things, but a lot of the candidates I was meeting with were cheating the old fashioned way. Our problem was posted online. People would go look it up, practice it, come in and crush it. And again, I didn't even really blame those candidates or those students in your case. I did the same thing when I was a student. People are trying to get a job or trying to finish their homework, right? It's hard to blame people in that position. But anyway, I watched hundreds of these. And you can imagine I'm working on LLM tools.

3:44I'm running a bunch of rather boring interviews that kind of got me thinking about the combination of the two. And that's where Ropes was born from. I would say when you think about Ropes, it's really a pretty simple idea. and think of it on two phases. One, you know, there's a reason why I was in those interviews, right? Like interviews are biased. They're not perfect, but they work. They work in the sense that if I watch you code or I watch you work for an hour, I have a pretty good read, at least technically, of who you are and where you might excel and where you don't. And that comes out through something I call vibes.

4:12But it's really all of these like subtle signals, like how somebody approaches a problem or what they do when something goes wrong, all that stuff. Our bet is that large language models can pick up on those same things, just like a human can. So we're trying to build these async sort of challenges that give the same amount of insights as a live interview. I bring up the second phase because AI is a big part of that. We are not believers in, we're gonna, and we do have all these anti-cheating tools and obviously we're aware of what candidates are up to, right? But you're gonna use AI on the job, right?

4:41So especially in the practical domains, let's build tests where people are encouraged to go use LLMs. Let's see how well they can steer them. Maybe later let's educate them on how they can use these models better. but these things aren't going away. I think that's a lot of my underlying thesis. Yeah. You mentioned the comment earlier, they're only going to get better. And I had an engineer from Stripe on the podcast the other week, and it's probably an episode, probably one or two or so before this, if anybody wants to go check it out. But that was what he mentioned. This is as bad as it's going to be today.

5:14And you have to have that framing in your mind. It's so critical. And it's what he did is he basically, somebody put out a tweet about how expensive DocuSign is. So he's like, oh, I'm just going to go build it. He took two days. He built a clone of just a version of DocuSign because it's a very simple thing at the end of the day. And then that went viral. It's all just bringing to market. And it's got like thousands of people. It's just a free alternative. The DocuSign, this like huge publicly traded company, which that in itself, I want to get into some of that too, like how teams are different in startups with you being.

5:49a startup as well. Quick break in the pod. If you're listening to this podcast, chances are you've been thinking about how to actually use AI inside your business. And that's exactly why we built the AI Opportunity Finder. It's a free tool that helps you uncover high impact, tailored AI use cases based on your business, your goals, your pain points, and your industry. No fluff, no generic use cases, just real ideas that fit your business and the rank by ROI potential. It takes about three minutes to run and it's like having your own personal AI strategist for free. If you want to try it for free, check out the link in the show notes or go to hatchworks.com backslash AI dash opportunity dash finder.

6:29I mentioned, I think the other thing too that's interesting is new companies today are, they're building things with AI from the start. Like when you think about building ropes, AI is native to the entire experience, I'm assuming. 100 % versus incumbents. They're kind of like, oh, how do I weave AI into my existing solution? They graft it on. It's like this kind of just bolted on thing. But I'm curious, like what's your take there? And you're relatively, relatively speaking newer in your career, but what has that been like starting from zero? Because this is your first startup, I believe. What is it like from starting from zero with AI as a tool that you can use?

7:15Like how did that, Like, what was that process like as you were thinking through the problem with this new novel thing? Yeah. To be honest, I think it's a huge advantage. I have a couple of things. First, I think people underestimate how effective these tools are. Remember, I started the company over a year and a half ago. At that point, the models were, they were, I still thought they were incredibly good. But they were nothing near where we are today. They could write. But you saw the writing on the wall. That's the key thing, right? That's the big thing. a lot of it you were able to get ahead of it yeah a lot of it's the slope like we're lucky right now i think the hiring point where you kicked us off is like extremely visceral everybody is feeling this because the amount of cheating going on is insane and like the old way of hiring is just completely broken doesn't work anymore but the slope of that was obvious i think in 2023 or even before 23 when i picked it out maybe it was yeah before that but i go back to from a team perspective from like me building the v0 the product perspective people underestimate how helpful these tools are because they're really helpful at the early stage.

8:17If you're a software engineer at Google, you have a giant code base. You have a ton of reviews. There's a ton of context, right? These models probably aren't particularly helpful on day-to-day dev work. I'm sure they still are, but you're still doing a lot of the work yourself because the complexity is not going and writing a bunch of code. It's writing in the line with what already exists and the necessary approvals and what have you. When I was starting Ropes, there are no approvals. I'm the approver, right and there's no context to really pick up on the repo because they're i'm building it from scratch right you can drive these things extremely quickly and that's why you hear a docusign mvp you could probably spin up very fast right because you're moving from zero so first people underestimate how helpful these models are when you're kind of like very early on the other thing that you mentioned on incumbents i think is like really interesting and it's hard to generalize because i do think in many verticals incumbents have an advantage obviously they have like distribution, right?

9:12Yeah, there's, it's very nuanced, depending on where you are. But I do think your point is salient. I think of not to keep bringing up Google, but I think of like perplexity versus Google. I'm a user of both. I think there are times where the Google search gets the job done. There are other times where I'd opt for perplexity, which I think does like an incredible AI search. If you're Google, you have built an entire company around the former, right so it's very hard for you to conceptualize and go forward with kind of throwing out the old guard and throwing in something new right it's hard to move backwards to your point and you can see where people kind of like slap it on or what have you but one functionally it's difficult to rewrite the product two you have a ton of customers that are happy with what exists today you don't want to go to them and admit that you're wrong right that's an area where like it's easy for complexity, people who are going to AI search are there for AI search.

10:04So I do think that startups have some advantages, but yeah, there's, it's a dogfight. I do think you mentioned the example of the incumbents using AI. Obviously, if you're starting from nothing, it's just you. It's, Hey, let me go ask my coworker. That's you basically at AI. But I think that is, I think there is still opportunity to solve it in large companies. And I've chatted with some folks on this it's just a different type of problem that ai can be applied to i.e like when i have a very large code base lots of dependencies and things like that ai is still very well suited to solve that but it has to be like defined architected in terms of how best to do it but i the point you mentioned though is it's so freeing it's like democratizing in a sense to where anybody can go prototype, build something and bring it to market very quickly to validate it.

10:54And I think that's the key thing is you can get to validation very quick. It's, it's beautiful. And I do think there are incumbents that do this really well. A company I'm very bullish on. Ramp puts out their, every month I think they put out their fastest growing company by card spent. Yeah. And the fastest growing by spend in April was not, you know, cursor. I think by number of customers, cursor was on the list, but by cash it was HubSpot. So there are interesting examples of incumbents who have a distribution advantage and can ship this stuff really quickly. I think it's actually like an interesting like practice to go through and figure out like why are they doing so well versus why are the ones that aren't I think part of.

11:29You look at their CTO and I don't know how much he's still involved, but Darmesh, he has this whole this agent network that he's created. It's crazy. Instead of hiring people, you go to this agent network and you can hire agents or build agents. It's just this marketplace for agents. So it's like when you have those innovative people in there and even Google, man, like some of the stuff they're coming out with, I don't know if you saw yesterday or the day before, they now have like their Firebase, I think Firebase Studio, they call it, which is like their version of V0 or lovable, which is very easy to build.

12:01And it's directly integrated with Firebase for the backend authentication, all that kind of fun stuff. Yeah, I think your point is great, which like democratizing is like the great word. Like people can move up very quickly. I encourage my non-technical friends to go into a V0 and just see if you can build something. Like a very rewarding process. So yeah, it's cool. Yeah. So you mentioned something earlier, the idea of like vibe, like in the interview, how they're thinking through problems and like vibe coding is the big thing right now, but I think there is this essence of like how you problem solve.

12:31But I'm curious, like the skills, people aren't only using AI, like in the interview process they're using in the job, which means the skills you need in the future are different than the ones you needed before. Or like curious from your perspective, being a tool that helps with interviews, how do you start to think about assessing AI skills? And when I say AI skills, I don't mean like data scientist, AI, ML engineer. I mean like your ability to work with AI tools, orchestrate AI tools. Like where do you see that going? And then how do companies start to assess that? Because it's very new. Yeah, I think it's a really important question.

13:11It's like one of the most common use cases that we see on the platform, especially when working with startups or earlier teams. Look at Ropes. We're trying to hire great engineers who can drive these models really quickly. I think earlier this week I was on a panel and the panel is called like hiring for slow. I think the takeaway is that we all know that AI is going to be a bigger and bigger part of our lives, especially when we think about what we're doing in our jobs, especially in software development. But we don't know what that future is going to look like. Right. So when you're hiring people, we don't want to hire necessarily that you're perfect at Java syntax and things like that.

13:42Right. But I want to hire people who are flexible or agile. And when these new tools come out in 2027 and eight, they're going to adopt them really quickly. I think we have some opinions of like how to do that. Again, it's a very common use case. The first thing I'll say is we allow AI usage. That is like a lot of the bar to begin with. It is up to our customers. Some of our customers prefer not to use it. For the majority of customers on route, you can allow and steer these AI models however you want. That's half the battle, right, in my opinion. And the challenge on our side is creating problems where that is possible.

14:13If you can go plug a problem into an LM in one shot and you get the right answer, then it's hard to allow AI. But if we can build kind of these real-world environment challenges where you're steering these models, that's where you can allow and encourage usage. The second thing I'll say is that the types of problems are different. And I foreshadowed it there, but we're trying, and again, we're not trying to test, like, do you know perfect syntax? But we're trying to test things that are more design related. Can you take a user story and implement a front end from it? Knowing what I want to see in the application, can you go steer a VeroZero to build something that's cool and interesting?

14:47How quickly can you do that? Do you stumble on your way? Do you catch these models when they fall off guard? The problems themselves, I would say, are a little higher level, a little more design related. They definitely encourage AI usage. And then we're looking at how well are you able to steer these models? And I think that will adapt and continue as we move on over the next few years. Yeah. And I think it's, that's the core element. Like we have our generative driven development methodology and there's some core principles we talk about, but it's that idea of like, you're the orchestrator, AI is kind of executing the task, but your ability to have that conversation with AI and then how you approach that conversation is critical.

15:24Cause I think a lot of people, when they use it, they're like, it's almost like they're a dictator telling AI, go do this, go do that. but they're not having that contextual conversation with AI in a sense. Yeah. A lot of it's easy to do well when they're on the right track, but they all fall off. And are you able to go catch it and push it back? And when do you do that? Like those are types of... Yeah, exactly. And with ropes specifically, so there's an element of the tools, talk us through this because I'm curious, it's creating unique and customized tests effectively for the interviewee to take.

16:02So it's not like some off the shelf template thing. You're leveraging AI to create this customized test in a sense. How does that work? And then I think that's like the other unlock that people like listeners, like the thing you need to think about is the ability to customize has just been made so much easier with AI, which opens up millions of different use cases and businesses and problems that can be solved because you can get very unique in terms of what you're customizing for. Yeah, I think this is what's always excited me about AI and ML and NLP. I was leading projects at Microsoft and then it's always been in my mind even before I can like the LLM area, which is like things are just so much more relevant when it's unique to you versus something off the shelf.

16:50That's true across the board. That's true again in education, right? Think about like course content. Like we're very bullet on this idea of like personalized learning. So I mean, you are different. You have prereqs on being a great podcaster that I don't have. Right. So when I take a podcast in courts, I need a little more handholding and effort versus otherwise. That's where I think all this stuff is notably headed. Where that applies in assessment is our idea of problems. So we stay away from what a lot of people in the industry call like library problems. We don't have a giant library of 3000 tests and you can pick problem 2731.

17:23One, you don't want to do that because A, that problem is online and shared by other customers. B, it's really boring for your candidate. Again, those students at Starbucks are definitely going to take that problem and pump it in because it's just busy work. You know what I mean? So instead, what we do is we take a job description or we take even just like a natural language description of here's what I want to serve. We are a bank here in New York. We're going to hire for our fraud department. we need software engineers who pay great attention to detail and are able to look through like very long csv files yep what ropes will do and our secret sauce is how we do it but we'll take a bunch of dev agents essentially and we'll turn this kind of description into what we call a draft problem and that draft is going to have an instruction for the candidate so it's going to say here are the major moments here's what we want you to do right this here's the repo here's already provided here's a task and then there's going to be a bunch of boilerplates so everything the candidate needs to solve the problem.

18:18And in engineering, that's going to be a bunch of code, right? But you could imagine in Excel, that's, I don't know, an existing model, right? Usually we want to give candidates something and ask them to make some tweaks or edits. But yeah, that custom part of the platform is obviously good for employers, keeps the problem safe. It's a lot more relevant. You're actually like testing what you care about, right? But for the candidate side, it's just like a lot more interesting, right? It's actually engaging. It pitches, hey, come take a look at this repo. So come take a look at what it would be like to work on the job versus go solve these homework problems.

18:51Yeah. And then are you using AI to actually assess the performance on the back end as well? Yeah, we are. Again, think about back to earlier, which is a lot of the signal or a lot of the insights that come out of the interview are not just in the result, but yeah, it's in the thought process, right? It's in how you got there. That's why we run so many live interviews is watching somebody work very quickly gives you good read on their abilities. road steps the same so there's no human on the phone but we watch how people work in our workspace we literally look at how folks approach a problem or what they do when something goes wrong right and then we summarize that at the end we make it available for evaluators what that does again we frame it as we want to be fair for candidates that's my number one goal and then obviously number two is a business right but our bet is that if we can build the best platform for candidates evaluators will have to adopt it and will want to adopt it but anyway i bring that up because there are cases where i'm an engineer i took some of these old tests right and i can i know it's solved the problem but the syntax draws me up or i can't pass one test right does that mean i'm a bad engineer probably not so there are cases where people get into that zone they do really well but they can't figure something out but even though they might not scored very well in the end result the thought process looks really great and those people might pass in otherwise cases they won't But yeah, to answer your question, yes.

20:13We're looking at how people solve problems. It's a key part of the platform. And that's, again, for listeners, breaking down what Ken's done here, right? They've used AI to create these unique customized tests. They're using AI to assess the test. And then you look at the human element. You don't necessarily need the human in the interview trying to assess, which allows for much greater scale as well. And that's the way to start thinking about problems in which could be in your own company, could be a new business, but that's the new and novel way you got to think about this stuff is how can I solve this existing problem or job to be done with AI in a way better way.

20:55But the other thing I'm curious to was playing around with an idea because the AI voice agents are getting very good. Yes. I don't know how much you've been playing with them lately. And people can say, oh, I can still tell it's AI. But back to your point earlier, they're going to progress. You just need to know that they're going to get better. I think it was Sam Altman that said, if your product gets better as we get better, meaning AI or whatever tool you're using, then you're in a good lane. Another idea is with the actual first round interview, right? A human's time to go in, I think there's plenty of companies probably trying to do it, doing this now or whatnot, but a human to schedule the interview, figure out the time, conduct the interview, do research before the interview, do the assessment after the interview voice versus a voice agent that has the context and can do it.

21:48Right. So that frees up a ton of time for the human. And then it's also a better experience in some ways for the candidate because they don't have to wait two weeks to do an interview. they could literally do it right away right which also allows you to assess more candidates but that again there's just so many things that can be refined in terms of how we do it but i'm curious like the voice agent stuff have y 'all played with that at all i don't know if it necessarily gets into what you're doing with the assessments but what's your take on like voice agents and where they're going yeah i think voice agents is a technology first like super exciting the stuff coming out of like 11 labs and labs and all that stuff is nuts i totally agree with you Like even today, it may have some shortcomings.

22:28Like it'll be there soon. You're going to see applications across all verticals. I think sales is a huge one. Support's obviously a huge one. And recruiting definitely will have interesting implications. I think that space is interesting. We haven't touched it a ton, partially because what we have is working, partially because... It's a different modality too, right? Because it's really the engineer doing the test where you don't really need a voice agent in that process, I guess, right? Exactly. I would say in the engineering use case, people prefer it. People prefer not to have it. We hear this from engineers a lot.

22:59Like it's hard to talk about my thought process and solve something. You prefer to. Oh, that's interesting. Just, you don't have to talk. We can just implicitly see what you're thinking about through how you approach a problem. So that's one angle. The bigger thing for us is there is, I think in the business, you want to be either like cut a cost or you want to be increasing revenue, right? You want to be on one of these two ends. And I think like you can definitely argue reps cuts costs. You can replace live entities, that sort of thing. really what we focus on is revenue generation, whether that's an internal team, right, and we're able to assess way more at once and find you a greater set of engineers, or certainly on the staffing side where folks basically use us as like a core competency to go and place candidates and boost their offering versus competitors.

23:40Like, we like being in areas where people can use ropes to go win big contracts or grow a better team, right? So I think there's interesting applications there. We have our hands full with what we've currently got going on. But, uh, yeah, I think in general, it's a safe bet that these tools are going to keep getting better and better and better. Yeah. One or two other things for you. So one, obviously you, you, you haven't just built an AI solution you're building with AI. And I think you've mentioned cursor is kind of your go-to tool. Like what is that? Uh, how are you using AI in the development process?

24:11What is your, um, process look like in terms of how you're using it? I'm just curious from engineers that are now very much natively using these tools within their everyday. What does that look like for you? Any like interesting nuggets or nuance in terms of how you interact with some of these AI native IDEs? Yeah, I think the better question is like, where don't we use it? Uh, I think it's, it's a huge difference. I think probably if you wanted to judge my speed ups personally, like in the dev world from using all of them, I'm going to put it like one five X, maybe, maybe higher. like so like another human and a half almost uh yeah it's it's a it's a huge and a half kids um uh so again from a dev perspective using it everywhere the one i think it's more interesting the one like uh the one area i would try my best not to use ai very tempting to use it is email so when you receive an email from my handle it's coming from me it's not coming from quads speaking for me here what have you uh and it's very tempting honestly to use lms because i have a lot of email that's basically right but you know to me email is thinking like uh when i'm writing an email to you i'm saying hey i'm super excited for this podcast here's what i want to talk about right or here's what i think we should go with this like uh i'm not you know i'm it's not me blasting off and sure you're going to get the message but really what's valuable of that is like the process of me thinking, what should we talk about?

25:43Right. What would people find interesting? So that's an area that like, again, the value is not the text that I send across email. It's the thought process that went into like pairing it. So I try my best to keep it away from email at all costs. You get an email from me, came from these hands. But yeah, everything else, the dev side, sales side, like, you know, looking over meetings or what have you like uh lms are extremely helpful and they'll only be more so later this year sure yeah that's awesome one specific question i got because i've got this from a few different folks a few different takes yeah but in cursor specifically um um for those that don't know what cursor is an ai coding tool integrated development environment but there's the uh the chat you have with the agent right and if it starts to go on too long it can start to go off the rails a little bit of hallucinating and there's this pain point of like okay when do i start a new chat and kind of start fresh and then okay i gotta get the new chat agent up to speed on kind of where we are um any any thoughts on that in like your approach or process of like when you actually start a new chat.

26:57Do you start a new chat? Any thoughts on that? Yeah, that's actually the example that I was thinking of when we were talking about like evaluating someone's AI skills, like you can imagine. Okay, it's like, when do you cut that off? Do you know how to? Yeah, because for better or for worse, that stuff happens. These models are not perfect. It is a bit sad when you're really cooking with a chat interface and they go off the rails, right? I think I've kind of just developed like an actual tendency on like the right time to go switch over by definitely the only other thing i'll say is when i'm about to switch over sometimes i'll say okay it's weird these models are like my friends you know so i'm like yeah i'm gonna go i don't tell them i'm going to a new model i'm just like hey i'm gonna go bring this to like a smart friend uh can you write everything we talked about so far in as much detail as you know so you basically get this like really long page just like context then i'll paste that into the new chat and get moving um that just helps like kind of speed up some sorts of things uh but yeah you just develop a tendency i mean my my advice from to everybody that i talk to you from developers here to uh you know people who are not technical at all to people who are just touching ai it's just like use the tools use perplexity go use chatgmt go use card and if you use it for a week and you don't like it fine that's that's totally okay right but yeah um that's where the biggest advantage is gain today it's just like people who know how to these tools will drive much faster because as much as it's 1.5x today i think next year it might be two or two and a half three right and you want to be in a place to command it very much is this hurdle of like or flow of oh my god this is amazing oh this thing sucks oh my god it's amazing you kind of go through this like sequence of emotions always with the chat yeah with any any tool and even outside of ai but like the chat thing specifically it you mentioned it's like sad when it starts to degrade it's like it's it's slowly getting drunk like it's taking more shots of tequila you gotta like cut it off yeah but one one thing that uh somebody was talking to yesterday the way they do it very like modular in nature it's like they do one feature then they just go to a new chat and that's kind of their working process uh but i love the idea you mentioned of like okay give me the context i'm gonna take that to the other one but it's weird i do i have the same habit like you almost get like connected in a sense with the, the agent you're working with, you tell it stuff, you don't need to tell it, but you do, because it feels like you're almost working with a human, which I would tell people like, take that approach because if you act like you're working with a human, you're gonna provide better context.