Brainfood Live On Air - Ep408 - Assessing the Skilful Use of AI

0:05take him out of orange and stuff like this so really ancient ways in which human beings are economically active. And I think we're all probably going to find a way to do that, either through spiritual expression or through practical purpose. You can make some goat's milk or something and do something like that. Yeah, we can always eat more cheese. That's for sure. That's it. Okay, folks, we're live, everybody. Welcome to Brain Food Live on Air. I don't even know what episode we're on, something in the 400s or so. We're going to keep cranking out these episodes, folks, because it is got so much to talk about in the world of work today.

0:42I think we can't keep to that like once a week schedule. Whenever we have an opportunity to have a great conversation, we're going to go ahead and grab the opportunity and do it. And today I'm really pleased to do a special with you because the early conversations I've had with my next guest have been so interesting that I think it's going to be quite difficult to kind cut this down to one hour David. We're going to explore a ton about the future of software engineering, the future of recruitment of software engineers, how do you assess those folks, and what it means for the work when in fact you end up interacting and organizing agents to do it rather than doing it yourself.

1:25So the topic of today is going to be how to assess the skillful use of AI, what it means to recruit software engineers in 2026 and beyond. My guest today is David House, Senior Engineering Lead at G2i. David, great to have you on the show. Welcome to it. I probably need to give you an opportunity to say a little bit about who you are and what G2i does. So do you want to kick us off with maybe 60 seconds of that? Sure. Yeah, thank you. I'm really happy to be here. Really appreciate the opportunity. I've been at G2I for about 10 months, which is about the amount of time that we've been working in AI.

2:09G2I is a namestay in staffing and recruiting for the last 10 years. But recently we've had the opportunity to work with Frontier Labs and needed to recruit a lot of software engineers very quickly in order to build a lot of software. um so i i if you rewind to 10 months ago that was about the time that uh agents started stopped becoming scary for software engineers and started becoming the de facto way that you have to do your work in order to be competitive um so i i you know hired um dozens and dozens of engineers through that transition in our ecosystem. And not every hire was a successful one, but I think I have some lessons that I can share for the audience, both for recruiters who are trying to find the best candidate and for engineers who are trying to show their skill with AI.

3:08Yeah, fantastic stuff. And let's kind of start with a kind of a statement on this moment of AI agents? Because I think within the software engineering milieu, it's probably widely accepted now that the AI agent sort of moment was a decisive sort of break from what happened before to after. But from people outside of that space, they may not entirely be aware because they're thinking, I'm still using ChatGBT as I always did. You know, what is the big deal? Can you sort of describe to that audience what an AI agent is and what it's meant to software engineering? Yeah, absolutely. So if your use of AI has been only in chat GPT, only in the browser, you haven't really experienced anything close to AI's capability to expedite and improve your workflows.

4:08And I'm not even getting into the realm of job replacement. I'm just getting into the realm of automating the boring stuff, taking the things that you do every day that don't require a lot of focus and just doing them really quickly. The difference between these two workflows of using ChatGPT in the browser and using ChatGPT on your computer is in the browser, you're just talking to the model. but on your computer the model acts as an agent and can act on your behalf in your computer. So its ability to read information that is available to you in your work, its ability to synthesize information across multiple different applications, and its ability to take actions on your behalf is increased exponentially.

5:01If you want to see a really clear example of this, the promotional video from GPT Astra that dropped recently, that paints it clearer than I ever possibly could. Seeing what people are able to do with GPT on the desktop app versus just in the browser, it's pretty amazing. Yeah, fantastic. And this obviously has had more impact in the world of software engineering than in any other field. And I do have a feeling that software engineers were obviously the first to adopt sort of artificial intelligence in their work. And it is still the place that's having the most dramatic transformation on the job itself.

5:49So can you tell us what it is to be a software engineer sort of pre-AI agent compared to post? What are the main elements of work that have shifted in your work, for instance, David? Yeah, absolutely. So if you think about coding before agents, every engineer, every engineering team, every company that writes software was limited by the speed that engineers could write code. I'm sure there are people out there who will quibble with this characterization. organization. But for contrast purposes, now it's much easier to write code. So the expertise of a software engineer, the expertise of an engineering team, the expertise of a software company becomes how do you know what code to write and how do you know that the code is good, how do you know that the code will work?

6:53Essentially, it shifts the responsibility and the skill shifts from authorship to verification. Because in a world with agents in software engineering where you can write a single prompt and two hours later have an entire application, how do you know if it's right? How do you know if it's good? How do you know if it's correct? How do you know if you can go to sleep and not be paged at 2 a.m. because prod is broken? Yeah, I'm getting kind of a visualization of a barbell where the world of engineering work is at the beginning, is moved to the beginning. So you've got to architect it, you've got to prompt it, you've got to have the systems designed.

7:37And then it's kind of moved to the end where you've got to verify, confirm, integrate, and then release. Whereas all of the co-generation bit in the middle, which previously was the bulk of the work, is now conducted by one agent, a swarm of agents. Basically, agents do the middle bit, so it becomes a thin kind of human observation in the middle, whereas the setup and the direction at the front is human, and the verification, confirmation, validation is at the end. Yes. And I, and I think that's also exactly where, where engineers can be steered in the wrong direction. I think that's also where people using AI to speed up their workflows can be steered in the wrong direction.

8:24Right. Because if you sort of set it and forget it, then it's very easy to let a small mistake that the AI made. just slipped through. I think in my mind, I, you know, if you are, okay, I'll tell you a quick story about an interview that I did recently. I interviewed someone for a role with one of the Frontier Labs, and they, I always do like a live coding exercise, but it surprises every single engineer because it's very easy. I give them a very easy coding exercise. And I tell them up front, like the goal here is not to assess the code that you produce. The goal is to assess your ability to use AI.

9:17And we had 25 minutes for this exercise. Not a single line of code was written because the candidate got so distracted by pitching me on how amazing and how sophisticated their tools were for orchestrating agents, swarms of agents, that agents, verifying agents, self-agents, like it was that nothing was written. So as a hiring manager, if I am interviewing someone and I see them do that, I think like, okay, well, what are they going to do on my team? Or what are they going to do on this team that I'm interviewing for with one of the frontier labs? I don't want to trust them to try to deliver value if I give them a simple coding task and they don't write any code because they're so excited by all of the agent orchestration.

10:09So I think that's sort of the tension where it's very, very important to continue to push boundaries and continue to take agents and try them on new tasks because the models keep getting better. But you have to make sure that the agent doesn't work ahead of you. And there are lots of very creative ways that you can verify the progress as it goes, both for a software engineer doing work and an interviewer assessing whether or not someone has AI psychosis. There's a term, and I'm careful to caveat this, because I'm not sure if it's grammatically correct or it's accurate according to the Oxford English Dictionary, but I'm going to give it a go anyway.

10:55There's a term called neophilia, where you are just kind of very predisposed for the new. It's very exciting. It's revolutionary. This is the way it's going to transform how it used to be done. And it's wonderful. Yeah, and it's shiny toy syndrome. But the shiny toys are real. It's not fake. It's actually real. But the problem we have with AI is that the shiny toys are shinier and kind of the frequency with which the shiny new shiny toy arrives is such that you could just constantly be looking at the new shiny toy and not actually creating valuable output. And yet we want to have people who are forward looking.

11:35So how do we balance between that? Like, what would you say is the mental sort of makeup of a person that's able to make that judgment?

11:49That is only a question that I have been able to answer through gut feel, honestly. And I think that is, it gets into this realm of, you know good software when you see it. You know good software when you feel it. You know someone who is accountable when you speak to them. You know that someone is listening to you with the way that they talk. I think if I had a really easy answer to your question, then I truly would have solved software engineering recruiting. And I don't think I have. I think I've made some progress, but I don't think I've solved it. I think the more I use AI, the more I experience its benefits, the more I experience its pitfalls, the more I ingrain in myself an understanding of experience and sort of wisdom even with when to use AI in certain situations and I can recognize that in other people when I talk to them.

13:16Maybe like I could speculate if anyone out there is an expert in something. Say you have a hobby that you're really good at or you want to explain software to your family, you want to explain recruiting to your family, you go home to the holidays for dinner and you try to explain your work or your expertise to your family, you have to change the way you talk about it, right? There's like certain phrases that you can't say because the way that you talk about it only makes sense to someone who is also experienced. I can say that I have that experience all the time with using AI. Like using AI is something that I have experience with.

14:08And if I'm talking to someone who also has deep experience with it, who has also developed that wisdom, then I find that the way that I am able to talk about AI with them is different than if I talk to someone who has perpetual shiny object syndrome or someone who is so nervous about using AI that they haven't actually dove in. I want to use the term resonance here, you know, in that kind of not irrational, but let's say it is non-intellectual way, like you're able to vibe with someone. And I think vibe is the right term because it's vibrational like the the the resonance of the conversation is such that you know and both parties know that you're syncing and your understanding is very very uh sort of uncannily um a map to each other and folks if you're watching this i'm sure you can all resonate with that idea like you've had conversations with people who may have been strangers to you but you remember when you immediately synced um and you immediately got each other um that type of resonance.

15:18I think this is what you're referring to here, David, that you need to share the experience, gone down maybe the same journey. Maybe at a neurological level, there's some sort of rewiring going on, which is comparable to the other person that's gone down the similar path. And that's when you can kind of communicate in that really rich and deep way. It kind of, I mean, this is now diving into some really interesting kind of philosophical kind of talk, but we need to go there because what we're hearing in 2026, David, is that, hey, AI is going to do all of this stuff, which is great. But, you know, one thing you can't do is human judgment, quote unquote.

15:56And it seems to be like a big full stop on that. And it's like, I'm thinking, okay, great. But that starts the question. It doesn't end the conversation. What do you mean by human judgment? How do we get better at human? Can I acquire this? Is it a skill? is it universal? For instance, are we great at human judgment across domain or is it domain specific? Now, what's your take on this, David? Firstly, human judgment, is it something that we can get better or worse at? Is it a skill? Yeah. I mean, I think there are, the way that I try to be honest with my own assessment of my judgment is as close to the scientific method as I can get.

16:50Right. And I think this is, uh, there, there's like, I'm not a philosopher, so I'm going to, I'm going to say things that someone who's actually trained in philosophy will, will scoff at maybe. But you could say that there are, you know, scientific appeals and then there are political appeals, right? I think I have a deeply intuitive sense of when someone is making an appeal that's based in evidence that is verifiable, where they are clear about the methods that they use to arrive at that conclusion versus whether someone is trying to persuade me on the basis of their charisma.

17:45I want engineers on my team to be scientists. uh because i think that it's like scientific reasoning researcher reasoning is a just a good effective way to make sure that what you're getting out of ai is useful uh it's it's sort of if you contrast that with like a political appeal a like maybe we'll throw out the gpt4o dynamics right um for those of you who might not remember this is um 10 years ago in ai time about a year ago in the real world uh gpt4o was highly sycophantic so it it was very very likely to tell you how amazing you were and how awesome your thinking was. And, oh, you've developed something, some new branch of quantum physics, right?

18:56That's a political appeal. That's not actually grounded in anything real. And people love to be told how amazing they are. So you can sort of, if you're not careful, if you're not very principled in your reasoning and your judgment, then you can be complicit in a lie that the AI hallucinates and then convinces you of. So I don't know. I'm not sure how clearly that answers your question, but I do think you can train judgment. I think it requires a lot of experience. But for me personally, the way that I value it is through something approaching a scientific method. Yeah, I get it. I mean, I'll share my opinion as well, David.

19:46I don't want this to sort of be overly loaded and you to come up with all the answers. So folks, shoot me down if you think I'm wrong on this. I think that judgment can be trained. And I do think it is domain specific. So I think it's possible for people to be very good judges of a certain thing, but also terrible judges in another thing. So what does that mean? It means that we need to kind of have the reps in order to get through into the intuition, which is what judgment is. So without the reps, having done it and seen that situation over and over again, and got that feeling that yes, this generally happens when X generally happens when Y occurs.

20:28I think that intuition is worth paying attention to, which is completely different to a person that's making a gut feel sort of judgment, a knee-jerk judgment on something they've got no experience on where they don't have any prior experience at all. So experience matters. You're an experienced engineer, David. Brilliant stuff. You might be in an ideal situation because you develop your skillset pre-AI and now you're AI enabled. So we've got this like perfect balance. But what happens to the generation coming up sort of that don't know a world of software engineering outside of AI. Are we at potentially a risk where they don't have the reps to understand at a deep level and therefore their human judgment might degrade?

21:16And at a civilizational level, we might actually level down as we have people that don't have pre-AI experience. What's your thoughts on that? I'll just be very honest. I am not at all concerned about this. Not at all. So I'm going to speak directly to the software engineers, perhaps get a little bit of a technical slant on this and try to think of another way of framing it. I think about this in terms of low-level and high-level programming languages, right? Taking your framing of you can have good judgment in one context and bad judgment in another. I have excellent judgment in JavaScript and I have terrible judgment in assembly.

22:03Assembly being a very, very low level programming language and JavaScript being a very high level programming language because my software engineering skills are trained in that high level space and not in that low level space. But if I go to my friend's dad, who was an engineer at Microsoft and wrote assembly, he probably has really good judgment in assembly and nowhere near as good judgment in JavaScript. He doesn't optimize for that level. So the question becomes then, how does AI fit into this? I don't like to predict the future, but when I do, I try to be near term. I think near term, programming languages are going to all sort of converge into this lower level space, where in order to be an effective software engineer, it's less important that you are an expert in code.

23:11But it becomes so much more important that you are an expert in software. And I think all the junior engineers out there can learn that. And I hired an intern and taught him. He was fresh out of college, had never worked as a software engineer before. I mentored him very specifically on how to use AI well, how to use AI that encourages his learning, how to multiply his onboarding efforts using AI. And he was building things in his first three months that I didn't build for my first three years easily, easily. So I, it can, I think we, it's easy to get alarmist. but I think we're just getting higher level.

24:10I think if you go from the Turing machine to assembly, assembly to JavaScript, JavaScript to LLMs, I don't know. It remains to be seen but that's how I am orienting around it and that's why I'm not concerned. Yeah, it's great to hear optimistic stories, David And, yeah, I think a lot of junior people, they just need to have guidance and direction from somebody senior in order to move forward in the right way. Yes. Rather than sort of have this perception that, you know, AI is replacing particularly junior staff. And therefore, you know, we just got to double down purely on highly experienced, which from a recruitment perspective, that tends to be happening simply because of this, you know, short sightedness and the perceived sooner return to value of experienced hires.

25:10But as you as you mentioned with this personal example, you can get someone up to speed very quickly with the use of AI, as a matter of fact. So it's actually accelerating the ramp up with the right guidance. they can be a very useful person very early on in their career. And potentially, you set them off in a self-learning loop, and then you're going to be in great shape with a very strong resource in your team that might hang around a bit longer than someone who's already three, four companies in in terms of experience. I want to throw something on there too. There's a supply and demand dynamic, right?

25:48Right. There's lower demand for junior engineers right now because I think this mistaken perception that they are less valuable, which I, again, I disagree with. So there's a higher supply of junior engineers, a lot of people who are really hungry to get in the industry. If I have like one word of advice for all those junior engineers, just hang in there. Keep trying, keep learning, do personal projects, get your reps in. so that as soon as people start to realize that you're extraordinarily valuable to the profession, that you'll be a shoe-in hire. Yeah, that's exactly right. I mean, leads us on to the next question that I want to move that conversation on to, more recruitment-specific.

26:29How do we surface up signal of this high-potential candidate when we're no longer relying on the traditional signals of capability? In software, in general recruitment, we used to look at their CV, their resume, their LinkedIn. In software engineering, we might look at personal projects, their GitHub and so on. But of course, today, everyone can produce the code. Everyone can produce the apps. And it's like, okay, that kind of might make the situation more noisy and make it more difficult for me to identify the signal. Is it possible to find the signal somehow without the direct face-to-face or sort of one-to-one interaction?

27:15How do you do it, David? Yeah, without the face-to-face, one-to-one interaction, that I haven't solved. That is, I think, I wouldn't want to say that AI changes literally everything. thing, I think looking at people's resumes, looking at people's CVs, still very valuable. If you can find ways of leveraging AI to verify the things in that person's CV, right? Like, give your AI to GPT and say, like, take a critical eye to every single one of these lines, search the internet to verify that these companies exist, you know, take a critical eye to whether or not this person could have done these things at this role, surface any contrarian evidence that might suggest that these claims are inflated, right?

28:08That's a great way to use AI ahead of actually scheduling a call with someone to evaluate their skills. And again, it's one of those things where like, should you do that? Like, yeah. Is that valuable? Absolutely. Do you have the time to do that without the use of AI? Probably not. Maybe you do. Maybe all the recruiters out there are so much more capable than I give them credit for. But I think there are ways to take all of the things that you would otherwise do if you had unlimited time and make more of them possible using AI before you meet with someone. Once you meet with them, And that's, I think, where you get the best signal because that's where you get into the vibes, I think.

28:59Yeah, that's good stuff. And you know what, for the recruiters listening to this, I actually think this is a very optimistic situation because it means that, you know, that huge efficiency decade, two decades where we had, okay, everything is getting more efficient. We went shifted from phone to email. Everything went digitalized. We were processing more and more data, et cetera. And the human element kind of retreated significantly in the recruiting process in all of this because actually it was all about managing systems. You know, recruiters ended up becoming reasonably technical, you know, good at Boolean searches, the interrogating systems with, you know, scripts and what have you.

29:42Not necessarily human related job. perhaps the world of recruiting becomes more human because it's it's about having the interactions with high potential people and then figuring out okay how can we unlock that potential um and this was basically recruitment vintage you know this is recruitment as it was prior to uh prior to the internet yeah really you've discovered the high potential people David and you want to now assess them and I guess this goes to the meat of our conversation um how was the assessment for First, let's set the scene. How has the assessment of software engineers changed pre-AI agent, post-AI agent?

30:20So can you give us a quick sort of overview of what a technical assessment looked like before AI, specifically before AI agents, and what does it look like now? Yep. Yep. So before AI agents, I was much more likely to test people's recall about specific behaviors of code and of software that are directionally accurate to their experience having written code and software. things like oh you say you're an expert in Python okay let's talk about list comprehensions what's a list comprehension oh I guess you're not an expert in Python so those kinds of things were were very useful just as as quick filters

31:15post post AI let's take that list comprehension example you're not writing list comprehensions anymore. So think about what is Python useful for? I might ask a question about, okay, like if you're starting software from scratch, like what kind of languages would you pick? If they can bring judgment to that question and say anything other than, well, I asked the agent to tell me, well, I could ask the agent to tell me, what about you? Like, do you bring any value to this equation? Or are you just like a proxy for your agent? Aligning those sort of tactical questions along with the level that people would be working is, I think, important.

32:02But I think I really have moved away from questions that have very specific answers that I'm looking for and much more towards very open-ended, very conversational questions that explore three particular themes of skills using AI. So one of them is delegation. How much do you delegate? Do you not delegate enough and foreclose on productivity gains you get from AI? Do you delegate too much and end up with something to review that is too big to review or impossible to review? How do you prompt, right? So are you baking your judgment into your prompt, or are you asking for something that could be reasonably interpreted by the AI to be something not an all that you want?

32:58And then how do you verify? And that's probably the most important one, is verifying things that are mission critical in very, very different ways than things that are not mission critical. Verifying ambitious projects in very different ways than you verify small projects. Knowing when an agent can verify your output versus when a human actually has to verify your output. And then maybe even building your own verification software for you as the human to verify the software. I have literally done that three times in the last few months because that's what I needed. I needed something to leverage my verification ability.

33:46So I think what I would say in general is that your ability as a recruiter to flag someone as a good candidate to pass along is governed by a few things. Number one, your experience and skill with AI, because you have to have the vibes there in order to be able to feel the vibes in someone else. Number two, the tailoring of those questions around prompting delegation and verification to the specific role. Is this a healthcare role where verification is so much more important? Or is this a scrappy startup where delegation is a much more important theme? And then trust your gut. Ask questions that you can put in the recording, pass along to the hiring manager, and then they can verify the technical stuff.

34:52But it's like you said, Hung, it's a very human process. yeah very very interesting um everything you've said there i'm not hearing technical tests in this um so are we i mean i think we can all agree that take-home technical tests might be done um simply because like what are we getting back we can't we can't verify that but what about online technical testing you know we've got some great products out there historically software engineers have relied on that as a recruiting server to do tech vetting. And these days they're offering things like proctoring and, you know, anti-AI type detection stuff.

35:33Do you think they still have a place in this world or do you think actually, you know, we've moved beyond the testing for stuff we don't care about anymore? Like, what's your take on that? I'm so glad you mentioned this because I actually disagree with the idea that take-homes are done. um i used a take home for i mentioned earlier that i was doing some interviews for one of the frontier labs we used a take home for that process um we had people work for a whole day on a take home um now we took that product and here's what we did we did two things number one we had AI vibe coded. So we had everyone build it.

36:20We had AI vibe coded in one prompt, and we built a bunch of different verification tools for the result. We tested the AI vibe coded result against the verifiers, and it got nearly a perfect score. So okay, if all you do is ask the AI to vibe code something, you're going to get a nearly perfect score. Turns out three quarters of the people who did the take-home got worse than the vibe coded score. So what does that tell you, right? That tells you that the human in the loop actually resulted in a worse outcome than just the AI on this bench. That was a very valuable signal for taking people and saying like, you know, we probably don't want to pass you along to this lab because if you, if your involvement results in a materially worse outcome than just the AI alone, that's not a good signal.

37:23So that's number one. Number two, we interviewed them after the work assessment and actually tested their understanding of what they built. Because if someone just decided to Vibe code a solution, we also don't want that. So we tested their understanding of what they built at a very fine grained level with questions that they would not have been able to answer well if they vibe coded it. And we actually had someone who got nearly a perfect score but got knocked out of the process because he couldn't answer any detail-oriented question about his trial and it was clear that he vibe coded his solution.

38:04Yeah, very, very interesting so it's almost like don't throw the like don't mistake the the assessment for i guess it's not it's not repurposing the assessment but rethinking how you get value out of the assessment um yes like 100 priorly it's all like hey did the guy execute sort of uh they produced the code um and then you know create the solution we want check the code line by line see how it looks you know these days of course it's all going to look perfect quote unquote um But can the person explain it? And can the person's output be tested against something that you can generate on the fly?

38:43Very, very interesting. I don't know how people feel about that. Do we feel good about the return of the take home? I don't know. I think maybe so. Because I'm not sure with that embracing of the idea of proctoring sort of coding scenarios, that creates kind of like a very artificial way in which work is done. And I do wonder whether that actually creates, again, a bunch of signals that may not be accurate. When you're assessing someone in that high surveillance type of manner, are you going to do the best work? I mean, I don't know. I don't like having my boss look over my shoulder. I don't think anyone else does.

39:22Exactly. I mean, just the sense of performance. I mean, I think even people who are doing non-technical work can understand how that can put you off your game. Imagine you're a salesperson making sales calls. How would you perform if your boss sat down next to you listening into your calls? It's going to affect you one way or the other, and it will not be the same. So what are we even assessing? Okay, very good. How do we score this, David? So you've produced all of – so this is the way we're going to do it. We're going to have people do take-homes. They're going to produce these things, and then we're going to try and say yes, no.

40:00How do we score it? Is it possible to quantify or do we need to move to the kind of a traffic light system because quantification here, you know, we're just making up numbers? We tried a bunch of different verification methods, a bunch of different sort of go no go heuristics and tools for this process. In the end, the one that gave the highest signal, I think, was a person sitting down, looking at the results, looking at the material and making a call. There were like we tried to have the latest model make a call. And it ended up being so overly conservative that it was entirely unfair to the majority of candidates.

40:54Meaning they were too harsh a judge? Yeah. If you ask an agent to make a yes or no decision on hiring someone, in my experience, it was too harsh. Because the behavior that I saw, at least in that context, was that it was too likely to punish any mistake. uh but and it was too likely to treat this mistake as just as impactful as that mistake when in reality for as a hiring manager i would totally excuse someone making this mistake five times but i would never excuse making some someone making that mistake um and i think this this For me, this only reinforced the importance of proper verification, not actually delegating the final decision to AI, not delegating that final judgment to AI, but using AI to give me as much high signal information as possible so that I could feel very confident that I was making a good judgment.

42:05Yeah, very interesting. The ability, the weighting of the mistake, right? So AI agents, I guess, to date might perceive any mistake as egregious, whereas in fact, they may be trivial. And yeah, as you say, forgivable, like all fixable, something that doesn't tell you there's a fundamental error in how this person is operating. They've just made a slight mistake. Very, very interesting. Um, quality control. Um, is there a way in, actually, let's have another question here, um, on kind of team and team fit, um, software engineering, at least pre AI was often considered to be a collaborative exercise.

42:54You know, a bunch of code is each contributing different types of code to different parts of the system. All systems need to, all the code sort of contributions need to work together. And the best way to do that is the engineers need to work together. Is that still the case in the AI agentic world? Do you think team play has gone up or down in terms of value? and if so, what's the impact on how we assess team fit? I can only speak to my experience leading the engineering team at G2I.

43:38Something that I just want to caveat to say that I think there are infinite effective ways of running a team to build software. So I don't want to put what we've done at G2I on a pedestal, even though I think we've done a rather good job. But my experience at G2I is that team fit and team collaboration is just as important as it ever was. And there are ways where you can use AI to have even more effective collaboration. So one of the things that we've done is integrate a documentation authoring process into the way that we write the code so that you sort of get documentation for free. And then when one engineer is working on a feature, there's already documentation that exists that describes how that feature relates to something else.

44:38the agent reads it for that engineer and now you have agent to agent collaboration that doesn't require a person to do anything differently um this is uh this i i think has been really effective especially for teams that are newer to ai is um using a system that begins with documentation because it puts front and center both AI's capabilities and its limitations, where an engineer is like looking at a doc and they're saying, oh, well, this is really, really good. This is good. Oh, what is it talking about there? Okay, well, now I just learned something about how to use AI. And then, but at the end of the day, I think the best software teams are teams.

45:36I think if you have a group of software engineers, the collaboration style, listening skills, proactiveness, you know, just trust building, none of that has really changed with AI. We're still people, right? Yeah, fantastic. It reminds me on the documentation side, it reminds me of the perennial joke in coding, which is, you know, a line of documentation where it says, don't delete this, don't delete this line without speaking to Bob. Right. It's like, you know, that's Bob. The line of code stays there forever. But, you know, it's fascinating to see how this sort of communication layer, agent-to-agent communication layer might kind of be integrated in how code is produced.

46:27and I think inevitably it's going to change the way kind of the structure of teams and do you think it will fundamentally flatten teams for instance because oftentimes the requirement for someone to coordinate and to you know ferry information up and down may reduce does it make teams smaller fundamentally and then maybe you have more small teams to work on different systems or does it enable actually mega teams? Because, you know, the problem with the mega team has always been the comms. So can you solve that with AI now? What's your general sort of take on this? Well, Meta tried to create smaller teams and then they backed off from that strategy.

47:13So I think that is, for me, reasonable signal that we are either nowhere near ready to have teams produce the same amount with fewer people or it's not worth pursuing and it's better to take the teams that you have and make them 10 times more productive using tools. The question of super teams, that's an interesting one. I think it's, if I think about this as an engineering manager, I think, like, what are the things that certain people have to say to other people in order for the work to be done well?

48:00And then you get to a question of, okay, do agents help with that? Yes. Do they remove that need? I don't think so. I don't think we're quite there yet of being able to substitute agent-written documentation for the creativity and the spark that happens when two people talk about how to build something. So I think I tend to fall maybe more on the conservative end of things in terms of organizational change from AI. I think there's so many more opportunities for us to improve what teams are able to accomplish simply by improving the team's ability to use AI to complete their work. That we have a lot of run room to go before we start fundamentally reconstructing teams and potentially introducing unknown regressions.

49:05yeah absolutely and we i get the feeling these kind of cultural innovations will become emergent anyway rather than through any like design i mean i'm i've always been a bit of a cynic on the concept of job design you know i don't think jobs are ever designed you know it kind of emerged through the gaps um of of the system that we have you know um there's a really wonderful um uh a museum tour in London here, the Tower Bridge, where they built this architectural monument, Tower Bridge. And within it, you see a number of sections which describe the jobs that emerged from the creation of this bridge.

49:46You know, you were a stoker in an engine room. You're a lamplighter keeping the lights open. You were a clerk, you know, making sure you took the records and everything. And half those jobs would not have existed if the technology system worked perfectly, but they emerged because it didn't work perfectly. You needed a bunch of humans to come in to triage it all. So I think we have to have some faith in the emergent capabilities and maybe not feel that we need to design everything because a lot of it would just happen more organically. I totally agree. Go ahead. Just prompt engineering is a perfect example of that, right?

50:27Because we had these agents, everyone had this theory that, oh, if you just prompt it well, then you'll get transformative results. So we tried to create this job based out of a perceived need, but it never fulfilled. We never saw the ROI from the role of the prompt engineer. So then it disappeared. Yeah. And basically, AI sort of innovation, its own sort of innovation cycle filled the gap, whereas the human, usually it's slower. And this is one of the bigger things that of this revolution, which might kind of this time is different type of scenario is that, yeah, human beings may not be able to respond fast enough.

51:06So some of these gaps will actually will appear and disappear faster than we can we can we can respond to it. That's a conversation from the time, though. Looking at the time, we're running rapidly as predicted. We've got so many more things to talk about. I hit two more notes before we let you go, David. I want to talk about sort of career progression for a software engineer. I think that probably has changed sort of pre and post AI, so to say. We might need to do different things and look for different things and develop different skills. Can you tell us a little bit about how you see career progression for a software engineer in 2026 and beyond?

51:47Yeah, I mean, I haven't seen any evidence that the latter changes. or that the things that are valued at different stages of the ladder changes yet. We might see that in time. What I have seen is that it's possible to climb the ladder a lot faster if you bring good judgment, if you're a really good learner, and if you dive headfirst into effective use of AI. I've seen that firsthand. Yeah, really, really good to know. And in terms of things like career development, for the engineers out there listening to this, um what kind of things do they need to do to you know understand okay this is how you you move forward in your career um and and become you know from from junior to to experience the senior to lead and so on yeah totally i think one uh number one side projects always always always side projects uh listen to podcasts so you're listening to this podcast you're doing a good job listen to more podcasts.

52:50Listen to podcasts that you don't understand everything that's being said. This is very, very important because the more you listen to a podcast that's just beyond your reach, the more you will learn. You'll start understanding maybe 10, 20, 30 percent, and then you'll start understanding 80, 90 percent. And suddenly you're saying the things that you learned on that podcast in your role and people are very impressed with you. Do that. Lots of podcasts. I think in terms of personal projects in this spirit of skill with AI, try to build something that you could not possibly understand how to verify if it's correct through the code.

53:31And challenge yourself to verify if it's correct from the result. I think this is worth trying. And can you unpack it for me as to why you think that's so valuable, David? Yeah, because essentially it forces you to get better at verification.

53:53I would hang my hat on the idea that the most important skill in software engineering right now is verification. At intermediate levels of implementing a feature, at the end of implementing a feature, gradually as you change software over time, verification of code health verification of software health verification of user patterns the the more creative you can get answering hypotheses about whether or not your software is good uh the better because that is what allows you to branch out from being hyper-specialized to taking your judgment and generalizing it across many more contexts. Yeah, very, very good.

54:48Final question, actually not the final question, it's the penultimate question, David. For the recruiters out there, professional recruiters out there, they may be heads of TA, they may be people kind of working for software companies, recruiting engineers. How can a non-technical person contribute to this changing scenario where we're looking for, as the title of the show says, the skillful use of AI in the context of software engineering? How do you think a non-technical person can support this process? What role do they play in verification, if at all? I think the first thing that I would do, if you haven't already, is to talk to your hiring managers or hiring managers that you know about what does skillful AI use look like on their teams.

55:48Because I can promise you we will be different. That skillful AI use in a legacy code base will be different than a modern code base. That will be different in a healthcare highly regulated space versus this. And I would hazard a guess that you can find ways of assessing someone's, you know, tendency to be able or be likely to use AI in that way. um i you know i and i'm speaking this primarily from first-hand experience having used ai a lot not necessarily from a technical standpoint i tell the difference immediately between an ai cowboy and an ai regulator that just stands out immediately to me and those two people are both very valuable assuming they're both skillful with ai just in different contexts so I think it's again a lot of this is new we're figuring it all out in real time we're all learning in real time embrace the learning embrace the ambiguity and try to train your vibe so you can detect other people's vibes yeah I love this I love the direction that recruitment is going by the way simply because it is going back to a sense of intuition that comes from science.

57:17It is not sort of one or the other. But in fact, the two things are actually completely intertwined and may be actually the same thing, just with different language. So yeah, very, very good. Final, final question, David. What's the one question that I haven't asked you about skillful use of AI that I should have done? Any sort of last things that we haven't covered during the course of this hour we've spent? I think that's a bad, I'm going to take a minute to think about that one. You've asked me some very good questions. I think one question that I like asking people is how did their trust in AI change over time?

58:13It was literally the subject of an AI engineer talk that I did. I did some research on my co-workers and I asked them, hey, how did your AI change over time? And I think it, for me, both for myself and for my co-workers, what we discovered is that it swings on a pendulum. that everyone starts from a different level of suspicion with AI. Some are like, yeah, let's go, let's use AI. And some are like, I don't know about this. And as you use AI, you have successes and you have failures and that swings you on a pendulum until you eventually converge into this place of like, okay, AI is really good in some situations, it's really bad in other situations.

58:56And I think that that's also potentially like a useful question to ask in a recruiting capacity, right? I have a limited sample, but every single person I've talked to who I would trust to teach me how to use AI is someone who takes a measured approach, who knows when it's good and knows when it's bad. yeah fantastic and i love the pendulum analogy at david uh lots of things and we need to reach into analogies and metaphors to understand this crazy world that we're entering into but we are running out of time we have run out of time uh we might be setting this up for a for a part two later on down the track david let's leave it at a a final comment about g2i what is g2i's business why should recruiters care about it who how do they find out more if you're training a models go to g2iai if you want to hire world-class engineers go to g2iai and if you want to connect with me you can find me at linkedin at david house dev fantastic stuff brilliant conversation david good luck with everything you're doing i love sort of your introspection and your ability to articulate this new world of technical vetting i think a lot of people will get value from this conversation and from following you uh for further commentary on this topic um brilliant stuff that's That's it, everybody.

1:00:20Thank you for watching. I hope you've enjoyed it. We'll be back next time, hopefully with a round two on this, because I think this could be a brilliant series to continue to do. Okay, that's it. David House, it's out for me. Thanks for watching, everybody. It's been a pleasure. Thank you.