You can't have AI without DevOps | GitHub’s Martin Woodward

26:33But that requires like a big shift in how teams like are aligned in their practices through DevOps, how they communicate and coordinate and assigned work yeah um are there like patterns that you see emerging in teams that are successful in the way that they co-work with an llm that maybe stands out to you yeah there's a few different things one again what's fascinating and we saw this already with like the early versions of copilot is that if you wrote better comments copilot did a better job for you of coding and so it kind of tricked you into writing better comments and and better interface names and better documentation because the more context you gave the model the more it was able to help kind of thing yeah we see a similar thing happening i mean it's early days again you know we announced it like as we're recording sort of three weeks ago but we'd obviously been dogfooding it for a while and dogfooding with all the teams for a while what we're seeing is that you spend a bit more time defining your issue uh because you find that you're writing it up less as an aid memoir for yourself but more of like a spec almost you know right like how what and actually like this is this is the whole freaking point of DevOps like it's to have that conversation early with the customer of like so what does this need to do what happens in these cases you know all that stuff we've been doing since TDD and then through Agile and then through DevOps it just kind of forces you to do that again because you're you want to write instructions down in a way that not just you'll get and figure out, but that an AI can help figure out for you.

28:10Or a human reviewing that code can figure out for you. So it encourages those kind of best practices. So that's the first thing that we see, which is kind of like accidental, but also cool. It makes me think, oh, maybe this AI is like a good thing. Maybe this is, you know, this is definitely helping. The other one that we see is custom instructions are the secret source that not enough people know about yet. and yet the teams that get the most out of Copilot, that's what they're using. It's basically a file you can have at different levels. You can have it personally in your local machine. You can have it in the repository or you can have it at the org level as well.

28:49And it's additional instructions that get passed to the prompt as part of every single indication. That means that there's a few things. One, you can define a personal style. For example, I make sure all my comments are in British English and that I add use in random places to words. I don't use emdash. Like, don't generate comments with an emdashing because I never use emdash. Like, why would I use emdash in a comment? You know, and some of those sorts of like personal quirks. But more importantly, at a team level, I can specify, hey, when we're using, when we're doing a React component, let's stick this.

29:25You know, we use a flagger as our internal feature flagging system. but say you're using something like launch darkly or something like this like wrap this within this feature flag automatically for me because that's our coding standard when you're logging use this logging framework when you know you and with with examples and by doing that it allows the code that's generated by the llm to not just to reflect where you want to go with your code base not the crap it finds in your existing code base because we all know our code bases are currently terrible and we want to make them better you know if an llm's context is just what it knows from what it's been trained on plus what it can detect from your code you need to give it additional instructions to also go and this is where we want to go this is how we want the future to be with our code base so that it's continuously improving continuously making things better that's probably the main trick i see not enough people have kind of figured out yet but yeah yeah there's two things i want to double click on i'm going to do them in in reverse and the first one is the one you to talk about the instructions because I want to echo that as a practitioner, someone who's using these tools every day that like the GitHub copilot instructions and being able to add that personal style level to to what you're prompting just kind of baked in.

30:40It really helps. It reduces like the iterative cycle of going back and forth with the LLM. But what it does actually is it forces you to understand what's important and what's not. And then that gets to the main point of what you said. And I think you said so well, something that like has been bouncing around in my head a lot. And you really like nailed it about how AI is kind of in a way tricking us into adopting and revisiting all of these like really great baseline foundational best practices about how to make great software. And, you know, doing things like TDD, building a spec that we can all agree on and has the full idea of what's supposed to be accomplished and really engineering for impact, right, for how folks are going to use your software.

31:23And LLMs, they require these guardrails. And so we have to go back to like the basics and really write that stuff out and make sure we're all on that same shared page. So it makes everyone a better developer. And it seems like the key to going fast is to be really strongly aligned with what you're building and using things like tools. It's in communication. Again, that's why I think the teams that today do DevOps well are getting the most benefit. it because the better you can communicate, the better an LLM can understand your intent and the better it can execute on it. But also, the better your team can understand the intent, the better your customer can make sure you've understood their intent.

32:06So communication is key and it just gets more important with the advance of AI, which again makes me very optimistic against some of the doom and gloom type scenarios and things like that i'm like well if it's tricking us into communicating better and tricking us into you know having better documentation and can actually help me create good documentation and can help me create good tests and can help my developers who don't speak english as a first language understand requirements that came in in english and they're now want to understand them in their language like these this is all this is all goodness you know right exactly and if it makes us all work better then that's the direction that we should all go in and that kind of goes towards the next topic that I want to dive into with you about like the skills that you see emerging for the developer of tomorrow.

32:55Like we're moving into a really a new skill set that developers have to adopt. And this is where it gets really interesting. It's where there's so much opportunity. And it's where like the folks who are listening to us now, people who read and listen to Dev Interrupted, this is what, you know, we're all on the cusp of figuring out together as developers, as engineering leaders of what are the things that I need to do for myself and for my team to make sure that we're upskilling and getting ahead and taking advantage of where this is all going. And in your keynote, you touched really well on like the evolution of LLMs from being like autocomplete to being more multimodal.

33:30And then you're moving into an AI native environment. And what you've iterated here today is core to that, that those teams with strong DevOps practices, they're the ones that are best positioned to take advantage of that. Can you walk us through what that would look like in practice for a team and how leaders could consider maybe revisiting or taking DevOps more seriously so they can mature into this new era? Yeah, I think if a customer could come to you and explain exactly what it is that they want and how to best implement that with technology, then an LLM could probably do that task entirely on its own kind of thing.

34:14But actually that's never the case for us. It's like why I love technology so much and why I enjoy it every single day is because we're solving problems. Like it's that creative problem solving, being able to parse apart what a customer is saying and like, okay so if i did this this would solve it in the minimum number of steps kind of thing and this is what computers can do and then also all those skills you have to build up over years of like how to break down the bigger the problem is the more you need to break it down into small steps so that you can iterate as you go and you can increment and you can make sure you're heading in the right direction because what you think is the answer at the macro level definitely isn't the answer when you get down to that small level.

35:05So that skill that you already kind of have, but it's hard to build of taking down a big multi-sprint activity into individual day tasks kind of thing that you can get done. You need that skill even more in the world of AI, because if you just give it a big, build me one of these things, it's going to try and it'll spend a lot of tokens and it'll spend a lot of time doing it. But it's more likely to not be exactly what you want because at any point along that random you know interest entropy path that it was doing to help interpolate what you wanted it can make a choice and it might make the wrong choice just like humans do and so by being able to figure out okay how do i break this down what components do i need how am i going to create those and what you do is you know you again you create issues for these you create pull requests for them but the the major difference is that you learn where to lean on to the AI for assistance.

36:03So, hey, well, this sounds like, like, you know, create to me a set of test cases. Well, let's let the AI make the skeleton for that. And then I can go see, go see what Copilot missed and go see what other edge cases I need. And actually, oh, that's now making me think about this problem even more. Let me go talk to the customer to see if this is an edge case or not that I need to take into account. And it just allows you to be more creative and kind of freeze you from the typing toil, the implementation toil and allows you to focus more on the kind of creative problem solving so you then as a as a team you have to make sure you're building those skills building the like how do we break this down into smaller chunks so that we can then have humans build some parts you know co-pilot help with other parts collaboration between co-pilot and humans for other parts yeah literally eliminate all the typing i even now prompt my llms with voice to text and so just cut out all the typing Yeah, yeah.

37:00I mean, I probably do it even when I was in an office. I'll be one of those people. But everyone can just see how they can just see how I blender through my prompts in the open. You know, we can all learn together. Which actually brings me to another point that I really was curious to know about at GitHub is how do y 'all foster like a sense of continuous learning within your engineering team? Because this is like a total like sea change event. And y 'all are leading the charge. So how at GitHub do y 'all celebrate and create this environment for continuous learning? Yeah, we call it growth mindset here at GitHub.

37:33You know, it's a kind of a term I think a few people would use. But it's sort of we want to be a learn-it-all company rather than a know-it-all company. You know, we want to be consistently learning. Now, luckily, because kind of at the heart of GitHub is the open source community, a lot of people we hire kind of come with that attitude anyway. they assume that somebody out there is smarter than them and so they can kind of you know figure out like do some research and figure out what the best way of approaching these things is that said like getting like we built copilots i've been using copilot what since 2021 ish i think i've been using it yeah for a while now you know it's like the the sixth person to try copilot in the company wow and it was cool but like going from six people using copilot to you know the entire engineering org using copilot didn't happen overnight what we find with adopting llm tools is they you know the mores kind of innovation curve where you have like early adopters then you have the innovation gap and then you have you know early majority late majority you know all that sort of stuff.

38:41Unlike DevOps deployments and unlike kind of rolling out TDD in a team or rolling out Agile in a team, which is very much a team by team, one and done kind of way of deploying with Copilot. And with, I think with all AI tools, it follows this more adoption curve because individuals have to take the time to start using it in their workflow and start learning how to prompt the ai learning what where copilot can help and so you get to 25 kind of 20 the people that listen to the show they're all early adopters yes and so they're the people that are just going to grab copilot grab any technology give it a try see how it can help amazing because they're listening to the show but then like if you think about all the people that you work with like they're busy and they maybe don't want to be trying they've got an environment that works They've got an infinite backlog of work that needs to be done.

39:35Like, when am I going to find the time to figure out how to use this? And so helping them find that time and helping them share when they come across something. Like, you've obviously had some amazing moments where, you know, Copilot or other LMs have kind of like surprised and delighted you. Finding a venue by which people can share those is invaluable inside of a company as well. we have a i'll say stuff copilot did like kind of channel where people can share like excitement or like amazement to like look at what i just did for me this is crazy kind of thing because people are going to believe those people a lot more than they're going to believe in me saying because you know saying copilot is amazing though of course you think that you know you're a vp you just delete emails all day you don't really code yeah exactly let me talk to you really codes yeah so we do some of that yeah but but so we we had to work on getting internal adoption even for us and took us a while to get to the point where the you know the vast majority of devs are using copilot inside the organization so if we did like be prepared that as you're building those tools inside your company you're gonna have to do real change management like help people come along on this journey with you because what we see is that people are using copilot like the numbers we have you know like what 50 % more effective they're 80 % happier like there's all these ridiculously big numbers that we have from doing surveys and stuff yeah which is cool but then the people who are not using that in your organization they're going to miss out on that like that's not fair on them either and they're not missing out on it because they're bad or lazy they're missing out on it because they're busy and they need the help to get on board you know so um yeah so that's one my biggest things be prepared for kind of it being an adoption journey not an overnight everyone's going to grab it and do amazingly well tomorrow you know yeah no it's it's really insightful to hear how like github itself is dogfooding and adopting its own tool and when y 'all were doing that process and obviously you're still in that process too because we're all still figuring out where it's going are there like things that you look at to really understand the impact or like the productivity or the developer experience gains are there like go-tos that you really measure as you as you have adopted this across your org yeah the primary measure that we take is around like code acceptance like how much people are accepting the suggestions from copilot and using them and this usage as well is another key metric like you know are you uh a one day 28 a 10 day 28 28 day 28 kind of user you know um how many people like a histogram of where we're seeing that happen inside the business and then what areas people are using and other metrics we're looking at are things like well that's cool that's usage you know but like what's what's happening to our pr merge times what's happening to our security vulnerability uh response times you know are we seeing correlative improvements in those alongside the percentage adoption of copilot that we're seeing and turns out we actually are like even if i look at across all of github like every user on github i can see we we introduced you were talking about getting copilot into the hands of people that's one of the reasons why we made a free skew of copilot available so that people can just try copilot and not have to ask permission you can just give it a go when we introduced that i i can see the graphs i can see like this marked increase in the people using copilot and this marked decrease in globally not just in github but globally the average pr merge time and also an increase with the amount of prs being created like it's and it's is day and date like it lines up exactly with that graph of copilot adoption that's cool so so the pr merge so it's not just the number of prs going up is like is cool but the size of prs is going down and the amount of time it takes to merge a PR is going down.

43:44The amount of time it takes to merge, to fix a security vulnerability from it being detected is like gone from sort of weeks to hours. You know, it's like, it's ridiculous how much it speeds people up that way. But again, back to our like old school things, all in positive ways that I wasn't sure at the beginning of this journey, like would PR sizes go down? I was a little bit afraid they would go up because now LLM is helping me. Just show me everything in. Most of PRs. Yeah, exactly. But it turns out, no, people need to iterate more. And so their PRs actually get smaller. So cool. That seems like a positive thing, but not something that we could have known for certain until we started on the journey, you know?

44:28Yeah, it's almost like the global adoption of the AI that you can see on trends. It helped everybody move into a better development work style because we all know that having smaller very focused prs you know it's like a best practice and being able to iterate on them you don't want a massive pr that changes like eight different things so you see these like best practices get like globally implemented as part of like the llm adoption so it makes me excited about like how much more efficient and safer engineering can get in the future as we continue to like adopt these tools which kind of brings us to improvement in communication because because Because you're getting now to build, you're having to communicate more.

45:06And so because you're communicating with an agent, with Copilot, rather than with another developer, that's what's forcing you to actually make those smaller PRs and things. Because if Copilot gives me a monster change, because it's my job on the line if that change breaks, like I'm the one who's going to get called out. So Copilot is not going to respond. And well, you know, sometimes it does, but it's not going to answer the bleeper at 4am. You know what I mean? Like I'm responsible for this code. So I have to do that code review. I have to make sure it's good. And the only way I can do that is if I do smaller PRs and make sure this code is good.

45:42It just goes to say that having those fundamentally good best practices is going to help you move faster with these new tools. I think so. And it kind of brings us to also to like a larger question of or rather opportunities that I see within like the world building new technology, having this like democratization of development with AI and seeing a boon around things like open source. You know, Martin, you and I both like are from an open source world. We're both very, very, very pro open source. And I'm curious to know your perspective on how you see open source contributions and communities evolving in an age of AI enhanced development.

46:17Yeah, one of the things that we're, I mean, we make Copilot available. So Copilot is available for free to anybody. But there's limits in terms of like how many requests you make and things like that. So, you know, but for a hobbyist stuff of doing open source work, it's plenty. But then what we also did is we made Copilot available, Copilot Pro, which is our page queue. We made that available for free to, it's like over a million kind of students, teachers and maintainers of popular open source libraries. Because again, we want to just get it in the hands of people to help them improve how they can build open source.

46:53So we do that. And what we're actually seeing is people really starting to use it. Like we've started enabling a few projects with the coding agent and we're sort of scaling that out. And people actually, you know, using that, learning how to use it, learning what instructions they need and what actually the improvements they need to make to their builds and tests to then allow Copilot to automatically run the builds and tests, which allow Copilot to make better suggestions and things. Again, better DevOps. So we see a bunch of that happening. We see the ability to respond to security defects actually dramatically improving from it as well.

47:34Because again, from an open source maintainer's point of view, they're busy. They're trying to build stuff on the weekends and maintain stuff on the weekends. So if there's help available for me to do that, then that's great. And then the other area where it's actually hugely beneficial is code review. so one thing that copilot can do ridiculously well is tell me what this code actually does not what the human is telling me it did it is supposed to do and then spot the delta usually from that that helps you with a few ways one is it sees what the code does and helps find defects helps finds bugs where it can see what the human saying they intended and actually what the code does is a delta this is probably this bit of code that's wrong like somewhere um yeah what it can also do is spot those um you know there's the human says this fixes an important security vulnerability in your library mr open source maintainer code buyer says this adds a bitcoin mining uh wallet stealer or something like that you know right and then the maintainer's like oh maybe i need to take a look at this code a bit more than i was going to and it makes it harder to kind of sneak those things through.

48:48So there's a whole bunch of ways there. I think it can kind of help improve open source. But when we first introduced these tools, we did see, it's calmed down a lot now, but early days we did see a spate of actually open source maintainers getting quite a lot of, we politely call them low quality submissions. Some people will call them spam or whatever, but they're not really spam. They usually buy well-intentioned people pointing an LLM at a project and saying, do something for me without the skill to know if the answer was correct or not. And then submitting it anyway, thinking they were being helpful kind of thing.

49:29And then because, you know, Copilot does a great job and all LLMs do a great job of like sounding authoritative. Like the person sort of the maintainer has got to review that and take a look. and what was it doing? It's harder to detect that the thing that this person's coming from isn't as skilled as it might sound like they are based on their PR descriptions and things like this. So we had to do some work there, a bunch of education. Thankfully, things have got a lot better, but it's something that we're definitely, you know, we're being stewards of the open source community. So that we're very, very mindful of is that we don't, we don't accidentally overload these maintainers with you know we don't make it too easy to overload these maintainers with low quality submissions kind of thing so here's a bunch of work happening there to try and help improve that and like make sure that the tools are helping reduce workload from the open source maintainer and don't add too much workload to them yeah and one of the last things i want to touch on here is it's kind of like that persona of that person who might pick up this tool and not have the coding knowledge or understanding of the code base and and you know use the llm and the llm is very authoritative it's convincing and it's easy to fall into this like false trap of understanding what it is that you ultimately prompted through and that like you said creates friction and some like things like open source projects it also creates opportunities like for junior developers of actually really understanding what they need to understand in order to have an impact, right?

51:01And so in our listenership, we have a lot of active developers, a lot of leaders, but there's also a lot of aspirational developers and folks who want to try and use these tools. And I'm curious, what would you, in your position, say to someone who's maybe more of a junior developer about where they fit into the developer world now and what they should focus on for being the best developer they can be tomorrow? Yeah, that's great. First of all, a warning to those senior devs. I see a lot of snobbishness that goes around. It's like, oh, yeah, I can use these LLMs, but that's not for the juniors.

51:33That's not for the kids kind of thing or whatever. That's one of the less experienced people because they're never going to understand what this stuff is. They're not going to know how to use it properly. I'm like, no, sorry. The people who are coming on are just as smart as you. They just don't have the experience that you've got. And if you think about your, like, I've been around. I've got no hair anymore. for people that are not watching i've been around a long time and when i when i first came into the difference between when i was building stuff at home as a hobby to when i came into the workplace is that i went from building code that i completely knew inside out knew everything and i knew every single line of that code to being in a code base that was like just didn't like there was bugs in this code base it was by other people it was just did stuff and like having to understand what this code did, this next round of junior developers are going to be a lot more experienced to that.

52:33And to be honest, that's how most of our real work is because the LLM helps create the scaffold form, helps create some work. And then they're trying to figure out where it doesn't work. Like they're trying to figure out the debugging skills. They're trying to figure out all those sorts of things. So actually, in terms of the work that we do, I think they're in an ideal place. In terms of answering your question around like what skills, learning about the different technology stacks, different architectures, learning about what different languages are capable of doing so that you can, there's less of a barrier now to go from say Python to Java, to go from like this amazing, you know, whatever the JavaScript framework is that you're familiar with to a more recent one, to one that's more appropriate to this particular task that you're using.

53:19So the more you can build up your knowledge of what frameworks are out there, what languages are out there and what are the right types of solutions for these types of tasks i think you've been better positioned to then be able to guide the ai to give you the right kind of solution rather than doing what we did as developers which was i know how to build one of these how do i solve a problem with what i know how to use you know what i mean so it's a great time to be a junior dev i think obviously um you know bearing in mind like all economy shifts and everything else like that but um in terms of the skills like if i was a kid now learning to code if i was you know switching careers and learning to code the fact that i can like highlight some code and have it explain code to me in my language is amazing like i had to go to i had to order books from the library and wait two weeks of them to arrive and then read the books you know yeah and then those books are barely related to the problem right you had the guess of like how that overlapped and then and then you had stack overflow where you did the same thing oh how does this person's question actually relate with what i was doing and so being able to have that personal answer is is incredible but also learning that it's an unreliable narrator that it's a fallible narrator explaining that code to you as a human is and that then helps you as you're building your career because i'm sorry i i probably give just as many wrong answers as i give right answers when people ask me questions you know and so it helps you build up your internal molog and your internal skepticism of like, okay, how can I prove this?

54:50How can I test this? How can I break this down into small chunks that I can build up that are testable that I can increment with kind of thing? Yeah, I think that's a great way of looking at it. And so I definitely take that to heart. I completely agree with you. And Martin, we're coming up on our time here and we've had such an incredible chat about where the future development is going, how y 'all have been building GitHub agents and Copilot and your own opinions on how DevOps practices are able to kind of accelerate that adoption. I completely agree with them. I know many of our listeners will as well.

55:23As we wrap up, where can folks go to learn more about stuff that's top of mind for you? Obviously, everyone knows where to find GitHub, but are there resources that you want to point them towards that you think are really salient? Yeah, well, if you want to, gh.io slash Copilot, and we'll provide links in the show notes, is a link to kind of all the co-pilot stuff so you can learn what's new and then following along with the github social so you can kind of the the pace of change is ridiculous like we add new calls every week it feels like and so keeping up to date with what shipping is probably key folks can also feel free to we've got a conference coming up called universe towards you know in um october time frame there'll be a bunch of cool stuff shipping then there'll be lots of announcements then as well so i would I would encourage people to look out for that.

56:09And then finally, like martin.social is my blue sky handle, but it's also my website, the links to all of my socials and things. So people can feel free to just go there and, um, and figure out where to follow me if they like a British kind of accented to their tech news. And that's always good as well. Great. That's awesome. We'll, we'll make sure we get all of this in the, in the roundup. So folks can go check it out and get hub universe. Is that what you said is later in October, right? Is the big one yeah i'm i'm pretty keen for that one i'm i'm excited to see what happens maybe maybe dev interrupted will be at that one too so i'm excited to see person that'll be amazing yeah maybe we could maybe we could do this in person and and have like a chat because because like what you just said this stuff's going so fast so um we're gonna have to keep talking about it i think because there's just so much more to uncover so folks thanks for joining us today this has been a really fun chat if you enjoyed it made it all the way this far you clearly need to like the episode, you need to share it and you need to subscribe.

57:05But more importantly, you need to go on socials and you need to tell people what you thought about today's conversation. You should tag Martin and I. We're both active on places like LinkedIn. It's really easy to find us, contribute to the conversation. We want to know how you're using these tools and what are the conversations we need to have to make it more effective. So thanks for joining us. And as always, stay building and we'll see you next time.

57:32you