The future of AI in software development | Inbal Shani (CPO of GitHub)

0:00the user of the AI tools to develop software needs to form a different thinking. You need to start figuring out how are you using these AI tools to help you be successful. And it's no longer just the actual code writing. It's really evolving your thinking to the big picture, to the connected experience to the connected systems, which today we just find it more in the world of more senior developers and less and less in the junior developers. The junior developers, when they start, usually we expect them to be able to write simple codes. But if now there is an AI assistant that is helping them write in code, they can spend more time from the get -go understanding the system, understanding the environment that they are building or understanding that product that they're building, which today they don't have time because they are still learning how to code.

0:46Today my guest is Inbal Shani. Inbal is chief product officer at GitHub, formerly a general manager at AWS and at Microsoft, and also senior software developer on Amazon Robotics. In our conversation, we explore the end state of co -pilot and AI -based tooling for software engineers, what the future of software development looks like with AI, what's underhyped and overhyped in AI and software development, what metrics the teams look at to understand if co -pilot is doing its job, what mistakes teams and companies make, jumping into AI, the design philosophy of co -pilot, plus what's helped Inbal most become the leader she is today, and also work GitHub is heading as a product and an organization.

1:27With that I bring you Inbal Shani after a short word from our sponsors. You fell in love with building products for a reason, but sometimes the day -to -day reality is a little different than you imagined. Instead of dreaming up big ideas, talking to customers and crafting a strategy, you're drowning in spreadsheets and roadmap updates, and you're spending your days basically putting up fires. A better way is possible. Introducing Giro Product Discovery, the new prioritization and road mapping tool built for product teams by Atlassian. With Giro Product Discovery, you can gather all your product ideas and insights in one place and prioritize confidently, finally replacing those endless spreadsheets.

2:09Create and share custom product roadmaps with any stakeholder in seconds, and it's all built on Giro where your engineering team is already working, so true collaboration is finally possible. Great products are built by great teams, not just engineers, sales, support, leadership, even Greg from Finance. Anyone that you want can contribute ideas, feedback and insights in Giro Product Discovery for free, no catch, and it's only $10 a month for you. Say goodbye to your spreadsheets and the never ending alignment efforts. The old way of doing product management is over. Rediscover what's possible with Giro Product Discovery, try for free at Atlassian .com slash Lenny.

2:49That's Atlassian .com slash Lenny. This episode is brought to you by sanity. Your website is the heart of your growth engine. For that engine to drive big results, you need to be able to move super fast, ship new content, experiment, learn, and iterate. But most content management systems just aren't built for this. Your content teams wrestle with rigid interfaces as they build new pages. You spend endless time copying and pasting across pages and recreating content for other channels and applications, and their ideas for new experiments are squashed when developers can't build them within the constraints of outdated tech.

3:24Forward thinking companies like Figma, Amplitude, Loom, Riot Games, Linear, and more. Use sanity to build content growth engines that scale, drive innovation, and accelerate customer acquisition. With sanity, your team can dream bigger and move faster. As the most powerful headless CMS on the market, you can tailor editorial workflows to match your business. Reuse content seamlessly across any page or channel and bring your ideas to market without developer friction. Sanity makes life better for your whole team. It's faster developers to build with, intuitive or content managers, and it integrates seamlessly with the rest of your tech stack.

3:59Get started with Sanity's generous free plan, and as a Lenny's podcast listener, you can get a boosted plan with double the monthly usage. Head over to sanity .io slash Lenny to get started for free. That's sanity .io slash Lenny.

4:18Involve, thank you so much for being here and welcome to the podcast. Awesome, thank you for having me. I'm excited to be here. I want to start with just like a broad question about where software development is going. It feels like you're at the center of the storm of what is changing in software development. It almost feels like GitHub kicked off the storm with the launcher co -pilot. Let me ask, what do you think is overhyped in terms of what is changing in software development? And then what do you think is underhyped? I joined GitHub as a CPO. Basically a year ago, today's ago I celebrated my first year.

4:53The initial approach is like, let's look at the platform, let's look at the software development lifecycle and draw some conviction. And for me, the biggest conviction is that developer happiness and productivity strongly depends on their environment and their tooling. And as such, AI is really critical to that mission. When we're looking into what's happening right now with AI, it really became stable stakes on the expectations among developers. And it's really central and critical for them to be able to do their job. We know that something like 92 % of developers are already using AI tools. But in that landscape, some of the things that are overhyped is that generative AI will replace humans.

5:34I don't see that happening in the near future. The way I think about it, you always need that human in the loop because AI cannot replace innovation, right? That creative spark, that creative thinking that is the center of humanity, this will not be replaced by at least not in the near future. If we think about what does that mean having an AI tool that is successful, you need data. In order to generate data, you need humans using tools, acting with tools, doing something, something. So I think the overhyped that is happening right now is that generative AI will replace humanity. It's going to be the solution for everything.

6:14And I think it's a tool. If I'm drilling down to the things that are underhyped right now in the world of software development, I'm starting hearing a little bit more on AI driven testing. But there's not lots of mention on it. When you're thinking about the world of AI, where we're going to generate much more product, much more efficient, more lines of code, more application, then testing is becoming such a critical part of that journey of developing software. And I would like to hear more about that. And what companies are thinking about how do they use AI to generate a larger suite of testing to be able to validate the work?

6:53And by testing, you mean like unit testing, integration testing, things like that. One of them, low testing, multi testing, rupture testing, security testing, penetration testing. Think about that when you need a human to write all these testing capabilities, you need a lot of humans that needs to form kind of different thinking. Can we leverage AI to really generate that testing suite, that test, everything that you need from unit testing and functional testing to low testing, to performance testing. There's a lot of testing that we do that we don't even consider them as testing. So we don't necessarily spend a lot of time in doing them.

7:30But if we're now saying software is in the center of everything, all companies will become a software company. Some form or another, there is more code being generated, hence the importance of testing is becoming much more creative. Let's follow that thread. You talked about how you don't see software engineers completely disappearing. What's your general sense of how software development will be different in like three to five years? What do you think is going to be different? And then also just how do you see the end state of all of this of co -pilot and tools like this getting better and better and better?

8:01What do you think this ends? For me and I keep on saying that again and again co -pilot is a co -pilot, it's not a pilot. You still need the human in the loop. But that means that now the software developer or the user of the AI tools to develop software needs to form a different thinking. One, you need to start thinking more systems, more architecture. You need to start figuring out how are you using these AI tools to help you be successful. And it's no longer just the actual code writing. It's really evolving. You're thinking to the big picture, to the connected experience, to the connected systems.

8:36And what we will see in the fullness of time that developers one will adopt much more of that mindset, which today we find it in software development. Don't get me wrong. We just find it more in the world of more senior developers and less and less in the junior developers. The junior developers, when they start usually we expect them to be able to write simple code. But if now there is an AI assistant that is helping them write in who they can spend more time from the get -go understanding the system, understanding the environment that they are building or understanding that product that they're building, which today they don't have time because they are still learning how to do it.

9:12So I think that's one element. The second element that I expect to see changes is in the world of hardware development. Because we know that AI has a big footprint and requires a lot of resources. And in order to be able to meet the scale, to meet the demand, we need to be able to improve our hardware, our CPUs, our GPUs, our compute, and optimization of code. And that's another area that today it's a niche of people that are spending time. And I think it will become much more aligned across the software development lifecycle integration with AI is something that will definitely become something that is more normal for every developer is today something that we start seeing that revolution, that acceptance, that AI tools, I need to figure out how to integrate with AI.

10:00I need to know how to use AI to become something that is more custom. And we will start seeing that chips from very early on when developers are start shaping their thinking, they will start understanding all these components. I saw a stat that at Shopify, over a million lines of code in their code base was written by co -pilot, which I don't know if I can do the math, but I imagine that's more than one person would have written in their lifetime. And probably many engineers write in their lifetime. And I think that number reminds us just how fast things are moving and how much is actually changing.

10:35You also mentioned 92 % of engineers are using AI tools already. Is there any other stats that you can share that might surprise people, but the usage of co -pilot, growth of co -pilot, things along those lines? Yeah, well, we have over a 37 ,000 organization and more than 1 .5 million developers that are using co -pilot. And they're writing codes based on our surveys, 55 % faster. So if you imagine that you can give a software developer even few minutes, half an hour back in every given day, how much more productive they're becoming and also as a little hunger happier they're becoming. So we know that in a survey that we have run through our users is 85 % of the persistent and self -more confident in their code quality.

11:24And also code reviews were completed 15 % faster without a co -pilot. And we know that usually no one likes to do code reviews, so if we can help accelerate and may developers take their time and do code reviews. And also, if we think about removing frustration, then 88 % felt less frustrated and more sopus on their ability to reporting. And that is just the beginning. We have different examples from our customers. I think Accenture is a recent one that we got where they had 88 % of suggested code was retained. So we have a lot of these stats that are coming together to showcase the size and the impact of using code pilot in AI tools for software development.

12:07Say companies here, these stats of like, oh, we're going to be 25 % more efficient. Some people may think we need 25 % less engineers. We're going to do all the same things. Imagine what you're actually finding is your engineers can do more. So it's not like you need to cut your people. It's that people can be faster and better. No, let me be very clear. You cannot cut your people. You have to have a human in the loop. Co -pilot is a co -pilot is not a pilot. And I keep on saying that sentence again and again. I think the second thing that what companies need to realize and that's another developer experience survey that we have run recently, that throughout the years we have added so much tasks on a single software developer.

12:47From writing code, writing testing, interacting with people, collaborating with 21 people on every given week, going to meetings, waiting on builds, digging into legacy code, all of that is added to the fact that they need to write code, which is their basic work. So most developers spend less than 25, some say less than 20 % of their time writing code. So if we're able to give them half an hour back, one, they can write more code, safe can, they can have a break and take a breath. So we don't burn them out. And they're more happy. They're giving them more time for collaboration and creative thinking.

13:25So that sparks innovation. That is not something you want to cut back. This is something you want to be able to give as a gift to your developers. So they're happy with you. And you can retain them. They can do their job, they're productive, hence they're happy. I've talked a bunch of engineers that use co -pilot. One thing that surprised me is the most amazing engineers love co -pilot. You think this would be like for junior engineers learning to code better and faster, but like 10x engineers say that like someone told me it made them 50 % better and faster too, which is incredible. What mistakes do you see product teams and engineering teams make when they kind of rush to start adopting AI and integrate AI into their workflows?

14:06I think there are a couple things. The first one is companies expect to change to happen. Magicly, here's the tool go use it. And it's not always flying the same way across all companies. So investing time in really taking the company through a change manager and process. The second thing that I'm seeing is that since AI is so hyped right now, there is often the questions of, oh, what should we do with AI? So there is that sense that I have to do something with AI because if not, I'm not doing the right thing versus the question that for me is a big focus for for my product team is what is that problem that we're trying to solve?

14:48And how can we leverage AI better to help solve the problem versus what do we do with AI? So it's really working backwards from the customer problem from what we're trying to solve and then realize what are the best tools that we have in order to do that work better and things through the AdLands game versus oh, AI is here. We have to use it because it's not doing something right. Is there an example that comes to my if someone that did that incorrectly or we sit a lot of time? It's not necessarily examples, but I've been talking to some of our customers and they're coming to us. We heard about AI.

15:25What do you think we should do with AI? And how should we adopt AI? And can we learn from your experience? And I'm sharing our experience when we started thinking about AI, it was with the idea to make developers much more productive. And what are some of the things that we've seen that developers have multiple tasks on them and they don't have enough time to spend on coding. So how can we give them time to spend more on coding and what are some of the coding element that we can automate? And from that comes the need to, okay, let's incorporate open AI, let's incorporate chat GTT, let's connect all of them to help create that tool that helps develop where we're being more productive and is AI based.

16:06So I'm sharing how we went through that journey to think about how to use AI and that I see a lot of the times like, folks with the customers like, oh, we didn't think about it like that. It's more about we wanted to plaster AI or a lot of things versus I have this workflow and it requires a lot of manual work or it requires a lot of configuration. Maybe I can think how to incorporate AI into that and really shorten the time or make it seamless or make it less friction for developers to adopt that tool and so on and so forth. So it's really sharing more how we are thinking about it and helping our customers to understand the same.

16:42Along the same lines you're talking about how you think about it with your teams feels like your product teams are probably living in the future of how other product teams will operate because you have access to stuff everyone else doesn't necessarily have access to. They probably adopt these tools a lot faster than the other teams would or some ways that your product teams operate that other teams may not yet be operating in. In GitHub in general, it's not just the product team. GitHub is GitHub. I'll say we eat our own drinks and that means that we are the first to try every feature, every capability that we're developing and it's not just that our engineering team expands across GitHub.

17:20So for example, GitHub is running on GitHub. There are several blog posts and talks even in the recent universe we had a specific talk in terms of how GitHub is using GitHub and adopting AI across software development cycle. And the biggest focus for us is if we're saying that GitHub is a developer platform and it's more than that it's a software development platform. How are we making sure that all the personas that are taking part in the development of GitHub are also using GitHub. So for example, finance teams using descriptions and posts and PRs and reposts to communicate RAR numbers and so on and so forth.

17:57We have our legal team using where our HR team usually like it or not. It's a different question, but the idea is that we're using GitHub to run GitHub and the product team is one of the first early adopters along with our engineering team. So for example, when we've tested chat and we wanted to see the quality or the recommendation, the search, this is something that the product team was one of the first users to go and figure it out. Is it giving me the right thing that I need to do that? Is it summarizing the PR the way I want to summarize and so on and so forth? But we are really strong in advocating that we have to be the one testing our tools.

18:33If we cannot use them, our customers cannot use them for sure. So nothing is shipped before it spends enough months cooking inside GitHub. So we have a save. This is working for us or not before we put it at the hand of our customers. One of the magical elements of Copilot is how it just kind of fades into the background. You don't have to chat with it. You don't have to tell it anything. It just infers here's what you're doing. Here's the answer for you if you want to use it. It's cool if you don't find is there anything you could share about just the design philosophy of Copilot? Yeah, the design philosophy for Copilot is very much aligned with the working backwards concept that I just mentioned.

19:12It's really putting yourself at the shoes of your customers and triggering out what is it that they need? How is that experience going to work for them? It's an extra tool and if you need to ask for it and if you need to ask for it or if you need to wait for it, then developers will not adopt it. So because we developed it for ourselves first and we wanted to experiment with that with design philosophy, what's what's your self at the shoes of a developer? How would you like that experience to be? You know, in 2020 we had a group of GitHub engineers that were working with the design team and we open AIGPT to really figure out how we're going to build that and how it can work to help developers to do their job more efficiently, more productively improve their collaboration and happiness and so on and so forth.

19:56But the grounding principle was that the developer needs to want to use this tool and the more you add friction, the more you add churn, the more you add complexity, the developers will not want to use their tool. We just talked about all the things that the developer needs to do in every given day. Now add to that mix, I need to learn how to adopt AI. We knew that it will not fly. So it has to be a seamlessly integration. It needs to be something that is very intuitive for developer schedules to be successful. What are the success metrics for co -pilot? It's such an interesting problem of measuring efficiency gains and productivity gains for engineers.

20:31What metrics do you focus on to tell you this is doing what we want it to be doing? We are in a world that there are no right metrics. There is no one one metric to roll them off. It's a combination of the things that you're looking to measure out of adopting AI. You can think about measuring quad quality using AI. Can you improve the security of your code by introducing, for example, or have advanced security using AI? There is a lot of different metrics that if you combine them together, providing you with that developer productivity and then there's developer productivity, developer collaboration, time gain that if you're bringing them together, are you dealing kind of that developer happiness?

21:15The biggest area of focus for us right now is the definition of productivity. Because we have so many users that are coming to get them and they're looking for that productivity gain. One of the things we're very conscious of is we continue evolving, co -pilot, and AI across the software development lifecycle across all the tools. Productivity is not the right metrics against each one of these components. When we're implementing a young to get up advanced security, writing more secure code is the right element. It's like how many secrets were we able to prevent from leaking? How many issues in the code were able to detect and find and fix before you shake that?

21:51I think that the world of the metrics for AI is really a big one. We are working a lot with our customers as they're asking the same questions and they're running their own experiments within their companies to figure out what is it that they want to measure. The most easiest one is time, but time is funny what I'm going to say. Time is not quantifiable as a success metrics because you can write really bad code really fast. It's really about how are we taking time and translating it to efficiency for the productivity to something that are harder to measure. And then how does that lead to the viral core happiness, which is another harder metrics to measure.

22:33So the combination of all these input metrics leading to eventually developer happiness is what we're focusing on. Yeah, so interesting because engineers could end up spending more time coding because they're enjoying it more. They're getting more done. You also like more lines of code isn't a good metric because you don't want to know if those are good lines of code. That's like a classic way to not measure engineers as well. So to me, it feels like developer happiness is the ultimate metric and there's we had Nicole on the podcast who came up with a framework Dora. That's an interesting way of just measuring developer experience, developer happiness.

23:05Right. I think one of the things that we're talking to customers and we're trying to figure out how we're taking late is instead of time, can we talk about time to value? So from the moment you put a developer on a task, how long did it take you until you realize the full potential or the full value of that if it's generating revenue, if it's in adoption, if it's time to market, like you can define your time to value in different ways. But eventually you're investing in AI tools to help your developer to be more productive, to be more efficient, to be more happy and then easy impacting your time to value, which is something that that's the business side that they're looking to measure.

23:44This episode is brought to you by Help Bar by Camillean, the free in -app universal search solution built for SAS. Your help content lives outside your app and is scattered in many places, forcing users to waste time hunting for answers. Help Bar solves this. It delivers answers directly inside your app and eliminates context switching. Users can search or ask questions to get AI generated answers and lists of the most relevant documentation from all of your help sources, including your knowledge base, docs, blog, and video libraries. You can also use Help Bar to navigate your app and launch actions, such as scheduling a meeting or viewing an interactive demo.

24:22The best products today use Command -K for in -app search and navigation. Help Bar makes that readily available within your app without engineering or new code. Give users a faster and more delightful self -service experience reduces friction and increases in -app engagement. Upgrade your user experience with this modern component and supercharge your product -led motion. Sign up for Help Bar today. It's free and easy to set up in minutes. Check it out at helpbar .ai slash Lenny. That's helpbar .ai slash Lenny. Coming back to where this all goes, imagine you've seen all these demos of people like sketching a website, taking a picture, setting it to JGPT and it builds the website for you.

25:04Yeah. How do you see all of that sort of thing playing into this? Do you see like CEOs being like, here's the iPad app I want. I'm going to sketch it for you and then just run it through Copilot, version 10, and it writes it for you. I know that you're saying engineers won't be cut and only increase and become more efficient, but I guess where do you see that version of it going? I just like here the sketch. Go for it for me. I'm thinking about that as a better collaboration tool versus a production tool because right now a lot of the time where we see challenges in articulating ideas is the communication.