Forget vibe coding. Say hello to vibe entrepreneurship. | Shopify’s Andrew McNamara

0:05Welcome to Dev Interrupted. I'm your host, Andrew Ziegler. And this week we're bringing Andrew McNamara, Director of Applied Machine Learning at Shopify, on the show to discuss a new topic, vibe entrepreneurship and the future of e-commerce. And Shopify has been killing it recently, and it was really fun to chat with Andrew about all the things they're cooking up over there. But first, we're kicking off the show today with something a little different. Instead of our standard news roundup, we're presenting a special report to dive deep into the topic at the forefront of our industry, the economics of software innovation and how AI is impacting it.

0:41And to help us do that, we're joined by a special guest and my dear friend, Fatima. She's developer advocate at GitLab. And Fatima is here to walk us through this brand new GitLab report. And we'll be breaking down some of the most compelling and potentially provocative even findings about what's inside. So Fatima, let's go ahead and dive right in. Welcome to the show. Yeah, thanks so much for having me, Andrew. It's always great to hang out with you. Amazing. Well, you know, without any further to do, let's talk a little bit about this report. So what's it called and what do we learn from it?

1:13Yeah, so this is a research report called the Economics of Software Innovation. And it was this global survey of C-level executives about AI, how much money they're spending on AI, how much adoption they're having with AI, and sort of thinking about, you know, how humans are learning with AI, the governance of it, and sort of strategic things across their organizations. It's a whopping like 40-page PDF. And so together, Andrew and I have pulled together a few stats that we thought would be interesting to talk about on the show. Yes. Well, great. That's great framing. And we're going to include links to this report so folks can go check it out.

1:46As you know, our Dev Interrupted listeners, we've been covering these kinds of stories for a while, looking at surveys from all across the industry. So I'm really excited to get GitLab's perspective here. So let's just dive into one of the first things that stood out. Fatima, you want to talk about it? Yeah, I thought this one was interesting. And so the report talks about how executives want to see a 50-50 human and AI partnership when building things. And currently, they've assessed it to be 75 % human and 25 % AI. But they're saying the, like, ideal balance is this 50-50. And, like, in my experience, I've seen the 75-25.

2:22I would say it's even less than 25. And I don't know if 50-50 is something to aspire towards. Like, I kind of disagree with this. I do as well, because also 50-50 in terms of work split feels very arbitrary, because a huge amount of that work could be toil. That could be freed up even more by the AI, right? But let's say that the AI does 75 even more percent of the work, and there's less for the humans to do. That doesn't mean that what the humans are doing is less valuable or less critical. And if you were to try to delegate some of that to AI, it could cause major issues. I can think of a lot of industries where aiming for a 50-50 split feels really arbitrary.

3:05So that's kind of an interesting finding in terms of the balance. Absolutely. Even like if you're thinking specifically about software as an industry and a lot of the use cases that we have for AI and LLMs right now with like, you know, turning spec into a starter project or debugging or researching, like I can't imagine that 50 percent of your role as like a senior software engineer can be done that way without some level of management of the AI. And so maybe this number is indicating like 50 % of your time is going to be managing a set of agents and the other 50 % of your time. And the report talks about how like human creativity, human vision, human collaboration, like can't ever go away.

3:48And so maybe that's the split, but I'm not totally sold on this stat just yet. Yeah. Well, I guess we're going to have to see what develops. You know, what else stood out from you in here? Another one that I thought, and it's funny because I'm pulling the things that I don't quite agree with. So there's a lot of other great stats in the report. So people should totally go and check those out. Some of those are highlighted on the website. So you don't necessarily have to download the report to see them. But I pulled out the ones that I thought we'd disagree with because I think that's where the fun is.

4:16And so 94 % of executives say that AI will pay for itself in two years is the other stat. Ooh, that's an interesting take. They think that so all of the money that is put into an AI investment will pay for itself in two years or less. Yeah. And if you check out the report, like the investment sort of varies across training, like bridging that training gap for skill sets, like hiring and things like that and like infrastructure. So it's not quite like individual people's salaries specifically, but it's also just like building out infrastructure for being able to host maybe their own custom models and things like that.

4:54I kind of disagree with this. I think if you look at it only from the infrastructure perspective, maybe it makes sense. Like you're getting some velocity in return for building things out long term. But I think from like a code perspective, and we talked about this a little bit before the show, like you have all this AI generated code. Who's going to maintain it in two years? Yeah, that's right. I mean, it also on a macro level is an interesting point that the AI pay for itself in under two years because at the rate that we're going, how does anyone know what their own infrastructure or architecture is going to even look like in two years?

5:29So to make that kind of future looking prediction is, I think, a little fuzzy for me. But the biggest part, too, is about really how I think AI is multiplying the way that we work is it actually increases the disposableness of a lot of the tools and software and architecture that we like cement into the ground and build so much stuff off of. AI reduces, in many ways, the tech debt and the initial lift of creating one-off tools and specialized, you know, fits for your own architecture. So gone are the days of having to have the behemoth that you specifically bespokely customize. And here are the days of creating those specific one-off and small, almost idiosyncratic engineering solutions, right?

6:15So if you think of leveraging AI in that way, it certainly could pay for itself if you're, you know, having that instead of paying for something else. But that is a limited privilege, I think, for a lot of companies to be able to work and build that way. And frankly, I don't think that we're there yet to know what that future will look like. Absolutely. And like if you're in client services, that's a really hard sell because every client thinks that their project is so unique and needs its own customized build that it's hard to sell to them that like, we're going to use this template that we built and we promise it's going to work for your needs because it's going to be maintainable in the long term.

6:50And like, isn't that where support contracts make money, right? Like we're going to do this thing and then we're going to support it for the next two to three to five years. And so I'm not really sure that the code itself will pay for itself. I think that they might, organizations might end up spending more money looking for people to maintain this. And that kind of reminds me, there was this article on Hacker News front page. I still like notoriously check the front page every few days. It was called AI first and the bus factor of zero. Like I wanted to talk about your listeners a little bit about the bus factor and like how that's changing and how that's going to impact the way that, you know, we continue to work.

7:33No, I love this. Why don't you give us the scoop on this article? I love this one. Cool. So the bus factor is this idea that like the risk of losing the knowledge about a project is the people that are on the bus. And please add to this a little bit if we need to give more details. So it's like, you know, if you and I built something together and like we were part of it and then you left the company for whatever reason, like the bus factor went from two to one. So like I'm now the person that can be asked in the future if there's technical doubt. Like even if I move on to another project and they hire a junior, they'll have to like pay like follow the documentation but then ask me questions and and so like now with ai generated code there's not necessarily an individual on the bus anymore so we're calling it the the author talks about how it's a bus factor of zero and so like now you don't know you know they're about the documentation about like there's nobody to go to and there's nobody who is responsible for making sure that whatever is left on the bus is enough for the next person to be able to like drive that and maintain it and make sure to keep it updated.

8:38And so that's kind of scary. Yeah, I think it's interesting to about how a lot of trends in software engineering sometimes are going to go backwards during periods of like fast innovation because you're so concerned with moving fast that you forget all of the fundamentals that made moving possible in the first place. So you kind of like outrun yourself and you fall on your face or it's like the Sisyphean experience of rolling that boulder up the hill of making those incremental security base test driven changes within your org, but then letting it all roll down because you want to chase the shiny new AI boulder, right?

9:14And the bus factor innovate all in the sake of innovation. And the bus aiming for the bus factor of zero is exactly that. Every software engineering leader, when asked before AI got on the scene, what were they aiming for with their bus factor? A high number. They wanted to go the other direction, not to zero. They wanted to go from three to five. They want to go from five to 10. They need more people to know about their code because that is how the code survives. Because remember, like we talked about a moment ago, this is a cemented layer of their architecture, of their technological world. They need people to know what's happening inside of it.

9:46So now in this world of more disposable software solutions, bespoke things, that bus factor can be zero because it could be entirely generated on the fly as almost like a shim, right? In a situation to fix something very particular. So aiming for the bus factor of zero runs counter to everything. But it makes a little bit of sense if you think that the software is entirely AI made and maintained. It depends on what its surface area is, but you really just need to know what that is doing. A bus with no one on it is scary, especially one that's moving on a road that you're also driving on with other people.

10:25That's the reality of having this in your code pipeline, right? Right. Interacting with other projects. Yeah. Yeah, exactly. So I think it really raises the importance of observability as well. Yeah. Like, you know, where is this bus headed? And if it gets lost on the way, like, how are you going to find this bus and make sure you're getting it back on track? And so, yeah, bus factor of zero is pretty scary to me. I understand, like you said, like it helps organizations move faster. But even as like models change, right? Like the models that you've relied on are now different or the LLM that you use to build that project is gone.

11:00like is even the zeroth bus factor, which is, I guess, in that case, the AI that you use to build it, like is that still consistent, right? Like the new model isn't going to give you the same answers that the first one did. And so that also scares me. It's like the meta version of this article. I should go and leave a comment on that Hackney's post. That's a very provocative thought to end the battle. I think about, you know, when the model itself then changes, your bus factor zero goes from zero to what? Like even less than... Is that negative? Is that negative technical debt? Negative imaginary number?

11:30Like it's something bad. So is someone who's good at math, please let us know. We need a mathematician on the show. You know, there was one other stand here. I just wanted to double click on before we move on to learn a little more about the GitLab release. And that's about how things like AI is impacting productivity. You know, were there things in the report that stood out to you on that topic? Because this is what we've been talking about from a lot of folks in the industry and how it's perceived. Most of the numbers in the report, and I can't remember the exact percentage, but they were like relatively positive that AI is improving productivity across these organizations that were surveyed.

12:07Okay, cool. You know, I think it'll be interesting to get a deeper dive into some of the findings behind that numbers. And our listeners, I know, have been following closely that understanding of how it impacts the developer productivity. So, folks, if you're listening, make sure that we include that in the newsletter so we can go do a deeper dive. And you said that you've looked at other reports. Like what sort of findings did you find about productivity? and where they're different. Yeah, so, you know, we've been covering this a bit from like surveys, like from Stack Overflow and from Lead Dev, really zooming in on the inside of an organization and understanding the actual impact of AI for engineers within like the day-to-day process and how executives are framing that conversation and the expectations they have around it.

12:48And what a lot of these surveys, especially a recent one from Lead Dev, for example, is finding is that there's a polarizing divide in the perception of developer productivity with AI. Folks who are very close to the AI and using it in a day-to-day process as an IC typically find that or report qualitatively less of those gains than their executive counterparts claim to perceive or think that they have. And so this creates this like polarizing situation within some orgs about the efficacy of AI, right? So that's what some of those reports have double clicked into. Yeah, and I think what you just mentioned there, like qualitative versus quantitative, like it's possible that the systems are reporting up the usage of the AI, but not necessarily what exactly it's being used on.

13:38And so I can imagine there's a reality where the IC is using the AI for scaffolding tasks or supportive tasks, but they're doing a lot of the strategic visioning of the project on their own or the work that the sprint that they're working on. And that maybe doesn't always bubble up to the executive level. But I'd be interesting to see the diff between the different reports and sort of what people at the highest 10 ,000th level are thinking of and what people like on the ground are experiencing. Yeah, I agree. I think we're going to keep learning more. So we'll keep exploring that one with you.

14:09More numbers. More numbers, please. And, you know, just before we start the wrap up and head into our guest for the day, I want you to tell us a little bit about the GitLab release that came out last week. Yeah. So we release every month. And so this month, it's 18.3. This release is sort of our vision for AI native software development. So more agent capabilities, some multi-model AI support. We're trying to be like integratable with all these third-party models and sort of bring them into the ecosystem as well. Some notable features, we on self-hosted, which is our sort of compliance and bring your own and set up your own environment.

14:46You can now bring your own models. So, you know, depending on what you're working on, like maybe you want Mistral to manage some code. Maybe you want Llama to help you write things. And so you can configure those models for your specific deployments. You can also do like hybrid. So maybe your coding standards are running off a cloud model and maybe you're like the stuff that is more proprietary. Maybe there's a formula or something that you're doing predictive modeling on. You can host that one locally and sort of fine tune it for your needs. So I think that really brings a lot of flexibility on self-hosted.

15:16That's pretty cool. And there was another thing that stood out to me, too, I want to ask you about it. I see there's a CLI agent that you can create on the fly and interact with your favorite tool. Yes. It's pretty cool. This one's very sci-fi. As someone who I have a very positive vision for the future, which, you know, when you really think deep into it, you have to kind of tweak a little bit. But basically, on merge requests, you can tag in CLI agents. So integrated with Cloud Code, OpenAI Codex, Amazon Q, and ask them to do things for you. So as long as you bring in, this is another bring your own API key.

15:46I feel like this release is bring your own. There's like a bring your own episode. It's a bring your own party. Bring your own LLM, bring your own API key. I brought my own co-host today. I love it. It's bring your own. So basically, it could be just like, cloud code, can you fix this? Or can you open a merge request that addresses this bug? And it will go ahead and do that for you. So it's sort of having a partner in the CLI, right? Built into the platform. Amazing. Well, Fatima, thanks so much for coming on the pod. to give us the scoop on GitLab and the report. We're going to include details for all of this in the newsletter.

16:20For those listening, stay tuned because after the break, you'll hear from Shopify's Andrew McNamara. Thanks so much for having me. This has been really fun.

16:30Are you investing in AI but struggling to see the real impact on your engineering team's productivity? Well, you're not alone. In a free 35-minute workshop that I'll be hosting with Linear B, we're going to show you how to translate AI metrics into business ROI, just like Expedia and Adobe have. And you'll learn a simple framework for understanding where AI is helping, where it's hurting, and where to focus your next investment. And besides, you're going to get a nice takeaway report on AI productivity as well. So don't miss out. The workshop's coming up on September 17th or 18th. Grab your slot and we'll see you there.

17:06We're back on Dev Interrupted and joining me is Andrew McNamara, the Director of Applied Machine Learning at Shopify. And his impressive background, it includes so many great things like applied research at Microsoft after Maluba, where he built and scaled machine learning algorithms was acquired in 2017. And that really sets the date for you because he's been building human centered AI since before it was cool. And we're actually going to get to that a little later. But until recently, it was more just called machine learning. And right now, Andrew is bringing Shopify Sidekick to e-commerce with the goal of unlocking the full power of the platform in everyone's hands using only natural language.

17:47And with Sidekick, we're not just talking about automating tasks. We're watching the rise of something new, a generation of merchants that can almost run a vibes first business and no spreadsheets, no MBAs, just good ideas, great products and the right AI tools. And I love this idea because it's so quintessentially 2025. So I'm excited to dive into this with you. Andrew, welcome to the show. Yeah, thanks so much. I'm excited to be here and really looking forward to our chat. Great. So starting off, you know, I wanted to pick your brain about this like personal assistant to AI pipeline that's been your career.

18:23And we talked about this a little bit when we first met. And I wanted to kind of explore that here with you today. So could you maybe tell me about your time, like building assistance for the last 16 years at Maluba and Microsoft and beyond? Yeah, I think just having gone from that scrappy startup back in 2010, even starting the personal assistant before Siri came out, which was crazy. So when Siri did come out, we were thought we were in a lot of trouble. But, you know, being pretty scrappy, we ended up getting some pretty big licensing deals with Samsung, Microsoft, or not Microsoft, Samsung, LG, their phones and TVs, and even BlackBerry at the time.

19:04And then, you know, through that, we kind of started up a research lab and then eventually got acquired, like you said, and we became, you know, Microsoft Research. and then we got back into, you know, so we were doing research for a bit in natural language, then got back into the product side of things with the chat with, you know, Sydney now called Copilot. Back then it was called Sydney and Bing Chat. But yeah, the mix of research and engineering has just been like super helpful from that whole scrappy startup to big company and then bringing both kind of experiences here to Shopify. And it's really interesting how your background spans this applied research world.

19:41You know, it's not every day that you really talk with somebody who comes from applied research. And often when I do, they fall onto one side of like, they're really strongly in the research camp or they're really strongly in the like applied camp, the building. What about you as an applied researcher? We've had many of them on Dev Interrupted. I'm kind of curious, like what, how does that you approach your work philosophy? Yeah, I think one of the big things from being on both the, you know, the very research side in Microsoft Research and then, you know, the scrappy engineering and applied research side in the startup at Maloobo.

20:10but was like just the value of evaluations and how that's really shaped kind of everything that I've done moving forward. Evaluations are big in the research community. They're also big on the product side, but they're hard to get right in both cases. And I think with LM these days and vibe coding, you know, and whatnot, proper evals seems to be kind of falling off a lot of people's radars, honestly. So I think it's just like a huge value that sometimes people are missing these days. I'm really glad you're touching on evals because this is a topic that I've been learning more about recently, more in depth.

20:48And it's something that I know many folks building in the space are really caring about as they start to build these products, you know, production-ready applications, understanding the decisions that they make to quote-unquote improve it. You know, does it actually improve it? Does it make it better? And what about it did improve and why? And, you know, Andrew, you recently gave a talk, actually, at the International Conference on machine learning about building production-ready agentic systems. And it touched on this need for like a robust eval system for tools. And so I want to know from you, you know, what is vibe testing compared to what you are talking about?

21:22And how can folks like level up to reach those real evals? Yeah, it's a great point. And yeah, that talk at ICML actually gave, there's a lot of great discussions afterwards, both in the talk of people sticking around and then coming by the Shopify booth after and just having lots of deep discussions on it. So I think it's, it is a huge topic these days and, and it's super important. It's a big part of machine learning and LLMs have really changed a lot about machine learning specifically around, you know, validation testing and evals. And in my opinion, like LMs have almost polarized evals. Like everyone used to have a standard way of doing evaluations.

21:59You know, you'd split up your training data into, or you split up your data into training and validation and then you train and then you test and then you'd have this like holdout or golden set. that would be hidden developers. And then you would test that on after the model was done training and see how you're doing. And you would kind of optimize for that. But these days, LMs have completely changed evals. You can make insanely powerful evals with LLM as a judge or even agents as a judge, which was a paper that I saw at ICML. But you can also get away with just vibe testing and just asking an agent, hey, can you rate this from one to 10?

22:32Or the LMs are so good that you just try a couple of things and it works. And so you kind of ship it. So we've gone from like this very principled approach in classic ML to LM enabling super powerful evals. But then LM has also worked so good that some people kind of just vibe tests and ship. So yeah, it's an interesting spot that we're in right now. Very polarized. Yeah, very polarized. And you're kind of calling out some of the, I think the pitfall here of, you know, you can use, you can leverage AI so much to get so much of that done. That along the way, you kind of lose the rigidity of like the benchmark.

23:07and what it's looking for. It's like, sure, you're getting that result faster, but what is like the efficacy and the accuracy of that result? That starts to get thrown into question, right? And so this actually brings me to an interesting question because you're working on a product that's, you know, it's bringing AI, it's bringing agentic work to really close to a huge customer base, to a huge user base, right? So what's like the hardest parts of getting a system like that to be production ready? Is it the evals? Is it something else? Yeah, I think the evals are probably the most important thing to get a production ready and probably the hardest because it's pretty easy actually to get an agent up and running these days or any kind of product with LLMs.

23:50Like, for example, what we built in the first two years at Maluba, I could literally do in an afternoon, like a day at the most I could rebuild Maluba, which is like insane on how far we've come in the last 15, 16 years. But for it to be in production, you have to have very high trust that what you change is making it better across the board. And you really need evals to do that. And it's really, it's especially difficult in a conversational setting as well. But I think it's, you know, it's a big part of the talk at ICML conference on how to, you know, really make your systems production ready by having very strong evals.