How AI is changing software engineering at Shopify with Farhan Thawar

22:34We have a leaderboard where we actively celebrate the people who use the most tokens. Oh, really? Yeah, because we want to make sure that they are, if they're doing great work, of course, if they script, like that's not what I mean. I mean, they're doing great work with AI. I want to see how did they spend$1 ,000 a month in credits by a cursor, or maybe that's something, they're building something great, and they have an agent workforce underneath them. So we have this proxy, and again, it's used to build anything that you want to build, and you can ask for a token and use it. And then we have, we are very early on MCP.

23:03We're on the MCP steering committee. We are fans of MCPing all the things. And so when I talk to people about how to get access to some piece of data inside the company, we quickly will spin up an MCP endpoint for them so that they can just use it. So for example, our internal wiki is called The Vault, It has information about all of our projects that are happening. It has information on every internal talk that has been done. It takes the transcripts of all those. You could say something like, when did the point of sale at Shopify launch? And it would come back and it would say, oh, in 2012, there was a hackathon where the point of sale was started.

23:38Based on wiki docs. Based on the wiki. And in the 2013 Q1 board letter, it was mentioned. And it would tell you all of the history of what happened in Shopify. And so we do that by just having all you do is put up an MCP. And now the Libra chat has that. It's like an internal perplexity or something like that. Exactly. And so we basically, any internal document is crawled by this thing, and now you can have access to it by MCP. And so any new system comes online, we put an MCP in front of it, and now everybody has access to it. This episode is brought to you by Sonar, the creators of SonarCube, the industry standard for integrated code quality and code security that is trusted by over 7 million developers at 400 ,000 organizations around the world.

24:15Human-written or AI-generated, SonarCube reviews all code and provides actionable code intelligence that developers can use to improve quality and security early in the development process. Sonar recently introduced SonarCube Advanced Security, enhancing Sonar's core security capabilities with software composition analysis and advanced static application security testing. With advanced security, development teams can identify vulnerabilities in third-party dependencies, ensure license compliance, and generate software bills of materials. It's no wonder that SonarCube has been rated the best static code analysis tool for five years running.

24:49So join millions of developers from organizations like Microsoft, NVIDIA, Mercedes-Benz, Johnson & Johnson, and eBay, and supercharge your developers to build better, faster with Sonar. Visit sonarsource.com slash pragmatic security to learn more. And how many MCP servers do you have, Kevin? Point two dozen. Two dozen, yeah. And they're growing quickly. They're growing because people are building a new thing. Again, there's one for Salesforce. There's ones for Figma. There's an MCP in front of anything. But I tell anybody who's looking at building AI systems internally, the way to make this easy is start putting MCP in front of everything, and now your data is accessible.

25:25We happen to use LibreChat. You might use something else. But you don't have to be even in Cursor, right? You just use the chat interface. Yeah, you can do chat. You can do the LLMs. And I like how you spun up basically a platform team to build the infra, to do monitoring, to worry about things like a security, decide on models, all those things. Does this team have a name? Is it an AI platform or is it just a platform team? Infer team just took it over. Yeah, data platform. I don't know. I'm trying to figure out because they're inside the data platform team. I don't know if they have a name outside.

25:53I call them LLM proxy. And was this more like you or someone that said, like, hey, we need to do this? Or are they just like, hey, this is a good idea. Let's kind of bottom up or top down? It was more top down because we felt like this was going to grow internally. We wanted to make sure that we could, we didn't want people signing up for their own personal tokens. And it's also a good way for us to see, hey, this team is growing. Is the usage growing? What's happening there? This person's usage is growing? Yeah, because number one pushback I get from people or even engineering directors working at companies is the data security issue.

26:22And obviously when you host it yourself, it's good, but then you need the infrastructure. So it sounds like, at what point did you decide? What was this month ago? Was this more than a year ago to like invest? Probably, I mean, I would say every week it's getting better, right? but I would say probably a year ago we started thinking about it this way. We had the LLM proxy. When ChatGPT came out, we actively have a warning. If you go to ChatGPT, it says, hey, by the way, you can go to the internal version. That was for a long time. You mentioned cost, and I am hearing stories of CTOs, engineering leaders saying, okay, we were paying for GitHub Copilot.

26:56I don't know, it's$10 or$20 per month for engineer, but we have all these seats, or maybe we have for everyone. And they're telling me Cursor feels a bit too expensive because they're not assuring 30 or 35, and we're not going to go there. And generally, there's a big pushback on cost. People are saying, look, AI should make engineers more productive, but it's now kind of ridiculous to spend$1 ,000 on an engineer. You think about cost differently. Tell me about it. Yeah, I did a tweetstorm on this because I think that people are looking at it differently. So think about it this way. If I could give you a tool that could make your engineering team more productive by even 10%, would you pay for it?

27:32The answer is yes. would you pay$1 ,000 a month for it is the question. And my hypothesis here is$1 ,000 a month is too cheap. Like it's too cheap. If I can get 10 % more, it is way too cheap, like we should be$5 ,000 a month. It's insane that people are looking at this like$100 a month, whatever. Now, I did meet somebody who said their engineers were spending$10 ,000 a month per engineer. And I said, I want to talk to them because I want to see either they're doing something very, very smart or very, very stupid. But I want to meet them. And so I literally said, please introduce me because I'm unclear how you can use that much.

28:02Either they've got fields of agents building amazing things or something is going very, very wrong. But I would like to understand because there's something interesting happening. And you should not be penny-pinching on AI tools because actually the productivity gains are there. We don't know what they are because we actually don't know how to gauge developer productivity. But it is clear that there's something happening there and you should be very open with your pocketbook there. And so, again, we celebrate those who spend more. And, of course, we talked to them, and what did they do? So they were getting the product.

28:32Do I understand that even though you cannot measure it right now or maybe not as accurately, you believe that there's productivity gains and you believe that people need to use it to get these? So you're kind of treating it as an investment right now, like for a year, for two years, whatever that may be. And then you're going to figure it out. Probably forever. I mean, again, you wouldn't turn off spellcheck and grammarcheck and G Suite. You wouldn't turn off these tools, right? I mean, there's probably a debate on when you turn off Slack because that could be a productivity killer. But you're seeing the gains, and I don't think that we should be penny-pitching.

29:05Now, over time, we might realize there's an equilibrium. I'm like, oh, this is the amount of tokens that people should be using. Actually, if anybody's using Cursor here, the number one thing I'll tell you is, please move away from the default model, because the default model is nowhere close to actually using a sophisticated model. And by default, Cursor puts you there. And so we tell people, please move off the small models, move into something more bigger. will even say things like, please use O1 Pro for this project, or please use O3 Pro because those are... They're more expensive. They're more expensive.

29:34Actually, I'll go even further. Mikhail, the CTO, will say something like the following. If you don't pay personally for O1 Pro or Gemini Ultra or Claude, whatever it's called, Advanced, the$200 a month one, you are crazy because you can afford it. And actually, you are missing out on actually all of the progress happening in LMs because you are just stuck to the$10 a month or$20 a month chat GPT. Like that's how far he would go. One interesting thing that I've heard when I talk with engineers, they're saying leadership is leading by example. Toby is hacking. And if you go on his Twitter or social media, you're going to see all the stuff that he's doing even a year ago.

30:12I want to ask you, like, what are you doing on the side or experimenting or playing with? And what tools are you excited about? Yeah, so when I was building, when I was pairing with that engineer with Onanthropic, Like I was building something for myself. Right? Yeah, I was building a couple of things. It's a great way to use company time, I guess. Yeah, exactly. Well, I was like, let me see what they can build. Because we didn't want to go super deep into the Shopify API. Yeah. But we do build a lot of things that, you know, one was actually a commerce flow. I was trying to build a commerce flow via like operator where you can actually record the browser session.

30:42And I wanted to see how the session was storing like credit card information or secure tokens as we were getting through a commerce flow. and I was trying to figure out, I was trying to build almost like a subscription product without using our subscription product from Shopify. And so it was an interesting experiment where we were trying to see how do we store those credentials, especially when you want to store like credit card. Is it stored in the cloud already? Is it stored by the vendor? Are we storing it in the browser session? That's how I kind of stay as close to it as possible. I use Cursor pretty often to build my own workflows.

31:13In this case, it was Cloud Code. We actually even use, I don't know if you've heard of Gumloop. Gumloop is an automation platform that is very interesting at its own layers. Here's one example. We tried to build... I did something super dumb. I was like, let's figure out when the next SanDisk Extreme Pro 8-terabyte drive comes out. We tried to build in Cloud Code. It took us an hour. We did not get there because it actually tried to do something at the wrong layer. It was trying to read the JSON and try to get the image from the SanDisk side and figure out if the 8-terabyte drive is out. It was actually quite complicated.

31:45We built it in two minutes in Gumloop. What did Gumloop do? So Gumloop is basically like it was browser based. We said, go to this website, go through the search box, using English commands. And they came back right away and said, there's no 8 terabyte drive available. And I said, okay, cool. Run it every week and send me an email. It was just, they're different tools for different jobs. And an hour with cloud. Again, nothing wrong with Anthropic Cloud Code. I use it all the time. It was not meant to do this kind of web scraping. Because what does it do? It tries to build a web scraper versus Gumloop.

32:14It already knows how to scrape websites. So it's actually just a different layer of the stack. And what I would encourage people to do is figure out what you're trying to do. And we have a lot of automation happening in Gumloop now because people have to do something. I want to scrape a LinkedIn profile. I want to find out what platform this company is using behind the scenes. I want to scrape a website. Like all of these things are, it's built for the right thing at the right length. You have a lot bigger tool set. Exactly. So you should figure out where you want to work. Right? If you're writing code, plot code.

32:40If you're doing a web skyscraper, use Gumloop. So far it was AI, AI, AI. And when this letter came out, externally, there has been a lot of some speculation saying, hmm, is this a sneaky way for Shopify to freeze headcount because it says you shouldn't hire unless you can check if you can do it with AI? But then you told me something interesting. I'm not sure how big Shopify is, but you told me that you're applying to hire 1 ,000 interns. Can you put that in context with the size of Shopify? And tell me, why are you hiring interns where a lot of companies are saying, let's not hire entry-level people because we have AI?

33:15Yeah. What are you seeing that others are not seeing? Yeah, so a couple of things. One is there is this real notion of, like, generations as they come up in different, like, graduate from different programs out of university or even coming from high school. And you really want to be close to the next generation of people. One, they're like, you know, they know things that you don't know. And so we restarted our intern program in full thrust this year. So last year we had like 25 interns a term. I convinced Toby to hire a thousand interns this term, mostly because I felt that they would be more AI reflexive than everybody else.

33:49That was our hypothesis. So my tweet actually said, I want you to come to work with an LLM and a brain, not one or the other. The idea was that bringing these like centaurs, AI centaurs, they know how to work with an LLM to solve their whatever workflow it is, whether it's a finance person or an engineering intern. And we really focused on them coming in and changing our internal culture. And so a thousand engineers over the year meant like 350 a term. And what we saw there was, one, they're excited, they're hardworking. We also have them as a cohort, meaning they come to the office. We're a remote company, but we have the interns come in because we felt like the younger people need to have, like, younger people around them to work together versus just being remote in their, like, condo or whatever.

34:30We're changing the culture that way as well. And we did see this. They come. They have their reflexive kind of. We saw the same thing in mobile. You were in mobile as well, and so was I in my last decade of work. Yep. And I hired lots of interns because they grew up with mobile phones. And so this next generation is growing up with the Internet, and they're growing up with phones, and they're growing up with LLMs. So we wanted them to come in and change our culture around that axis. So do you feel like people are learning from... We always were. Interns are the secret weapon. You always learn more from the interns.

34:57People think about internships as community service. Oh, that's so great, Shopify. You're helping the next generation. We don't do it for that. We do it to learn from the interns. That is always the reason. So if you don't have an intern program, an early talent program, I encourage you, you should be bringing those people in because you will learn from them more than you will get out of them. You will take a net positive benefit from them. And of course, for those who tweet at me and say, please don't hire interns. You have to pay them. I'm like, we pay them. I don't know why people think that you don't.

35:21We pay them very well, actually. And we do hire, I think I've hired almost 100 this year from the intern pool to come into our program. And we're very much thinking about the only way to get into Shopify as an entry-level engineer is through the intern program. I'm fully with you with On the Paying. I never understood why companies are cheaping out on this. Because sometimes interns will reject some of the offers because it doesn't cover the living costs, which is wild. Yes, of course. You have to pay them. Actually, one of the best intern, it was another company's loss. And in Amsterdam, they offered 500 euros per month, no housing.

35:53And that intern would have loved to work there. What an experience. But they applied to Uber because they said it's just too low. And then we gave them a new grad salary. They could afford housing. They could afford to travel. Of course. And that intern could have done amazing things at that company. And now this person is still senior at Uber. Been there for five years. So their loss. So think about it. 350 engineers interns to about 3 ,000 engineers, about 10 % at any one point, yeah. So if I joined Shopify as either a software engineer or an engineering manager, can you tell me how stuff gets done?

36:27First, as an engineer, like what tools do people use? I'm assuming it's just no longer, you know, VS Code, do code review. There's going to be like AI tools here and there. Yeah, well, first on, we have a very specific way of writing software to the point of we built our own project management system. Like we don't use Jira. We don't use linear. We don't use, yeah. And the reason is, is because we believe that there's that line, like first you make the tool and then the tool makes you. And so we believe that. And so we make the tool so that it can make us. Versus, it's nothing against those amazing tools.

37:02I'm a big fan. Like linear is beautiful. And like, you know, I've known Jira for a long time. I'm a pivotal frackle guy myself, being a pivotal, ex-pivotal person. but we do believe that we have a specific way of writing, of building software. And if we use one of those tools, we would be adopting someone else's way. And so instead we have a tool called GSD, which stands for get shit done. And it is a tool that allows us, it's a program management tool, like a product management tool, which allows us to think about what are we building? Who's on the team? What's the latest weekly update? It has metrics.

37:31It actually pulls in PR reviews. So you can see like it pulls in PRs and you can see there's activity happening on this project. Here are the core contributors. Here's how long it's been going on for. this tool is how you kind of get work done and you're forced to like write an update every week and now actually we have an AI. Here's an interesting one for you. We have an AI tool which will pull in the latest PRs and the latest conversations from the Slack channel and it'll write an update for you and then you can look at it and say this looks good and you can also tell it what did I not think about and say please emphasize this and it'll help you write the project management update every week.

38:02So this is interesting. I want to push you a little bit on this because I think there's two schools of thoughts here. One says your weekly update, like I appreciate that you're doing it. The weekly update, the point should be for you to look back on what you did. Obviously, everyone hates doing this, by the way. And by AI doing it, would you not lose that kind of reflection? Well, this is why we pair you with the AI. So we don't, we do auto-publish it if you don't do anything. But if you auto-publish it and don't do anything, and we look at the stats, there is a little bit of loss from you because you are now losing out on the context that you were supposed to gain as the champion of the project by looking deeply.

38:40However, we expect everyone, just like a PR written by AI, to have read the code and read the update. Yeah, so you're responsible. You're responsible. And so we're early days in this. We're literally three weeks in on this project, and we may revert it and say, by the way, it turns out everybody stopped looking at what was happening week to week. Yeah. But we also, every six weeks at a company level, with Toby go through every project in the company and we see if we think that it's on track, not on track, should be kept working on, has the right resources, is it not too long, is it aimed in the right direction?

39:11You better know what's in there because literally we're going to review it at the leadership level. Yeah. I think you have no... I like how... Because I feel like with AI, I kind of want it to automate the stuff that I really don't want to do. Yes. And ironically, it's really good at automating stuff that we like to do sometimes, which is coding. But writing an update, writing documentation, I don't think as an engineer we loved doing that. And so as an engineering manager... Again, we're trying to reduce toil. If we find it reduces actual context, we will revert. I love the experimentation and going back and forth.

39:44As an engineering manager, what additional things might people use? Do people build their own tools to stay on top of things? Do people go deeper because of... Yeah, as an engineering manager and a director, we try to surface, again, metrics so that you can see how your team is doing. So, for example, we have tools that allow you to see things like focus time of your team. How often are they in meetings? AI adoption. Are they using, like, AI tools or not? Like, we're trying to surface this information. How many of your people are on a GSD project versus not? Because maybe a project ended and they're, like, freed up to work on something else.

40:15So we try to surface that. And engineering managers, too. We try to get them to be, a lot of the best ones come from IC land. Like, they started as ICs. So we try to get them to do, if they're hired outside, start as an IC at Shopify before they move into management. oh here's a fun one that I think we should talk about you mentioned something super interesting to me when you're hiring engineering directors and above in the past it was the usual interview you know culture of strategy all that stuff you added a coding interview for every single engineering director and above hire can you tell me about this yeah so it's interesting like it's maybe shocking to folks and I know like Rodney's sitting there he used to work for me I did the pairing interview with him and so what happens is it is shocking for VPs especially to be like whoa oh, there's a coding interview?

40:57I'm like, yeah, because we believe that the best leaders here were ones who were not running away from coding. They just felt like they got better leverage from running a team. They still are deeply in love with technology, and they still, in Rodney's case, still codes on the weekend. And so it worked out super well. But our whole idea is that you're not running away. You're running towards technology, and this is just a better way for you to express it. So I pair with the candidates, and they also see that even though I'm not writing code every day, I'm still deep in the weeds of technology. I still love technology, and I still want to talk about technical topics.

41:26And so we pair. And some people believe that that's not the best for them. And there's lots of great companies out there where that's not the requirement. But at Shopify, we believe people should be as close to the details as possible. And we are such a tech nerd company. Like, again, our salespeople are vibe coding now. We want our engineering leaders to be coding as well. And so that doesn't mean coding day to day. But you should understand code and how code works. And a lot of it comes back. The muscle memory of coding will come back in these pairing interviews. Yeah, and I guess especially now we have these coding tools when you can generate something.

41:54as long as you can look through it. Like, you're in control. Well, this is the best part about co-pilots is that a candidate comes, the co-pilot generates tons of code, and now I'm like, great. Is that good code? Bad code? I love... Oh, hold on. So you're using AI during your interview process. Yes. Oh, you're not running away from it? No. You know, one of the interesting stories that we just learned, Cursor has disbanded on the interview process. They're the AI tool company. Right. So you're embracing it. We're embracing it. Okay, how is it working? Tell me. I love it because what happens now is the AI will sometimes generate pure garbage.

42:25So you're screen sharing and you say, let's use that everything you want. And they're using, I say, let them use whatever they want. Here's what I'll say. If they don't use a copilot, they usually get creamed by someone who does. So they will have no choice but use a copilot. Sometimes I will shadow an interview and you do the questions myself. I've never seen them with a copilot and send it to the interviewer and say, please mark my assignment as well against the candidate. I have not lost yet. If they have not, if they don't have a copilot, they will lose. But when they do have a copilot, I love seeing the generated code because I want to ask them, what do you think?

42:54Is this good code? Is this not good code? Are there problems? And I've seen engineers, for example, when there's something very easy to fix, they won't fix it. They will try to prompt to fix it. And I see, are you really an engineer? I get the nuance of just prompt and prompt and prompt, but sometimes it's right there and they will not prompt. I'm like, change the one character and they won't change it. And I'm like, okay, so they're... I don't want you to be 100 % AI coder. I want you to be like 90 or 95. I want you to be able to go in and look at the code and say, oh, yeah, there's a line. That's wrong.

43:24So I want to ask you, what does a standout senior or staff engineer today look at Shopify? And what has changed, let's say, three years ago when we didn't have any of these tools? I'm trying to prod here, like, how you think AI or AI tool usage might have changed of what we expect an amazing engineer. And you think of the person who you think is, like, this fantastic person. Well, I would say the thing that's changing is I'm seeing some of my engineers now actually use these AI tools to like do the infrastructure that they've always wanted to do, but never felt like they had the time. So like debt reduction, refactoring, making things easier to read.

44:00Like those are the things I'm seeing the best engineers at Shopify use AI tooling to actually do. And they just never felt like they had the time or the resources. like the engineer who always felt like I wish I had like six months to do a code red now can use an AI refactoring tool like a cloud code or a open AI codex to like unleash these agents in parallel to then and then review all the PRs but then unleash it and have it and feel like they have a team themselves so that's what I that's where I'm seeing and they're not afraid of trying things that may or may not work they might unleash something for 24 hours and like a devon-like tool and it comes back and like utter garbage they delete it all but they said it was worth trying Yeah.

44:35And as a closing question, a lot of people are thinking we'd love to be like Shopify. Like I'm either a staff engineer, director of engineering, et cetera. I'd love to transfer my culture to be a bit more AI first. What would your advice be? How can people start? Like not everyone has the luxury of having the CEO saying, okay, I love these things. Do you have tips that you give to your peers? I know you're in CTO groups, for example. Yeah. So I think that the number one thing is role modeling is the best example. Like, I haven't seen anything work better than role modeling. And the way that happens is, like, you have to do it.

45:09So if you are coding and you are showing people your workflow and you are in those same channels asking, like, hey, I was trying to get this working and it wasn't working, or I was able to get this workflow going by this prompt, we have a prompt library internally where you can literally grab prompts and, like, say, like, you know, hey, this prompt worked for somebody. Let me try it and modify it for my use case. That's the number one way in which I see people trying to adopt AI is because they've seen other people do it. And so we try to share like AI use cases. We have, again, prompt libraries.

45:38We had a hackathon like as a part of Shopify Summit a few weeks ago and we leaned heavily into AI. And it wasn't just junior people learning AI. It was senior people trying to embrace cloud code. And one guy said, I haven't coded in six weeks. I've only used cloud code. Then I pushed him and he said, okay, I have to make changes like here and there. I said, cool, you're 95 % AI, but that's the point. You're not supposed to be 100. And role modeling is the number one thing I've seen. That and sharing examples. So role modeling and then sharing. Especially because no one has it figured out. So we have to learn.

46:05No one has it, exactly. No one's there yet. Well, this was wonderful. I love how much you're experimenting. I love how you are not afraid to, you know, like take some turn to learn more. And I love how much you're sharing internally. I think we got a little window of this, but I think it's just something I hope more people, more companies will do. So this was wonderful. Thank you so much. Thanks for having me. I found it refreshing to see just how practical Shopify is about LLMs. It feels to me that they understand that you need to invest and experiment to get results. One surprising thing for me was understanding how Shopify expects to become more effective using AI tools by hiring interns who come in with an open mind towards AI tools and can come up with clever use cases for it.

46:47For more details on how Shopify operates, check out Deep Dice in the Pragmatic Engineer linked in the show notes below. If you've enjoyed this podcast, please do subscribe in your favorite podcast platform and on YouTube. This helps more people discover the podcast and a Special thank you if you leave a rating. Thanks, and see you in the next one.