Shopify Distinguished Eng (L10) on Principal+ Engineering, Career Story, Regrets

0:00People that have to ask that question by definition are not principal engineers. This is Ilya Grigorik. He grew to a distinguished engineer at Shopify, which is a VP level role, and I asked him what it took to get there. To be a really effective principal plus engineer, you need to have very good dynamic range technically. He turned down a clear path to director at Google to go back to being an engineer. This is a path that is vastly underappreciated and is actually surprisingly common for a lot of principal and then higher up roles. And his best advice might change how you plan your career. A better way, more resilient way to build your career is to optimize for being the only person as opposed to the best person.

0:41Here's the full conversation.

0:46You mentioned you went to Waterloo and I got to ask the smartest interns that I ever worked with. They came from MIT, Caltech, and Tsinghua University in China. But the best interns I ever worked with all came from Waterloo. And so I'm curious, what is it that makes these Waterloo interns so strong? For me personally, the most important thing that Waterloo got right and better than most, and I know that more schools have adopted this pattern, but it's still not well understood, is their co-op program. so what Waterloo did really well is they realized that yes there is the academic portion of learning how to engineer and like what software design is all about but then there's the hands-on and applied part and instead of modeling it as hey you're going to study for three years and then you have an optional kind of intermission style co-op which is what what most universities operate on, right?

1:49You could take a year off or something like that to go and work in industry. Waterloo said, you know what? We're going to just do a rotation where every semester you study, you work, you study, you work, you study, you work. You know what that means? You don't have a break in between. You're just constantly iterating through this loop. But what it gives you is at least six shots on goal for trying things, right? And this actually has very positive of effects on many dimensions. So first of all, it removes the stigma of trying things, right? Because many of us, like, we don't know what's going to stick with us.

2:28Coming back to my earlier point about consulting versus something else or something else. I had so many of my peers going through university where we all started with some preconceived notion of what we wanted to do. It's like, oh, I really want to go to Wall Street and work there. And they try one of the co-op terms there and they're like no I actually like that for whatever reason you know I don't like the city or I don't like the industry I didn't like the people it's like it just didn't click for whatever reason right same thing for like medicine or others or consulting and that's really important to discover early and the really nice thing is it's a concentrated it's like it's a three-month sprint you get to try right and you have permission to try five more things at least without the stigma of looking at your resume saying, hey, like you went to a role for three months, you jumped.

3:18You went to a role for three months, you're like, are you really like committed? Are you like, do I want to hire you? It's like, no, that's my co-op, right? Like that's the whole point. So you could, first of all, try a bunch of things and get a kind of a good range of experiences under your belt. But also see the kind of the gamut of like, hey, here's how a large company operates. Here's a smaller company, here's a medium, but like which one resonates with me more, right? So it's industry, it's the type of work that you do, it's the type of environment that you're in. And I think that is really helpful.

3:51On top of that, you actually get exposure to engineer things, not learn the hypothetical and academic of like, I know how a big O notation expresses this particular algorithm. It's like, I actually had to build the thing, right? And it turns out this didn't work or that particular solution worked or we had to solve it under these constraints. And the whole big old thing was an issue because I was like sorting 10 items in an array. Who cares? Right. It's like pragmatic engineering. So I think that is key. That is still a superpower of Waterloo. And I wish more universities, like it's an open secret.

4:28And I wish more people would copy it because it's such a lever for success for people that go through that program. After you went through all of those iterations and all the co-op loops, I understand that you started your own company, which became PostRank. That was then acquired by Google. And I'm curious the story behind you choosing to found a company right after graduating from Waterloo. I graduated from Waterloo. And as most students, I didn't know what to do. I wasn't ready to commit. it. So my escape, which was a well-trodden track for many, is I'll go to grad school. So I applied and then got into the University of Toronto, actually.

5:14And during that summer, in between, I had a couple of project ideas that I wanted to get off the ground. So I had a particular itch that I wanted to scratch. And the itch was, if you think back to how Google was founded, the key insight, if you simplify it and boil it down, that Larry and Segi had back in 1997 is, hey, we could treat the link crap of the internet effectively as a feedback loop on what is good content. So if you and I create pages and we link to each other, that's a mutual signal in both directions that there's something there, right? So disconnected pages are not worth much, but in effect, it's kind of a social currency.

5:51Links are a social currency, right? Now, fast forward to 2010, 2011, when I'm sitting there, the web two revolution has already happened or like well underway, you now all of a sudden have dynamic content on the web, not static links, not static web pages, but we have comments on blogs, right? Blogs became a thing. Comments became a common thing. So my observation was, hey, if a conversation is happening on Reddit about an interview that Ryan has recorded, like that means there's something there. I don't, I can't qualitatively say whether it's good or bad. I just know that people are talking about it and that a signal.

6:27So what if you built a system that went out and aggregated all of these interactions and then built a better version of PageRank effectively, right? Links becomes one input, but, you know, thumbs up, a number of conversations, a number of likes, and all of these things become additional inputs. And that was PostRank, right? It's kind of embedded in the name. It was an algorithm for ranking posts based on social engagement, kind of a child of the Web 2.0 era. Still remember the day, it was like July 8th when I released it, and then I didn't sleep for three days because the servers were melting down.

6:59not strictly because it was like such huge demand although there was it was also because my code was terrible so it's kind of like brute force and just throw more servers at the problem and then eventually you know optimize it out over the next couple of weeks to to get it back to a manageable set so that becomes kind of this nucleus of hey there's a there's a feature there right and it got it got good publicity which then turned into conversation hey is there a product here so i remember the structure points that somewhere late in may we're kind of in the middle of all these conversations with investors kind of unanticipated fork in the path right and i'm thinking what what do i do like i'm uncertain are we going to be able to close around can i do this because in a couple of weeks i'm supposed to start my my grad work and i'm looking at that syllabus i'm like what are the courses i'm most interested in and i go through the list and it was like the ones that immediately stood out to me about like entrepreneurship and like how to do a company i'm like wait a second so i could go and learn about this stuff academically and read books or i can like i'm literally having the conversations right now with the investors like or i could just like go and like i can read those books on my own time and i could i can go and try which led me to reaching out to my advisor and basically asking, hey, can I take a year time out?

8:23I was planning to do this this year, but how about I start next year? And he hammed and hawed and said, okay, I understand and come back to me in six months. To close that story, we did end up closing the round and that became post-rank the company and I never did get to finish my so-called graduate degree. So in the process of building this company, I know eventually it was acquired by Google, but I'm really curious because I feel like a lot of acquisition conversations are behind the scenes, hush, hush. What does that kind of conversation look like when they reach out for an acquisition and maybe you can tell that story?

9:01It's a lot less stressful than you make it sound to be. I remember we were actually, we were building the product that I was describing and we were presenting at one of the conferences in the Bay Area. And, you know, we got off stage and Phil Moody, who was one, who was the kind of lead PM on Google Analytics kind of walked up to us and like, Hey, this, this is cool. Do you want to have a chat? And I remember taking a train from San Francisco to Mountain View later that day, sitting in the conference room and him basically interrogating us about what is it that you build and then um ending the conversation with hey maybe we should do something together it's like okay well tell me more right um that's a that's a blank check for the follow-up conversation um so it was it was very friendly there's there's nothing uh kind of uh more to it than that and then uh what we learned uh after is this is right in the the time span when Google and Facebook were like at war in terms of drawing the battle lines for social.

10:10This is when Google woke up to the fact that Facebook is on a runaway trajectory and all hands on deck. Google needed, this is when Google starts working on Google Plus, which of course has since been sunset. But it became this catalyzer within Google for, hey we need an answer and they were looking for all the help that they could get and all the experience that they could acquire to speed run this whole thing and we became that catalyst for Google Analytics because what they saw in us is like they frankly they did not care for the PR product that we were building what they cared about was the experience and some of the infrastructure that we've accumulated that we could bring in to fast run and fast track this whole thing to say, hey, if I'm using Google Analytics, I can get really good insights on how my social engagement is going.

11:05And I can figure out how Google Plus is helping me and all the rest. So that became the kind of the mutual point of win-win situation that led us joining forces with Google Analytics. In being brought in as an acquisition, how'd they determine leveling or compensation or any of that stuff? Google, as many other tech companies, run the same process. It's usually kind of accelerated batch model, right? Where you bring in the whole team and you kind of do a wholesale evaluation. But otherwise, everyone on our team went through the panel, through the full panel, myself included. So that was the bar.

11:44And really, the exercise for them was to vet the quality of the team and then to figure out where kind of on the leveling do all the engineers stock up. I see. Okay. So it's, it is just a normal interview process. What about in the case though, because you're bringing in company and assets and stuff, is that just like a giant signing bonus or something? Or how's that? It depends on the company that, you know, and then the deal of course. So there's both players to it. Exactly. As you said, right. There's the, like, there's the IP, there's the customers there's the people and you can treat those as kind of distinct conversations right so as far as negotiations concerned are you acquiring the product what are you going to do with the customers right is there ip there are you going to use the technology many like there's the aqua hire acquisition where you're effectively acquiring the team you're saying cool thing you built but that's not the thing i need you to be building let's just put that on the shelf and not just focus on like rebuilding the thing in our infrastructure.

12:47Postrink was closer to that. It was not a full kind of aqua hire because they brought in and they used the IP. But for all intents and purposes, the thing that we built ourselves, we had to rebuild from scratch with Google Analytics because you're just operating at a completely different order of magnitude, right? And it was an amazing trial by fire, if anything, for our engineering team, because we went from managing tens of thousands of analytics accounts. And granted, yes, we were crawling the web at a fairly large scale to operating in a service that was literally executing on half, more than half the internet, right?

13:22So like to make a modification to a data pipeline. I remember the conversations fondly, like sitting with our SRE team and they're like, okay, well, you're going to add this bit field right here. That's going to be this number of exabytes of data. I'm like, oh my gosh. Okay. Let me think about this more carefully, right? Because before that was not even a conversation point for me and my team. So a lot of it was about the talent and ensuring that they're bringing in the right folks with the right experience and that they're capable of kind of hitting the ground running and working with the rest of the Google Analytics team.

14:04So I want to get into your career at Google. And it sounds like you were placed initially in Google Analytics. I'm curious what the leveling was and the role. Were you a manager? Because you mentioned you brought your team with you. So I was founder and CTO of PulseRank and I joined as an engineering manager because effectively that's what my role was at Google. And I was very invested in making sure that we build the right thing the right way and kind of manifest the vision of why we started this damn company to begin with because I strongly believed in like, I want to help publishers understand.

14:44That was still like me scratching my own itch of like, I'm going to make Google Analytics solve my problem, damn it. And I think we succeeded at that. It took us about a year to effectively rebuild our stack. And the team was on a good trajectory. And that's where I took a step back and looked around for, hey, what's the next adventure? I was in a really good direction within Google Analytics. I remember a conversation with one of our VPs at the time where we were having one of these career sit-downs. And you're like, look, you have all the things you need. You're on track for a director. Here are the things that I could see you take on and all the rest.

15:32And I'm like, huh, that sounds all exciting. And I'm very honored that you think I'm capable of doing this work. But at the same time, I could not let my curiosity get the better of me. Google is such an amazing technical playground that even though that first year was a pressure cooker of like, I need to get this thing shipped, right? And I'm going to work my butt off to get it working. I also was spending every free moment of time just like diving deep into various design documents and wikis. Google is an open culture, which is the thing that I loved about Google. And I could approach anyone and everyone about any piece of infrastructure, any product.

16:15And I leveraged that. And I learned a lot because, you know, I remember building PostRank and reading papers that were published about Bigtable this and Hadoop that. and trying to figure out how to replicate some of the magic that Google had. And then I'm reading the internal design documents, and I'm like, oh my God, this is three generations ahead of this paper that I was reading. This is amazing. I wish the world knew about this, right? And then half the time I would talk to the engineers. I'm like, how come we never publish the follow-up on this thing? They're like, we're too busy. It's not that it's a trade secret.

16:46We're literally too damn busy. She finds us the time, and I was like, huh, well, that's interesting. Like, wouldn't it benefit the entire world, the entire like intranet if we did more of this? Which led me down some interesting conversations. And I discovered this group within Google, which was called Make the Web Fast, which was basically a skunk course project started by Sergey around, well, how can we make the intranet fast? And it's a very self-referential, self-motivating thing. that there's very direct evidence that the faster the internet is, the faster users browse the web, the more they use Google.

17:27So it's a mutually beneficial relationship, right? So within that group, we had all kinds of fun projects. We literally built radio towers trying to figure out how to build more effective cellular networks. You know, dog trenches built towers. We worked on TCP IP and trying to figure out how to reduce congestion. We worked on optimizing proxies for, hey, turns out that most humans write terrible websites. Could we just optimize it out on kind of dynamically as a web server plugin to, hold on a second. When we say make the internet fast, what does it even mean? Like we know it when we see it and when we feel it, but if you had to put a number on it, how do you express that, right?

18:08So if you walk the full dynamic range of those questions, like it's just absolutely amazing laboratory of like experiments to run. And I was really fascinated about that. And I remember at that point in time, I realized that I had an opportunity to cash in some chips. And I could say, look, I see a path towards kind of career growth as a director in this. And that would be a great achievement. On the other hand, I could take a step back. I could take this lateral step and go and work as an IC in this field, I'm actually not quite even sure how I'm going to contribute. I just know that this is a really cool area that I'm passionate about.

18:51And I'll hopefully probably find something useful that I can do there. But that would definitely take me off the career track, right? I remember that conversation. It was like, well, you know, if a IC bar at Google is actually really high, so if anything, you're going to be down-leveled likely and you're going to reset your track. I was like, okay, I think that's a trade worth having because I get to control my time. I get to work with people I look up to and I get to work in a particular domain that I think is just absolutely fascinating. So I made that pivot. And in retrospect, it was an amazing right decision for me because it allowed me to kind of flex my curiosity and technical muscle in a completely different direction.

19:35You turned down this more clear cut director path, which would be the clear cut growth path for career to optimize for other things, which sounds like intrinsic motivation in the work, curiosity, and also sounds like maybe you got some time back. Is that the math you were doing in your head and making that choice? Exactly that. And it was a lot of uncertainty in that process, but this is kind of a repeating arc through my career where as a manager, your time is managed by others right like you are you are in the service of others and i wanted to shift gears and and go into a mode where i have more self-direction and go pursue some like interesting research or uh projects that like where i can apply my own skill set in a unique way so later i saw that you switched back to management and you're director of uh developer relations at Google.

20:38So what was the rationale then switching back given what you just said? Yeah. So I've seesawed a number of times through IC to manager. And I think this is a path that is vastly underappreciated and is actually surprisingly common for a lot of like high performing ICs and like you'll find in principle and then higher up roles. And the observation is, I think it was a tour of duty when I'm doing management. To be a really effective principle plus engineer, you need to have very good dynamic range technically. You can work well at the low level of the stack. You can also zoom out and look at the business requirements, deeply understand what's actually necessary, where can you relax constraints, where you need to hold the line and all the rest.

21:32and once you get attached to certain projects like some projects you can execute and you call it done you're like a great assault that problem right but many problems that are handed to you they become they come as nebulous statements that then evolve into huh this is actually like a team's worth of effort like now that we've understood the shape of this problem and what often happens is you you kind of you see this pattern of um some individual doing the trail blazing work path finding where where we need to climb and then they find themselves okay now i need to recruit a team of people to help me actually build a thing like i know how to get to the top of that hill with a machete but now we need to build a highway right and to build a highway i need a team okay well let me go find a team before you know it you're kind of you're you're tl right and you're exercising your soft power to help direct people and all the rest.

22:25It's not a far stretch to switch to a manager, right? Because like, well, now you're doing performance management as well, plus a few other tricks. But oftentimes you'll find kind of yourself managing or being responsible for the direction of a broader team. And before you know it, like, I think it's a natural transition to say, well, okay, fine. I'm going to like manage this domain. I'm going to manage this as a product team. I'm going to manage this, like the growth of the individuals and the rest. And that's completely fine. That's what happened to me with developer relations. When you think back to the kind of the set of problems that I shared with you, I ended up gravitating towards the question of how do you measure performance?

23:02And I found myself doing a lot of web standards work. I got engaged with W3C and ITF and trying to figure out, hey, what kind of metrics can we define in browsers such that we all have a common ground truth for how to measure performance? And that was great. I did a lot of that. And that was my kind of IC contributor role. Then you get those metrics into browser. And then the problem becomes great. How do I get community or ecosystem adoption in this thing? Well, I need to go tell people and I need to convince them to adopt it, right? And integrate it. Well, now I need to go talk to all of the analytics vendors.

23:37Let me go talk to all the open source projects, right? That problem does not scale with one human. Like you really need help of other humans. This is where the go-to-market and the kind of developer relations part came in. And I quickly realized that it would be really helpful to have a team of people that can focus on this thing. And this is the thing we deeply care about. We want to accelerate the adoption of these things. It's kind of my model based on prior experience was, look, if we just leave this ecosystem be as it is, in three to five years time, we will probably see the adoption that we want.

24:11But I'm not happy with three to five years. Like my question is, how can I compress it into one to two years, ideally one year or less, right? And for that to happen, we need to inject energy into the system. The energy is a team of people. Great, fine. Let me switch to my manager role. I'm going to go recruit people and I'm going to put my organizer hat on and we're going to go and execute this problem. So when you say tour of duty, it's this starting as an IC, doing pathfinding, and then becoming a manager potentially is all in service of a mission or a problem. Like you see some problem, maybe EM is the best way to solve that problem.

24:51Maybe IC is, but it's really all about the problem. Exactly, exactly. And it's kind of like your relationship and your commitment to the problem, right? Because not every problem has to become your mission. I'm often engaged in projects where I'm participating, I'm actively contributing, and then I say, look, guys, you're well on the way. You have the right vector, go execute. You have everything you need. You don't need me here. And in fact, that's my kind of measure of success as a principal engineer oftentimes, right? I don't, if the team is dependent on me, that means that I'm probably doing something wrong because my job is to up-level the team and make sure that they can deliver this thing self-sufficiently.

25:28So if you know the Homer Simpson meme where he like disappears into the bushes. Yeah. Right? Like it's like kind of like that. Like if I can pull that off successfully on a project and the team continues executing in the right direction with a good velocity, like that's success for me oftentimes. Now, occasionally I stumble into a problem where I'm like, either I'm like, just it's in my bones and I feel like I need to be like, I want to own that problem and see it to a logical conclusion or I have some unique position or leverage in it where like yes I'm the right person to take on the broader responsibility and kind of execution of the team to see it through completion and it's a judgment call for which you know when you pull that off and when you say okay this is going to be my tour of duty like my job is over the next I know that I'm committing for the next two years or something to like really run this thing on this ship and that's my responsibility.

26:23And you know what, at the end of it, I'm going to do the same thing I did before. I'm going to like my success, I need a succession plan. And my succession plan is either we solve the problem when it's done, or I've set it on a trajectory where I can successfully pull back and focus on the next thing and focus on the next thing. I'm going to become an IC again and go find the next thing to solve. I know later the story continues with you went to Shopify, you became an IC, and you went from a principal engineer to a distinguished engineer. I'm curious, what is the tour of duty there that got you promoted?

26:55One of the, I think, underappreciated or misunderstood aspects of being a principal engineer is when people ask, what do you do? Like, you know, give me a job description because I want to become a principal engineer. Like, what are the boxes I need to tick? And the answer is actually embedded in the question because people that have to ask that question by definition are not principal engineers. because the problems become ambiguous enough at that level where you kind of have to figure it out yourself. My expectation for, I mentor a lot of principal engineers that come into Shopify. My expectation for someone coming in that level is, look, you basically, to use an analogy here, you're going to be dropped, parachuted into foreign terrain.

27:40You're in a reconnaissance mission. And within the first seven days, you should figure out the lay of the land and figure out where the problems are. build alliances with your directors, VPs, whoever it needs to be, figure out what their problems are, and then kind of understand the situation and figure out where you can apply your particular skill set to help them, right? I don't have a prescription for you. Like, it is your agency. It is your problem to figure this out. And that requires a particular kind of skill set and a toolkit that you have to develop to figure that out, right? Because there's a lot of ambiguity in that.

28:17Going to a level above that, kind of as a distinguished engineer, it's even more amorphous in many ways. There isn't any shared definition, right? But the way I think about it is proof of the dynamic range of the problems that you can solve. So, and then dynamic range kind of goes in a couple of different directions. One is technically. So are you able to, have you shown a repeatable record of being able to operate at all layers of the stack? If you're engaged with a team, are you able to kind of go all the way down to the bare metal and understand what are the constraints, what are the problems, kind of from ground truth, reconstruct the problem.