In short
The Peterman Pod - Episode Summary: Meta Senior Staff (IC7) Eng's Honest Demotion Story
Podcast Overview Podcast Title: The Peterman Pod Description: Sharing software engineering career stories to help you accelerate your career. Hosted by an ex-Staff engineer at Instagram. Episode Title: Meta Senior Staff (IC7) Eng's Honest Demotion Story Episode Description: Igor, a senior staff engineer with experience at Meta, Google, and Cruise, discusses his journey wanting a demotion at Meta and the challenges he faced in that process.
Key Links
- YouTube: [Watch Here](https://youtu.be/i1iBweuOQI4)
- Apple Podcasts: [Listen Here](https://podcasts.apple.com/us/podcast/the-peterman-pod/id1777363835)
- Transcript: [Read Here](https://www.developing.dev/p/meta-senior-staff-ic7-engs-honest)
- Igor's LinkedIn Post: [View Post](https://www.linkedin.com/feed/update/urn:li:activity:7401415295409700864/)
Episode Timestamps
- 00:00:00 - Intro
- 00:00:37 - Why he wanted a demotion
- 00:07:32 - Differences as a Senior Staff at Meta
- 00:16:01 - Meta vs. Google culture
- 00:19:09 - Downleveling at Google
- 00:23:17 - Transparency in sharing experiences
- 00:25:11 - Quality of life at different engineering levels
- 00:30:42 - Senior Staff promotion at Google
- 00:42:27 - Mentorship experiences
- 00:43:11 - Career regrets
- 00:46:46 - Advice for his younger self
- 00:49:06 - Outro
---
Summary of Discussions
- Motivation for Demotion
- Igor's desire for demotion stemmed from feeling out of place at the Senior Staff level (E7) at Meta. He perceived that he couldn't meet the high expectations set for this level and preferred roles that allowed him to engage more in coding and debugging instead of management responsibilities.
- Challenges at Meta
- High Expectations: At Meta, the performance review process placed Igor against longstanding, high-performing colleagues, which created immense pressure.
- Lack of Appropriate Support: Igor found the onboarding process challenging, as most senior hires at Meta were already familiar with the company’s culture and processes. He felt that the management did not have adequate experience to support new senior recruits effectively.
- Comparison: Meta vs. Google
- Cultural Differences:
- Ambitious Goals: Meta often set aggressive deadlines, leading to constant pressure on engineers to provide updates.
- Work Environment: Google was described to have a more reasonable approach to deadlines, encouraging a better work-life balance.
- Experiences with Demotion at Google
- Igor faced challenges in transitioning back to Google at a lower level (E6). He noted that while Google has a process for promotions, demotions are less common and can raise red flags for recruiters.
- Mentorship and Learning
- Igor shared personal mentorship stories, emphasizing the importance of guiding junior engineers and the potential for them to rise quickly within an organization.
- He expressed that sharing personal struggles and career journeys can help others avoid similar pitfalls.
- Personal Reflections and Advice
- Career Regrets: Igor reflected on his idealism about working at major tech firms and how it was tempered by corporate realities.
- Advice for Younger Self: He advised younger engineers to work on projects that add real value, and to seek opportunities that align with their passions.
- Career Goals and Self-Assessment
- Igor mentioned that he found greater satisfaction working at lower engineering levels (E5, E6) where he could engage more with coding rather than management roles that come with higher expectations.
---
Key Takeaways
- Self-Awareness: Understanding personal strengths and what brings satisfaction in work is crucial for career decisions.
- Culture Fit: The importance of aligning with a company’s culture and processes significantly affects job satisfaction and performance.
- Mentorship Matters: Investing time in mentoring can lead to exponential growth for both the mentor and the mentee.
- Transparency: Sharing personal experiences can foster a more open environment and assist others in learning from challenges.
---
Final Thoughts This episode of The Peterman Pod provides deep insights into the intricacies of career progression within major tech firms, focusing on the personal journey of Igor. It combines practical advice with personal experiences, making it a valuable resource for engineers at any level seeking to navigate their careers more effectively.
Written by AI. May contain mistakes. Listen to the episode to check what was said.
Chapters
Tap a time to open that second in VOThe Desire for Demotion
0:40 to 1:50
Igor explains his desire to drop a level at Meta and the challenges faced.
“You mentioned in a post, a pretty famous post that I'll link so people can see, that Meta didn't have a process for demotion and you were looking for demotion.”
Expectations of Senior Staff
1:50 to 3:40
Discussion on the expectations of senior staff engineers and the challenges of ramping up.
“You need to be an expert, a big expert in the field.”
Personal Experiences at Meta
3:40 to 5:50
Igor shares his experiences and feelings of inadequacy in his role at Meta.
“And I believe that within this year and two months that I spent in meta, maybe if I'm generous with myself, I maybe achieved like E6.”
Transitioning from Google to Cruise
5:50 to 8:00
Igor discusses the differences in his experiences at Google and Cruise compared to Meta.
“And I understand that for the company, it's a difficult thing to implement because I already have some granted stock.”
Challenges of Ramping Up
8:00 to 9:40
Comparison of the ramp-up process at Cruise and Meta, including difficulties faced.
“They got promoted and then a year or two later asking again and then climbing the ladder.”
Learning and Management at Meta
9:40 to 12:20
Igor reflects on his learning process and the support from management at Meta.
“What were the differences between those two ramp-ups in your opinion?”
Job Mobility Concerns
12:20 to 14:00
Discussion on the risks of job mobility for senior individual contributors.
“If they say, like, this other person just ramped up, like, you know, three quarters ago, and here's what they did, I would have followed that recipe.”
Expectations and Project Challenges
14:06 to 15:18
Discussing personal expectations and project complexity at Meta.
“But yeah, I was, so the part, like just the few months leading to my leaving meta, I was working on a project that initially I thought it would take me like two weeks to accomplish.”
Cultural Differences: Meta vs Google
15:18 to 17:40
Exploring the cultural differences in work environments between Meta and Google.
“Meta tends to set up very ambitious goals.”
The Process of Demotion Request
17:40 to 21:42
Detailing the challenges and considerations of requesting a demotion at Google.
“I've heard some people saying that Google as well has felt a little bit of pressure too, but hard to say.”
Show all 22 chapters
Sharing Personal Experiences
21:42 to 24:17
The importance of sharing personal career experiences and the privilege it entails.
“Um, one was, uh, sound like sounded like the best match for me.”
Reflections on Career Progression
24:17 to 28:00
Reflecting on career ambitions and the desire for a better work-life balance.
“My kids are already grown up and off to college and don't have mortgage on my house.”
Navigating Imposter Syndrome in Tech
28:00 to 29:50
Learn about the nuances of imposter syndrome and its impact on career decisions.
“been E7 long enough to get comfortable in the role, maybe if I just kept doing it for some time longer I would have started enjoying it Yeah.”
Promotion Pathways at Google
29:50 to 31:34
Discover the pivotal projects that led to promotions within tech companies.
“I just felt like I was not enjoying the ride essentially.”
Building Machine Learning Infrastructure
31:34 to 33:40
Understand the challenges and collaborations involved in creating ML infrastructure.
“And my role was to make sure that this ADS training infrastructure can run on those TPUs.”
Testing Infrastructure Without Hardware
33:40 to 35:51
Explore strategies for testing software infrastructure in the absence of physical hardware.
“And, you know, so it's also like where you need credibility.”
Balancing Data Center Resources
35:51 to 39:35
Learn about the complexities of managing data center resources and throughput.
“And the floating point arithmetic is relatively slow.”
Predicting Demand for TPUs
39:35 to 42:01
Delve into the challenges of forecasting hardware needs in tech projects.
“and you say, I need more disks, not for storage, I need more disks for the throughput.”
Mentoring Experiences and Insights
42:01 to 43:08
Explore the host's experiences and lessons learned from mentoring interns.
“Yeah, embedding lookups in certain types of embeddings where essentially the chip is very good at doing matrix multiplication and accessing memory by big chunks of memory.”
Disillusionment with Corporate Culture
43:09 to 45:06
Discuss the speaker's disillusionment with corporate life and changes at Google.
“When you look back on your career, is there any regret that you have that some people could learn from or could help people avoid?”
Navigating Career Decisions and Project Relevance
45:07 to 47:26
Learn how to assess project significance and make informed career choices.
“So it was just like you feel like, yeah, at the end of the day, I'm just a sell in the spreadsheet.”
Switching Projects in Tech Companies
47:27 to 48:46
Examine the ease or difficulty of switching projects within different tech companies.
“Like ask yourself, like, does my company really benefit from what I'm doing?”
Transcript
Automatic transcript. May contain errors.0:00Then I asked the management, can I actually drop a level?
0:03Igor:This is Igor, a senior staff engineer at Meta, Google, and Cruz who wanted a demotion. You cannot still justify my level with the project being completed. He failed to get the demotion at Meta, and going back to Google at a lower level was harder than expected. They don't have a process for bringing people at a level down, but they made it possible for me. We went over that experience and some transparent discussion across levels. which engineering level has the best quality of life. There are many things that can go wrong when you share stuff like this. What makes you willing to share? Here's the full episode.
0:45You mentioned in a post, a pretty famous post that I'll link so people can see, that Meta didn't have a process for demotion and you were looking for demotion. can you talk about what led you to wanting it to motion at meta yeah so when you join a big company meta is not the exception here but you're joining a big company at the very high level like senior staff there are certain expectations from your performance right come performance review like it met it's called psc uh you you're being judged against other senior staff engineers in organization. As of a year ago, they started to lay off people.
1:33They started doing the Amazon thing essentially, like let's lay off the 10 % lowest performance. What it means is that you have at most one year to ramp up to be comparable in performance to other old timers of the company. And also, what does it even mean to be an E7 in a company like this? You need to be an expert, a big expert in the field. You need to know a lot of people. You need to understand the infrastructure that you're working with very deeply. You probably need to be easily familiar with any piece of code that your team is working with. and also you need to know all the surrounding, like what all the surrounding teams are doing.
2:22And you need to know all these people, like all core people around you, and they need to know you and they need to trust you, right? That's a very difficult thing to achieve within like a relatively short period of time. I initially saw that, okay, when you were, so again, joining Meta day one, I know less than an intern sitting next to me, who was hired two weeks prior to me. So I start from level zero. And then hopefully you'll be able to get to something that L3 would perform. So you can take some very easy task that under supervision from other folks, you can kind of promote yourself to a junior engineer.
3:11Then you do some more stuff. You learn additional things. you can get promoted to like a less junior engineer, like a E4. And then slowly you ramp up, you know, to more and more like higher levels. Essentially, like you start from zero and you climb the ladder as fast as you can. Yeah. So you ramp up yourself to, you know, the level that you're supposed to achieve. And, you know, the more senior you are, the more levels you need to jump through. And I believe that within this year and two months that I spent in meta, maybe if I'm generous with myself, I maybe achieved like E6. So I don't feel like I reached E7.
3:55What did you see that made you think you weren't living up to the expectations? It's self-judgment, mostly. This scale where we call numbers, it's not like quantum states, right? There are many things in the middle, right? It's a continuous scale. So it's very hard to say where you are on the ladder. But again, comparing myself to other senior staff engineers, the work that I knew, I felt like if anybody is going to be on the chopping board, that's going to be me. Right. So that was one thing about this ramp-up process. And the second thing, I also noticed that I really enjoyed doing the coding stuff, like just sitting down, debugging things.
4:53That's what I really loved about the job. So during my ramp up, when I was around like E5, E6 territory, that's what I really wanted to do, right? I felt like as someone who was programming from the age of 12, I really liked coding, debugging, designing, mentoring more junior people. But like a senior staff engineer, it's more than that. It's like a leader who spends most of the time in meetings and design docs and touches much less code. It's a slightly different type of work. And just by going through this process, I realized that actually I want to be in the E5, E6 territory. And then I asked the management, can I actually drop a level?
5:50And I understand that for the company, it's a difficult thing to implement because I already have some granted stock. I have a certain compensation package. And so how do you execute that? It's not a well-polished process. I know that they can switch levels when you go from one title to another. So let's say you were a director and you want to become an AC. So they do have some of that demotion process. And frankly, I don't know how they do it. But yeah, staying in the same job category and dropping level, they said it's not possible. I see. So when you asked if it's possible for Demotion, your manager went to HR and kind of the result was just it's just an impossible thing to do.
6:47You know, I asked the question and I got the answer. No, maybe if I pushed harder, if I like went to talk to the, you know, VP or something, they would have made it possible. but at the same time I also felt like I'm too far outside of my comfort zone and before coming to Meta I worked a little bit at Cruise and before that I worked at Google for 14 and a half years and that was my comfort zone I just had this option the easy way out essentially I had a recruiter who was contacting me periodically like i already better to come back to google so this time i just said yeah let's talk when you were at google and at cruise you were at these senior levels as a senior staff engineer what's the difference that makes it so that you can perform there but not at meta so when i came to google i started from the lowest e3 like l3 level uh so junior software engineer and slowly Over the many years, I raced through promotions, many times failing.
8:02They got promoted and then a year or two later asking again and then climbing the ladder. My last promotion to senior staff happened because I was able to accomplish something big. The project that I worked in that project for several years, so I was already an expert. I knew everybody, right? Everybody knew me and I was able to build something that I can be proud of. But again, coming then, then leaving Google and going to cruise, it was also very difficult for me to ramp up there because you're coming, starting from zero. And the first year was very difficult for me in cruise but over time i found like a relatively safe uh like you know my found some comfort zone there until cruises a company uh went the bad like they had the accident and then they had layoffs and then like people were just quitting all the time like losing people so I decided that I want to leave as well.
9:13But before the accident at Cruise, I actually was quite happy. I was able to do some coding. I was able to do mentoring of VacerFox, and I was able to also act as an UberTL a little bit, guiding the team on things that they need to do. You successfully ramped up the senior staff at Cruise, and then at Meta it was a bit harder. What were the differences between those two ramp-ups in your opinion? Cruise was a smaller company, so less infrastructure to learn. The team is less crowded space, I would say. You can easily carve out space for you to grow into. There is lack of people and you can say, okay well this project needs someone to work on and and you can just go and work on on this stuff at meta i felt like uh it's quite crowded like especially among the city of hogs it felt a little bit like they have too many frankly at least in the work where i worked so i don't know about all of meta but it felt like the space is a little bit crowded you need You need to find scope for yourself.
10:38Is there anything that you would have changed in your onboarding process that could have made things different? Probably. Again, I didn't switch companies that many times in my career. I started at Google, spent so many years there, then I only switched to Cruiser, then Meta. So I'm not very well experienced in switching companies. and essentially, you know, this whole ramp-up process is new to me. And I also felt like the management at Meta also doesn't know how to import senior folks because most of the senior folks in the company, they grew within the company to those senior levels. Even if they came to this specific org, you know, from another org within Meta, but at least they already have the meta knowledge.
11:35They know how to run a job in the cluster. I think I spent a lot of time just reading docs and trying to build this foundational knowledge. I think I spent too much time doing that. It is much better to just get your hands dirty and just try doing stuff. And then doing those things in parallel, like you build something, accomplish some tasks, and also you're learning in parallel. And I did more of the learning part, less of the... The building. Yeah. Is there something that you wish was done or maybe something that would have helped you from the manager side? You're just guiding me, right? If they say, like, this other person just ramped up, like, you know, three quarters ago, and here's what they did, I would have followed that recipe.
12:33We just didn't have the recipe. Like, I'm not, I don't have any complaints against my management. Like, they, you know, they tried to support me the best. They just also lacked this expertise of, like, ramping up with senior folks, you know. So it was a learning experience for all of us. I've heard in conversations from other people as well that I guess job mobility as a senior IC actually becomes progressively riskier or maybe scarier, I guess, because you get so used to your existing org and all of that. um is that something that you've seen in other people or like peers and and as well in that uh senior ic job ability is a lot tougher i've seen folks at google that uh left google and then came back a year or two later saying that it didn't work out well for them um And I, yeah, it's, you know, people don't share things like that openly usually.
13:40So it's much harder to hear those stories. You hear the success stories, you don't hear the failures. Also, I think that if I kept trying, like instead of going back to Google, I went to, I don't know, some other company, I would be much like, you know, more experienced in ramping up and then maybe maybe if you do this often enough uh it becomes a habit and and then and then you can do it easily you mentioned that you you thought you weren't meeting expectations based on your own judgments um i'm curious did you ever get any feedback from your manager or anyone saying hey you need to do more for your level's expectations yeah i did get feedback i just you I don't necessarily want to openly share all of that.
14:28But yeah, I was, so the part, like just the few months leading to my leaving meta, I was working on a project that initially I thought it would take me like two weeks to accomplish. It was like a small thing that I thought would be easy to do and turned out to be actually a lot more involved and a lot more complicated. and I felt like even if I finish it, which I almost did, like I almost brought it to completion, even if I finish it and launch it, everything's successful, it is still not an E7-level project. You cannot still just justify my level with the project being completed. Probably it's my fault in which projects I pick.
15:18uh you know or how i underestimated the complexity of the thing because you're going back to google um are you going back to an org where you have all the existing context and and relationships and all that no no everything will be new to me new people new infrastructure new new everything but at least I know I know how Google operates I know the culture there are differences in cultures also between the companies and I think that Meta also in terms of internal culture at Meta I didn't feel like it fits me the best What's the biggest cultural differences between Meta and Google? Meta tends to set up very ambitious goals.
16:10They give you, like, oftentimes they will give you a very arbitrary deadline saying like, okay, this project, you need to finish it within like, you know, by September 15th, whatever, like one month from now. And then everybody works hard. Like there is a lot of pressure. Like you need to constantly send the updates to the leadership, how the project progresses. Comes the date, the deadline, the project is still unfinished and it just keeps dragging on. and everybody is fine with that. Again, I don't know, I haven't seen all of Meta. I've seen the specific org where I work and it feels like, so then what was the purpose of setting this aggressive deadline, right?
16:53Then like, you know, are we like, you know, you can do it once, but like after five times going through this project, through this like artificial pressure with like non-realistic goals, then people just say, okay, you know what? 5 p.m., I'm going home. I'm not going to try to work hard because I know that this whole pressure is artificial. I think Google is much more reasonable in that regard. If there is a deadline, it's probably for good reasons and people would work hard, but usually there will not be pressure. again talking about google like as of 10 years ago i'm not sure that it's still true today but you would be pressured to to fix something or to accomplish something when there is really uh you know exceptional case for that like you you wouldn't be pressured to to work on the pressure you know for years yeah i mean i get the sense that the industry as a whole is kind of becoming a little bit more intense when it comes to execution and deadlines.
18:06I've heard some people saying that Google as well has felt a little bit of pressure too, but hard to say. It depends on the org, I'm sure. Yeah, it could be. And again, I didn't feel personally much pressure at Meta, but just talking to other folks, seeing how they work and operate. And as I said, there is this cultural thing where the leadership wants updates and everything, and then the people on the ground, they actually kind of dismiss it, essentially. It's like, you know, it almost feels like elementary school, where the teacher is yelling, but kids are still playing. If you're yelling too much, it stops working, essentially.
18:55If you constantly put pressure on your people, it just doesn't work anymore. So what do you do next? You start laying off people. Yeah, it might work to some degree, but again, people adjust to everything. That's crazy. You mentioned that when you reached out to Google, you explicitly asked for a demotion or going back as a L6 before you were in L7. Was that a challenge or was that just a very straightforward process with the recruiter? It was a challenge for the recruiter to also make this happen because they have a process for bringing people in back at the same level or even level up. They don't have a process for bringing people at a level down, but they made it possible for me.
19:41Again, I can totally see how a person comes back and says, I want to be a level down. There might be some red flags. There might be things that I'm not telling. It's a gamble for a manager to hire a person like that. Also, just by giving me offer at the level down, they understand that they will not be able to match my compensation at the previous place. So would they even accept it? right and so so i had to like and assure my recruiter that yes i'm happy to accept the offer at the lower compensation like you know a lot of people will not do something like i did because it's like a significant drop in compensation you mentioned um stuff that people might be hiding from the recruiter what comes to mind when you you mention that i don't know maybe you maybe you did something in the company that you know you like a fireball offense or something like a like i i don't know like if if you're a manager and somebody comes to you with like you know saying like yeah previously i was the this level now i want to be level below please hire me it does sound fishy like why don't you like what's wrong with you like you like you know there are so many companies out there like why why would you come back to do this company like why don't you try something elsewhere like i i don't know what questions popped up in my hiring manager's head when he saw this google's current process do they do host matching or what was the like did you meet up with the hiring manager beforehand or just through the recruiter?
21:34The recruiter sent me a few openings and, uh, asked me which one sound interesting. I spoke to a few hiring managers. Um, one was, uh, sound like sounded like the best match for me. And, uh, then I spoke to a few engineers on the team. Then I spoke to the manager's manager. and that was it. Another nice thing for me is that they didn't require me to re-interview.
22:04Igor:Oh, interesting. But it had been like three years maybe? Over, yeah, three and a half years. So yeah, they didn't ask me to re-interview. Actually, that was another interesting thing that they said. Like if you're coming back as E7, you definitely don't need an interview because you were already L7. But if you're going to be L6, then you might need to do a coding interview because who knows if you wrote code while you were out. That's kind of funny. Yeah. I got an exception from coding. I see. That's funny. You got promoted out of competence in the lower levels. Yes. But it's understandable when people go to higher levels, they write less code.
22:53and maybe they get rusty who knows it's kind of crazy that you could boomerang after had been three years or more yeah to be fair i don't know if that's a general google's policy or it was just a special case for me i don't i don't have any friends that did this recently to to understand if if like they did it for anybody else you mentioned earlier that you know a lot of people don't share this kind of stuff about emotions and these types of things and i'm curious what makes you willing to share i'm generally a very open person like i i like sharing my my life experiences positive and negative with friends and the you know i'm yeah i'm generally quite open and also i'm quite confident that's another thing like you wouldn't share if you're worried about your career, if you're worried about, you know, there are many things that can go wrong when you share stuff like this, right?
23:58So I'm very privileged, essentially. I don't need the work visa, so I can live in this country without being attached to a certain employer. My spouse has health insurance. My kids are already grown up and off to college and don't have mortgage on my house. So I can afford not to work for a few years.
24:30I'm fine with... Let's say I said something in this chat right now and Google says, we don't want you for some reason. I'm totally fine. I will just... So that's a position of privilege, right? Definitely. I mean, you're free. Yeah, you're entirely free. You're your own person, which is awesome. Yeah, so that's a position of privilege that many other people wouldn't have. So if another person asked me, should I post about my challenges at work? I would probably say no. unless you feel it's so so safe and secure and i mean you mentioned that google you were asking for promos you went from their lowest level up and you were pushing and pushing for promos and that's a very different mindset from now which you were kind of pushing for the motion i'm curious is there did something major change that made you um not motivated to go for promos anymore in your career?
25:36Essentially, if you are a senior engineer, right, E5 or E4, you don't even know what the life of two levels above you looks like. You can imagine what happens at one level above you, but it's very hard to see what does a principal engineer do. Most people don't know. And I don't know. By going through this, I realized that my happiest time was when I was working in a relatively small team, doing a lot of coding, debugging, designing, mentoring more junior folks. That's where I felt the happiest. let's say money is not a constraint at all and it was just which engineering level has the best quality of life in your opinion which one would you say uh senior uh engineer like e5 l5 uh yeah that's probably least pressure you're still shielded by uh you know probably like if If you're working on a team, you probably have an ETL who is six on the team who, you know, sends all the updates to the upper levels and shields the lower, like the team from all this stuff.
27:04You have the management, also like the lower level managers who shield you from all this stuff. So you can just do the stuff that hopefully you enjoy what you're doing. Like not everybody enjoys doing work. Like a lot of people do it just for money, right? I personally, I was programming since I was 12 years old. So I really like this stuff. I got into this job because I love it and it happens to pay well. So like, but I would do it even if it wasn't paying well. Like a lot of people come to software engineering because just because that's the money thing. So then if it was easy, as easy to do, would your ideal situation be two levels of demotion that's too extreme I think I can still enjoy being like E6 I'm quite comfortable with and maybe I haven't been E7 long enough to get comfortable in the role, maybe if I just kept doing it for some time longer I would have started enjoying it Yeah.
28:16Like, you know, one thing is like, was I really... Like, you know, imposter syndrome is something that everybody has regardless of what your level is. I'm sure Elon Musk has imposter syndrome. So you know, I'm questioning myself, like, was I really, you know, qualified to be E7? And my answer is like, partially yes, but not fully. it's like a multi-dimensional thing and in some dimensions it probably was good and in some dimensions was not this good you mentioned working with other peer ICs that were also really high level is there skills that you saw that they had which would have closed the gap for you personally or something that you thought made them so strong it's often a mistake to compare yourself against a group of people.
29:08That's what gives you a lot of imposter syndrome, where you're saying like, oh, those people around me, they're so smart. They are so good at talking. They're so good at doing presentations. They're so good at communicating. They're so good at leading. But it turns out that it's like one person is good at this, another person is good at this. And you're comparing yourself against a team of people. And so that's always you need to be careful not to make that mistake. So yeah, I think I, you know, given enough ramp up time, I would have been able to reach like full productivity and being useful to the company at E7.
29:50I just felt like I was not enjoying the ride essentially. And I didn't have to. And I debated a lot before making the decision to leave. I was talking to friends, I was talking to family, And pretty much everybody was telling me, like, are you crazy? Like, you're not laid off yet. You're not, nobody showed you the door. Why would you do something crazy like that? And even if you're leaving, like, you don't enjoy it, why won't you try E7 elsewhere? Like, why do you need to go level down? That's, there is a lot of peer pressure. And my answer to that was just like, yes, I can try that. I can do that.
30:29But, like, I don't want that. It's like, why, you know, I'm capable of running a marathon, but I don't like running a marathon. I don't like running. It's like, why would I do that? You mentioned that you kind of rose through the ranks at Google. I'm kind of curious, what was the project that got you promoted to senior staff at Google? It was a project in ads. Ads builds a lot of machine learning infrastructure because, you know, the ranking of ads is a essentially a recommendation system. And I was working on this machine learning infra and when I came to the team, they were doing training on CPUs, not on GPUs, but on CPUs.
31:17And TPU was a new thing, new hardware that Google was developing. It was about to be released like maybe a year or two when I joined the team. And my role was to make sure that this ADS training infrastructure can run on those TPUs. And additionally, when you train machine learning models, the scale matters and the utilization matters. And there are lots of nuances that can, like, you know, your model may be training, but things go wrong. It's online training, which is also, like, something that most people don't know. Like, LLMs don't usually train in online training mode. So there are lots of experience on the team.
32:18Like, the whole infrastructure that existed within that team was built for the CPU training, and it was polished over, I don't know, over 10 years. It was polished to be super reliable, super like you've well monitored, unit tested. Everything is like very rigid, very like polished, right? And here I am coming and building a completely new piece of infrastructure to run on TPUs. And it has to be as good as the old, you know as polished as as monitored as tested as reliable um so it was a big project and uh you know it succeeded yeah it's something to be like i was proud of what i built and so that got me promoted what was the operating model of that project were you tl or were you kind of writing a lot of it yourself or how would you describe the execution Yeah.
33:18Initially, it was a quite exploratory project. So it was just two people. It was the TL plus one person to building a prototype. Plus a lot of people from other teams helping, like the TPU team, the compiler team. There are lots of other people who were helping us succeed. Like the TensorFlow team, we were writing TensorFlow back then. And, you know, so it's also like where you need credibility. Like that's why knowing people and them knowing you is very important. Like trusting you. If you don't have that, you cannot succeed. So it's a really like cross-team collaboration, very big project. And again, hardware like this, you need to decide how many of these chips you want to order.
34:12and they will be delivered like 18 months from now. And right now you don't have any. It's not like you can try out and see. And those chips are super expensive, right? So it's a very risky thing to do for the company and yeah, it's like a lot of risk taking, a lot of like leadership skills that they needed for doing stuff like that. So you were on the, well, kind of the product team, not actually product, you were machine learning infra for the ads org. And then all the underlying infra teams were helping collaborate with you. You mentioned compilers, maybe some training infra, other infrastructure teams.
35:04Yeah. And once the thing started working a little bit, the team grew quite a bit more. A lot more people joined the effort and started working polishing. And then again, first-generation TPUs come in, and then they already tell you, we already designed the next generation for becoming next year. So you're already rushing to adjust your infrastructure to the next thing. And that's pretty much every year you're upgrading your infrastructure. How did you test the initial builds of this infrastructure if you didn't have the chips to begin with? A lot of things were just about input reading and processing, which you can do without having the chips.
35:50The difference between CPU training where you have the CPU and Intel CPU, they can do floating point arithmetic and integer arithmetic in parallel. And the floating point arithmetic is relatively slow. So input reading and processing was free on the same CPU. Going from that mode of operation to you have a relatively weak computer with eight TPU chips. each one of those TPU chips is like order of magnitude faster than the CPU that you had before. If several orders, I think it's like two orders of magnitude faster. Now there is no chance you can do the input reading on the host of those TPUs. So you need to like build the input processing pipelines and everything, but that's something you can test outside of, like, without having TPUs.
Read the full transcript
36:56Actually, we did, many times we did this test where people say next TPU will be, like, two times more performant than the previous one. And a nice test to test your infrastructure is to say, like, okay, we will just remove all computation from our models. Just let's see how fast we can, you know, Feed the data in, take the outputs out, and process this whole thing. How fast your system can run if the KPU was infinitely fast. And that's a nice test to do to see that you don't have bottlenecks. And you do find a lot of bottlenecks all the time. Things that previously were like the thing that schedules which data to train on, it's you know previously was never like an issue and then suddenly this is your bottleneck in this model and many things like that if I'm understanding correctly the TPU consumes data orders of magnitude faster than the CPU so everything around the processing unit like the data loading and maybe I don't know the scheduling and all the other things that are around the TPU needed to be scaled and tested.
38:19And that's a lot of what you did to get promoted. Is that right? Yeah. Plus, you know, there are always like funny things that you get where like certain resources are more available than other resources. For example, so you know, you're training on some insane amount of data, petabytes of data. This data is stored on spinning disks because there is like not enough SSDs in the world. So spinning disks, if you look at the history of spinning disks, they're getting bigger in terms of storage space. But the speed at which they rotate is staying constant. And the speed at which the head moves is staying constant.
39:01So if I gave you previously, let's say like we go back in year 2010, a typical disk would be like maybe 200 gigabytes. and it has a certain throughput of how much it can read. Now fast forward to today's, the typical disk is like 6 terabytes. So your data center has a lot fewer disks to store the same amount of data, but this throughput is like an order of magnitude less.
39:34And you're coming to the people responsible like on building data centers and you say, I need more disks, not for storage, I need more disks for the throughput. And you usually would hear an absurd thing that says, disks are very cheap. Compared to everything else in the data center, super cheap, but you cannot get them. You cannot easily go and buy a few thousand disks and easily install them. You need racks, you need power, you need...
40:05Yeah, it's similar to how during COVID, we all ran out of toilet paper, right? Like the stores just couldn't, couldn't, like, why would you run out of toilet paper, right? It's just because it's bulky item that the stores cannot easily store on the shelves. So that's roughly it. You mentioned having to predict 18 months out how many TPUs you'd need. And I'm curious, when you look back on that prediction of how many you needed, did you over order or under order? Usually under. we try to be relatively conservative because your main goal is to make money like with LLMs if you look at the current world of LLMs they just order more like we want more gigawatts of data centers they don't even really none of those companies is profitable with LLMs they're all losing money and they're all fine with that but when you have a mature business like ads you need to be making money on that so if you wasted if you're over provisioned and this goes to waste then it's not good but then it constrains what kind of models you can train also models don't stay the same so when you're saying 18 months from now we'll not be training what we're training today but you don't know what you're going to be training the attention mechanism wasn't in use 10 years ago and suddenly it's needed.
41:39Your chip might not be even capable of doing that. TPUs are quite specialized hardware. They are not as generic as GPUs. There were certain things that we had to work around a lot of times where certain functionality was very difficult to implement. Like what? Yeah, embedding lookups in certain types of embeddings where essentially the chip is very good at doing matrix multiplication and accessing memory by big chunks of memory. And embeddings are things where you need sparse access, like random access essentially. And that's difficult to do in the chip. Yeah, one thing I want to ask, because you said you enjoy mentoring others, and I was curious, do you have favorite, maybe stories of mentorship or favorite advice that you like to give when you're mentoring other people?
42:40Well, I once had an intern on the team. I was also not very senior back then. I was like L4 maybe, and I had an intern that I hosted. And then later that intern converted to full-timer. and a few years later he became the manager of the team that he interned in. That's funny. Were you reporting to him eventually? No, by that time I already switched projects. But don't underestimate your interns. They can be really, really good. When you look back on your career, is there any regret that you have that some people could learn from or could help people avoid? Not everything was very smooth in my career, But I think it was still like a learning experience.
43:27Like I wouldn't be who I am today if I didn't go through those periods of time. I was very idealistic when I was younger. Like I believe that like Google is a really positive force in the world. That it's really like, you know, I would work here just because it's Google, not because it pays more. like you know not because of some like you know I really wanted to be in Google because of my personal values and how the company operated and then I got disillusioned over the years that yes it's just a corporate like maybe in the as of 17 years ago it was really a more a different company but today it's it is a corporation um just like any any other corporation it has positive things it has negative things but what matters to the company at the end of the day is the bottom line on the financial reports and the uh you know they will do whatever it takes to to get there to increase that number is there something that led to that disillusionment And Google had a lot of remote offices, for example.
44:41And I was working in the Pittsburgh office, which was a relatively remote office. And at the same time, they shut down an office in Atlanta. And it was roughly the same size. And it was quite shocking to everybody. Like, why would you shut down an office with a few hundred people working there? and the answer we got from the leadership they said well we have those big senior vice presidents of the company who decide how to allocate the head count and where to invest like you know I want to hire in Bay Area I want to hire in Seattle I want to hire in this place and just so happened that the big lead who was sponsoring that Atlanta office decided to pull out and nobody else was willing to take over the head count of that office.
45:35So it was just like you feel like, yeah, at the end of the day, I'm just a sell in the spreadsheet. You know, it's like there was very little empathy that company showed to those people. They said like, yeah, we can help you relocate if you want to do other places, but if not, here's your exit package and good luck. And that was like many, many years ago before all the layoffs that happened after COVID and, you know, nowadays people are much less, it's much more understandable that the company can lay off anybody just because this, this project here doesn't make sense anymore. We'll just like shut it down and let people go.
46:21Like this is now a common scene. It was not the case as of 10 years ago. yeah i remember i think when google was a lot earlier it was um you know don't be evil and the culture was very very set on that yeah i think they still like they're still trying not to be evil but again the bottom line often drives they overrides that decision i guess it's true for all
46:48Igor:public companies yeah and then the last question i'd like to ask is if you could go back to when you just entered the industry or you were working at Google and give yourself some advice, knowing everything you know now, what would you say? I worked on some projects that didn't make sense to me, especially when I was more junior. I was working on some projects that, like, why are we building this in the first place? Who needs this? And then a year later, the leadership realizes the same thing and they just shut down the project. So maybe work on what matters. Like ask yourself, like, does my company really benefit from what I'm doing?
47:35Like, and if not, then maybe you shouldn't be there. In that situation, let's say you recognized it. And then your org's going in that direction. You think it's useless. but you're a junior engineer what what could you do to kind of adjust your direction i mean talk to other managers hey find another project there are different companies have different uh treatment of people who want to switch projects uh before google actually briefly worked at intel their switching project was almost impossible like it's it's much easier to just quit and then reapply. Versus Google was very fluid. You could easily switch projects within Google.
48:19So this advice doesn't work for any company. I know some companies like IBM is another example I've heard where switching projects is impossible. Some companies have certain reputations. I don't know. I've never worked at IBM, so I don't know, but that's the reputation they have. Within Google, it was easy to switch projects it was in meta i believe it's easy to switch projects yeah i don't know about other companies yeah i think most modern day or silicon valley companies are inspired by google meta those types of companies so similar culture on team switches so awesome well yeah thanks so much for your time igor i really appreciate it yeah thank you very
49:02Igor:much. And yeah, I hope somebody finds it useful. Thanks for listening to the podcast. I don't sell anything or do sponsorships, but if you want to help out with the podcast, you can support by engaging with the content on YouTube or on Spotify. If you want to drop a review, that'll be super helpful. And if there's any guests that you want to bring on to, please let me know. I feel like sourcing very senior ICs. There's no well studied list out there on Google that I can just search this up. So if there's someone in your org or at your company who you really look up to and you want to hear their career story, let me know and I'll reach out to them.
From the publisher
In this episode, I talked to Igor, a senior staff engineer who has worked at Meta, Google, and Cruise. We discussed his experience of wanting a demotion at Meta and the challenges he faced in that process.
𝗣𝗼𝗱𝗰𝗮𝘀𝘁 𝗹𝗶𝗻𝗸𝘀:
• YouTube: https://youtu.be/i1iBweuOQI4
• Apple: https://podcasts.apple.com/us/podcast/the-peterman-pod/id1777363835
• Transcript: https://www.developing.dev/p/meta-senior-staff-ic7-engs-honest
𝗘𝗽𝗶𝘀𝗼𝗱𝗲 𝗹𝗶𝗻𝗸𝘀:
• Igor's post: https://www.linkedin.com/feed/update/urn:li:activity:7401415295409700864/
𝗧𝗶𝗺𝗲𝘀𝘁𝗮𝗺𝗽𝘀:
00:00:00 - Intro
00:00:37 - Why he wanted a demotion
00:07:32 - Why Senior Staff at Meta was different
00:16:01 - Meta vs Google culture
00:19:09 - Downleveling at Google
00:23:17 - Why he's willing to be transparent
00:25:11 - Best quality of life eng level
00:30:42 - Senior Staff promo at Google
00:42:27 - Mentorship stories
00:43:11 - Biggest career regret
00:46:46 - Advice for younger self
00:49:06 - Outro
𝗪𝗵𝗲𝗿𝗲 𝘁𝗼 𝗳𝗶𝗻𝗱 𝗜𝗴𝗼𝗿:
• LinkedIn: https://www.linkedin.com/in/igorts/
• ML basics youtube videos he made: https://www.youtube.com/playlist?list=PLVcptlT8D7DgN5FtLMFUdHb5pJXW1g0YL
𝗪𝗵𝗲𝗿𝗲 𝘁𝗼 𝗳𝗶𝗻𝗱 𝗥𝘆𝗮𝗻:
• Newsletter: https://www.developing.dev/
• X/Twitter: https://x.com/ryanlpeterman
• LinkedIn: https://www.linkedin.com/in/ryanlpeterman/
• Threads: https://www.threads.com/@ryanlpeterman
• Instagram: https://www.instagram.com/ryanlpeterman
• TikTok: https://www.tiktok.com/@ryanlpeterman




