PyTorch Eng Director: Promo Hacking, Industry Shifts, Regrets | John Myles White

0:00You have to play the game.

0:01John Myles White:It's totally irrational not to play the game. This is John Myles White. He was a director of engineering on PyTorch in MSL. And since he quit recently, we talked freely. I feel like our goals should not be written in a way where shipping a thing we intend to delete is a success. The general perception is the supply of engineers is like way over supply. We also talked about the incentives in big tech. You ran promotions in AI Infra for a long time. You know, the only reason you do anything is because there's a clear story about how it's going to get you a promotion. You saw this. I saw this. What could you do, though?

0:39Here's the full episode.

0:46John Myles White:I'm curious if you're bullish on MSL, like after you were working there for a bit and kind of seeing it from the inside. Oh, this is a, I guess, like a more radical question, but I guess I will be honest. I am very bullish on Meta as a company that I am now a stockholder of, but not an employee. I am very bearish on Meta if you're an employee who's not a stockholder, which just turns out to not be really anyone. But like, you know, I think you can think of yourself as mostly employee if you're mostly getting cash and mostly not equity. The more senior folks are the ones who are more mostly stockholder than employee.

1:27And I think that's the change in this. I do think like my sense is that it's enforced where they don't get the perception is unique to MSL. My perception is that I think just meta as a place to be an employee is less enjoyable than it used to be. but it actually is being run very effectively if what you care about is the bottom line of the business. And so like I continue to invest in Meta and I suspect I will continue to invest because I do think it's actually from a business perspective run quite well. But I do think pretty uniformly, I think it's not unique to MSL. I do think it's a much more stressful time to be an employee there than before.

2:03John Myles White:What's the part that makes it where you're saying employees might not enjoy working there as much as they used to? I think that, I mean, I think this is true of all of Silicon Valley. I think in general, like there was a lot of sense that like it was incredibly easy to lose your glute employees and you had to do everything impossible to sacrifice to retain them. And you were constantly constrained by an undersupply of employees. And I think basically everyone in Silicon Valley's view, as far as I can tell, is that that's mostly not true outside of like a small set of like AI researchers in frontier labs, where I think people do still behave this way.

2:39In fact, maybe behave this way more than ever before. But I think for everyone else, the general perception is the supply of engineers is like way oversupplied. And especially I think actually people's concern is with AI, maybe they're really oversupplied. I think what happens is a new market dynamic, which is like, I think as an employer, you're thinking, I don't actually have to make so many sacrifices to acquire and retain talents. And so therefore, I'm going to make fewer of them. And this can play out in a million ways, but it can be like compensation, but it can also be things of like, do we do things that upset the employees or do we not?

3:15And how hesitant are we? How much do we give them a voice and how much do we not give them a voice? I think that stuff has changed a lot in the years I was at Meta. But my impression is Meta is not in any way unique here. This is just a general property of all of Silicon Valley. but I do think you know as an employee I think the like the labor supply and demand situation is super different from when I started and I think it is a thing that is going to make generally being an employee rougher periods.

3:44John Myles White:It feels like it can be subtle as well like what you described of I guess employees having less leverage you could see that affecting maybe even like reorg decisions or things like that where it's like you know if people leave that is part of the calculus of the reorg and that is less of a downside in now and maybe in the future so i can see there being a lot of downstream effects where uh things just don't go as well for for employees oh yeah i mean there's just so much stuff that in the previous versions of meta was done that i think there was a bad decision for the business but made sense from the context of retaining talent um you know and i think that the company is just doing less of that but on the flip side I mean I guess to be clear I didn't say as I supposed to before I do think for most people like fundamentally just like the possibility you might get laid off is the number one emotional thing that causes people stress and living in a world where you know that is both happened recently and may happen again in the future I think sort of like is probably beats out all the other questions of sort of lifts and like food and stuff I think I think when prior to the layoffs when those things got taken away people freaked out and were like Like, this is unforgivable.

4:54But then once they're like, oh, wait, actually, you guys might just fire me. So now I'm much more tolerant of you taking away my food away. But I do think that fundamentally, it probably drives for most people, the vast majority of it. And I actually think for us, well, this probably is one of the things that makes, say, MSL more stressful than the other frontier labs. It's the sense that it might have layouts in the sense that I think, you know, so far there have been fewer rounds or at least perceived to be fewer rounds. some of the smaller startups like OpenAI and Anthropik.

5:24John Myles White:I think another thing that is oftentimes used for retention is the promise of growth, or I guess career growth for an employee. And I know you ran promotions in AI Infra for a long time. I was curious if you saw things change over the years. I observed a lot of divisions of meta would really stop talking about, like, well, your comp is why you're here and we pay you good money and you enjoy the work and that's why you stay. And a lot of people were like, no, the only reason you do anything is because there's a clear story about how it's going to get you a promotion. And you would like, especially, and this was really striking when I moved from data infra, whereas one of the places I was in earlier in infra and moving to AI infra, and I was in AI infra for a while before I came into sort of PyTorch part of AI infra and then later MSL.

6:13One of the things that blew my mind is like there was not a single person I would meet on any team in AI Infra whose first and foremost goal wasn't promotion. This was a thing that sort of came up in Data Infra, but was not like 90 % of people's attention. And when I started at Meta, it was just like not ever a thing. Actually, like, you know, I used to manage one of your previous guests, Adrian, and like Adrian and I were talking about career growth back in the day. And one of the things that was really amazing was like we started this org growth that actually had never had anyone above ic7 ever in history um on the sweet side to be clear there were some other roles that had it but like um at least this is what we were told i actually don't know for a fact it was objectively true but we were told this by several people um i think in that world you just you weren't focused on promotion so it wasn't a big thing but then it became a thing where it was like this is the thing like you will one get more money more promoted but also you'll have a title that you can use for your next job and like everything is driven by promotion dynamics um actually i would say the org i sound this by far most strong in was monetization um which is like when monetization would hire people out of a info it would always literally be like here is a very detailed plan of work you will do in order to get promoted and that was a thing that worked very effectively but it also my experience like wrecked tons of teams and i managed a bunch of those scenes now to fix them but they were teams where like basically everyone agreed the thing we were working on was bad no one thought it would succeed but people like oh but i have to ship it because that's my promo bar and you get into the state where it's sort of just impossible to even make good decisions about software anymore because the promotions were so important and then i think what happened as the market became less pro-employee is that people were like, oh, no, no, no.

8:06You should be afraid of being laid off. You should not be worried about the positive chance of a promotion. But I don't know that that was a, I don't think that was a good cultural fix, but I do think it's been a bit of like an attempt to undo some of the damage from before. But I do think that like sort of promotion mindset was incredibly intense everywhere. And ironically, it led to this thing which I found really troubling, which was the thing that would actually let you develop real skills that would do better in your career going forward increasingly got decoupled from the promotions you know and i've seen a lot of people on twitter over the years say this and i find it very compelling which is people like the main thing that's wrong with the engineering cultures with the big tech companies is the promo culture and i really agree uh actually met ironically my perception is seems to be doing better at this than many of the other companies that have even more formal processes and more anonymous parties involved.

8:58But I do think they're like, oh, what really matters is that you ran a rollout that affected 10 other systems. This causes people to not build clean systems. It causes them to build systems that maximally are coupled to the other systems in order to be able to hit the promo bar. And that I think actually I met so many people who didn't even like the work they were doing because they're like, well, this is what will get me promoted. And so do you think that stuff was really unfortunate? And especially I think actually in the AI world has been especially unfortunate because it means like unless someone is clear how using the AI tooling is going to get them promoted, they may not do it.

9:36But it is without doubt in my mind, the only thing that's going to matter for actual professional development and growth and actually being able to do more interesting work in the future.

9:45John Myles White:yeah i definitely saw some some unusual behaviors but we're all trying to play the game because it helps people get promoted and you retain people and all that so yeah i mean unless your vp is ready to fix that and totally shut it down like you have to play the game it's totally irrational not to play the game and i think you know you either if you don't want to be in that we have to leave the org if you're there you've got to play the game you cannot be the one who sort of unilaterally disarms but it is i mean it was it was mind-blowing to me to see this culture i think was unique to both modernization ai infra but it was completely dominant in ai infra and it was really it was not great for any parties and funny is it was like it wasn't even good for the people who were in it like they themselves had gotten to this rat race that seemed to be demoralizing to them you saw this i saw this what what could you do though when i was in it too the people who were talking about it they were not necessarily saying, yeah, this is good.

10:43John Myles White:They were saying, yep, I'm doing this that I don't believe in, but we both know that I need to do this for this reason, so I'm doing it. I think there are sort of two high sources of uncertainty that compete, and I don't quite know how they balance out. But I think one is, I think people have a ton of agency in choosing the culture of the team they want to be on. And if they're in a situation like that and they don't love it, meta has a lot of teams that don't have that culture there are a lot of teams like pytorch was one of them where people were just like genuinely in it for the love of the craft and like people loved engineering as engineering and pytorch in a way that i think was not present to a bunch of other parts of meta i think people wanted to come to pytorch for that reason and i think people came and were happy for that reason so i do think people just have agency in that sense which is like yeah within the context of that machine you got to follow the rules but it's not like you're forced to be in that machine.

11:34There were other roles. Not everyone would get them. I mean, Beto was very choosy, but I think many people could and it was worth doing. I think the alternative there was also was like, even within a team, managers can differ a lot on how much they push on this. And I have personally been a person who I think mostly benefited from actually trying to bet on, be on the stable team that will gradually succeed over time and will have a healthy culture and not collapse and not do things like overlevel ourselves. But it does mean that you get promoted slower. And this is where I think I do struggle with this a bit, which is I think like if you're at the end of the day, what you really want to do is do something like compute your total sum of earnings over your entire lifetime.

12:18It may be better to be in the orgs that get promoted really fast and get fired really fast. But there are a lot of those orgs at meta where people are like, well, they're the stars. Oh, actually, no, it turns out they're terrible and lied to us. So we got rid of all of them. I've had this happen a bunch of times in the year. It might be that that is actually economically rational. I'm not sure. But certainly I think for myself about emotionally rational and feeling like I enjoy the craft of stuff, being on the teams, and PyTorch was like this. I mean, one of the challenges actually PyTorch had with hiring was actually the perception that PyTorch held a higher bar.

12:53People would be like, oh, I've heard you guys are actually much tougher about promotions. And I'd be like, yeah, honestly, we are. like honestly like r8s are like as good as you're going to get anywhere in this whole world and you know if you want to be the best engineer you're ever going to be in your entire life you should work with them but like r8s probably are better than the 10s and a couple of other teams um if you want to be a 10 you maybe should be in that team instead uh and i think for some people that would really turn them off but i think part of this was again the sort of like agency and selection mechanism as i think pytorch selected people who actually just loved engineering as engineering and then there were people who were willing to tolerate slightly fewer promotions and ironically when things i think was interesting though was that it became a bit of like a magic thing that kind of turned out well which is because people perceive pytorch to hold such a high bar it was much easier for us to convince people outside of pytorch that actually our people were ready for eight or nine or ten whereas other teams when they tried to make this argument they're like oh but you you guys are not well known for holding a high bar maybe you guys are just overselling these people but in pintorch people like oh yeah like you you guys hired our guy and thought he was pretty bad so actually we think you probably are credible um and i think that that like again it's like in the short term doesn't actually accumulate as fast but i think in the long term does have a ton of benefits whether it is the absolute like compensation maximizing algorithm, I'm not sure.

14:21And that's where I have a little bit like torn up torn. But for me, I was also like I was willing to get 20 % less compensation to be in a place that I was more proud of.

14:30John Myles White:Actually, that's funny, because I remember someone from PyTorch would join one of our collaborations, for instance, you know, someone would join and go, oh, he's a five, but really, he's like a seven, like he's, he's better than all of our engineers. But we don't know why he's a five but he's a five that was not uncommon working with that org yeah i mean i mean yeah this is like i mean it would it would come out like flat out in like opportunity chats where we tried to hire someone and they're like hey you know pythroach is cool but like aren't you guys like really under leveled that would be like the first question people would ask and have to be like maybe or maybe we're holding the right levels uh you know you gotta decide where you want to take a chance on us but you know but i think the flip side is you know that you know we really did train people.

15:11Like a lot of people who were in PyTorch really learned to be remarkably good.

15:15John Myles White:I know before PyTorch, you worked on some of the data and experimentation tools at Meta. How does that work? And like, how was your experience growing in early Facebook before all this promo craziness? Well, I came into a wild ride, which was actually honestly an amazing experience. And, you know, some of my happiest years of my life from like my first two years in Mena, but also some of the most fearsome and scary years were also there. But I joined the team that I signed up for. And when I signed up, it was supposed to be called Data Science. Before I arrived, because I asked for a six-month leave to work on Julia, the programming language I was one of the core contributors to, I asked for a six-month leave between finishing grad school and going to Facebook at that time.

16:00And during that six-month period, the guy who hired me quit. And then the team that was called data science that I was hired into got split into two teams, one called core data science and one called data science infrastructure. And then that itself became really tricky because I wound up joining core data science, but really loving collaborating with the data science infrastructure people who were the ones who own the experimentation tools. Working in the experimentation tools was like, honestly, like I think to this day, the most, most people who know me from that are like, oh yeah, John was really helpful for that stuff.

16:31I actually think almost nothing I worked on as an IC ever went anywhere close to being as valuable to the business as the experimentation stuff. And I don't think I was ever as good at any of the other stuff. I really loved being in that space. And I think it was really influential to the business. That said, I was on this core data science team that was an insane ball of stress. I joined, it was radically reorged, it had new managers actually wound up really liking the new managers but that was still a source of churn but then a few months into it someone who actually was on the data science infrastructure team which is particularly what's amusing but attributed his team to being core data science uh because he perceived that to be sort of the team he really was on when he wrote this paper published this paper in pnas the proceedings of national academy of science called something like emotional contagion social networks that wound up just becoming like the absolute singular worst piece of PR for meta as a business that year.

17:31Just absolute disaster. I mean, I was like really, really trying to prevent this from happening, but I didn't have the authority to prevent it. But like, you know, at least people perceive I was on the side who was not happy with this decision. But it meant that I was on this team where I was doing the data science infrastructure work that was sort of very inward facing and very safe, but I was affiliated with a more researchy division that was publishing these papers that went from becoming sort of a PR win to a PR nightmare very rapidly. And that team, I think, was one of the most formative experiences at Meta because it really was like, well, what happens if this team just gets fully disbanded?

18:06And this was in a world where there weren't layoffs. It was like, well, Meta doesn't do layoffs, but this team maybe has got itself to a state where it's going to get laid off. And the guy who wrote this paper wound up having to do like a company-wide Q &A where effectively he sort of apologized to the entire business as his Q &A. It was really just like a mind-blowing experience. And it was actually, it was like one where it's like, you know, I came as an IC4 and all of a sudden we were like in these meetings where like, you know, the head of legal being like, well, why did you guys do this? And, you know, having to have these discussions with them on a regular basis and really trying to figure through like, what was the future of our team?

18:43But it was great. That blew over for what it's worth. And then I spent several years just working on our experimentation tools. You know, it's like one of the main developers of Deltoid 3, which is, I think, now just called Deltoid because I think it's been Deltoid for so long that it's referred to as Deltoid. But at one point, it was like the third iteration. And I worked a ton on that. And then especially it was a real example of how career growth can actually happen, which is like I was on the team. And then all of a sudden, almost all the senior engineers left the team in the span of like six months.

19:12And then I went from being like, and at the point, maybe I was already in IC5 when they all left, I'm not sure. But I went from being like one of the people on the team building experimentation tools to the only one who remembered how anything worked left. And suddenly I went from being sort of random IC5 to like de facto TL for a bunch of stuff, which was actually an amazing opportunity for growth. And I think people understate how often these things can happen in tech. But it meant that I wound up like really being like able to drive a bunch of the vision for the A-B testing tools for years, which were hugely successful at Meta.

19:45And it really was some of the most fulfilling work I ever did at Meta.

19:51John Myles White:Just to give people context, Deltoid is the A-B testing framework or the thing that you opt in one code path to A, one code path to B, and you measure all the downstream benefits of ideally your tests, right? Yeah, I mean, I think one of the things that's actually kind of mind-blowing to me is like, you know, actually, it's one of these decisions where maybe I did make bad career decisions, which is like, as far as I can tell, every time I've ever looked at it, the Statsig product, which I now don't know what its state is after they got put OpenAI, is just like, is deltoid. This is one of the things where like, you know, stuff in meta, when you've been in meta, you work on these things like awesome in a data swarm.

20:28But when people are like, what's data swarm? Like, it is literally Airflow. And it's not like metaphorically Airflow. Like the guy who wrote Airflow built Data Swarm, quit. And like a week later open source to airflow. And the Deltoid is like not quite the same because it's not the same people left and built Statsig. But my perception is that for most people who will be watching this, if they've ever seen Statsig, my perception is that like almost the entire UI is the same, almost all the functionality is the same. That's one of these things where I like, you know, probably should have built a Deltoid startup many years earlier and I did not have the wisdom to do

21:00John Myles White:that i think there's another one called optimizely as well yeah so i've i've seen many companies it seems like a very repeatable playbook where you just take something that people take for granted that's state-of-the-art from a big tech company and you just give it to everyone in the industry and it actually creates like a billion dollar company is pretty it's hard but at least the product market fit and idea part are relatively solved since it's creating so much value for these big companies. For this podcast, I produce transcripts for every episode for convenient skimming, and I built a custom tool to automate that.

21:37John Myles White:Recently, I noticed in the Barbara Liskov transcript, my simple speech-to-text tool was getting a lot of things wrong. For instance, the CLU programming language is spelled all caps CLU, not CLU. So to fix this, I used cursor 3, picked the strongest version of Opus 4.7 extra high, and had an agent make a plan to fix that. And while I was waiting, I figured I'd trigger a few more agents for code cleanups and front-end improvements. It generated a reasonable plan with rich system diagrams. It applied all the changes within minutes and worked on the first try. So if you want to build something with the flexibility of sending off a bunch of agents with frontier models of your choice, you can go to cursor.com to try out Cursor 3.

22:22John Myles White:You know, I saw before you worked at Meta, you were working on the Julia programming language, And I actually wasn't familiar about it. So I read into a little bit and it looks like it was part of these data science language wars, basically, where there was R versus Julia versus Python. What is Julia and what is the context on that war there? Yeah, well, certainly I think I made it more of a part of the war. I don't think it had to necessarily be part of it. Although you think also like the simple fact of the reality is like programming languages are products and products exist in an ecosystem where they're in zero sum competition and claiming that they're not in zero sum competition is like a very cute thing that people say is appropriate, but it's clearly false and I think just makes everyone worse by misleading them.

23:08But like, I mean, so Julia, for me, and it's actually sort of why did Julia so appealing? I mean, for me, what Julia's pitch was like, we should be able to write code in a high level language that looks like Python or like MATLAB, which is really the language it was originally designed to destroy. It was really designed to get rid of MATLAB. It was made by MIT math people who wanted to get rid of MATLAB. And it really like targeted that market much more than data science at start. and sort of I think I was involved in pushing it towards data science. But you know, to me, the thing I always do when I give talks about Julia is be like, listen, let's look at the R function for distance, like compute a distance matrix between a bunch of vectors, you're like, you know, pairs of vectors, and you get all the distance matrix.

23:52If you look at like, that function, and then you actually try to figure out how it's implemented in R, what you find is like C code, that is very reasonable C code, that is just a bunch of for loops, like, you know, loop through all the rows and all the columns and then compute the distance at that row and you're done. If you basically take that code verbatim and just translate it naively into R, you're going to take some type information away, you're going to get rid of some like ints and float signatures. But otherwise, you're going to basically write for loops that look exactly the same. The R code is going to be like somewhere between a thousand to ten thousand times slower than C.

24:28And this to me was the thing that just like drove me insane where I'm just like, wait what like these two programs are like 80 the same why is one not as fast and julia really was all about this notion that like that was unacceptable and that's what made it so appealing when the first like the first post by the original founders went out i was like oh you guys are doing the thing i wanted people to do which is like not claim that it is impossible to make high level languages fast which is like so much of actually how the Python R community sometimes behave is to be like oh well we can't be fast but also fast isn't important and to me like that that double hit of like well we can't be it and it's not important really didn't work for me so Julia really resonated I think I probably is the guilty party of trying to make it more part of the day of science wars because I was myself a heavy user of R and was just so disappointed in R just so incredibly disappointed in how often I would try to do a project and R just like fought me at every step of the way.

25:32But Python is also like this. I mean, if you look at all the really great libraries like PyTorch, like, you know, deep down at the end of the day, you're going to look at C++ code or you're maybe even looking at like handwritten assembly or handwritten like kernels for GPUs. Or at least you're looking at something written in a much lower level language. And so Julia was really like about trying to solve that. And I don't think it totally won, which I think is probably why you didn't know about. I think it was very hip at one point and has become less hip, but it's actually doing okay. Like it's, I think it's in the top 25 programming languages by users in the world.