Mozilla Firefox CTO on Browser War Stories and the Path to Distinguished Engineer

24:19And then we would look at it and be like, yeah, this is a real issue. And then we had a bounty program, right, where we would pay them a small amount of money as a thank you and an incentive to bring it to us and not sell it on the black market. But the black market certainly paid more. And sometimes there would be zero-day exploits where you'd have to sort of jump on some new thing and fix it and ship a fix within 24 hours to make sure that users were protected. And sometimes these attacks got ahead of us. And so there was one situation where somebody flagged me that there was a zero day exploit in the wild that had been discovered.

25:03I think it was on like MoscowTimes.com. And I was disassembling it. And then I had this realization that like, I knew who had written this exploit, because it was one of the security researchers that we work with. And without thinking too hard, I fired off email to this guy being like, hey, like, you're double crossing us, right? Like, you're not supposed to sell these things if you report them to us. And he got back to me and the thing about security researchers is that they're extremely paranoid. He's like, no, no, no, I didn't. This must mean that my devices must be compromised at a very deep level.

25:35And the only thing I can think to do here is to go off the grid. And we then discovered basically like an hour later that actually it was Mozilla that had been compromised, that Bugzilla, our bug tracker, had been compromised by a threat actor. And I tried to write back to the researcher, but at that point, he'd already gone offline, and we couldn't reach him. And so it was several weeks later that we got a phone call from a bowling alley in upstate New York, where this researcher had gone into hiding and was trying to, you know, throw off the people that were after him. And we managed to explain that actually, sorry, this was our bad, and had to help him get back on his feet.

26:13But But it was a really big problem that Bugzilla had been compromised because that is where you have the records of all of the interesting security attacks that people have discovered against the browser. But the amazing thing was that Slaughterhouse had just wrapped up and that out of, I think, 45 of the, I don't know, something on the order of 45 of the bugs that the threat actor had exfiltrated, we had fixed all but two of them. And with this sort of new architecture that basically fixed it. And so that really impressed upon me the importance of getting ahead of things on security and making sure that you have the right architecture, because sometimes you find yourself in a situation where you don't have the luxury of the time to fix it.

27:00Bobby Holley:How'd you know that that security researcher wrote that vulnerability? people have very particular styles and in this particular area of security or of exploits which which is different than other kinds um there was just a lot of people's fingerprints on it right you could tell because these were things that were handwritten logic attacks where you are using very particular parts of like i'm going to take this api and then i'm going to bind this function to it and then i'm going to send this call like around async through the event loop twice and then I'm going to do this, right? Like there was a very particular way and these were all handwritten.

27:38There is a different type of security attack that was also a major issue at the time, which were memory safety vulnerabilities. And these were much less of a artisanal thing and more things that you would produce by tools, particularly by fuzzing tools. And so this was also a major issue that Mozilla was grappling with because you would have people who would run these fuzzers against the code base. And the code base was written in C++. And there are lots of ways to have memory hazards in C++. And really any one of them is the sort of like web page takes over your computer situation. And so we were really struggling to find ways to deal with that in a robust manner.

28:23And that was really one of the things that motivated Mozilla's investment in Rust.

28:28Bobby Holley:So I could see, and I'm just trying to understand here, the security issue here is there's some front-end code that abuses or does some special path in Firefox and then can take over Firefox in that case in the past. But how does it take over the computer? Wouldn't the operating system kind of shield, ideally, from Firefox doing crazy stuff? So certainly not at the time. Like originally, particularly on Windows, applications largely ran with system privileges. And if you took over the, you know, the classic thing that security researchers would do to prove that they'd pwned your machine was to launch the calculator, like the Windows calculator, right?

29:09And so you'd run this exploit. You really have to trust the person who sent it to you that it wasn't going to do anything bad, but, you know, calc.exe, that was the proof. As time went on and, you know, Chrome had a major hand in improving the table stakes of how browsers worked, there were additional layers of protection that were put on, right? And so some of these include in-process sandboxing, as well as operating system-level sandboxing. And so the architecture that browsers like Firefox and Chrome have today involves a very, you know, defense in depth, multiple levels of protection, including operating system-level sandboxing that is applied to things like the rendering and content processes.

29:55That said, this was all bolted on over time on Windows. And Windows, unlike a platform like iOS, iOS is designed from the beginning with an application sandboxing model where applications are just applications and they have their own little world. That was not how Windows worked. And so evolving Windows and evolving the security critical applications that sit on top of Windows, like browsers to deal with these sorts of attacks is honestly an ongoing challenge.

30:24Bobby Holley:Did you ever figure out how Bugzilla was compromised? I think it was a credential leak from an employee. Like, I'm 85 % sure that that's what it was. And that was, I think, before we had things like two-factor authentication for logging into Bugzilla. You know, Bugzilla was a very old tool that dated back into the 90s. And so similarly, it was something that had to evolve with the threats over time. One thing that I was curious about is, as an organization, how do you prioritize security? Because, for instance, at the time, there was also these big investments in performance, and I'm sure there was a bunch of other things.

31:00For sure, yeah. Security is one of those long-term things that is very easy to take a short-term perspective on and ignore. And that is perhaps a decision that some products will make. given where they are, but it's not something that you can do as a long-term browser. And so honestly, from my perspective, it's a cultural thing in Mozilla that from the very beginning, back before there was any sort of management structure and performance measurement and company objectives, it was just understood that we had an obligation to protect our users and to keep them safe. And I think that persists. That means that there's a lot of, I would say, ongoing bottom-up energy towards making sure that we improve security.

31:47And there's a lot of pride that people at Mozilla take in the security of Firefox because Firefox does not have, you know, I think that the Chrome security org, you know, on its own is like on the order of, you know, 70 plus people. And we don't have anything close to that, but it is considered to be a shared responsibility. And we have a lot of people who are very dedicated to that and trying to think ahead, right? Like slaughterhouse, the project that I led, that was not any kind of top-down edict of we need to go fix this. It was that I was responsible for the code and I felt that this was important that we needed to get it right.

32:21And one of the things that I'm very proud of at Mozilla today is that there's an annual security competition called Pwn2Own where researchers show up and test their exploits and receive prizes for compromising the browsers. And generally speaking, they always find a way to compromise every browser, right? Like no browser is perfectly secure. And Firefox for the last two years has won the award of being fastest to patch within, you know, on the order of 24 hours. And this is only possible because we have people literally in time zones around the world who are just working nonstop to get it out.

33:03And it's not something that we ask people to do. There's no management edict that somebody needs to stay up late or work on a weekend to do it. People are just motivated to do it because they care.

33:11Bobby Holley:You mentioned no performance reviews and this scrappy bottoms-up culture. How does that kind of culture work? I think in the world of prior to being much management, it was just uneven, I think, as you would guess, right? And I think that it was generally things worked well because the caliber and dedication was very high, but it was very much a bottoms-up culture, right? And if there was somebody who was owning that particular part of the code that wasn't doing a good job, everybody just kind of lived with that. And also, I would say, there was very much a notion of people having ownership of areas and that being somewhat final, right?

33:55Mozilla had a governance structure that was very much insulated against any sort of executive power. It was this module system and people were owners of the modules. And even the initial access control system, I think, was called despot. And it was very much a notion of distributed and decentralized decision making. I think that that culture was very powerful. And there are still elements of that culture that I think are important to preserve today. But there are also ways in which it fell short and needed to change. And one of the things that I've spent a lot of thought on over recent years is finding myself in a leadership position is how do we preserve those aspects of the technical culture that make it great while also addressing some of the shortcomings.

34:42Bobby Holley:You mentioned a little bit about C++ and memory safety issues. That's a very classic thing. And I know that Rust was developed at Mozilla to tackle those problems in large parts. So I'm curious, can you tell me a little bit about the adoption of Rust and how it solves the problem? Right. So in the early 2010s, it was pretty clear that Mozilla and Firefox, the code, had some real structural issues. And a lot of these were issues that we could address through sheer hard work, right? This included building a multi-process architecture and evolving our extension architecture such that stuff, you know, didn't have its tendrils into all the undocumented interfaces within the browser and building native video playback so that we didn't have to rely on Adobe Flash anymore.

35:39Like these were all large but incremental projects to address specific architectural issues. But there were two architectural issues that were very difficult to address in that way. And the first one I already mentioned is memory safety, right? Because when you have a code base that's written in C++, a large portion of the defects that you can make are the sort of defects that compromise the browser and in the absence of sandboxing compromise the computer. And that was just a very difficult thing to deal with in a non-whack-a-moley fashion. and there's another piece which was that particularly at the time there was a increase in the number of cores that were shipping on devices but browsers were still largely single-threaded and this meant that you were leaving a lot of the available performance on the table if you didn't have multi-threaded code but if you did have multi-threaded code especially in c++ you just had all sorts of very difficult non-deterministic bugs that were very difficult to track down.

36:41And some of these also became memory safety issues that became exploitable defects. And there was this notion that both of these were rooted in deficiencies of the C++ programming language. And all browsers were written in C++. But, you know, in 2010, Firefox had just taken on the world and opened up the web. And there was a sense that we could do the impossible. And so we began an effort, you know, sponsoring some research that some people internally had around building a new programming language. And this programming language was Rust. And the idea of Rust eventually became that we would have a programming language that had static guarantees at compile time.

37:22That number one, you didn't have memory safety issues. And number two, that you didn't have race conditions. and this idea was that first we would build a new programming language with these properties which was difficult to do and then we would use this programming language to build a new browser engine for firefox and so the rust project co-evolved with this project called servo which was a ongoing evolving test bed for the rust programming language with the intention of building a browser that was memory safe, but more importantly, was parallel. Because there was a sense that we could solve these architectural issues and close the gap with Chrome, but that also we needed something to leapfrog Chrome.

38:08And there was a belief that concurrency and safe concurrency powered by a new programming language was the way to do that.

38:15Bobby Holley:Yeah, I want to talk about the competition with Chrome at the time, because you said they launched with a more performant browser and also they were hammering you guys with distribution advantages. I'm curious, what was the strategy at that time from the Mozilla team to compete with Chrome? So the strategy to compete with Chrome was multifaceted. One of the parts was a recognition. I think there was a little bit of denial in the very early days about the deep value that some of Chrome's architectural advantages would provide. And this was, you know, they came out with a JavaScript engine with what's referred to as a method JIT.

38:58We had a tracing JIT. We thought the tracing JIT was good and was going to be better. But in the end, it wasn't. Chrome came out with multi-process architecture. I remember the first thing that I heard was, yeah, good luck doing that on Mac. And then they figured out how to do it on Mac. And there was a belief that some of these things were not possible until they were. And then once they were, that they weren't the most important thing. But eventually it became clear that things like a multi-process architecture and things like native video playback were really essential. And we had this issue, particularly on video playback, where because Chrome first had a better plugin architecture for Adobe Flash, and then was also early to introduce native video playback, they were able to have a much more stable experience on YouTube, which was blowing up at the time.

39:48Whereas Firefox on YouTube, which was what everybody wanted to do on the internet, was just crashing left and right. And it wasn't stuff that we could fix. It was Adobe's fault. And we really couldn't do anything about it. And so we had to work on a lot of these really hard but discrete challenges of closing the gap with Chrome. But then at the same time, there was a desire of you can't just close the gap. You have to do something new and you have to do something that is better. And Rust and Servo were our strategy for doing better.

40:24Bobby Holley:So you mentioned the strategy with Servo and the rewrite in Rust. How did it go? Was it successful? Well, we know that Google's distribution eventually dominated, but how did the efforts with Servo go? So on the one hand, Servo and Rust had, against all odds, captured something really real. And Servo, as a browser engine prototype, had a lot of really amazing stuff in it, right? It was not complete, but it did have the skeleton of certain features that had never been done before, including a multi-threaded CSS engine. But the problem was that we had a team of, you know, a dozen people working on Rust and Servo.

41:11Google had hundreds and hundreds and hundreds of engineers who were working on Chromium. We had less than that, but still most of Mozilla's efforts were directed towards Gecko and Firefox because that was the product that we had in market. So we had this situation where in order to build a competitive browser engine, it's not like you need to just do this one time it's the accumulation of all of the work over all of these years of hundreds of engineers working on improving it and improving the architecture improving the performance improving the features and so every year there was more and more stuff that server would have to catch up with and i think if you just look at it there's it was very difficult to believe that we would ever end up in a situation where servo would be competitive with a regular production browser engine.

42:03And so circa like 2014, 2015, that was a situation where there was really a sense that we needed to deliver a fully refreshed version of Firefox that really got us back in the game. It included this parallel CSS engine that we uplifted from servo and this was a project that i personally led and was really the highlight of my career working on it because it was so exciting to be doing something really new in a new programming language that and every step of the way there was this notion that like that's definitely never going to work right like how are you even going to like hook this thing up and then we hooked it up and we had it traversing the dom over c++ um you know we have a c++ dom and we're having this rust CSS engine and how you get these two things to work together.

42:52There was no tooling for it. We had to build it all ourselves. At the end of the day, it was really a dramatic performance improvement. I think it improved rendering time on Amazon.com by on the order of 25%, which is just enormous in terms of web performance. And because it was parallel and it can run on all of your cores, it was and still is the fastest CSS engine in the market. Firefox really got back in the game and was really performance competitive with everything else out there. And that was a really exciting moment for everybody, realizing that if we just coordinate and we do all the technical effort and we make the right bets, we can win.

43:32Bobby Holley:Okay, so you mentioned that this was one of the most proud moments of your career and like a major launch, very exciting project. I'm kind of curious to talk about the career parts of it. So where were you in your career at this point? And maybe you can talk about how it grew as you worked on these projects? Sure. So I think when I started, I don't know if Mozilla had levels. I certainly wasn't aware of them, that they existed. It was a culture of choose your own title. The title that I chose for myself was Negative Entropy. That was sort of what I wanted to do on this incredibly complex and gnarly code base.

44:09As a fun fact, my favorite intern made his title Entropy, but we were really on the same side. I think by the time we were shipping Firefox Quantum, Mozilla had developed, I would say, a more industry standard career progression. And so, you know, you had, you come in as, you know, just title of engineer, and then you've got senior staff, senior staff. At the time when I was working on Quantum CSS, I believe I was I had reached the level of senior staff. And upon the successful launch of Firefox Quantum, I and the engineer who led the other half of Quantum Flow were promoted to the level of principal engineer.

44:58And at the time, there were not very many principal engineers, like probably fewer than five. And so that was a pretty significant jump. But my long-term dream was to be like Boris Zabarsky. And Boris Zabarsky was one of Mozilla's three distinguished engineers. And so that was really what I had my sights on in terms of the level of impact that I wanted to have. And the way that I really aimed at having this impact was by trying to contribute to as many parts of the code as possible and just be prolific. And so after working on quantum CSS, I went and worked on web render and I worked on mobile and multi-process and layout and all sorts of different parts of the code.

45:52And I remember my manager at the time, Joe Hildebrand, who ran engineering, told me that he wanted me to spend one day a week when I didn't open my terminal. And I was just shocked. I was like, what are you asking me to do here, right? You're asking me to not work for an entire day. And he was encouraging me to start thinking about it in the impact that I could have in a much broader way, as opposed to direct contributions to the code. And so this was something that I started to think about, but it was really not the natural way that I thought to work.

46:27Bobby Holley:When you talk about the individual contributions, because you mentioned that was your way of having a lot of impact, do you have a way of thinking about which contributions are more impactful than others? I was very much motivated to work on whatever felt like the biggest problem. And so that was early on, that felt like the DOM bindings and XP Connect and all the security stuff. And then at some point, it was the media playback to solve the YouTube problem. And then it was multi-process. And then it was this Rust and Servo CSS stuff. And after that, I thought that WebRender, which was the new graphics backend that we were uplifting from servo so i went and worked on that then mobile so i was always drawn i think to what felt like the most pressing problem and my model of how to have impact on that was to jump in and learn all of the code in the most pressing problem and help work on it and not just necessarily typing all the code myself but working with the other leaders in those areas on figuring out what the right things to do would be and reviewing code and setting direction and setting priorities.

47:38So there were aspects of leadership, but it was all very much oriented around the code that needed the most help.

47:44Bobby Holley:So you were prolific, but it was a guided search specifically on the biggest problems for the organization. That was how I thought about it. Yes. One thing that I'm curious about, because in the industry, when it comes to career growth, I often hear two methodologies. One is where people, they focus on the promo first and foremost. And as a byproduct to get the promo, they, you know, grow and level our skills and various other things, behaviors. And in the other school of thought, it's ignore the promos, but focus on the behaviors, focus on the skills, and then the promos will come. So I'm curious your thought between the two methodologies and which one you think is better for structuring an engineering career.

48:33I have pretty strong opinions about this. I am very much of the mind that it is important to focus on impact and not the title. And I think there are a number of reasons for this. But that is certainly what I look for now that I'm in a situation when I am helping people with their careers. And I am often in the decision of, you know, figuring out, you know, who's the right person to advance. And what I am looking for is I am looking for the impact. And my general philosophy is that if you are having that level of impact, you don't need to sell yourself in that way. or sell yourself is not quite the right word, but there is a particular philosophy where you want to promo hack, right?

49:21You want to pin your management chain down on what are the things that I need to do to receive this promotion. And that just speaking, frankly, is very annoying, right? There are aspects to this that I think are just intuitive to how I, at least as a person, perceive things, you know, like I'm a father of kids. And when my daughter asks me, like, when, you know, I ask her to do something and she's like, you know, will you give me this if I do that? Right. It's it just does not land well with me. But I think that there's also a substantive aspect to it, which is that if you are really focused on the impact, that is how you are really going to achieve things.

50:03Right. Whereas if you're focused on checking some boxes that somebody has presented to you, then you're in this awkward situation where it's like, well, I did technically say that if you did this, you know, that would maybe make a case for this next level. But the way in which you did it and was very sort of focused on checking the box as opposed to having the impact. And at the end of the day, impact is what matters and impact is what needs to be recognized and rewarded.

50:27Bobby Holley:One thing that I hear often, because I think this is a failure mode for engineers that are a little more soft-spoken is they're great at writing code and landing it and having impact, but they're a little weaker on the, you know, selling yourself or kind of marketing your work. And so that kind of bites people. And so sometimes I hear career advice that is focused on, you know, hey, make sure you create a brag document and you share the wins when you land the thing. And, and I think a lot of those people don't have an ulterior motive of, let me, you know, kind of connive my way into the promo.

51:10Bobby Holley:Um, but I think there is this, this real thing of promos are decided by people and there's some value and I guess a balance. Cause obviously if you were completely conniving, you say, Hey, I'm going to talk to my manager and sell my work to him, that's kind of not ideal. But making posts or sharing it and thinking about the levels and stuff seems to be a good way for those little quieter or less proactive engineers in structuring their careers. So how would you kind of balance those two frames of thinking? I think there are often multiple ways to frame similar activities, but that the framing of how you think about it is going to have a big impact on how you carry it out and therefore what it actually achieves.

52:03And so I think a good example of this is this notion of that you brought up selling yourself or brag documents or something like that. I think it is important that people have visibility and visibility around the work that they're doing. And this is just a very, like a basic fact about impact. Because if you want to have impact, you need to be connected to people who understand what are the biggest challenges, and you need to be coordinated. And you also need to make sure that people understand what you are capable of, such that they can put you in a position of having more impact. But if you think of it as I am trying to brag more about me versus I am trying to communicate and coordinate and create visibility for work that I think that is important, you will have very different outcomes.

53:02Bobby Holley:Yeah, it's interesting because it's subtle. I feel like you could have two people with different incentives. One goes into it saying, I'm going to create this update so I get visibility. and the other person comes in to say, I'm going to create this update so the right people know the right things. And they get to the same path at the end of the day, but it's like this subtle... It's worded slightly differently and people can tell. A lot of it is the degree to which, you know, how much you make it about you versus how much you make it about other people and the work and the team. And everybody knows, you know, that the person sending the update likely had a significant hand in the work.

53:44And a very important thing to do as part of those updates is to make sure that you give visibility to other people and to thank the team and not make it so much about you. Because also that is a way to work well with people, to have people feel like you are a, you know, humble leader and somebody that they want to work with. And that is an important part of leadership, is making sure that you create recognition for other people's contributions and don't make it all about you.

54:19Bobby Holley:It is an ideal world that people have impact and they get recognized exactly as much as their impact. One thing that I can imagine happening, though, is, you know, there's that idiom that the squeaky wheel gets the grease. basically if you had a loud person that is really vying for their promo and you know making all aggressively trying to make the posts and you know constantly bothering their manager hey i want to get promoted that oftentimes assuming they have impact if they don't have impact then it's not gonna nothing's gonna happen but assuming if they're also a high performer and they're doing that compared to the high performer that's, you know, reasonable, quiet, and solid, not selfish, that actually that loud person, unfortunately, I think in most organizations will get rewarded first or faster.

55:15Bobby Holley:What do you think about that? I think it does depend on the organization and it depends on the technical culture. You know, one of the things that we try really hard to do in our culture is to not reward that. But I think there is a balance to be struck because my message is not don't be ambitious, but rather think about your ambition in terms of the impact that you want to have as opposed to the status you want to achieve. And if you have the impact and create visibility around the impact, the rest will follow. And yes, there are edge cases, And there are probably some situations where people lobbying for themselves will kind of grudgingly get themselves a little more consideration just because it's top of mind.

56:01But the best kinds of conversations that I like to have with people are not conversations where people come and ask, how do I get promoted? It is how do I grow my career and have more impact? Right. Like, am I working on the most important thing? Like, what could I do to grow? What could I do to have more impact? Right. And I think that those are important conversations to have with your leadership. to make sure that you're focused on the right things. Because the worst and saddest thing is when somebody is just working so hard at something that is just not that important and not gonna have the type of impact that they ultimately wanna do.

56:34And they might do a great job at it, but it's just not the right thing. And so I do think that some coordination of like, this is the type of impact that I wanna have and not everybody wants to have that kind of impact and making it known that you wanna have that impact and you're looking for swings at the bat and you're looking for opportunities to do more. I think that that is helpful. but trying to pin people down of like what are the boxes that I can check and you know I did this like are you good to promote me now is just honestly extremely annoying from my perspective and I just think zooming out at a level of life you don't want the people in leadership to find

57:12Bobby Holley:you to be annoying so in your case you were focusing on impact you had these role models of these DEs that you wanted to be like. And I'm kind of curious, what eventually did the growth look like and how did your impact grow? I was really focused on continuing to work on what was the hardest and gnarliest challenge in the browser and moving on from team to team. And that was really the model that I had in my head. And I did have in my head this idea that I eventually wanted to have the level of impact that Boris had. And I assumed that that would eventually mean becoming a distinguished engineer.

57:49And then at some point, the CTO at the time, Eric Riscorla, sat me down because at this point, you know, Boris and David and this guy, Robert O 'Callaghan, had been distinguished engineers kind of since the beginning, you know, since maybe like 2010

58:04Bobby Holley:or something. And we hadn't made any new ones since. And the CTO sat me down and he said, look, I want to let you know that for the first time in almost a decade, we're going to make some new distinguished engineers. And I thought you should know that you're not going to be one of them. And this was, I think, a little bit surprising to me because I do think that, you know, I thought and that I think anybody at the time would say that at that time, like probably maybe not quite Boris level, but that in terms of knowledge of all of the Firefox code, that was something that I had more than almost anyone else.

58:41But what he explained to me was that that was a more traditional model of distinguished engineer inside of Mozilla, but that he was looking for something different now and that he was looking for something that was more oriented around industry impact. And that was very much how he saw the world.

59:04Bobby Holley:The definition of impact seems to have shifted while you were thinking about your DE journey. Did you ever debate the definition of impact or what was your thinking when you saw that the criteria was updated? Honestly, it was just kind of inspiring for me. I think there was, you know, some people were concerned that I would be upset, but I think it was more like, you know, you work out and you think that you're really strong and you kind of go do some Pilates move or something. You're like, oh, my gosh, like this muscle that I have, I didn't even know that I have, but it is extremely weak. And the truth is, is that there are lots of different ways to define DE and it's what matters for an organization.

59:40The thing that mattered early on in Mozilla and what was the make or break for Firefox's success was just the management of this incredibly complex and sprawling code base by very few people. And the thing that was needed most was a kind of prolific contributor and enabler of that. And that was the archetype of Distinguished Engineer at Mozilla. There are other companies where Distinguished Engineer means something else. I know that some companies, generally it means you built a billion dollar business, but that is not what Mozilla is about. And what Mozilla is about is changing the world. And so the criteria for DE are really focused around that.

1:00:22and this was I think an inspiring thing for me and I still think that I you know didn't ultimately you know I still probably have not had the level of industry impact of some of our other distinguished engineers and so probably in the end when I got to DE it was maybe a combination or a hybrid of the old and the new definitions but understanding that model of thinking about the world and that model of impact, I think was very profound for me and really oriented how I directed my efforts.

1:00:55Bobby Holley:I'd love to hear some examples of what a DE level of impact looks like. Can you talk about some of the people who have done really impressive engineering work? Sure. Yeah. So at the time there were, you know, when we had this conversation, there were three new distinguished engineers that were promoted. So the first one was Martin Thompson, and Martin still is at Mozilla today. And Martin is an international standards extraordinaire in terms of the development of the technical standards that power the internet. And so his name is, you know in the author line of protocols like http2 and quick and um web push and he was heavily involved in the creation of web rtc and geolocation he authored other protocols like oblivious http like i don't know i'm honestly not super familiar but it wouldn't surprise me if there's really nobody else who has had that level of prolific impact on all of the core technical standards of the internet.

1:02:07And this isn't just about, you know, doing some editorial work on making sure that all of the drafts are formatted correctly. This is about core design challenges of making these protocols better in a way that advances Mozilla's objectives. And so similar to the situation around TLS, there were generally problems around the privacy and security and performance of the legacy HTTP protocols. And by designing new ones that made it faster, while at the same time, making it more private and having less metadata leaked, and having fewer security issues, meant that there was a business incentive for the entire industry to move on to these protocols, because they were faster.

1:02:53And by designing them that way, he managed to really increase the privacy and security of the internet. So that was one archetype. The second was Luke Wagner. Luke was many people. Luke was probably, if there was anyone in the world, the most single-handedly responsible for the development of WebAssembly. So historically, people wanted to run code in browsers that was not written in JavaScript. maybe you wanted to run AutoCAD and they didn't want to rewrite the entire code base in JS. And the original way of doing that was through plugins where there was this basically native binary blob that got shunted into the browser window and it was awful had lots of terrible security properties and bad user experience.

1:03:45And so we tried to move away from that and the first idea was actually from there was an engineer who worked on a side project alone sky who made this tool at Mozilla called Inscripten. And the idea of Inscripten was we just take the C++ and then we compile it to some very naive JavaScript and then run the JavaScript, which technically worked, but it was extremely slow. And Luke was one of the lead engineers of the Spider Monkey JavaScript engine in Firefox at the time. And he wanted to make this something that was actually viable. So the first thing they did was there was some back and forth of, well, how do we optimize the JavaScript engine for what mScripten is emitting.

1:04:25And that was obviously not a sustainable approach. But then they had this idea of ASM.js. And the idea was that mScripten would output a subset of JS, which would be detected by the JavaScript engine. And the JS engine would then verify that it was from this restricted subset, which was sort of a proto bytecode format, and then run it in a more natively unoptimized way. But, and so this was what got the flywheel going of people actually being able to say, hey, we can deploy these amazing native experiences that are compiled and that they're running in the browser. Because the ultimate goal was that we wanted basically everything on the desktop to be running in the browser, or most things on the desktop to be running in the browser.

1:05:10And there was some stuff that just couldn't. And that flywheel then created enough industry momentum to say, why don't we actually create a new bytecode format, which was WebAssembly, where you could have code that was native, but that was also sandboxed and portable and running inside the browser. And this was at a time when Google was pushing a very different technology that was more akin to plugins. They had this thing called Native Client or NACL, and then they had this thing called Pinnacle. And there were a lot of very complicated industry dynamics of getting all of the browsers to get on board with this WebAssembly train.

1:05:49And so Luke managed the technical pieces and the political pieces to eventually forge a browser consensus on WebAssembly, which has been a very transformative technology in terms of what browsers can do and even what can be done outside of browsers, because WebAssembly was built with such great technical properties. It's now being used all over the place on the back end, things like serverless. So this is a bunch of the work that the Bytecode Alliance is involved in. So that is another archetype of impact of the creation of WebAssembly. The third distinguished engineer at the time was Tim Terabury, who was our codec lead.

1:06:26And so one major problem that Mozilla and honestly the web had back in the early days was that all of the video playback happened through proprietary plugins like Adobe Flash. And you would think that the obvious answer, which is the obvious answer, is to build native video playback into the browser. But the problem was that the formats that are used to compress video are heavily patent encumbered. And there are lots of reasons for this, but it's just the way the world works, where in this particular space, everybody is fighting over patents. And this meant that an open source browser like Mozilla Firefox couldn't ship native support for the H.264 codec, which was really what was powering most of the video on the internet at the time.

1:07:21It was the most efficient codec, and that was what YouTube was using. And this was a pretty significant problem for us. And so Mozilla worked over a number of years to develop truly open and free and non-patent encumbered and royalty free codecs. And the team led by Tim first developed an audio codec called Opus, and then they worked to develop a video codec called Dala. And Dala was then incorporated into AV1, which became an industry standard video format that is now developed by the Alliance for Open Media, which is a consortium across many companies like Mozilla and Google and Apple and Netflix.

1:08:09And so this is basically what powers video today. And a lot of that vision around what would it mean to actually develop truly patent encumbered video codecs came from So those, I think, are three archetypes of the types of world-changing impacts that you can have as an engineering IC at a place like Muzo.

1:08:29Bobby Holley:yeah the the achievement of these engineers is absurd i mean you mentioned web assembly i mean i think of a business like figma as possible because of a lot of that work um and that's a tens of billions of dollars industry or i mean a company just by its public market cap And so the value that's being created by these distinguished engineers is absurd. What's interesting to me is, I guess because it's a more open source company, the value capture is very different. I'm curious, is that something that people at Mozilla are ever not feeling great about? Because they've created so much value, but the capture is happening by these capitalist other companies.

1:09:25Yeah, I think from Mozilla's perspective, we don't have shareholders, right? We're not trying to make gobs and gobs of money. The Mozilla mission and what we are here to do at the end of the day is to create a better Internet and one that works for people and it's open, accessible for all. Right. And sure, it is very convenient for big tech companies to have a royalty-free codec for video streaming so they don't have to pay anybody. But they totally could pay, right? What Mozilla's angle here is to make these formats and make these changes in a way that really empowers everybody. And obviously, that creates economic value for everybody.

1:10:06And some people capture that more than others. I think from Mozilla's perspective, we want to be able to continue to exist. and that does mean that we have to think about revenue. And at the moment, a lot of Mozilla's revenue comes from one of the dominant advertising players on the internet, which is Google. And so we would like to change that, but it's hard. And it's particularly hard because it's not necessarily where our heart is, right? Our heart is around having impact on the internet and making that better. And so it is a tension that I think does exist within Mozilla of how much to focus on one and how much to focus on the other.

1:10:42But I think that Mozilla at its best, we have some necessary business stuff that we have to manage. But the end goal of what Mozilla is about is having this impact that benefits everyone.

1:10:54Bobby Holley:So then after these examples of distinguished engineers, I'm curious what advice you would give to someone who wants to become a distinguished engineer? In terms of one thing that I think that basically everybody can work on at whatever level they are is writing. This is a particular hobby horse for me because I think that this was an area where Mozilla and myself included did not have a lot of developed capability. And I think it is one of the most powerful things. we have this internal memo, which is actually, you know, public on the internet, directed at Mozilla engineers called writing things down.

1:11:36And one of the things that this memo does is it makes the case that writing is both about the artifact that you produce, which is useful for coordination and wide review and getting feedback, but also useful in the process itself. And that is because the process of writing is the process of thinking and refining your ideas and articulating them. And this is just an infinite game that you can always get better at. And I think it is one of the most powerful ways to create clarity of thinking for yourself, because clarity of thinking is, in my view, one of the most important things. And you only can really get that by forcing yourself to write down your ideas.

1:12:18And so I suppose one, you know, pertinent or timely piece of advice is if you care about this, do not use an LLM to produce text for YouTube. Because first of all, if you're putting your name on this thing, at least today, the LLM is like, it's pretty easy to tell when somebody does that. But more importantly, you are losing the opportunity to do the thinking yourself. And that thinking that happens when you write is one of the most important types of thinking for growth.

1:12:49Bobby Holley:Do you have tips on how to write well? Yes. I would say one of the most important things is that it doesn't need to be any longer than what you have to say. I think I always prefer shorter documents. And I think there's some notion where people run out of things to say. And at that point, you either need to refine your ideas more, or you're just done. And it's never useful to anybody to add fluff. And dealing with documents that are fluffy and overly overwrought is one of my least favorite things. so I think that's one piece related to that I think that it's important to focus on substance rather than rhetoric which is sort of related to fluff but I think that some people say oh well I'm not a native English speaker and therefore I'm never going to be able to write as beautifully as others and I think that's the wrong way to think about it like there is some basic baseline that you need to work on in terms of articulation of the ideas.

1:14:06But it is not about using the most flowery words. It is about clarity of thought. And clarity of thought, once you have it, is relatively easy to express in written form if you practice.

1:14:17Bobby Holley:Yeah, I think every single person I've ever asked this question to has always said, be concise. It's shocking to me how unanimous that advice is. Okay, so going back to the browser Wars, I think we all know how things have played out. And I see that there's this new era of browser wars that things are heating up again. There's a lot of competition for an AI browser or bringing AI into existing browsers. And I'm kind of curious for your take on what the future of the browser wars is going to look like as the CTO of Firefox. So I think one thing that's really worth clearing up is that you hear that there's, you know, every month you hear about a new browser from this AI company or that.

1:15:04And what I think is really important to understand is that these are not browsers as I think of browsers. These are different front ends, different window dressing and monetization keys on top of Chrome. Because much of what powers a browser is the browser engine. And browser engines are also extremely complicated and difficult to build. And there have been fewer and fewer browser engines as time goes on. And today, Firefox is the only independent browser that has no dependency on a browser engine from somebody else. So you've got Safari, which is pretty vertically integrated with WebKit. Nobody else really uses WebKit directly.

1:15:51Then you've got Firefox, where we build Firefox with Gecko underneath it. And then you've got Chrome on top of Chromium and all these other browsers like Opera and Perplexity and even Edge, they all use Chromium under the hood. Because what they're seeking to do is not necessarily to evolve the web platform and the web experience. they are trying to control the touchpoint with the consumer in this world that is coming with AI. Because on desktop, the way consumers experience the internet is through their browser, and everybody is jockeying over what this experience is going to look like, and people want to control that experience.

1:16:32So I am very much of the opinion that AI is coming to consumer technology like water running downhill. And that this is not something that is going to be stopped or reversed because there is so much disruptive potential of the technology. But where that water goes and how it ends up, it can end up over here in somebody's reservoir, or it can end up in the ocean, or it can end up as a river. There are lots of opportunities now to steer how it plays out. And there is also a lot of desire for people to capture it for themselves. And Chrome, as an example, is making a very aggressive play to control the AI stack by leveraging its dominance in Chrome.

1:17:21And they're doing this at multiple levels. They're obviously doing it as part of the Chrome browser, where there's 5 ,000 different integrations with Gemini as part of Chrome now, but they're also doing it as part of the web platform. They are advancing some proprietary APIs that we do not support to allow websites to have local, to access to an LLM inside of the browser. And this is attractive to web authors because it allows you to offload your compute and you don't have to pay for the inference and you can just ask the browser to do the inference. But the thing that is different about this from other web standards is that there is no way to specify an LLM or to have it have deterministic behavior that is interoperable.

1:18:07And so the API calls that you have into this LLM are fundamentally unspecified. But as the owner of the world's dominant web browser, they have a lot of market power to get websites to tune their usage of the LLM to what Gemini expects. And there is obviously a co-evolution of AI experiences and the models underneath them. And creating a world where more and more AI experiences are built on top of Gemini is part of Google's strategy to achieve and dominate control of the space. And we at Mozilla are not building our own foundation model. We don't have any particular horse in the race. But what we are trying to do is to make sure that the space remains open in the way that the web has been open and that users have choice of how to integrate with these things and which ones they use.

1:19:05And also that the web remains important and relevant because there is a big risk that the web becomes a backend detail that's used by RAG from your chat GPT query or that's used by operator on the backend, where it is no longer the focal point of the user's experience. But if this were to happen, there would be much that would be lost because the web is a very unique system in the sense that it provides users agency and control that they would not necessarily achieve elsewhere. Right. It is not a direct experience that is produced and delivered in a way that can't be modified. It is reinterpretable content that the browser can modify and render as the user chooses.

1:19:51And there are lots of ways that this matters in practice. You know, things like ad blockers are a great example of how the reinterpretable nature of the web allows users to control their experiences in important ways. and that the web is a safe environment where it's safe to open a link even from an untrusted source and that all of this is not created and controlled by any one company but that it is built as an interoperable system where anybody who produces code that implements the right protocols can show up and do part of the system and this leads to a multi-stakeholder environment where no single entity controls the evolution of the web platform.

1:20:32And that is not how most platforms happen, right? It's not how the platforms happened before the web, and it's not how the native mobile platforms happened after the web. It is, in some sense, a very historical accident that we got something that is this well aligned with the vision that Mozilla has for how the internet should work. And it's very important to us to make sure that the changes that happen with AI do not sacrifice these properties.

1:20:58Bobby Holley:You mentioned all the components of the browser and there's the web engine and a lot of those AI browsers and also other ones like Opera, etc. They're actually built on Chromium at the end. I'm curious, what do you think would happen if Firefox was built on Chromium? What is the advantage of Firefox building its own web engine? So it's a lot of work and we like doing hard things. But the strategic advantage from the perspective of Mozilla's mission is that it gives Mozilla a seat at the table when web standards and the future of the web is decided. And it used to be the case that standards bodies were wide groups of many different vendors.

1:21:44You know, you had Opera back in there with its own engine. You had Internet Explorer. You had Safari. You had Mozilla. And as time has gone on, engines have dropped off the map. because if you look at it from a bottom line perspective, there's basically no justification for doing it. It is a giant cost center from a business perspective. But from a mission perspective, the benefits of being part of the conversations that shape the web are profound. And that is why Mozilla is the last independent player that is doing it because we are not driven by that bottom line. We're driven by the impact. And so imagine if there was a world where there was no Firefox at the standards table.

1:22:24You would have Apple running WebKit and you would have Google in charge of everything else. And Apple and Google, very capable and innovative companies, do not get along particularly well and are very entrenched in their own interests. And in a bilateral environment, as you can even see in the mobile space, there is a sense of entrenchment that can prevent progress. And one unique thing that Mozilla brings to the table, even though we have relatively small market share compared to these other browsers, is that we bring a fresh perspective and the ability to tilt the conversation back towards what is best for the user and what is best for the web.

1:23:12one example that i'm particularly proud of is not necessarily one of web standards discussions per se these happen all the time in working groups all over the you know editor of the html specification is a mozilla employee who works with everybody and continuing to refine html but the example that i would go to is actually one about performance. So it's long been the case that competition is what drives the advancement of browsers getting faster. And it is just this amazing achievement that browsers continually get faster, right? It's like the semiconductor supply chain. It's something that people don't even think about.

1:23:56And just like every year, the stuff underneath that is powering all the experiences getting faster and faster. And the thing, if you look at the history, the thing that makes browsers get faster is competition from other browsers. And the way this has historically been measured is that different browsers would produce a benchmark that they would design in a way that was, you know, particularly well suited for their engine, and then they would optimize it. And then other browsers would say, well, yes, that's your private benchmark. And there was an initiative a couple years ago around this benchmark called Speedometer three to try and produce a shared benchmark that we could all agree on that would be representative not of just what worked well in any particular browser, but would be representative of the web and the experiences that we wanted to get faster.

1:24:45And this is a type of engagement where if you just have the bilateral relationship between Google and Apple, it would predictably go nowhere because there is just too, there are too many opportunities for somebody to tilt it in their own interests in a way that the other side says, aha, I see what you're doing there. But I think that Mozilla's involvement was really critical in keeping everybody focused on the stated mission of the operation, which was not to make any particular browser look good, but to provide us a shared incentive to make browser engines faster together in order to make the web faster.

1:25:22And in Firefox, the impact has been pretty incredible of what has been driven by this benchmark over the last couple years. I think that really across all of the browsers, at least Firefox and Chrome, both Firefox and Chrome have improved their score on the benchmark by a factor of two. So, they've gotten twice as fast. And we have actually measured in Firefox the impact that this is translated to on real-world performance on the same machines in terms of page load. and time to first paint and all these things. And it's on the order of 30%. So I think it is reasonable to say that having this benchmark has made the web 30 % faster.

1:26:21And that is something that benefits everyone. And I don't think it's something that would have happened without Mozilla in the room.

1:26:27Bobby Holley:Coming to the end of the interview, I just want to ask you a few questions kind of reflecting over your career because I feel like you have a lot of experience. When you look back on your career and you made a lot of contributions, You were an IC through and through for almost all of your time at Mozilla. I'm curious, is there any technical decision that in hindsight you wish you'd change? One thing that comes to mind is a situation where I think the outcome, the technical outcome was good, but I think in some sense the process wasn't worth it. And this was around WebRender, which was the graphics backend in Servo.

1:27:06When I was working on Quantum CSS to take the Servo CSS engine and put it in Firefox, I was very excited about the momentum of uplifting other parts of the rendering pipeline of Servo into Firefox. And I advocated very heavily that the graphics team should replace its rendering and rasterization pipeline with the one from Servo. and this was not based on technical expertise on my part around it it was very much a vibes thing of this is where it's going and that vibes thing was infectious and i was not necessarily strictly in a decision making point at this time but there were i think that point of view really caught on with management and i remember there was uh the the basic issue was that the team many engineers on the team had very strong doubts about this.

1:28:02They thought that it was not ready, that it was too much, that it was a very risky project. And they tried to express some of these concerns. And there was an engineering leader at the time who heard them and responded with, C++ is not the future. And that vibes-based enthusiasm for a risky technical direction over the concerns of the people who knew it the best ended up being very toxic. And a number of our most senior and most impactful graphics engineers ended up leaving. Now, the rest of the team ended up rallying and WebRender did turn out to be a major and difficult and expensive and many year project.

1:28:48The actual technical result is awesome. It is a fantastic engine and it is really performant and safe and does all these great things. And I'm really proud of the team for what they built. But I think that pushing it on the team when the team wasn't ready and wasn't aligned on it had collateral damage that I regret.

1:29:11Bobby Holley:Is there a top book that made an impact on your career? I would consider myself to be an avid reader. But the truth is, I don't read a lot of books. I do read some. but honestly you know the books that I can think of that probably have had the most profound impact on my thinking are not books that I've actually read and that's just a personal style of mine I tend to be very wide in terms of what I consume and reading a whole book on a topic is a depth first approach and I certainly wouldn't want anybody to take this as encouragement not to read books. But, and I think that there's a lot of value that can be gained from that.

1:29:54And it's something that many people do and do very successfully, but it's not the only way. I think, you know, there was this book that I just read a book review of called Good Strategy, Bad Strategy. And I just read three pages and that has just stuck with me as a description of like what is wrong with many strategy documents that I read. Just a very clear and crisp articulation of the muddled that gets created and what the right thing looks like. And that's something I never actually read the book. I just read a three-page summary. Other, I think, ideas that have had a profound impact on me from books, book Turn the Ship Around, which I also haven't read, which is about empowering teams on the subject we just discussed.

1:30:40Innovator's Dilemma, Clayton Christensen. I think it's very powerful analysis of how the industry works. I definitely do like to consume content from a lot of different areas. I really encourage people to look wide, not just within tech. Within tech, you know, I do really appreciate Ben Thompson's writing, I think is really good for understanding the tech industry. But I like to also spend a lot of time consuming writing from other intellectual disciplines. One example is I've found that the whole geopolitics, national security space is an area where there's actually a lot of thoughtful writing that I had never been exposed to.

1:31:28And there are people who basically spend their lives really thinking about what happens on a battlefield and what goes well and what doesn't. And a lot of those ideas actually translate very similarly or, you know, it certainly rhymes with situations, you know, battles between big tech and how do you manage and motivate teams. and so I would definitely encourage people to take a wide view of the thought that is out there because there are many things that can be learned from things that are outside of just tech and business.

1:32:08Bobby Holley:Last question for you is if you could give yourself some advice when you just entered the industry is there anything you'd say knowing everything you know now? At the end of the day to be humble is the most important. I think that I had a particular attitude, not strictly about myself, but about engineering and the role of engineers and how important engineers were. And there was definitely this attitude of that was the center of the universe. And as I've gone on, I've started to realize that there are many other important things that go and, you know, that you don't necessarily understand when you're first starting out at an organization of why there are certain processes and why there are certain functions and what do we need a legal team for and all these kinds of things.

1:32:58And as I've progressed and found myself more in a leadership role, I have really managed to develop an appreciation and empathy for all of the different players within the organization and within the industry and realizing that everybody has a point of view and a place that they're coming from and that working to understand where people are coming from is the way to make progress.

1:33:29Bobby Holley:All right. Well, thank you, Bobby, for your time. I really appreciate it. With that being said, is there any last words you want to say for the audience? Yeah, I think the most important message I want to send is that the web very much matters. And the web is something that underpins so much of what we do. And I think for a lot of people, it is invisible in something that they take for granted, right? it is like water flowing through the taps or the semiconductor supply chain of this amazing resource that continues to get better over time and continues to deliver us more value that we take for granted.

1:34:06But it doesn't have to be that way and it easily could not and we could easily backslide into a situation of unilateral control by big tech over the experiences that people have. And I think that simply losing focus on something because we've had it so good for so long would be a tragedy. And I encourage people to think about what they gain from the existence of an open and free web and to participate in trying, doing your best to keep it open. And, you know, from a selfish perspective and from the perspective of the web, a great thing that you can do if you are building websites and experiences is making sure that they work in all browsers and that you don't do the lazy path and build something that only works in Google Chrome.

1:34:51Because that is the first step to ending us back in a situation where we were 25 years ago.

1:34:57Bobby Holley:Thanks for listening to the podcast. I don't sell anything or do sponsorships, but if you want to help out with the podcast, you can support by engaging with the content on YouTube or on Spotify. If you want to drop a review, that'll be super helpful. And if there's any guests that you want to bring on to, please let me know. I feel like sourcing very senior ICs. There's no well studied list out there on Google that I can just search this up. So if there's someone in your org or at your company who you really look up to and you want to hear their career story, let me know and I'll reach out to them.