


Scaling and refactoring production systems at Slack and GitHub, focusing on database/query redesign, handling thundering herds/event floods, and building realistic load tests (including 500,000 WebSocket connections). Also covers migrating Slack’s channel membership data and GitHub’s Rails monolith scaling pain.
Maude Lemaire, principal engineer; author of Refactoring at Scale. Background includes 8 years at Slack (performance/enterprise scaling), database scaling work at GitHub, and improving time-to-first token at Cursor. Earlier roles include Rent the Runway (front-end) and being influenced by Camille Fournier’s leadership at Rent the Runway.
Written by AI. May contain mistakes.
Maude shares her background and journey into computer science.
“I will show you a photo and I will ask, what am I seeing here?”
Maude discusses her experience at Rent the Runway and moving to Slack.
“Complete opposite from the molecular biology class where I was like, I really liked it.”
Discussion on the scaling challenges faced by Slack in 2016.
“But she was still instrumental in helping me land a job at Slack a year later.”
Maude explains the database architecture and message handling in Slack.
“In startup land, when you need to get to the next place and just get that next customer or build that next feature that you think is going to capture.”
Learn about the evolution of Slack's database architecture and its challenges.
“But you're saying that there was a time where this charting worked where a large company might just have its own database instance and for a while they...”
Explore the complexities of migrating Slack's database tables and maintaining performance.
“Yeah, I think it came about from calculating mentions.”
Understand how Slack approached load testing and handled sudden spikes in usage.
“Shadowing meaning that you're doing whatever you're doing and you're doing a new operation and comparing that as the same result, right?”
Learn how Slack handles emoji reactions during high traffic events.
“You are very motivated to plop a really good custom emoji on that notification.”
Explore the differences between event floods and thundering herds in messaging systems.
“And I think, I mean, this example I keep coming back to because I had multiple conversations with folks over at Slack about it, like, we should just limit the total number of emojis that are allowed on a message, period.”
Discover how Slack prepares for load testing with potential customer demands.
“Maybe they had an external customer team.”
Understand the complexities of simulating user loads for Slack's infrastructure.
“And with this let's get back to mode and how Slack load tested for a customer expecting to have 500 ,000 users.”
Learn about innovative strategies for effective load testing at Slack.
“But yes, the$150 ,000 still feels like pretty excessive for that one test.”
Discover how to enable product managers to contribute to load testing.
“At first, it was just like a very sort of silly like JSON blob of like, call this thing, then call this thing, then call this thing.”
Understand the circumstances and challenges faced when transitioning to GitHub.
“Speaking of reliability, after Slack, you joined GitHub as a principal software engineer.”
Explore the use of Vitesse for better data sharding and management at GitHub.
“But I was also seven months pregnant at the time.”
Learn about the complexities of managing scatter queries in a Rails monolith at GitHub.
“The bots keep posting in the channels maybe that will have enough data associated with it that it can never run on a single ultra large database, right?”
Learn about the complexities of code migration and the challenges of working with different levels of abstraction.
“it might be harder to fine tune your queries, etc.”
Discover insights into how GitHub manages complex MySQL queries and the impact of caching.
“And Prisma, you can sort of, it's not, you know, plain SQL written out as a string between quotation marks, but you can sort of see the structure of the SQL query being made.”
Explore how AI tools are utilized in software engineering and their limitations in performance optimization.
“And if you have low counts of things, sure, okay, maybe it matters that there's three open pull requests on like this very small repo that like no one really does anything with.”
Learn about the complexities of instrumentation in software and optimizing performance with AI-generated insights.
“do because it's like, oh, like, find me a place where we can send fewer traces to save money.”
Learn about the importance of building a solid foundation in product platforms.
“Because I think a lot of the problems that I work on are more of the like product platform problems.”
Explore paths for software engineers to join product platform teams.
“I'm such a strong advocate for more product folks, more interchange actually between product folks and product platform teams.”
Discover the benefits of collaboration between product and platform teams.
“Like what you're reading your code, what?”
Hear about specific scaling issues faced by Slack and GitHub.
“or we can actually build better software, hopefully.”
Understand the importance of verification and testing in platform engineering.
“I mean, having worked on GitHub and Cursor, she still verifies pretty much everything by hand.”

Get the gist, explore an idea or ask about something said.
On The Pragmatic Engineer, what are the key points of this episode?
Automatic transcript. May contain errors.
0:00How do you scale a product that millions of people use daily when the architecture underneath it is starting to break? Today's guest did this both at Slack and GitHub. Maude Lemaire is a principal engineer and the author of the book Refactoring at Scale. She spent 8 years at Slack, worked on database scaling at GitHub, and improved time-to-first token at Cursor. Today we cover... Low testing with 500 ,000 WebSocket connections, a job where using headless Chrome instances were too resource-intensive and too expensive. Why GitHub had only 16 database charts for all pull requests and issues, and why scatter queries were a massive headache inside a large Rails monolith like GitHub's.
0:33Why platform and product engineers should rotate between teams, and more. If you want to understand what scaling up looks like inside of products you probably use every day, this episode is for you. This episode is presented by TurboBuffer, Vector and Full Text Search built on object storage. It's fast, cheap, and extremely scalable. This episode is brought to you by Antisysys, the platform to release with confidence. In today's episode we'll talk about rewriting massive production systems as they scale up, which is a kind of engineering challenge that's rebuilding a 747 mid-air. But why not just use CodeReview when using AI coding agents?
1:07It's because CodeReview is an increasingly naive and ineffective approach to do so, which is why Antisysys goes well beyond this practice. Antisysys runs your whole system in a hostile simulation. This simulation includes both targeted testing and fast testing. By running this simulation it finds every bug before your users do and because the simulation is fully deterministic it doesn't only find bugs it gives you a perfect reproduction of every issue which makes it much easier to fix issues. The first thing I thought when I heard about Antisys is that automated bug discovery and fully deterministic testing sounds like science fiction but actually it's hardcore engineering under the hood.
1:43Jane Street's Supa-based Rapid Railway and Datadog ship agent-written code with full confidence because they know it's been verified by Antisys. And to give a sense of how Antisys help with large production systems, Datadog use Antisys to verify their new architecture when rewriting the Datadog event platform, a system that collects more than 100 trillion events per day. Today, Antisys is one of many correcting strategies being used as part of Datadog's harness for their coding agents. To see more case studies and details, head to antisys.com slash pragmatic. Mode, welcome to the podcast.
2:16Maude Lemaire: Thank you for having me. It is so nice to have you here. This is a bucket list item. Very much so. Well, this is the first in the podcast. Usually I ask, sorry for the question. I'm not going to start with a question. I will show you a photo and I will ask, what am I seeing here? I'm seeing a baby who looks like how many months? Maybe about a year old. Maybe a little before a year old. Yeah. Sitting in front of a computer. We're also putting this on the screen if you're watching it on video. Same haircut. So my question is, were you born with computers? Very nearly. There are pictures of me much younger than that, actually, maybe like a couple months old.
2:59Maude Lemaire: I truly like I'm barely keeping my head up where my dad has me sitting in his lap, like resting against him. And he's I don't know what he's doing. He's probably like grading some kind of assignment or because he was teaching. He taught computer science for 17 years before we moved to the U.S. And yeah, there's a bunch of pictures of me like holding ASCII tables, which now in hindsight is hilarious. But I ended up doing this sort of by accident. You'd think that all of the variables and the environment would have led to like a very logical sort of linear career trajectory. And that is not at all.
3:43Maude Lemaire: So anyway, he had to think about that. like eons before um but i thought it sounded really boring um i was like uh no no no and i really like my bio classes i thought i was going to become a genetics researcher that was my plan and my parents always told me to chill the heck out because i had plans i had when i started university i started microbiology i signed up for molecular biology organic chemistry all the things I was already I'd already accrued enough AP credits for all the Americans listening that I was technically like a second year college student rather than a first year. Oh, yeah. Rather than a first year college student.
4:22You want to speed run college.
4:25Maude Lemaire: Oh, yeah. I was ready. Skip the fun years. Yeah. Get straight to work. Which I kept doing that. I had like a 19 credit semester at some point. I was like, why did I do that? That was such a poor decision. But I mean, I had fun doing it. So I guess, you know, my my brand of fun is, I guess, different from other people's brand of fun. And yeah, I was going to do that. And then like three weeks in, there were a couple of things that really bothered me. One, I it's not because you're good at something that you like doing it. And I think I conflated the two for a really long time. I mean, to be fair, when you're 18, like who like what do you know?
4:57Maude Lemaire: But I conflated being really good at memorization with liking memorization. and I don't know if you've ever taken a molecular biology class, but it is a significant amount of memorization and you're learning the Krebs cycle. So I was going to switch into organic chemistry, into chemical engineering. And to do that, that's a different faculty at McGill. I mean, it's a very big school. There's faculty of science. I was in faculty of science. There's faculty of engineering. And I was looking at all of the requirements for first year engineering students. And there was a programming class. There was a intro to computer science.
5:32Maude Lemaire: So I swapped out one of my bio courses the next semester with the intro to computer science class. And I told my mom I swapped it out for a computer science class. Please do not tell dad. Please do not tell him that I swapped it out for a computer science class. Like, I don't. I'm scared I'm not going to like it. And then he's going to be sad because he loves this so much. He loves it. This is like true love for him. And she was like, sure. OK, I'm pretty sure she told him like two minutes later. Because that's what my mom does. I love her. But like, man, I think she knew that it was not going to be, you know, a wreck.
6:09Maude Lemaire: And sure enough, eventually, like I tell them I'm taking this class and three weeks and I'm like, oh, I really like this. Complete opposite from the molecular biology class where I was like, I really liked it. It was like, oh, with the power, I'm sure you've had this feeling. It's like, I only have a computer and I type into it and then it does things. I feel like a god. And then your first job, you did internships, but your first job, which followed the internship, was at Rent the Runway. It was, yeah. It's a pretty well-known company. Thanks to anyone who's entering management will probably know of the book, The Manager's Path by Camille Fournier.
6:48And she was at Rent the Runway.
6:50Maude Lemaire: She was, yeah. And actually, that was a big part of why I wanted to go back after my internship there was because I you didn't see very many women in those CTO type positions at the time. Also, like I mean, I my mom very nearly became a fashion designer. And so we grew up watching Project Runway from the very beginning. That was like the one thing I was allowed to like stay up late to watch was Project Runway. and so it combined two things that I loved uh combined you know programming computer science I was mostly doing front-end stuff at the time and and fashion I could just like rent stuff for dirt cheap and like walk around in a like ball gown if I wanted to it was it was so cool I think the there were so many women on the team in general I think partly because of the space that the company was in.
7:41Maude Lemaire: But I also just for folks who've had the opportunity to be in a room with Camille, she just commands this presence. And I had so much respect for that. And I mean, even if, you know, I was many levels removed from her, I was like, I want to learn from this woman. Like, I really admire this, this presence that that she has and this confidence. And so it was an easy sell. It was a very easy sell to go back. But then she left a couple months after I came back. But she was still instrumental in helping me land a job at Slack a year later. This is Slack in 2016. So at the time, I vividly remember because in 2016, I joined Uber.
8:26We still had HipChat, which was Atlassian's Slack offering or like Slack competitor. It was used by a lot of companies and then they suffered two terrible and undefensible data leaks. Uber almost immediately migrated off to a product and I remember that they evaluated Slack and Slack would not scale. I'm not surprised. I thought this was later in 2019, Uber would move to Slack, but I wrote a thread about this on Twitter and I said like, well, we built something called uChat, which was built on top of Mattermost.
9:01Maude Lemaire: Yes, yes, yes, yes. And I was thinking like, did our in-house team just want to build something? And did they just like say that Slack didn't scale? But one of the engineers who built UChat, because they were working hand-to-hand with Mattermost, they were actually expanding it, the open source offering. He actually said that they sat down with Slack and Slack said that they cannot support 50 ,000 or 40 ,000 people in one room at the time. So there was no way. So like just to set the stage in 2016, Slack still maybe struggles to serve more than 10 ,000 people in one room, if I remember. I think it was about 30.
9:35Or 20 ,000 or 30 ,000.
9:36Maude Lemaire: I think 30 ,000 was Comcast, 30 ,000. But they can do 40 ,000. So this is the state of Slack. This is early Slack. It was already wildly popular. People loved the onboarding. Teams was in infancy. I'm not even sure if it was born at the time. so yeah i don't think it was you you get to slack yeah yeah and what were the problems you had to work on that you got to work yes yeah um yeah i very much got the the performance itch from from at the runway it just wasn't i just had to work on it you know on my own time but i still enjoyed doing it and then when i showed up at slack um i my onboarding buddy was um just like very senior engineer who was working on performance issues specifically at the time.
10:25Maude Lemaire: And I burned through some of, you know, my starter stuff. And I was like, what are you working on? What are you working on over there, Moysha? And he was like, oh, let me send you some tickets. Like, do you want to work on performance? I was like, yes, yes, I do. So he sent me a couple things, like relatively easy stuff. In startup land, when you need to get to the next place and just get that next customer or build that next feature that you think is going to capture. Like you make a lot of short term decisions that, you know, probably inside your soul, you will revisit at some point. And a lot of the performance work that we were doing at the time was just a lot of revisiting of shortcuts.
11:05Maude Lemaire: I mean, like well calculated shortcuts, but like shortcuts that we knew were going to bite us in the butt eventually down the line. So a lot of that work I was doing. And this was before the like enterprise, the Slack enterprise product was launched. But it started a couple months before GA. IBM was already trialing the product. They were sort of on beta there. They are a large company. Lots of people work there. That's 100K plus people, if I recall. Yeah. So one of the distinctions actually was it was sort of fine to have tens of thousands of folks on one in one channel at one time. And we were sort of spreading them out across different teams within the same Slack organization.
11:55Maude Lemaire: Obviously, they could DM each other among some other things, but that sort of worked. But it was IBM was sort of falling over on a regular basis. And I think I was not part of these conversations, but I get the impression in hindsight that when we agreed to GA, then they agreed that they would start scaling and adding more users to Slack. Like once we reach GA, you know, we can't hold the reins on you adding more people as much. So we hit GA and they were started adding more people like pretty aggressively. uh we started we ended up having to spin an actual team up that was just the enterprise performance team I ended up being able to join that team because I just hadn't to be working on odd random performance things um I was definitely the most junior person on the team but I love that I feel like I've spent so much of my career to date being the most junior person on the team I mean less so in the last couple years but I that's how I've learned the most and it's really just like pushed me out of my comfort zone over and over again and it's just like it's the most it's the most fun and can we do a bit of a slack arc in the 2016 2017 slack architecture one-on-one 101 now this is as we know slack has changed a lot this is now how not how it will look like today at all but when we think about slack workspaces then channels private groups dms behind the scenes how was it built up in terms of databases in terms of what happened when you sent a message to someone or to a channel in terms of fan outs or rights?
13:34Maude Lemaire: So the pattern before Enterprise and still with Enterprise was everything belonged to a company, a workspace, a team unit. And all of the data also was sharded by team. And so if you were on a really big team, well, we just would give you a bigger database. and eventually you run out of the largest size of database that you can throw money at Amazon for. But you're saying that there was a time where this charting worked where a large company might just have its own database instance and for a while they... It would work. It would work, they didn't fill it up. Yeah. And smaller companies which shared, they would be multiple on the same...
14:16Maude Lemaire: Multi-tenant on the same database instance. But you as a customer, your data was not spread across multiple database charts. It was just on this one. It was also how we distributed messages. So for the WebSocket connection, when you were connecting, logging into Slack, establishing that WebSocket connection, you were establishing that WebSocket connection with a server that was associated with your company. And so if everyone at your company was very busy, very busy chatting on Slack, you all were on that one same server. And again, like if we had really, really busy customers, eventually we could just say like, Like, oh, well, this is your instance of the chat server.
14:58Maude Lemaire: And that was mostly fine for a very long time. Enterprise started complicating that in some special ways in that we, I mean, again, startups, you like sort of make decisions that are fine in the moment, but then sort of don't scale. We sort of made the assumption that like inside really big companies, it was smaller companies. and so like marketing at say ibm is its own company so you sign into the marketing workspace at ibm or the engineering works i mean engineering actually is too big it's some subset of engineering at ibm sounds like it's a little bit of wishful thinking that you could just turn this large problem into smaller problems right which and then you can just make your current architecture work But there still needed to be like an overarching body.
15:47Maude Lemaire: So we created an enterprise, which was really just another row in the team table that just said, is enterprise one? And then all of the teams associated with that enterprise had an enterprise ID. And you still within IBM would want to say, you know, say you were working on one of the engineering teams. I was working on a different engineering team. We want to collaborate. I want to be able to DM you. We're at the same darn company. I should be able to send you a message. We pretended that these teams within a company were their own companies. But then we had to create this other entity that overarched everything where you could still cross teams.
16:27Maude Lemaire: And that got very complicated very fast. But then the part that became really terrible was everyone across all of IBM, whether they were on the marketing team, the engineering team, whatever, were all querying the enterprise shard for all of their DMs and the big company-wide channels, all of that stuff, all at the same time. And then within that data, when you're booting up Slack, you have your channels that show up in the left sidebar. You'd think that it was just like, cool, get four user ID, get all channel membership entries. That was not the case. And I think it came from a good place in that DMs seemed like a separate thing from a regular public channel, right?
17:10Maude Lemaire: Like they seem different. And so that was a separate table. And a public channel is a public channel. And so that was a separate table. But then private channels seemed different. And I think like from a product perspective, you can totally see it. And from a product security perspective, you can also totally see that. You want more scrutiny when you are querying for private channels. And so that was a separate table as well. And then group DMs came about later and it seemed like they were just sort of like a private channel. And so they were just glued into the private channel table. So when you were making that query to populate your sidebar, you were actually doing a big, big old join union.
17:48So what happened once a team had the idea or figured out that this through these tables don't make sense, but now you're talking about a lot of customers, including the likes of IBM, just, you know, putting a bunch of load on it, they're using it. How do you migrate from these three tables that are actively being used into one.
18:07Maude Lemaire: All the time. Yeah, I think it came about from calculating mentions. So calculating mentions, like to get the badge icon, was also a very complex query, whereby you had to join across all these tables, very latency sensitive. And we had gone through the process of doing as much as we possibly could at that time to ease the load. and in one of my teammates doing some work on mentions he's like we gotta we gotta join these tables together it will not like there's only so much we can do before we have to like really start digging the hole so i i was like sure yeah i i will endeavor to do this very scary thing that no one else wants to do that seems totally fine i have so much appreciation for this in hindsight too in that if you look at flamework on GitHub, which is the PHP framework that Cal Henderson open sourced after writing Flickr, it will explain many things about what Slack was like early on.
19:13Maude Lemaire: Yes, I highly recommend it. It's a good, I think it's still there. It's still there. It's a good read in terms of like, oh, this explains so many things. So we had, everything was super procedural for lack of better term in terms of how code was organized. We had sort of like API entry points and it was like chat.login or whatever. Right. And it was like, OK, this is the API and this is all the code for the API with some helper functions in there. There were a wild amount of functions that were like many, many, many, many, many lights long. But there was just like straight SQL queries like embedded inside the code.
19:50Maude Lemaire: Like an ORM did not exist in this world, whatever you want to call your data access layer. We did not have one. And there were actually there was a lot of opinions about whether we should even do anything to hide the SQL queries. And. Yeah, now, I mean, I totally understand why why they had that stance and why we did the super minimal approach that we ended up doing, because it is so easy to write a SQL query that will like shoot you in the foot over and over and over again, unless you're forced to actually understand what it does in the context that is being called. But we had like 400 something call sites that called into any number of these channel tables.
20:39And at Moise call sites, these are SQL instructions in the code.
20:42Maude Lemaire: Just like select star from Teams channels where user ID blank. Yeah. Where team ID blank, user ID blank. And so you just copy paste the pattern that exists, right? That's always the problem with migrations is like you have to make it so clear that like this is the old pattern, stop doing this because there is still while it's going on so many examples of it in the code. So I had to write some, I started off by going through doing some kind of regex approach where I was finding all the places that we were querying for this. Thankfully, the table names were like pretty easy to find and identify.
21:15Maude Lemaire: I got very lucky there. And then I just one for one took the call and just pasted it into another file. Yep. That is all I did. You created a layer that did the same thing. Yeah. Just a class, basically. Yeah. I didn't optimize anything. I literally just copy pasted. And now it was calling into a function instead of straight up calling SQL. And you had a minimal abstraction layer, if you will. Yeah. Which already was like a little bit spicy. but I tried to make sure that like the function name was like real darn clear as to what it was doing and I got the spicier folks to be okay with this approach so I felt that felt like a win and so I migrated everything over everything was feature flagged basically I had I don't know how many feature flags at some point but I had so many distinct ones just to make sure that I was doing it as carefully as humanly possible even then you know I still like definitely borked some things along the way.
22:11Maude Lemaire: I accidentally, when I thought that we were basically there and ready to start flipping on to prod traffic, I'd run all the backfills because obviously like I was moving everything to a new table. I was doing, um, we were doing a double rights for a very long time to make sure the data will be up to date. Yes. So that's where we started. We started doing double writes, making sure that that all looked sensible. I was doing a bunch of... Did you also do shadowing? Yes. Shadowing meaning that you're doing whatever you're doing and you're doing a new operation and comparing that as the same result, right?
22:50Maude Lemaire: Yeah. We did a bunch of that as well. But obviously, we had to make sure all the data was there first. So we did the double writing, did all the backfills. And then we were shadow reading and comparing if there were any differences, throwing away the new call, obviously, the new results and keeping reading from the old ones. I was about to flip all of Slack to reading from the new table. Which at this point you confirmed works. So that should have been a little risk change. Yes. Yeah. All the data looked right. I had a bunch of data warehouse dashboards that showed me how many differences existed between the two sets of tables within reason.
23:27Maude Lemaire: And then I was about to turn it on to all of Slack. This was like Thursdays back when we had gather hour, which was like after after work hours everyone would sort of hang out if they wanted have appetizers whatever and i turned it on and then i realized oh wait no something's not quite right um and oh we didn't just do a gradual roll out because with these things you don't really do a gradual roll out internally we were like not that big at the time everyone was like gather hour you know i was like oh this seems safe we've been shadow reading it's fine yeah i turned it on i was like, oh, there's like data isn't there's some data that's like not quite right.
24:08Maude Lemaire: And I had been in so used to essentially like running the script that like would delete all of the backing data and rerun the backfill or at least chunks of it because I had been seeing discrepancies. And honestly, it was so much easier to just nuke the data re backfill than it was to try to like pinpoint and like scalpels fix the the call sites so i forgot that the flag was still on to everyone and then i promptly which flag was on the flag to have everyone read their channel memberships from the new table okay yeah i forgot that it was on for slack not everyone externally like just us and i was like oh there's some discrepancies oh right but i fixed those let me just nuke the table for our team and backfill, which like took 25 minutes to do.
24:59Maude Lemaire: Like it was not. Oh, but did everyone lose their messages until then? Oh, yeah. People had completely blank sidebars for about half an hour. Inside of all of those. Inside all of the Slack. And Slack obviously uses Slack for everything. For everything. But everyone was at gather hours. So like it was only there was this only one guy, Kevin, who had been sort of like helping me like coach through this, who DMed me. And he was like, what's going on with my sidebar? I was like, I'm so sorry. Oh my gosh, let me flip the flag. So I re-flipped to read from the old data while it was backfilling, it finished backfilling, and then I swapped it back.
25:33So what did you learn at Slack about load testing? And I'm asking because right now, load is going up on all systems. We see this. AI agents generate more pull requests, more code. They put more load on the systems. I assume we will have the point where we generate more data here and there. But at Slack, you had this interesting challenge where a customer with 500 ,000 users wanted to go onto the platform. Slack was architecting for it, and you had a plan of 18 months. And then COVID happened, and you had to bring it up forward. But I'm more interested in that. And as you you built load testing of something called Koi Pond on this, but when you think of load testing, of checking how a service will be able to handle this type of load before it hits, what are the techniques that you've used and would still use today?
26:26Maude Lemaire: Tested production. But how did you test the production something where you don't have the load just yet? Right? Yeah, no, I mean, I think there's a lot of different ways to do it. The simplest way that I think a lot of people can manage to do it, it depends on what kind of usage you are worried about with your product. And nailing down those usage patterns first and where you're really concerned it's going to fall over is the determining factor in how you figure out how to simulate load. In Slack specifically, what was usage pattern? There were a couple a couple of key ways that it had historically fallen over thundering herds, where I mean, I think most people have some kind of thundering herd problem at some point where you say we spell out with the thundering.
27:11Yeah, just for those. Oh, yes. Yes. Yes.
27:14Maude Lemaire: Suddenly heard. Suddenly everyone shows up suddenly like that. That is a thundering herd. A thing happens and then suddenly everyone shows up and does the same thing at the same time. I guess a good example is if your service goes down, like you're, I don't know, you're running, let's say, imaginary, like Google search, and somehow all your machines die for a bug. And then you have like a thousand machines. Obviously, Google has more, but we can just imagine. And then you start to bring up machine one, machine two, machine three, but each of them gets hit by the complete traffic and they go down again, right?
27:47Maude Lemaire: Yes, yes. This is a classic example. Oh, yeah. Yeah. The classic example in Slack is say CEO says, oh, great quarter, everyone. You all get Friday off. Posts this message in the big company channel that has like 100 ,000 active users in it. That notification gets sent out. You see that notification. You are very motivated to plop a really good custom emoji on that notification. You click that notification. You immediately land in the channel. In the channel, you're fetching channel history. And then on top of that, you are adding an emoji reaction because you're really pumped about getting Friday off.
28:21Maude Lemaire: But so is everyone else also getting putting custom emojis on that message because they're also in the same five seconds. Yes, in the same five seconds. So the message that came in to tell you that you're getting Friday off came in over the WebSocket connection and it was fanned out to all 100 ,000 active users. but then all 100 ,000 within a couple seconds, right, of those people show up to that channel, issue a channel read, and then are sending emoji reactions. Every single emoji reaction fans out also over the web socket. So one custom emoji turns into 100 ,000 custom emojis being sent to everyone who's connected.
29:00Maude Lemaire: And if you add 10, well, then you can do the math and it just like balloons and balloons and balloons. It's like prime example for like pure explosion. And as much as I think we we had done some work because we knew that emoji fan out was like actually really expensive to rate the importance priority order, the types of messages that get sent over the WebSocket. And so like message like actual text messages were the most important. And then emojis, if you were in a very high load situation, would just get dropped on the floor, which like is the thing to do, obviously. I'm sorry to say, but the next generation, I'm not sure if Gen Z or Gen Apple might disagree that emojis are more important than text messages.
29:41Maude Lemaire: I need to know if someone reacted to my message. No, but I hear it makes sense of prioritizing, especially based on size, right? Yes, absolutely. And there's some amount of, and this is where like, I think product thinking is, you can't go around thinking you're going to load test strictly from an engineering perspective, you have to be putting your product hat on. You absolutely have to, because that's where understanding that like messages with text are more important than emojis. That's more of a product call than it is like an engineering call. I'm coming to you and I'm saying like, we have to drop something.
30:17Maude Lemaire: Let's reason together about what makes sense to drop and understand it. And sometimes it's like not what you would expect. And I think, I mean, this example I keep coming back to because I had multiple conversations with folks over at Slack about it, like, we should just limit the total number of emojis that are allowed on a message, period. So we currently limit or I think last I checked Slack still currently limits. You can only have 23 distinct custom emojis as a reaction on a message, but you can have as many like plus ones on those emojis as you would like. why the 10 ,000 and once emoji does not matter we should just cut it off at some point and I think like whoever I mean I think part of the problem is like no one really owned custom emojis it would have been the messaging team I guess but like it's one of those things that's just so ubiquitous is that this word to the slack experience that like kind of no one owns it um so who would have made the call on like what is the proper limit to put on this but thundering herds was definitely super top of mind as one of those issues i wanted to ask are event floods and thundering herds the same thing no they're different what what are event floods so event floods are when your client is personally overwhelmed by the events coming over the web socket connection and so any i mean that could be from that specific example that's sort of a thundering herd and maybe an event flood event in that you're getting hammered with emoji reactions after the fact as well but an event flood is just say you know this message is happening the co you get friday off but it's happening in 16 different channels at the same time that you personally happen to be in not everyone else is in super, super busy channels.
32:12Maude Lemaire: But maybe you happen to be. This happened to, there was this guy who was our like main Slack guy at IBM. His name is Jeff Robke. He eventually became a PM at Slack, which I think is like the most interesting thing ever because this guy was on the receiving end of so many event floods. And I felt so bad for Jeff all the time. I mean, I was in some channels. I'm like, tell Jeff, I'm sorry. But I still don't fully get like what the event flood experience is can you tell me like what yeah so one of the things that would happen to him right is that before some of the re-architecture work whereby you didn't have a bunch of distinct web socket connections for each team that you were logged in jeff was an admin at ibm so he was in hundreds of different workspaces at one point i think the maximum number of workspaces you could be logged into was around like 30 something and you're saying he was in this because IBM as an enterprise had this idea, this concept of like the marketing team, the finance team, the thing.
33:14Maybe they had an external customer team. And as the admin, he was in all of.
33:18Maude Lemaire: He was in all of them. And they had so many teams and he was in all of them. And so if you were getting a message in an enterprise level DM or a channel that is shared across all 30 of those workspaces that he was currently logged into, well, he would get that message 30 times. There was no deduplication of that whatsoever. So his client, his machine, his notifications would be both overwhelmed from maybe a little bit less in processing. That's fine. But like from a UI, from a being unable to see what's going on, an annoyance point of view. Yeah. I mean, also his client was just dying all the time.
34:00Maude Lemaire: Which was not good. Yeah. And interesting. This is the first time I've heard about event floods. Yeah. And they were very user specific, but they were particularly painful for power users. This happened to power users all the time. And power users were also some of the biggest champions of Slack. And so it's a double edged sword of like, I'm sorry, you get the absolute worst experience of our product, but also you love it, I guess. And you are trying to make sure that people adopt it more. It's really, really rough. Mo just made the point that the heaviest users of Slack got the worst of the event floods.
34:34Today, the heaviest users of Git happen to be your agents that often push code in parallel, which brings us to our seasoned sponsor, Entire. A few weeks ago, I told you that Entire was building a Git forge. Well, it's finally here. You can now host private repos directly on Entire in the US, Europe, India, and Australia. And Entire has just reinvented a pull request for the agentic era. The problem with PRs is how agents write it faster than how any of us can review it. So Entire just replaced it with something called Trails. A trail collects everything about the work in one place, the intent, the code changes, and the agent sessions.
35:08A tail can be spread across multiple repos, so the change to your backend and your web app gets reviewed once. With a trail, you can configure customized evals like code review, security reviews, slot reviews, and others. They look like a heads-up display in an airplane and score your work on every push. And you can see how each agent from passes or fails your user-defined gates and then decide when you merge based on these. The point is that you control when you allow your agents to be allowed to merge code, like merging only after human review or after all gates pass. Oh, and one more thing.
35:39Entire ship continuous memory. This turns your repo into your brain so your agents can search past coding sessions to find the context behind earlier changes. Basically, answer the why behind the code changes. Memories get shared across the team and across the repo. If you want to see what an AI-native GitForge looks like, head to entire.io slash pragmatic and check out trails. I'd also like to mention our presenting sponsor, TurboPuffer, who were rebuilding their storage engine to make storage faster and cheaper at the kind of scale we're talking about in this episode. The new system, called TurboPuffer v3, is not yet in prod but it already passes CI.
36:12And when the rewrite started in September, it was not nearly as performant as the current production version v2. The TPuff team has been perf-maxing query plans until they are as fast or faster than v2, doing all of this in public all the way up to the v3 production launch. And it's so cool to see their progress. The new system went from being 10 to 100 times slower across most metrics to several times faster on things like cold vector search, cold full-text search, or counts. The v3 system is still slower in areas like hot full-text search, but only 1 to 2 times slower versus 100 times slower when the perfmaxing started.
36:47And the Teephuf team is sharing exactly how they optimize things in their development log with things like vectorized query execution, group count aggregation, compressed column encodings and many more. But hang on, why is the team doing a full rewrite? It's because search is changing and so is TurboBuffer. Teebof started as a vector database but it's no longer a vectorDB. It has evolved into a high performance database for search. The v3 disk cards the vector primary index and all indexes will share the same columnar storage system. Every query will run on vectorized query engines which is something that you don't usually see in search databases but which will both results in very fast search and just generally a very fast database.
37:27The Turbo Buffer team is keeping a work log of their journey and you can follow along at turbopuffer.com slash v3. It's a blast to see how it's being built. And with this let's get back to mode and how Slack load tested for a customer expecting to have 500 ,000 users. And then back to load testing. So how did you simulate the load that you would be expecting to get? In this case you knew that a big customer would be coming with 500 ,000 or a million users. Yeah. And you didn't know if Slack could handle it, but you want to figure out before testing and production, before they all came in, did you build a custom load testing tool?
38:03How did you go about it?
38:04Maude Lemaire: Yeah. So we'd had a couple of like half conceived load testing options to date. Mostly that tested just API throughput and making sure that we could withstand the underlying load, primarily on a database. And so if you are loading up a channel that issues a particular API request, I think it's conversations.view unless they've last changed it. And that makes a particular database request a SQL read on that channel shard now because everything started differently. But it makes a particular request. We wanted to know that we could handle that RPS. The problem with that is, and any of the load testing solutions before this particular moment in time, was none of it had active WebSocket connections.
38:48Maude Lemaire: And the two particular cases that we just talked about, the WebSocket is the problem in some ways. I mean, we've never really had latency issues with sending messages through the WebSocket. Like those stateful services have one of the co-founders was like the mastermind behind all of these. And like the amount of time and energy he sank into making them as performant as humanly possible is commendable. They were never really a bottleneck. It was either how the client handled it or the read downstream event caused by you seeing and receiving this message that was the further bottleneck. And so we needed a system that had an active WebSocket connection in order to be able to adequately test the whole flow.
39:36Maude Lemaire: It's not just sending like, as though you're reading a channel. It's not just an API. Yeah, yeah. So we really, and it's not just like flooding a database with like. But when you say you need an active WebSocket connection to test for 500 ,000 loads, you needed 500 ,000 active WebSocket connections. We needed 500 ,000 active WebSocket connections. Yes, we did. Okay. Yeah, yeah, which is not easy to do. And so we had a solution before this customer came in, which I'm 90 % sure it's public knowledge now that it's Amazon. But we had a solution that would spin up Slack as a headless Chrome instance in like a bunch of Kubernetes pods.
40:14Maude Lemaire: and try to run a tiny little JavaScript script and click buttons, like pretend to be a Slack user. Pretend to be a user, yeah. Which was sort of fine, but I don't know if you've tried to wrangle more than 50 ,000 headless Chrome instances in Kubernetes. I don't think many of us have. I don't think many people listening or viewing have. It is not a good time. I mean, the problem with the headless Chrome instance, right, is like just running Slack and like 200 Chrome tabs and whatever on your laptop is like kind of a time. And now you're running like, you know, LLMs on top of it. But like it's an experience in really pushing your laptop to the limit.
41:00Maude Lemaire: And now you're running like per pod, like attempting to run like 10 ,000 headless Chrome instances of Slack, which is a very memory and CPU, mostly memory intensive application. The amount of money we were spending in these EC2 instances. The problem seems to be that per node, per VM node, depending on the size you will be able to run. You know, you'll know, like, I don't know, like 10, 20, 50, 100. I don't know. You figure out how much, but then you pay for every one of those VMs, those nodes. You have a number of them. Yeah. I mean, look, like Slack raised a bunch of VC money, so I guess. Sure, sure.
41:39Maude Lemaire: Yeah. We know where some of that went. And also from customers, of course. Yeah, yeah, yeah, of course. That is wicked expensive. It is. So it was 37 cents per headless Chrome instance, per five-hour tranche of load testing. Yeah, we're talking about$150 ,000 for that one test. yeah yeah yeah which like okay that's like financially not sensible like we like it just didn't work but but but i guess if if you compare it like of course i'm playing devil's advocate but 150 000 is great but if you have a customer coming up with 500 000 seats if they would get a discount they they would pay five million dollars per month like you would think there is there is a point of investing to make sure that it works for them and they don't complain i don't know where that comes in.
42:26But yes, the$150 ,000 still feels like pretty excessive for that one test.
42:30Maude Lemaire: Oh, yeah. Yeah. I think for and for that one, I mean, that's only a five hour chunk of time. And that's assuming that you're able. Yeah, that's only five hours. OK. Yeah, that's getting a problem. And you're that's assuming you're testing all of the feature tests that you wanted to run during that five hours, which was not the case. That was not the case because we were spending half the time trying to bring pods back up because they were crashing. Okay. Did you figure out a better way to do it in the end? Yes, yeah, we did. Lower fidelity, but definitely a better way. We gave up on the whole headless Chrome instance situation, which was obviously like the highest fidelity and best option if you wanted to make sure that it was truly testing how Slack would work.
43:11Maude Lemaire: But instead, we spun up a bunch of little Go processes that would each establish a WebSocket connection and run through a series of API requests that made it look like they were actual clients. And so they would still receive everything over the WebSocket, send everything over the WebSocket, as you would expect, and still have a connection to the regular API that the client was making requests to and also our edge cache. We didn't talk about our edge cache at all, but there's a bunch of data that needs to be really snappy to load when you load up Slack. Like when you go to switch channels, the channel list is in an edge cache closer to where you are because all of the main API requests still hit US East 1, which is great.
43:56Maude Lemaire: I mean, we are so boned if US East 1 goes down. All those main requests still hit US East 1. So we still had to also test the latency associated with making requests to those edge caches. And so we also established that request pattern there. It sounds to me that with load testing, all I've heard from you is like, you know, step one is understand how your product will be used, what will likely top it over. Because once you know you're not going to just randomly load test, you'll actually can test target it for that. and then building something that simulates it good enough, knowing the resource constraints.
44:33So in this case, don't boot up a whole slack in a headless Chrome, but just mock, not mock up, but build out that critical path that you care about. And if you combine this to your domain knowledge and your expertise, knowing how the product works and knowing how, you know, you knew how it used to fall over, you can now be reasonably confident that if it falls over, it will not be this way. It might be a different way, but I think it's nice that we're talking about the cost of testing, especially like we've talked about the cost of LLM. But like there is cost considerations and also repeatability, how lightweight it is, how much easier it is to control it and to even tweak it.
45:12Or it sounds like your solution, you could later, if you find a different failing those case, you can just rewrite it to that. It's not a big deal.
45:19Maude Lemaire: Yeah, yeah, it was really lightweight. It just did what it needed to do. We created a pretty, I mean, we evolved the system whereby you could create like routines to send to these like fake users to run through over time. At first, it was just like a very sort of silly like JSON blob of like, call this thing, then call this thing, then call this thing. I wrote a like mini recorder tool where I could via slash command in Slack say like slash start recording, go through a whole flow in Slack. It would record every single API request I made across both to the normal web app backend and to our edge cache.
45:56Maude Lemaire: It would record every single outgoing and incoming WebSocket message. And then I would sort of sort through it all to just the pieces that I wanted to simulate and then inject that into another tool that I wrote that turned that into a script that I could then send to the simulated clients. Because the other half of the battle was figuring out what are all of the APIs that get called when I do this one product flow, which is like not a small number. And you wanted to be sure that you captured the entire thing. So that's where the recorder tool came in, came in pretty handy. And we used that over and over and over again.
46:35Maude Lemaire: And I would, again, to go back to the product thing, because I'm going to keep hammering the product thing. I think like the best product managers are mindful of, like performance and reliability are important aspects of any given feature. and the best PMs know that and care about that and optimize for that. And I was trying the last like several years that I was there, I was always trying to push more tools into the hands of PMs and say like, hey, you want to add your use case to be automatically tested with KoiPond with our load testing system? Here's the recorder tool. Go record the flow that you want us to test.
47:11Maude Lemaire: Send us back the file. We'll plop it into the system and we'll run it at the frequency that you want it to run at. Because they knew, you know, oh, files do this weird thing when we like pop open a canvas and it like does this thing. And like, I don't know. I don't use canvases the way that like most of our users probably use them. You do. You're the PM. So like go record it and then bring it back to us and then we'll just shove it into the system we wrote. Speaking of reliability, after Slack, you joined GitHub as a principal software engineer. And this was 2024. So this was at a time where GitHub Copilot was already very big.
47:46In fact, Cursor and other competitors were already, as tools, were starting to get up there. The load on GitHub services already started to increase and GitHub's reliability started to visibly become shakier, not as bad as it is these days. What did you observe inside of GitHub? Can you tell us the vibe when you joined? If you can give us a bit of an understanding and appreciation of what GitHub has been built on, this massive Ruby monolith. And when you joined, what was it like?
48:15Maude Lemaire: It was a really interesting time, particularly because one of my friends from Run the Runway, actually, we're going way back, had been at GitHub for many years at that point. And we'd sort of been keeping tabs on each other's careers. And she told me, Moad, we are going to run out of space. Like GitHub is still primarily on data centers and it's like increasingly less so over time. They used to have their own data centers. Now they're moving to Azure. This is public information that is shared. Yes. Yeah. The move to Azure is front and center, but still a significant, or at least at the time when we were talking, a still significant amount of load was going through the data centers that like they had whole teams managing the metal.
49:00Maude Lemaire: And she was like, we're going to run out of space. The space is running out. And we either, well, we both have to figure out how to move to cloud. And we also have to figure out how to do less. Because we're doing a bunch of dumb stuff, as everyone does, that we shouldn't be doing. And we can make the system a lot more efficient with what we have to give bias more runway. I was like, oh, this sounds like a very interesting problem. Would love to work on this problem. But I was also seven months pregnant at the time. And so what that ended up being was I showed up. I was seven months pregnant, which is actually kind of if you can swing it a really great time way to join a company because you are you sit there and you're like, cool, I will just drop a bunch of opinions like, you know, nothing very strongly held, but I will just drop a bunch of opinions on people.
49:55Maude Lemaire: I will own nothing for the next two months and then I'm going to go have a baby and then I'll come back. Let's see what happened. Let's see what happened. And in my first couple weeks, I was sort of floating around some spaces and I, having done a bunch of data migrations at Slack and some of the stuff that we didn't talk about, but was moving from this like team sharded tenant based model to a more dynamically sharded model with Vitesse. where it turns out, no, a team was too big. So you mentioned Bitess. What is Bitess? Bitess is essentially a wrapper on top of MySQL that lets you shard more dynamically.
50:36Maude Lemaire: And so you have a... So instead of just having MySQL run on one database and when you outgrow it, good luck. You manually move the tables or copy them or whatever. It's a nightmare. It allows you as it grows and as it outgrows, it will, I don't know, predict or it will now have a way to more seamlessly move over to multiple machines. You can reshard, redistribute your data a lot more seamlessly than you would be able to otherwise. And you can choose different sharding keys more easily as well. And so like before, it's like everything was sharded by team. There's still data that is sharded by team where team is a logical unit that makes the most sense for fetching that data.
51:17Maude Lemaire: But actually turns out that the smallest, most logical unit was channel. And so we ended up re-sharding a bunch of our data to be fetchable by channel ID rather than by team ID. And then that allows for much better horizontal sharding structure as well. and the idea being that there is no channel that will ever be big enough. Maybe one day it will be. The bots keep posting in the channels maybe that will have enough data associated with it that it can never run on a single ultra large database, right? It's a much better unit from a scaling perspective. So Vitesse has had some of its nascency embedded in GitHub actually and it was really interesting because when I was interviewing at GitHub And I spoke with a bunch of the principal engineers that I would later end up working with.
52:08Maude Lemaire: And they were sort of trying to figure out, OK, we have Vitesse, but I don't think we're actually using it to its potential to where we should be using it. And so they were asking a bunch of questions about like, are Vitesse set up and what we were doing? Caveating this with I did not ever operate Vitesse or deal with the internals. I was more of a I'm a product platform person. And so I just like find that like, no, we should not be sharding this thing by this thing anymore. Let's migrate it. And so that was my perspective on it, which was also mostly their perspective on it as well. And I came in and I started poking at the Vitesse setup and I was like, what are we doing?
52:46Maude Lemaire: What are we doing here? I was used to thousands of shards per sharding key. So say four channels, we have a bajillion because there's a bajillion channels and you want to sort of spread load evenly throughout those. And when I arrived, I was looking at the Vitesse setup and there was, I mean, I don't know if this is still the case, but there were 16 shards for all of pull requests and issues. Across all of GitHub. Across all of GitHub, only 16, which is not very many given the amount of data that we know is associated with that whole space. And my first question was like, wait, why do we only have 16 charts?
53:33Maude Lemaire: Oh, my gosh, this makes me deeply, deeply nervous. And part of the answer was we can't have more because we already have so many scatter queries. So those are queries that are coming in. What are scatter queries? So those are queries that are coming in where you do not have the primary key in them. So say you are querying for a pull request or an issue and you don't have the pull request ID, which is how this data is being distributed. Well, you have to find it. So you have to go through, you have to find and match the criteria. So say I'm querying for all pull requests authored by me. Yeah. I don't know that that's an actual scatterer, but this could be in the scenario.
54:15Maude Lemaire: Let's just assume. And so because I don't have a pull request ID, well, I'm going to hit every single one of those 16 shards. Wow, this will be bad in so many levels. Uh-huh. Because, well, so many people are doing it also. And I understand then why you don't want to have many shards because if you have more shards, you would need to hit all of them. We're talking like it's almost like a linear scan. Yeah. Yeah. Oh, no, it's quite, quite, quite inefficient. And again, I don't know that this is an actual example of one of the... This was hypothetical. This is completely hypothetical, but there were still very many significant scatter queries that were hitting almost every single shard before it knew that it could complete the query and return the result.
54:56Maude Lemaire: Which, yes, I understand why that was a constraint then. You don't want to add more shards. But then again, you also cannot make the shards much bigger and they're under very heavy load. And so I was ready to apply the playbook that I'd been using for the last several years of like, well, let's just burn down the scatter. Let's just get rid of the scatter queries. Don't have scatter queries. Just don't have scatter queries. That can't be that hard. Well, it turns out that can be very hard when you're dealing with a Rails monolith. Yeah. And you've got ActiveRecord. And I always sort of thought it was funny.
55:32Maude Lemaire: There were a lot of ex-Twitter folks at Slack early on. And everyone was like, ha ha, ActiveRecord. And I was like, what are you? Like before they did the whole like Scala Java thing. Right. And I was like, OK, I guess like we're all, you know, joking about ActiveRecord. I did not understand until I like came face to face with it. And I was like, oh, I really don't like this. This is really terrible. So ActiveRecord is Rails's abstraction layer for the database, right? So you define things in, I guess, code and it kind of maps nicely to database, except it might be harder to fine tune your queries, etc.
56:07Maude Lemaire: Yeah, I mean, it does a very good job at hiding the underlying data model from you, which again, pros and cons. I came from a world where like SQL was, you know, religiously written out by hand for performance reasons. And then I was being thrown into this world where at one point we were trying to figure out we were doing a bunch of work around domain extraction. And so how do we rip out everything related to, say, pull requests from the Rails monolith and extract that into a separate Go subsystem, take everything that deals with pull requests, just like make the requests go through, say, the monolith still, but then just like immediately send them over to the other system.
56:53Maude Lemaire: Really standard sort of migration procedure. We started doing that. We started with Starz, and I think that project was eventually abandoned. but I was working, I was pairing with someone who had like a ton of Rails experience. And I was like, okay, you gotta like, because I, as someone who was so used to reading just code that was very sort of straightforward and didn't have so many levels of abstraction, it's a little bit daunting when you're going into a world where now there's abstractions upon abstractions upon abstractions. And I was working with her to try to rip out more of the stars related queries inside of the Rails Molleth.
57:31Maude Lemaire: We'd found sort of the the root caller that created this query that would call into the stars table. And it was like a five line function. And I was looking at it with her on Zoom. And I said, wait, Kate, where where is the call? Like, where where is the sequel being made? Like, what is happening? It's a five line function. She's like, Mo, that's the million dollar question. And I was like, what are we looking at? What is happening? Because I half expected to see, I mean, at this point, I think so many people use Prisma, right? And Prisma, you can sort of, it's not, you know, plain SQL written out as a string between quotation marks, but you can sort of see the structure of the SQL query being made.
58:16Maude Lemaire: You do not get any of that in this case. It's just, it just looks like a regular function call. And in that moment is when I realized this is why this has been impossible to date. And there is because no one had and why the problem has gotten so bad because no one sees the impact of just writing a new line of code. And also fixing it now will probably, it's not as easy as just tweaking your abstraction layer. You would probably have to change how code is being written, being called, starting to call more either functions that call SQL, etc. Like it's just, you took a technology that worked great when you were small, you were able to scale it as long as you, and they figured out how to grow it as long as things are going linearly.
59:04Yeah.
59:05Maude Lemaire: And I think one of the things that blew my mind the most that both gave me a lot of respect for how they've managed to date and also filled me with much dread was when you work inside of GitHub, obviously you have the admin view of GitHub. and there's this bar at the top where you can load up a page on GitHub and it'll tell you exactly which MySQL queries were run to populate the page, which clusters they hit, how fast they all were, totals around all of this stuff. Just like the internal employee view. Yeah, yeah, yeah. And what's called the files disambiguated page, which is sort of like the repo homepage that you land on with the readme.
59:47Maude Lemaire: That page last I checked, you know, before I left. Before you left like a year ago. Actually less than that. But yeah, yeah, yeah. So some amount of time ago was it was over 100 MySQL queries to populate that page across at least three different MySQL clusters. And the thing with ActiveRecord is that it only caches the output automatically if the query is exactly the same. And so if, say, you're fetching user information, like full blanket, like select star from users where user ID equals blank, and you have that full user, but then later on you do select star from users where user ID equals blank and something else, it's not going to reach back into the results that it cached for that first query because they are not exact same queries.
1:00:39Maude Lemaire: every single count or at least when I last left almost every single count on you see on a github page is live calculated none of those are cached there's a lot of reasons why that is mostly product reasons actually um but I that's one of the first things that I was like but why and to go back to like my sort of silly emoji example of like the 10 ,000 and month emoji no one cares? Why does anyone care? Right? And if you have low counts of things, sure, okay, maybe it matters that there's three open pull requests on like this very small repo that like no one really does anything with. But once it gets to like 99, don't just don't show anything, just say 99 plus.
1:01:20Maude Lemaire: Don't bother counting it. Why are we counting this? Really interesting that so far we talked about you did interesting thing that Slack, I get how you also worked at Copilot Code Review. Afterwards, words you worked at at cursor again they've they've really pushed how we're coding with with ai tools but so far we've not really talked about ai it just didn't come up naturally and i wanted to ask firstly what do you think ai so far you have you obviously use these tools all the time increasingly intensively what parts of the job does ai not help you with and what parts of it Has it helped and has it changed how you work?
1:02:02Maude Lemaire: Oh, gosh. Yeah. It has not helped me with performance optimizations basically at all. At least the investigation part. Maybe the implementation part is a little bit faster. What once you know what you want to do? But yeah, figuring out what I want to do. So say I was looking at improving time to first token, time to first visible token. At Cursor. A cursor a couple of months ago where and there's a bunch of factors that feed into how fast you start getting those tokens back. There's a significant amount of time that goes into when you are typing your prompt and you hit submit. There's a significant amount of time that goes into building all of the context associated with the request that you are about to make to send to the LLM.
1:02:50Maude Lemaire: And then there's some amount of data that we are able to cache, some amount that we are not able to cache, either for product reasons, privacy reasons, but also strictly because of load and, again, cost of these systems. It is not cheap to say cache absolutely everything that you have ever queried cursor for in Redis for 24 hours. That costs a lot of money. So we're not going to do that. And then eventually sending that request over to inference proxy, which then sends the request over to the underlying LLMs and providers. Given that there's so much that happens, but also back and forth, a lot of this process is not really as synchronous as the experience leads you to believe.
1:03:37Maude Lemaire: and in the regular flow if you're lucky enough to be able to use HTTP 2 which some poor folks out there are stuck on HTTP 1 you establish that gRPC streaming connection with the backend servers and you're able to split that connection into like four or more concurrent streams and send and receive different bits of information across those streams it's very complicated and also AIs think that they're really good at instrumenting these systems. They are not, turns out. They think they're, it's like writing tests. They think they're really good at writing unit tests, but they're really good at just confirming what they just wrote.
1:04:18Yeah.
1:04:18Maude Lemaire: Which is why some of the skills that I ended up crafting run through like intensive series of mutation tests on the code to then force the LLM to realize, oh, I didn't actually test any of the edge cases. Let me rewrite those tests. Thank you. So the LLMs right now, I find, are good at doing the really basic level of instrumentation, but not great at figuring out, especially in an asynchronous context, what makes sense to instrument how. I love traces. I think they're amazing. You can get so much really good debugging information from them, particularly from a performance perspective. We had this problem at Cursor where we had like too many and too little traces.
1:05:04Maude Lemaire: And by that, I mean that we had traces with millions upon millions of spans, which are completely useless. You cannot do anything meaningful with a trace that has that many spans. And then we also did the very classic mistake that basically everyone does, which is we started sampling for cost reasons, but not in one place, which is, again, really easy to have an LLM do because it's like, oh, like, find me a place where we can send fewer traces to save money. And it's like, cool, I'll just like sample it right before I send it to Datadog or whatever. No, no, no, don't do that. Please don't do that.
1:05:44Maude Lemaire: Always do it in the same place. the problem with duplication is just like so much worse when you're dealing with this. And so one of the first things that I ended up having to do was like, honestly, manually instrument a lot of these flows and actually read through the code myself and understand, OK, this chunk of processing is synchronous. This chunk of processing is asynchronous. And put all of those together. And one of the ways that I started doing that was I had it generate a like big sequence diagram of everything that goes into time to first token. So like from the time you hit submit to when you receive that first visible token.
1:06:28Maude Lemaire: And it was huge. And I went to Staples and I had them printed out on like big blueprint paper because I'm still a really analog person. And then from that, I went through and I was reading the code and I was annotating in on my big piece of paper like, OK, this this is correct or this is incorrect. This piece we could probably do something about this other piece. No, actually definitely needs to happen before this other piece. One of the other things that I was finding that it was doing really painfully wrong is that it was finding sort of the classic things that could. What was doing what the LM was doing wrong?
1:07:02Oh, yeah.
1:07:03Maude Lemaire: So what it was doing wrong was finding, well, wrong is kind of the wrong word. No, it's supposed to be opinionated. I like it. Naively, naively what it was doing was say you have a bunch of things, you're finding a bunch of tasks that don't depend on each other, right? But they're latency sensitive. What's the classic solution? You paralyze them. You paralyze them. However, again, product hat, right? What if basically no one is ever doing all four of those things? And why are you doing all of them at the same time? Like you're wasting resources, latency, you're increasing load for no reason. And so I remember, I think it was Opus that was like very excited that it like found this opportunity to paralyze a bunch of these tasks.
1:07:52Maude Lemaire: And I was like, oh, like on first glance, sure, that seems like a good thing to do. Except that the problem, even if you paralyze them, the longest one is always going to be the bottleneck regardless. And the longest one was always so much longer than the other three. And the other three tasks were being run such a small 1 % of users were doing any of the other three. And so it ended up being a complete waste of time. Like this was not the optimization that it was like selling to me as being like the panacea to improving time to first token. And a lot of the things that were actually quite helpful were really silly things like, I mean, silly, like increasing the TTL on what we stored in Redis where we could within reason.
1:08:41How do you use LLMs? Because so far, my sense is, talking through your career, you look at a problem, you understand it, you figure out how to do it, you fail sometimes or make mistakes, you learn from it, you learn from others, including right now in the last few years where AI is increasingly everywhere. But as you're making systems more reliable, more performance, it sounds like you're still doing so much thinking, so much. You're now I sense that you're using the LLM on the side to generate some ideas to say, no, this is wrong. No, that was a good idea. OK, let me use it.
1:09:16Maude Lemaire: But unlike a lot of software engineers I talk to, even on this podcast, most of them are saying like, oh, so much of my work that I used to do, the LLM does it. I now can do more things. I can focus on higher. I'm not getting that sense from you. Yeah, I mean, I'm spending all day like chatting with an LLM in some way, shape or form. I think like I still do a lot of the normal engineering stuff, which is like I'm on call. Sometimes I just have to fix a silly bug. But I do tend to I do a lot of sort of preliminary exploration, but then verification manually. And I sense that a lot. Like it sounds like even when you get something from an LM or whatever, you will verify, you will make sure that you understand that this is right.
1:10:02It sounds like you don't have a judgment if it's going to be right or wrong, but you need to verify it, right?
1:10:06Maude Lemaire: Yeah, yeah. Because I think a lot of the problems that I work on are more of the like product platform problems. And the whole point is to solve a problem holistically and to create that foundation that you can build on top of. Right. And if you only built the foundation to cover like 80 percent of the house, like. What's the point? Because that other 20 percent that was not built on the foundation is going to be really shaky and impact the rest of the house that you did build on the foundation. And if you already did that over here, well, like nothing's really stopping you from, say, like tacking on another 10 percent off past the foundation and then that spiraling.
1:10:52Maude Lemaire: Right. And so I think especially especially now that like I think AI is so good at just duplicating things in places because that seems fine and it doesn't have a sense of it's not great at going, you know, but why? But why? But like 20 levels removed, it leans more towards I will just tack on I'll just build an extra bathroom past the foundation. That's fine. Right. Like you asked me to build an extra bathroom. That's where I'm doing it. Rather than questioning the underlying fundamental assumptions, which are, should one build a bathroom not on top of a foundation? And those are the questions that I'm asking myself all the time.
1:11:36Maude Lemaire: And I think part of it is because I'm coming from a product platform perspective. I have a sense that the product platform or the platform or reliably performance, it will only be more important because, again, we'll have systems that have so much load, so much test. What advice would you have for a software engineer who may or may not have much experience, but who is excited about the idea of performance reliability and would like to get closer to eventually work on platform teams? What are paths that you've seen people make it onto platforms that you worked on? How did they transition? I'm such a strong advocate for more product folks, more interchange actually between product folks and product platform teams.
1:12:19Maude Lemaire: I've seen it so many times where product platform teams are spun up out of product engineers who care a lot about the fundamentals, right? They care about reliability. They care about performance already. They're trying to improve that and inject that into the way that they're building product all the time. And then I think the best product platform folks that I've worked with get to a point where they've seen a pattern enough times in the things that they are building as product engineers. And they're like, I want to dig in and solve this problem across the board. Like this specific problem, I understand really deeply from having implemented it two, three, four times.
1:12:56Maude Lemaire: I want to go solve it once and for all and be done and like have this thing that other people can also use. Right. They get annoyed with the toil and then they go and maybe embed with the product platform team or they like spin one up with some other folks and then they do that. And then a lot of times those folks end up staying on product platform teams and continuing to really like the product platform work. But then they get further and further detached from what product development actually feels like. And I think that's where you see sort of the chasm that ends up being created between product platform teams and product teams is just there's not enough cross-pollination between the teams.
1:13:35Maude Lemaire: And so one of the ways that I've seen that chasm merge back together is having like rotation cycles between those teams, sort of always having someone on the product platform team, having a core set of folks who are like maintaining the main systems and platforms that they've built, but sort of always having someone on rotation from a product team on the product platform team, bringing like, oh, well, I don't know what you all are doing here, but like on product, we're just like YOLO shipping all this AI slop all the time. Like what you're reading your code, what? um it's a thing it's a thing we on on product code reviews are are totally dying it's like completely different right but if you're not close to that then you also don't know how to bolster your platform to be the best that it can be so that they can do that i see my role in this you know phase of of the world right now as a product platform person to be like the guardrails need to be better than they have ever been before because product people need, I mean, need is a strong word, but should ideally be able to slop out whatever the heck they want.
1:14:42And the platform should stop them or warn them or...
1:14:46Maude Lemaire: Yes. So do I understand that your suggestion is like, look, if you today are on a product team, you're building products, you're shipping to customers, to users, obviously understand what the business cares about, build, you know, try to build quality stuff, but also ask yourself, like, are you carrying out performance and reliability? If not, first of all, why not? And second of all, as you start caring about this, and if you are working with a platform team, start working with them, hang out with them, see how you can, these problems that you're having, can I help build a solution for that? If you have the option, transfer onto this team, it's pretty useful.
1:15:22And if you are on this team, tried to have a way to work a little bit, like implement the feature with the team, understand why they're shipping so quickly and so seemingly carelessly, because this will, both ways will just make, like, it seems like we need both, especially now that product teams are pushed to ship a lot faster, and they are shipping faster, by the way. And there's new tools. Some of them work, some of them don't. And, you know, this is how we're going to be better engineers and how we will build better systems that allow us to, like, ship reliable and quality experiences, which I'm all for.
1:15:54I hate seeing quality degradations on software.
1:15:56Maude Lemaire: Oh, no, no, I know. And I mean, like we can stop it or we can actually build better software, hopefully. Yeah, yeah, I think so. I think it's it is really similar to for folks who are familiar with like the embedded SRE model. Like there's a reason why that's really, really effective. And I think that there's real parallels with product platform as well. I kept talking to my team about like, I mean, we always have to fight against spaghetti code, right? It's so easy. But now we're at this point where like the spaghetti has like a life of its own and it like actively wants to spill out of the plate.
1:16:31Maude Lemaire: Whereas before people were like, OK, like I got to kind of keep it on the plate. No, no, no. The spaghetti like wants out and we have to almost like red team our own situation. We do. Well, Mo, thank you so much for this conversation. Thank you so much for having me. It was nice to get an insider look of what scaling up looks like inside of products we all know and use, like Slack, GitHub, and Cursor. Until now, I'll admit, I never thought much about custom emojis causing a kind of scale problem. But most example of the CEO sending out a everyone gets a Friday off message to 100 ,000 users was a good visualization.
1:17:07Hundreds of people had a custom emoji, and this causes an event storm of custom emoji binaries sent all across the network and messages. Slack decided to fix the problem with a very simple solution, limit each message to 23 custom emoji reactions. I'm sure they could have done a more proper fix, but who will notice the 24th custom emoji not appearing anyway? It was also fascinating to hear GitHub's limit of 16 shards and why they couldn't add more. It was because of scatter queries, queries with no primary key that had to run on each and every shard before they could return. More shards would have slowed these queries down even more.
1:17:40The solution, of course, is to not have any queries without primary keys. But when you work with the legacy system with hundreds or thousands of queries without primary keys, rewriting those queries takes a non-trivial amount of time, not to mention that even as you're rewriting, you might have new queries added with no primary keys. Finally, it was interesting to hear how, while Maud is clearly AI-pilled, I mean, having worked on GitHub and Cursor, she still verifies pretty much everything by hand. Because when doing platform work, she says that you want to have a stable foundation. She also found that LLMs tend to write tests that just confirm the code written.
1:18:13Mode is now a fan of using mutation tests to have them cover edge cases. Do check out the show notes below for related to pragmatic engineer deep dives on Cursor, GitHub, reliability, engineering, and performance topics. If you've enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube. A special thank you if you also leave a rating on the show. Thanks, and see you in the next one.
Brought to You By:
• Antithesis – verify your system’s correctness without human review or traditional integration tests – and avoid bugs or outages.
• turbopuffer – a vector and full-text search engine built on object storage. It’s fast, cheap, and extremely scalable.
• Entire – The Git forge for agent-human collaboration. Built for the scale and throughput of agentic coding.
—
What is involved in scaling up large, popular products at places like Slack and GitHub?
We get an insider’s view on this from Maude Lemaire, a principal engineer and author of ‘Refactoring at Scale: Regaining Control of Your Codebase.’ She spent nearly eight years at Slack, where she led the backend performance infrastructure team, before joining GitHub to tackle database scaling challenges. Most recently, she worked on improving performance at Cursor – including time to first token.
In this episode of The Pragmatic Engineer, Maude and I dig into the challenges of scaling some of the world's largest software products. We explore how Slack handled growing numbers of enterprise customers, how it evolved its approach to load testing, and the lessons Maude learned during nearly eight years of scaling Slack's infrastructure.
She also tells us what it was like joining GitHub in 2024 when Copilot was taking off, the challenges of working with GitHub's gigantic Rails monolith, and how Maude uses AI in her work – while keeping judgment calls and decision-making firmly in her own hands.
Timestamps
00:00 Intro
03:15 Early years and path to software engineering
07:32 First job at Rent the Runway
09:17 The HipChat data leak
10:54 Joining Slack and the challenges of scaling
26:30 Load testing and thundering herds
32:29 Event floods
38:30 Evolving load testing at Slack
48:38 Joining GitHub in 2024
56:52 The challenges of the Rails monolith
1:02:26 How Maude uses AI
1:12:40 Advice for engineers interested in product/platform work
—
The Pragmatic Engineer deepdives relevant for this episode:
• Real-world engineering challenges: building Cursor
• Pushing software engineering limits with “napkin math”
• What is happening with code reviews?
• Why reliability is hard at scale: learnings from infrastructure outages
• What is Reliability Engineering?
• The Platform and Program Split at Uber
• The present, past and future of GitHub
• Why performant code matters (but gets widely ignored), with Casey Muratori
• Designing Data-intensive Applications with Martin Kleppmann
—
Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].