How AWS S3 is built

17:55Mai-Lan Tomsen Bukovec:Well, you know, I mean, as you know, I mean, you're an engineer yourself. And, you know, as you know, a lot of engineering is about constraints. Right. And that is the fun part about working on S3 is that when you think about constraints, you think about constraints that we have for availability, you think about constraints that we have around the cost of storage, we start to get really, really creative. Okay. And in S3, because, you know, we build all the way down to the metal of S3, you know, of the drives and the capabilities that we have in our hardware, we're able to drive, you know, efficiencies at every single part of our stack.

18:39Mai-Lan Tomsen Bukovec:Okay. And so our engineers, when they get together and they talk about the constraints, they talk about the design goals, we'll do something like we'll set a target for the cost of a byte, and we'll drive for that. And we'll drive for it at every single part of the process. And the part of the process that we are also including is, it includes a data center. How do our data center technicians be able to operate the service of S3 from a hardware and a data center perspective, like the physical buildings, just like we do the same thing for the software and the layers of S3 itself. And when you have that, when you have that ability to run across the whole stack all the way down to the physical buildings, and we're thinking about so deeply about the cost and the lifetime of every byte, you're able to do things like Glacier.

19:37Pragmatic Engineer:You mentioned something really interesting, that when S3 started, it was eventually consistent, which means that data eventually arrives, it might not be there, and you might be behind. And there's a lot of things that you can do with this, and it gives you some constraints. But you mentioned that the reason that the team launched this is because durability and availability was more important. And I assume, of course, cost as well. But during those initial phases, while SD was eventually consistent, what kind of benefits does it give to have eventual because is it a cost constraint? Is it just easier to do high availability systems from an engineering perspective?

20:14Mai-Lan Tomsen Bukovec:Well, I mean, from an engineering perspective, the main optimization was it was availability. It was not necessarily durability, but it was availability, okay? So if you take a step back and look at the original design of S3, we were really focused very hard on availability. So let's take a step back, okay? So when you talk about consistency, It's the property where the object retrieval, the object get, reflects the most recent put to that same object. OK. And so if you think about, you know, what parts of the system of S3 that really hits, a lot of it just kind of starts with our indexing subsystem.

20:55Mai-Lan Tomsen Bukovec:So if you think about the indexing subsystem in S3, that holds all of your object metadata. And so that's like its name, its tags, its creation time. And the index, our index, is accessed on every single get or put or list or head or delete, any API call like that. And so every single data plane request where you go back into our storage system to go get an object goes through our index. And if you think about it, more requests go through our index in our storage system because, for example, it's serving things like head requests and list requests. that don't actually end up going back into our storage system at all.

21:33Mai-Lan Tomsen Bukovec:Those are metadata or index requests. So if you think about our indexing system, we have a storage system in there. And that is a really central concept, a storage system in the middle of our indexing system.

21:49Pragmatic Engineer:So you need a storage system for your index system, right?

21:52Mai-Lan Tomsen Bukovec:That's right. And so we have to configure and size the system to deliver on our design promise for both availability and durability. Okay? And so the data is basically in our index system is stored across a set of replicas, and it uses something called, you know, it's basically a quorum-based algorithm. Okay? And a quorum-based algorithm tends to be very forgiving to failures. And so if you think about how we implemented Quorum in our index system, we start first from servers that are running in these separate availability zones. And the reason we do that is that it lets us avoid correlation on a single fault domain.

22:38Mai-Lan Tomsen Bukovec:Okay. And since the failure of like a single disk, a server, a rack, a zone, it only affects a subset of data. it never affects all of the data for a single object or even a majority of the data for a single object, which we have sharded across a wide spread of servers. So this core of availability for us is this idea that we spread everything. And so when a read comes in, it's coming into the S3 front end, and we just heavily cache objects across our system. When a read comes in, it could route at random and you could create a situation where you're creating an inconsistent read. And so when we have quorum at the index storage layer, we can see reads and writes overlap.

23:26Mai-Lan Tomsen Bukovec:But in the cache, they don't because we're optimizing for availability.

23:30Pragmatic Engineer:So just so I understand, the first part, the eventual consistency, correct me if I'm wrong, that you can just write to all these distributed nodes and you ask one of them. And if it doesn't have it, no problem, because it will be eventually consistent. You now have high availability because you don't need to worry about all of them being in sync.

23:47Mai-Lan Tomsen Bukovec:That is correct.

23:48Pragmatic Engineer:And that's phase one of AWS. And it gives you availability. And now you're now explaining how you're able to, behind the scenes, turn this into a strongly consistent. The strong consistency means that it's guaranteed to have the whole system's state, which is hard to do because you could have distributed failures, et cetera.

24:09Mai-Lan Tomsen Bukovec:And this replicated journal, you know, it took us a while to build. I won't lie. We don't talk about this stuff very, very much. OK, because this is kind of the secret sauce of S3. But, you know, again, like our engineers who are in the room, they were thinking about how do you deliver on both the strong consistency without compromising availability? availability. So I go back to constraints, okay? So in that case, we were not trading off the consistency and availability anymore. And so the engineers had to come up with a new data structure, basically. We do this in S3. Vectors, basically, is a new data structure that we came up with as well.

24:51Mai-Lan Tomsen Bukovec:But, you know, if you think about what we had to invent for strong consistency at S3 scale without relaxing the constraint of availability, is we had to build this replicated journal, okay? And the replicated journal is basically a distributed data structure where we're chaining nodes together so that when this write is coming into the system, it's flowing through the node sequentially, okay? And so a reader write in a strongly consistent system for S3, it flows through these storage nodes in the journal sequentially, and so every node is forwarding to the next node. And when the storage nodes get written to, They learn the sequence number of the value along with the value itself.

25:31And therefore, on a subsequent read, like through our cache, the sequence number can be retrieved and stored.

25:40Mai-Lan Tomsen Bukovec:And so now you have this strongly consistent and highly available capability in S3. And the heart of that is actually this replicated journey.

25:51Pragmatic Engineer:Okay, but what's the catch? On one end, because there's always something with trade-offs. You always have something. So one, you obviously have more complicated business logic. And then I guess the second obvious question is what about failures? Because in the case of eventual consistency, you don't worry too much about one failure. Clearly, in this case, what if a node in the sequence fails either at the first time or later? Or does the system monitor this, recover? Because I guess that's going to be the tricky part, right?

26:20Mai-Lan Tomsen Bukovec:There's another piece to this puzzle that we implemented, which is, you know, it's basically a cache coherency protocol. And the idea is that this is where we built what we think of as a failure allowance, where in this mode, we needed to retain the property that like multiple servers can receive requests and some are allowed to fail. And so it's kind of this combination of this replicated journal as a new data structure. Plus, we implemented this new cache coherency protocol that gave us a failure allowance. And those two things working in concert gave us this strong consistency. I will say, too, this does come at some actual cost.

27:06Pragmatic Engineer:I was about to say, like, nothing is free in engineering, right?

27:09Mai-Lan Tomsen Bukovec:There's hardware cost in this because you can imagine we've done some more engineering behind the scenes. But I remember sitting in a room with our engineers on S3 and we did a debate on this. We debated it. We said, you know, there's costs. There's like actual costs to the underlying hardware for this. And do we pass it along to customers or not? And we made that explicit decision not to. We said.

27:35Pragmatic Engineer:Really?

27:35Mai-Lan Tomsen Bukovec:Yeah, we said that when we launch this, we should launch strong consistency. We should make it free of charge to customers. And it should just work for any requests that comes into S3. We shouldn't sort of say it's only available on this bucket type or what have you. This should be true for every request made to S3. And part of that mindset for S3 is like, how can we provide these type of capabilities? And how can we make it something that becomes a building block, like part of the building block of S3? And you shouldn't have to think about the cost of it.

28:14Pragmatic Engineer:This was the very surprising thing of this launch, by the way, that suddenly AWS said like, okay, everything is strong, existent. It does not cost you more. Latency-wise, your latencies shouldn't have changed significantly. I mean, I'm sure when you roll out initially your measurements, etc. But that was the promise. And that was why I couldn't really believe it when I reread history because it typically doesn't happen. And typically, strong consistency does add latency or it increases costs if it doesn't add latency. There's always this tradeoffs. And I mean, it sounds like you either swallowed the costs or costs caught up, but it's very unusual.

28:51Mai-Lan Tomsen Bukovec:So if I think about that, one of the things that was also very important for us, we haven't really talked about this as much, but we think about it a lot on the S3 team is correctness. OK, so it's one thing to say that you're strongly consistent on every request. It's another thing to know it. And so when we built this strong consistency, you know, I talked about our new caching protocol. I talked about this replicated journal as a new data structure. You know, that took a little bit of time to do and to get right. But at S3 scale, we could not say that we were strongly consistent unless we actually knew we were strongly consistent.

29:31Mai-Lan Tomsen Bukovec:Okay. And so what does that mean? How do you do that at S3 scale when everybody is using it for every last workload? In fact, one of the reasons why people use it is because our scale is such that we're decorrelating workloads and you can run absolutely anything on S3. But how do you know?

29:49Pragmatic Engineer:Mylon just talked about how strong consistency made it so much easier to trust S3. Trust is something that is just as important when writing code, especially when with AI we write more code than before. And this is a good time to talk about our season sponsor, Sonar. What is the impact that AI is having on developers? Let's look at some data. A new report from Sonar, the State of Developer Survey report, found that 82 % of developers believe they can code faster with AI. But here's what's interesting. In this same survey, 96 % of developers said they do not highly trust the accuracy of AI code.

30:21Pragmatic Engineer:This checks out for me as well. While I write codes faster with AI agents, I don't exactly trust the code it produces. This really becomes a problem at the code review stage, where all this AI-generated code must be rigorously verified for security, reliability, and maintainability. SonarCube is precisely built to solve this code verification issue. Sonar has been a leader in the automated code analysis business for over 17 years, analyzing 750 billion lines of code daily. That's over 8 million lines of code per second. I actually first came across Sonar 13 years ago, in 2013, when I was working at Microsoft, and a bunch of teams already use SonarCube to improve the quality of their code.

31:00Pragmatic Engineer:I've been a fan since. Sona provides an essential and independent verification layer. It's the automated guardrail that analyzes all code, whether it's developer or AI generated, ensuring it meets your quality and security standards before it ever reaches production. To get started for free, head to sonarsource.com slash pragmatic. And with this, let's get back to the importance of strong consistency at AWS.

31:21Mai-Lan Tomsen Bukovec:How do you know that you're strongly consistent? And that is why we used automated reasoning.

31:26Pragmatic Engineer:What is automated reasoning? for those of us who are not as familiar with this, which will be most people outside of very few domains like S3.

31:34Mai-Lan Tomsen Bukovec:Yeah, it's, I mean, S3 uses automated reasoning all over the place, okay? And automated reasoning is a specialized form of computer science, okay? And Gurley, if you kind of think about if computer science and math got married and had kids, right, it would be automated reasoning.

31:50Pragmatic Engineer:Is it formal methods or based on formal methods?

31:53Mai-Lan Tomsen Bukovec:That's exactly right.

31:54Pragmatic Engineer:I mean, I studied computer science, so yeah, that's fun. So it's actually proper formal methods that you're using.

32:00Mai-Lan Tomsen Bukovec:That is right. And we use formal methods in many different places in S3, but one of the first places that we adopted was for us to feel good that we actually had delivered strong consistency across every request. So what we did is we proofed it, right? We basically built a proof for it, and then we incorporated our proof on check-ins into this index area that I talked about, right, where you have your caching and then you have your storage sub layers of the index capabilities. And so when somebody, anybody is working on our index subsystem now, and they're checking in code into the code paths that are being used for consistency, we are proofing through formal methods that we haven't regressed our consistency model.

32:46Pragmatic Engineer:And can you just give us a rough idea? Because the formal methods that I have studied, they were pretty abstract. The things like designing languages, how to have like the different operators and of course there are some maths involved as well but what are the are they like primitives like servers network etc and models being built data flows like how how can i imagine a a simple proof of of something yeah inside s3 roughly at

33:15Mai-Lan Tomsen Bukovec:a really high level yeah i mean if you if you go back to the fundamental notion of a proof you are proving something to be correct, okay? And so the places that we use these proofs, we use them in consistency, where we built a proof across all the different combinatorics to make sure that the consistency model is correct. We use it in cross-region replication to prove that a replication of data from one region to another arrived. And we use it in different places within S3 to prove the correctness of API. In all of these cases, you know, we talk about durability, we talk about availability, we talk about cost, but just as strong of a principle, a design principle for us across S3 is correctness.

34:01Mai-Lan Tomsen Bukovec:It's a correctness of, you know, a thing, an API request, you know, an operation, as it were. And the key thing for us too is that you You don't want to just proof it once. You want to proof it on every single check-in, and you want to proof it on every single request so you can verify, you can validate and verify that you are doing, in fact, what you say you do. And I think for us, you know, at a certain scale, math has to save you, right? Because at a certain scale, you can't do all the combinatorics of every single edge case, But math, math can save you and help you on this at S3 scale. And so we use formal methods in many different places of S3.

34:48Mai-Lan Tomsen Bukovec:We have some research papers, too. I can send you some links to some research papers where you talk about that.

34:53Pragmatic Engineer:Yeah, please do. And we will put it in the show notes below so anyone can check it out because I think it's really interesting. I feel formal methods are not really a thing in a lot of startups and even infrastructure startups yet. but it sounds very reassuring to me to actually have an ongoing proof of that and speaking of which i i want to ask about one thing that is related to this durability uh amazon s3 has very very like high durability promises i think it's 11 nines which i i had to like do a double check on because in in uh backend systems whenever you say three nines it's like when you say four nines of availability we're not talking durability availability four nines is already hard to achieve and beyond that it just gets very expensive and i have never heard of 11 nines of durability now this is durability and not availability one question that i i got when i when i shared this that uh publicly what people one thing people were asking and i was also thinking how can you prove that not just in a formal way but you're now storing as you said 500 trillion uh objects which is now large enough that just by this durability promise you should be you might be losing some of them.

36:05Pragmatic Engineer:Do you actually validate it on the actual data as well outside of the proof? Because I assume in the proof you will have assumptions on hardware failure rate, which might or might not be true. So my question is that at Amazon S3 level, when you are able to look at the, are we living up to, for example, our durability promise, how do you go about that? And what are your findings?

36:26Mai-Lan Tomsen Bukovec:Yeah, so we just spend a lot of time talking about our index subsystem because that is the subsystem that is related to consistency. But when you think about durability, I mean, you think about it all, you know, on different levels of the S3 stack, but we really think about it in the storage layer. And so if you think about it in the storage layer, you have this design, this promise of, you know, the design here. And underneath that is a combination of things. It's software, but it's also the physical layout of where our data is across everything that we have in S3. And, you know, one of the things that I talked about is that we have, you know, disks and servers which sit in racks, which sit in buildings, and we have tens of millions of these hard drives.

37:11Mai-Lan Tomsen Bukovec:We have millions of servers, and we have 120 availability zones across 38 regions.

37:17Pragmatic Engineer:Yeah, and one availability zone is, like two availability zones are two physically separate locations, just to be clear.

37:23Mai-Lan Tomsen Bukovec:They're physically separate, and sometimes they're a ways away from each other. And in some of our regions, we have more than three available. I mean, the availability of loans gives us a different domain, a fault domain. If I were to think about durability, I think the most important thing for us is our auditors. So you think about a distributed system. We talked about the put and the get. We have many, many, many microservices that are all doing one or two things very well in the background. OK. And so we have many different varieties of health checks. But we also have repair systems and we have auditor systems.

38:00Mai-Lan Tomsen Bukovec:And our auditor systems go and they inspect every single byte across our whole fleet. And if there are signs that there is repair needed, you know, another repair system will come in place. And these are all, you know, in the world of distributed systems, these are all microservices working together, loosely correlated, but communicating through well-known interfaces. And so that, you know, collection of systems, which are over 200 microservices now, that all sit behind one S3 regional endpoint, and a fair number of those subsystems, those microservices, are all dedicated to the notion of durability.

38:36Pragmatic Engineer:So they will go and check and log and report back. So do I understand correctly that in any given time frame at S3, someone or some people or some systems can actually answer the question of what is our durability the past week, month, year and so on?

38:54Mai-Lan Tomsen Bukovec:Yes.

38:55Pragmatic Engineer:OK, great. So you can actually verify your durability promise that. Check if the math is mathing.

39:01Mai-Lan Tomsen Bukovec:Yes. And, you know, part of our design is that at any given moment in this conversation that you and I have had, just today, we're having servers fail because servers fail. And so what we are building and what we've built in S3 is an assumption that servers fail. And so a lot of our systems are always, you know, they're first of all, they're they're they're checking to see, you know, where any failure might hit an individual node. How does it affect a certain bite? What repair needs to automatically kick in place? And so this system is constantly moving behind the scenes, if you will, while and that is a completely separate thing from the get in the put.

39:40Mai-Lan Tomsen Bukovec:Again, the put is what the customer sees. There's this whole universe under the hood of how do we manage the business of bytes at scale.

39:49Pragmatic Engineer:I'm just thinking because for a lot of us engineers who are building like moderately sized systems, I'll say compared to S3, they can already be big. But a failure is a big deal. Like, you know, like a machine going down again. I have a small side project and my storage filled up and I started to give errors. And this is a big deal because it rarely happens to me. This is the first time it happened in three years. But I understand in your business or when you work at S3 scale, this is just every day. And the question is not when, it's just how often, how do you deal with it? I guess it's a different world.

40:23Mai-Lan Tomsen Bukovec:It is a different world. And the trick is to really think about correlated failure. OK, so if you're thinking about availability at any scale, it's the correlated failure that'll get you.

40:36Pragmatic Engineer:And what is a correlated failure?

40:37Mai-Lan Tomsen Bukovec:Okay, so that's super interesting. So if you think about what I talked about with, you know, eventual consistency, we talked about quorum, okay? And quorum is okay for one node to fail, but if all of the nodes go south, for example, and they're in the same availability zone or on the same rack, then you're really going to be messing with your availability of the underlying storage, okay? You just lost your failure allowance that I talked about with the cache because they all fail together. And so like a correlated failure is an incredibly important thing to think about when you're thinking about availability.

41:11Mai-Lan Tomsen Bukovec:And so when we're designing around correlated failures, the thing is that we have to think about is like, do we expose or how are those workloads exposed to different levels of failure? So when you upload an object to S3 with a put, we replicate that object, okay? We don't just store one copy of it, we store it many times. And that replication is important. It's important for durability, but what's interesting about it, it's also important for availability. Because if any of those correlated failure domains fail, like if a whole AZ fails, there's still a copy somewhere else. And the data is still available somewhere, even though an availability zone has failed or a rack has failed or a server has failed or so forth.

41:57Mai-Lan Tomsen Bukovec:Okay? And so that idea of how do you manage and design around correlated failures with both our physical infrastructure as well as our logical infrastructure is super important for S3 for both availability and durability. We also do things like we think about something called crash consistency. I mean, Greg Lee, you can tell I can go on and on about this. So you just have to stop me.

42:22Pragmatic Engineer:No, but this is the interesting stuff.

42:25Mai-Lan Tomsen Bukovec:All right. So the whole idea of crash consistency is that a system, any system that you build, it should always return to a consistent state after a fail stop failure. And if you can do things like reason about the set of states that a system can reach in the presence of failure, and you just always assume the presence of failure, then you also assume the presence of consistency and availability, then you just design all of these different microservices to all work together in an underlying capability like S3. But that's what our engineers do. They think about like crash consistency. They think about correlated failures.

43:07Mai-Lan Tomsen Bukovec:You know, they think about failure allowances and caches. Right. And it's it's all that deep distributed system work that that our engineers come in every day to work on.

43:18Pragmatic Engineer:can we talk about how you think about failure allowances because again there there is a concept of error budgets outside in other companies as well i feel it's a bit like loosely handled whereas i feel this is kind of your bread and butter so what is a failure allowance and how do you measure it and what do you do if you if you overstep it or overspend it yeah i mean

43:39Mai-Lan Tomsen Bukovec:i think that the idea of a failure allowance is want to have it like you have to have it if you assume for no, you know, that you'll never have a failure, you will, you'll actually have a very bad day for your customer. And so we account for failure allowances. And but the most important thing is, let's just talk about the failure allowance in our cache. So how do we manage that? Well, we manage it in such a way that you'll never experience it because we size it, right? And if you're sizing the cache, and you're making sure that the underlying capabilities in the hardware are always there. And we have, like I talked about those distributed subsystems, those microservices that are all in our operating under the hood.

44:18Mai-Lan Tomsen Bukovec:We have a ton of them that do nothing but just track metrics, right? And like, you know, the sizing of our cache is all related to the metrics and the size of our underlying system.

44:32Pragmatic Engineer:All the metrics, yeah. Yeah, that's right.

44:34Mai-Lan Tomsen Bukovec:And so one of the really big benefits of running on S3 is because our system is so huge, you have these massive, you know, layers, right? And the massive layers are all managing things like correlated failures and failure allowances. And because they are so huge at the scale of S3, any application that's sitting on top of S3 gets the benefit of it.

44:57Pragmatic Engineer:Let's take a break a minute from S3 to talk about a one-of-a-kind event I'm organizing for the first time, the Pragmatic Summit in partnership with Statsig. Have you ever wanted to meet standout guests from the Pragmatic Engineering podcast, plus folks from cutting-edge tech companies and learn about what works and what doesn't in building software in this new age of AI. Come join me 11 February in San Francisco for a very special one-day event. The Pragmatic Summit features industry legends and past podcast guests like Laura Tacho, Kent Beck, Simon Willison, Chip Huyen, Martin Fowler, and many others.

45:29Pragmatic Engineer:We'll also have insider stories on how engineering teams like cursor, linear, open AI ramp, and others built cutting-edge products. We'll also have round tables and carefully curated audience where everyone and everyone is interested to meet and chat with, something I'm hoping will make this event extra special. Seeds are limited, and you can apply to attend at pragmaticsummit.com. Talks will be recorded and shared, and paid subscribers will get early access afterwards as well, as a thank you for your additional support. I hope to meet many of you there, and I am so excited about this event. And now let's jump back to S3 and the massive scale of the service.

46:03Pragmatic Engineer:To get a sense of what the reality is like working as an engineer, an engineering leader inside an organization like this, I read a quote from a distinguished engineer, Andy Warfield, who said, I'm just quoting what he said, early in my career, I had this sort of naive view that what it meant to build large scale commercial software, that it was basically just code. The thing I realized very quickly working on S3 was that the code was inseparable from the organizational memory and the operational practices and you know the scale and the scale of the system. Since you've now been more than a decade in S3, how do you think of this beast, this really complex system, hundreds of microservices, data that is hard to fathom, you know, unless you think of the hard drive stacking all the way to the space station?

46:50Pragmatic Engineer:And how do you in Nigeria kind of wrangle this? Because it does feel a bit intimidating. I'm not going to lie.

46:56Mai-Lan Tomsen Bukovec:Well, I think so much of this just comes back to the culture and the commitment on the team. And yeah, I've worked on S3 for a very long time now. And I have such deep respect for the engineering community on S3. And, you know, honestly, I mean, this is true for all the services in our data and analytics stack, but we have engineers on S3 and they come in every single day with this deep commitment to the durability and availability and the consistent of your bite. And so the type of conversations that we have are so interesting because we have people and really, you know, these are people who are early out of school.

47:38Mai-Lan Tomsen Bukovec:They're people who've been working on S3. We have engineers who've been working on S3 for 15 years and everything in between. mean, the creativity and the invention of S3, like you have this tension, which is like on one side, you're like, you have to be very conservative with S3, right? And on the other hand, like, I mean, we have this principal engineering tenant called respect what came before. And that's an Amazon engineering tenant, which is if it has worked for many, many years, you have to respect that. But then there's also this tenant. These two tenants are a little bit in tension with each other, which is kind of what makes it so fun.

48:14Mai-Lan Tomsen Bukovec:Amazon engineering tenant is called be technically fearless. And I believe that the S3 engineers are just amazing at this, at respecting what came before. Because if we build new capabilities in S3, we have to maintain the properties, the traits of S3, which is it just works and you get that durability, availability, etc. But at the same time, we have to be technically fearless because our ability to go into the world of conditionals. Our ability to go into the world of native support for iceberg or for vectors means that we are extending this foundation of storage in a way that helps customers build whatever application they need now and in the future.

48:57Mai-Lan Tomsen Bukovec:And so that combination of the two things, that is sort of, when I think about our Estree engineering team, I think they come in every day and they embody that.

49:07Pragmatic Engineer:Now, going back to the evolution of Estree from unstructured to structured data. You were mentioning how Hadoop, the data warehouse, was a big use case where customers start to use it on top of S3. And then at S3, you noticed what a lot of customers or some of your biggest customers doing. And then you kind of built it yourself with more structured data. And then S3 tables came along and then vectors. Would you mind sharing a little bit more on how you evolve S3? Because this was another question that when I asked people about what they'd like to know about S3, one of the questions was like, is it done?

49:42Pragmatic Engineer:Is it finished? Or is it still evolving? Because there is this notion that S3 can store anything already, right? Like any object, any blob. What new thing is there? And yet we have a lot of new things.

49:54Mai-Lan Tomsen Bukovec:Yeah. And if you kind of go back in time a little bit and you think about, you know, the rise of Parquet. Okay. So the rise of Parquet data in S3 started about 2020. And we started to see more and more people store their tabular data in S3. And if you think about what Iceberg provided, it provided a replacement for Hive. Okay. So if you think about Hive and Hadoop, Hive was basically giving your file system access into S3 unstructured storage. Iceberg is giving that iceberg that tabular access, including the, you know, the compaction and all the table maintenance that goes along with it into your parquet data.

50:36Mai-Lan Tomsen Bukovec:And I actually think that the world's data for tabular data is going to live in the future in S3. And if you just think about the launch that, for example, Supabase did last week, Supabase announced that their Postgres database is just gonna do secondary writes directly into an S3 table, just like their Postgres extension for vector is going to integrate directly with S3 vectors. And so if the world of database, if the world is data as a source, if you will, goes directly into an S3 table, what does that mean for the world's data? Okay, so SQL, as we know, is the lingua franca of data. And the world's LLMs have all been trained on decades of SQL.

51:24Mai-Lan Tomsen Bukovec:And Python, SQL and Python. And Python and the stuff that's already out there. And so if you think about this, we have many, many AWS customers who know the S3 API pretty darn well by this point. It's a pretty simple API. But now you have the ability to interact with data in S3 through SQL. And what that means is that you don't have to be somebody who's building cloud applications or know S3. You just need to know SQL.

51:52Pragmatic Engineer:And this is with S3 tables, right?

51:54Mai-Lan Tomsen Bukovec:With S3 tables. And so you can just write SQL into an S3 table. And whether you're an AI agent or a human, right, you're introducing the lingua franca of data as a native property of S3 with S3 tables. And I think you're just going to see that take off in the upcoming years.

52:15Pragmatic Engineer:And your latest launch is S3 Vectors. Can you share a little bit what it takes to build a new data primitive like Vectors? Just behind the scenes, how long it takes, how the seams come together, and maybe what are some engineering challenges of launching something like this? And again, we're talking about Vectors, right? So like you can use embeddings whenever you have LLMs, you create an embedding, it's a Vector, you want to store that somewhere, you will need to search on it, there's specialized Vector databases, there's specialized Vector additions, etc. So I'm assuming this is the function that S3 Vector supports very nicely.

52:50Mai-Lan Tomsen Bukovec:Yeah. And, you know, I mean, today a lot of customers use vector databases, just like back in the day. A lot of people put their, you know, their tabular data in just databases. OK. And they just use the structure of the database in order to, you know, take advantage of being able to query their data. But they didn't really need to use a database. They just put it in a database. and then S3 came along, and then we introduced this way, you know, with the help of open formats like Apache Parquet, of being able to store that structured data in S3. That's kind of what we're doing with vectors right now, okay?

53:29Mai-Lan Tomsen Bukovec:And if you think about vectors, vectors are basically a bespoke data type. A vector, at the end of the day, is a very, very long list of numbers, and vectors have been around for a long time and they've been in vector databases for a while but they really kind of took off in people's you know data worlds in the last couple of years with the rise of as you said the embedding models okay and so if you take a step back and you think about one of the great ironies of data it is that you have to know your data to know your data right you have to know what your schema is you have to know what the data types are you have to know where it is And as these data lakes become data oceans, you have this situation where it gets harder and harder to know what's in your data, right?

54:18Mai-Lan Tomsen Bukovec:And the beautiful thing about embeddings is that embedding models will understand your data so that you don't have to understand your data. And the format that these embedding models puts this semantic understanding of your data is, in fact, a vector. And so when we talk to customers and we, you know, they're so excited about how these embedding models are getting better and better, they want to apply more and more basically semantic understanding to their underlying data, whether it's unstructured or structured, that they have in storage. And so they kind of want to store billions of vectors.

54:57Pragmatic Engineer:But just to say, when you say they won't understand, to correct me if I'm going to hypothetically, you have a bunch of text data or maybe some image data, and you're saying that a lot of people, customers, teams, they would like to write queries to say, like, hey, can you find an image that looks like a puppy? Or can you find an article that contains this or this? And embeddings, as we know, are great for that, but then you need to actually create the embedding, build a system, et cetera, right?

55:23Mai-Lan Tomsen Bukovec:Yeah. And, like, exactly what you're saying. I mean, if you think about what vectors can do, if you think about all the data that a given company has, you know, your knowledge across your business or your knowledge across your life isn't organized into rows and columns like a database. It's in PDFs. It's in your phone, right? It's in audio customer care recordings, which capture the sentiment of how a customer actually feels about their interaction with you. It's whiteboards. By the end of this day, this whiteboard is totally filled up with ideas, and it's in documents across dozens of systems.

56:03Mai-Lan Tomsen Bukovec:And so it's not that you don't have data. You have tons of data. But understanding what data you have across all of those different formats is a real problem, and it's one that AI models can help you with. And so the capabilities of those AI models have gotten so much better in the last 18 to 24 months. But we needed a place to put billions of vectors, billions of, you know, the semantic understanding of relationships. And that's what we built S3 for. The state-of-the-art embedding models combined with the ability to have vectors across S3 is like a really important part. And it's not a database.

56:48Mai-Lan Tomsen Bukovec:I mean, it's the cost structure and scale of just S3, but it's four-vector storage.

56:54Pragmatic Engineer:And then do I understand that, did you need to build new primitives to store this? Like going down to the metal, figuring out exactly where we do this, or did you build it on top of your existing, you know, like again, existing primitives of like blob storage, et cetera?

57:08Mai-Lan Tomsen Bukovec:It's actually a new primitive. And so, you know, we had talked about S3 tables. S3 tables is building on objects because those individual parquet files, at the end of the day, they're an object. Vector is totally different. So with vector, we built a new data structure, a new data type. And it turns out that when you're building vectors, searching for the closest vector in a very high dimensional space, which is basically vector space, it's often really hard to find the nearest neighbor. And so you basically, in a database, you have to essentially compare every vector in a database. And that's often like super expensive.

57:51Mai-Lan Tomsen Bukovec:And so what we do in S3 is because we aren't storing all of our vectors in memory, we're storing it on our fleet of S3, a very large fleet, we still need to provide a super low latency. And in our launch last week, we were getting about 100 milliseconds or less for a warm query to our vector space, which is actually pretty fast. It's not database fast, but it's pretty fast. And the way that we do that is we pre-compute a bunch of, think of them as vector neighborhoods, okay? And so it's basically a cluster, a bunch of vectors that are clustered together in similarity, like, you know, a type of dog as an example.

58:35Mai-Lan Tomsen Bukovec:These vector neighborhoods, if you will, they're computed ahead of time offline. They're computed ahead of time asynchronously so that when you're doing your query, it's not going to impact your query performance. And then every time a new vector is inserted to S3, the vector gets added to one or more of these vector neighborhoods based on where it's located. And so when you are executing a query on S3 vectors, there's a much smaller search that's done to find the nearest neighborhoods. and it's just the vectors and the vector neighborhoods that are loaded from S3 into a fast memory, that's where we apply the nearest neighbor algorithm, and it can result in like really good sub 100 millisecond query times.

59:22Mai-Lan Tomsen Bukovec:And so, you know, if you think about the scale for S3, we'll give you up to 2 billion vectors per index. You think about the scale of a S3 vector bucket, which is up to 20 trillion vectors, and you think about that combined with 100 milliseconds or less for warm query performance, that just opens up what you can do with creating a semantic understanding of your data and how you can query it.

59:48Pragmatic Engineer:It sounds very interesting and also challenging because you had to build this for scale from day one. I guess that's one of the, I guess, benefits and curses of working at S3, that everything that you launch, you need to prepare for what will be extreme data elsewhere. But here, it's just Monday.

1:00:02Mai-Lan Tomsen Bukovec:We have S3 service tenants as well. And one of the tenets and one phrase that I use all the time and our engineers do too is scale is to your advantage. So if you are an engineer and you think about that and you think about one of your tenets for anything you build is that scale must be to your advantage. It just changes how you design. It means that you can't actually build something where the bigger you get, the worse your performance gets or the worst some attribute gets. It has to be constructed so that the bigger you get, the better your performance gets. The bigger S3 gets, the more decorrelated the workloads are that run in S3.

1:00:47Mai-Lan Tomsen Bukovec:That is a great example of scale is to your advantage. And so when we built vectors, just like we build everything in S3, we ask ourselves, how can we build this such that scale as to our advantage. How can we build this such that 100 milliseconds or less is just the start of the performance that we're going after? And how can we make sure that the more vectors we have in storage, the better the traits of S3 for vector?

1:01:13Pragmatic Engineer:I have a different question about the limitations of S3. I read that the largest object you can store in S3 is 50 terabytes. Why is there a limit on the largest object? I mean, I think we can imagine this will be through either multiple hard drives and so on. But why did you decide to have a limit? You know, I'm just interested more in the thought process of how the team comes up with like, OK, this will be the limit and this is why.

1:01:39Mai-Lan Tomsen Bukovec:I mean, I think, first of all, that limit of 50 terabytes is 10 times greater than what we launched with. We launched with five terabytes and now we're 50 terabytes. And sometimes we sit and tell customers that and they go, what am I going to store that's going to be 50 terabytes? And we're like high resolution video. Right. And so, you know, if I think about customer. Right. And so if you think about this sort of thing, you know, like if you think about, I don't know, size, size limits. Generally speaking, we we do try to optimize for certain patterns. and when you raise the size of an object by 10 times like we did, we're just optimizing for the performance and scale of the underlying systems.

1:02:24Mai-Lan Tomsen Bukovec:It's like we increased the scale of our batch operations by 10 times last week too and the idea behind that is that the underlying systems, we're just optimizing for distributions of work that are the new norm for how people are doing things. And we'll just keep on changing. We don't have too many limits, to be honest, but we'll just keep on, you know, looking at what customers are doing across the distribution of workloads and seeing if there's something that that needs to be changed. The big thing for us, you know, again, we did have a lot of conversations with customers and they're like, really?

1:02:59Mai-Lan Tomsen Bukovec:Like, I don't have that many individual objects that are that big. but with the increase of you know cameras and phones and things like that we are seeing more larger size objects and we just wanted them to be able to grow unfettered in s3 and so

1:03:17Pragmatic Engineer:how does s3 evolve and how is the roadmap changed because so far what i picked up is everything you told me is saying well you know our customers were doing this or that and you obviously hear you live and breathe data so you see the patterns you see stats you see the objects you also talk with them. Is it only you talking with customers, seeing what's happening, what's what they're struggling with, what they're using more of, and then deciding to improve that? May that be the limits? May that be figuring out we need a new data type because they're now building their own data types on top of it?

1:03:50Pragmatic Engineer:Or is there also some some kind of more kind of, all right, here's a vision, here's a roadmap of what we'll do?

1:03:56Mai-Lan Tomsen Bukovec:It's a great question. And in fact, one of the things that we talk about all the time is the coherency of S3, right? And so there are certain things that people always expect from S3. It's the traits of S3. It's the durability availability attributes that we talked about. And so a fair amount of engineering goes on under the hood for that, okay? And it's a set of capabilities that, you know, we may or may not have talked about today. In fact, if you think about, I think, back to 2020, I think we've launched over a thousand new capabilities since 2020 in S3. And some of them are what we think of as the 90 % of the roadmap, which is what people ask for explicitly.

1:04:42Mai-Lan Tomsen Bukovec:Okay. And so, for example, you know, some of our media customers want the bigger object size. And so we delivered that. We have other customers that do a lot with batch operations. But then we have some things that we invent because, you know, we look at what customers are doing with the data and we ask ourselves, how can we build that? Vector kind of falls into that category. For Vector, when we looked at S3 and how S3 is evolving, we told ourselves like, look, you know, we can continue to make S3 the best repository for data on the planet And we will. We will. We have engineers that come in every day working to make that so.

1:05:20Mai-Lan Tomsen Bukovec:But there's this other element of how do you make sure that the data that you have is in fact usable? And how do you make sure that it's usable in a way that's, you know, industry standard, like that iceberg layer on top of our tabular data? but it's usable because AI models have now gotten so good at embeddings that you can have AI give you a semantic understanding of your data if only you had the cost point of putting billions of vectors into storage so you could actually understand and use your data in a different way. And so for us, a lot of it is kind of taking a step back and looking not just at what customers ask us for, but we want to remove the constraint of the cost of data, which is what we do in S3.

1:06:08Mai-Lan Tomsen Bukovec:And we want to remove the constraint of working with your data, which is what we do in S3 too. And when we can do both of those things, if we can make it possible that your data grows as your business needs it, and you can tap into all the capabilities that you're getting with AI and how the world is changing for data, then we have a shape. We call it a product shape. Then we have a product shape.

1:06:34Pragmatic Engineer:What's a product shape?

1:06:35Mai-Lan Tomsen Bukovec:It's sort of like an emerging, like when I think about S3, I think of it as almost like this living, breathing organism where the shape of the product is evolving, but it's evolving with coherency around what you expect for the traits of S3, but it's evolving in a way that lets you steer into how you want to use data. And how do you want to use data, not just now, but in the future? And we will continue to evolve the product shape of S3 based on what you want to do with data. And so in a lot of ways, we're sort of transcending the boundaries of what object storage was, or what a database traditionally was, because now we have tabular formats, we have conditionals, and we're evolving into this new shape.

1:07:22Mai-Lan Tomsen Bukovec:And it is ultimately uniquely S3.

1:07:26Pragmatic Engineer:It kind of sounds like you have all these microservices. It's kind of evolving almost like a plant or a living organism, no?

1:07:32Mai-Lan Tomsen Bukovec:Yes. I am, in fact, a former Peace Corps volunteer from forestry. And so, you know, a lot of times I will go back to the natural world for my metaphors. and yeah I mean S3 is this living breathing repository of data that lets people do things with data that they never thought possible.

1:07:52Pragmatic Engineer:It's just interesting because I think as engineers we don't often think to relate the systems that we build with like a living organization when in fact I mean obviously there's code but as you said there's people there's servers there's failures that now happen at a cadence you can humble it's just you can probably predict how many hard drives are failing today, in fact, at your scale already, which, again, maybe it's, do you think it's because of the scale when things become large enough, they start to have these characteristics? Because what I find fascinating talking to you is the way engineering works inside of S3 feels very different to how it works inside a smaller organization, your kind of startup, which again does, you know, like terabytes of data or maybe even a few petabytes, but that's kind of it.

1:08:39Pragmatic Engineer:And you've seen some of these organizations. what changes at this large scale? What do you think that makes it? It feels pretty different, the world that you and the S3 teams work in.

1:08:50Mai-Lan Tomsen Bukovec:It does, but in order for us to sustain the traits of S3 and to evolve it over time, we have to constantly go back to simplification. We have a very complex system with all of our different microservices, but I kind of go back to those microservices have to do one or two things really well. And we have to stay true to that. Otherwise, you know, the complexification of a distributed system, you know, it's unmaintainable over time. And for S3, this concept of, okay, there's a simple in S3. And the simple in S3 is a couple of things. One, it's a simplicity of the user model, where not only do you have a simple API, but now you have the simplicity of using SQL with S3, or you have the simplicity of being able to leverage these AI embedding models, which makes semantic understanding of your data so much easier than having to annotate a whole metadata layer.

1:09:51Mai-Lan Tomsen Bukovec:And so that concept of simplicity is in the user model of S3. But under the hood, if you are sitting on any of our engineering meetings, you will hear our engineers talk about how do we make sure that we implement this capability with the greatest simplicity that we possibly can.

1:10:09Pragmatic Engineer:Speaking of which, what type of engineers do you typically hire to work at S3 in terms of what kind of traits, potentially past experience do you look for?

1:10:19Mai-Lan Tomsen Bukovec:Well, we hire all kinds of engineers. You know, we have a lot of engineers on S3 who are early career. They're straight out of school or they're at a, you know, undergrad or graduate school. And like I said, we have like a ton of engineers who have been on S3 for a long time and everything in between. I think there's a really strong element in our teams that work on data around ownership. It's, you know, people feel this like personal sense of commitment. I feel it. I feel it every day I come in, where I feel a personal sense of commitment to your bite, to the preservation of your bite, to the usefulness of your bite, to the ability for you to think about what your application does next and not the types of storage that you need or how you grow it.

1:11:10Mai-Lan Tomsen Bukovec:And that deep sense of ownership and that deep sense of commitment is a very, very common thread across our data teams because we know that at the end of the day, every modern business is a data business. And everything that people are trying to do with traditional systems, AI, whatever, is based on your data as shaping the core of your application experience. And so that data is our responsibility and we feel it very deeply.

1:11:42Pragmatic Engineer:And what would your advice be to, let's say, a mid-career software engineer, someone who has a few years of experience working at different places, who is actually, after listening to this, gets really enthusiastic and decides, like, one day I'd love to work on a deep, strong infrastructure team like S3. or like, let's say, like more experienced folks, what are experiences, activities that you might look for that might help you consider these folks more?

1:12:09Mai-Lan Tomsen Bukovec:There's a strong value in relentless curiosity. Okay. And, you know, I talked a little bit about coloring within the lines and how when you work on S3 or a large scale distributive system, which continues to reinvent what storage means, you're not really coloring within the lines. You're just kind of looking, you're taking a step back and you're saying, you know, I will draw what the lines are today and I will know that I might have to rub those out and draw new lines in the future for wherever things go. And so, you know, I have three kids who are in university. I have two kids in university and one in grad school.

1:12:45Mai-Lan Tomsen Bukovec:And that is one thing that I, you know, I think is really important is to always take a step back, take a look at the latest research. And some of the papers that I'll share with you are around how we, you know, we either took formal methods and we brought them into storage systems, right? Or we thought about failure in a different way, where that creativity, that relentless curiosity and that creativity with engineering, I don't think you can go wrong with that. I think the next generation of software, no matter if it's built in S3 or elsewhere, it is all driven by the creativity of the engineering mind and it is in all of us we just have to kind of unlock it and unleash it and we will brand we'll build amazing things like s3

1:13:32Pragmatic Engineer:and i also love that with s3 not only has s3 created something that did not exist and i think like it just was unimaginable because it didn't exist but now i'm hearing startups that are building on top of s3 i think turbo puffer is a good example you know they're building innovation because now they have a base layer. And I feel there's different levels of innovation. You decide where you want to innovate at the very lowest level, one level higher, and so on. And you just use the right primitives, right? In your case, this is just doing hardware and storage better than anyone. In the other layers, it will be using the right primitives better than anyone.

1:14:08Mai-Lan Tomsen Bukovec:Yeah, it's very exciting for us to see so many different types of infrastructure built on S3 now.

1:14:14Pragmatic Engineer:And as closing, what is a book or a paper that you would recommend reading that you enjoyed and why?

1:14:21Mai-Lan Tomsen Bukovec:I read a lot of different papers. I am fascinated by how quickly the evolution of embedding models are coming along now. And in particular, a field of science that I'm quite interested in is the multimodal embedding model. Because as you know, the world that we experience is multimodal. And therefore, the understanding that we have of data should be multimodal as well. And so there's this whole field of science that's emerging quite rapidly around multimodal embedding models. And so that is something that I encourage people who are working in the field of data to look at, because I think that is the next generation of data.

1:15:00Mai-Lan Tomsen Bukovec:If you think about, you know, the next world of data lakes, I think it's actually going to be on metadata. It's going to be on the semantic understanding of our data. and understanding how that is created through vectors and how it's being searched and done across multiple modalities, I think is an important area of both research and advancement. And so that's what I would encourage people to look at in the world of data. I think vector is going to be quite big, particularly at the price point that we've introduced for S3 storage for vectors. And I'm excited about it. I think, you know, I think we're just getting started.

1:15:40Mai-Lan Tomsen Bukovec:with data and an understanding of our data. And I can't wait to see what comes next.

1:15:45Pragmatic Engineer:Amazing. And do you have any book recommendations?

1:15:48Mai-Lan Tomsen Bukovec:I will give you a book recommendation, just in case your readers are interested. It won't be in the field of computer science. It will be about the evolution of the ecology around us and supporting the bees, the native bees and insects around us. So a tiny bit farther afield, but I'll give you a book recommendation and if your readers are interested they can take a look at how to support the bees of the planet

1:16:16Pragmatic Engineer:Well, Myline, thank you very much This was fascinating and very interesting to get a peek into this massive world of scale of data and respecting the bite and treating it and making sure that it's durable

1:16:29Mai-Lan Tomsen Bukovec:It was great talking to you and thank you to both yourself I know you're a fan of Esri and to all of your listeners who use Esri We quite literally wouldn't be able to do what we do without the feedback and the encouragement from everybody who uses S3 today. So thank you for that.

1:16:48Pragmatic Engineer:Just wow. I always suspect that there's a lot of complexity behind a system like S3, but I just did not realize the scale of it. Whenever I worked on a systems with even hundreds of virtual machines, failure of one machine was a rare event and not something that we really counted on. During my conversation with Mylon, she casually mentioned that several machines have failed during our conversation, which is something that the S3 knows and prepares for, and treats it like an everyday event. I personally really liked how AWS has two conflicting tenants heavily used on the S3 team, respect what came before, and technically fearless.

1:17:21Pragmatic Engineer:For such a massive system, it will be easy to say, let's move conservatively because of how many companies depend on us, but if they did so, S3 would fall behind. Finally, I'm still in awe that AWS put strong consistency in place, rolled it out to all customers and did not increase pricing, nor did they increase latency. At S3 scale, this is an absolutely next level engineering achievement. In fact, it was probably one of the lesser known engineering feats of the decade. I hope you found the episode as fascinating as I did. If you'd like to learn more about Amazon and AWS, check out the exclusive deep dive I did with AWS's incident management team on how they handle outages in the show notes below.

1:17:58Pragmatic Engineer:In the Pragmatic Engineer, I also did other deep dives about Amazon and AWS. They are also linked in the show notes. If you enjoy this podcast, please do subscribe on your favorite podcast platform and on YouTube. A special thank you if you also leave a rating on the show. Thanks, and see you in the next one.