Distributed databases with Peter Mattis

32:52Peter Mattis:something else yeah yeah so i mean my recollection here is he was working on this kind of the big um internal system i think it was called gaia that actually had the mapping from you know you log in you have your user id and you have to look this up and they were storing you know all like this map from user id and email to whatnot to the metadata about the user in stl maps you just know it's like well there's a lot of memory usage here and it shows up on profiles and then we're like well what can we do to do better and that was kind of the genesis of it and you know he happened to be working on it and he happened to be working with me like we just started kind of noodling on this problem like oh can we do something better and it's not one of these things like i think now with google they have a whole team working on their kind of internal libraries at the time it was more of like you know everybody working on their own systems and contributing to a shared base but but i guess it still goes back to what you were just saying of like just go down the layers try to understand and if something just doesn't add up like suddenly like oh there's this big explosion memory usage like you know just ask the questions what why is this and if you're able to or you happen to be like oh can we do something about it right and one of the things that you know he observed earlier on i think part of one of the things was it was like a map from integer id to something else and like if you look at red black tree every node you have your kind of value that you're storing the map and then you have two pointers you might have an integer id that's like four or eight pipes and then two yeah and you look at that and you're like oh that seems like a lot of overhead and you're just you could use a bit well the bean tree actually has a lot better um it has better spatial locality and that's what made it faster um but it was actually smaller as well at the same time because you had less pointers involved you also contributed to go right yeah well that came later yeah yeah it came a lot later but can we talk about that yeah i just one of these other things you know i pay attention to like you know when there's research papers coming out about new data structures and like hash tables.

34:41Peter Mattis:Hash tables are like the, the, one of the earliest things you've learned about in college and data structures. Like how do I map keys to values where the ordering is unimportant? That's when the hash tables come in. And there is like, you know, the, the very earliest ways to do this. I've implemented hash tables multiple times. It's like you take your key and it might be a string and you, you put through a function and it spits out an integer, and then you map that into an array of buckets. And if multiple things map to the same bucket, you have to have a link. You have a linkless. Yeah, this is a naive implementation, right?

35:09Peter Mattis:Naive implementation used quite frequently. And over time, people discovered a lot better ways to do hash tables. That's called chaining of your hash. There's another technique called open addressing, where instead of actually having a linked list, you just kind of hash it again and move on or kind of walk down to subsequent buckets or subsequent slots to find out where you should be. And I remember reading about this new technique. It came out of some folks at Google. I believe it came out of their Swiss office. It was called Swiss tables. I believe that's where the naming came from. I'm not 100 % sure about that, but I remember reading about it.

35:45Peter Mattis:And then I was working on Go for a long period of time. And Go has this built-in map structure. And it's a hash table. It's a very highly optimized hash table because the Go team is very competent. The Go runtime team. And various folks have taken an attempt at putting together a Swiss table implementation for Go. and i tested some of them and i was like this is kind of fascinating what swiss tables do and i'll explain how it works in just a second but i looked at it's like well it's really hard to beat the performance of the runtime the runtime was really good and i kept on a new lift on this for a little while and uh eventually i ended up having to take this business trip to uh to india to bangalore and so i was on a long flight no yeah it always starts like this and i just like i'm just gonna to try to pull on this.

36:29Peter Mattis:I pulled on it sufficiently that I could get some of the benchmarks to be faster. Wow. And then I'm like, you know, that like is kind of like catnip for an engineer, like kind of make it all faster. Can I figure all the rest of it? You know, got some help from the runtime folks. There's a, an issue on the Go issue tracker that, you know, where other people have been attempting this because people propose like, hey, let's use Swiss tables and like, like, well, you know, we don't quite know all the details. You're going to have to navigate this and that. And there were some ideas there that combined them together and got to the point where it had kind of a complete implementation that was faster on most benchmarks, not quite all of them, but most of them.

37:02Peter Mattis:And then the Go folks eventually picked this up and pushed it over the finish line. And then so you came with the idea, you got to the point where you were able to show an implementation that showed how some of the benchmarks were faster. And then you started to work with some folks on the Go team too. Well, it wasn't quite that. I came up with an implementation that we ended up using at my company. It was good for our use case, but actually putting it into the runtime whole other you know kind of level but but then you just showed like here is this implementation and then they they kind of took the inspiration and the ideas yeah they're like well this is great we want to make all they always are looking for ways to make it run time faster and there's like you know oh wait and then so so you did this in this contribution this was a lot of years after you left google right yeah this was from from the outside this is from the outside awesome yeah and other people contribute stuff to the outside as well you know we had another colleague um he contributed one of the CRC implementations, you know, adapting some stuff.

37:55Peter Mattis:CRC? Cyclic redundancy checksum. You know, Intel published some papers about here's how to do the CRC and assembly very fast and contributed one of the implementations. You see a number of those things where, you know, people just like are contributing externally. It's not a lot, honestly. I mean, I actually don't know the full details, but, you know, people are regularly contributing to these things. Peter just described how the Swiss table work came together using an issue tracker in the the Go team pushing all of this over the finish line. This is where I need to mention our season sponsor Linear, which is a place to coordinate work between humans as well as agents.

38:30One thing I've noticed about how most of us work with agents is how it's a pretty single player thing. You open a terminal UI, go back and forth with an agent, and it usually produces a PR. But the rest of your team has no idea what happened in that chat unless you tell them or copy the whole history. And when everyone in the team works like this, a lot of work happens that's invisible to the rest of the team. Linear's take is that agent work should be teamwork. Even today, teams already use Linear to define the work to be done. Now, Linear can already delegate an issue to a coding agent. This agent could be an AI agent that Linear integrates with like Codex or Cursor, or Linear's own agent, or a custom agent.

39:05Either way, the engineer in delegating stays responsible for the outcome. What I really like about how Linear works is how the work stays visible. Your teammates can follow the session of the agent, check out the PR it produces, and join the review. We've gone from single-player work to multiplayer engineering work with agents. Oh, and one more thing I like about Linear, a focus on costs. Linear agents' autorouting chooses a model that is the best suited for the task. Teams can also inspect usage and set limits, track usage, so you can use capable agents without having costs balloon out of control.

39:35Hop on board at linear.app.pragmatic. Peter also previously mentioned Gaia, Google's internal system that mapped logins to users. It's not surprising that Google custom built all systems, including this one, but most of us won't build our internal Gaia. This brings us to our seasoned sponsor, WorkOS. You can think of WorkOS as something like Gaia for the rest of us, identity infrastructure you've otherwise spent quarters building yourself. WorkOS includes single sign-on, skim directories, sync, audit logs, role-based access control, basically everything a big customer security team asks for, delivered as a handful of clean APIs.

40:08It's how companies go from we have a login to we can sell to a fortune 500 without standing up their own internal identity platform and work was is already building for the next version of the agentic authorization problem their newest product is airlock the authorization layer for ai agents think about what happens when you hand an agent a task like clean up the sale opportunities in our pipeline the last thing you want is for this thing to have standing permissions to delete whatever it likes airlock sits between your agent and the tools they call. It evaluates every request against the agent's intent and your rules and then it allows it, denies it, or routes it to human for approval.

40:46You write down the policies in plain language, the agent never sees your credentials, and every call and verdict gets logged. It works with coding agents like Cloud Code and Codex and with MCP gateways. So if you're working out how to let agents do real work in production without over-permissioning them, take a look at WorkOS Airlock at workos.com slash airlock. And with this, let's get back to Peter and why he left Google after building Colossus. You're at Google, you're building Colossus distributed file systems. You're at this point probably working on probably the largest system on the planet, honestly.

41:20Why did you even consider leaving?

41:22Peter Mattis:Yeah, yeah. Well, after Colossus, I kind of dabbled in this other project called Google Goggles for a little while. Remember the glass holes? Yeah. It keeps coming back, the idea, by the way. so yeah yeah now it's still here and present it seems like google's early yeah yeah and you know i think that was a technology before its time i don't think it was ready to do at that point and looks like the uh actually doing the glasses is quite a bit harder you know the android phones we were trying to power it on were you know not powerful enough and then you know i just kind of got wanderlust you know like you know am i just kind of stagnating here google which is a strange thing to say but you know some other people feel it as well and decided to go off and try my hand another startup that didn't work out we got aqua hired by square so yeah i want to pause for a second so this company was the company name uh the company that we founded is called viewfinder viewfinder yeah it was in the mobile photo sharing space which should sound familiar this is like instagram this is like snapchat this was in 2012 yeah right as as instagram and all the more we're taking off yeah yeah no we were right there in the play and uh we just didn't have the the right go to market kind of like how to attract the users how to get viral growth kind of thing because from the outside like what i read when i when i checked you know the story just like oh you know like you you you co-founded a startup it got acquired by square hooray like it sounds like you had bigger ambitions and this was a decent outcome but not the the dream right yeah no it wasn't dream at all so uh the term i used was aqua hired so sometimes a company will get acquired get bought for you know their ip for their product for their business and other times they get bought just for the talent the people the people and we got bought just for the talent they acquired the ip but i don't think square ever did anything with it it wasn't kind of like where they were working but we built up a kind of strong technical team and that's what we were hired for and you know like i can't remember the details we'd raised a small amount of money we were able to pay our investors back make them whole maybe they got a little bit of a haircut but maybe they actually got a little bit of a but it was it was not they were okay essentially they got their money back which is like you like you know as a founder you uh you know investors are big boys they're used to losing their money but you kind of feel bad if you lose a lot of money for them so you know getting them paid back kind of makes you feel a little bit better yeah yeah and then you you spent a little time at the company that acquired you were just square yeah and then you started itching a little bit again yeah yeah yeah because you know we've been working on these you know distributed file systems and storage systems you know colossus uh one of the kind of sister projects to colossus is Spanner.

43:48And how is Spanner different to Colossus?

43:51Peter Mattis:Well, Spanner is essentially a distributed database. Colossus is a distributed storage system. And the way I think about the difference between a distributed storage system and a distributed database, you might think, oh, they're both storing data. I was about to ask. Yeah, yeah. Because a distributed database will at some point be a storage system. Right, right. So what's the difference? Yeah. So for Colossus, it was targeting large files, large append-only files. You can't update them. Append-only, yeah. Yeah, large append-only files. So, you know, 64 megabytes, maybe up to gigabytes in size.

44:23Peter Mattis:But if you're a database, you want to be storing like kind of small, like, you know, kind of, you know, if you're using SQL or like the relational data, you might have a table with, you know, billions or billions of, you know, rows. Those rows are broken up into columns. The columns are typed. That just has a very different nature to the engineering challenge for a database than it does for a distributed storage system. And usually distributed databases are implemented on top of some kind of distributed storage system. And that was the relationship. So Spanner was implemented on top of Colossus.

44:52Peter Mattis:And some of the design decisions in Spanner kind of directly fell out of the append-only nature of the files in Colossus. You can't update a file in place, so you have to make the files immutable in your database. And this is where log-structured merge trees come into play. and they weren't invented at google um but google really popularized them with level db which emerged out of the work on big table and spanner um that got popularized into rocks db i subsequently re-implemented one of these things and this is what we use at cockroach labs it's called pebble so i'm very familiar with the internals of that but it's kind of all based on this idea that the the data is kind of stored in immutable files so how did you decide to found cockroach labs while we were working at square um my co-founder and i it was actually three of us we were all at Square and one of them is Spencer.

45:41Peter Mattis:I mentioned earlier, he was working on the GIMP. He's my college roommate. And he was also at Google. He was also at Google. This other guy, Ben Darnell, who was also at Google, he joined us at Viewfinder and ended up at Square. We, you know, like we're kind of just noodling on a project to do. And we'd actually had this design back in Viewfinder. We were like, ah, we didn't, we looked around for a data base to be using in order to build Viewfinder on top of. We didn't really like the things that were out there. The technologies that said Google looked better we had the big table we had spanner and whatnot we were looking around you know like each base existed but i wasn't quite happy and there's some other systems like react and others and you know at one point we're just kind of like you know came up with the design for um cockroach to be an initial design and i was like no no guys we're doing a mobile photo sharing site we shouldn't build a distributed you know database so we put it on the back burner which i think was absolutely the right thing maybe or maybe we should just pivoted away from doing the mobile photo sharing site given the way things worked out and then we got to square and we saw some of the same problems that they were experiencing with data storage systems and spencer is very convincing managers convinced some of the management that like hey you can just work on this part-time you know to see if it had life behind the the design and then kind of conscripted uh ben and i into it and eventually started getting you know attention externally and we're like hey can we go and spin this off into a company and that's what ended up happening and so you started a company but i understand you didn't raise vc funding initially right that was the case at viewfinder at uh we did differently at viewfinder we kind of eschewed the vc money and you know in hindsight i wouldn't recommend that um but you wouldn't recommend yeah i wouldn't recommend taking the vc money because my experience the vcs are very very intelligent they can help you navigate a lot of challenges you know i think sometimes there's this perception you know the vcs will push you into various areas and maybe there's some bad ones out there that do the vcs i've had experience with just like some of the sharpest people you know i've ever met and so you kind of have like an extra like like person helping you on the team pretty much mentoring you giving you guidance telling you what they're seeing they give you advice seeing what they're seeing in the market where things are going that is very hard for sometimes for a founder especially the technical founder right that you're focused on the engineering part yeah yeah no we actually took money right away for cockroach labs um it was almost like as soon as we left we got did a little road show you know kind of in the the bay area and got some interest and you know got an investor right away i have to ask about the name though yeah how did the name of cockroach yeah i mean uh so you know we named the gimp yes that was uh that was mine pulp fiction come out in college and like which we named this thing oh the new image manipulation program i think we're thinking image manipulation program initially and i'm like oh gimp it's obvious and it just stuck at some point you know we're newly on the this new database and you kind of want to give things a name um you can't just say oh we're working on this distributed database you kind of need something and this master was like oh cockroach db like cockroaches are unkillable i want these things this database to be unkillable you know cockroach is going to survive the nuclear apocalypse apocalypse so that was where the genesis was and just stuck yeah we're we're whenever a bunch of our nodes goes down this thing will still be up yeah yeah and you know this is where we're at today with cockroach db it's like one of the things that i point out i'm like holy crap this is awesome you kill a node we did this whole campaign last year which was really just to prove out something that It had already been present.

48:56Peter Mattis:The campaign was performance under adversity. But just like you can run a workload against it, you can kill a node, you can sometimes kill a whole region and the system keeps on going. And it's like stories like that, you know, like what we did on the marketing side there, but also we hear this from our customers too. They've had fires in data centers and all the other data systems go down and CockroachDB keeps on going. I'm like, that's awesome. When you started out, who were companies, startups that who wanted to use CockroachDB and how has it changed since? because, you know, like just making the case, like, okay, I'm starting a startup.

49:28It's a small startup. Like I will need a database and I'll, I don't know, I'll typically choose a Postgres, right? Because it's free. Everyone's using it. I'm running it on Node. At what point did you see that typically tech companies are like, okay, like this is not enough for me that it's running on a Node either because it can go down or because I'm outgrowing it. What was it they are growing? Like I'm trying to get a sense of like, at what point did companies say like, tell themselves like we need something distributed yeah in a database yeah i mean oftentimes we have companies calling us up after they've had a disaster so so it's like a node went up or a

50:06Peter Mattis:hard drive failed that kind of stuff you know we it's not quite like we're ambulance chasers but if you see an outage like a big outage from some you know company you know like we'll sometimes be trying to knock them up but also they they uh will call us you know i mean like there's a very big bank who's now a customer um don't think i can name them but you can go read they had a very serious outage due to a weather event and after that which probably knocked down i'm assuming a region or a database or an networking cable or tree fell on something it knocked down a whole region you know it was a region-wide power out you knocked down the region and there's a mandate from the ceo it's like no we just have to you know be able to survive these things that gets pushed down all the way and you see this in other places where you know one of our early customers they're running on AWS and they just got to the maximum size you can run in Aurora instance on and then what would typically happen at that point is then you have to shard your database you know this is very standard practice you you take your single node database you create 10 or 20 or 100 shards and this is what Google done for some period of time and that's a heavy burden on the application developer and the way we always phrase this is like I mean the application developer is becoming a database developer at that point and they're doing it poorly you know they're trying to implement distributed transactions or indexes and whatnot.

51:18Peter Mattis:We felt the burden for that belongs on the database developer. Can we talk about automatic sharding? I think it's safe to some, most of us will know what sharding is when you're, but actually let's start from like manual sharding and then how you can implement automatic sharding. And if you can tell us like, you know, tactics that a database like CockroachDB can do to actually just take that load off of you. Yeah. Yeah. So I think the very basic form of sharding is a little bit like the hash table. let's say, you know, you have a fixed number of shards. Like, let's say it's just 100 shards. Your data model is a user with a lot of data associated with the user.

51:51Peter Mattis:You just take the user and you say like, oh, they map into one of the shards. And, you know, you kind of just rely on the hash function to get like fairly even distribution. The problem with this is at some point, you know, one of your shards will get full and you have to kind of re-shard. And that's a very, very onerous process. And re-sharding, I guess, simple way to do is like, if it's just a hard drive, I don't know, per node, where you write the user data, gets it gets full and you're like okay well i now need to split it somehow i need to move it i need to remap it i need to rejave my metadata which knows where this data lives yeah yeah and depends on exactly how you're doing that mapping from like you know the user id or whatever your shard key is to the shard you might have to remap them all right this is very typical so exactly so i mean this happens in hash tables where you know oftentimes in order to grow the hash table you just have to essentially create a new hash table double the size and copy all the data over now That's kind of like the very basic straightforward way.

52:42Peter Mattis:And there's various levels of complexity you add on it. One of them is called consistent hashing. And there's various techniques to do this. It's kind of fascinating, like how they all work. But in consistent hashing, you can add an additional node, and then it only moves a fraction of the data from each shard over there. There's various systems to do that. And I believe this is like 100 lies Cassandra. The way CockroachDB does it is more into Bigtable, more into Spanner, more into HBase, where instead of actually hashing, we actually take, you know, you can imagine all your keys in a system. And this is always true in any system.

53:14Peter Mattis:You can imagine them just in one big contiguous key space. And then you kind of partition contiguous spans of that. And then you have to build up an index on top of those contiguous spans. And what I just described there actually sounds a lot like a B-tree. So there's this index on top that is like that maps you from, you know, like I need to have this range, which node is it on? And this is a little bit like a B-tree. You know, he kind of squint, you know, it's like, I think you squint and everything's either B-tree or it's a hash table. And, you know, that index structure. But it's now starting to make sense because when I remember when I read, it might have been the Wikipedia article on B-trees.

53:47It said this is data structure that is frequently used in databases. It's now coming back to me because I didn't think too much of it. I'm not someone who builds databases. But now that we're talking, we just organically keep touching on B-trees again and again.

54:00Peter Mattis:And the other place it comes up in databases. So this is where it kind of comes up in distributed databases. and no one ever really calls it a B-tree. I just kind of squint sometimes and I see like actually kind of a B-tree. But the other place it comes up in databases is for your indexes. So if you have a table and you have like an index on your email address and you want to be able to scan over those email addresses in order, that's a B-tree under the hood in a database. And, you know, like any kind of index you have usually provides sorted order. There are hash indexes, but oftentimes it's the B-tree index and they're ubiquitous in databases.

54:32Peter Mattis:There's actually a paper called the ubiquitous B-tree and basically just identify that. I think that paper was written back in the 80s and they're still ubiquitous today. They're the foundations of single node databases and pretty much every data system I've worked on has had B-trees at some point in place in them. I want to ask about strong consistency. So CockroachDB offers strong consistency. Now for people who are a bit more newcomers to distributed systems, can we talk about the consistency models and then why strong consistency is important and why it's hard to implement it? So, I mean, there's multiple ways to kind of approach this, but I mean, if you've used a database, you probably heard of transactions.

55:09Peter Mattis:And transactions are a way to perform a whole bunch of mutations atomically. So databases talk about atomicity, consistency, isolation, durability. The durability is really easy. It's like when I write data to the database, it has to be durably written. So if anything crashes, it comes back. The atomicity is just referring to the fact I want to do a whole bunch of changes. I want them all committed or all aborted at the same time. I don't want to have like some kind of partial operation. And why is this important? Why is the atomicity important? Well, the atomicity is what gets you to the point where it's like, as an application, I can do a bunch of operations.

55:40Peter Mattis:And if there's an error and an error occurs, it all kind of gets rolled back. And it's a much simpler development model to work within. And then there's the consistency and isolation, which kind of get, you know, kind of muddled. The isolation is referring to isolation between transactions. I don't just want to run one transaction at a time. That's easy to do, right? I want to run a lot in parallel. Oh yeah. and make it so that when they're running in parallel they are running as concurrently as possible but you want to have the appearance when they're running as concurrently as possible that there is kind of a serial order to them so this is like kind of the whole trick the kind of the gold standard for isolation is called linearizability don't worry about that there's a step down from that is called serializability and that literally refers to having a serial order of your transactions but you're having to construct that in a way that you're doing everything as concurrently as possible.

56:29Peter Mattis:And the benefit of this, the serializability is, again, it's a very simple model for the application program. They don't have to worry about weird kind of defects occurring in their program. And some of the ones that can occur, it's like the classic description is of a bank, right? Where I might want to read, you know, like have an operation that reads and says, like, do I have$100 in my bank account to transfer somewhere else? And you could like arrange for lesser isolation levels that you might be able to subtract that$100 twice. And that's bad, right? You know, we want to keep accurate track of your bank account or, you know, what's in your shopping cart or, you know, it's kind of like the use cases are endless there.

57:07Peter Mattis:And you do this wrong and you have very egregious bugs. But now going back to weak consistency and strong consistency. Anything less than linearizability or serializability might be considered kind of weak consistency, but there's also like, you know, kind of strong consistency and eventual consistency. so the eventual consistency is like sometimes i can do an operation and it might not be immediately i might not be able to see all the updates the read results will not necessarily give me the current update but they'll come back you know at some point you know i've written part of it and the rest of it will show up at some point and oftentimes the easiest thing is your credit card balance right yeah yeah yeah and uh it's faster to do it that way faster in terms of just what the performance you can get out of the system but again it's a little bit harder for the application to deal with.

57:49Peter Mattis:One of the places this often comes up in distributed databases or databases that have any sort of replication is that I could write to the primary replica and then I read from the secondary and it's not there yet. That's eventual consistency. Yeah, that's eventual consistency. And you can often work around this, but just like puts bigger burden on the application developer because you have to pay attention to that. And then with CockroachDB, you have strong consistency, meaning as soon as you're writing it, when you're reading from the database, you already get the written value back. You get the written value back.

58:19Peter Mattis:You know, it's like you read whatever you wrote. It doesn't matter if you're reading from the same, you know, node you wrote it to. If you read it from another node, you still actually get the data you just wrote. Is the trade-off logically not that you would have higher latency? Because clearly to implement strong consistency, you would somehow need to, in the naive approach, you would need to write all replicas, right? Yeah. What are you doing inside Cockroach TV? Well, we are writing to all the replicas. Well, yeah. Yeah. You're just doing it fast. You're just doing it fast. You're making that efficient.

58:48Peter Mattis:I think this is one of the things that kind of also fascinates me about the software industry is we keep on finding ways to be more and more sophisticated in order to provide, you know, like do things that make it easier to write the applications, but do it in a very high performance way. And we've gotten, you know, very, very good at this over, over the years. And this is the area I know about the databases. This is happening everywhere. Like I'm just fascinated by how fast graphics have gotten, where when I entered the industry, like you were literally like writing out each individual pixel.

59:14Peter Mattis:And now you have these GPUs that are doing like billions of triangles per second, whatever the current numbers are and just like kind of astounding like how much sophistication I've got has gotten into every area of computer science wherever you look at it. One more thing on CockroachDB I want to ask about is the raft consensus. What is the raft consensus? Yeah I mean consensus protocols the original consensus protocol is called Paxos and it was famously hard to implement. Raft was you might think of it as a variant of Paxos it was kind of like an alternative to Paxos but in my mind today. And another consensus algorithm being that you have like a number of no's like three to a lot more and then how do you get them to agree on what do you typically agree on yeah yeah so you agree that the write occurred so like the way to think about consensus so you might think i want to replicate data and i i write it to a primary and write it to a secondary and you you can't actually have consensus when you only have two replicas and the reason you can't have consensus is if there's a crash and i come up like if i'm on the secondary how do i know if something was written to the primary if i'm on the primary how do i know it was written on the secondary right and you're always going to be in this kind of confusing place where it's like you either have to roll back a little bit or like you know you lose some data and consensus requires at least three but you can have consensus across more than three replicas and the idea with consensus is i'm going to write to three places and normally you don't actually when you're doing a read you don't read from a multiple of them but if there's a crash i have to do recovery then i'm reading oh i can read from uh two of the uh any two of the three and i know I can kind of determine what had happened previously.

1:00:44Peter Mattis:And it's usually just on the recovery time that you're actually doing that consensus read. So reads are typically just happening from one replica. You have to write to all three. And on a crash during that kind of failure is when the consensus read occurs. And inside CockroachDB, how many replicas do you choose for the consensus? It's typically three. For some system tables, it can be five. And then customers also have control of this. At the database level, you can write to five, seven. And five is like, you know, if you're really concerned, you know, about the data durability, you might use five.

1:01:14Peter Mattis:But there's a slowdown, you know, the more you're running to, it's like the more storage space. And obviously, like we're talking like slowdown with nodes. But of course, if they're like between regions, there's now you have a lot more resilience for, let's say, an earthquake or power hours or whatever. But now you will have additional latency, just speed of light basics, right? Speed of light latency, right. And it's, you know, tens, you know, or up to hundreds of milliseconds or even higher if you're going across the globe. So, you know, you have to be very careful with that in terms of how you architect your queries.

1:01:43Peter Mattis:And one of the things that is just a general truism of distributed databases is you don't want to have a lot of back and forth. You want to kind of like do all your reads in one kind of parallel read set, get them back, then do your writes. Right. But if you're doing like kind of serial operations where I read a row, I write a row, I read a row, I write a row. I mean, the latencies just add up. We talked about founding CockroachDB, but how has the company grown and where are you today? Yeah, yeah. I mean, we're being used in like, we're powering mission critical applications. This is our bread and butter.

1:02:10But by the way, can you elaborate on mission critical? Because it's like, if you're not, you are in the industry where you know what this means, but from the outside, it can feel hard to put a thumb on what is mission critical? Is my SaaS that is like showing as mission critical? Probably not.

1:02:25Peter Mattis:Yeah. So mission critical, in my mind, is like these kind of, the other term of art is tier zero applications the ones that are like just the the core crown jewels of what's running a company you know like a trading system you know your trading system can't go down if the trading system goes down this is kind of a critical problem or the firm that is running the trading system you know banking systems um but also you know like we work with doordash you know like some people might think that delivering your burrito is a kind of a mission critical system it certainly is for doordash right you know if that goes down is problematic we power shopping carts um you know other stuff like this where it's like well if the shopping cart goes down you know that company is losing you know hundreds of thousands millions of dollars per hour so that's kind of the criticality you think about yeah i guess of course that they're losing but but this is like when yeah their customers are also like they're used to this just working like running water and then when it's not the same thing as when your utility breaks right your water electricity is out you'll survive but it's not what you expected Yeah, yeah, yeah.

1:03:24Peter Mattis:And everybody's like, what? What age are we living in? The electricity goes out. We were kind of had this ingrained into our heads at Google. It's like Gmail cannot go down. People are relying on it. Search cannot go down. You know, if it goes down too long, people are going to move to other systems. And it's not like, you know, like in some ways search isn't as mission critical, except, oh, wait, every single search is ad dollars behind it. And you can actually notice the blip in the revenue. And it's not just the blip in the revenue. It's the blip in reputation as well. I mean, I think that's the one that really poisons companies is like, if your bank is down for a serious amount of time, the reputational damage there will be horrifically bad.

1:03:59Peter Mattis:And, you know, like we often talk about like, oh, and then grandma won't be able to pay her rent and she'll get evicted. You have to take this like responsibility really, really seriously. like no but also like just gmail being mission critical just on the way here we only exchange numbers later but we were communicating over email like oh i'm i was telling you that that you were telling me that you're here i emailed and i never for a second thought that it could go down and i think we were like responding within 30 seconds right and it's just i just know it's there like i i didn't like bother setting up a secondary communication line so yeah yeah it's like when people just like when you have that kind of level of trust with your users you got to maintain it and invest in it but then it leads to this kind of freedom for the user as well where you just don't think about like i don't have to worry about it it's just going to work and then in terms of the the company like how many engineers do you have roughly we have some hundreds i don't actually know the precise engineering number 110 but there might be 150 in r &d overall um there's other folks besides just engineers i'm clear as engineering managers those are engineers as well and then you know like we've been growing steadily it takes quite a while to build a distributed database not for the faint of heart yeah so it took a couple years a couple times yeah uh yeah well i did distribute storage system i did distribute database it's not for the faint of heart right so there's a lot of work getting it to a level of stability then a level of kind of quality beyond that level of stability um getting all the bugs out um and then continue to innovate and put more performance into the system adding functionality to integrate better within within enterprises a revenue has been you know kind of steadily growing over the years and you know it's at this place now where we see a path to future success as well and i want to ask about your coding habits so when you co-founded the company how much code did you write for the first few years i wrote a lot so i've always been a very prolific coder there were a lot of code early days and early days i mean i was uh kind of we were all technical co-founders ben spencer and i um and we were already a lot of code and i was no exception but you know if like i look back at my github output you know it's like kind of peak years maybe a hundred thousand lines of code in a year which is a lot yeah yeah no so uh we're talking pre-ai pre-ai right this is when you're back doing this manually right you know at some point you know we started out using a system called roxdb which is an lsm at some point i think it's back in 2019 you ran into limitations with it i decided um we wanted to rewrite it and did a big push to you know right that might have been like 40 50 000 lines of code and then a bunch of other people come up and helped and like you kind of look at that output i'm just like oh my goodness that was a lot to keep in your head it's a lot just to type you know 100 000 lines of code the average kind of like uh that the industry talks about is 3 000 lines of code from an engineer in a month and so if you multiply that out maybe 36 000 in a year that's good right so i was doing a lot you know i kind of look at that it's like there's kind of a max that you can hold in your head at a time um the tools have gotten a lot better since i first entered the industry we gotten better debugging techniques better testing techniques but still um quite significant and you were cto from from from the beginning co-founder of cto but there was a time sometime around like 2022 when you decided to kind of be a bit more hands-off right yeah yeah can you tell me about that i mean the general rule of thumb for uh engineering leaders is well you got to have your team you had to manage your team and we had a vp of engineering um but i was kind of getting to the point of like okay is my are my coding days done you know like i just direct from a higher level and you know i got this advice for a long period of time and i pushed back on it but you know i kind of acquiesced at some point and i think it was the right advice i'm not saying that the advice was wrong at the time um but there was time period from about 2022 to 2024 whereas like my my output declined i think i actually did the swiss tables thing in that time period but i wasn't doing much on the the core the business yeah the the core business you know i would get in there and do some work but like it's really hard that if you're in meetings all day to also do coding i think this is the fundamental tension if you like so you kind of took on the kind of the meeting burden the coordination burn and the yeah the the stuff that was if i'm reading correctly before you spent a lot of your head in the code and now you're spending a lot of your head like above the code the business the engineering the or the whatever customers that kind of stuff the customers and just being an executive as well you know so very hard to wear all those hats simultaneously and then i got back into it because ai started to emerge so how what when did you get when did you start using and when you start to find it useful or in terms of coding yeah it was interesting because you know those initial versions of like kind of glorified autocomplete came out and we're talking about the github copilot the cursor yeah the early one pilot that was the one i i had the first exposure to you we dabbled with cursor at the time but they're all like kind of glorified autocomplete and it was of crazy that you could just start typing something you know like build the rest of the function you look at you're like wait you kind of got that right this is crazy right and um you know we're trying to encourage our engineers to use this and at some point you know i can't remember if this is my idea or my co-founders or someone basically like you know like in order to like really guide people about how to use it you have to be a user yourself you know i think this is true of like engineering management in general like you want to like manage engineers you have to know how to be an engineer like if you if you don't know how to be a good engineer it's like really hard to manage other engineers i feel you'll have a hard time like just relating to them at the very least exactly exactly so kind of took it on me like no i mean this is like it was clear very early on like this is probably going to go somewhere but it wasn't quite clear how far how fast it would go and you start dabbling this and i was like oh okay well it's not quite good enough it's not quite good enough but you know let me start getting back into the coding very rapidly you know you started seeing the signs of life like you know kind of the opus models coming out it was sonnet first then the opus and you're like you're looking at these and you're like oh wow okay well they seem to be able to do quite a lot but you know the code is still not great but then it was just like the just this cadence of continuing improvements and you know i was starting to do a lot with sonnet and then starting to use opus and you know i had that same moment everybody else did and this was last year last uh november november december winter break right yeah it was thanksgiving i distinctly remember it because you know you were not chilling you were coding weren't you you were agenting i was uh agenting just like everyone else i mean i had this thing i'd wanted to do for a long period of time um on cockroach db which is like i wanted to like test like all these configurations of cockroach db across like you know different vertical scaling like how many cpus you have on a node how many stores you uh discs you have on a node how many nodes you have in the system and test across all this huge matrix of it this is one of these things they couldn't ever quite get prioritized appropriately because it never seemed like kind of critical but i've like i always had this intuition there was something there and then over this like four day span the code just like materialized as i was using um i think that was opus four seven or four five whatever the number was like the recollection i had is you know like a pretty fast typer.

1:11:08Peter Mattis:And then I just remember having this feeling of like, well, the code is just like materializing before my eyes. You know, you would ask for these things. I got out of the habit of actually typing and you just kind of like ask for something materializes. If you've ever seen like some of those, you know, a movie where they had this kind of archetype of a software engineer gets in front of the keyboard and they start typing, show the screen. It's just like, it's going way beyond human speed. And then it was gone that, right? Like it wasn't. I think as software engineers, we used to laugh at these.

1:11:37I still remember swordfish, the thing when they're visualizing the things are going, or the code appearing. You're seeing that the person's typing, and then it's a big line. And as software engineers, we're laughing because it's not how it is. But it's crazy, that effect, right? Yeah, yeah.

1:11:54Peter Mattis:But now it is. It's even better than that effect, right? Because it actually works this time. And it's that slow in comparison. It materializes faster than that. like you can literally go like we've been talking about b trees a whole bunch i implemented another b tree in in the past month you did yeah and it took about 30 minutes to implement probably 10 000 lines of highly optimized rust i mean it's just like it boggles the mind um i mean i we should look up later on 10 000 lines it's like a crazy amount you physically cannot type that fast you you start getting back to coding like are we talking about kind of vibe coding prototyping or are we actually talking you start to contribute like proper production ready code that is up the level of what you're doing at HomebookDB?

1:12:35Peter Mattis:Well, it started with this tool that was like this kind of benchmarking tool that tested this matrix. I am the CTO. We have an office of the CTO. The office of the CTO's mandate is to innovate. And I was looking for places like we can have innovation. And one of the things that we want to innovate in was, you know, better auto-scaling of a CockroachDB cluster. And kind of in the January timeframe, I came out with like what I would think is like kind of a research breakthrough, you might say. And it came about because I was dabbling in this area and just working with models, trying to understand it.

1:13:09Peter Mattis:And they're not just good at coding. They're also good at like helping you explore design ideas. And I think this is kind of the fascinating thing where it's you have to get out of the mindset of like, I know exactly what I'm going to build. But more like, hey, we have this problem. Talk through it. Be a partner with me. And be a sparring partner. Be a sparring partner. you know like there's this advice you might have heard that like you know if you're stuck on a problem you should go yellow duck it go rubber duck rubber duck it uh i think i've heard his yellow duck you know go just go talk to something you don't even need it to respond now you can talk to this system uh this intelligence and it will give you back stuff and it's not always right i mean this is the thing even today these models are fantastic fables fantastic astros fantastic and they're not always right but they they kind of like they know so much it's just encyclopedic and you can explore ideas super super fast and then be pointing out well that doesn't sound right to me and it'll be like oh yeah you're absolutely right you know like i hate that sick fancy it's like it kills me or your right to push on back on your right to push the back on that that's a new episode right yeah and yet like just able to make such fast progress um and i i i find it absolutely incredible so it quickly moved from just doing kind of side things of building up some tools and whatnot to building you know essentially over the last eight months it's january um we've been building towards a you know a new launch and you know like a lot of that has been produced a gently powered engineers and like the entire company's gone on board with this now where i think it's like probably 100 of engineers are using it to a greater or lesser degree and a lot of code is being produced and i think it's very high quality code these models will not test things adequately they don't look quite intently enough about performance but if you're like you can guide them in the right way and i think there's actually a huge advantage anybody's done management before it's a little bit like being a manager of people where you're a manager of a large group you're not looking at every line of code but you're definitely kind of helping architect the system i think there's a very strong analogy there i sometimes push back on this analogy the reason being i was an engineering manager my take is that working with these agents it's not like management because management has so much of the human stuff like as a manager when i think of all this a bunch of stuff i dealt with which was the people side of things the conflicts between people the performance reviews the meeting etc and you have none of that you do have the orchestration like again this is like almost like i guess a really like naive way of management where like you know they don't push back they they start to do it sometimes they're like unreliable but you know i almost like to use like orchestration a bit more because i feel management is is so much more involved.

1:15:45Like, I think this is a thing where like a tech lead who has no management responsibilities, but they have a group of interns, but they don't need to do with their performance, with their anything. Like, it's a lot closer to that, if you know what I mean.

1:15:55Peter Mattis:Yeah, I do agree. I use the engineering manager shorthand, but it's really about being a tech lead for like a 30 or 40 person organization, you know, or being an architect for one. I think that the term architect gives me a little bit of a distaste, but it's been an architect. He's also on the ground. A hands-on architect. A hands-on architect. And you don't have any of the management and stuff which is a blessing and curse but it's kind of remarkable that you can spin these things up if they make a mistake you can keep on correcting them until they get it right we've always been able to do that on the human side and i can do it faster i think the ultimate result for me is you just have to be more ambitious about everything you do you can produce more higher performance higher quality more secure so our ambitions have to raise up you also mentioned with AI, like you're ever using and you're building stuff, but also with CockroachDB, you're now building something that is also related to AI.

1:16:47Can you talk about that?

1:16:49Peter Mattis:Yeah, yeah. We're building multiple things. I mean, AI is the future. It's here to say. I think that's for sure. Yeah. I mean, like, you know, one of our theses, which is not crazy. Every application in the future is going to be written by AI. You know, I think there will be some kind of bespoke software, you know, handcrafted software, we'll see that, continue to exist just as people write assembly still, you know, but it's going to be diminishing in size. So, you know, asymptotically approaching 100 % of software will be written by AI. Generated, right? Yeah. It's hard to say if like when it's going to end where the humans are the agent, you know, kind of like supplying the agency and the vision behind it.

1:17:25Peter Mattis:I think that might exist for many, many years, but like, I think the code will like fundamentally be written by the AI. And I think we're going to just see this explosion of applications. And we're seeing that inside Cockroach Labs. We're seeing it elsewhere. Earlier this year, we kind of rolled out this internal platform where non-engineers could write kind of mini applications. I mean, this is like, you're hearing this at other companies. We did the same thing. And over the course of just a couple months, you know, 500 applications, a thousand applications. By non-engineers. By non-engineers, primarily by non-engineers.

1:17:58Peter Mattis:And I thought it was awesome. Like our HR team is building like these little applications. Like this is the thing I've always dreamed about doing. and like they were never serviced like your cfo's all over the place and ours is no exception producing dashboards that they can never create it before and i think it's very empowering i'm married i have a wife she needs software she cannot produce that software on her own and i have never actually helped her produce the software which is my own failing but um i think there's a world in the future where she gets like everyone's getting custom software built for them and you're just going to also see greater and greater applications and higher quality systems being produced by every company as well.

1:18:33Today, what is your stack? What do you work with in terms of harness, model, how you run agents? Is it one agent? Is it multiple? What kind of terminal do you use?

1:18:43Peter Mattis:Yeah, yeah. It's evolved over time. So, you know, like when I first started dabbling the AI stuff again, it was, you know, GitHub Copilot. I was an Emacs user for like 20 something years. I got convinced to move to VS Code. But that's all gone now. At some point, I moved to using Cloud Code. That was the thing that whatever reason I just got started using it was cloud code in the terminal of late I use a mixture so I'll just grab my current setup I use the cloud desktop app cloud code via the cloud desktop app. It's fantastic. Good job anthropic I also sometimes use the codex desktop app just so I have an alternative model to turn to For some very critical things we're working on I will get you know one of those models sometimes like the best model oftentimes I'm using the best model like Fable.

1:19:26Peter Mattis:You know, I've recently started using that sometimes it's Astra, sometimes it's the other ones, but you like you have one producer design, you have the other one being like, Hey, my colleague produced this. Can you kind of tear it up? You know, adversarial review it. And it's not always perfect. Right. You know, but I think there is utility, especially for something that's super, super critical. We're pushing towards the launch really soon and kind of rushing towards the finish line. I'm telling folks on the team. It's like, you know, especially our very senior folks, like you should use the best model right now yeah um it's worthwhile to do that i'm generally just using the best model and part of the reason is i don't actually know that i'm getting a lot more intelligence from it but i don't want to have the cognitive overhead of deciding on a case-by-case basis yeah should i use sonnet should i use opus should i use fable should i use soul or astra and i think you know maybe if you like i really need the speed i would make that decision but oftentimes i'm like i'm doing things in parallel so you're asking how many agents i'm spinning up well it depends like oftentimes there's like a certain number of sessions you might be using i find like kind of cognitive overhead is about five to ten sessions concurrently but sometimes those sessions will have many sub-agents doing things yeah and they'll be long running yeah it'll depend on what i'm doing so as i was coming here i was on the train i spun up something to do a little bit of research and that was just probably using three sub-agents.

1:20:43Peter Mattis:Other times, it might be 20 sub-agents. Anthropic has dynamic workflows. I can't remember what the codex thing is called, but they have various ways of spinning up graphs of agents and prosecuting them. I think sometimes I'm probably using 100 sub-agents. Others, it's only five. It depends on what's happening. Sometimes I'm not doing any of that agentic stuff because I'm often thinking about something else. Since you were an Emacs user for 20 years using the terminal, right? Well, Emacs, but how come you moved to graphical interface with agents a lot of people still use terminal uis yeah yeah well i so i moved from emacs to vs code i was just like a colleague was like hey these ids are really good you need to try them out i was like emacs is an id and then i i kind of moved and like you know some of the stuff was just more seamless and like emacs keeps on catching up it always feels a little bit you know six months to a year behind and like every time they have a new version my setup would break and i just got tired of that so i moved to vs code but then i was like i'd have many terminals open in vs code my setup would always be like terminals in vs code tabs the terminals and tabs you know have like eight of them open and then they started taking over used to be like just cloud code in the terminal yeah yeah and then you know just some point recently is like six weeks ago colleague was like well have you tried out the cloud desktop app recently it's really good i was like yeah no i'm very very happy and i made the switch and it was a little bit awkward at first and then suddenly i'm like holy crap this is awesome you can like manage the agents a bit better easier i mean it's like you have the sessions there the sessions i'm down the side you know and it's like they're kind of like tabs in some regard and shit but the whole integration that anthropic's been doing in open ai is doing the same thing now by the way like i'm not dissing cursor and factory and cognition they're all pushing the same thing it's incredible how fast these systems are innovating right now and evolving What do you think good software engineering looks today compared to like, you know, four years ago before we had AI?

1:22:37Has it changed? Has good changed or that's not really changed?

1:22:40Peter Mattis:Yeah, I think the ambition has to increase, you know. I think quality has to be higher. Security has to be higher. I'm glad you mentioned quality. Can we talk about that? Because I'm seeing across the industry, just quality declines, which you cannot fully put your finger on AI, but oftentimes it is people pushing out more and more and just not paying attention to just small regressions here and there. Again, like you're building a database. Like, have you noticed any or have you gotten feedback of any quality regressions? Or if not, like how come? Because like when, like this is just basic, you know, like we talk about the law of physics.

1:23:19This is an observed thing that when you start to have more output, you increase your deployment frequency. you often not always but you often have uh like more regressions and more bugs if you produce a

1:23:32Peter Mattis:certain number of lines of code you're probably going to have a certain number of defects per line of code and now you can produce more lines of code so you probably would have more defects right but the thing that pushes against that is that you can be telling these agents you have to give them quite a firm hand at this this is a thing that i i hope the the model providers are listening to but you need to give them quite a firm hand they kind of get a little bit lazy on the testing side and you have to make sure the tests are comprehensive but also they're using all the testing techniques and there's a lot of testing techniques out there we have a lot of knowledge about how to do testing well and you know what the agents are lazy humans are a little bit lazy so getting the humans to actually be very disciplined about their testing is also challenging i think it's actually easier with agents you know you can kind of instill that you can get them set up you have to give them the guidance use property-based testing use metamorphic testing use like these advanced testing techniques deterministic simulation testing you know there's like technique after technique you can use and humans have always like i'm lazy myself like i'm pretty good about being disciplined about testing at some point you're just like okay that was enough we gotta ship and like now you can get a little bit like you know a little bit stricter and firmer i think the same thing applies over on the security side the performance side i mean we've always had at cockroach in the industry we've known security coding practices yep you can literally have every single line of code on every commit reviewed by a security expert doing like an adverse or adversarial uh security review by yeah an agent or multiple agents or multiple agents right and you know we're going through a tough time in the industry right now with like kind of um hacks and security leaks and whatnot there's only a limited number of bugs that can be in software i think we'll get it you know out of it on the security side and on the quality side and then also just on the pulp when i say quality such as bugs but it's also like you And it's just a little thing.

1:25:18Peter Mattis:It's like, oh, that UX element is wrong. And there's no excuse for that right now. Like fixing it is so, so easy. And what we're also starting to see evolve is like, you know, designers using Figma. Well, I think that should be a thing of the past right now. Our designer just deals with HTML and CSS and JavaScript directly. And sometimes is even just producing pull requests, you know, PRs, which she loves. We all love. Everybody's happy with this. There's none of this waterfall. So she's producing pull requests for the production code base. Yeah. Yeah. And you know, this is on the UX side, not on the core database.

1:25:51Of course. But this is the area that she owns, right?

1:25:54Peter Mattis:Yeah. It's just, I mean, she loves it. Everybody loves it. I mean, there's no downside. This is not new. Yeah. For sure. Yeah. What about code review? What's your take on code review? I think it's pretty, it's starting to get a bit controversial. Like, is it going to stay or not? Cause there's been a practice that's been around like, think about, I mean, you, you worked at Google. Google has been so big on code review. As I understand, there are two code reviews, right? The domain experts reviews it, and then there's a language expert reviews it. So you went through that. How did you think of it, and how are you thinking of it now?

1:26:26Peter Mattis:I used to be part of the... I was one of the early members of the C++ coding review team. I'm asking the right... Okay, you will have strong opinions. Tell me. Yeah, yeah. I mean, I see that code review, I think, was fairly essential before the age of AI. but now we're getting to the point where i'm reviewing and looking at code i look at most of the code that the agent's producing still and yet it's like you're not giving it the same level of scrutiny right and this has always been the case if you get a pull request from a junior engineer you have to give it more scrutiny than if you get from your most senior engineer and you ask any you know tech lead any engineer manager any software they'll say yeah yeah so the junior engineer or someone new to the code base you know you just have to give more scrutiny and what i'm finding is the agents are getting better you're having to give less and less scrutiny and yet you do still have to give scrutiny to some things it's like i said it's like the testing will be incomplete and maybe they didn't follow the security stuff maybe you know like there was a performance regression and i think it's like you know assuring there's kind of constraints on the system so it's hard for them to do the wrong thing i suspect that you know i don't know if it's going to be this year or next that we're materially going to stop looking at the code in the same way that we don't look at assembly anymore yeah like you will we trust the compiler to produce pretty good assembly we trust the compiler to produce good assembly unless you're one of those people where it's really important you're and you're maybe a game developer or or someone and you look at it and but there's fewer and fewer of those folks yeah and even then i think what will be migrating to is why why does the human have to look at the assembly the ai should look at the assembly like i'm doing something right now where i want to have a zero overhead abstraction, you know, for doing something that is done test time.

1:28:07Peter Mattis:And what it's in compression is compiled out. They will set up for me a little system where it actually looks at the decompiled code to verify like that. There's only a few extra instructions put in place. I never would have put that in place, but now it has this guard rail that every time it changes this, it can verify that there's no regression. And I think you're going to see more and more of this where you kind of put guardrails in place. I think the model kind of likes it because it can work within that guardrail. now for what like 20 plus years you were writing so much code like i'm sure you were in the zone you remember being in the zone and just churning out you wrote a lot of like production ready code now that you're coding with ai like do you get into the zone yeah absolutely and how how is that zone is it the same is it different feels it definitely feels a little bit different it's Maybe a little bit less intense, but you're managing more things cognitively.

1:28:59Can you describe me, like, what is this right now when you're in the zone?

1:29:03Peter Mattis:Yeah. Well, I'm thinking of ideas that normally would have taken me a week to experiment with. And I think of multiple of these experiments, and then I fire them off all simultaneously. And then I'm kind of, like, reviewing, like, what else should I do while that's being complete? Sometimes I'm kind of reviewing, like, they'll be giving me progress updates of, like, oh, hey, this is coming in. We're seeing this stuff. and i'm being like well that doesn't sound right hey what about this or did we do this you know correctly you know maybe i had design and it's not implementing the design quite perfectly but i have this little feeling that you know i haven't been a college professor but maybe i'm a i was a you know if i was a college professor i had a whole swarm of research assistants and they're all got off doing things and it's coming back but it's coming back just really rapidly rapidly you're not waiting months or weeks i'm not waiting months or weeks and then i'm iterating like oh that one failed that's fine you know you just gotta let go and this this is the nature of the the software i build i think there's other pieces where it's like you just whip out a website you can whip out something that doesn't have this level of kind of scrutiny you know very quickly but i i talk about this in a i have a whole bunch of analogies about like the let's talk about what analogies do you have about about using ai i mean the one i was uh you know advocating for uh i had kind of two that i was advocating for like late last year and this year which is you know ai is coming for us it's here right and it's like uh you know you're producing software like walking down the road and sometimes you know someone will pass you they're running they got an efficient gate and whatnot but everyone's under their own locomotion and the these agentic coding agents came and it was like a car pulled up next to you you get into the car you don't know how to drive you don't understand the controls but you got to get in you start you might figure out the gas pedal and it takes off and crashes into a tree but you have to learn how to drive you know i think using all these coding tools it just isn't doesn't just happen naturally it's learning how to drive i think we might be in the era of f1 driver right now which is like the really good people can drive these systems a lot harder a lot faster than the people who just picking them up if you've never used an authentic coding tool there's a vast difference between someone who's like really expert in them knows where they break can pay attention that versus someone who's just picking up for the first time one analogy i've heard is we used to talk about 10x engineer you remember like this used to be a debate for a very long time is it or is it not but now what i'm hearing is the 100x engineer yeah and so you're saying that you are seeing some folks who maybe let's not use the term 100x engineer but like this like f1 driver who is just really really good at it like how would you describe a person who you've seen this is is it just like rock solid engineering basics and they picked up uh they lean into using these tools or what are they like yeah yeah i mean There is quite a bit of a, it feels like, you know, it's directional that they're good at software engineering before.

1:31:48Peter Mattis:I mean, I sometimes think it's like, you know, everyone's in this kind of spectrum of capability of software engineering. And this is just like, you know, taking that line and spread it out. And it's not quite true. You know, I think it's helped some people more than others, but, you know, it feels like it's just stretched it out. So, you know, your ability before is now amplified. I'm always interested to learn that, for example, Boris Churny, Thibaut at OpenAI. they both have been really really good software engineers boris wrote one of the first typescript books the first typescript book for o'reilly he built some massive system same same with tbo who built it and now you're kind of seeing oh these people are building all these tools and innovating like yeah they worked really good before yeah well i mean this is a i mean i have two other analogies to give you about like the what feels like an ai you you've undoubtedly heard about the term pair programming and programming yeah you know and the idea behind pair programming is it's good to just like you know have one keyboard one monitor and two engineers at it one of the keyboard and the other one sitting beside them kind of like looking over the shoulder and giving guidance and i think there's an aspect of that feeling where i actually got chachapiti to do a little image of this where you know it's like the android is at the computer typing and you're just there giving instructions but that was maybe the way it felt like a year ago i think it feels a little bit different now um the one i i've just started recently saying is that i feel like the domain experts the people who were really strong before are now massively amplified have you seen all this mathematical stuff coming out like the crazy proofs well i don't understand it but i i don't understand them either yeah yeah so i told you i wasn't very good at math you know it's not complete i just was like i stopped in the freshman year of college right yeah but you know i kind of like watch along with these uh advancements and you know terence tau he's like probably the most famous living mathematician you know super genius he actually posted this session there's this recent breakthrough it's called the jacobian conjecture i don't even know what it meant but he posted this chat gpt session where he's interacting i think it was chat gpt and you can see him interacting with this intelligence and it was crazy because he's talking to it as a peer colleague it's responding and literally it honestly looks like i'd encourage everyone to go look this up it looks like you know almost a foreign language it's like his domain expertise is getting amplified by the system he's interacting with you can see how he's like kind of learning and exploring ideas just really rapidly.

1:34:05Peter Mattis:Now there's all this controversy about AI mathematics, but I feel like the analogy that comes to mind to me though is the domain experts, they're a little bit like sorcerers in their particular domain. You know, you have the earth sorcerers, the earth wizards, the water ones, whatnot. And if you know the magic incantations, the right words to say the right order, you actually get something kind of magical. And if you don't, you just get sparkable stuff that doesn't have anything there behind it. You know, you would probably admit, you don't know much about distributed databases. If you're going to ask Fable or Astra, build me a distributed database like Cocker TV, you will get something out, but it'll kind of be ultimately hollow inside.

1:34:41Peter Mattis:But if you're an expert in databases and you ask to build a distributed database and you can point out all the various things you have to know about a distributed database, here's what you have to worry about, the storage layer, the networking layer, here's the various data structures, runtime inside, you can actually get something quite magical very, very rapidly out of it. So what would your advice be for engineers with like mid-level to senior level who have been figuring out how to do coding to become strong engineers in this, I guess, age of AI? Well, the first off is you have the most amazing tutor kind of readily at hand.

1:35:16Peter Mattis:And I mean, one of the things that I would always do throughout my career, and now I've kind of stopped doing it, but it's like the reason why is going to become obvious, which is like, I would always look at other people's code. So I was at Google early on. You probably heard of Jeff Dean. jeff dean was amazing coder his colleague sanjay gamelot also just an incredible coder and you know i would be looking at their pull requests i'd be looking at their changes it wasn't called pull requests at google there's a different name for it but i'd be looking at my cl right yeah cl um this is uh p4 it's a different version control system but i'd be looking at their changes be like how'd they do what they did right be looking at their code oh my goodness sanjay's code is really always very elegant you know jeff's high performance how's he doing that you know like how's he going about it it's almost just like you acquire via osmosis but now what you can do is not just acquire via osmosis but you can literally like i mean i would be encouraging if you're a junior engineer you know there's a senior engineer nearby like you could ask him how they're doing what they're doing but you can just ask the ai to dissect what they've done and explain it to you and explain to you at various different levels like how does this code work what is it doing give me a diagram explain it to me like i'm five explain it to me like i'm 10 explain it to me in French, whatever you want.

1:36:30Peter Mattis:I mean, fundamentally, to some degree, AI is a translation tool. They translate it from whatever kind of language or understanding it's in and keep on interrogating it until it increases your understanding. You were saying how domain experts are very much amplified. I guess one strategy as a software engineer is like, obviously become a great software engineer and use it as a tool. You can get a lot faster.

1:36:57You But I use AI to explain a few things for me to understand upfront, which was very helpful. And it would have taken me a lot longer time beforehand. So I can use this. But I wonder if there's another part of like, as a software engineer, you can use it to become more of a domain expert wherever you're working. If it's a payments company, I mean, use it to learn about payments as well. So you can help the business, you can help your team. And honestly, you'll just learn more, right? Yeah, yeah.

1:37:21Peter Mattis:I think, you know, I would encourage everyone. You have to have a little curiosity, right? don't don't be bound in by you know the area you're working on explore outside of it you know i was working on gmail but i was fascinated about how like the the main google search engine worked i was fascinated by like how the internals of big table worked even though i wasn't directly working on a big table just explore in look at those things and now it's so much easier because you have this super advanced patient intelligence there to explain to it like why do you think it was done this way and then like i mean once you become an expert you can be integrating well i see us done that way why don't we change this would this be helpful and you know that's where you go from just kind of learning to actually contributing back i think everyone has to have the personal agency to do this you know if you're just sitting there waiting for someone to educate you on how to do this it's going to be really hard right now because anyone who's coming in explaining how to use ai or explaining how to be a better software engineer they're going to be out of date right you just got to get in there be using these tools all the time yourself and using it like easy to learn i feel like i've learned more in the past probably even year than the previous five years combined which is weird given given your trajectory and given the environment you were working right i mean like anybody's been in the industry for a while like i'm definitely a better coder i was a better coder 10 years ago than when i first got in the industry it's like i can look back every decade and realize like i got a lot better and i feel like i just got a lot better over this past year and and this was also one of the reasons i was really excited to talk to you Because when we started to just exchange messages, the first thing you wrote to me and I asked you, like, hey, you know, how are things going?

1:38:54You said, like, you wouldn't believe, but my coding output is insane and it's high quality and it's database quality level. And those were the things I don't really usually see. I usually see, okay, I'm not producing more code, but it's slop. But again, like, to me, this is a bit of an inspiration. Like, look, like, you can use these tools to just, like, amplify yourself as a software engineer. Like, you are one example, right? Hopefully one of many.

1:39:15Peter Mattis:Yeah, yeah. Yeah, no, and I'm not the only one in Cockroach Labs. We have other people doing this as well. I find it very exciting. You know, it's a little bit exhausting right now, but it's very exciting. Like I got into software engineering because I like building stuff. I can build stuff faster. You know, the stuff you might have had to compromise on in the past, and you can take away some of those compromises. I mean, you see this in the UX of software coming out. I think the UX is a lot higher. You see all the fancy like web animations and whatnot. That's only just like the surface level.

1:39:43Peter Mattis:It just extends way, way below that. Peter, this was awesome. Thanks for coming on the podcast. Yeah, this is wonderful. Thanks for having me. One reason I was excited to talk to Peter is because he's been a very high-profile and productive engineer, pre-AI, building some of the most resilient distributed systems in production. CockroachDB is known for...

1:40:20I found both stories a good reminder that you can improve the existing library or even the language, especially if you measure which parts feel. slow. Another part of his conversation that I liked was how Peter came a bit of a full circle. He used to write 100 ,000 lines of code per year, being a very productive engineer and CTO. He didn't stop writing code aiming to coach engineers between 2022 and 2024. And then he started to code again because with AI tools, he wanted to coach his engineers better, but it's hard to do if you don't use the tools yourself. And now he finds himself being extremely productive.

1:40:55And this time, the team around him is productive as well. And we're not talking about vibe-coded software, but database-worthy, high-quality code-generated and committed to production. Peter is convinced that AI amplifies existing expertise, and this is one reason why he probably learned more this last year, building with AI, than the previous five years combined. And I find it a valuable reminder that learning and building deep expertise in software engineering, this is very valuable. And as closing, I appreciated that Peter said that not only is he excited, but he's also exhausted. There's a lot to learn, but it's tiring and neither him nor anyone I know is immune to this.

1:41:30So if you're also exhausted with all of the things going on with AI, know that you're not alone. Check the show notes for more of the pragmatic engineering deep dives on Google's engineering culture and on distributed systems. If you liked this episode, please make sure you're subscribing to your podcast player and a special thank you if you leave a rating. Thanks and I'll see you in the next one.