Mitchell Hashimoto’s new way of writing code

33:49You know people like the product and want to use the product and maybe want to pay for the product, but you don't know exactly how to get everybody to pay for the product. And then CD and so on is just continuing to build the repeatable revenue machine. And so with that framework in mind, we were on the right track. It was basically like to build the product. We had clear product market fit by the A in terms of the open source, right? We had millions of downloads, a lot of stars on GitHub, all sorts of signals that showed that this was resonating. We had zero revenue. And so, you know, it was raise money and slowly, slowly get closer and closer to solving the business problem.

34:29And I think we're just a year or two late or like later than the average startup. But the general keyframes were the same, just on the slightly wrong timeline, I guess. And then when you decided to do a business, this was you already had the hashi stack and then you built managed offering. I remember. Yeah, our first foray into commercialization was a total failure. It was this idea. Oh, really? Yeah, we had this product that some people, you would have to have been a diehard HashiCorp product fan to know this, but we had this first product that was called Atlas. And the idea was commercially shipping the vision of running all the products.

35:07And so there's a couple death knells there. One of them was that you had to run all the products. and so if you were just like a vault user you had a really impossible time buying or buying into our commercial product and the second was just that it was just a huge problem to like attach on to regardless of the option required you're trying to solve the problem that multiple different buying organizations in a company were fighting over so like even the people who had adopted all our tools we ran into the problem of who pays for it it wasn't the simplest engineering thing for it Correct. I think one of the lessons that I would have, you know, I would have for engineers that become founders that don't have a business background.

35:44And one of the tough lessons I had to learn is that companies want to pay for software, but they will fight over whose budget owns that. Budgets are important, right? Yes. So the budget has to exist. And if it looks like a networking problem, they're going to say, oh, networking should pay for that. So I have more budget to buy my other toys that I want. Or I could hire more people. Yeah, it could get broken down into like vendor budgets. It could already be earmarked for external purchase. But yeah, so we had this product that was like, does security perform for it? Does networking perform for it?

36:13Does infrastructure pay for it? Like, does dev tooling pay for it? Like, where does this go? And it's just that Spider-Man meme where everyone's pointing at each other. Ultimately, you don't sell anything. And so that was a failure for that reason. So I don't remember the total time we chased this down. but we had a board meeting for sure on a Friday and board meetings were usually on Fridays and we had this board meeting. We're based in the city of San Francisco. Board meetings were an hour south in real Silicon Valley and it didn't go well. It wasn't, there was no yelling. There was nobody saying you guys are messing up.

36:53There was nothing like that. It was just the way I describe it is when your parents aren't happy with you but they don't have to say that they're not happy with you. You know. But you know they're not happy with you. We had this board meeting. We drove home. Armand and I, complete drive home was silent. And it's Friday night. So usually what we do is we go straight to, Armand lived in the city and I lived in LA already, but we'd go straight back to where Armand's place and just like have a glass of wine, debrief, talk through things. And we didn't talk in this car ride home. Armand drove straight to the office.

37:27I didn't question that. uh we went into the office um sat at a table not much larger than this the only difference was there would be a whiteboard here i think one of us at that point said well that didn't go well uh we both knew it we didn't feel good and uh like the the sequence events here is now very fuzzy but at a certain point we decided let's play this experiment where if there was no sunk costs if we were starting from scratch what would we do differently today we whiteboarded all this stuff what we whiteboarded out was per product enterprise products and doing vault first and all this stuff we wrote it out spent some amount of time there it's still friday it might be saturday in terms of the time of day but it's still friday i think it was armand who looked at the board and goes why don't we just do that like why not like and and and i was like yeah why not so we decided over the course of that weekend.

38:26Just throw it all away. Just throw everything we were doing before away. We had two paying customers. We were like, just breach contract. I don't know, like figure it out. Like get out of it. We're done. And we convened an all hands meeting on Monday. Probably only about 20, 30 people in the company at that time, but we convened an all hands meeting over Zoom. And we might not have used Zoom then, but whatever video chat. And we said, okay, we're switching directions. We are now enterprise as our customer. open core per product. We would have this open source and we would have a forked version internally that had closed source features.

39:01It was a fork, but yeah. Open core business model. Armand and I thought people would quit. We thought we would lose. We didn't have an exact number. We thought it would shatter some level of confidence and wow, these guys have no idea what they're doing. We didn't have any idea what we're doing. Open core, even then, had a bit of an icky taste in people's mouth. And so like we thought people would just like philosophically quit being like, no, I came here to work on open source. I'm not going to do open core. Enterprise was kind of just like a sooty, boring thing. There was like multiple facets of why people might quit.

39:36Nobody quit. The vibes in Slack were amazing. Super positive. Oh, what happened? Do you think like why? Yeah, we asked about it in one-on-ones and follow-ups. We asked about it and it was really like everyone was kind of just like buzzing that we had a clear direction and a conviction. And, you know, there's fear of the unknown, but before there was this feeling of like, we're just throwing darts at the wall and doing this thing, and we don't know exactly who our customer is. And there was all this uncertainty in a different way. And now it was like, we don't know if this will work, but at least we're just going to sprint towards this.

40:15Like, there's these clear things, which is like, definitely enterprise, definitely open core, definitely vault. Like, all these things are set in stone that gave us a different set of certainty that suddenly the company was like, let's go. So yeah, nobody quit. It went super well. And we started, I don't know what time of year, but it was like in the fall. We built Vault Enterprise by the new year within like the first quarter of trying to do sales. We could just like tell that it was different. It wasn't like obviously successful yet, but just the caliber of conversation we're having, the distance we're getting in the buying process and the speed we're doing it, it just felt different.

40:53And what was different of this approach? Yeah, I mean, part of it just comes down to like the classic startup, like listen to your customer. And we should have listened from the beginning because our potential customers were screaming at us to do what we ended up doing, which is we would give these pitches about adopt all the products and buy this pie in the sky thing. And there were so many meetings where someone would be like, okay, I'll think about that. But how do you replicate your secrets involved? You know, they would just like ask these questions where if you If I was just listening, I was so blinded.

41:24A lot of us were blinded, but I was so blinded. If I was just listening, I'd be like, wait, a lot of people are asking about secrets replication. And that's an at-scale problem. Maybe we could close source that, right? That's what we ended up doing. That was our first feature was secrets replication, not even across data centers. The first feature was just a cluster of vault servers in a single region. You would sell this more focused product, But now kind of the problem I talked about earlier, security was definitely the buyer. There was an obvious budget, obvious person you were talking to. There was a feature that it resonated with that scale.

42:02And so we were just having much higher quality meetings in terms of getting this done. Michelle just talked about how HashiCorp managed to build a product that enterprise customers cared about and wanted to buy because it resonated with their scale. This brings us nicely to our presenting partner for the season, Statsig. Static offers engineering teams a tooling for experimentation and feature flagging that used to require years of internal work to build and is especially important at enterprise scale. Here's what looks in practice. You ship a change behind a feature gate and roll that gradually, say, to 1 % or 10 % of users at first.

42:34You watch what happens. Not just did it crash, but what did it do to the metrics you care about? Conversion, retention, error rate, latency. If something is off, you turn it off quickly. If it's trending the right way, you keep rolling it forward. And the key is that the measurement is part of the workflow. You're not switching between three tools and trying to match up segments and dashboards after the fact. Feature flags, experiments, and analytics are in one place, using the same underlying user assignments and data. This is why teams at companies like Notion, Brex, and Atlassian use Statsig.

43:05Statsig has a generous free tier to get started, and pro pricing for teams starts at$150 per month. To learn more and get a 30-day enterprise trial, go to statsig.com slash pragmatic. And with this, let's get back to the episode and what came after they built Vault.

43:44like, you know, we need support. We need proof of concept to prove it works. We need some white papers in terms of, like, other customers' scale, blah, blah, blah. And, yeah, that's what we had to build up after that and get going. And then, so you started selling with Vault, and then you did it for the other products as well, right? Yeah, we did Terraform, and we did Console. We had it for all the products, but, you know, all this data is public. You could look at it, and, well, for a period of time it was public. You could look at it in, like, the public reports of when Hosh Group was a public company.

44:12you know it really broke down to vault terraform one thing i i remember is terraform just became so so so popular across the industry so like you know like there's a hashi stack but i i only later that all the other parts existed because like terraform just seemed to be everywhere what why do you think that sudden popularity was it's so funny to hear that because i i accept and know that now and i feel the same way that you feel now that terraform is this huge thing but for the longest time like we were the vagrant company like all the other tools were like no one knew the other tools and not only that like terraform uh i one of the things that kind of frustrates me i haven't heard it recently but for a period of time one of the things that frustrated me was like oh they they only won because they were first to market i hear that a lot and we were like seventh to market okay so like to market in in what category in terms of that infrastructure codes so there were like other like players who so many yeah yeah and and no one was a clear winner it was a warring market but like that first year 2014 when we came out terraform i you know at that time one of my marketing strategies was i was at every conference i i went i traveled an obscene amount i was speaking wherever i could but even if i couldn't speak i was going just to talk to people.

45:31And there's actually a little anecdote here was when the COVID lockdowns happened in March 2020, my wife and I had nothing to do at night. We didn't have kids yet. And we opened up our calendars and we realized that it was a, we had been dating since 2012. And the first time in almost 10 years of our relationship that I will have been in the same place longer than eight days no for for almost 10 years at nine years for nine years straight i had been somewhere different at least every eight days that's how much you traveled that's how much i traveled yeah and i know there's consultants that travel a lot more and stuff but like i was traveling a lot i was coding a lot i was like doing all these things you must have coded that while you traveled as well all time yeah i had a whole system when i started traveling in-flight wi-fi didn't exist yeah yeah exactly even now it's kind of patchy yeah so i wrote these scripts that i ended up iterating on but mostly used where I downloaded all the GitHub issues and I categorized them and I would just break it down into tasks that none took more than 10 to 15 minutes and I just created this list and and when I was on the plane I would just one by one bust them out there's no internet so just commit them locally yeah and then I would get back and and some people used to notice this because I would land and you'd get this push and people would get these email notifications where like 30 issues were closed all at once wow but i found the key was pre-planning what issues we were going to work on i did that online on the ground yeah and then breaking them down into 15 minute chunks because i found it was really hard to get into like multi-hour even when i was traveling to japan or something it's really hard to get into like multi-hour flow on an airplane so i was like i'm only going to work on the stuff that isn't like heavy design work none of that it's just like bug fixes, right?

47:16Like just cleaning stuff up. And so that was my process. In 2021, HashiCorp went public. What is it like to go public, both in terms of preparing for it? How did it feel? What changed after? On the prep side, I don't have the full answer because I also stepped down from the executive team about maybe six months before we went public. So I was part of some of the planning. And obviously, I was very aware that we were planning to go public. But like, for example, I wasn't part of the roadshow or any of that but yeah you know from my seat the the parts that i was part of the parts that i had visibility onto i mean it's it it takes over a year to do it so there's a lot of prep and and there's some funny things that you do like you do you start running like a public company at least two quarters before you're public um i don't remember what the drop dead date is but there's a date where you could just like cancel going public and it's pretty close like it's like very close to when you actually like have that day so you you kind of run like a public company and to the point where you do mock earnings calls like you actually with a conference room table your investors are the public investors that aren't in the room they go somewhere else and they talk over the speakerphone and ask you the types of questions um your cfo or vp of finance gives the full report of the quarter.

48:39They try to frame the types of questions you get, and you run it, and you try to figure out whether it's running well enough, I guess. And that's sort of what the prep feels like. And there's an obscene amount of secrecy because from a regulation standpoint, you can't talk about any of this. And so, I mean, you could look back at even the dumb stuff like hacker news comments. Like, I just want radio... It's the clearest signal that a company's gonna go public because I want radio silent on every topic because everything became questionable. I remember there was just a point because there was a Hacker News comment I gave like eight months before we went public and our general counsel, like in the middle of the night, was like, you have to delete that.

49:17After he talked to me, I was like, I could see how that might affect things, but like I didn't realize it was a matter and I ended up deleting it. And is this because you're not supposed to give public information away or something like that? I don't remember the exact regulation, to be honest. Yeah, but there's some regulation about not leaking information. It's not really. I mean, it's all information, but it's more about you can't influence the market in any way. And so, yeah, and you can't make promises because if you say we're going to go public, it might cause even private funding to froth up.

49:53Then it's a form of fraud. So, yeah, basically, I just stopped talking about everything. I don't know how seriously other people take it but I took it to the point where I planned this trip to New York to go public and I invited my parents and I didn't tell my parents why we were going to New York and I just told them I want you to New York it's really really important it has to do with HashiCorp and they were like sure and I can't tell you about it and they said sure and I told them maybe a month in advance we had a dog we had to get our dog sat by my aunt and I just told them we're going on a family vacation up to the point we left.

50:32I didn't tell, nobody except my parents knew basically. None of my friends, nothing, except the friends that worked at the company. But yeah, that's what it's like leading up to it. Yeah, I was at Uber when we went public and previously I read that well before going public, HashiCorp VMware made an offer earlier That was way early. That was like a super early days. That was like two years into the company. We went probably like 10 years into the company. Yeah. So like when they tried to buy you, like, what was it like? Did you almost sell at some point? Was there any point where you were close to potentially selling?

51:06It felt close. And I got a lot of accounts afterwards that it was very close. It came down to like one vote on the VMware board was what I heard. About two years in the company, we were only three employees, me including me and Armand. So we had one employee, I guess. It was two founders and employees, three of us. We got approached by VMware. you know, I didn't know what this would be like. And it is not what it isn't is they don't show up and say, we would like to buy you. No? No. That would be too obvious. The way it happens is you get an email from some low-level business development person that wants to just like talk vaguely.

51:42And the vague talk is they're not interested in buying you. One of the jobs of business, BD people at large companies is just to have an understanding of the ecosystem. So it's really just like, let's have an understanding. they might have had an executive tell him or her to go talk to this company. There might already be an executive kind of poking around. But yeah, so it kind of starts out that way. It turns into, would you like to come by our offices and meet in person? Oh, our VP of engineering swung by. Let's talk to him. Nice to meet you. Then I think this is our actual timeline. And then I think there was a dinner where there was three VMware executives at the dinner.

52:21At that point, we thought they might be interested, but it was still so... Wow, so much dancing. Oh, this is months before there was even an offer. It was still so social. We drank, we talked about our hobbies and interests, and very not about, I mean, very basic about tech. It's really more of a vibe. They'd go to dinner, and then it started to get more serious. we spent more time in Palo Alto at the VMware offices where we started talking about partnerships, about how can VMware help our products more. And it starts about partnerships, and then it turns into hypothetical, if you had the resources of VMware, what would you do?

53:04We're like six meetings in at this point. There's no offer of anything. And then at a certain point, honestly, we were getting tired of it because nothing was happening. It sounds like you're a startup and you're going to all these meetings. Oh, and I don't even live in the Bayard. So I was flying up all the time. It was a waste of time. And to a lot of founders, that is the warning I give them is M &A becomes a waste of time. So I have another story. M &A merged in the acquisitions. Yeah, merged in acquisitions becomes a waste of time. So I'll tell you another anecdote after this. But ultimately, we kind of politely had the like, okay, let's like shit or get off the pot kind of conversation.

53:39And they put an LOI in front of us, which is a letter of intent. Letter of intent. The LOI was one page. It's basically like a non-semi-binding promise that we're pursuing buying you. No number on there. It's just like kind of vague. Still no number. Yeah, well, verbally. Yeah. They're not writing anything down. They're not putting anything in email. None of that. It's just verbal. And so at that point, verbally, we had gotten a drop of$20 million, which doesn't sound that much. Well, yeah, but we're 23 years old. oh yeah the three of you 23 years old 23 years old me and armand together own 70 percent of company um okay yeah yeah it you know it it sounds interesting to say the least what i tell people is you start you start thinking about the things you will buy is you start that's the that's the it's a dangerous path that's the that's what happens and we had advice from people who said it's like phenomenally too low like wildly too low so go ask much higher and we asked i don't remember anymore but we asked for maybe like 40 or 50 or something and they just said yes they said okay and then you know it's like way too low and uh and that was verbal too so there was nothing binding about that yes it was just it was when it wasn't like yes it was more like okay we'll work on that you know but very positive yeah a bit like in this indirect it's an indirect In an indirect business sense.

55:11In an indirect, yes. And it turned into, come meet the CEO of VMware. You know, like, clearly they're interested because we're, like, climbing still. Armand and I, we kind of started getting cold feet because it's, it's, the way we described it is it's a dream-killing amount of money. It's like, you would take the money, but you're too small to be important to a company like VMware. So they're going to just... Because even though it's, like, so much money... Personally, it's so much money. But, you know, at a VMware level, I guess you see the revenue there, all that you realize that for them it's not a big it's meaningless to them yeah it's meaningless crazy that's that messes with your mind you know yeah yeah so it becomes this thing where like personally your life could change but this thing that we both were truly passionate about like the thing i wanted to work on more than anything else would end in a sense because you know i would probably get thrown into like working on esx or something you know and you would get a manager of vm where you're not even the ceo the executives make it sound like they're going to do all this stuff with your products but like that's just one executive in a cog of corporate machinery so we started getting cold fee being like So if they're interested, maybe we're on to something.

56:18If we're on to something, we want to sell out early and sell out in a way where our dream dies. That's why I was like a dream killer. Armand very maturely, and he's two years younger than me, so he's 21 at this time. No, he sounds like the older one. Yeah, yeah, yeah, yeah. He's very mature. And Armand very maturely came up with the, I forgot where it comes from, but the risk minimization, not risk, the regret minimization framework. He was like, what? Personally on your own, go think and I'll do the same. And let's come up with a number that if we walked in the next day and they said, we're killing everything, you're going to go work on ESX for the next four years because we were going to have a walk-up no matter what.

56:59You're going to work next four years that we would be like, cool, this is worth it. Like, what's the minimum no regret, or minimum regret? We came back, and I don't remember exactly what our numbers were, but they were pretty close, and we ended up at 100. And so we're like, it felt so wrong. Like, how could we possibly ask for 100? But we're like, we said this is what we're going to do, and we stuck to it. So we went back. We asked for 100, and it wasn't a no. And it wasn't a yes. This one had a lot more hesitance. It was a lot more like, we'll get back to you. Right? Like, I don't know. But it wasn't a no.

57:33And basically, they came back to us and said, this requires board approval. So we're convening a board meeting next week. Like, unplanned. That's not when they're board meeting. We're convening the VMware board. We're going to vote on this. And then we heard that the vote didn't pass. That was that. It's just crazy how such small things could, you know, like influence. If that was an extra yes, Who knows what your story... Yeah, the one person. You might have, you know, like... It's hard to... But, you know, in VMware, you might have been clogging away on, like, this project. Yeah, yeah, yeah.

58:06I mean, we didn't build Terraform yet. So Terraform... Terraform might have not been... Probably never would have existed. High confidence, I know who the vote was. I know why they voted that way. Like, I know a lot more details, but it's like I... It worked out, obviously, in my favor, but yeah. So you've left HashiCorp and you're independent. And one thing cool about being independent is you're just very honest about stuff And there was this really interesting thread where on Twitter, you wrote about, you said, like, ask me anything about the big cloud providers, because at HashiCorp, you've worked with all of them.

58:37What was your experience back then of, you know, like Azure, AWS, Google Cloud, like your kind of honest view of how they work back then and possibly like, how has your views changed on them? the precursor to that is while I was at Hoshko I always had to be very careful about what I said about any of the cloud providers because we're partners with all of them we're partners and I didn't want to insult anyone and so I was just very professional about all their relationships and then like we like all of them like yeah or just say nothing or just say nothing nice to say don't say anything at all and then I left and I was still I kept that up because it was too close I was still flying too close to the sun as they say and then enough time passed where I was like yeah like my opinion doesn't really matter and um yeah so my to answer your question um my broad view of all them was that aws was really arrogant annoyingly arrogant was how i describe it and then when you say arrogant like can you help us understand like how you work with them or what part of them or like is it just general yeah i'll start disclaiming this though that you know we worked with so many people there that there were individuals and all of them who are awesome and nice and kind.

59:48So I'm not trying to make like individual judgments here. It was just more of like how all of it came together and how it felt as a whole. So by arrogant, I mean, it always felt like they were doing us a favor at every turn in terms of partnerships, in terms of just getting a meeting with them. It always felt like you should be thankful that we're spending time talking to you. And not just that, but also like there was always this subtle vibe of like, we will just spin up a product and kill your company. You know, it felt that no one ever said that. Well, it kind of got to a point where it was sort of like, if we don't come to terms, we're going to build this service.

1:00:22It did kind of come to that. But, you know, we did see that later on with Elastic and Oh, that had already happened. Oh, it happened already. Yeah, just not with us, but with other companies. With OpenSearch. Yeah, and they always publicly spun it as like, oh, it's so great and builds the ecosystem larger and we're doing it by the letter of the license. all has truth elements to it, but it's still not a nice thing. No, I think like, I don't think people paying attention to open source appreciated what Amazon did with Elastic's business and it showed how open source can weaponize against a company that spends, you know, their blood, sweat and tears.

1:01:01And I guess, you know, HashiCorp, you had the same thing, right? Because you were publishing permissive. Well, I mean, open source needs to be permissive. It was MIT or MBL licensed. Yeah, so like Amazon could have spun up anything they wanted. There was like a two-year period where I think for the entire two years, the entire leadership team was terrified that at any moment there would be like a vault service or something would pop up. And so yeah, that's sort of my characterization of AWS. It really took like, for example, teeth ringing to get them to help with the AWS Terraform provider. We had I don't remember the exact number, but we had something like five full-time engineers employed working on only the aws provider for terraform which you know maths out full benefits and everything to like a million dollars a year and all of that was pure open source pure integration with a commercial entity and they were not helping us at all and and they were the last of any of the cloud providers that provide any sort of help there and it it came down to some drama where we went to a meeting and basically said that we're gonna publicly say that the aws provider is deprecated and we're done like the community could pick it up or whatever but we're not we're gonna yeah because you didn't get any help from them yeah and it's taking up too much work and there's too many bugs and you're shipping honestly aws is shipping features too fast and like it's just like not worth it and that freaked them out and finally they started helping you know they might recount their side of things differently but that's pretty much it felt like no movement for years and we said that and movement started happening really fast so yeah there was that um microsoft i would i have the most positive view on microsoft they had a really hairy technical product is how i describe it it was very difficult to use azure azure and a lot of nouns like like principles and i didn't i still to this day and i've integrated with the service don't fully understand the iam hierarchy of azure um i just kind of bolted it and got it working with a team and and that was that but so technically kind of but from the business side super competent um professionals and team players was like how i describe it they we we went into every meeting with them and a lot of our meetings the first question was how do we both win that was like the first question and yeah very pleasant awesome they were the first people to jump on board uh supporting terraform sure that's some kind of bias but like they were consistent throughout the years so positive on microsoft um and google cloud you know my my yeah google cloud in general it was always like the best technology the most incredible technology and architectural thinking and I swear none of them, it felt like none of them cared or thought about the business at all.

1:03:56It was like every partnership meeting, we'd spend hours talking about the coolest edge cases and scalability and how this is going to work and I think the best public example that you could just see in history was they were the only company that when they partnered with us to write the provider they spent a lot of time building this very good, I think they called it magic something. They fully automated the whole thing. So when they shipped the new Google Cloud thing, it had a Terraform provider resource right away. And not just like, it didn't feel automated. It felt very ergonomic and like, it was good.

1:04:31It was really good. And so they had that. But whenever we would get into, how do we do co-sale? How do we like attribute your sales engineer's quota to selling like infrastructure that's spun up by Terraform. Like how do we do this? So like the business side of things? Crickets. Like impossible to get anyone. Not just impossible. It was like even if you got someone, they would say something for 20 minutes and be like, okay, cool. We have two more hours. Let's figure this other thing out. And yeah, that's what it felt like. And then the other disclaimer I'd give is all this knowledge was circa, I don't know, 2019, something like that.

1:05:09So maybe in the past seven years, things have dramatically changed. but that's what it felt like. Yeah. Going to open source, you're actively involved in open source and open source today. And it seems open source is changing a lot, especially with AI and you're seeing stuff at Ghosty. Can you tell us how open source has changed with Ghosty with the AI contributions? And what are you seeing with open source maintainers? Seems like there's a bit of drama or worrying stuff happening. Well, I would say more broadly, the issue facing open source today um is i mean there's there's multiple but the one that i feel is most prevalent across industries right now is ai contributions and the specifically the ratio the signal to noise ratio being incredibly low or in other words just being super noisy with low quality contributions it's just stressing the system quite considerably and yeah and and so So after you left HashiCorp, you started Ghosty.

1:06:13How many years ago was that? Was that like two years or so? Well, I left HashiCorp over two years ago, or a little over two years ago. I had like poked around with prototypes of Ghosty like maybe three years ago. But after I left HashiCorp, I started just like kind of working on it like 20 hours, like much more just because it was the thing that I had. What drew you to Ghosty? What was your kind of vision and why did you start working on it? It's a better terminal, right? Right. It's a terminal. Better is subjective. Well, I installed it because I like it better. But yes, a terminal and opinionated terminal, right?

1:06:48Opinionated. Very modern in terms of like supporting as many of the newer specs as possible that enable functionality like displaying images or, you know, clicking on your prompt to move the cursor and like dozens more examples like that. The original thing that drew me to it is the exact opposite of good advice that people usually give to people, which is that you find the problem and you build a solution. And what I did, and you pick the best technology that then solve that. What I did was I found a set of technologies and I was like, what could I build with these technologies? I went the opposite direction.

1:07:23And I had spent over 10 years, 12 years at HoshCorp Incorporated and three years prior to that doing infrastructure open source. So 15 years in total, just thinking almost all the time about infrastructure and cloud services and things like that. And so I had felt that I was rusty. I had sort of like my skills have had weakened on desktop software, systems programming to a certain extent, because I was so constrained by networking challenges distributed systems. So like low level systems programming had had had atrophied. I had never really worked with GPUs and GPUs. I guess crypto was happening, but I kind of ignored that whole trend.

1:08:01but this is pre-AI but GPUs were obviously in use and I just felt like I had no idea how they worked so I wanted to go to desktop so I picked all these different technologies and I said okay Zig because it looked cool to me I just wanted to try it Kenton for those of us I'm not into Zig I heard good things about it can you explain why Zig is so interesting innovative and why does it grab so many so many devs attention I don't know why it grabs other people's attention but for me it was It just felt like the best better C that I saw out there. And I am someone that's coming from the position where I actually enjoyed writing C.

1:08:38So a better C sounds great to me. To me, it's not very annoying in terms of like, if I want to blow my own foot off, please let me blow my own foot off. You know, a bunch of qualities came together where I thought on the surface it looked cool, but it's very hard to judge a programming language on the surface. So I wanted to build something with it. And so yeah, I picked the GPUs, desktop software. What could I build? for all my time at HashiCorp, I built CLI's. And I was like, well, I live in a terminal. Like, what does it take? I live in a terminal, and yet I understand very little about a terminal.

1:09:09So why don't I just, like, build a toy project that's a terminal? And that's how it started. And as with AutoStuff, I find that once you dig beneath the layer of taking something for granted, you realize that everything is way more nuanced and complicated than you imagined it to be. And terminals were the same way is once I dug beneath the surface, I realized how much they were doing, how brittle some things were, how much better certain things could be. And I got sucked into being like, I want to do this better. Okay, for like someone who's a dev, you know, I use terminals as well. I'm going to ask the stupid question.

1:09:46How hard could it be? What does a terminal actually do? And then can you maybe tell us like how Ghosty is structured or like what are the things that it needs to do just to give it a little empathy of all the work that you're doing. Yeah, yeah, yeah. I actually get that a lot. I get that question a lot. So it's definitely not a dumb question. It's really like, it gets asked less now, but a lot of people were like, I thought they were done. It's usually the most feedback I get. It's like, what is there to do in a terminal? So at a basic level, they don't do a lot. The problem is that the functionality has grown significantly of what terminal developers want to do.

1:10:18But let me just give what they do. It's kind of like an application development platform, right? It's not an operating system. You're not dealing with hardware. level problems, but it is like an application sandbox on top of that, and that other applications run within it and need to render text, they need to render colors and images and widgets and mouse events and all this stuff. Like, the best description is it's like a browser, but for text content. And so all of the complexities that a browser has, a terminal has similar ones, a smaller scale, but similar ones. And if you try to extend what a terminal is capable of, then it it gets, you know, you start bringing in more and more problems.

1:10:59Like as soon as you brought images into a terminal, you've introduced like a whole new ecosystem of problems. But the tongue-in-cheek answer I like to give to Ghosi's complexity is that it's 30 % of terminal and 70 % of font renderer. And yeah, that's what it feels like. It's really like a problem of, you know, that terminal screen you see, whether it's GPU or CPU rendered, that terminal screen you see, it's like you're drawing on a canvas. so you are building a renderer for text in there everything kind of bubbles from there so from a rough architecture standpoint of ghosty i like bringing it down in terms of threads because ghosty is multi-threaded not most terminals are not um but i'm not saying that as a positive point just a good way to describe the architecture we have a central ui thread which just draws the windows and stuff that's pretty standard for desktop software and then we have an io thread which runs the actual shell that you're seeing so any bytes that we send or it sends back to us it's processed by the io thread and then we have a renderer thread which is actually drawing it so it's it's the best way to think of it is it's on a v sync clock through 30 60 120 frames per second is it's just sampling what the terminal state is and then drawing it and the renderer itself uses a font subsystem on the same thread but we have to take the fact that this grid has this character at these sets of characters and map them into fonts and do that all on our own a lot of people think oh doesn't the operating system solve that for you but they don't unless you're much higher level like you know you can't just draw easily you know monospace text in that way you have to really put pieces together that's the the big picture um it's quite simple at that level and then just you know extend all the functionality the terminals have into that so you're kind of like building a like a 2d graphics engine a little bit that has like very focused on fonts yeah yeah Yeah, it's from a renderer side, it's very simple.

1:12:51The renderer is actually not that complicated, and I won't overcomplicate it. The hardest part is actually maintaining the terminal state. So the way terminals work is they're a grid of monospace cells. So you'll have like 80 by 24, 80 columns, 24 rows, and there's commands that the program could send to move the cursor. Say, I like to say, think of it like a paintbrush. It could say, make the paintbrush red and bold, and everything after that is red and bold, and now change it. and you're just maintaining the state and drawing around and then there's all the scroll back, right? People are used to terminals going back.

1:13:22And that's where the challenge is, is doing that in a fast, performant way. And that's what I try to do with GoSee. I mean, I show this. There's so many benchmarks we run, but one of the most obvious ones that shows the speed, which also gets a lot of criticism, is just catting, reading a large file. If you just dump a bunch of text, how fast can it get through it? And you'll see a stark difference between modern terminals. I'm not just going to say ghosty here. Like if you take ghosty, kitty, alacrity, any of these newer terminals, they're all going to do great compared to terminal an app on macOS or traditional like Linux terminals.

1:14:01The criticism is why does that matter? And you know, the easy answer is when you accidentally catafile, like a lot of people will force close. The creator of Redis posted a great comment for me, a great comment on Hacker News about why he loves ghosty which is that he previously previously used to tail production redis logs and you know just spews logs out and he used to have to send them to an intermediary file and then read them out later so he could render it so he could render it and actually work with it and he doesn't have to do that anymore because ghosty's fast enough that he could just let it dump while he's going through it parsing it like like mentally parsing it things like that and and that just saves him time and um yeah so there's something to be said at some point we should probably talk more about the fact that a lot of software these days does not care about performance and i think it's refreshing to actually have examples and i i hope we will at some point maybe get back to it you know we'll talk about it i might not help but there's a level of craftsmanship right just like not wasting resources or being efficient or i i think we all like i see in my day-to-day life like we have more powerful resources laptops phones and they're not getting any faster and it's just frustrating at times it's kind of like the love of the game i mean a lot of a lot of ghosty is just the love of the game um like like i like to say like our renderer because because like i just claimed before like it's not complicated i'm not i'm not ever going to say that ghosty is like a 2d game because a 2d game from a rendering standpoint is much more complicated um but i do care a lot about the render and we got our renderer down to for a full screen on my Mac set of grids, each frame updates in roughly, I don't know, it's something like 9 microseconds or something.

1:15:48That doesn't include the draw time. That's just like taking the state and submitting work to the GPU. It's about 9 microseconds, and the GPU takes some time. A 120 hertz, 120 frame per second frame is 8 ,333 microseconds. So if you have 9, you know, again, we don't have the number of how long the GPU takes, but it's super, it doesn't take much time at all. You're leaving a lot of options and work for what I'm saying is like, we could have made it 2000 microseconds and it wouldn't have mattered. It like you would, you would still get that performance, but that's not fun. Like I want to make it sub 10.

1:16:22I like it. The fun. Yeah. So we spent a lot of time just like it was a big, I blogged about it. It was this thing where we got it down from, it used to be about 800 microseconds and got it down to like nine. and I thought that was awesome even though for end users it doesn't make a difference. But as you say, the craft and the love of the game. So when you started out building Goalsy that was around the time where I think ChatGPT was out there were some tools. How did your tool set change in terms of how you're developing day to day? There's two sides to that. So one, AI gave a huge boost to terminals which is a funny thing.

1:16:56Like, how so? The number, because of cloud code and all these things the amount of time spent in a terminal has gone up, which if you told me in 2023 terminal usage would go up, I would say, no, it's not going to go up. I had no disillusions that I was going to like save terminals. And I didn't, right? Like AI came out and came out all these CLI tools. And, and even when you're seeing like Codex apps and Cloud apps, like it's leaving the terminal, they're still executing so many things in a pseudo terminal the number of terminals out there is is massively larger than there was in 2023 which is hilarious oh wow yeah so random super random and so that's part of why uh one of the things i'm doing with ghosty is extracting it's actually extracted already what i've called lib ghosty which is everyone reinvents this very small surface area of a terminal and because they do it it breaks like all sorts of things break.

1:17:53Like if you run a Docker build or push to a platform like Heroku and you do enough weird things in the terminal that aren't actually that weird, just like draw a progress bar. It renders it like chaos. All over the place. All over the place, yeah. And it's just because they've poorly implemented a tiny subset of a terminal because they're more complicated than people think. And so LibGoC is this minimal zero dependency library that people can embed terminals anywhere. Oh, cool. And yeah, MIT license. And just, it's really like, I'm tired of seeing broken terminals everywhere. so please use this.

1:18:22So okay, that's the one angle. Really funny. The other angle is actually AI usage. It's hard to say I'm a big fan, but within the right categories of things. I think that it's a revolutionary tool and I get a lot of joy using it. Yeah, I use it every day. I use tools like CloudCode and AMP and Codex and the chat tools every day for some aspect of my life. And it's really allowed me to choose what I want to actually think about. I think that's the most important thing is that I always felt limited in terms of, oh, I'm going to have to spend the next two hours doing this boilerplate annoying stuff that I don't want to learn about.

1:19:04But now I don't have to learn about it, which is, yeah, I'm not getting skill formation in that category, but I could now spend those two hours doing something else, and that's the best to me. in your workflow do you just use a single agent do you use multiple agents have you have you experimented with them i've tried a bit of everything i would say my standard workflow what i try to do is i try i endeavor to always have an agent doing something at all times maybe not when i sleep i don't go that far a lot of people do go that far i don't go that far but while i'm working i basically say i want an agent if i'm coding i want an agent planning If they're coding, I want to be reviewing.

1:19:44Or, you know, but there should always be an agent doing something. So you have a separate tab? Yeah, separate tab. And sometimes it's multiple. I don't, there's a lot of work that I do around cleaning up what agents do. And I don't run, like, Gastown-esque, like, things. And so I'm the mayor, so to speak. And so I don't want to run too many. I don't find it that fun to clean their stuff up. But periodically, I'll run two in competition with each other because it's a harder task and I don't have a high confidence that they're going to just like crush it. I'll just run Claude versus Codex or something like that.

1:20:20Or I'll have one coding. I'll have one doing like some sort of research task. I absolutely love them for research. That's awesome. And then I'll be doing something else. But no more than two, I would say. The code that they generate, do you always review it or have you kind of got a bit more loose? and some people swear on closing the loop, having validation for it? Or are you still like, I want to see the exact code and I'll review if it's correct and what I expected? Matters what I'm working on. And if it's ghosty, I'm reviewing everything that's going into it. If it's like I set up a personal wedding website from one of my family members, I don't care at all what the code looks like.

1:21:00Did it render right in the three browsers that I tried? Yes. Did it render right on my phone? Yes. Don't care what the code looks like. Like, does it make any network requests? No, it has no secrets access. I don't care. Like, ship it. It's only going to be online for two months, so ship it. Yeah, and then how did the AI policy at Ghostly change? I remember that maybe a year ago or so, you asked for disclosures if someone is using it. And just very recently, you kind of cracked down and said, like, all right, no more. Yeah, we're going to change again. Well, I'm not going to change again. Iterate.

1:21:30So yeah, a year ago, I started asking for disclosure. and people you know the the very fair question there is what does it matter how the code is produced and the reason to me it always mattered was because it dictates how much effort i go into fixing it because if if you produce the code of ai and you did it really quickly then i'm not going to spend hours fixing up your code you you spend your time yeah because because you know that that person that puts much time and not much human time. You're kind of trying to mirror it, right? It's ever for effort. If you put in hours, I'm going to put in hours back and I'm going to help you.

1:22:09But if you put in a few minutes and never read anything and threw it over the wall, then I should be able to read it in a few minutes, say, no, thank you, and close it. It's fair. And I need to better understand what that is. And, you know, it's not about bad code because open source has always gotten bad code contributions. But the difference before is usually those bad code contributions came from people that were genuinely trying their best and put in a lot of effort just to get to that bad code point. And so people behave differently. I would always try to reciprocate by being like, this is someone very junior or this is someone just new to the project.

1:22:43And I would try to educate them and be like, okay, we should do this better and give these careful reviews. But if it's bad code that there was low effort, I'm not going to give a careful review. So again, I wanted to know these things. And the disclosure worked decently well. The issue wasn't the disclosure. The issue was that the quantity of low-quality AI PRs that we were getting reached a point where it was too high. Do you know why that might have happened? More people instructed agents to contribute a PR to fix an issue? Do you have theories or actually seen evidence of why this happened?

1:23:21I have theories, and I've seen some evidence. Obviously, there's the rise of just AI usage in general but the real trend a step change that i saw at a certain point and i don't know when it happened because i don't use agents in this way but at a certain point they started opening prs you know before it was like you generate code and maybe they commit and stuff but you would still like push it to a branch and open the pull request at a certain point they started opening prs and there was a dead giveaway at ai because at least to this day to the point we're recording this the way claude opens a pr is it opens a draft with no body and then it edits a body later and then reopens it for review which is not how human would do it oh like one human a year would do that and now it's happening three times a day and so even if they're not disclosing ai or they're hiding it it's like oh and it happened to the speed that's unrealistic it opened the body came in less than a minute later and it opened less than a minute later like yeah pure ai i i just suite about this a couple days ago, which is just like, I wish that these agentic tools would put a pause on opening PRs for a second, because I think that's the point where it's really causing a lot of friction.

1:24:33How did you change the policy? Are you considering closing down PRs? You mentioned that recently, that the thought crossed your mind. I would say I was crashing out in that moment, but i but kind of um so we shipped this policy update where prs written by ai are no longer allowed anymore unless they're associated with an accepted feature request so you can't just drive by and be like i did this thing that i've never talked to you about here you go we and we we get about two or three of those a day and so we just close those i don't even i literally don't even read the content i could see it's ai i could see there's no fixes issue number i just close it No idea if the code is good.

1:25:14Don't care. It's just policy. Don't have time for that. That's pretty much where we landed on currently. And we're recording this in the middle of another transition, which I already have the PR open, where we're going to switch to a explicit vouching system for the community. So you're no longer able to open a PR at all, AI or not, don't care anymore, which is I think the people who criticize where it came from doesn't matter. It doesn't matter anymore. Now all that matters is that another community member has vouched for you. And if they vouched for you, you're added to a list where forever or indefinitely you could open a PR.

1:25:52If you behave badly, then you, the person who invited you, and the entire tree of people they ever invited are blocked forever for the repo. This reminds you a little bit of the social lobsters. Lobsters, yes. That's what it's based off of. So the idea is that you're putting your own reputation on the line by vouching for somebody else. I'm a reasonable person. If this happens and I or one of our maintainers of community made a mistake, if you just like hop into Discord or email and seem like a reasonable, apologetic person, I'm not going to spend a lot of time like there's not going to be like a, I don't know, a mock like court type session.

1:26:29I'm just going to be like, OK, I'll give you no chance. So, yeah, we're sort of moving to that system. I think one thing that's a little bit different is, and I should say that this is, one, inspired by lobsters, but specifically in the AI space, it's inspired by this project called Pi. They do this. Well, they do, they - The quality is built on Pi. It's a self-improving - Like, build your own agent toolkit. So, you know, kind of ironically, it's an AI tool, but they care a lot about code quality and anti-slop and things like that. So they have a similar mechanism, a little bit less of the tree and some other, but similar you can't open a pr unless you're vouched for and the other difference here that we're going with is in addition to vouching where you could positively mark someone you could actually denounce users so if there's a bad actor you could actually ban them not not just like you can't even attempt to contribute again and um that's just a yeah we had one yesterday where someone opened pr we closed it because it violated they had no associate issue and it was ai and then they just reopened it.

1:27:32Like not the same one. They resubmitted a new branch and reopened it like less than 10 minutes later. I was like, oh my gosh. So stuff like that is just, the problem is it's just wasting time. It feels like most of open source will have to change because of AI, right? Like it's, you probably know more maintainers, but I hear this, your story is not the only one, you know, like the project closed down PRs. GitHub is, I think, just shipping a feature that projects can automatically close or reject PRs? Yeah, I think open source will have to change in a lot of ways. I mean, I think, I forgot who wrote this, but one of the logical extremes is if agents are so good, you don't need open source anymore because you can just build it, right?

1:28:12Theoretically, yes. That's the extreme. I don't want to describe that extreme, but that's one of the extremes. The issue is there used to just be this natural back pressure in terms of effort required to submit a change, and that was enough. and now that that has been eliminated by ai it's i like the wording that pi uses which is that ai makes it trivial to create plausible looking but incorrect and low quality contributions and that's the that's the fundamental issue you know open source to a certain extent has always been a system of reputation right like you you earn some trust and you get more access that you know and that's how it's supposed to work um but yeah it's been that reputation system has been taken advantage of in a certain sense with AI or the default allow PRs has, you know, has been taken advantage of.

1:28:59And so I think like this vouching system that we're proposing for my project, I think it's like very true to what open source is, which is that open source has always been a system of trust before we've had a default trust. And now it's just a default deny and you must get trust by somebody. Do you think we might see a lot more forking happening though? I hope so I hope so because until now forking used to be a you know like a like a fork off a little bit because it was a lot of effort it wasn't to to keep up like it never seemed viable to fork a proper project right yeah and I okay I am separate from AI and everything I have always been a huge proponent or I guess in the past few years I've been a huge public proponent of there should be a lot more forks like a lot more forks because open source I think And one of the reasons maintainers have been taken advantage of to some extent is that contributors have some sort of entitlement, you know, whether it's toxic entitlement or not, but there's some sort of entitlement, which is I've made a valuable change.

1:30:03And it's clean and it works great. So you should accept it. But you really don't have to. Like, you absolutely don't have to. And then I've seen this time and time again where you have a high quality PR, like perfect PR, but you say no. And there's anger in the community. but the thing is, I've said this since 10 years ago in the Hotspur days, hitting the merge button is the easiest step. Getting to and hitting the merge button is the easiest step. Undergraduates should be able to do that. After that, it's the years of maintaining whatever you just merged within the context of your roadmap, the bugs, customer needs, all that stuff.

1:30:41That's the hard part. You're signing up to keeping this forever. It's very hard to remove features or remove anything. The core privilege you get with open source, like OSI open source is forking. And you should take, that's the right you got. You should fork it and maintain your own software. Yeah. One interesting impact of AI, someone tweeted about how there's a rumor that big tech is looking into re-architecting their monorepos because of agentic tooling, AI tooling, just a lot more code being churned out. What's actually happening? What's the problem with Git? The problem with Git, I mean, I think there's a lot of problems with Git, but the monorepo problem with Git is that Git is relatively bad at very large repositories because you you pretty much have to clone the entire repository there's there's some extensions to like fix that but like official mainline git can't really do that right and so for very large uh changes the very large repositories um it's sort of annoying to maintain and then if you have a lot of churn in it it's very hard to get changes into whatever your trunk is your main your master branch right you constantly rebase merge queue solves that to a certain extent i think merge cues works for humans at a certain scale but the merge cues could get quite deep but then if you sort of 10x that like conservatively i think 10x that and then if you buy into like hype cycles and you 100 or 1000x that i think it gets completely untenable in terms of how are you ever getting any semblance of cohesiveness onto the main branch quickly.

1:32:10And so, yeah, I think there's a confluence of problems there, which is the merge queue problem, the disk space problem, the branching review type problem. I also tweeted another time where Git has this you branch and you push up your branches but the branches are only the positive. When you close a PR and you don't accept it, you pretty much are the branch. In GitHub you could reaccess closed PRs but a lot of people don't even get to the PR stage. They experiment. They're like, oh, this isn't the right way. And they never push the branch. And that's like relatively important information, relatively important.

1:32:49It's not as important as the positive, but I think there should be a lot more branches and get a lot more information that we just never throw away. Like we're at, to me, we're sort of at the like Gmail moment for email, for version control, where like you used to really have to like curate, delete all this email. and then Gmail came out, gave a gig away for free to everybody. We never had to think about it. Their tagline or something was like never deleted email. I remember seeing that in some set of marketing. It's like archive it, right? Never delete it. And that's where I feel like we should be at with code, which is like just this huge repos, a lot of context.

1:33:23We need better tooling in order to find relevant context in that Git repo or version controlled repo. I would say that the real, you ask for like real examples. I do advise a company that's currently stealth but working in this space. And the real example is driven by the highly adjunct companies. The companies that are going really all in and drinking the Kool-Aid, and they're struggling in terms of the amount of churn that these agents are causing is so much greater than humans. And it's not an AI review problem or anything. It's really just a release problem, like managing the merge queues, humans getting access to the right set of data in the repository and things like that.

1:34:03So are other problems performance problems? mainly with Git or just like even the workflow of... Yeah, all of it. Performance for sure, but workflow, yeah. I mean, like every time you pull, you can't push because every time you pull, there's another chain. Like every time you push, it's rejected. Oh, yeah, there's a lot of parallel work happening as well. Do you think Git will be around with the JGC in a few years? Who knows, but what's interesting is this is the first time in like 12 to 15 years that anyone is even asking that question without laughing. We're not laughing. Right. Like if five years ago you said, well, Git be around in five years, you'd be like, yeah, of course it'll be around.

1:34:42Like, that's crazy to think, right? But now people could ask that question. And of course, some people laugh. But like there are people that critically think that Git might not be around in five years. Well, I think you do want to save the prompt history because often reading the prompts is actually, if it's a bunch of code generated, the pull request is meaningless. Changes will happen. Git and GitHub, forges in their current form, do not work with agentic infrastructure today. And it's nascent today. So, yeah, change will happen. And I'm not exactly sure. And that's not something I'm trying to change myself.

1:35:16But I'm on the receiving end in terms of agent user and a maintainer where I'm like, this isn't working. What other engineering practices that have been relatively stable for like 10, 20 or even more years you think have to change or are looking to change, thinking things like CICD, testing, code review, other ways of doing? yeah you know amp has a saying which is is it's kind of clickbaity but it's so true as everything is changing and this is this is the first time really where it feels like is the first time in my you know short relatively short to other people but still a 20-year professional career that so much is on the table for change at one time and i'm an optimist so it's really exciting to me.

1:36:02It's a lot of fun, but we've never seen so much editor mobility. Editors used to be one of those things that once someone picks an editor, it's very hard to get them off that editor. They're stuck. The level of editor mobility in the past few years between VS Code and Cursor and just jumping around is unreal. So there's a bunch of mobility there in terms of Cursor itself is a great example of a company that reached an insane valuation that you could never have gotten pre-AI on an editor product. So Editor Forges um cicd for sure and i think that testing in general because to make an agent better it needs to be able to validate its work and so tests go from even the best test case scenarios don't have like i mean the best i guess have full coverage but that that's a very extreme the the very good test case scenarios just test like one of the edge cases and one of the happy cases and you know bad case and they just kind of go through and if it passes it's probably good paired with a human who's thought about the problem.

1:37:00But AI is more goal-oriented in terms of I want this feature to work this way that if it doesn't see a spec somewhere or a test somewhere that other things should work in a different way, it'll just break it on its path to its own goal. And so I've heard this called a lot of things. I mean, the one I like the most is kind of like harness engineering. Which is like - Harness engineering. Yeah, and one of my goals for this calendar year has been to spend more time doing that, which is that anytime you see AI do a bad thing, try to build tooling that it could have called out to to have prevented that bad thing or course corrected that bad thing.

1:37:38And so it's sort of like moving from the product to working on the harness for the product or product development. And so, yeah, there's a lot of that where I think testing has to change to be far more expansive, but CICD is not set up just resource performance-wise to be able to do stuff like that. so yeah I'm not sure how it changes but that's going to change too so everything is on the table it's really interesting yeah and a lot of tools to be built one other thing observability yeah and then and I guess on that same topic I mean of the volume and scale and observability it's also like the sandbox like I didn't think even being in infrastructure and being heavily into infrastructure you know containers blew up the amount of like minimal compute units we had like floating around everywhere i didn't think that was going to go up i mean it'd go up like predictably up but i didn't think it was going to like slope change up and it is like slope change up already just due to the sandbox environments that agents need and yeah i mean that's super interesting to me because that stresses a whole lot of new systems i think you know the things that i worked on like all the products i worked on but also things in the ecosystem like Docker, but like Kubernetes, they're going to be stressed significantly because they're engineered for some level of scale, but this is a different type of particularly non-production workload scale that you have to support.

1:39:02So yeah, it's fun, fun problems. Going back to hiring, you've hired a lot of engineers and you previously talked about something really interesting. This was, I think, in the context of maybe HashiCorp, how some of the best engineers you've hired had really boring backgrounds can you talk about that like who were the best engineers you hired and like how yeah that's a better way to frame it yeah i i stand by this most the best engineers i can remember from my time at hoskirk but also just in every job that i've had are notoriously private not because they want to be private because they just don't care to be public i guess it would be the better way to put it i don't want to like carefully describe anyone without giving them away but you know they're just they don't have social media profiles very often they honestly are nine to five engineers they go back and they don't code at night they just spend time with their family but because they don't do anything else during their working time they're like locked in and and they're really good it's not about putting the hours it's also just skill wise um super strong um so yeah i always found like when i when i was reviewing resumes and stuff when you find the person that has a resume where they like they don't have any github even a github account like some people are like oh you have to public contributions to stand out like that is a way to stand out.

1:40:17But also, if you have zero public contributions, and you've just worked at companies that also I've never heard of before, it kind of is interesting to me, which is like, okay, you might know something like deep. So yeah, I think that, you know, the problem is, and the funny, the ironic thing is, I spend a lot of time on social media. And these engineers are better than me. But the funny thing is, every moment you spend on social media time is zero sum so any every moment you spend on social media is taking away from something else and the issue is it's not one for one because as every engineer knows that time it takes to really get your mind into flow to get going with something is it varies but it takes time and so when you context switch to social media if you if something's compiling and you tab over and you spend time you you've given something up in terms of thinking i i think one of the best things I do spend a lot of time on social media, but maybe unhealthy amount of time on social media, but also an unhealthy amount of time at night.

1:41:19I don't have insomnia, but it takes me a long time to fall asleep. And it's because I just sit there in the dark. And I love, some people do this in the shower, but it's not long enough for me. I love to just sit in bed, lights off, my wife's sleeping, and I just think through, like I'm writing code in my head, I'm thinking through products, I'm thinking through website copy. I'm thinking through, I'm running CLIs in my head of how it's going to feel. And sometimes, last night, I went to bed at 9.30 because I'm a dad. So I go to bed early. You have to wake up, and you don't know when you have to wake up.

1:41:52Yeah, yeah. And I didn't even feel like I was up that long. I was like, oh, I got to go to the bathroom. I should really actually go to sleep. And I looked, and it was 12.30. And all I was thinking about was, it's so dumb, but all I was thinking about was this vouching system. of how vouching might work. It might not work. And I've always had this thing where I'm willing to, I like competing. I think competition's fun. But I always feel fair game to compete with anyone in product building space because I think I'll spend more time thinking about it than they will. I think people turn it off and I try not to turn it off.

1:42:26So yeah, I mean, I think the point of all that is the best engineers are the ones that context switch the least probably. Having used AI, AI agents, do you think this might change? Because these agents can go on and think or do work for you. How would you hire in this new world where using AI is kind of a given? Most devs will prompt and fewer and fewer write, even though best devs clearly know how to write code as well. um i would definitely require competency with ai tools you don't need to use them for everything that's not important to me but it's an important tool to understand the edges of like it's like any other tool where sometimes it's useful and sometimes not useful but if you ignore it completely you're gonna do something suboptimal in a time and i mean the best example to me is proof of concepts like constantly in real product organizations you have an idea and you need to like demo it out to figure out if it works i would much rather someone just like throw slop at a wall that you're never going to ship and spend a day doing that you know may less than a day doing that rather than spend a week doing it organically as a human like because you're going to throw it away anyway and you don't even you might throw it away because it's a bad idea but i'd rather prove it out and so just slop it up and so this is why it's so nuanced i'm I'm so like, I'm so, get so worked up about sloppy PRs to open source, but it's because there's a time and place for them.

1:43:59And that's not the time and place for them, but there is. And so I would hire in that way. And I think the other thing that I don't know if it's the right thing to do, but I would strive that, that goal that I have, I would strive for everyone to have an agent running at all the time. Again, like it doesn't need to be coding, but to be doing something extra for you, I would strive for that because. I do driving. That's my biggest one. On the drive here I had some deep research going and it's like I will always spend 30 minutes on the boundaries when I wake up and before I stop working and before I leave the house or something.

1:44:33I spend 30 minutes, stop working. What can my agent be doing next? That's slow. What's a slow thing my agent could do for the next time? I know I was going to drive here for an hour. It finished far faster than an hour but it was just like, oh, I need to do some library research. okay find all the libraries that have these properties that are licensed in this way and i was looking up some like hdp3 stuff quick stuff and so build that ecosystem graph for me right before i left i was working on something to do with this vouching system and i didn't quite understand the edge cases of what i was doing and i will think about that manually but why not just start just start an agent to like look at this repo and i use amp so like consult the oracle like think deeply about what the edge cases might be what am I missing if I had another two hours to work I wouldn't need the agent to do that I would have done it myself but I don't so why not have it do it so it's just part of my goal to always have one going and I unfortunately don't have one going because they finished it all right now interesting and so this agent running there is kind of do I feel correctly that it's now so natural that it doesn't get in the way of your own thinking like you do your own thinking and you do your work but every now and then you glance and you ping it or you start it or it's now so it's not distracting right because i think that's yes i actually turn off all the agent tools do this and i turn off the desktop notifications yeah i think the desktop notifications are for the most part a mistake um so yeah i turn those off i choose when i interrupt the agent not it doesn't get to interrupt me um so for sure and there's another aspect where i think my engineering has changed where i try to identify the tasks that don't require thinking and the tasks that do require thinking and and just delegate like delegate the work to an agent like sometimes it just feels productive do the the non-thinking tasks and you're like yeah i did a lot today i got they got this but but a lot of times i just try to just delegate that out there's a lot of people that that you know say like you think less and i think if you use the tools wrong you do think less because you just like launch an agent and I don't know, go watch YouTube or scroll social media or something.

1:46:42But if you instead view it as a way to choose what you think about, then I think that you don't need to sacrifice that thinking. But I think the problem is the majority of the population probably won't do that. Yeah, but it's still, I think it's good food for thought. And it's good to hear from you on how you're using it. It's working for you. When did you start to have this second agent running? What made it switch? Was it the models getting better or? Yeah, I don't remember which model it was, but there was a certain, I tried Cloud Code right when it came out, which was like March or May last year.

1:47:12Yeah, it was March, the beta, yeah, and the May public release. Okay, I don't think I used the beta, so it was probably May. Wasn't a huge, wasn't super impressed, honestly. And then, I mean, really quickly, by like the summer, at some point during the summer, oh, I remember, I remember, I saw so many positive remarks about it that then I started to get scared that I would be behind on how to use a tool. And so I actually started forcing myself to, I still didn't believe in it, so I would do everything manually, but I was forcing myself to figure out how to prompt the agent to produce the same quality result.

1:47:50I was working much slower because I was doubling the work and it was more than double because they're slow and we're going back and forth and I already had the work done and all this stuff. But I was forcing myself to do it and you find stuff that I couldn't figure out. It just wasn't there yet. But then I found other stuff where it's like, oh, I naturally got to the same point that thousands of other people got to, which is like, oh, if I do a separate planning step, it does so much better. And everyone got there. And then I figured out, oh, if I have a better test harness for it to execute, it does a lot better.

1:48:25And then I think everyone starts with no agents.md or clawed.md or anything. same thing i realized oh if it makes a mistake and i add that just to agents.md it never makes that mistake again like oh and like these these are just like incremental things that i recognize when i see people that are new or i've watched a couple live streams like lurked on live streams or like kind of anti-ai people like try ai and it's one of those things where i'm like they're just swinging the hammer way off, right? Like it's because you haven't, it's the thing is like, it's as if someone tried to like adopt Git and they used it for an hour and decided they weren't more productive with it.

1:49:07Like it takes much longer than an hour to get proficient with Git, but you put in the effort and then you reap the rewards later. And it's sort of the same thing to me with AI tools. What would your first advice be for someone who was like not there yet? My first advice would be reproducing your work with an agent. And if you really, really don't want an agent to code, reproduce the research part of your work with an agent um like there there's a lot of people it's like i don't want it to write code for me for whatever reasons like um but yeah just kind of delegate some of the other research part there's so many places it could be helpful so it doesn't need to take you know you don't need to pick up on the it must replace you as a person kind of propaganda you could just find the the corners of where you work and and replace those parts one thing that you give people is you give advice on for potential founders because you're a successful founder you've had an exit you built up this awesome company you get a bunch of emails from people asking hey i want to be a founder what was your advice and you you wrote about this you shared the email but can you tell us like what advice you typically give people and how is it received uh well i usually ask something more specific uh because yeah if someone's like what could I do to be successful?

1:50:16One, I always will always disclaim that you're consulting someone with survivorship bias. So you need to take that into account. But I'm willing to share my experience as a survivor, but just understand that there's survivorship bias. But usually I ask for like, what's, what's something more specific? Like, what are you trying to do? And so we usually get to like, should I open source my project or not? Or should I be remote or not? Or should I do enterprise? And, and I don't know. But my, my, the most general advice I usually give people is startups are much longer than you think um you're gonna probably work on it for i say imagine 10 years a lot of people say five years but i say imagine 10 years like is this really something you want to work on for 10 years and is it something that like you need to have a certain amount of hubris in order to say i'm gonna work on this for 10 years and i truly believe i'm gonna do it better than anyone else there's nothing behind that no substance by that other than hubris.

1:51:11So you need to have a certain amount of of ego and hubris in your head to make that, but not too much where you'll be blind to change coming in. So that's usually like the first advice I give because a lot of people have cool ideas, but they're going to burn out relatively quickly. So that's where I start. So currently you're advising some companies. What are you seeing with them? Like what are servers doing these days? What are they doing differently than, you know like earlier how's that landscape uh again it's really contextual in terms of like if you're an ai startup it's very very different how are how are ai stars working differently they are there's a lot of pressure to go faster than i've ever seen any startup um i i think the industry is moving so fast that i i don't advise any ai startups but i've talked to some of them and it's even as an advisor i feel like it's too much pressure because they are just being pushed to prove themselves quickly, whether it's through traction or revenue or something.

1:52:08It's sort of like there's this mentality within that ecosystem where AI should allow you to go crazy fast. And in addition to that, there are a lot of companies moving crazy fast. So the change is happening. I think that's the one thing. Outside of that, I mean, like I said, it's just a ton of opportunity in every space. Otherwise, it's a lot of the same stuff. I mean, it's remote versus non-remote, open source versus non-open source. Do you see the role of software engineers changing? now especially at the any of the companies where engineers like like yourself they're actually being way more productive they can produce a lot more code a lot more output are they being pushed into being like you know like wearing more hats talking to the business of being a bit more like a mini founder if you will i hesitate to say more productive i i i view that there's an expectation they could do more i don't think that's necessarily more productive but it's more like you should be able to for example build a full demo design everything for your you don't need a team to do that anymore right like you should be able to do that at least from a demo perspective there's no reason not to because again you could ship slot for that that's fine i mean this is still the same but you should be able to research effectively and and in a sense to handle more vague tasks i'm seeing that a lot more just like just the capacity to experiment is so much higher i would say but then when it turns into productionizing something uh it feels similar to what it's always been i i think that there's a lot of companies that are eating the yeah the dog food of of of the ai companies of shipping whatever and i think that's a little scary yeah they look at entropic and they're like oh they build cloth co-work in 10 days and it'll be a billion dollar company they're freaking out of why they're not doing that there i think a big change is from like a pre-seed perspective or yeah pre-seed perspective where you would be like i need to raise a seed in order to build a prototype that's like like show me the prototype because yeah you should build that really quickly for most things there's still hard tech out there that you can't do that so you do a bunch of coding you do a bunch of thinking about coding as well even as you're trying to fall asleep what refills your bucket outside outside of coding outside of tech obviously like the stereotypical things like just taking breaks and being with my family and things like that but i mean i think the biggest thing is you know i am introverted so just quiet solo time refills the most energy for me i live pretty close to the beach and just if i'm in a bad mentality things aren't working and feeling unproductive or something something's going on like just closing my laptop and taking a walk outside it like stuff like that helps a lot i have a lot of hobbies and stuff but it's i think like just as a general recharge it's it's that more than anything.

1:54:54I know there's a lot of people that's like going out with friends or something like that. Then I like that, but that's not the full recharge for me. And what's a book that would you recommend and why? So I only, I pretty much only read fiction outside of news. Great. Great. Okay. The most recent book of fiction I read is an older book and it is an easy read. So I hope people are like, not like, oh, he's an idiot for reading this. But it was, what is it called? The Something Life of Addie LaRue. It's just like kind of a romantic type of fiction novel, but yeah, it's just about, I think it's like 10 years old.

1:55:31It's older now, but it's just about a woman who kind of sells her soul to live forever, but the cost was no one remembers her once they walk out the room. And yeah, it's just going through her whole life of losing all human connection, but she gets to live forever, what that is like. and I know I like reading fiction so I like reading fiction at night I don't know I don't know if it's escapism or just like you just like you know you get to live in different roles it's so so different to the coding or anything it may maybe just helps me turn off the thing I personally probably read way more fiction than I do professional non-fiction honestly yeah yeah I'm the same way it's my version of TV too TV to me is more of a social activity like if if my wife wants to watch something together like we'll watch a show But if I'm alone, I'm not going to watch a show.

1:56:23I'm going to read probably. Awesome. Well, thanks so much for going through all of these details. It was just not great to hear from how you're working, the history of HashiCorp. This was all just really interesting and motivating. Yeah, thank you. Thank you. I hope you enjoyed this long and interesting conversation with Mitchell. One thing that really stuck with me from this conversation is Mitchell's own rule for himself. Always have an agent that does something. Not necessarily coding, just doing something. For example, while he was driving to this podcast recording, he had deep research running.

1:56:55Before he leaves the house, he asks himself, what's a slow task that my agent could do while I'm gone? An important part to all of this, he turns off all notifications. The agent does not get to interrupt him. He interrupts the agent when he's ready. Michel is in charge and he has a buddy who does the work that he has delegated while he focuses on a problem that he is solving. This is a nice challenge for anyone listening. Next time you step away from your desk, before you close the laptop, ask yourself, what slow tests could an agent be doing while you're gone? If you enjoyed this episode, share with a colleague who's thinking about where software insurance could be heading.

1:57:28And if you've not subscribed yet, now's a good time. We have more conversations like this one coming. Thanks, and see you in the next one.