


18 Nov 2026 · 1h 27m · 31 chapters

Harjot Gill (CodeRabbit) argues human code review is “dead,” while AI code review and agentic “software factories” are now essential. He describes scaling to 5,000+ PRs/month via AI agents, shifting review from per-PR to per-release risk management, and using “driving license” rules where developers earn reduced review requirements based on production regression history. He also discusses agent “hill-climbing” loops (local verification, emulators, cloud tests, progressive rollouts), quality/CI challenges (“de-slopping” tests, keeping CI ~3–4 minutes), and compute constraints (CPU/GPU scarcity), plus how agents must extend into production, even marketing.
Harjot Gill is CEO of CodeRabbit. He’s coded ~30 years, is on his third startup, and previously built NetSail (Kubernetes observability) and Flux Ninja (web-scale load shedding/graceful degradation). He was an early adopter of AI coding tools (Tab9, then GitHub Copilot).
Human LGTM-style PR review is theater; humans should review higher-order risk/blast radius at release time. Teams of ~30–40 can ship “like Facebook” if they build software-factory tooling for testing, rollouts, and operations.
5.3 Codex/Opus-driven agent throughput; agents running for six weeks to build Git write-ahead logs in Rust; agents refactoring to add effect/xstate; CI de-slopping and custom CI; ads deployed via PRs to Google Ads; “content studio” marketing in code.
Written by AI. May contain mistakes.
Discussion on how Harjot Gill merges thousands of pull requests using AI agents.
“Like the last month, it was more than 5 ,000 pull requests.”
Harjot shares his career journey and the evolution of his startups in tech.
“Like earlier, it used to like nudge you that you have to hold the steering wheel or something.”
Insights on the role of AI in code writing and software development processes.
“Yeah, I mean, I started my career in very early stage in a telecom company, which was actually a startup acquired by Alcatel-Lucent back in the day.”
Explore how AI is transforming coding from a manual to an automated process.
“they might have been a bit more hesitant to just go there because you kind of have to get rid of that really familiar UI.”
Learn about the concept of hill climbing and its applications in AI software development.
“I mean, my agents are writing Rust code.”
Discover how small teams can adopt the software factory model and compete at scale.
“Like you're first making them hill climb locally.”
Examine the changing landscape of code reviews and the role of human oversight in a world of AI.
“It means that all the sophistication they had in terms of testing in production, managing the risk, managing your CICD, everything, caching.”
Understand the new dynamics of accountability and decision-making in software releases.
“price per pr like mines i don't know like 150 bucks a pr like for some people might be thousands Then you have to question, like, are they really doing your work or someone else's work?”
Explore the roles and responsibilities in software releases and their implications.
“So it's about who can understand the risk.”
Discussion on the accountability and responsibility of developers when shipping code.
“I think the human judgment is scarce now, right?”
Examining how rapid shipping affects code quality and developer practices.
“And it's nice to tie back, you know, the feedback loop that look like I think it's human, right?”
Analyzing the impact of AI on code quality and maintenance practices.
“and what happens to code quality when agents write most of the code.”
Discussing the challenges of compute capacity and efficiency in development.
“So the thing is like these agents are now, because they're like writing code and CLI commands, they're driving a lot of compute.”
Learn about the benefits of Rust and optimizing CPU usage in code.
“But now, a little bit talking about this CPU crunch.”
Discover how to evaluate AI models and manage new releases effectively.
“which means that, I mean, every day we are like evaluating something.”
Understand the implementation of formal methods in software projects.
“And a big reason for that was the feedback loop, because with open source, each review that happens every few seconds goes back into our pipeline to fine tune the hardness.”
Explore the financial implications of AI tools in software development.
“than 5 ,000, we're talking weekly commits at the last data point.”
Explore how small engineering teams can drive significant financial impact through AI investments.
“If you're going to mass produce, you're going to see like companies, composite startups now pin up, which would be a brand, but with like many products, they're going to mass produce just like how you do in a factory.”
Discuss the changing landscape of software development, market competition, and the need for speed.
“Like one of the things you're going to see is like this point SaaS solution.”
Learn about the compressed timeframe for startups to gain market traction and avoid being copied.
“And the bet is that you'll be expanding your market, you'll sell more to customers.”
Understand the evolving hiring landscape and the skills needed in modern software development.
“after building your MVP in maybe one or two months, that's the window you have now before someone just gobbles up that idea and runs with it.”
Examine how team collaboration and structure, such as 'pods', influence software development speed.
“Well, again, I just want to be clear, like this is for like a new startup VC building from zero because for existing companies that have a business, often like physical or in the world.”
Learn about collaborative workspaces called war rooms and their impact on software development.
“background of like playing this like strategy games and all have been very successful.”
Discover the high-pressure environment of startups and the challenges of maintaining work-life balance.
“At Uber, we used to have up to 2017, something called the War Room, which is like this.”
Discuss the impacts of sleep on productivity and mental health in high-stakes environments.
“And we haven't figured out the right interface to work with these agents, fleet of agents at scale.”
Explore the concept of polyphasic sleep and its implications for managing work-life demands.
“I don't know whether that's the right analogy.”
Hear insights on how burnout is linked to outcomes and personal experiences in startups.
“You wrote about burnout, and I'll quote you here.”
Delve into how autonomy and visibility affect employee motivation in fast-paced startups.
“Like we are like seeing the revenue grow like crazy.”
Learn about the long-term benefits of taking risks and gaining experience in startups.
“Like even financially, I wouldn't say it's crazy.”
What skills and practices should software engineers focus on to excel in dynamic settings?
“Like you will get the VC money, you will get the team and people will join the mission.”
Explore how Harjit views AI advancements and his unique practices around it.
“And it's like even some of the capabilities we are still like discovering with what these models are capable of.”

Get the gist, explore an idea or ask about something said.
On The Pragmatic Engineer, what are the key points of this episode?
Automatic transcript. May contain errors.
0:00The Pragmatic Engineer Host: What does it look like when a startup CEO merges 5 ,000 pull requests in a single month thanks to going all-in on AI agents and software factories? This CEO is Harjot Gill, heading up CodeRabbit, the AI code review startup. Harjot had been writing code for 30 years, is on his third startup, and never shipped as much as he does now. Today we cover Why Harjot says human code review is dead, but AI code reviews are more important than ever. The concept of driving license for engineers, how devs at his company lose their right to ship to prod if they ship too many bugs. His spicy take that$8 million seed rounds are no longer enough to build a software startup and me pushing back on it.
0:35The Pragmatic Engineer Host: And many more. If you want an unfiltered look on how a truly AI-pilled founder slash software engineer operates who is extremely hands-on with AI, this episode is for you. And while I am nowhere near this AI-pilled as Harjit, I found it educational to understand his thinking and his views on the dev tools industry where competition feels more intense than ever before. This episode is presented by TurboBuffer and I want to show you the new and improved the Pragmatic Engineer search at pragmaticengineer.com slash search that I built on top of it. I was frustrated with Substack's search, how their search was keyword only, and I couldn't do a search like the article where me and Ken Beck made a fun of a big consultancy.
1:11The Pragmatic Engineer Host: I'm obviously looking for the piece about McKinsey, but Substack cannot find it for the life of it. So I decided to build a search that I needed. I did the whole thing from idea to deployment in three hours with Codex and TurboBuffer with TPUF doing the heavy lifting. I dumped all my articles into TurboBuffer. it handles creating embeddings and getting full-text search working. The result is a much better search than what Substack has. Check it out for yourself at pragmaticengineer.com slash search. But let's think bigger. What would it take for Substack to give every single publication this kind of advanced search?
1:43The Pragmatic Engineer Host: With most other search systems, you would need to create a bunch of shards and it will be expensive to operate the search indexes. But with TurboBuffer, each publication is simply a namespace. Idle namespaces live in object storage which has no compute cost. Active ones are pulled into NVMe and memory caching tiers, so they are served very fast. And this is how TurboBuffer can scale to hundreds of millions of namespaces and why multi-tenant AI products at companies like Ramp, Harvey, and Notion are built on top of it. With TurboBuffer, you really do get the best of all worlds. Ridiculous scale, low latency, and cheap operating costs.
2:15The Pragmatic Engineer Host: Whether you're building a side project like I did or a platform with millions of accounts, check it out at turbopuffer.com slash pragmatic. Harjit, welcome to the podcast. Hey, thanks for having me here. It's great to have you here. This is your pull requests per month. Like the last month, it was more than 5 ,000 pull requests. That's right. How? How? At this point, like, it's like really machines are building machines. Like, I mean, just good at keeping the agents busy at this time, I would say. What are these pull requests doing? When are you building it? Because, you know, like you will have a lot of responsibility and time spent just running the company, I guess coordinating people, pulled into fires, the usual stuff, right?
3:00Yeah, I mean, it's all started like, I think, when the models, I would say, 5.3 Codex was kind of the first inflection when we started seeing these agents getting good enough to do like real world work, like we couldn't even ship to production. And around that time, like it was very clear to me that, okay, we have been around for like two years. I mean, so generative AI has been going on for a while, but last three or four months were like a big inflection point.
3:24The Pragmatic Engineer Host: I also see it here. It was up to July, you had like, I don't know, maybe 100 or 200 max, and then it went to more than 1 ,000 in July. It was just something else. And you wouldn't believe like everything just breaks when you start shipping at that scale. So as a leader, like one of the key things that I keep doing is like, how do I make my team productive? And a lot of my pull requests go in just fixing some bottleneck in our SDLC at this time. I mean, of course, I'm doing like maybe one or two tactical features where I'm driving the agents with each message to craft the UI or something. But most of my volume of the code will be like, believe it or not, fixing things like janitorial work, like fixing CI or flaky tests or building some like distributed system, like fixing something and caching or something.
4:08The Pragmatic Engineer Host: Right. So all of that hill climbing is what I'm doing with these pull requests. Right now, it's not very optimal. I would say that number should probably be 10x of where it is. We're not even hitting the peak here with what these agents can do. And what agent or what tool really kicked this off around, I see, it was July, August. Was this Opus? Was this Codex? Was it something else? Yeah, both the model providers have been in lockstep in terms of capabilities. A lot of our team members use Opus 5.5 with Cloud Code, but I personally use Codex. I mean, I've been a big fan of the Codex desktop app.
4:45I've been coding like 30 years. I've done three startups. And most of the time, I've also been like kind of a CTO hat in every company I've been. And I have been in a NeoVim person. I've been a code editor person all my life. Even like I put out my.files, which a lot of people use. But now we are in a different zone altogether. I have not been in a code editor since I shifted to Codex desktop app. It's pretty awesome. and ever since then like it's been always about like how many agents can i keep busy at a time so i'm actually not sleeping much that's the other side of like you get all this anxiety that you have to really um because the kind of value you are creating i mean if you know really what to build then the kind of value you're creating is is phenomenal like i mean i've never seen things move in such a compressed time frame quickly within the companies so it's not just the volume like these are not like some silly pr they're like actually moving the needle forward for our company
5:39The Pragmatic Engineer Host: Okay, you're not sleeping much, so it's eating into your night, clearly. And you're running ED agents there mostly with Codex and some other tools. But when you're commuting, you know, a lot of people listen to podcasts or music or just drive. And you have a cheat code there, right? Yeah, so I kind of live in the East Bay and I take the Bay Bridge on my way to work here. And I have a Tesla FSD and I love FSD. I'm like roughly 97 % self-driving on FSD. like pretty much it's like driving on its own. And what I do is like, and FSD has gotten really good. Like earlier, it used to like nudge you that you have to hold the steering wheel or something.
6:16But now you can like, as long as you're like looking at the road every few minutes, it's okay. So as a cheat code, what I'm doing on this side is like I have got my laptop with the codec session and I'm just like sending the instructions while I'm driving. So it's kind of funny, like the AI is driving the car while like the other AI is driving the company. Wow. Which I'm stating.
6:35The Pragmatic Engineer Host: It's wild that were there, right? Right. And also, I assume, you know, you can now talk as well or like, you know, there's Simon Wilson was telling me how I think a year or two ago he was already talking. It's just crazy how different hands off or in different areas you can you can just use AI to like build stuff. Yeah, that's right. Like one of the things with companies like CodeRabit is like as you start shipping faster, every part of your business also have to move at agentic speed. like what we are finding is we're shipping so much that the attention the marketing has become a bottleneck so i had to go back and also like refactor the marketing foundation to be very agentic which means that we built like a separate repository we call it the content studio all our slide decks are now in code html css we built a whole portal to like share those download pdfs and so on we actually are running even ads through code like to like we have millions of dollars of Google Ads spend.
7:30And the way we are going after all these keywords and the different copies on that is we have a loop where we look at the ads performance and then some agent comes up with the code changes and we actually create pull requests to deploy ads. So when you merge that pull request, it deploys it on Google Ads. So it's like everything is inside CodeRabbit at this point.
7:50The Pragmatic Engineer Host: Okay. So you've been in the industry for closer to 30 years. I just want to go back before we get more into like how we're building now, but how did you get started? How did you get into the industry? Where did you first work? Yeah, I mean, I started my career in very early stage in a telecom company, which was actually a startup acquired by Alcatel-Lucent back in the day. That's where I first got away from, okay, people could start companies and build deep tech. I mean, that was like career grade systems, high availability systems. So I learned a lot about distributed systems in my first job.
8:20At that point, I kind of like felt that, okay, I want to do my own startup. And that was the reason I came to the U.S. And the route I took was, I mean, I always wanted to do a startup. I mean, that's very clear. It's not like something. Very clear. And now you're on your third one anyway. So the route I took is I came into my PhD at University of Pennsylvania in the East Coast at Philadelphia. And the goal over there was very clear that, I mean, I'll incubate some good idea that I'll take out and then commercialize. And that's exactly what I did. So we did a technology transfer from the university.
8:50My first startup was kind of also like an inflection point, right? When microservices and Kubernetes were taking off, we saw that the observability is going to become a bottleneck. So that was my first startup. It was called NetSill, listen spelled backwards.
9:05The Pragmatic Engineer Host: Oh, yeah. And this was observability for Kubernetes, right? That's right. So the idea there was like these prior to microservices, you had monoliths and you had like a Java monolith where people will install an APM agent like New Relic App Dynamics to instrument that. So your function calls were in the code itself. And when function calls became API calls, when you were in microservices, so you started decoupling, then you had to like really observe the network. Your network became the app in a way. So that was the idea behind NetSail was, can we listen in to these network events to understand application performance?
9:38And that was kind of precursor to EBPF and some of those technologies that came out later.
9:45The Pragmatic Engineer Host: After the startup, you moved to a larger startup, right? or did you move or was it an acquisition? We got acquired. It was a pretty... But no one six, right? Yeah, this was a public company at that time. And we got a pretty decent offer. Like we just did one round of funding, but then the offer was pretty good that we took an off-ramp. Then I stayed there for like a few years trying to spend time integrating our technology into their stack and the product offerings. Then I quit to start another company called Flux Ninja, which was in like web scale. like how do you do load shedding, graceful degradation, controllability, right?
10:23So we built like a whole PID controllers to make sure that the applications don't crash with cascading failures during like Black Friday spikes and so on. Now, very interestingly, like at that time, the market had appetite for all these Kubernetes service meshes, but then generative AI happened all of a sudden. We saw that first, yeah, when I touched GitHub Copilot. I mean, we had Tab9.
10:42The Pragmatic Engineer Host: This was 2021, right? Yeah, so at Flex Ninja, we were using some AI, like Tab9 is one of the products we were using. Tab9 was the OG of AI autocomplete. Yeah, and it would create its own local model or something. I don't know what it used to do, but it used to work, but it wasn't as great as when the GitHub Copilot, that was the aha moment, right? Yeah. And this was, I believe, even before ChatGPT had come out. At that point, it was very clear. We were a fast-moving team, and at that time, the code review was already a bottleneck. so people were doing stack pull requests we built our own tool called spr we open sourced it but the but the thing is that like it was still not ideal like given the volume of the code just started growing after github copilot it was very clear that ai is going to write most of the code and so just to pause there when you saw github copilot in 2021 this was before chat gpt you already predicted or kind of like fast forwarded saying, oh, AI will write most of the code?
11:42The Pragmatic Engineer Host: That's right. This was not a commonly held view, like a select view might have held it, but what made you assume this? Can you talk us through your thinking process at that time? I was in the arena, like as a leader, like I was actually coding on the ground. Even at that company, I mean, a lot of the work I was doing janitorial, like just making the developers were productive, adding all the linters, the tooling, everything to go faster. As a leader, my goal has always been how quickly we can get to the next milestone. And when I touched that system, I don't know, it just came to me that, okay, this is going to change everything.
12:16The tool was really good in sensing what I needed to do next. And I was accepting the suggestions. It was like writing really complex algorithms. You just provide a few comments on what you want to do, and the code came out. Tab complete, right? It was pretty magical. And at that time, it was just clear to me. I don't know how, but I mean, once you are in the arena, I think you see these patterns long enough, the second order effects. And the same thing was clear to me three months back on how software is going to be built. Like I would say I'm still learning. It's not like we're not on a stable ground yet.
12:47Things are moving too fast. And some of the things I'm seeing now, we could not have predicted even six months back on how the software development is going to evolve from here.
12:56The Pragmatic Engineer Host: You found a code rabbit on this kind of notion or prediction. Was that how it happened? Actually, we didn't realize this would be such a big opportunity. One thing was clear, it will become a bottleneck, but we thought, I mean, the market cap, we were looking at Snyk and a few other companies in the GitHub ecosystems, right? Because we have an installation which is like inside the version control systems like GitHub. Like there was some precedence around previous generation companies like SonarSource, based out of Europe, which are like code quality agents. So we were kind of like thinking, okay, the market cap would be like in that territory.
13:27What we didn't realize is that this is like much, much bigger market cap opportunity because people who were not even traditionally buying those tools would line up to buy AI code reviews. Even though we don't sell to individual users, a large number of our users and individuals paying for the product.
13:43The Pragmatic Engineer Host: So you've been a NeoVim user. You wrote a lot of code for like decades, pretty much, right? You're handwritten. How easy did you find it to just move over to AI? Because a lot of people, especially some folks who use NeoVim, you know, like they're kind of into the craft as well. they might have been a bit more hesitant to just go there because you kind of have to get rid of that really familiar UI. You no longer even have a UI when you're using agents. For me, writing the code is just means to an end. As an entrepreneur, you want to get to the final value and validation quickly. I mean, I did enjoy writing code, but I wouldn't say that was the most enjoyable thing in building a company.
14:22It's about how quickly you can get an idea out, get validation, grow that business, and so on. Coding was kind of a necessary thing you had to do, and a lot of the work around maintaining those code bases, DevOps, all comes alongside that, right? So for going from, I don't miss it. If you ask me, like, do I miss writing code by hand? I actually don't. What I love is now I'm able to go from ideas to very quick implementation and just validate. So I'm doing actually the fun part of software development. I'm still doing software development. It's not like I'm not doing software engineering, but now what I'm focusing on is a lot more fun than figuring out the syntax and autocomplete.
14:58I mean, my agents are writing Rust code. I have zero idea how Rust works. We recently had an agent goal running for six weeks to... Six weeks. Yeah, so for running, so building like Git, write-head log on object stores. Yeah, write-head log. When Cursor's origin blog post came out, I pointed the agent, it ran for six weeks, and now we have that in production, powering a lot of the knowledge bases at CodeRavit. So I didn't even have to have the real implementation. It just hill climbed over time on its own, all written in Rust. And I have not read a single line of it.
15:33The Pragmatic Engineer Host: And for those of us who are not as familiar with this specific term, what does hill climbing mean for agents? Yeah, so it's some sort of a loop. Like it all starts at a very basic that agent can verify its work against something. Yep. And these loops are like inner loops, outer loops, multiple types. So when you're looking at software development, you want to provide agents, means to verify its work close to where it's doing the software development. So we do a lot of, believe it or not, a lot of formal methods. So when we first did the Slack bot product like eight months back, we invested a lot in first modeling what this distributed system is going to do in TLA +, then implementing it.
16:12And also investing a lot in simulations. So the way we are scaling code development, code review is just one piece, which we, of course, market, but there are like 100 other things you have to do, right? So we invested a lot in local deterministic simulations, right? Very recently, we started investing in lean theorems as well. So a lot of that is like...
16:32The Pragmatic Engineer Host: So lean is another form of formal methods, right? Yeah, like you have theorems and even like used for mathematical proofs, where you can also try to build provably correct software, things like billing service and all. We have got like a bunch of lean theorems. I can actually show you after the podcast. It's pretty cool what we're doing there. So what we have been doing is like, these are all like hill climbing. Like you're first making them hill climb locally. Then you're going to the next environment where let's say emulators is another thing we have invested a lot in our own 12. So you can now very quickly mock a real emulator, like the real APIs, like Google Cloud APIs or whatever, right?
17:06Then you are actually testing against a real Google Cloud or Vercel environment. Those are the cloud providers we have in dev. You can have the agent actually spin up services, actually test them, right? I mean, this is all hill climbing. And then in production as well, like the thing is that I kind of see each release. So we release multiple times a day, right? Every 30 to 45 minutes, we ship code. And each release is almost like a turn in a strategy game. If you have played Civilization, you end your turn, you learn. That's how we see releases. Like each release is an opportunity for an agent to learn something from that environment.
17:37So a lot of the times we will release features under a feature flag, do like progressive rollouts, like maybe 10 % for our open source users and so on. And the agent will observe the performance, the logs until it's stable, then do wider and wider rollout. And each release is the opportunity to learn for the agent.
17:54The Pragmatic Engineer Host: Interesting, because what you're describing, this kind of stage and wider and wider rollout, this is something that Facebook had been doing for a long time. And they just have a massive user base. We're talking billions of people, right? And they built this automated system. And outside of Facebook, I don't really know. I didn't know of many other companies that had a kind of automated rollout for a few reasons. One is this only works when you literally have hundreds of millions of users. It's really hard to do it otherwise. It's a large investment to build. Automation is hard to build. You need even pre-AI, you needed some sort of ML, which again, just goes up and up.
18:30The Pragmatic Engineer Host: And now you're describing that you had a pretty small company in terms of headcount compared to Facebook, 75 ,000 people. CodeRabbit is like... 40 engineers, maybe? Yeah, and you've been able to build this for yourself. That's right. So the way we are seeing it is like the way Netflix, Facebook used to ship with thousands of engineers and the kind of tooling they needed, like all the way from the process on merge queues and everything like they needed to ship at scale is something that small teams are going to need now with AI. Now, teams of 30, 40 people, if they use agents in the right way, They have the right abstraction and this whole concept of software factory.
19:09They're going to ship as much as Facebook. It means that all the sophistication they had in terms of testing in production, managing the risk, managing your CICD, everything, caching. There's a lot of stuff around it. And they will need it in a small team setting as well. It's a golden opportunity if someone was at Facebook at that time, like when they were shipping and figuring out all these concepts, to go out and now actually build products. The market's grown. There's a lot more people who need those products than ever before.
19:36The Pragmatic Engineer Host: And you mentioned software factory. It's a term that is going around a lot. What does it mean for you? Yeah, it's many things. I think, I mean, we have a slightly, maybe it's the same take as others, but I mean, I'll give you my take on it. Even when you're building with AI, there are, I think, two ways of using AI. One is like more like on a potter's wheel, you are like shaping the software with each interactive chat with cloud code or codex. Like, for example, like with UI development, I have the browser open and doing annotation and each time I send a message, I'm crafting the software. But with AI and I can like manage as a human, like my decision capacity is to do only like one or two of those tactical chats in a day.
20:19I mean, I usually have like 20 chats in Codex right now. Two would be of that manner. Around 18 are more like the factory concept, like where I know that I can have the agent hill climb on something. whether it's like building a distributed system, I can be hands-off with a goal. Just like the example I gave you with the Git on object stores and also SQLite LTX, that's the other thing we did with KnowledgeBase is we removed our vector database provider to build our own in-house version on SQLite LTX, right, on an object store. Those kind of things you can hill climb. The other stuff is like each time you have a deploy, CI, CD is happening.
20:57It's an opportunity for the AI to learn and continuously improve it. So I have agents running continuously to remove flaky tests, right? Each flaky test resets the merge queue. It's really bad. So you have to have that continuously happen. Each time you have a deploy or a CI, each run is an opportunity for the agent to optimize something, do the next thing, right? And then you're running the agents against production as well. So you have agents which own health of some components I care about. For example, I care about billing. I care about the knowledge-based performance. Like I care about the code graphs we have.
21:27So each deploy, I have the agent running in a loop to observe the next experiment now and do either two things, like add more telemetry if it's like flying blind or fix something. So those are the kind of loops I'm running. And those are factories. So the thing is that right now, the chat is not kind of the right interface to run those agent fleets at scale. The chat is great when you're doing crafting the software one message at a time. So some of these things we are still figuring out. First of all, that whole maintenance, the janitorial work around shipping software and maintaining the factory should probably be in the cloud on sandboxes and all.
22:03And I shouldn't be the one waking up at like 3 a.m. I mean, it's more like morning shift, night shift now in the factory. We luckily have a team in Bangalore. So we have a follow the sun model. So I want like, if I'm babysitting the agent, so I'm not owning the Kuhn anymore. We are owning agents. Let's say I have an agent looking at the health of a billing service and it stalls on something. It has a question. Then if I'm around during the US time zone, I'm able to answer that question. But I want someone to be able to unblock the agent while I'm sleeping. So I would say we haven't figured it out yet.
22:29So we are going on the upward trajectory, but most of our development right now is constrained by how many sessions people can run locally, how many releases they can like make their PRs land in, like look at the release coverage and the PRs they landed. And we're also constrained on local resources. One of the things we found, and that's where I have a cheat code as a CEO, I have a very powerful machine. The developers don't have those. They have like 48 GB RAM, 24 GB RAMs, and they max out at like three, four sessions. Beyond that, if the machine just dies with the unit test, it has to run locally, right?
23:01So we are feeling like the real compute constraints right now in terms of the local development. And we are investing very heavily to go and make some of our cloud-based environments work.
23:10The Pragmatic Engineer Host: But I guess it's pretty obvious at this point that a lot of development will move to the cloud because of a lot of the constraints that you mentioned. And I guess the only downside I can think of moving to the cloud, of running coding agents in the cloud is, a there's a bunch of work that needs to be done that you have an environment that's similar to your local one security all all that stuff but you need to build that and there's potentially a cost uh question as well because now you know like you have your hardware and when you're utilizing it pretty well that's pretty that's pretty good right you buy it once and you do it and there will be an additional cost part but the the cost i i wonder if that's a smaller question because you know like we already know ai has a very real cost and i know you're paying a lot for ai already it's actually better because in the way like if it's in the cloud and more like a collaborative setting you clearly see the value created because right now we have some people who are spending tens of thousands of dollars on local tokens we don't know if they're doing like side projects like everyone's building their like home automation systems and see what you mean yeah right so there's no auditability as well on on local development right so sometimes we look at the price per pr like mines i don't know like 150 bucks a pr like for some people might be thousands Then you have to question, like, are they really doing your work or someone else's work?
24:22So the thing is that, like, the cost perspective, I think I will be also, as a leader, like, comfortable spending the money if I can see where it's adding the value. Like, if it's in the Slack, so we have a Slack bot over there when CodeRabbit responds to all the Datadog alerts. I clearly see it's adding value.
24:37The Pragmatic Engineer Host: CodeRabbit, one of the main things that you do is AI code review. Now, I have to ask you this question, is code review dead? because specifically human code review, there's so many signs that it is on its last legs. The teams who are still doing it, who I talk to, it's kind of theater. It's like, you know, LGTM looks good to me, but people no longer have comments. They no longer read it. And companies like Uber are now making changes to just make it so much easier for people to, you know, like see what a code review does because they also know people no longer, they spend less and less time reading it because there's just more of them, right?
25:14You're right. I mean, the human code review is dead, but the human review is not dead. Some review still has to happen. If you look at the software development space, there used to be like a upstream kind of a guard, like when you were like planning the work because your engineering was an expensive resource. So when you were using the tools like linear Jira, one of the reasons to do this agile and sprints was to make sure the developers are working on most important tasks.
25:39The Pragmatic Engineer Host: We don't like spend a bunch of time and effort building something that was thrown away. Exactly. So there was a lot of regulation upstream earlier. Now that floodgate has suddenly opened with AI. Now you can go from idea to code without going through planning. Like you have a support ticket coming in. People will just ask DevIn to look at it or CodeRabit Slackboard or something, right? At the same time, your designers and people who were not software engineers are now doing coding as they should be because they know what to build. A lot of designers are shipping code. You have background agents like Datadog is now going to be opening PRs.
26:09Each time it sees an alert, it's going to say, I'm going to do it for you, right?
26:12The Pragmatic Engineer Host: Yeah. So what that means is that your planning layer and the review gate still has to exist somewhere. And that becomes a pull request because everything upstream, there's an influx now happening, right? And when you have a pull request open, it doesn't mean that you have to ship it. At that point, you decide whether this is worth spending my time, whether I take it to production or not. Because the human taste is the bottleneck now, human judgment. So when a human is coming in, they're first making a decision. Is this something I need to ship? Is it slop I need to close? I mean, and you make a decision after the code is there, right?
26:42Second thing you're doing is like, what's the risk? Can it just go through production without me looking at the code? Or do I need to understand the broader implication? Is it like a database migration it is doing? Why is it doing the migration? You want to sometimes look at the schema. So what, because that's more structural change, right? So we do all that like risk management within CodeRabbit. So some PRs are on fast track to production. We are even letting our interns ship directly to production. So human code review is dead, but I would say humans still review some of the higher order abstraction.
27:12and how we are like doing this with SOC 2 compliance is also interesting. So we don't, so the definition of change at CodeRabbit is now a release. So we have a Slack message. We kind of get all the indicators around the risk we are taking. Each PR, you have a minimal risk, high risk. And over there, you kind of make a decision on whether this can go or some things have to be rolled back. So each release is what we are reviewing now, not PRs, as a human.
27:37The Pragmatic Engineer Host: And then when you say human review, just to be clear, human means software engineer or can it also mean non-software engineer? I'm just curious if you still think the accountability needs to sit with people who kind of know the internals or can it be someone else? Can it be product manager, designer? I think it's still software engineers that code rapid. Like they're not necessarily looking at the code anymore, but they are actually understanding the risk that we are taking with each reel. So it's about who can understand the risk. Because if something goes bad, then they are going to be the one dealing with it, like rolling back, like it takes like 10, 15 minutes to roll back.
28:08I mean, that's the worst that can happen if something goes bad. But they are still taking the responsibility. So we ideally want like developers to be around when something goes out so they can babysit what they've just shipped. That's why we try to release so frequently. But at the same time, like with the person who's approving the release, we have SREs and all. So they also like are taking some responsibility at that point.
28:27The Pragmatic Engineer Host: I think it makes perfect sense, especially for a business. Because we're talking about a business where you actually have revenue, users, you know, like if Slop goes out and it takes down production, you're going to just lose customers or goodwill or cost frustration and so on. So I think it's great that you're saying, okay, let's have someone accountable. But the problem that us devs have is there's just a lot more changes and I'm not going to have time to read through all of them. What ways are you thinking either now or in the future? What do you think needs to change so that as engineers, as the person who's making the decision, I can just have better information to, you know, like focus my attention where I need to focus.
29:03The Pragmatic Engineer Host: Like, for example, I don't really want to review a manual refactoring that just move things around and it compels. I mean, I could, but, you know, there's good use of my time and there's poor use of my time. And I feel, as you will know from yourself as well, I mean, you've shipped like so many pull requests. You will probably focus your time on the critical stuff. That's right. I think the human judgment is scarce now, right? And you want to like, first of all, not look at the code. There's higher order abstraction. you want to understand the risk, understand the blast radius especially if you are familiar with some area of the code you might also want to have inputs on the architectural change like for example if you're writing a migration on a large table like it's going to crash in production you have some of these like tribal knowledge you want to bring in right?
29:46Other thing is like also like I kind of see like now driving these agents almost like getting a driver's license right so we basically even on the developers put some responsibility on them who are shipping. So we let people ship if they are in the review exempt list. So we also have different lists and groups and teams where based on their experience and their past performance, they can either go to production without a human review or we put them in a group where they need a human review.
30:12The Pragmatic Engineer Host: That sounds like based on a trust that you built up with them. The way we have done that is we do look at how long there was a regression in production. Because we look at the code, when you fix something, we are able to tie that back into the PR that introduced it. And now if someone has been irresponsible, like they did hard prompting there or they did not follow through their change into production, you will have sometimes 100 hours of exposure. I mean, someone had to point out, a customer had to point out that something regressed, you didn't QA it or something. And once you have like enough strikes, we take your driving license away.
30:43We make you go through the code review. Then you go through like a proving period with some human reviews and then we let you ship on your own. So we're trying to like set up a culture where like I think responsibilities also with the developers to not drive these agents off the cliff.
30:57The Pragmatic Engineer Host: I like this driving license analogy, honestly, because we talk a lot about, well, I talk a lot about or worry about quality because I can see that quality seems to have dipped across the industry. And clearly it's because there's a lot more things going on and a lot less review. And it's nice to tie back, you know, the feedback loop that look like I think it's human, right? Like the more you ship, the more likely you'll mess up. And as long as you own it or there's consequences or you say like, OK, from now on for this time period, either come up with better practices or have a second person, have a second pair of eyes, which is expensive.
31:29The Pragmatic Engineer Host: But this is what we used to do even before AI, right? The intern for you, you give the trustee intern to ship, but we didn't have it. The intern needed to have like two sign offs, for example. Right. I mean, it's almost like that. I think people have to have responsibility. Like we have seen some developers are really good in asking the AI the right questions around the implementation they just generated. they're more diligent. And some developers will come and give like an ambiguous high level goal. They will say, just solve this problem for me. And AI will like completely re-architect it. Sometimes we have to pull the plug when we start shipping, seeing those PR show, we're saying, okay, we need to stop that right away.
32:02We have seen those kinds of things. So some people are like good, they've been prompting and steering these agents and others are just reckless. And that's a behavior we then try to penalize.
32:11The Pragmatic Engineer Host: Yeah, I do like the idea of responsibility as well. But there's a question of like, why, if the AI will get even better, right? Like we're just speculating here. But at that point, why would you pay software engineers? And I kind of keep coming back to the idea that, well, you know, why do I pay a plumber to come when I could just watch a YouTube video and I could fix my toilet? The problem is that, you know, like nine out of 10 times, it probably works. But the 10 times I made it, it might either clog it or break it or something and i'm paying the professional because if they come in and they mess it up they're responsible and they will fix it and i'm kind of thinking that as software professionals you know like we will be responsible and and uh you know one of the reasons we will hopefully be paid the big bucks is is we can take responsibility for a complex system where an outage is just very uh costly right no i think the responsibility to some extent matters like the things you can't foresee.
33:09We all have to be reasonable with responsibility. Software are still like complex systems. They break in very complex ways, right? And you saw that today, the tweet around like railway employee hacking and then they blame agents going off track. But in the end, like it's really human who have to be responsible with how they're driving these agents, right? And they can do your bidding, but they can also go in crazy directions, right? I think that's important. And the other thing is like with this kind of, when you're shipping at agentic speed, your operations also have to be agentic. So one of the things we are agreeing is that, okay, our goal is not to prevent every outage.
33:43It's to make sure that the exposure time of something bad happening in production is as little as possible. Which also means that agents need to be looking at production.
33:51The Pragmatic Engineer Host: Yes. If that's out of lockstep, you can't ship fast. You're going to be more like AWS. You burn your hands with AI, and you're going to say, ban AI from my code base. What you're saying, as I understand, is, okay, we'll have agentic feedback to verify the code or give AI code review, whatever that is, or run maybe false tests, etc. But the observability layer should also likely be tied. It's all part of the QA. It has to be because your defect rates maybe are not going down, but now you're shipping more defects because the volume has gone up, right? Which means that you have to stay on top of them.
34:25Like you have to have agents always looking at not just alerts. In fact, alerts are limited by your imagination on the failure scenarios you imagine. Like actually these agents are really good at looking at logs. They're great looking at traces, right? So the way, in fact, those are the areas we are still not like fully scaled yet. Like there's so many areas of our code where we have had longer exposure in production with bad code because no agent was looking at it. Wherever we have agents looking, we are able to quickly resolve stuff at agentic speeds as well. So entire software development from left to right is now going to be driven by agents, including production and including marketing.
35:04Even stuff going out, you have to now be agent-a-ground.
35:06The Pragmatic Engineer Host: Harjit just said how software development is going to be fully driven by agents. And when agents write, ship, and fix most of the code, you want to know why each change happened, which brings us to our seasoned sponsor, Entire. A few weeks ago, I told you that Entire was building a Git forge. Well, it's finally here. You can now host private repos directly in Entire in the US, Europe, UK, India, Australia, Brazil, and Singapore. And Entire has just reinvented a pull request for the agentic era. The problem with PRs is how agents write faster than how any of us can review it. So Entire replaced it with something called trails.
35:38The Pragmatic Engineer Host: A trail connects everything about the work in one place. The intent, the code changes, the agent sessions. A trail can be spread across multiple repos, so the change to your backend and your web app gets reviewed once. With the trail, you can configure customized evals. They look like heads of display in an airplane and score your work on every push. And you can see how each agent run passes or fails your user-defined gates and decide when you merge based on these. The point is that you control when you allow agents to be allowed to merge the code, like only merging after a human review or after all gates pass.
36:11The Pragmatic Engineer Host: Oh, and one more thing. Entire ship continuous memory. Entire turns your repo into a brain so your agents can search past coding sessions to find the context behind earlier changes. Basically answer the why behind code changes. Memories get shared across the team and across the repo. If you want to see what an AI native GitForge looks like, head to entire.io slash pragmatic and check out trails. And once agents are observing production and pushing fixes, someone has to decide what they're allowed to touch, which is why I'd also like to mention our season sponsor, WorkOS. WorkOS provides the author infrastructure B2B and AI native companies need to sell to enterprise customers with SSO, SCIM, RBAC, and now authorization for AI agents.
36:52The Pragmatic Engineer Host: WorkOS Airlock is intent-based access control. Most access control only asks who you are. Airlock also looks at what the agent is trying to do at the moment it tries to do it. Clear-cut rules like no agent deletes cloud resources are run as plain code. For the fuzzier calls an AI judge evaluates the policy at runtime. You write these policies in plain English and Claude can pull the ones you already had out of Notion or Jira and import them for you. Your agents, whether that's Claude or Devin, connect to a single access point and your MCB servers are behind this access point. Airlock is a new layer on top of your existing permissions which means that an agent can never get more access than the person who invoked the agent has.
37:31The Pragmatic Engineer Host: When something risky comes up, Airlock can escalate to human for approval, say over Slack. The idea is to write the rules once and spend human attention only on the calls that need it. You can request early access at workquals.com slash airlock. And with this, let's get back to Harjit and what happens to code quality when agents write most of the code. Let's talk a little bit about quality because in your previous startups, you know, pre-AI, you've built some infrastructure tooling, which, you know, reliability is very important at those places. Accuracy, when you have the monitoring, I assume these are all non-negotiable.
38:03The Pragmatic Engineer Host: But what's happening now with AI, what we're seeing is when you're using AI to generate code, it's great. It does it faster than I could do, usually similar or better quality than what I could have done. But it does get verbose. It can just repeat itself. It can just add a lot of tech depth. You know, like once you start generating enough, you see that it's just gone a bit more bloated or a lot more bloated. What is your observations here on how did you see this used to be for earlier models? Have newer models changed it? And is it still a problem? and what are you doing about it? Yeah, I mean, once you unleash the agents on your code base, there is some chaos.
38:40I mean, there's a saying like, let the chaos reign and then you reign in the chaos. That's a philosophy. So rather than prevent the entropy, the entropy is going to increase of the code base. Maybe your security posture also weakens because you're not reusing the same auth layers or something, like reusing stuff. Then you need to have agents which are self-maintaining that code, the janitorial, like the Roombas, right? So that's the approach we are taking. let people ship quickly let some of the chaos rain maybe things are not scalable right now but then you have the agents also cleaning up the mess like all the way from flaky tests to bad abstractions to refactor the code for example recently we ran agents to introduce effect and xstate into our code base so refactors constantly keep happening right so you only make sense of that mess once you actually have the mess there like then you figure out what's the best way so I think both have to be like it's like raining in the chaos
39:33The Pragmatic Engineer Host: as they say and then rain in the chaos right yeah and i know you you previously talked about things like de-slopping tests yeah can you tell me what what is that is is that a skill is that a oh it's massive like i'll tell you our ci has gone through the roof like i think we're probably spending as much on on ci as tokens and you're spending a lot on tokens yes and also on ci like ci i heard last time like vendor quoted us like 13 million or something a year we built our own by the way now in one week. Yeah, but that would have been your run rate if you would have stayed on the vendor. That's right.
40:05So the thing is like these agents are now, because they're like writing code and CLI commands, they're driving a lot of compute. And all these code changes, you're still going to make them go through CI gates, your merge queues and everything. And you're in fact writing more linters, more simulations, because you want to be sure. There are two problems there. One is like a lot of these tests are actually useless. So you already have agents which are de-slopping the tests, removing unit tests, doing more integration end-to-end tests, redundant tests, like we have those running all the time. Now at the same time, it's a constant battle, like your CI time goes up, but you want to keep them within three minutes ideally, like that's how much, I think we are like roughly four minutes right now, and at the same time, then you're sharding and all like the costs go up, and this is the reason like one of the things is like at our scale, like it It makes sense to just build our own.
40:53So we built our own CI, right? Running. Right now, what we are hitting is like compute capacity.
40:59The Pragmatic Engineer Host: Are we talking about CPU capacity? Even GCP, unfortunately, has very little compute available. Anthropics took it all. So we were in the central region, and we couldn't even upgrade our database instance. We run out of compute. And we had to like migrate. And you're like, just to be clear, you're not the smallest of companies. I mean, you will not be a Google, but you have plenty of budget. you have good funding like you're you know you're ready to give the money to well yeah we have signed good contracts with them but unfortunately they're all locked in by the large because they're all reading sandboxes rl environments i don't know what they're scaling yeah but they're also not just scaling gpus anymore but they're even the ram so because i i wrote about the the cpu crunch and in the primatic engineer a few months ago but you're a good example where it's it's happening right it is happening in a big way right now we have a big cpu crunch as well and it's just like a few companies shipping at that scale.
41:50Like, I mean, for the rest of the world to get there, I think the biggest bottleneck is going to be the data centers, the energy, and those kind of things. The models are commoditized. Like, the models are there, but they drive a lot of that compute as they work.
42:01The Pragmatic Engineer Host: But now, a little bit talking about this CPU crunch. So, because I think this is, I'm super curious to learn more on both what you can do to get more CPUs and what you can do to maybe use your CPUs more efficiently, right? You mentioned Rust, and I wonder if this is about it, because Rust is as efficient as you can get your code down to, right? That's what we did. Like, I mean, we were on TypeScript earlier. Like, we just moved to Bunn. I mean, there was just another. I was in Tahoe on a beach. I ran a goal. And over the weekend, all the services were migrated to Bunn. Dude, your idea of holiday.
42:32Yeah, I mean, I have the pictures of the team. I was so surprised. I just ran a goal command. And all the services moved to Bunn over the weekend.
42:38The Pragmatic Engineer Host: So just so I understand. So you have this thing where you set up a goal. And then you do a long-running session in the cloud or on your local computer? Unfortunately, it's local. Like, I mean, we do have a cloud agent, but I should be the one using it, but I have like a powerful machine, so I always go and... And then it goes off and it runs on your machine and it runs long-running tasks and it usually completes it. It does. Like, I mean, it will sometimes like stall to ask a few questions, but some of it is like, interestingly, the distributed systems are now far easier to build. They don't need much hand-holding because there are only a few ways to get something right when it comes to the architecture.
43:14The product is like open-ended, like the UI you can build in 10 different ways, And that's where the taste, the judgment comes in a lot more. But we used to pay a lot for people doing complex scale distributed systems, things like Kubernetes and advanced caching observability scale, right? To be honest, those things are super cheap now.
43:30The Pragmatic Engineer Host: So you're saying that that part, that kind of this backend, even though they're complex, has been commoditized thanks to the models? Yeah, because they're like, as the saying goes, like there's only maybe one way of building Unix. If you were to build it today, you're going to come up with the same abstractions. Yeah, because going back to, for example, Kubernetes. I had Kat Cosgrove on from the Kubernetes team, but she was saying how Kubernetes is, on one end, it's complex to onboard to because it's complicated, but they also have really good documentation. And I guess when you have something like this, and there's only, I guess, I assume there's only one or maybe two ways to do things, but when you have this, agents are pretty good.
44:03The Pragmatic Engineer Host: When there's good documentation, no matter how complex it is, they can navigate the complexity. Those kind of things they're like great at. Like things like, okay, I'll optimize my Docker files. They can just keep running it against Orbstack locally and optimize it. optimize my CI cache or do this bundt migration. Those kind of things are super easy right now. I mean, if you have good unit tests and testing, like you can validate that, hey, things are coming up correctly and the smoke tests are passing, you can even try on production. And usually things are working. It's not like we are shipping a lot more defects.
44:31Our defect rate, I would say, is going down. I mean, this might be a hot take, but like humans were writing slop, not AI. I mean, that's what I'm realizing. Like AI is actually writing less slop at this point if it can validate against something.
44:42The Pragmatic Engineer Host: So you're saying that in the past, we shipped, like most teams would ship kind of slop or low quality stuff when we didn't validate it. And to be honest, if we're just being honest, we didn't always validate it, right? We had more outages in the past. I mean, that's my observation. I still have to measure it. But I would say when we were shipping with hand. And you're talking about just CodeRabbit, right? Now I'm seeing actually fewer outages. The more our code base becomes agent ready, the better validation layers become. We are seeing less defects, not more actually, as the velocity is going up.
45:10The Pragmatic Engineer Host: So one more thing about quality is evals. Because of your business of offering a tool that gives you AI code review, model quality will be really important and costs and all of those things. How do you evaluate new model releases? And I'm obviously talking about the big ones, the open AI, the Entraffic, but also open models and so on. Because I assume it's kind of a big deal for you to know how good of this model is in terms of the quality, the cost, latency, and so on. And that's right. I mean, right now we are also bottlenecked on our evals capacity. Like, I mean, the models, the cadence that they are coming out every few weeks now, because the RL thing has taken off, right?
45:51So people are coming up with back to back releases, the training runs are shorter and shorter, right? which means that, I mean, every day we are like evaluating something. And they're like, of course, three or four classes of open AI models, anthropic models. And we have like an ensemble of models in our products. So it's like, unlike other products, we don't let the user drop down and choose a model. Like we do it for them. We know which sub-agents, which workloads, the model is going to be a good fit. So a lot of the time that our applied AI team spends is figuring out the price to performance of each model and the workload to the point that we have zero bandwidth to even onboard another vendor.
46:28Like we are a big token spender. We have been working with like, even you can see the Gemini's end up, but we don't have bandwidth. That's been a big constraint, even with the open models. I know they're getting good enough. Each time we think that, okay, we could go use these open models. OpenAIR brings down their models in price. Like Luna is a great model. Like given the price point that that's being offered. Sounds like you're using it a lot. Yeah, we use a lot of Luna, like in our product. We use other models. We use expensive ones for deep inference and reasoning, but a lot of code exploration, a lot of the context preparation is done by cheaper models these days, right?
46:59But the trick is, like, the open source is lucrative. The only thing it's helping is, like, putting downward pressure overall in the industry. We ourselves didn't get, like, much time to go and evaluate those.
47:10The Pragmatic Engineer Host: Do you build your own models? Do you either fine-tune or do some, like, post-tuning, or have you experimented with this? Early days we did, and that was an interesting realization that the frontier was moving faster than we could maintain those fine-tuned functions. I mean, the shelf life of those investments were very short. So it was a little bit better for a short time, and then a new model came, and boom, your advantage was gone. Then you're in a different zone altogether, right? And the other thing is we are in the coding space where I think other people have plenty of data. That's what they're running on.
47:43I mean, if we, let's say, had really very different data stream, then maybe it makes sense because the models are not good at training on that. But if you look in coding, code review, they're trained to some extent. Where the real trick comes in is in fine-tuning the hardness. There are two parts of fine-tuning. One is a model. I think the bigger problem is, like, what's the right context? Because these reasoning models can go off track if they overthink. I mean, you might have noticed extra is actually sometimes worse. It overthinks, right? Same thing. Like, I mean, depending on the context, we use a lot of like semantic analysis of the code, code graphs to make sure the token costs are like lower, while we also get very high quality leads to go and track.
48:20So that's where like the effort goes in, is in fine tuning the hardness. And we have very large open source user base. So we are like very popular in public repositories.
48:28The Pragmatic Engineer Host: Yeah, yeah. You have this thing where you make it really easy for open source projects to use CodeRabbit, right? Yeah. And a big reason for that was the feedback loop, because with open source, each review that happens every few seconds goes back into our pipeline to fine tune the hardness. So basically, just from a business perspective, I mean, you're losing money on every one of these because you're paying for it, but you're getting it because now you're able to make your product better, which hopefully it will be more valuable on the long term, right? That's right. And this could be a win-win.
48:58Yeah. So the reason we have the price to, and this is like one of the very few use cases by the industry where a specialized harness is still applicable.
49:06The Pragmatic Engineer Host: Like everything else is getting eaten by general purpose cloud code, even legal finance, like it's doing general purpose work. But code review is one area where it's a breadth and a depth problem. You need a good recall precision. And over there, like a custom harness is very effective. And this is where like all this open source data is helping us tune that. I mean, I have a couple of agents just running, looking at open source all the time. One more thing about quality is you mentioned briefly formal methods. I'm interested. When did you start to experiment with it? You mentioned TLA plus and lean.
49:39The Pragmatic Engineer Host: How is it going? Where are you using it? Because when I talk with Hillel Wayne, he was on the podcast and TLA expert, you all know him. He said that he thinks formal methods, especially TLA plus is probably an overkill for almost software even with AI, although he did say this about six months ago, so the progress was not there. I'm just really curious on where you're seeing success. So we started using it back in February or March when we built the Slack bot product. So that product had like some distributed states, like it had a queue, like queue up the messages, you could steer the messages.
50:15It had like complex states around like shutting down the sandboxes, like there was a lot of state machine there on the warm idle state and so on. So that's where that TLA plus models, they're not great for like things which are like algorithms. They're great when it comes to like messages and actors and interactions. And that's what like we used because at 5.3 codex wasn't as great as now Astra. Now I think it might make pure mistakes, but the way we made 5.3 codex successful was first model it, make sure the state machines were correctly done before writing the code. And we did find issues in the initial implementation after we deployed TLA plus.
50:48It could find all these like race conditions which we had overlooked. Yeah, simulations also helped, by the way. So there were two techniques we invested back in February or March. We built like an entire like simulation harness where we could like fuzz and try like different interleavings. At the same time, we use TLA plus to model different states and explore those states and then find bugs.
51:09The Pragmatic Engineer Host: Nice. And I want to go back to the, not your commits, but the cold rabbit commits. And this is the weekly commits, which are kind of similar to yours, except it's more granular. you can see, but there's a huge inflection point in August, very similar to how you went. It went from about less than a thousand commits per maybe like 500 or 300 in July to more than 5 ,000, we're talking weekly commits at the last data point. I wanted to ask, what about the cost of AI tools? You obviously have AI code review tools, you have the AI generation tools. Where is it at right now, roughly? So our AI cost, I mean, mine alone would be$250 ,000 a month,$275 ,000, something like that.
51:54The Pragmatic Engineer Host: Wow. You're really spending it. Yes. I mean, that's just me alone. And some of the people on the team are shipping close to me right now. And a lot of the also like the number two code commit is actually the CodeRabbit's background agent. So we also have like the agents looking at health and just creating PRs and we just take them to completion. So we are spending like several million dollars a month. on tokens and then we are also spending a lot on the ci cd right i want to sandbox environments and it sounds like that was the ci cd was on track to be a million a month but you now brought it in-house so you probably decreased it up a little bit 50 roughly is what i'm hearing i have to see like but we just shifted like over the weekend this weekend all all to the local ci now okay so for almost everyone else this was absolutely freaked them out like having this thing and they were thinking, how can I rein it in?
52:43The Pragmatic Engineer Host: I also heard a lot of, I did research on startups and how their costs went. And for most of them, the costs spiked in August with Opus 5.5 and then it went down. Most companies are kind of just using it for coding. They're not running as many loops as you're doing. How are you thinking about the cost of AI and why are you not nervous as a CEO and a founder? So I think you're going to use both kinds of models like this. So if you have a good model routing layer then of course for some of the most of the tasks you can even use cheaper models but some of the things you still have to bring in the expensive model the other aspect is the fast mode like you pay double with ultra fast those are really great right i mean as a startup you want to get there faster than the next competitor right then the thing is like even though ai is getting commoditized but it's now driving the second order effect of the the ci cost and the cpu crunch we are seeing right i think that's real that's probably not going to go way unless you are getting rid of unit tests or finding some other way of validation.
53:39Some people are doing local validation and using that attestation to pass the central CI. There could be some tricks, but the fact is the software factories are going to be expensive. Earlier, when I wrote that tweet, I know that I got a pushback on that tweet.
53:55The Pragmatic Engineer Host: We can talk about it. Let's maybe talk about this tweet. This tweet, I'll also put it on the screen, but I'll read it to you. You quote tweeted this founder or engineer who shared how they're spending 32 million per year in clotspan building this massive multiplayer game and you said that what we're seeing at code rabbit and elsewhere is this parable trajectory that even large seed rounds like 8 million plus are no longer enough to build the software business software factories need quality tokens at large scale compute to run ci sandboxes and other infrastructure to ship software at scale at this point some smaller startups exist only to funnel ideas to larger startups with more tokens there's a lot of spicy stuff in here.
54:30The Pragmatic Engineer Host: But let's talk about, I know X is not necessarily great for capturing the nuance, but you had an observation there. You said that you are seeing this trend. Is it startups spending a lot on software factories? I think that's going to be inevitable. There are multiple software businesses you can run, but if you're looking at hyper growth, VC funded startup in these competitive spaces, you would probably need more capital right now because now your bottlenecks have shifted. Earlier, you needed to pay mainly for the talent. So you'll get ex-Netflix, ex-Google, ex-Facebook engineers. They themselves will be founders.
55:03They will hire from the same pool. Give them nice talk, some really good laptops. And that's it. It was all the brain game. The IQ was the people.
55:13The Pragmatic Engineer Host: But you needed to pay for them, right? In Silicon Valley, the base salary would have been$300K plus and stock and all that. We're talking a million plus. The salary was the bigger piece in terms of running a business. In early stages, I'm I'm talking about seed series A, even before you hire sales. Like a lot of it is like the team, like the dev team, right? Now what we're seeing is like the IQs on the tap, like the tokens. Like in fact, even the smartest people need to be driving tokens. And a single person like me, I can drive two or$3 million of tokens a year, like blended models, not just always using the best model.
55:44And perhaps it should even be more once we have the factory concept taking off. So my point here was like the bottlenecks are now shifting in how much you can like drive these tokens number of agents the CI and the other infrastructure humans can make a lot more decisions in a day to drive millions of dollars spent now you need like it's almost like marketing team if you remember like I mean 10 people can drive 100 million dollars of spend same thing we're going to see with engineers now a small team can drive 150 million dollars of spend of 30 million in that company right and that's how you stay competitive because now you're going to compete with everyone who's writing software in an inefficient way not in an economically viable way.
56:23If you're going to mass produce, you're going to see like companies, composite startups now pin up, which would be a brand, but with like many products, they're going to mass produce just like how you do in a factory.
56:33The Pragmatic Engineer Host: I feel you're still a huge exception in everyone I talk to, including inside of NVIDIA, there's a monthly budget of$10 ,000 and you can go above that, but most devs I talk to don't go about that. So I'll just play a little bit of devil's advocate with you. So you're spending a lot on AI, like way more than most. You're also a founder, so you have a really good idea of where the business is and what value is adding. What value do you think you and people spending this much is adding in terms of like, does this mean that you're clear you're moving faster, but what is this resulting in? Are you getting more customers?
57:09The Pragmatic Engineer Host: Are customers spending more? Are you just, I don't know, getting more free user? You see what I mean? Because you're in a great position, usually for an engineer, it will be maybe harder to connect, but you can clearly connect like what either you're seeing right now or what you're expecting to see as a result of this investment and spend. Yeah, all of the above. Like one of the things you're going to see is like this point SaaS solution. The days are over. Like now you're going to see really like companies doing a lot more software, right? And this is adding value in all parts of our stack. First of all, like, I mean, as I mentioned, like earlier, I would like rely on vendors for some of the things.
57:40Now you can just build in-house much cheaper. There are many tools like CI we moved in-house. We moved vector databases in-house, a bunch of this stuff. Those are very clear areas. you can like go and build those differentiators quickly, right? And the other thing is even for the market, like the people have infinite cap appetite for software and these solutions, right? So there are companies which are going to like, for example, we are in the SDLC space. So all of our customers want to go and ship faster. Code review is not the only bottleneck they're running. Now they're on the market looking for the next thing, which is how do I solve my CI bottleneck?
58:08How do I do this? Like, so we're in that business. So we have to, first of all, go and learn quickly where the bottlenecks are going to be. I mean, we're getting arrows in the background, whatever, right? but what that also puts us uniquely in a position is that we can offer these products faster to the customers but even everyone wants to ship faster like that's the thing with with the with the vc funded companies right so that the the kind of like the range of tech you want to go quickly build that's your usually or more like i mean you have like a few months now the problem with the with the software right now is like even like so a lot of these startups they used to have a moat of a few months before they got copied but now they're getting copied within days and weeks so you can't just go to the market in this competitive market with idea, which is shallow.
58:49Like you, you, you get like even, um, and tropics going to copy you in like two weeks right now. Oh,
58:55The Pragmatic Engineer Host: so I see where you're coming from, but I'd still want to push back on the fact that my sense is, okay, you're spending this crazy amount of money. You know, you personally are spending or you're on track to spend like 2 million per year, or let's just talk about per month. Like you're, you're, you and the team maybe are spending 500 K in this month. Maybe next month it'll be 600, maybe then 700. but in return what you're getting is you know you you are in the dev tool space with ai code review you're now realizing oh dang we also have this bottleneck with ci maybe we'll do something with ci or help our customers do something with ci and we do this because we're now so fast we're seeing where they're going to be in like six months or 12 months so now we're now going to start this product we'll spend more then we realize maybe it'll be a feature flag and we'll be able to roll out and so on and by doing this you're almost like doing r &d for your business and you're also building your products.
59:44The Pragmatic Engineer Host: And the bet is that you'll be expanding your market, you'll sell more to customers. But what I'm wondering is, are you not a little bit unique that this might be true for anyone who's in a dev tools market, but for some company who's not? In any other market as well. So basically now you're moving in a compressed timeframe. Let's say it took you like a while to go to market and do a PMF, six, nine months if you're an YC company, right? Now you can like go there and try that idea quickly. And if it's a shallow idea, now the risk is you can get copied. so basically we probably have to like do more but you are now moving at a compressed time frame the earlier the bottlenecks were in scaling the team like so i needed to have the talent you will like get all this VC money scale the team i don't need to scale the team at the same level right i mean maybe the sales still need to scale because they have to do the meetings but product marketing engineering you can now do few people burning a lot more tokens right and this token span right now if you and it's quality tokens like you can get cheap tokens it will take you longer to get there, if you have the quality tokens, your price per task will be high, but you get there faster at a high quality, right?
1:00:48The Pragmatic Engineer Host: Okay. So what I'm hearing is what you're actually saying with this tweet, which wasn't as nuances. If you are in the VC game, meaning you're raising venture capital because you want to build a billion dollar company, which means you actually want to win your market. Like inside of DevTools, you might have a niche, may that be like AI code review or AIS or CICD, something like that, if you're starting today, you will probably need to raise a larger amount because you need to win this quickly. The game will be over pretty quickly, and you want to get meaningful traction before the big guys and Tropic OpenAI can maybe copy you.
1:01:25The Pragmatic Engineer Host: Oh, and as an added benefit, you can have a small team to start with, and you don't need to worry too much about hiring. But by the time you would hire anyway, you will want to build some solutions and get customers and so on. So everything is just compressed and you can pay more. And when the attention is compressed, like if you don't get the mindshare after building your MVP in maybe one or two months, that's the window you have now before someone just gobbles up that idea and runs with it. Okay, so that's pretty brutal. Yeah, I mean, that's the situation right now we're dealing with. The window of opportunity to get the mindshare has become shorter.
1:01:58It used to be that you had smart people, they had built something, right? and maybe it was a lot of product iterations. It was a SaaS product with a lot of like graft and care or it was a complex backend system which just had a moat, like people couldn't replicate easily. So they would have like a shelf life of, I don't know, six to nine months before something. And they had that breathing room to get the mind share. But now it's like few weeks, unfortunately. And the company that's like spending a lot on tokens and all can get there faster. So you're going to see more companies like Rippling, which was a composite startup.
1:02:30You're going to see a lot more composite companies with these very efficient factories and the processes, gobbling up a lot of that software market. You will still have people tailoring at the last mile, but if you're buying these SaaS solutions and all, that's where you're going to see a big disruption.
1:02:44The Pragmatic Engineer Host: Okay, I think this is fascinating insight. I mean, just a caveat, you're seeing this in Silicon Valley where everything has accelerated, always has been compared to the rest of the world. Let's say Germany will be always a lot slower, right? We've seen that with the Industrial Revolution, same thing happened. I mean, one example is, for example, toys. Like an example, like earlier people used to make metal toys and very good craft. When I grew up getting those Matchbox cars, like all metal, pretty good. Oh, yeah, the old Matchbox. They were a lot heavier, right? When I was a kid, like they were very heavy, really good quality.
1:03:12But now you're seeing plastic. But mass produce, the market has appetite for that. So, yes, the quality is going to go down. Right. But the companies that are able to invest in these factories, which need now actually as much investment as running a factory. It's almost like building a steel plant now. 50 million out of the gate. are going to pass-produce software.
1:03:31The Pragmatic Engineer Host: But just to push you a little bit more on that, because I really want to push you on this here, these software factories probably cost so much because there's not a turnkey solution. And let's say, given that everything is so fast, in a year, there might already be software factories you can buy. But you're saying that by that time, I mean, first of all, everyone can have access to it. And your window of opportunity might have closed for a lot of businesses. Software factories are expensive right now, tokens. But then CPU, all the, like we talked about the CI, Like you can drive a lot of spend just with the velocity alone.
1:04:01A single person cannot drive it. Right. So the economics are going to force you into that factory model. It's almost like how we've seen in the industry. Even like, let's take another example, like watches. Like when the quartz watches came out, you could mass produce and then you pay for Rolex, which is crafted. Now, the question is like, would artisanal software, which is very well crafted, people took a lot of time, will command a premium? Or is the market just going to be OK with generic looking, maybe slightly lower quality software, which I think is what the case is. And that software is going to be designed in a factory.
1:04:32Yeah.
1:04:33The Pragmatic Engineer Host: No, I see where you're coming from. I really want to disagree with you in a lot of ways, especially, but I see your point and I think I have a hard time disagreeing with the fact that, look, if you do want to win a market right now, you want to give yourself the best time, the best opportunity. And if the theory is that to win, you want to be either first or fast and iterate and get it right, you have less time because an interesting thing that i'm also seeing which which is a one more thing we could reflect on is so far we've only talked about we assume that your composition is either the big labs who you know see what everyone else is doing and they launch something or the other startups who some of them raise more money and might be faster than you in their software factory but the reality is everyone else is a competitor like ramp launched their coding agent spotify is launching a platform for feature flags so you have the existing tech companies who are also realizing, hang on, we built this tool inside because we're now big and we see and we think there might be an appetite.
1:05:32The Pragmatic Engineer Host: And what you said is every small company, like 50 % company like yourself will need the tools that Netflix has. Netflix at some point might say, like, hang on, let's open source. So now I feel there's just a lot more competition for the same thing, just coding harness. Again, what have you thought that a credit card company would have a coding agent and people take it seriously? Right. And the thing is that whoever is not on this parabolic trajectory you just saw, they're going to get disrupted. A lot of people are not even ready for what we're seeing already. Like their culture is not going to let them.
1:06:01So the way they're doing software development, like they have to rethink everything. Like first of all, how many people they need. Like people who are just good at writing code, they should probably look for other jobs if they don't know what to build. Like people who are not software engineers, like artists, people who have taste, maybe should start coding. And then how do you allocate budgets? It's like you have to now really think about if you're hiring someone at like half a million a year, expect they're going to spend more than a million on tokens now. So earlier, like when you do this plan to show the VCs, you will have the salary as a main component.
1:06:33But now we're starting to show tokens, right?
1:06:35The Pragmatic Engineer Host: Well, again, I just want to be clear, like this is for like a new startup VC building from zero because for existing companies that have a business, often like physical or in the world. Okay, they could be disrupted, but, you know, they have something stable to be there. And they're the companies who are spending the$200 max plan for everyone, right? I mean, if you have a hardware business and all, you probably have a better moat at this time. But if you're like just software with very high margins, those margins are going to compress. And you probably also have to like rethink. I mean, if you're not on the parabolic, then companies coming from behind.
1:07:02Now, this explanation is scary. I mean, these kind of things start world wars, right? It's that scary. So that kind of companies are going to start coming from behind and start decimating some of these lower dinosaurs unless they go and transform themselves on this parabolic. And those companies, they're actually investing in factories more than talent. They're investing in just, I mean, paying for the best model possible. They're not cutting corners. They're just going fast.
1:07:24The Pragmatic Engineer Host: Let's talk a bit about hiring because this is also a very, very interesting topic. First, can you tell me how did CodeRabbit grow over the years and how has your hiring changed? Because you did mention that you're hiring a lot slower, right? In some areas, yes. In many areas, we are hiring rapidly. Like sales is one area where we have really doubled down on the field sales. because one of the things we found with generative AI is that the last mile is still a problem. People still need a lot of hand-holding in how to set up the systems in the right way. So those kind of things you can't accelerate with AI.
1:07:58You have to have those meetings. You have to have conversations. So those areas we are hiring rapidly, but other areas now we are rethinking what we need to hire for.
1:08:06The Pragmatic Engineer Host: Let's talk about software engineering specifically. Software engineering is the most trickiest area right now. Every month we are learning something. like one of the things we have found is like the problem solving skills system skills like people like who came from like even i wrote a tweet about like i'm looking for people who have played strategy games let me read this tweet because that was really interesting one so so you're you're you are hiring and you said i mean there's a kbeat we're hiring the person who applies must meet one of the following two criterias to join either has tastefully shipped at least 1000 prs in the last 30 days or has played at least 500 hours of simulation games such as civilization age of empires transport tycoon factory etc almost like now if you ask me what the software building looks feels like it's like playing one of those games right now i mean i talk about civilization like each like you basically make the moves and each turn is like a release for me right and you learn something in the next turn and while you are stating the civilization you still have like i don't know workers improving your roads and building fields and so on that's how the agents are and each turn you're like just making these agents work on something meaningful right which we think is a great ROI for your civilization right almost like now running these agents feels to me feels like building if people have grown playing strategy games I think they will be very successful in with these agents right now they can multitask they have like range of things they can I mean at least they have a good problem that these agents can then work on and how is hiring going for you so far how easy or more like hard are you finding to hire these folks who have this kind of experience so the way we are looking at is like either someone who's already a tenured senior engineer at a good place they have to come and first unlearn right or you hire someone even who's like fresh out of college but they have high agency but can learn faster what we found that the people who are like coming with like are not set in the ways and have come especially from the background of like playing this like strategy games and all have been very successful.
1:10:05That's just my take on it. And that's what we're looking for. I think the human agency now matters a lot more than it ever did. Person who was 10x knew what to build is now suddenly becoming 100x with these agents, right? Because now you at least have a problem list or the most impactful areas these agents can work on. And you could just unleash these agents as much as possible in battle.
1:10:25The Pragmatic Engineer Host: So we talk about individuals, but I know you have a concept called pods that you also call as war rooms. That's right. What are our war rooms? So what we found is like that's another area like I think we have like the collaboration has become even more important. So I've like in previous startups done remote teams we are like building distributed systems so it doesn't matter like teams like in India or Eastern Europe have done all. But now like what we're seeing is that now we are moving so fast like somewhere one person is on the steering wheel but other people are shaping the software as it's getting built.
1:10:53And everyone is able to touch the software by the way even product managers get to touch it. And the kind of PRs they're making is relevant to like change a copy on something or some information architecture flows, right? But everyone's like touching the software at the same time. And it's great if they can be like in a wallroom setting. So we kind of create these wallrooms where we share like plans on a Slack canvas. We all agree on it and we all collaborate in a room in a more face-to-face setting.
1:11:17The Pragmatic Engineer Host: So you get everyone in a room, into a table. It's all real time. You kind of close the door and like, all right, let's solve the problem. And we're crafting the software in real time. Like sometimes you will like lay the foundation by running a goal. But when you're like at the stage where you want to like really craft the UI and experience, you want to get people in the room and really craft it live. Yeah, it's funny. At Uber, we used to have up to 2017, something called the War Room, which is like this. There's an important project, get everyone in, sometimes fly people in because they're distributed.
1:11:42The Pragmatic Engineer Host: And then spend maybe a week there, build up. This was pre-AI. So with AI, it's a lot faster. But just build a proof of concept, whiteboard, code, and then have a prototype. And then everyone goes back to their teams and just kind of builds it from there. but it's now working, the core thing. Yeah, now we're doing that more often. That's exactly the idea. And you need people with different tastes. You have product managers in the same room, you have designers, you have people who understand back-end, the front-end, and they are crafting the software together. It is hard work, and sometimes it is late work.
1:12:09The Pragmatic Engineer Host: Because one of our first interactions on X was, I think I was writing something about how teams are working really late up to the evening or into the night, and you reply, it's almost 2.30 a.m., and we're still shipping. and you're like, yeah, yeah, the kind of, you know, like crying smiley face. I'm not sure this is good or bad. But can you tell me a bit on the pace? You talked about you're not sleeping as much before the recording. We talked about how like, okay, at some point you need to get that under control because clearly there's a time for excitement, but also when you're thinking of building something longer term, like you'll want to find the balance, right?
1:12:44The Pragmatic Engineer Host: How are you thinking about it? It's crazy right now. Like, I mean, it does like impact your health to some extent because ever since these agents gotten better, as I said, like, there's like a huge opportunity cost if they are not working all the time and they do stall they have questions you have to like steer them i read the reasoning traces i guide them whenever i have to and for that like i mean i've almost like changed my sleep cycle i mean i take a nap i wake up at like 3 a.m to send like a few instructions because every hour like one or two agents are going to get stuck and you have to go in like it's almost like Roomba is getting stuck you have to like unstuck them that's what it feels like but then then that's kind of become a habit, but it's not sustainable.
1:13:19And we haven't figured out the right interface to work with these agents, fleet of agents at scale. Maybe it's really like a factory, like companies have to have a morning shift, night shift, because instead of owning the code, maybe you're now owning agents. During the day I make the decisions, but I can be rest assured we are seeing the same session, the same agent, that other people can make decisions while I'm sleeping. Right? So we are right now, because we're developing on our own machines, we are like bottlenecked on that collaboration right now.
1:13:45The Pragmatic Engineer Host: Yeah, the one thing I'll add is I wrote this book called Why We Sleep. I read it a while back and it actually forces me to prioritize sleep because there's this thing with sleep, which there's some scientific evidence. I'm not an expert in this, but there's a suggestion that not getting enough sleep now, it's fine for tomorrow and it's fine for next week and it's fine for next month but decades later it can come back and bite you uh and i ever since i don't know how scientifically sound it is but just as when i was like oh it could happen i'm like okay so like me getting a good night's sleep is not only important for me for tomorrow because i'll be you know like you'll know like you won't do as well but for my long-term health like i i you know i i'm starting to hopefully try to do stuff that will make me more healthy when i'm 60 or 70 etc so that's my only kind of like addition to like not sleeping i i would push a little bit like is is it as important to wake up and and steer the agent can it not wait until the morning so i think it's more like a polyphasic sleep i maybe i'm like exaggerating the health but i think even if you look at how people used to work pre-industrial a this nine to five thing is is a recent thing and i mean i come from like agriculture community so people will wake up tend to livestock because you are like working based on other aspects and you have to wake up and do that thing yeah now it's the agents you have to wake up to tend to that livestock, which is now agents, almost like farm well.
1:15:09I don't know whether that's the right analogy. So people used to have polyphasic.
1:15:13The Pragmatic Engineer Host: Or like tamagotchi for a little bit, right? So people used to have polyphasic sleep and I'm doing that right now. Oh, you're right. Like this was a thing in the middle ages. People would wake up in the middle of the night and they would often like dine and stuff like that and then go back to sleep. That's what I'm doing right now. But I will say the only reason I'm doing that right now is because I have to tend to the agents which are on my local machine. Now, once you've really figured out the right abstraction of the factory, once you move to the cloud, then maybe it's going to be like a factory, like nine to five, morning shift, evening shift, or like round the clock, because these are not physical factories.
1:15:44You can have all of the sun model, like we're still figuring out, but right now it looks more like polyphysic with this generation of agents, at least for me.
1:15:52The Pragmatic Engineer Host: Yeah. Let's talk a little bit about burnout. You wrote about burnout, and I'll quote you here. CodeRabbit is my thirst startup now, and I have worked just as hard for my previous two, NetSail and Flux Ninja. Burnout never happens due to hard work. It kicks in when your work does not lead to desired outcome for a long time. On the family front, my two daughters were born during startups. They actually help prevent burnouts. In just your personal experience, like how have you thought about burnout and also just working so damn hard? Because it sounds like, you know, you're on third startup. Like you don't seem to be the, you're the intense type.
1:16:23Yeah, I mean, I really, when the second startup wasn't doing well, I mean, that's where I did feel dragged down, burnt out because there was no outcome. That's the first time I felt the signs of a burnout. Because yes, you were like working very hard, but without a very clear result or an outcome, just waiting. The anxiety kills you. Nothing's happening. You're doing all this work, but nothing's happening. Nothing's moving. So that's a bad scenario to be. And I think that's a status quo, which I would never want to go back to. But for CodeRabbit, like I'm working hard, but things are on the upside.
1:16:53Like we are like seeing the revenue grow like crazy. Our team is growing. The funding's coming in. It's always something happening. We are actually moving the needle as well. customers are loving the product, the new capabilities. So even though I'm working hard, but I don't feel that I'm running out or I have to stop at this point. And I think I can keep going on.
1:17:10The Pragmatic Engineer Host: And I wonder, like, I think in your case, it makes a lot of sense because you were founder in all the companies. So you immediately, you were connected to the outcome or, you know, for example, if the revenue goes up, you know, this is just good because it will help your company grow or raise the next round or get a sale or whatever, you know, like, you know, what can come. I'm wondering, just being empathetic for employees who are not founders themselves, they might have equity, but it will not be as big of a stake. Could it be that having autonomy, having visibility for the business, just knowing how their work is connected, it might actually help?
1:17:43The Pragmatic Engineer Host: Because in startup, people typically work hard and long, especially early stage. But I wonder if the autonomy here and seeing how your work is connected to the things, that actually just helps, you know, get this kind of kick that you're getting, basically. Yeah, it motivates people. Like, first of all, we don't force people to work like I do. I mean, I lead by example. That's what I believe. And even if people can follow to some extent, it's great. But I do what I do. Like, I mean, rather than managing the people, I'm on the ground in the trenches with the developers. If there's incidents, I'm always like first there to help them out no matter what the time.
1:18:16And I think that does motivate people. They're all like happy that, okay, the leaders, right? They feel like much more empathy to their life, right?
1:18:23The Pragmatic Engineer Host: Yeah. Secondly, like I found like the people who join startups are looking for this adventure. Like it's a nonlinear career trajectory. So they are leaving like very predictable salary RSUs in some big company, right? To get like cheaper stock at startups, but then they're looking at the upside. And when that happens, it's like even more inspiring, right? When the companies are on the upward trajectory. So I would say like when the people who go and join the startups are looking for breaking that status quo, which things can be very mundane and boring in a larger setting. Yeah. And you're right.
1:18:53The Pragmatic Engineer Host: Like on a fast growth startup, there are a lot of advantages. Like either, you know, if the fast growth startup goes great and you have equity and there's an exit, you could have life changing amount of money, meaning that you can buy a house. You could have education set up for your kids and you now have the freedom to like either take a risky startup or do a boring job, whatever you want. And if it goes like this and it drops, for example, fast was a typical example in the, I think in 2022, which went bankrupt. You often, if the startup attracted really good people, you know, those people all go to different companies because now their company shut down.
1:19:26The Pragmatic Engineer Host: But there's also often this network, this camaraderie that was there. And they often call each other. Years later, they will, some of them will end up at the next, you know, like really successful company. So I've actually changed my mind. And, okay, the only downside, I guess, for joining a startup is often you will need to work hard. It will be often be in person. But typically when you're young and when you don't have other huge priorities in your life, it can be a huge career boost. It is. Like even financially, I wouldn't say it's crazy. Like that's how the Silicon Valley is. Like I think it does reward people who took that kind of a risk.
1:19:58I mean, there's no fatal failure somehow.
1:20:01The Pragmatic Engineer Host: Yeah. Like and I've seen like people like who just been through that experience have gained a lot in like even like if it's like a bad lesson, at least I know what is not going to work. And that's valued in whatever the next job they're going to go look out for. So you're kind of saying it might be more valuable than sitting in one place, not doing much, not learning much, right? Yeah, that's where you don't grow. But if you go out and do this thing risky, I mean, there's no fatal failure there. Like, people love, in the valley at least, people have taken risks, whether you're founders or early employees, foolish enough to join.
1:20:33But at the same time, people reward that experience, right? And if it's successful, then it's like you're on an accelerated trajectory. Like, and I think that's what I would encourage. Like, even though, yes, I mean, it's becoming a little bit cost prohibitive on the factory front, but still, it's still the best time for people to go out and try this out. Like before all these, like, factories become more established and start blocking some of these individuals. But right now, it's a very magical time right now. Like, even a single person just driving these agents in the right way can create immense value.
1:20:59The Pragmatic Engineer Host: I wanted to ask, looking back pre-AI to like your 25-ish years as a software engineer, as a founder, what are the things that you learned there or skills that you built that you think are really useful and helpful right now working with AI and in this like, I guess, AI era? I mean in terms of like entrepreneurship it's always been like I mean again it's almost like a video game like I mean you're playing all these like games where yes you have to think about like the product market fit I mean all that is exciting I think what I love is like creating value like that's end of the day like software is just means to an end yes you have to write code and I enjoyed it but it was like just means to an end for me like the real thing was like are people like it's really like really like making someone happy with the value you provide is always like the biggest reward, right?
1:21:44Everything is like then follows. Like you will get the VC money, you will get the team and people will join the mission. So it's almost like being a missionary in this kind of a world, like you're spreading a religion. And that's the fun part of doing a startup. Like, I mean, I've always been on the frontier. Like I've seen like early telecom, I've seen SaaS cloud takeoff. And each time it's always, what excites me is like, okay, there's a lot of non-linear value going to be created at the frontier than at other places. And I always try to be at the frontier.
1:22:12The Pragmatic Engineer Host: What would you advise a software engineer who one day would like to work at Code Rabbit or at a place like Code Rabbit, just as a fast moving place? What kind of skills should they invest in? What kind of practices? What kind of activities should they do outside of playing 500 hours of Vectorial? Yeah, I think the agency is the number one thing I'm looking for right now when we interview someone. because right now, like even if they're in a bigger company, because now a lot is happening on the side, if they're not doing side projects today, it's a red flag because right now it's so easy to build that people who have agency are actually building something right now.
1:22:47They can hold themselves back. And that's all always like a great sign. If you have side projects, I love it, right? And the second thing we are looking for is like people who have a range of experience. Now for the depth, you have AI. It can go deep, drill, infinite to performance, optimize something. But if you have like a range of experience, Now you can like build, let's say distributed systems. Let's say you came in from one space doing, let's say, observability databases. Now, if you were to give agents, now you can do 10x of that. You're not just building that, you're building 10 other types of distributed systems.
1:23:20So the range also matters now. So those kind of things I'm looking for, right? I mean, it's like systems thinking, whether they have range in skills and a high agency to keep these agents busy.
1:23:31The Pragmatic Engineer Host: As closing, what's a book or a paper you would recommend that you read and you found it interesting or fascinating? I think we're living in a black swan moment. I read this book. I mean, I'm not like, I spend a lot of time reading books these days. I've just been like building. But I think we are in that like the black swan zone, like what we are seeing right now. And what was the book? I mean, this was like the black swan book. I think I'd read, I forgot who the author was, but it was really. Yeah, we'll put it in the show notes below. Yeah, yeah, yeah. But when there was like pretty insightful of what we are right now going through is kind of a black swan moment.
1:24:00Like if you look at five years back, I would not have predicted that AI would get so good that it's first of all like driving my cars, it's driving the software. And it's like even some of the capabilities we are still like discovering with what these models are capable of. And we have not even hit the peak yet. So, yeah, so everything is going to change. And some of the things we can't even predict today, the way we're going to see the after effects.
1:24:24The Pragmatic Engineer Host: Well, Harjit, thank you. Yeah, thanks. Well, I cannot recall talking with anyone nearly as AI-pilled as Harjit is. To be clear, I was skeptical to start with when hearing about his$200 ,000 AI bill for a single month for a single person. And I remain pretty skeptical about several things. But Harjit explaining his thinking put a few things into perspective. Let's start with the part that I cannot agree with or recommend. Harjit's approach to waking up in the middle of the night to prompt his agents. He calls this polymorphic sleep. But to me, it sounds like deprioritizing sleep all to unblock agents.
1:24:56The Pragmatic Engineer Host: This feels very inefficient to me, and I would rather look at wrong or learning tasks in the cloud if you really want to be there to have your agents running it all the time, even when you sleep. But really, I don't see the obsession with agents always running. This is not a competition about utilizing your local laptop to always run at 100%. The part on how Harjit doesn't think a startup with$8 million of funding has enough money to build a software startup, I'm also skeptical. I get it that when Harjit assumes that all startups will send millions of AI like his startup does with 40 people, you burn through funding fast.
1:25:28The Pragmatic Engineer Host: I also get it that in DevTools, it can make business sense to build new products like a custom CI really fast like CodeRabbit did. But outside of startups that sell AI DevTools, I just don't see it making sense to spend massively on AI just to be a little bit faster to market than your competition. The part I do get, though, is how much competition across AI DevTools startup is there today. It's not just CodeRabbit launching new products with their 40 % engineering team. It's also larger companies like Ramp or Shopify launching DevTools products, AI Labs, and other large companies all trying to get the piece of the same market.
1:26:01The Pragmatic Engineer Host: So with this context, I do get why speed matters, especially for small startups, and why spending on AI to build so faster can make some sense. Finally, one practice I really liked is the concept of deslosing their driving license, if they cause too many regressions. AI pilled or not, I like this idea to force engineers to care more about reliability and shipping quality code. For more deep dives on software factories and engineering practices with AI, check out the link to Primatic Engineer Deep Dives below. If you've enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube.
1:26:32The Pragmatic Engineer Host: A special thank you if you also leave a rating on the show. Thanks, and see you in the next one.
Brought to You By:
• turbopuffer – a vector and full-text search engine built on object storage. It’s fast, cheap, and extremely scalable.
• Entire – The Git forge for agent-human collaboration. Built for the scale and throughput of agentic coding.
• WorkOS – everything you need to make your app enterprise-ready.
—
In this conversation, we step into the world of an AI-pilled CEO to explore how their startup works. This episode might as well be a way of succeeding that contradicts the growing convention that AI spending costs are too high and must be clamped down on.
Harjot Gill is founder and CEO of CodeRabbit, an AI code review business, and he shipped over 5,000 pull requests last month, spending on AI at a monthly run rate of $250,000(!) It’s certainly not a conventional approach and Harjot gets into detail about the thinking and reasoning behind it.
In this episode of The Pragmatic Engineer Podcast, we talk about his background in building software and founding infra startups for the last 30 years, and how things changed into the era of building software factories at CodeRabbit. We talk about why he believes human code review is dead, and how his team maintains quality while shipping at unprecedented speed.
We also get into the rising costs of AI development, why Harjot is convinced that tech startups need more funding than ever, and how AI is changing recruitment and the skills for success.
Timestamps
00:00 Intro
03:17 How Harjot shipped over 5,000 pull requests in a month
08:46 Harjot’s path into tech and his first startups
12:25 Early conviction around AI and founding CodeRabbit
14:40 Switching to AI coding
20:35 Software factories explained
25:37 What’s replacing human code review
38:44 Technical debt and deslopping tests
46:05 Evals
50:25 Formal methods
52:05 AI cost at CodeRabbit
54:57 Startup funding in 2026 and the value of AI spend
1:08:20 Hiring
1:13:03 Work pace, sleep, and burnout
1:21:55 Skills for the AI era
1:24:30 Book recommendation
—
The Pragmatic Engineer deepdives relevant for this episode:
• Inside OpenAI’s agentic software factory
• What is happening with code reviews?
• What’s the ROI of software engineers’ spending on AI?
• Why Ramp built its own in-house coding agent, Inspect
• AI SRE: what is it, how could it change on-call – and does it work?
• Context engineering with Dex Horthy
• Formal methods with Hillel Wayne
• Building Codex with Tibo Sottiaux
—
Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].