1. Podcasts
  2. The Pragmatic Engineer
  3. Uber’s AI infra, from the inside
The Pragmatic Engineer

14 Oct 2026 · 1h 33m · 35 chapters

Episode artwork

Uber’s AI infra, from the inside

In short

Uber’s internal “AI-native” developer infrastructure—how agentic AI is built on top of long-running developer platforms, including internal systems for code, workflows, security, model access, and tool integration.

Guest backgrounds

Udaykira Mendicetti, Uber engineer since May 2015 (joined when Uber had ~600 engineers). Worked on real-time API and mobile protocol, then fulfillment/marketplace platforms. For the last year leads agentic AI work on Uber’s developer platform.

Key claims

  • Production-ready agentic coding is harder than the AI coding harness; the surrounding infrastructure is the real work.
  • AI foundations are designed to be model- and harness-agnostic to handle fast-changing models and tools.
  • Retrofitting AI on “battle-tested” developer infra is easier than starting from scratch.
  • Central gateways provide consistent policy/guardrails, observability, and billing.

Notable examples (internal systems)

UOwn (asset ownership), M3DB (metrics foundation), Michelangelo (ML platform; GenAI foundations), Cadence (durable workflow engine), Morpheus (integrated experimentation/feature flags), DevPod (cloud dev environments), Model Gateway + AI Guard (single entry point for model requests), AIFX (wrapping harnesses with personas/plugins), MCP Gateway (auto-generates tool specs from Thrift/Protobuf APIs; exposes vendor tools), Skills marketplace + quality gates, Context Graph + “Cortana” assistant (grounding via live org graph).

Written by AI. May contain mistakes.

Chapters

0:45

Understanding Uber's AI Development Tools

A discussion on the various internal systems Uber has created to support AI development for engineers.

“This episode is presented by TurboPuffer, Vector, and Full-Text Search built on object storage.”
1:57

Uday's Journey at Uber

Uday shares his experience and the evolution of Uber's engineering team since he joined.

“So I'm super honored to be here and talk to you today about what you're doing.”
2:58

The Evolution of Uber's Microservices

A deep dive into how Uber transitioned from two services to a vast microservices architecture.

“And for those people who didn't work there with us, can we recall a little bit what real-time API or RT API as we used to call it inside, what it was and why it was a big deal and why it had so much load?”
5:34

Uber's Internal Tooling Changes Over Time

Uday discusses the major tools developed at Uber to support engineering needs and their evolution.

“Like we're still running one single global tech stack across all the countries we operate in.”
8:01

Feature Flagging and Experimentation at Uber

An exploration of Uber's unique feature flagging system and its importance in experimentation.

“products tech stacks you worked you still work on on the platform team right?”
14:01

Uber's Sharding and DevPod Development

Learn about Uber's infrastructure for sharding cities and the evolution of DevPod to enhance developer efficiency.

“And we also have like a logical deployment groups where a group of cities are routed to a group of containers.”
17:45

AI Foundations and the Model Gateway

Discover how Uber built the Model Gateway to support AI development and model flexibility.

“Yeah, I wanted to talk a little bit about before we got into what's on today, because I think a lot of things are connected, right?”
20:59

AI Guard and Cost Management Strategies

Explore how Uber ensures security in AI models and manages costs effectively for engineering teams.

“I can show a quick primer into what we had built.”
28:00

Understanding AI Foundations at Uber

Learn how Uber engineers are adapting to new AI technologies and their efficient use.

“And so that engineers are also aware of working in this new age.”
28:40

The MCP Gateway: Centralized Tool Access

Discover how MCP Gateway streamlines access to internal and external tools at Uber.

“This is one gateway for all of the tools.”
Show all 35 chapters
30:30

Integrating CLIs with MCP Tools

Explore the implementation of CLI for MCP tools and its impact on efficiency.

“So these might be the APIs from those services that are projected as MCP tools and we auto-generate the request pack, response pack, the descriptions.”
32:20

Creating a Skill Monorepo for Teamwork

Understand the creation of a skill monorepo to enhance collaboration among teams.

“So when you have it, that's what I'm starting to see with Uber.”
34:30

Improving Skills Through Continuous Evaluation

Learn how Uber evaluates and improves engineering skills continuously.

“So that even if you switch across harnesses, you have these plugins auto-installed.”
36:50

Leveraging the Context Graph for AI Assistance

Discover how the Context Graph enhances AI assistance and improves query efficiency.

39:20

Building an Advanced Knowledge Graph at Uber

Explore the development of a knowledge graph that interconnects multiple data sources.

42:00

AI Assistant at Uber: Overview and Functionality

Explore how Uber's AI assistant integrates with various tools to enhance productivity.

“or our production graph with billions of nodes, they also helped build this graph.”
46:48

Feature Development Lifecycle at Uber

Understand the lifecycle of feature development and the role of AI in enhancing it.

“in terms of tools used by devs at the company.”
52:34

Minion: Uber's Background Coding Agent

Dive into how Uber's Minion automates coding tasks and improves productivity.

“Yeah, so this is showing all of my DevPods.”
56:01

Enhancing Code Review with AI Tools

Learn about Uber's self-healing CI system and the Code Inbox tool that aids engineers in reviewing PRs efficiently.

“I go to this page, I can see all the things that we did so far.”
58:10

The Future of Code Review Processes

Discover the need for human oversight in code reviews and the potential for automation in low-risk PRs.

“which ones might be low risk in so many ways, maybe an impact, maybe in blast radius.”
59:27

Sanity Testing in Engineering Workflows

Explore the manual testing processes for new features and how Uber engineers test changes end-to-end using Slate.

“So when you log in, you are executing that branch.”
1:01:59

The Importance of Distributed Tracing

Understand how distributed tracing enhances the reliability of testing changes across microservices.

“And the same thing we have for iOS, Android, web and if you want to do backend APIs, then you can also test backend APIs.”
1:04:50

Building Robust Testing Frameworks

Learn about Uber's approach to creating a testing framework that captures production traffic and ensures proper validation.

“This thing took like years to add, to make sure that all existing services were retrofitted with this and all new services were built with this in mind.”
1:10:00

Routing Requests in Uber's AI Infrastructure

Learn how Uber's Deputy and Cerberus enhance routing requests within their infrastructure.

“But now that we have Slate, you don't we don't necessarily have to use Deputy and Cerberus because like it's already like you can deploy a Slate environment and we don't need to use any of them.”
1:11:23

Enhancements with Slate and Capri

Discover how Slate and Capri improve agent functionality and session management.

“After we moved to Monorepo, like then we moved to Monorepo because upgrading libraries was very hard, right?”
1:13:15

Automating Library Upgrades with Shepherd and Fox

Explore the automation of library upgrades across modules using AI at Uber.

1:14:36

Continuous Code Improvements with Fox

Learn about the Fox system that automates codebase enhancements regularly.

“you want to create self-improving code bases since we already have a whole bunch of agent skills.”
1:16:32

Bug Reporting and Debugging with Wisdom

Understand how Uber streamlines bug reporting and debugging processes using AI.

“from engineers and automate all of these things.”
1:18:39

Reproducing Bugs Easily with AI Assistance

Learn how AI helps reproduce and resolve bugs more efficiently at Uber.

“talked about yeah hey like coding agent looks good like why don't you create a pr for this minion will take all of that context that we have it creates a pr it will attach that pr to the Jira task.”
1:20:41

Impact of AI on Software Engineering Roles

Explore how the introduction of AI is changing software engineering jobs at Uber.

1:22:05

Measuring Engineering Productivity in the AI Era

Discover new metrics for assessing engineering productivity in light of AI advancements.

1:24:00

AI's Impact on Engineering Practices

Explore how AI is transforming the engineering workflow and feature delivery.

“And how is your mental model changing of software engineering compared to pre-AI and pre-T's tools?”
1:25:10

Evolving Definitions of Hands-On Engineering

Learn how the definition of hands-on engineering is evolving with AI tools.

“I mean, at least in my total tenure in engineering, there's never been this much change to how I work.”
1:26:47

Automating Customer Support with AI

Discover the process of using AI agents in customer support operations.

“Again, that used to take weeks, now it's like days for any policy.”
1:28:48

Automating Personal Tasks with AI Agents

Hear about personal AI tools that help automate daily tasks.

“Then, instead of doing manual evals, then how can you use all of the past context to create synthetic evals on top of that so that you can accelerate, you can deploy this and then verify this quickly.”
A man walking through an airport, asking his phone a question

Ask about this episode

Get the gist, explore an idea or ask about something said.

Claude

On The Pragmatic Engineer, what are the key points of this episode?

Searching your listening

Transcript

Automatic transcript. May contain errors.

0:00What kind of internal tools do software engineers at Uber use to build software today? We get an exclusive insider and visuals-heavy look from one of Uber's longest-tenured engineers, Udaykira Mendicetti. Today, Uday leads much of the work on agentic AI at Uber. In this conversation, we get into Internal developer systems Uber has built and used pre-AI like UOwn, M3DB, Michelangelo, Cadence, Morpheus, and others. AI foundational systems, a company built like its MCP Gateway, AI Guard, and AIFX. Recent systems used by nearly all engineers like Minion, CodeInbox, Slate, Shepard, Wisdom, and others.

0:33If you want to understand what AI native development looks inside of a modern tech company with thousands of engineers like Uber, and why it's much easier to retrofit AI on top of battle-tested developer infra, this episode is for you. This episode is presented by TurboPuffer, Vector, and Full-Text Search built on object storage. It's fast, cheap, and extremely scalable. Today's episode is about the close to two dozen internal systems that Uber has built to help engineers ship with AI agents. One of the takeaways will be that production-ready code is far more complicated than just using an AI coding harness.

1:04The harness is the easy part. What's hard is all the things around them. This is where I need to mention our season sponsor, WorkOS. One piece that is always a hard problem with agents is permissions. Do you want an agent to have standing access to everything it might touch? Or do you just approve every single prompt until you just stop reading the prompts? WorkOS built Airlock for this. Airlock is an authorization layer that sits between your agents and the tools they call, and it judges every single request against the agent's intent and your rules. You write the rules in plain English, agents never see your credentials, and every verdict is logged.

1:40It works with coding agents like Cloud Code and Codex. WorkOS Airlock is how agents do real work without over-permissioning. Check it out at workos.com slash airlock. Uday, welcome to the podcast. Thank you so much for having me. I've been following your work. It's amazing to see the kind of impact you've had and all of the great content you publish for all of the engineers. So I'm super honored to be here and talk to you today about what you're doing. Well, we worked together well before the Pragmatic Engineer was born back at Uber. Not together together, but I knew of your work as well. When did you join Uber exactly?

2:16Yeah, so I joined Uber in May 2015. In fact, like the two engineers who referred me, they're still here. so I joined in our real-time API team it was during the period where we were trying to break our monoliths into microservices and I was working on our API gateway for all of our mobile apps and I worked on changing our mobile protocol since then I worked on many different business platforms from our fulfillment, Uber for Business on our earner onboarding, marketplace pricing and the last one year has been all agentic AI both for our users, customer support, end users and so on So yeah, it's been like a roller coaster last one year on working on this.

2:57Yeah. And for those people who didn't work there with us, can we recall a little bit what real-time API or RT API as we used to call it inside, what it was and why it was a big deal and why it had so much load? So when I joined, we had just two services at Uber. It was called Dispatch written in Node.js and API written in Python. dispatch just to handle all of the rider and driver traffic at that point in time and the protocol was pretty simple at that point it was like a four second ping protocol we used to call that would give you the entire state of the world and when the trip is completed we would hand off to api it would handle all of the post trip processing payments receipts and everything rtapi was born at that point in time to abstract all of the internal services and mobile and also changing the protocol from simple four second protocol to back like where we have both rest apis and we also have server to client communication we used to call it push protocol so we would because so much of our app is very dynamic everything keeps changing every few seconds so we needed a way to efficiently update the state of the app across both riders and drivers as and when we get new updates as and when things change in physical world so it used to be at that point like one of the highest QPS services.

4:15And over the years, we kind of, because all of our stack was just Node.js and Python, so that was the natural choice at that point. But in the subsequent years, we have converted that into Go and it's still one of our highest QPS services at Uber. And I think for people listening, they're going to be surprised when they hear like, oh, there was only two services in 2015 or around that time at Uber, because of course Uber was known for the microservices boom. And Tuan was on this podcast. He talked a little bit about this, but would you mind just recalling how those microservices came to be and from your perspective, you know, like what it meant to suddenly have so many services?

4:49Yeah, I mean, I think there's like so many blogs and like so many people talk about that. So I think like when I joined, right, like we were, as I said, like we had just two services and there, and you also think going back in those days, the kind of hyper growth phase where we were in, we were constantly expanding to new countries, we were constantly expanding a product portfolio across mobility and later we started Uber Eats and that also was expanding. We needed a mechanism for teams to really build all kinds of functionality very quickly to match up to the scale at which the business was growing.

5:23So we have invested a lot in all kinds of service infrastructure, making it easy for teams to build services, metrics, tracing, monitoring, observability, deployment, the whole bunch of stuff around that. And there's not much off-the-shelf software for these things at that point at the scale and rate of change that we had and there's a lot of stuff that we had to build in those days internally and that created a boom cycle of a lot of we went from like two services to order of thousands of services then around 2018-2019 was when we standardized on our languages we picked go and java as our primary languages we modernized a lot of our tech stack we spent a lot of time in thinking about the layers of our architecture how do we platformize and over the years in fact when i worked on one of our fulfillment platform we actually reduced some of the services to few services even now we are kind of in the similar ballpark of number of services even though our business is 10 times larger we are doing many many more products and this whole bunch of platformization and years of investment that has gone into it.

6:31Like we're still running one single global tech stack across all the countries we operate in. There's many, many platforms that serve all of the mobility verticals and all of the delivery verticals using one shared building block. And I think that kind of, I love such to build more mobility verticals, more delivery verticals, because we have platformized a lot over the years. Would you mind telling us when you joined, how big was Uber's engineering team and how did it go over time? because I don't think people from the outside appreciate just how big that growth was. So when I joined, I was around the first 600 engineers.

7:10And in fact, the whole of marketplace team at that point used to be like maybe 50, 60 engineers. And over the next few years, as we've scaled our business, as we scaled our traffic, we grew to a few thousands of engineers. and around 2019 2020 we started also expanding across different sites now we have 12 global tech sites across the world in almost every continent we operate in in fact and that helps us build like products for those hyper local markets and some of those products are then rolled out globally across all the markets where we could roll out that that kind of functionality so i think that gives us a lot of ability to like you know tap into talent globally and also build nuanced local products and that also helps us build like one global tech stack.

8:00And speaking of building products tech stacks you worked you still work on on the platform team right? Yeah so at least like until my first 10 years I worked on business different business platform teams and in the last one year I'm more spending my time on on a developer platform on a agenti ki for engineers Agendic AI for the rest of the company. And in the process, I've also worked on some user-facing AI initiatives as well. Can you take us back to 2015 on how Uber's internal tooling changed? Because again, I was there and I was really surprised to see just how much custom tooling we had. We talked about RT API, but there was things like Morpheus, Edge Gateway and other things.

8:43Would you mind talking through like back in 2015, 2016, if an engineer joined Uber, what kind of tools did they use and how did those evolve? I'm talking obviously the main ones that most engineers back under mobile engineers touched at Uber. Yeah, absolutely. I think in those days, given the way in which we were scaling, there's not many off-the-shelf products that we could tap into. So I think we talked about RTPI and Edge Gateway. Some of the other stuff that we had to build was a system called You Own that will enable anyone in the company to identify who owns a specific asset. Like it was like services, right?

9:22And you could look at during on-call, if there's an outage, you could figure out which team owns it, who's on call for that kind of stuff, right? Yeah, absolutely. And as the engineering size grew, we also had to build like end-to-end like our engineering design docs, product design docs, getting all of those into one single place. We had to work on getting all of our incident manager into one single place. We are like really obsessed about how we handle our incidents, how the learnings from that. so we built a lot of really good tooling around that I think there was some of the more complex ones was our metrics like even back in 2015-2016 we used to probably have around like 500 million metrics emitted per second with with the cardinality and there's not there's no off-the-shelf metric system that we could use and that gave rise to m3db which was open sourced later and like there's even companies formed around that um and um that enabled us to build like the kind of like a handle kind of metrics volume that we had we also had to build our own end-to-end machine learning platform because we used to have like hundreds of ml models and if you take the whole life cycle from creating the features building the models monitoring serving this is michelangelo right michelangelo and like we recently just a few couple of months ago we open sourced our deep learning and like part of the michelangelo infrastructure so there's a lot of like really good solid foundational building blocks there.

10:48The other interesting one is also Cadence, which is if you think about some of the workflows, right, like they span across multiple days, whether it's a driver onboarding, whether it could be some CRM campaign, you could see a lot of engineers doing their own resiliency stuff, their own state management. And that gave rise to Cadence, which is a durable workflow engine. And later even that got open sourced. I think not many people will know Cadence, but a lot of people will know Temporal, right? Which was, as I understand, either inspired by Cadence or a lot of the ideas came from there. I'm not even sure it was the open source version of Cadence, but it had a lot of, like, it's kind of the external version of whatever Cadence is for Uber or used to be.

11:29Yeah. And I think those folks like this saw, like, you know, there's a real need for something like this. And they were able to go and like, you know, build, like, scale it even further beyond Uber. But I think that the pattern that I'm trying to say like there's a whole bunch of things that we had to like invent in those days for the scale that we had and over the years then we are like now as the industry also mature like now we have few few products that we build so some where we have we think there's some unique need we buy and if we build anything if we think that's interesting for the rest of the industry then we try to open source wherever possible yeah i think two more tools that i think is worth mentioning because i just remember them as one is morpheus are it's basically launched directly for uber but it was way more integrated into everywhere right like you could you could go in there and run experiments you could see the results immediately it was almost like feature flagging and experimentation but integrated into mobile feature flags and it was like i remember inside uber like all of these integration it was just integrated into a bunch of tools all of them right yeah and and it would and since i was since we were also expanding to so many countries like you need a really good configuration platform, whether it's for like configuring our backend services or also having feature flag system and also experimentation.

12:44And Uber is a very data-driven company and like everything goes through a feature flag, everything goes through an XP. Experiment. So then, and since we have also, it's not, and since we are two-sided marketplace and three-sided marketplace, some of the experiments are also not like really complex. It's not just simple AB. We also have switchbacks and we have to build really deep experimentation that's baked into all of our tools and it can automatically generate analysis and what are the p-values, what metrics to use. And that also led to working a lot on our metrics foundations, data foundations, so that whatever analysis that we generate from this metrics are trustworthy and we can make business decisions based on that.

13:27Yeah, and I remember I was always wondering, why do we have our own feature flag system? But then I was thinking like, show me a feature flagging solution from a vendor that supports cities. In fact, cities are first class citizens because inside Uber, I think cities were so important. And a lot of our tools had cities as, for example, for cardinality, every city had its own metrics. And then you start to realize like, oh, very few things treat, like Uber had a bunch of interesting constraints, right? Like that no one else had at this scale. Yeah, absolutely. Yeah, I think the city one is an interesting one.

14:00it's like so deeply baked into like our infrastructure like for example some of our like critical platform services we also do sharding by city because then we can also like observe across different tiers of cities because like we need to make sure like even now like across all of the regions where we serve our traffic we shard a city to a specific region We can also at any point in time fail over that city across regions. And we also have like a logical deployment groups where a group of cities are routed to a group of containers. And this is for few services. We don't do it for all the services.

14:42We do it for some of the services that are where we need that level of sharding and that level of reliability. One other system I wanted to ask you about that I remember using, and it was being developed in 2017, I believe, which is DevPod. And it felt really ahead of the industry at the time. How did that come along and what did it turn into? Yeah, so I think the backstory for DevPod is because we already talked about the growth of microservices. Initially, we had micro-reposts where every service was its own repository. And we didn't have a standard build system at that point. each service had their own like and we had at least two three build systems two three testing systems and it was very hard for miss even for a central team when we wanted to do library upgrades go across all the services and do upgrades so around 2017-18 we we we then changed the strategy to getting to mono repos even though they are micro services but all of the services were deployed from the mono repos we had one lang one mono repo per language so we had a mono repo for go one for java one for mobile ios one for android and so on then once the scale of that repository was pretty big enough then like it was pretty hard to run the build and like getting all of those systems set up on your local laptop also then you have to like go through all kinds of setup issues um so we took on this thing of like devpod which is like cloud developer boxes and and you can have any number of them we had one dev pod per monorepo so if I'm working primarily in Java monorepo I create the Java version of the flavor of the dev pod and it has all the build system cached I can have faster builds it's working on cloud machines as a beefier than even the laptops so then you could have like we could have really good SLA's on your build time on your test time and we don't have to worry about the setup issues because we can uh we can install the right tools in place and now the interesting part is as we got to the ai era like even the agents need the similar workspaces and cloud boxes and we are able to kind of latch on to that and we also created like now a mega dev pod with all of the repositories because then i can have my agents create prs across the repositories it has all the tools it has to work in any repository to look at context across any repository this also helped us to create the agents md cloud md at the top report at the top folder because we have standardized builds standardized testing once we set up the top level files most teams will get like really good value out of the bat they don't have to like really go into per module setup and like on the dev pod all of the ai tools are installed by default so you can get a new dev pod You don't have to go through even installing like Cloud Code, Codex, Open Code.

17:33They're all there in the dev pod. So this made it really easy even for non-engineers in the company to create a dev pod and start playing with Cloud Code, start playing with Codex out of the bat. Yeah. So I guess like having cloud development environments just like really early helped a lot. Yeah. I think all of those years of platformization bets also helped a lot as we expanded on our agentic journey because you could use all the things that we built, making the life easier for engineers and it makes it easy for our agents as well. Yeah, I wanted to talk a little bit about before we got into what's on today, because I think a lot of things are connected, right?

18:12So like there was so many things being built for platforms, for tools, often custom for Uber. But having talked enough about that, what does it look like today? What does development look like today? What kind of tools do you use? And maybe you could give us a little bit of like a primer of like the foundations that you put in place specifically for AI and what you built and what you built. why first thing is especially in the ai world one thing we realized very early on was because the landscape is changing pretty quickly harnesses are changing pretty quickly models are changing pretty quickly we wanted to invest in the right set of building blocks that are hardness and model agnostic so that no matter any of these things change we can easily port across different models different harnesses so few things that we invested like so we started working on a model gateway from the initial introduction of ChatGPT like late 2013, early 2014.

19:08So we had a model gateway back then, which would expose an OpenAI compatible API or Anthropic compatible API. Sorry, so like this is 2023, 2024, right? Yeah, so 2023, 2020, early 2024, when ChatGPT was first launched and GPT models were first launched. And we quickly noticed, you know, we wanted to make it easy for any teams in the company who want to like leverage and build agents with these models. And so we had a model gateway that could abstract all of the complexity behind where we had all of the security and policy guards in place. We had all of the visibility in place. So you could switch across different models or different vendors for these models.

19:49And you could just have the ID for the organization or the project, and then you can use any of the models exposed. So this is what you just expose it as like a model selector and it piped into the model gateway and it routed to the right, you know, may that be chat GPT at the time or Opus at the time, that kind of those models. Yeah, and it was built on Michelangelo itself. So we create a Michelangelo Gen AI. Oh, Michelangelo, but that's an ML platform, right? Yeah, and then now there's a Gen AI set of foundations on Michelangelo platform. So then you go to create a Gen AI project or a Gen TKI project on Michelangelo.

20:27then you can set up the project you can select the model that you want you can you can then you'll get a unique project uid that then you can use for your agents so then you have all of the same like you're able to use the same michelangelo infrastructure then to start building on ai specific evals and like stuff on the same on the same foundations looking back i think that was pretty early investment that gave a lot of ROI later. What is Model Gateway? I can show a quick primer into what we had built. And this was late 2023 when the first set of LLMs were launched. And we quickly realized that we had a whole bunch of internal teams wanting to use it for different use cases.

21:14And we wanted to make it really easy for them. And this was built on top of the Michelangelo platform and one more component inside that they can set up there. And Michelangelo, just to recap, that's the ML platform that was built from like 2015, 2016. So it was built a long time before LLMs. Exactly. And now we have like a Gen AI or Agentic AI set of components also inside Michelangelo. And we're also building a lot more right now that those are not in the open source version yet. But we are actively building a lot of stuff for Agentic AI world. But my model gateway was the first such one. And the initial version, we didn't have all of these middle version components.

21:55And we built some of these over the last one, two years. But the experience for any team in the company is, you know, they create a client. They give their project ID, which they initialized in Michelangelo. And then the caller token. And that's all. And we exposed one OpenAI compatible API or Anthropic compatible API. And all of the models across all the vendors, open-weight or closed-weight, they're all behind this gateway. And the first thing we added there was what we call AI guard. What is AI guard? Essentially, we wanted to make sure we are not sending any sensitive content back to the user.

22:35So this was one entry point where we could add this middleware that we can enforce consistently across all the projects. and security team had ability to like you know then they could iterate on all of the AI guard specific models whether it's on content whether it's on safety policy whether it's on like whole bunch of comms policy there are like five different set of fine-tuned models that they built there which would which runs on every request and every response with strict latency guard rails this helped a lot to make sure that all of our projects have same standard baselines and then And because every project is tracked here, we could have like per user, per caller attribution.

23:16We could have like then as we progressed later this year, we were able to add spent years for projects, spent years for users. And since all the requests go through one place, we could also add like audit logging, session traces and a whole bunch of infrastructure around this. Every model request at Uber goes through this. So there is no model request that doesn't go through this. So that gives us like one single entry point for across all the models. And I assume like you can just try billing as well, like costs and all of those things. Exactly. Like track billing, cache hit rate and all kinds of things.

23:51And now, you know, as we progress into like there's a lot of discussion of like, you know, model routing and all of that. I think this could be like a good entry point that we can leverage because all of our models go through this. I think on the same journey of like being able to make it easy for engineers to switch across models. the next thing was how do we make it easy for engineers to switch across harnesses like if i'm using cloud code then if i switch to codex then i need to like carry forward all of my settings all the skills all the tools uh all the plugins um so we internally wrap all of the harnesses with a wrapper called ai fx ai foundations and this is pre-installed on devpods or you can install on your local machine and we have a concept of personas so then that personas comes with whole bunch of default plugins and skills so that you don't even have to install like any engineer they get a whole bunch of default skills that they can use out of the bat since we wrap everything all of the settings are managed so even if you're using any of the hardnesses on your local or on devpod and or any of your dev pod the settings are synced so then centrally we can manage one set of settings that we can propagate across all the users and any user specific settings they are synced across all of their environments and again this makes it really easy because now we have one entry point for switching the harnesses switching the tools and then we also made it like there's a status line where we can give tips where we can have tracking of your spend across harnesses within that harness in that session.

25:27So all of those additional things that we could add without any explicit setup from the engineers. Nice. And then speaking of tracking spend, how do you approach that right now? Like as an engineer, I do, or if I just join as a, you know, like a senior engineer somewhere, like do I need to be aware of spend, of costs? What's the thinking there? See, we want to make sure engineers have access to any of the tools that are available and we have this notion of spent years so by default you get certain certain tier and then then we have many levels of tiers once you hit once you are close to that tier then then you can raise a request to go to the to the next year and you just need a simple manager approval so then you can discuss with the manager the roi for that project like how are you spending and it is a conversation around responsible use of this right but we don't want to restrict engineers like even if they like if they're working on like some uh if this is going to the right project use they should be able to spend uh whatever they need because ultimately we want to maximize we want to maximize the gain from this but you also want to make sure there's a responsible use uh so we try to do the minimum minimum amount of like you know paperwork needed to like cross the tiers and they will see like the the across any of our tools how much they're spending within that hardness and across all the hardnesses and we also have different tiers for different managed agents um so like then for all of the interactive tiers interactive harnesses we have some spent tier for some of our background coding agents we have a separate independent tier so we make sure like they can easily manage across all of these yeah but but i guess it just makes sense even if I do my side projects I will be aware of my own spend on my whatever limit I have and I guess I guess do I sense it correctly that the thinking is just don't use tokens on stupid stuff that is that makes no sense or or accidentally you know you might be just burning and you don't know so like just just get that feedback like these things are not free exactly like I think just having that feedback and we also have like a plugin called cost check that will run across all of your sessions and give you personalized tips based on your usage to say hey like this is the kind of thing this much you're spending on this project and this session this is these are the tips that you can use to further improve your cost profile and i think all of this creates a good conversation like for between managers and engineers but also engineers will also see like good feedback into like how they are efficiently spending across i think and then that helped us a lot with like you know like like all of the real-time coaching with context health, model sizing.

28:19And so that engineers are also aware of working in this new age. We also don't want inefficient spend. And what other foundations that you built are related to AI? MCP gave way. Oh, yeah. I think the other backstory was around, I think, early 2025. We started as Cursor kind of picked up. like we enabled cursor and mcp was um was any was a spec that was like created around that time and we quickly realized that hey the power of we can really increase the power of ai by exposing all of our internal tools in an easy way across things so then then we could potentially leverage this beyond just coding we could leverage it for all kinds of software engineering tasks so around April of 2025, we started working on MCP gateway, similar to model gateway.

29:15This is one gateway for all of the tools. So what we did was because we already had good IDL registry across all of our services, whether they're using Thrift or Proto, since we have the ideal specs, we auto-generate the MCP specs for any of these services. And just to add, I guess, if you also don't know, like uber services either emit thrift or product product buffers as strong contracts right yeah and like we initially we are just thrift and later we added support for protobuf as well but all of the new services in the last few years they're all protobuf the ones that were created like a few years back they're all in thrift and as they go through rewrites usually like team switch to proto but whether they're in thrift or proto like we we would have we since we have all of the ideals in the in one single registry we could use that registry to generate a default mcp spec for them so they could anyone in the company they can go to this tool mcp gateway they can see all of the mcp servers that we auto generated and if you go to any of that you could see all of the tools that were generated from that existing API.

30:31So these might be the APIs from those services that are projected as MCP tools and we auto-generate the request pack, response pack, the descriptions. Later, we also based on the API descriptions, you generate LLM optimized descriptions for the API sand fields so that we can, like we don't like we saw some tools with like 3 ,000, 4 ,000 tokens so then we can optimize it for model consumption and it is just one single config and and security policy change and then they can enable it they don't have to do any code changes if they want any custom mcp they can always lose we have like a framework for creating a custom mcp and and proxy their custom mcp also through mcp gateway the other interesting thing we did was even all of our vendor tools whether it's Google, Slack, all of them are also exposed through MCP Gateway.

31:27This reduced a lot of burden on configuration management since we always spoke about AIFX, right? So I could install any of the tools in the same manner, whether it's internal MCP tools or external MCP tools. In the last few months, we also projected all of these MCPs as CLIs. so you could execute any mcp as a cli so then if some agentic hardness wants to use it as cli they can use it as cli if they want to use traditional mcp they can use mcp and do i understand that this all comes because the apis to start with were already either generated or written in like a lot of them were generated that you already had a platform for the apis so adding mcp was somewhat i mean trivial i mean of course it's work but you know what i mean and now generating CLIs is also somewhat trivial because you had a foundation before AI, right?

32:20So when you have it, that's what I'm starting to see with Uber. You built a bunch of stuff before 2022 and you could just add AI infra pretty easily on it. Yeah, and the same team that works on our API Gateway also worked on MCP Gateway, so they're already in the business of building API Gateway. They know how to do any kind of protocol transformations, any kind of any kind of vpi gateways so it was like we were able to pretty quickly build this so i think this was probably like especially once we got our google workspace and all of our slack tools also into this it kind of like created all kinds of like amazing agent workflows people could automate anything right at that point like from slack to google jira linear like any any tool vendor tool or any internal service is you are at a CLI call away and then the other set of foundations as we like started thinking about this space was see I think all of these go in this camp of like vendor neutral hardness neutral building blocks that we think like you know as like you would have to do this irrespective of what happens because you would need agents need some way to access your tools then the other thing was around skills like you know we gemmed onto this earlier this year since we already have this notion of monorepos we created a skill monorepo in some sense that has all of the core skills that central platform teams build and all of the team specific skills individual teams can build and are these skills these are just like the the skills folders like the plugins that you can use for for cloud code etc exactly like so there's plugins and skills that you can use across all the harnesses then we had like one single uh like quality gates that can run across all of them.

34:09And we made it portable across all of the harnesses. So you install with the same AIFX tooling that I mentioned. And like if you do plug-in add that skill, it will update the configurations across all of the harnesses, whether you are at that point using that harness or not. So that even if you switch across harnesses, you have these plugins auto-installed. So you don't have to worry about like reinstalling them and reconfiguring them and so on. And since all of the skills are using the tools from the MCP gateway, we also have one standard mechanism for all these skills to access any of the tools.

34:49Now we are working on like, you know, some tools where engineers can go select their skill. It will tell them all kinds of tips to improve their skills. The next thing that in flight that you're working is how do we then collect the traces from the hardnesses. and generate improvement patches for the skills so that we can continuously improve the skills based on learnings because there's a lot of discussion around the skill quality, skill evals and how do we improve that continuously like when you first build a skill they might not be perfect and you need some mechanism to continuously improve that.

35:24I think both the MCP and skills this gave us like lot of like benefits and like if I look at like in the last few weeks there's around probably 30 000 skill executions per day this is kind of interesting for me to see because if you told someone like oh we're building a skills repo like what most people would think inside a startup right i'm not talking about uber but like oh you just do a get just a repo and you have a bunch of skills there just a bunch of skill folders and and that's kind of it but it's pretty clear that it's more complex right is it more complex because of how many people, engineers you need to support?

36:04Is it because you're also running evals? Because you keep cleansing these skills or tracking if they're used or not used, those kind of things. Yeah, absolutely. I think now with any of these latest models, it's very easy to create a V0 version of the skill. It's very hard to make it really good across all kinds of use cases. And that's why you want to really make sure like you know we have one one place we can like update all of these skills in fact when we exposed the CLI projection for all of our MCPs just couple of engineers using AI they were they updated all of the skills in this repository to use the CLI pattern because it's more token efficient because if every skill says I want MCP ABC MCP and EF then if I install these two plugins I'm gonna have like six MCPs across these two always in my context and it is so much context load right and like then we replaced all the skills with the CLI pattern so that that had it like a lot of impact on our like token efficiency as well and the other thing is because we auto install this skill marketplace for everyone it reduces one more step of configuration and so then like this the discoverability is easy the setup and configuration and management is easy and we could build a whole bunch of like eval pipelines around that discoverability pipelines around that we still have teams that have like their own repositories but majority of the skills go through this my favorite is uh like context graph um what is context graph the backstory is back in December like one of the engineers built a AI assistant that you can go to like we call it Cortana like you can go to web or you can go to slack and ask any question from any code base so it would do agent it would like search across all of our repositories and answer questions and then we go and then we saw people asking all kinds of questions like it could be research it could be like what is how to run this query like what is the data source for this who owns this like can you do RCA for this whole bunch of software engineering tasks and when we're looking at some of these traces like it was clearly obvious that you know we could optimize the number of turns it takes for the models to get grounding because we have so many systems like historically all of these are like siloed systems right like engineer has to go into like the the task management platform engineer has to go to our like on-call dashboard has to go to like our your own dashboard that we talked about then we're like okay how do we plug all of that into one mental map for the agents in just in the last few months and there's a cool visualization like they call it uber context graph like we have uh 100 plus node types like 130 plus edge types and this is live data as of like say like like three days ago if you click on any person node like you know you know what assets they manage what design docs they wrote what features they're working on what on call events they like worked on what are the configuration changes what incidents they managed what jira events and if you take any node it has a relationship to all other nodes and what this allows us is now the same assistant i talked about it has the context graph as a default grounding so if you ask any question it will first use this to like navigate okay like what is this person asking about once you get grounded into oh like this is about this domain this set of services then it can go deep and like read other documents read other things so it reduces your execute your your agent traces and the amount of turns it needs to give you a good answer and we already see like in a whole bunch of categories of questions the quality goes up the cost goes down the latency goes down and we have just in we're still in early in harnessing the full potential of this and like we are super excited about like all all of the things that we can build on top of this going forward and then the notes are they people are they tools this is the schema visualization so every person in the company will have like you know like there's a note for me note for like every employee and then it will have relationships to all of the instances like every pr that i generate there will be a note for that and there's a link saying okay would i created this pr they manages this asset i work on this product feature i contributed to this design doc so then the nodes are both people but they're also assets yeah exactly like one simple example is let's say if you ask any question like how many mobility trips around the world are with cash payment type like you first need to understand what tables exist what columns exist are there other people who wrote these queries are they from science team are they from mobility science like now all of this data is interconnected like you can go and check oh are there other queries that were done in the last 30 days around this and what kind of tables exist go and explore those columns because again all of this is uh interlinked then you can quickly get to like the blessed queries that we have and answer these kind of even complex questions pretty quickly otherwise you would have to like go through many many docs many many like tool calls um this makes it really efficient like it's much harder for us to manually write all of these cipher queries but agents are really good at navigating the graph like how they're really good at navigating file system with bash this seems very futuristic to me but it's pretty cool and i assume it's a ton of data to store just because uber has like 10 000 plus people working there easily so yeah the good part is uh you know similar to how our API Edge Gateway team worked on MCP Gateway effort, our graph platform team that manages graph or our production graph with billions of nodes, they also helped build this graph.

42:08So this is, in compared to our production graph, this is much smaller so this is, so it's not like a scale challenge for them anymore. Yeah, and we also, I already talked about like the AI assistant Cortana which kind of plugs some of all of the context craft tools skills and it's omnipresent in slack web cli so anyone in the company like even someone in ops in some country they can ask any question about oh how is this built where is this located who works on this what was the history behind this and like it is able to take all of this data all of our google documents and then and then give them like a really good answer and it breaks down kind of the information silos across orgs right like they're able to like self-serve ask anything execute any skill anyone has built and i guess it must have like access to the data sources so for at least for the right right people so it can go into like things like logs and and yeah monitoring those kind of things yeah it has logs monitoring you can ask okay how what is the health of this service it can go look at all of its dependencies what kind of storage systems it uses run all the queries and give you like a 360 degree health report for that asset and we carry forward all of the existing user permissions so like so then if i'll only able to i'm only able to access the stuff that i had access to otherwise we will that data is not accessible yeah yeah because inside uber i remember it used to be called break glass when you added a permission for a system you didn't have access to you go through an approval flow which goes through your manager or director or whatnot but i know uber was was really big at my time they were really big on this already yeah absolutely so i think those These are some of the foundations, we have some more in the workings, we are thinking about some more futuristic use cases.

43:55Some of these building blocks really helped us in our agentic journey and helping it take it from just pure, using it for just coding to all kinds of engineering tasks. Would I just mention how Uber wrapped up AI foundations work but already more foundations work is to come? Our presenting sponsor Turbo Puffer knows this all too well. They are rebuilding their storage architecture from first principles. They are doing this to make more types of storage faster, cheaper, and more reliable at scale. The new system called TurboPover v3 is not yet in prod, but it already passes CI. Still, it's not nearly as performant as the current production version, v2, and so the TPub team is now perf-maxing query plans until they are as fast or faster than v2.

44:39They're doing this in public, documenting how they increase performance all the way up to the v3 production long. For example, we can see how the team increased full-text search performance 9x by batching reads. A lot of this is basic foundational work right now, but they will be sharing some very cool systems work over the next few weeks. But hang on, why is the team doing a full rewrite? It's because search is changing, and so is TurboBuffer. T5 started as a vector database, but it is no longer VectorDB. It has evolved into a high-performance database for search. The V3 discards the Vector Primary Index and all indexes will share the same columnar storage system.

45:15Every query will run on vectorized query engines, which is something you don't usually see in search databases, but which will result in both very fast search and just generally a very fast database. The TurboPuffer team is keeping a work log of their journey and you can follow along at TurboPuffer.com. It's a blast to see how it's being built. I also want to mention our season sponsor Lanier, the place to coordinate work between humans as well as agents. One of the biggest changes for us software engineers is the sheer amount of code that we need to review. Uber has built custom tools to deal with this, but so has linear.

45:48It's called Linear Diffs and it lets engineers review PRs right inside of linear. One thing I really like about Linear Diffs is how a diff sits next to an issue inside a project and so it's super simple to see all of the context in one place and know why the change has happened without needing to jump through different tools. For large PRs, guided reviews walk you through the core change with an explanation of what has changed. Guided reviews also group the glue code and the supporting edits separately. These small but important changes make a large PR readable in one sitting. And linear also supports stacked PRs.

46:21If you view stacked PRs, you'll know that they're a great way to break up complex work. And if you've not yet used them, you'll probably love them when you try them out. How we do code reviews keeps changing as AI agents take on more of the work. These neat changes coming from Linear help us engineers understand code reviews more and expect Linear to evolve diffs to keep up with how we work more with AI agents. To learn more and to get six months off of Linear, head to linear.app.pragmatic. With this, let's get back to Uber and how building a feature looks like today in terms of tools used by devs at the company.

46:54PRD and ERD, I guess for those who don't know, product requirements documents and ERD was engineering requirements documents, right? The product spec and engineering plan for those not familiar inside Uber. Yeah, so I think we went through some of the AI foundations and now if you think about the lifecycle of a feature, then how can we truly uplevel each of those lifecycle with the right set of AI foundations and also rethinking those tools for the AI era. And some we are more farther ahead, some we are early in the discussion. like i think even the notion of a combined plan that agents and engineers can iteratively work to work together we like get in um there's like a lot of cool ideas but this is kind of like you know um we also have like all kinds of agentic skills for working with any of your design documents and as i was saying before like whether it's a product design doc engineering design doc launch reviews xp reviews production readiness reviews all of those things are in one place I'm like filtering some really old projects so that we can like look through them but like you will know exactly like who is working on them track the status of that and all of this data also goes into the graph as well so that we know like full history of what are the sorts of truth design docs approval status who works on them and each of these also have like the Jira link or the linear link so then we can trace from from this project how was the execution translated to this gives like a source of truth system for both humans and agents to know the specs for every project yeah i guess i guess planning is still really important especially for like something that is not just a trivial you know like one prompt or one shot project yeah exactly and because we need to like there's so many with the kind of product heterogeneity and like country heterogeneity like you need to like really think about all kinds of corner cases more than like you know the the happy path you have to think about all kinds of corner cases and like how they should exist and since we also operate in physical world the number of corner cases are like even more right i would really think about oh what happens if this goes on what happens if this goes on how do we handle the support case for this how do we handle yeah i guess just need needless but i'll just add it because it might not be as obvious but uber operates in so many countries and cities and even between two cities the reality might be different or might be different expectations or people are used to different things and so on yeah absolutely i think then and that's why it really becomes important to think about the right plan up front and this will also make it easy for both the engineers or agents to take that plan and decompose and execute on that like once we have the plan then the next thing is how do you convert that into decompose that and execute now we're kind of going into what we call minion internally so this is our background coding agent so around 15 % of all of the production PRs go through go through minion right now essentially you know this whole bunch of PRs that some PRs engineers want to do interactively but we also want to make it really easy once you have for the task that I have really good spec or i want to do minimal steering i can hand off that to uh to a minion and we had again we had we did multiple iterations this is the latest iteration we call it minion v2 but like you can use it from web you can use it from slack you can do a plan mode and then let it execute behind the scenes or you give one you can give the prompt once and let it execute behind the scenes if it's like a simple simple prompt and it has access to this is using the mega dev pod so then it has access to all of the repositories all of the code base that we have it can work across repositories and generate one PR or multiple PRs across repositories based on the plan and spec that you have when we initially started mostly maintenance kind of tasks used to be offloaded to minion as we progress on this journey as we are adding more and more tools into the minion environment where Minion can self-validate its work in web or mobile or backend APIs.

51:13Our goal is we can increase the kind of tasks people can do with Minion over time, increase the percentage of tasks over time. I think we can also see how it's integrated in Slack. You can just mention, do you mention at Minion or what do you say? You can do at Minion and then it will show you the status. and you can also like you don't need to go to the web you can steer it on slack itself maybe you have like a slack thread with the with the em with the pm you can iterate on the design if it's a simple thing you are just iterating and get alignment there and then say hey at minion follow this thread and can you can you create a pr and then it will take the slack thread context and then create a PR for you.

52:01So it's been really useful, even in our support channels. I would see like someone reports a bug, the on-call ask Minion, hey, can you see what is the issue? And then Minion will come back with, oh, this is the root cause. Oh, this looks simple enough. Let's just fix it right there. And then we would like fix it. Like there's no then loop of like adding it to backlog, doing in the next thing, like you're able to fix it right there. So we see like more and more cases of that as Minion gets powerful as well. So I think we kind of went through the build step. We also before talked about DevPod. It's improved since, hasn't it?

52:35Yeah, so this is showing all of my DevPods. Since I'm working on context graph, you see that. But yeah, if you want to create a new DevPod, you'll see which flavor you want. For Minion, there is a DevPod flavor that has all of the repositories. and optimize for minion usage. But for interactive, it will create a dev pod and within a matter of seconds, you will have that running. That's impressive. The seconds I'm sure you do some clever stuff like having things warm or pre-built so you can copy those kind of things. Yeah, it's running on all of... This is because this is developer shop. We have a separate multiple clouds that we are using for this.

53:22and it's like kubernetes cluster and like everything is warmed uh warmed up so then when you ask it like we are able to just clone from the pre-built image and it has build caches everything repositories everything already cloned like it's ready for you to build uh so i'm since we use basil across all of our repositories for build systems so i'm i was able to go into a repository like run basil for that and then just like we went we went through this in the order of this like you know seconds where any engineer like if they if i want to make a code change here like within order of seconds i create devpod it's one single way to build any module and then anyone knows this command and now agents also knows the same way right in the claude md agents md they know how to build test any module in any repository and they're able to use the exact same commands yeah so i think that covers kind of like you know what the kind of stuff that you have to do in co-generation so you can do interactive hardness you can do like through minion the next thing is okay like you have the pr what do you do after that then um um we will go through self-healing ci first one thing that we realized is because our pr volume has gone up year over year oh yeah it must have explored yeah our p50 cycle time p90 cycle time both for inner loop where like you know you have the PR and you want to get it to a green state and create the remote PR or once you have your PR created getting the human review getting all the CI things to pass I think reviews become a big bottleneck same as like any other company and we see year over year the delta and we want to like really we're like really focused on like breaking down every aspect of this and like how can we go really deep and optimizing it for the AI era so first thing that we did was we took a whole bunch of like simple CI jobs that used to run once you create the PR and we moved it to inner loop and we kept all of this behind one single command like AIFX validate so then when agent creates a PR it can run the linter the coverage rules the build rules the template rules like all of these things so that the probability of this PR in a is in a good condition like goes up then we hooked on ability to do visual validation where i can spin up emulator and test it i can spin up like a web and test it i can spin up like a staging environment where i can deploy this change run the apis and test it and now minion or even my interactive harnesses they know these agents know how to like have the skills for this so then they can go through this verification loop and let's say you create the pr and still something uh fails um then we we have like a self-healing ci you can apply to all the prs or per pr so we this is like a uber specific line that we add to all github prs that you can link to the issue that you're working on you can see which checks are failing you can have a whole bunch of automations auto request review auto merge or to fix essentially everything that we can we can do to reduce the p50 uh p90 cycle times and once you do all of this then we have this thing called code inbox this is showing a mock version of that where um it will any engineer can go they can see all the prs that are waiting for them all the prs that they created that they're waiting for reviews so if engineer send this for review.

56:56I go to this page, I can see all the things that we did so far. So it makes it really easy for me to get, oh, the AI code review passed. These are the things it checked. This is the sanity test screenshot. So I know, okay, I know. You can just quickly take a look. You have a video as well. Yeah, I can see the build checks and all of that. And we're also adding risk score, blast radius of this PR and so on. So all of this will give engineers really good signals to figure out, okay, like is it good to land and which parts of the pr i should focus on and yeah i think we're still like we are aggressively like working on this whole bunch of ideas that we have but we want to make sure like you know as we are ready for the influx of prs we in fact had to like had internal code red for our ci system because year over year the number of ci jobs also doubled or more than doubled so we had to like also like beef up whole bunch of foundations in this space so that we can be ready for the influx of new PRs.

57:57Yeah, and I guess you're also seeing that there's no way engineers can review all PRs with the same rigor as we did in the past. So I guess you're building tools to signal which ones to pay more attention to, which ones might be low risk in so many ways, maybe an impact, maybe in blast radius. What is your take on what will happen with code review over time? Do you think there will always be human code review or for some things it might go away? Yeah, absolutely. So I think one is for us, we want to err always on the safer side because the cost of an incident for Uber is really high. So even with all of this tooling, right now we still require an engineer to review every PR that goes in.

58:45Once the more confidence we get in our risk scoring and we can also back test it with years of data right we have all of this data and the more confidence we have in our ai code reviews then maybe for some class of low risk prs we could enable we call it green lane like auto landing but we want to be really really sure about all the dimensions that we want before we go there but right now the main priority is how do we make it really easy for engineers to get all the signals they need to review faster and if if we do a good job there then i'm sure like you know we can increase the percentage of like autoland prs over time but that's kind of p1 p0 is like first getting all the signals in right place nice yeah but code inbox is nice i like it this is like thing that we should touch upon is like maybe uh how i can do sanity testing and sanity testing just for those who don't know the internal that's the manual testing right yeah manual testing yeah i want to do let's say i'm i'm working on some new feature it touches two two or three services how do i know if this feature is working end-to-end so then typical workflow for engineers is um like pre-agent equaled we call it slate uh essentially you can it can you can create your own uh instance that deploys the branch that you're working on and you can route your test account UID to the instance that you created.

1:00:17So when you log in, you are executing that branch. Exactly, that branch and rest of the upstream and downstream goes to the other staging setups. So that way you can test end-to-end without worrying about other parts of the infrastructure and this will carry forward across the service graph. So you can say from your mobile app, this is the test account I want to test. It will go through other services and when it touches the service under test, then it will go to your Slate instance. If you're testing two or three, then it will go to the service under test across all two or three. So you can do even multi-service tests with this.

1:00:56Now that we had this, we had to just teach agents how to deploy Slate and we also give it all of the past APIs that you can use. Oh, nice. To validate any of these services. Basically, this was a thing where I, as an engineer, wanted to sanity test, meaning I want to manual test up my changes. I had to wait to do this in a pretty cool way. And now we can just give it to agents and they can actually test their own changes end-to-end in a safe way. Yeah, so here we are showing a web version of that and you can see in fact like this is like the CLI pattern we went through. It's using Slate MCP, create new Slate environment and it's saying this is the environment name and then once it creates the environment it's saying deploy service to this environment and giving the service name.

1:01:46This is a test service and now it had like it has its own environment this is like the Slate UI and it has like you know and if you see here you can pin test account UIDs to that particular instance and you will see all the services that are in that particular environment you can see the Gitsha and now like the the agent internally can like use all of these skills and tools so it if you see like what it did was it did a b testing of that web thing it ran agent browser in that slate environment with what is on main and what is on my branch and it created like a report for me to review the full set of changes that happened and so then like you know after this we can go to the like a report that it generates like see generated a report that shows before and after and showing me like all the there's no regressions and this is very easy for engineers to validate and we can attach some of the screenshots and videos to the PR.

1:02:58And the same thing we have for iOS, Android, web and if you want to do backend APIs, then you can also test backend APIs. I just want to add one of the things that again, you might not appreciate if you're not outside of Uber, but Slate is so important because Uber literally, today still has thousands of microservices or maybe not micro, But to run a change on just one service when you want to do end-to-end, that's just difficult, right? Like this is why Slate took years to build to start with. Yeah. And you also have to think about like the routing of the pinned accounts, right? And when the pinned accounts, like this is the test account that I want you to route to my branch that is just deployed in a staging environment.

1:03:42That's not trivial at all. So like in fact how this works is one of the things that we built back in maybe 2017-18. It's called Jaeger which is a distributed tracing infrastructure. It's open source now. So and all of our services have Jaeger spans. Essentially like wherever the call originates there's a parent span and at every hop there's a child span. and there is a you can add any baggage on that trace which can be propagated across the call stack so when you pin from the slate web we are essentially adding saying that like for these are the services under test and this is the account that i want to route so as it is going through our service mesh service mesh knows this baggage and then it can say oh i want to route this request not to this environment, but to the slate environment.

1:04:38So because we had tracing built out and because it's like propagated across the stack, now you could actually test and we can reliably route these kinds of test requests to the right environments so that it makes it really easy to test. This thing took like years to add, to make sure that all existing services were retrofitted with this and all new services were built with this in mind. So like I'm starting to sense a trend where like Uber builds stuff over years, pre-AI, just infrastructure that is easy to test for humans or easy to automate. And now with AI, it seems like, oh, you just built this the last year, but actually you built on top a foundation that was built for years, right?

1:05:17Yeah. I'm sure if we did not have a system like Slate, it would have been really hard for agents to self-validate. Then you cannot automate this thing. And as you said, this kind of stuff really helps um and um i'll also like talk about like one more uh like pretty interesting thing which is so uber specific and it's what is capri yeah and this is like very very hard to get right and like we had like even many versions of uh build like this in the past but i think this is probably our best version and we can like really scale it to all the teams but essentially the idea is um context. Because we have, even if you look at our core Rider app, we have hundreds of screens.

1:06:01Rider app is what most people think is Uber app, right? Yeah, Uber app, exactly. Yeah, we have different business, different product lines, different cities, different, like, a whole bunch of dimensions. Like, if I want to test any new thing, how do I know every combination of these has no regression? And if you were to, like, create all permutations across this, you would have to maintain thousands and thousands of tests, across the stack and that alone is like again really hard problem in itself so for some of our most complex platforms like we we call it capture replay like essentially you can capture real production traffic with just one middleware because now we have standardized language frameworks across all the services it's very easy to hook this into and it will capture not just the inbound and outbound at that service layer but also any of the downstream API's that service calls those requests and those responses are also captured so in some sense you have like full request response and all of your outbound request response is captured now with all of these captured requests they go through some pipeline and then you can build simple rules that says oh if you see this kind of field in the request this is the value for that dimension so then you can capture all kinds of dimensions that that platform team cares about and once you have all the requests and all the dimensions you can in some sense generate the full set of live dimensions and live combinations that that platform is currently supporting which we call it match guard that i can run pre-deploy in our ci cd that shows for all of the peers that are going in this deploy, let me validate against the full combinatorial set of combinations this platform supports and show the delta before and after.

1:07:57So then you get really high confidence if am I breaking maybe cache trips only in Brazil with this configuration, which would have been really hard for engineers to manually test or agents also to manually test. So step one is you capture what you want to capture and then you have the system that will rerun it check that there's regressions. Yeah, so like then I'm going to one of the domains that has this enabled, so you can see all the sessions that it captures. I can go to any of the sessions. Then you can see all of the metadata and tags. So essentially it's saying, oh, this request is tagged as marketplace type eats.

1:08:34It's tagged as line of business eats. It is in this kind of terminal state. It knows captured all the requests. You can also see all of the outbounds that it used. if we click on that it will also show you the the request and response for the outbounds and if you see at the top there is slate workflow uid or search environment name so then let's say if someone if i'm working on a bug that's only affecting for some reason san francisco uber each trips i i fix that bug then i can create a slate environment for that bug for my code changes. I can go here and then say, replay just this request on my Slate environment and show me the delta here.

1:09:20Show me the difference and you're expecting a fix or some change. Yeah, I'm expecting the fix because I know the before request, after request, what I'm expecting. And then I can do it for a specific dimension or I can use it. And then we also generate what we mentioned, like match guard, which is the full set of dimensions possible for that. So then you can create, in some sense, an automated testing suite for that team and also a more easy way to manually test one specific combination that you are focused on. When I was there, we didn't have this, but we had something called Deputy and Cerberus, but those were, I guess, very early versions.

1:10:00Yeah, so in fact, what Deputy and Cerberus allowed was ability to route any what Deputy allowed was ability to route any request from production to your testing instance and Cerberus allowed from your local testing instance back to the staging servers. But now that we have Slate, you don't we don't necessarily have to use Deputy and Cerberus because like it's already like you can deploy a Slate environment and we don't need to use any of them. I just want to stress because I think it's hard to see from the outside but that's really hard to build this because every single again you have thousands of services all of them have to support these things from routing for example to capturing production to replaying right like this was a lot of work i assume yeah and now i think the same like similar to how we exposed like slate to agents we can expose all of the capri functionality to agents so agents can check oh i'm trying to fix this let me get all of the sessions of this type and then see what request looks like can i compose a request of that type on my slate instance and see did it fix so it did not fix let me go in a loop until i fix this yeah yeah and and and now the agent can even though cannot see but it can see the delta and it can tell that nothing has changed for example so it can get feedback loops exactly exactly i think that that that that helps a lot now like the same it can agent can also do capture replay on slate uh with any of these we already talked about like the edge gateway edge gateway that's just the kind of uber's internal api right like we just we used to be rt api real-time api but now it's called edge it's a it's a go verse is it rewritten in go if i remember right yeah it's in go and now it supports whole bunch of endpoint types so there's a sectional api where there's grpc webhooks graphql so like again all apis go through one single entry point um where we can build whole bunch of tooling around that on monitoring on alerting um um and like even there is like a um uh like where there's all the reliability features the edge also has its own capture replay similar to the capri it's using capri behind the scenes and like if you want to look at reliability of anything it can like we look at the entire service graph for that service and tell you what is the potential unhealthy node that we need to look at the with the ager tracing we know exactly the service dependencies we've also automatically tagged which is a required dependency which is an optional dependency so that metadata is also there so as we are even when we are looking at alerts and diagnosing alerts all of this rich context really helps both engineers to debug faster and also agents to debug faster yeah we already talked about experiments uh the last thing uh we will cover is like what we call Shepard and Fox.

1:12:52After we moved to Monorepo, like then we moved to Monorepo because upgrading libraries was very hard, right? Yeah, on all of the different repos, yeah. Exactly. Then you moved to Monorepo, but then now you need to, someone has to like go and upgrade this library usage across all of the modules in that repository. So how do we automate that? So we first took a stab at like using open rewrite policies saying, okay, if it's a simple like you know just the signature change you can write that in open write policy and then shepherd knows what are the unique modules owned by different teams and then it automatically creates PR for each of those modules when AI came into picture then we could add like a AI based shepherd where okay if I want to do fix context propagation across all the services so that tracing works better I can create set of prompts and skills that then agent can run across every module and fix all the issues in every service faster so this is kind of our large scale changes whether you want deterministic changes or whether you want like agentic changes across the modules that gave us like we were able to do a lot of migrations over the years through this and we call each one as a campaign and if you see j unit 4 to j unit 5 there will be a yaml file with the policy and you'll see a whole bunch of PRs created for each module and you can a campaign owner can track it the status of that and and go through that so it makes it really easy to manage these migrations few months ago then we took the same platform and added like one more feature we also want to make it easy we want to continuously improve the code base not one-time migrations let's say I want to like code maintenance yeah code maintenance so we call it fox essentially you want to create self-improving code bases since we already have a whole bunch of agent skills.

1:14:46Essentially, any service at Uber, they can go here, enroll their service into different skills. I like that called cleanup. I want dead cloud cleanup or performance analysis or noisy logs. There's a whole bunch of these skills that you can onboard to. Oh, smart. And what Fox does is every Monday, essentially every Sunday, when we don't have that much CI load, it will generate a new PR for this service using that code cleanup. Like, oh, can I clean up any code? Can I improve the performance of this? Can I fix any noisy logs? So it's kind of automated, then code-based improvement. Then it's showing you how you can select.

1:15:30I like it because I heard people do this thing where they would set an AI loop, let's say in cloud code or codex saying, look for ways to improve my code base and they would execute it every night or once a week. But it sounds like this is just a lot more optimized. And I love that you can enroll your code base because a code base is owned by an engineer or team at Uber, right? Yeah, exactly. And you can see, because you're rapidly scaling, we are still in the opt-in phase. Once we improve some of the skills, will move some of them to opt out phase but anytime you obtain a skill for a service then every week in some sense you are getting a PR created like a managed agentic loop so then you don't have to like you are continuously improving this in some sense the more toil we reduce the more amount of time engineers can spend on the true feature work so like constantly thinking of how do I eliminate more toil from engineers and automate all of these things.

1:16:35And so we constantly look at all kinds of PRs with maintenance category PRs. Do we need to create a new skill? Do we need to improve that skill? So that class of PRs can be onboarded to this automated managed loop. This is nice. I like it. And then finally, I heard about wisdom. What is wisdom? Oh, yeah. So we use wisdom because all of our employees dog fooder products so if you're using a rider app or earner app and if you see any bug then you can from the app report that bug we now have all kinds of ai stuff where like it will automatically route it to the right team based on the stuff and then when you report a bug because it's from the employee from the employee like beta apps we can capture a whole bunch of logs and telemetry around that that makes it really easy for the team owning that feature to fix it so like in this case it's like an internal feature like it's essentially like if you want to test any feature in a different country you can mock the location on the device so if someone reported um that that that issue the first thing is okay now how do you reproduce that before it is it used to be a huge problem to reproduce so now we know the the set of screens the user has gone through because we have the in the logs and then we can give the engineers like this is the reproduction steps and this is the expected behavior and this is the observed behavior and so that we can boil down to the issue quickly and all of the telemetry that we collected from the app you can see all the logs in one place so like whether it's analytics network requests all kinds of feature flags so that also gives a lot of context and now now that we have all of this context we build a managed agent called debug assist that looks all of this looks at the code base and it will try to pinpoint what is the root cause and like what is the impact evidence timeline around that now i once i think okay this is the right right root cause i can hand off this to minion that we talked about yeah hey like coding agent looks good like why don't you create a pr for this minion will take all of that context that we have it creates a pr it will attach that pr to the Jira task.

1:18:57The Jira task also has the root cause analysis. So anyone who's working, they know the root cause analysis. If someone updated the root cause on the comments or like create a new PR, again, all of that is data that we can then figure out, okay, how do we improve the debug assist? How do we improve minion? We didn't get the right RCA or we didn't get the right PR from the RCA. Like both will be able to continuously improve. And then so do I understand correctly that you can capture and reproduce bugs so much easier, right? yeah and we also added ability to because we for the employee reported bugs we have all the logs i can taste take my test device and i ask i can ask it to simulate what that employee has experienced so then i once i give the bug uid we can essentially serve the same network responses that we served to the person who filed so that i can experience the same issue in my test device app to.

1:19:55So that way, okay, then once you fix it, then I can see, okay, this actually got fixed, and then it makes it really easy to reproduce. I love the recording of the thing, because that's, I mean, so important, right? Like, it's hard enough to do, even on the web, you need to build, like, custom tools for yourself if you want to do a website, but it's awesome that you did it. Yeah, I think this helps a lot, like, then in reproducibility and, like, getting to the right root cause. So you built all of these systems and especially last year it seems so many brand new systems that are just doing things like you know they're maintaining the cold quality agents are are validating themselves how do you think the job of software engineers inside of uber is changing and what are you learning from all of these new tools it seems like you're learning yourself you know like in some sense like if you're in the agentic ai world like you're constantly trying to find the next bottleneck and trying to like remove that and figuring out how you can like both using AI and both using like both systems engineering like reduce that bottleneck because ultimately our goal is to maximize the number of features that we can ship across all of our products and that will require us to look at the entire life cycle how do we rewire that for the new world and what kind of tools that we need to build and i think this is going to be like an ongoing journey right and there's a lot of discussion of you know this concept called capability overhang where i feel like even right now like there's so many things that we could you leverage the latest models for but we don't have the sufficient orchestrating capabilities or like the harnesses or the tools around that to be to leverage those models in the right places in the right way i feel like the more we work on this the more we can unlock um the more features we can ship the more productivity that we can unlock and like we are trying to measure a whole bunch of this and like constantly think about like what is the next set of uh foundations that we need to build to make it easy for engineers to work and to make it easy for agents to work you mentioned productivity how do you think about engineering productivity because when i was at uber this was again in like up to 2020 us managers did look at things it wasn't the main thing but we did look at things like diffs per engineer just to show you know if someone didn't contribute for like two weeks it was like what are you doing here or if someone did a lot of code reviews that's actually we wanted to notice but it sounds like a lot of those things the you know the code output the code artifacts are less relevant potentially you know what what is a productive engineer in your mind right now inside of uber or inside of your team yeah so i think like before the ai world like we were measuring you know like the standard stuff like code throughput and stuff like that those are the easy ones and then we would we also have like every few months we do the developer experience surveys we look at nps you know like sentiment across all kinds of dimensions we look at our code quality dimensions what's our incidence per def what's our code coverage and all of these metrics right um i think now once you go to like yeah i feel like we can leverage like the new set of capabilities to get more nuanced into this discussion like because because now we are internally talking about not necessarily just diff velocity but feature velocity like how much of all of this additional throughput is going towards unlocking new new features and how can we even change our organizational processes so that we can reduce the p50 feature time because like you can create 100 diffs but if they are not all feature diffs then like maybe it's not necessarily increasing the value that we can provide to the users right so like one for example one of the things we did is tagging all of our PRs to different categories like on complexity level and is it a feature PR bug fix maintenance testing documentation on different categories and we could see year-over-year how is that mix changing luckily like at least like right now I can see our feed our percentage of feature PRs is going up our percentage of maintenance PRs is going down so that gives us good indication that okay we're in the right direction okay we see the code volume go up and the percentage of prs that for features going up then we are investing a lot because before even though we had the uplan tool sometimes you might not have the right tracing from this product design dog to this engine design dog to this particular task so it becomes a bit harder to measure what is a cycle time into end reliably across all the features so now we're using ai to like automatically cluster these things and like tag any missing things so that we can reliably get what is the p50 feature velocity for different complexity of features for every half over the last two years three years and then we can go into is the bottleneck in the prd phase is the bottleneck in the launch phase is the bottleneck in the rollout phase what is going on there and what kind of things we need to invest to accelerate the number of features that we can ship per half.

1:25:01And how is your mental model changing of software engineering compared to pre-AI and pre-T's tools? Is it still the same job? Have things changed? I mean, at least in my total tenure in engineering, there's never been this much change to how I work. Like, for example, like, you know, we say like all the engineers had to wear our hands-on, like even even distinguished engineers at uber like our hands-on they like they create prs and i used to think i was hands-on engineer but my hands-on definition year over year has completely changed i'm like even personally i'm doing like way more prs than what i was doing last year like it's more easy to do uh all of these things uh all of these things right and i'm able to like explore more number of ideas than what i could possibly do in the past but the job of the engineering is you know in some sense how do you build the right context for the agents to produce reliable output like in some sense like you need to get really good at context engineering before i was operating with some context to do some tasks so i'm always thinking about okay if i want to offload some of this work to an agent what kind of context it would need and how do i set up the environment with that right context so that it can produce that work then you have to become really good at like you know how do you give the right specs right constraints for the agents apart from building the right context then multitasking has become really like a thing before if you're like it would take so much time to create a pr so like at least laser focused on that but now you're constantly switching between so many things like so you need to become really good at multitasking across all the sessions figuring how do you scope the how do you scope the work how do you track the work how do you verify the work really quickly because if you're not able to verify the work and like pick any code smells from the agent then then that's hard then you're shipping like some like some recreations into production right so like the context engineering ability to multitask ability to like review and like verify the outputs these things have become like much more important now because you can because you're able to parallelize a lot more than before.

1:27:13One thing I'm picking up is maybe a new primitive is we used to build software thinking how do I build this software and how do I verify it whereas now the question that you said is often how do I set up an agent to be able to build this and to be able to verify it so it's kind of like almost a meta like it sounds like there's more of that right? Yeah I'll give an interesting example I was working on with some of the engineers that work on customer support team like we are going through the journey of having ai agents handle some of our customer support policies that process would go from oh you have some runbook that human agents would follow so then from that runbook you are going through a whole bunch of steps to create a policy that ai agent can follow then you have to think about okay there is some dynamic elements here then which apis i need to call call to get that data which attributes i need to use then I need to configure my system change bunch of configurations then I need to do manual evals before launching then I need to do like continuous evals post launch and then like in the last few months we are able to like take down all of the step okay like how do I now take all of the context and all of the principles that I want to use for the policy conversion and teach agents to be able to do that so that I have a really good v0 policy that that an agent can create then you automate all of the configuration setup, manual testing, finding the right APIs, finding the right properties so that we can hook the right dynamic elements into this.

1:28:45Again, that used to take weeks, now it's like days for any policy. Then, instead of doing manual evals, then how can you use all of the past context to create synthetic evals on top of that so that you can accelerate, you can deploy this and then verify this quickly. Essentially what we were discussing, which is Now you're not, instead of doing that work, okay, how can you build the right tools and harnesses on top of this so that you can automate the workflow? Obviously, you need the right guardrails and human verification steps in between, but you're designing a system that can do these things instead of relying on so much manual work across all of these steps.

1:29:25Yeah, it sounds like you're learning a lot because these are all new concepts, which you're now using, you're learning about it, you're doing limitations. questions and you i i see what you mean by you know like more learning more prs and as closing what is a tool that you really like using um this this can be a digital tool this can be a non-digital tool something you find helpful yeah you know the last few days i've been uh hooked to uh like some of the personal agents like insting meos grokbots of the world i'm playing around with all of them it's helping automate a whole bunch of manual work that that i do in my personal life and I can in some sense follow the same process, give it all of the context it needs to do that task for me, give me all of the inputs I need, then I can have human in the loop verification step and do that task.

1:30:12But it's helping a lot to automate all of the manual personal work. Awesome. Uday, thank you so much for walking through all of the internals that are being used at Uber. I found it super interesting, especially that I haven't seen this for quite a while. Yeah, I know. Thank you so much for giving us the opportunity. I feel like we are still pretty early in this. There's so much to work on. There's so much to build. And we are super excited to be on this journey, to build the right stuff, and to see how the software engineering changes and how, at the end of the day, unlock the value for the company, value for our business by working on the right side of things.

1:30:50I hope you enjoyed this look into Uber's tooling as much as I did. It's now been six years since I left Uber, so a lot of the tooling that Uday showed me was also new to me. Uber was known for having about 5 ,000 services with about 2 ,000 engineers 10 years ago. But since then, the company started to consolidate services into well-sized services, and today it has fewer microservices than they did a decade ago. Today, it's in the ballpark of 4 ,500 to 5 ,000, even though these services are not as micro and they're much more well-sized instead. One piece of software that I really liked is Capri, the event capture and replay software.

1:31:23Back in my time at Uber, between 2016 and 2020, we had two tools called Deputy and Cerebrus that could do some level of production data replay, but it wasn't all that easy to use and it was just really hard to capture real base to start with. With Capri, it's far more trivial to capture sessions than replay them, and this works really well to reproduce bugs and also to test data fix work for a certain bug fix. Building a system at Capri is especially complex when you have closer to 4 ,500 or 5 ,000 services running like Uber has. Another thing that was interesting is how we're seeing where CodeReview is headed.

1:31:56It's slowly dying everywhere. But looking at the tooling Uber built around CodeReview with CodeInbox, every PR still gets merged after a human engineer decides that it's ready to do so. But Uber is building a lot of automated checks. Many of them are with AI to give more signal about the riskiness of the change and the quality that AI agents make of it. And so in the end, it's a lot easier for engineers to decide which PRs they should spend time looking at it in depth versus ones that are safe to approve with a symbol LGTM, aka you might decide you're not going to read the code. Finally, I had the sense that Uber investing 10 plus years in developer productivity is really, really paying off with their AI integrations.

1:32:36For example, they built cloud development environments called DevPots back in 2017. Uber got to reuse almost all of this infra in building out their cloud agents called minions. The same is true for existing systems to validate software, to safely rule them out, and now agents can use these as tools or as feedback loops. So it feels like companies that are getting some of the most benefit from AI are the ones that have had developer platform teams pre-AI as well, at least in the case of Uber. Do check out the show notes for more of the pragmatic engineering deep dives on Uber's engineering culture.

1:33:08Uber is probably one of the companies I have covered in the most detail so far. If you've enjoyed this podcast, please do give it a rating and tell your colleagues and friends about it. Thanks and I'll see you in the next one.

From the publisher

Brought to You By:

• turbopuffer – a vector and full-text search engine built on object storage. It’s fast, cheap, and extremely scalable

• Linear – the product development system for teams and agents

• WorkOS – everything you need to make your app enterprise-ready.

—

What do developer tools look like inside of Uber, today? They are very different from what you would see at most places. For example, can you guess what service a system called DevPods, Code Inbox, Slate or Wisdom would be providing to engineers?

Today’s episode is a deepdive into the day-to-day tools of devs at Uber, our tour guide being on of Uber’s longest-tenured engineers: Uday Kiran Medisetty. Uday is one of the few Distinguished Engineers at the company, and leads much of the work on agentic AI, today.

With Uday, we talk through how Uber’s developer infrastructure prepared it for AI agents. We look at the systems Uber built before generative AI and the new tools that connect agents to its code, internal services, and engineering knowledge. Uday explains how engineers use background agents, why tools like Slate make development (and bugfixing!) more efficient, and how code review is changing. We also cover Uber’s approach to token costs and how it measures productivity as the engineer’s role changes.

Timestamps

00:00 Intro

02:48 Uday’s early days at Uber

05:40 Uber’s move to microservices

07:55 Uber’s engineering team, then and now

09:05 Developer tooling before AI

16:03 DevPods

19:28 Building for AI agents

29:27 MCP gateway

34:37 Agent Skills repo

38:46 The context graph and Cortana

47:54 PRD and ERD tool

50:45 Minion

54:46 Testing and code review with Bazel, Slate, and Capri

1:12:26 Edge Gateway, Shepherd, and Wisdom

1:21:33 The changing role of engineer and measuring productivity

1:30:40 Uday’s favorite new products

—

The Pragmatic Engineer deepdives relevant for this episode:

• How Uber uses AI for development: inside look 

• Scaling Uber with Thuan Pham (Uber's first CTO) 

• Inside Uber's move to the Cloud

• The Platform and Program Split at Uber 

• Hypergrowth startups: Uber and CloudKitchens with Charles-Axel Dein

• How Uber Built its Observability Platform 

• How Uber is Measuring Engineering Productivity 

• Developer Experience at Uber with Gautam Korlam

—

Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email [email protected].



Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

More from The Pragmatic Engineer

All episodes