How Amplitude built an internal AI tool that the whole company’s obsessed with (and how you can too) | Wade Chambers

0:00I started showing a couple of people internally. It's like, oh, this is really cool. You got to look at this thing. And then a week later, it seemed like the entire company was using it. Moda is that internal tool that we have that unlocks all of the data that we have internally and then allows us to answer questions, to build artifacts like PRDs.

0:20Wade Chambers:What I love about these and other PRD generators is you can go from that little snippet of an idea to something much more robust. I got to see it and I'm all excited about it. I'm like, when is this going to be pushed so that I can use it? Monday, he added Push Live. I started showing a couple of people internally. It's like, oh, this is really cool. Okay, so this is my challenge to everybody listening. Mark the date. A month from now, I want you all to have your own internal tools just like this, or at least a prototype.

0:47Wade Chambers:Welcome to How I AI. I'm Claire Vo, product leader and AI obsessive here on a mission to help you build better with these new tools. We've seen a lot of workflows using a lot of tools on this show, but today we have Wade Chambers, Chief Engineering Officer at Amplitude, who's going to show us the tool they built themselves to do all their enterprise search, answer all their business questions, and I think build all their products. Let's get to it. To celebrate 25 ,000 YouTube followers on How I AI, we're doing a giveaway. You can win a free year to my favorite AI products, including V0, Replit, Lovable, Bolt, Cursor, and of course, ChatPRD by leaving a rating and review on your favorite podcast app and subscribing to YouTube.

1:36Wade Chambers:To enter, simply go to howiaipod.com slash giveaway, read the rules and leave us a review and subscribe. Enter by the end of August and we will announce our winners in September. Thanks for listening. This episode is brought to you by CodeRabbit, the AI code review platform, transforming how engineering teams ship faster with AI without sacrificing code quality. Quality code reviews are critical, but time consuming. CodeRabbit acts as your AI co-pilot, providing instant code review comments and potential impacts of every pull request beyond just flagging issues. CodeRabbit provides one-click fix suggestions and lets you define custom code quality rules using AST grunt patterns, catching subtle issues that traditional static analysis tools might miss.

2:28Wade Chambers:CodeRabbit brings AI-powered code reviews directly into VS Code, Cursor, and Windsor. CodeRabbit has so far reviewed more than 10 million PRs, been installed on 1 million repositories, and has been used by 70 ,000 open source projects. Get CodeRabbit free for an entire year at coderabbit.ai and use the code HOWIAI. Wade, thanks for being here. I am so looking forward to this. Thanks for having me on. One of the things that I think is so interesting about how you are approaching AI at Amplitude is you all have decided to build some tools yourself instead of plucking, you know, a bunch of various things off the shelf.

3:13Wade Chambers:And I'm curious, what was the internal thought process around this sort of build versus buy decision? Or why did you go down this path of writing a bunch of code? Well, let me start with first, it didn't take us as long to do it. So and it's in spare time, people's spare time. They actually put what we're going to talk about today. And so it's probably like three to four weeks spare time of some pretty talented engineers. one that's a little bit uh in the rearview mirror two in talking to a lot of my counterparts and other companies and asking them how they were approaching this and what they were doing i had it kind of split about half of them were saying hey i'm i'm pulling things off of the shelf and using it and the other half were kind of like no the way that you're going to want to use this and how you're going to get more leverage the more that you can do it internally as long as you don't invest a lot in, you know, overbuilding it and overengineering it, you'd probably be better off doing it yourself.

4:13And so once we got into it, I found that we were able to pull a lot of things off the shelf, like glean APIs and things along those lines, which just allowed us to move really quick. And there's not so much of an investment in it that I worry about. If I need to, I will revisit it. I'll throw everything away, do it again. It's really unlocking people and the data that sits in the enterprise that I wanted to do the most.

4:38Wade Chambers:That tends to be my whole theme when building things with AI. It's so fast and it's almost so cheap. I think, you know what? If in three months I throw the whole thing away, it will have been worth it anyway. So I can definitely understand that mindset. Okay, so we're going to see this internal tool you built and I'm excited to show it off. Apparently in the free time of engineers over three to four weeks or at least a little a little sprint. So tell us what Moda is. Like a lot of different companies, I think that right now Amplitude is going through that change of being AI native and wanting to move really fast.

5:19And so I think that in all of those companies, you need to get access to the data that you have internally. You want as many people as possible to see it. You want to see others being successful with it and say, oh, that looks really easy. How could I do that? And so Moda is that internal tool that we have that unlocks all of the data that we have internally and then allows us to answer questions, to build artifacts like PRDs, things along those lines internally, but with the full scope of information that we have access to. Great.

5:58Wade Chambers:So you took all of your business data across, I'm sure, like BI sources and documentation sources, and you exposed it in the tool. And one of the things you were telling me before we started the show is one of the approaches you took was the social engineering approach, which is you went into the decision of what platform Moda would be built on. So can you tell me a little bit about how you incepted the organization with the design of this internal product? The problem is everybody's starting in a different place, right? And so you use ChadGPT at home or you use Claude or you use something else.

6:32And some people are really advanced and others are not. And so how do you create this common language or this common fluidity, if you will, around AI? I, it felt like if we could do something and that made it really straightforward and we took a lot of the complexity and trying to push it down, it would allow a lot of people to be able to engage with it. And so step one, and by the way, great minds before me, well, I wouldn't say I have great mind, but like great minds that I can leverage. The folks over at Abnormal Security had built a little agent in Slack that I got a chance to see. And it allowed all their employees to ask really good questions.

7:16And I'm like, oh, that's genius. Because if I can see what other people are doing and it's adjacent to the work that I'm doing, I'll borrow the prompt. I'll borrow the question. Or if they've already done something that I can leverage, I'll just use the result on the other side. And I'm like, and that is the right way of doing this. The more that I can make this publicly visible and do real solid work on the other side of it, provide great answers on the other side of it, then it should catch like fire. Well, that's the thesis. Uh, the truth is, is that one of our engineers had this working on a Friday afternoon.

7:56Uh, I got to see it and I'm all excited about it. I'm like, when, when is this going to be pushed so that I can use it? Monday, he had it pushed Live, I started showing a couple of people internally. It's like, oh, this is really cool. You got to look at this thing. And then a week later, it seemed like the entire company was using it. It's been pretty incredible.

8:19Wade Chambers:I don't know. You might be wasting your talent in B2B. It sounds like you got some consumer thinkers there with how viral this product went. It's helpful. And anything that accelerates your thinking or allows you to get to a deeper truth. All of those things are just awesome. And so if this can help you with that, and you can see other people doing it. True story, right before I came on this, I was talking to one of our sales execs, and she hadn't heard of Moda yet. And so I was like, ah, well, that'll help you answer that question. And her first thing was, is she opened up the Slack channel, went in there, and she saw a colleague that's using it repeatedly as she just scanned down.

9:01It's like, well, if it's good enough for him, I'm all in. And so I'm like, okay, that's the social engineering aspect of it. If you can see people more credible or equally credible as yourself having great effect with this, it's an obvious thing that I want to use. Great.

9:18Wade Chambers:So let's take a look at it. Well, as an example, here's today's thing. And you can already see just how many people inside of the company have already been using it today. So let me go through and ask Moda to introduce itself. And so if I go through and just, hey, Moda, what are you doing? Well, he's chomping on data, nom nom. But it'll come back here pretty quickly and give an answer and say, well, I'm Moda, internal agent. I search internal knowledge sources. I can also search the public. I always cite my sources so you can verify the information. You can access me by Slack. There's also a proprietary interface, which we'll get into.

10:02You can even learn more about the implementation and things along those lines. And so if I wanted to know a little bit more about Moda, let's just ask it to tell us a little bit more about itself. And again, it's just going to go through, parse it, send it out, gather information, and put it back and just say that it was built. We have our own little stack that we've built to be able to process AI requests called the Langley framework. But on top of that, you can see that it leverages a lot of the things like Glean to be able to access certain parts of this, but it can do external search, and we already kind of covered that.

10:47Well, what about the data sets that it has access to internally? Because lots of people don't have the exact same things that we have, but let's go there.

10:58Wade Chambers:What data sets do you have access to? Well, Wade, I'm glad you asked. So we can get into Confluence, Jira, Salesforce, Zendesk, Slack, Google Drive, Product Board, Zoom, Outreach for transcribed meetings, those sorts of things, even some Gmail, Asana, Dropbox, GitHub, HubSpot. So it does not have access to private, personal, or restricted data sets. It's generally the public ones, well, enterprise public ones that we have internally. And so if I just was to go through and say, all right, well, who is using you? Completely different way of looking at it. Just how widespread is it inside of the company and how many different groups are using this?

11:43Wade Chambers:And so you're product managing your own AI product using your AI product here. Exactly. Because if I can understand where we have any friction inside of the company and what types of questions are being asked and where are people not having success on the other side, it's just a simple process of like, okay, where did we get the rules wrong? How can we help out with prompting? What data sets do we need access to? Do we need to do any grooming of that on the output side of it? But you can see product management, engineering, sales, customer support, marketing, our CEO, our head of sales. I've seen our chief product officer in here.

12:28There's, well, and you can see it just keeps going even as we're sitting here talking about it. So these are all, well, I'm even a reference inside of here. But you can see that lots of different people are using it for lots of different things internally.

12:44Wade Chambers:So could you show us one of the common flows that somebody might use with Moda? So it's pretty good at explaining itself, which is great. And it seems like it has access to a lot of data, but, oh, here we go. And then you can customize it. Well, you know, I want to make sure that I understand everything that's going on with Moda. So I'm going to actually give it a little bit of attitude and let's go through there. So I want to go through and actually start, I mean, almost everything that our customers do, they need to understand what's going on, then they want to make a decision, then they want to act on it.

13:21So we need to do the same thing here. Why don't we actually go use all of these data sets that we've got access to, to find out what's going on? I don't know that we actually need to see every, the top themes and queries, but maybe it's a good place to start. I'm just looking for an insight that we can kind of chase down. I think this is great.

13:43Wade Chambers:So I think that thematic analysis and sort of trying to quantify more qualitative or more qualitative feedback is a very common kind of product manager use case. And so anything that can have access to a broad set of business data and do that analysis for you. Oh, my gosh, these descriptions are ridiculous. Isn't this great? It really gave you Gen Z slang. Bet. Bet. Bet. No cap. All right. So this is doing something that I think would be very popular for a product manager. And so you're saying, let's take all this context I have, analyze it, quantify it. And then it actually gives you the scope of its research data.

14:33Wade Chambers:So it says it takes the 50 most recent Slack messages. So that's a good resource. But then you're looking for another one with more external resources. So that last query was really about the ModaBot itself. But this query looks like it's about your actual product and you're looking for real customer feedback. And this is a very common workflow that a lot of product managers do quite manually. Absolutely right. And so, you know, go into product board, go into Zendesk where things are going to be logged. actually go into the transcriptions of conversations that we've had with various customers.

15:09And let's get to a point where we believe we know where there's some energy, where people want extensions to the product or want to improve the functionality of the product, like connecting session replay to funnel analysis. That seems like a pretty good place to dig in. So why don't we just jump in and see what Moda's got to tell us about that?

15:35Wade Chambers:Got it. So you're taking this wide funnel of data analysis, trying to figure out what are the sub-themes, and then you can use Moda and say, okay, I've plucked a sub-theme. Give me even more data around it. And do you think that's a useful flow for folks? Basically, like start at a pretty wide data top of the funnel and narrow in? Is that how you approach things? I think it's useful when you're going through and saying like, okay, what could I learn from this? What's something that's new? And so this was me trying to start at the very top and say, okay, let's go in and dig into, well, let's start at the top, say, what are people asking for?

16:12And then if people are asking for this, let me make sure that I believe that what they're actually asking for, there's actual quotes, there's details behind it that I can look at. And we could even ask the opposite of this. And so there's plenty of conversations that come through in this of where you can see specifically, you know, from Zendesk and from Zendesk, from an outreach call. And so you can see the details of specifically what somebody is asking for and what they're trying to do inside of that, which gives me a sense of like there's some heat here.

16:48Wade Chambers:As an AI founder, you're used to sprinting towards product market fit, your next round, or that first enterprise contract. But speed isn't enough for AI startups. Buyers expect security, compliance, and transparency from day one. That's why serious AI startups use Vanta. With deep integrations and automated workflows built for fast-moving AI teams, Vanta gets you audit-ready fast and keeps you secure with continuous monitoring as your models, infra, and customers evolve. AI innovators like Langchain, Rider, and Cursor scaled faster and closed bigger deals by getting security right early with Vanta.

17:32Wade Chambers:Listeners can claim a special offer of$1 ,000 off Vanta at vanta.com slash howiai. For somebody who is building AI products for product managers, I know there's a whole cottage industry that is trying to build SaaS products for what you just what you just showed on this internal tool. And I'm curious if your product managers have by and large been quite happy with Moda of their as their source of insights or if you feel like there are pieces that this kind of general purpose internal tool is missing maybe to serve this specific use case better? I think that a lot of product managers that are here take this as a way of getting access to 100 % of the data and seeing what AI would generate for it.

18:22And if it comes back and it doesn't match their instincts, they'll dig in. And if it does match their instincts, then they'll look for more detail inside of it. And so I think that for the most part, we have a lot of product managers using it to great effect. And I'll go into more detail on how it actually does this. We have a proprietary interface. Actually, let's jump over there.

18:48Wade Chambers:Yeah, I would love to see that because I think my next question is like, how does it work behind the scenes for folks that say, well, I got I got a smart engineer in three weeks. How could I have this too? I'm curious how you've approached building this a little bit more detail. Yeah, and the nice part is you can even ask Moda on how it works. We have a framework that we've used internally, and we have both a little bot that sits in Slack that can call out to that framework. But you can see that I can go into a web UI and basically be able to access the same thing. In this flow, we've actually got a lot more capabilities that are associated with it.

19:29And so we can go in and create PRDs as an example. You can ask anything, you can do deep research on it, but you can create a PRD that then allows us to take some piece of this and go in even deeper. And if I was to go in, you can kind of look at just jumping into GitHub, right? Here's the YAML that kind of defines the high level how we would do a glean search on ask anything. We could go back up and even look at the PRD orchestrator on this side and you can see you know how it starts with a prompt and is able to sort of dig in a little bit more based on what you're trying to do. If it needs more details it's going to ask you for that.

20:14It's then going to take all of this and break it down of where it will break things into let's make sure that we do proper problem exploration solution exploration detailed requirements that come out the other side we like to move as quickly as possible to a prototype and so if it can do prototype generation or at least the prompts that then we can plug in we can copy and paste into bolt or you pick your lovable v0 pick your your favorite it and be able to move it around, then we can get through that fairly quickly. And then it will just go through using the Glean API, we'll get access to a lot of the content and be able to do a query.

20:57And so we use that as part of our reg to make sure that we go get the right content and then pass it off. And it's able to evaluate all of that and come back with a much better answer.

21:06Wade Chambers:Yeah, I have a couple technical questions because as a builder, I'm just so curious here. So it sounds like you're using the Glean API for a bunch of your enterprise search. So that simplifies a little bit the data access and controls and all that piece the rag piece of it then you've built these two kind of alternate interfaces which is the slack bot or or the web ui which is nice and then it seems like you've built also some kind of specialized tools in terms of kind of the general chat deep research chat or this prd flow and then just looking at that github if you don't mind pulling it up again i'm so curious who got this good at writing prompts because this is a this is a well-structured prompt.

21:45Wade Chambers:And I think one of the things that is very mysterious to people right now trying to build something like this is they vaguely know the scent, like a sense of like tool calls and they vaguely know about instructions, but no less about like these sequential instructions or multi-tool or parallel tool calls. And I'm just curious how your team up-leveled or up-skilled on how to build great prompts, how to build great agents. Was that just learn as you go? There was part of it that was learn as you go. I mean, we've got a couple of talented folks that have a lot of experience with AI. And so I think we were able to use those.

22:22But also, AI is a good tool to use for building prompts. And so you can just recursively ask AI to give you better prompts or things that would allow you to focus on very specific things that you want in the results set. And AI will actually generate the prompt for you that you can use. You'll probably have to edit it a little bit, but it does a pretty good job on its own.

22:44Wade Chambers:And then how does your team improve this over time? Is it open to anybody to do a pull request on it? Is there a team that owns it? Have you set it up operationally? Yeah, we've set it. I mean, it's all checked into GitHub. Most of our engineers can check it out. Even product managers and designers can probably do the same to it. We've had designers that are contributing to this and product managers who are contributing to this. we're currently going through an AI week this week of where there's a lot of things that are going on. One of my vibe coding ideas is I want to be able to add NCTs in addition to PRDs to all of this and see if I can do it totally on my own.

23:24Wade Chambers:Okay. Well, speaking of PRDs, is there any way I could get you to create a PRD with this flow? I'd love to see that. Let's do it. I was going to go through here and say, let's go through. I'm going to see if it will take it as is. Let's actually go through and create a PRD. So for people listening in our Slack flow, we got this insight that people wanted session replay attached to funnel steps, which makes total sense. And so now we're taking that idea and that context and going into the create PRD prototype flow in Moda, which is asking us what we want to build. So let's just give it the prompt that we had as an answer to a previous question and see if it's able to expand on that.