How Intercom 2x’d their engineering velocity in 9 months with Claude Code | Brian Scanlan

27:48And now you can just make it really easy to enforce those standards across the team, which I don't think is micromanaging. It's actually just making everybody's golden path much smoother to production. And so I think there's this just very interesting parallel to how we've approached CICD to how we approach things more upstream, even from the product management perspective. Totally. We're on this movement towards a software factory. And what factories are great at is, you know, like an IKEA factory or something. It's all the same furniture, all the different bits. And you know how to assemble it.

28:24And look, it's not your artisan stuff. It's not or it's not cutting edge or whatever. But it's very predictable and has a certain quality and meets certain standards when it comes out the other side of the factory. And so, well, pull request descriptions, again, they're not make or break for the factory or the pull request or whatever. It's one of those qualities of just good quality work that's reliable, predictable. And then when it's assembled together, you've got your IKEA factory. Well, and people don't want to feel, certainly engineers don't want to feel like they're part of a slot factory, right?

28:55And so these things that you can add into the flow that actually up level and meet the standards of the engineering team really help your human engineers on the team feel like they're working in a place that values quality. And so I appreciate that you've put that effort into these behind-the-scenes hooks and skills because I'm sure it reinforces to a culture that's being asked to move very fast, to ship things differently than they have before, that you still do care about their experience reading pull request descriptions. You meet their bar for quality, and I just think it makes everybody happier.

29:37Yeah, well, it's great when the robots just produce the work that you'd expect of your best engineers, you know? Yeah. And, you know, maybe as you get this live, I also think there are just still such more interesting problems to solve in software engineering. And we can talk a little bit later in the episode about some of the interesting problems that you are solving on the product side, on the technical side. I think there is no lack of hard, intellectually stimulating, creative problems to solve for customers. and coding redirects is just 100 % not one of them. So did we get my redirect live or are we close?

30:16It's still there. I'm waiting for an automatic review to kick in, but we can come back to it. So one of the things I would like to show next might be some of the telemetry that we have in place. So we saw that there was different skills getting invoked and we don't like flying blind. To run a system like this, you need to know how well people are using it. Are people using these skills at all? You know, the kind of basic information that you'd expect of when you ship a product to your customers, like, you know, where can I see the usage? How can I fight for the usage? What's going wrong or what's not going wrong?

30:53And so we collect a bunch of telemetry using different mechanisms and have different homes for us. the most open one that we have is we collect basic usage information for skills and the like and we send it to Honeycomb. So we just have a shared key that's deployed to all of our laptops and anyone can go in and kind of look through this data. So if you're developing a skill internally in Intercom and like hundreds of people do this it's very easy for you to go in to discover like, hey, who's actually using this? When are they using it? And you've got to use this as a kickoff to like follow up on just basic discovery of usage of your skills and all.

31:33And unsurprisingly, the main skills that we have are things like creating PRs. Admin tools is our admin internal tooling APIs, where we have an MTP in front of it. BuildKite is our CI system. Snowflake Log is where we put Snowflake. So you can see from this, a lot of work, a lot of the skills that are being evoked are all around the building and then seeing where my stuff is, and maybe some troubleshooting type information as well. And so this is the first kind of step. It's like, you don't have this. It's hard to have a large system like all these hundreds of skills and hundreds of creators working in this area without having decent telemetry.

32:10The next thing we do as well is we also collect all of the session data and put it into S3. And so we anonymize it. We do a few things to make sure we're not doing anything too private. You know, people put all sorts of stuff in their sessions. They yell at their sessions. Yeah. Yeah, people have personal relationships at times with Claude. And we don't really want to know about that, but we do want to be able to dive deeper into how things are going. I think understanding what the dropout rate of sessions did, how quickly people got to something useful, like whether it was a PR or something like that, this kind of information is pretty interesting.

32:48And so we're harvesting a lot of session data. We're doing different things. This is what I'm showing here on the screen. is like a very simple tool that we put together, which just gives you some personalized insights. And, you know, you can do this inside Cloud these days as well. And there's plenty of skills out there on GitHub where you can do session analysis. But I think we just built a little tool on top of our session collection system to give people feedback. And it's feedback that we're interested in giving feedback about how their sessions are going and how they're kind of fitting in, how you should think about your own, I guess, use of CloudCo compared to everybody else in the org.

33:26And, you know, I'm not doing too bad here. It's like 79 percentile. You know, someone has to be down the bottom of every percentile. And there's some interesting feedback here. Like, it's kind of getting annoyed at me. Oh, Ron, I was getting annoyed at Cloud a few weeks ago because I'd set up GOG to interact with all of our Google stuff internally. But it kept on trying to do the wrong thing. and I was kind of giving out to it and ended up adding stuff to Claude and MD and stuff. It's kind of giving out to me here. It's reminding me that this wasn't a very effective way to interact with Claude Kerr.

34:04So, you know, it's a good prompt for me to actually go and fix up my memory or whatever. And like, we all, like, people are at different levels, even at Intercom. People are at different levels of adoption. People are joining Intercom. They may not have, like, seen a system like this before and they want to know how things are going and get feedback. And so this is one example of how we're just trying to pull together this information to give useful, actionable insights to people so that they feel supported and that we're not just throwing them an API key and saying, best of luck. It's like, no, we understand what growth looks like and the progression that people go through when they're using these tooling and getting better and kind of self-improving.

34:44And we want to support all that. So this is one of the things that we're doing with the session data. There's loads of other things that's a work in progress. And like being able to, like we want to get insights to which skills are the highest quality, which gets you, which skills get you to your results as quickly as possible. And then which ones need work, you know, which ones aren't working out so well and might need a bit of attention to improve. This episode is brought to you by Cursor. If you all have been watching How I AI, you already know this. Cursor is my favorite way to code with AI.

35:17Whether I'm using plan mode to build out an ambitious feature, reviewing AI generated diffs right in my editor, or kicking off cloud agents to multi-thread our roadmap, I reach for Cursor as my favorite multi-model coding platform. Even better than building myself in Cursor, I love collaborating with BugBot to fix PRs for code security and quality, and have begun relying on Cursor's automated agents to keep our code base clean. It's not just me. The most ambitious teams love Cursor 2, including engineers at Stripe, OpenAI, and Figma. Ready to build more? We're giving$50 in Cursor credit to HowEye AI listeners.

35:57Claim your credits at chatprd.ai slash howEye AI. That's$50 in Cursor credits by going to chatprd.ai slash howEye AI. I have to pause before we look at your list of skills because I'm so excited about that part. But if folks aren't watching it, they may have missed how amazing what you just showed is. So I'm going to reiterate it, which is, one, you've instrumented all your internal skills with telemetry so that you're using Honeycomb. Love the Honeycomb team. You're using Honeycomb to see how often those skills are invoked over time. So this is just a tip for anybody building out a skills repository internally or even somebody who is maybe trying to get some visibility into their impact across the org.

36:47Let's say you build a skill and you want to go to your boss and be like, boss, my skill is being used by literally everybody every day. Find a way to put event level telemetry invoked in the skill, a little dashboard, and you can track those over time. Again, treating your org like a product, treating your repo like a product, treating your AI setup as a team like a product. And all products, all good products have tracking plans. And so figuring out how you put that telemetry in, I think, is really smart. And then the second thing for those that missed it or how to do it is you're taking all the raw session, I'm presuming JSON files.

37:25So for folks that don't know, CloudCode stores all your chats with CloudCode on your computer in JSON. And you can go look at those or query those at any time. It sounds like you all are uploading those files to S3 and then layering on top of it some anonymization, some user level views. And then you're essentially building sort of what I would call like an internal eval of how people are using Claude Code and what problems they are having over time so that individuals, one, can triage their own implementation. As you said, oh, it looks like I need to do this or that or improve my agent's MD. But then if you're seeing consistent themes over the organization on it's never invoking this MCP when we need it to invoke this MCP or people are yelling no every time the create PR skill gets queued up, you can fix that at a systems level.

38:22But you can't do that if you don't have the visibility. So again, my VPs of engineering, my CTOs, my friends out there, put some telemetry in your skills and then do some meta analysis on your cloud code sessions across the org. And you'll be able to identify places where some probably some high leverage fixes are going to get your team unblocked over time. I do hope and expect that this stuff will get easier over time. you know um i'm happy to kind of invest the work uh so that we can move fast and kind of be on the bleeding edge but there's something to be said also for being for having like last mover advantage and just getting all this stuff for free whenever entropic ship is or whoever ship um i mean maybe this is a product just that people should buy uh or build um but for us right now we've no choice we just got to build it we're we we're we're like we're fascinated with the insights that are locked away in these sessions.

39:16And so we just got to build stuff so that we can see what's going on. I love it. Okay. Can we see some of these skills? Yes. So it's a very exciting GitHub repo. Our lives are all GitHub repos and markdown files. Totally. And we have a lot of activity at the moment. We ran an AI day last week, kind of getting more people contributing to it. And so, well, so what this is, is it's a plugin repo and we have a series of plugins and they're growing daily at the moment. Kind of every team will have their own kind of specific plugins. In general, though, we're very liberal. We want stuff to end up in here, even if it's not great.

40:04But we do sweat the details on the core plugins, things that we think are fundamentals, foundational ones that go out to everybody. And so where we start off was we have like these base plugin, which gets installed. Oh yeah, so we distribute this not via the cloud code plugin mechanism. We found it was just a bit flaky. It was, you know, sometimes it updates, sometimes it wouldn't. And it ended up kind of like trying to manage a Python install on hundreds of different laptops. You know, you just don't want to do it. And so we ended up using our internal IT systems to synchronize all of the plugins to the disks of everyone's laptops.

40:44So this is a great cheat code. And yeah, strongly I recommend getting very close with your IT team to be able to deliver things like this reliably and not have to rely entirely on the cloud code plugins mechanism. Just our experience is a bit flaky and it just gives us a lot of reassurance. We don't have to do certain types of debugging once it's all on disk. So we know this stuff works anywhere because we've got our IT team pushing it out to disk. And so we've got some safety hooks. We have some of the foundational things like merging PRs. We don't want our agents going off into AWS. And then just different settings and the telemetry things as well.

41:22So these are the core things that absolutely everybody gets. But these are minimalist. list. We don't want anything that could be inappropriate in, say, a non-technical person's laptop or whatever. So this is the basic building block. The next main bit for us is what we call developer tools. Again, this would be things that we've been to all of engineering and beyond at this point. And these would be generally skills that would be appropriate to be used by any engineer in the course of their work day to day. And again, we would have a high quality bar again for all of these. These would all require evals.

42:03These would all require to pass different kind of tests or analysis that we do on the quality of skills. And so we try and maintain these and make sure that they're well updated and well used and we pay a lot of attention to. I can maybe go through one of these skills in a bit of detail. This one's near and dear to my heart. It's flaky specs. And I think the interesting part here is not the skill itself. The skill does reliably fix flaky specs. And I can pull up in the meantime, like here is a list of flaky specs that we have at the moment. I'm going to open up the skill and just start to run it on this issue.

42:43And so while this is running, just walk through what's in the flaky spec skill. And so there's a checklist here. And the fun part about how I built this was not that I was a world-class expert at fixing shaky specs. I roughly know the problem and have fixed a few of them in my time, but there's different classes. In a large test environment like ours, we have hundreds of thousands of tests. And if you're not super careful about data poisoning or race conditions and all these kinds of things that kind of kick in when you're running millions and millions of tests a day, you end up with these tests that end up slowing down your ability to deliver code to production fast and reliably and not confuse developers by things randomly breaking.

43:25And there's kind of known patterns and known ways you would go about this. I knew my goal, which was to have a skill fixing all of these flaky specs. And it was something that agents are pretty good at when you give them a kind of testable goal, you know, this wasn't quite open-ended. And I also had this huge backlog or yeah, there was a backlog of probably a few hundreds, but then also all of this historical flaky spec information. And so you can just harvest all of this data in your environment to go, hey, Claude, I'm going to build a skill. First of all, go and research every single flaky spec we've ever had.

44:00And then we're going to build a checklist. We're going to build a mechanism. And then we're just going to crunch through them over and over and over. And you get to this like 1x kind of, you know, it's doing a good job, probably as good as a job as I would do. But then as you keep building up all of these like little teeny steps, which are the kind of things that, you know, our best Rails coders kind of do. They've got all the stuff in their heads and all the different classifications of flaky specs and, you know, verifying it gets real data. But the really fun part is then you get, so you get something that's starting to be like 10x.

44:34It's fixing flaky specs that I'm not even sure if I could do. It might take me a day or something. And I probably wouldn't do it. But then you start to add in stuff into the skill along the lines of like, okay, when you fix something and it's novel, you need to update yourself as well. So in that session, it's updating the skill. So the skill itself is kind of learning as it goes along. And we also fan out. So it's like, okay, I'm very happy that you fixed that flaky spec. Now find every flaky spec that got impacted by that nature of it. And so I went from zero to like 100x in terms of this skill now is like, you know, see your distinguished engineer or being able to fix these specs.

45:18But it was more like the process that got there and sort of like working with a feedback loop, working with like a very clear goal and then giving it the freedom to do it, you know, giving access to the systems where it needed to pull in metadata, being able to run builds itself and having that feedback loop where it's learning and then, you know, designing the skill as well so that it's, you have to edit it every so often. It ends up taking up too much information that might confuse things. But then you break things out into like reference guides. So you're doing this like progressive discovery thing.

45:48And I've even accidentally pointed this skill as like a Python code base. And Claude has just gone, like, it's just Python, I'd be able to go. And it kind of uses the knowledge that's applicable to it. And so again, this skill is not going to make Intercom's revenue go 100x. but it's now this like perfectly reliable thing that we really no longer have to think about now we can expand out into many many different areas and we just have to maintain this and the maintenance work for a skill like this just isn't much and we have evals and stuff so that when we're upgrading models or maybe even moving to cheaper models or whatever that we can make sure yeah this thing isn't progressing it's still working as well as we think it is and we've got confidence and certainty that this is still a reliable building block.

46:36And again, the constituent part when put together, you've got like a very senior engineer who's able to get any work done in your environment. And so, yeah, we can take a look at what it's doing. Oh, it's asking me for permissions. Should have checked. You forgot to make no mistakes, dangerously skip permissions. That's the rule on how AI. One thing while it's running, I wanted to say is, you know, this skill is a perfect example of what I call the like and then AI workflow, which is I tell everybody like pull your skills and pull your workflows through a bunch of and thens. So I want to fix flaky tests.

47:13So I go to GitHub. I find a flaky test. I run through the skit. Let's say you fix it. And then what would you do? Well, I would document how I fixed it. And then what would you do? Well, I would go find all the other ones that are just like this and fix that. And then what would you do. I would go from, you know, a Rails code base to a Python code base and apply the same. You could just do that over and over. And because the cost of running these is so low, you can actually pull the thread of a bunch of stuff. Any reasonable human would have quit at step one because you're not limited, again, by headcount or coordination costs.

47:49You're limited by the technical capacity to solve the problem, which I think is a really interesting way to think about how you get from like the you know engineering intern that whose job is to go through and take a first you know gentle pass at all these flaky tests through to the distinguished engineer who has just speed run through 300 of them and has thought of a completely different way to architect your your testing overall in your repo so i think that's a really great model for four things. And then the other thing is like, again, engineers, go speed run your tech debt, fix your flaky test.

48:28Like these are all things that as somebody who has run engineering organizations, I have heard over and over, we can't because our code base, blah, blah, blah, blah, blah, blah. Like, can we pretty please allocate this amount of time to just fixing this really annoying front end flaky test? Like, you don't have to ask permission for that stuff anymore because there's just a new way to solve it. And I think, again, just going back to some of the stuff we were talking about earlier, I think your overall product quality is going to go up. I think your overall developer experience is going up. There's just so many good things that come out of using these tools and using them correctly.

49:04Yeah, I think backlog zero is a realistic thing for teams to be able to go after. You know, all the things that you wish you'd ever wanted to do, you know, it's now just achievable. But of course, you've got to balance it with all of the extra stuff that you can just deliver at the same time. But it's so sweet to be able to think that, hey, we actually have a path to getting rid of all of our backlogs and all of the kind of architecture changes or whatever. Recently, I was taking a Go microservice and re-implementing it in Ruby. And it was a single cloud code session. Before November, this was something that I would have had to advocate for on a roadmap and like, you know, plant some seeds in different engineers' heads and kind of build people towards it and kind of blame a lot of problems on the existence of this micro-crever.

49:53Wait, trigger warning first before you talk about that process. I'm sorry, I'm giving the secret sauce here of how to influence a dog. but yeah and but now it's like well i don't even have to think about this now it's a single session and in fact i can i can get claude to implement it five times and compare the styles or compare the you know get us to review them and figure out what the best way of implementing the thing is and this is just like this level of kind of creativity and freedom that where like your imagination is the blocker not not the the time it takes to actually knock out one of these things which was months in the past, you know.

50:30I completely agree. And I feel this at Chat Parity where people are like, what are your, I mean, I'm a product tool for product people. They're always asking what my roadmap is. I was like, I literally don't have a roadmap. We burn down the roadmap every week and then we figure out what we're going to ship next. And of course we have thematic ideas we want to pursue and things that are larger. And one of the things that I do to keep myself from overshipping absent product market fit is literally constrain the ideas to what I can do in my brain, which is there's like a natural throttle on not getting slop out because it's not engineering broadling me.

51:09It's actually just good commercializable ideas. And I think that's where we're going to see some of the limits start to come in play. Again, referring to Anthropic, another big news piece came out is that they're hiring a bunch of PMs because they have so much engineering capacity. They're actually limited at the PM capacity. And so it'll be interesting to see where the bottlenecks in your business, you know, end up and which bottlenecks are appropriate. It's probably good to have a product bottleneck a little bit because then you're not shipping anything which customers can't absorb. And so I just I think it's going to and it's going to evolve over time.

51:46And then, you know, product is going to have a whole set of skills. And then I don't know what we're going to do with our time, hang out on the beach. But I think it's a pretty interesting time to run orgs. Yeah, you know, I think engineers, designers, product managers, maybe it's just all going to be one blob of builders or something like that. And everyone, everyone just does things. Everyone just does things. And, you know, it's great. It's lowering the barriers to like just getting a lot of stuff done. And it's like so much fun when you can, when you don't have to ask somebody or get something on a backlog or whatever.

52:24you can just get it done yourself or even just get it done very fast in a small group. It doesn't matter what your discipline is. It's just like a great leveler at the moment. So yeah, so we're live. I think our lobster is live and it should be on app.intercom lobster emoji. Look at that. It's amazing. I need to get you all an affiliate code, you know. Yeah, I mean, lobster emojis, they're the new thing. They're the new growth hack. or the new growth hack. Okay, so we have seen your PR per R &D employee go up. We've seen how you can get from kind of cloud code to production very, very fast with a bunch of guardrails.

53:04We've seen your list of, it looks like hundreds of skills, but at least dozens of skills that you're invoking via hooks. You're using that to not only ship customer-facing product, but you're also using that just to make developer experience better, burn down tech debt, all those things we want to see. You all are, you're measuring it both from a telemetry perspective, both like quantitative and qualitatively. You're measuring your cloud code sessions and, you know, 2x isn't enough. You're going to get to 10x. So you all are on the edge, at least for folks that I talk to. And I'm sure you're like me where you're like, sure, you think we're on the edge, but then I see people and they're really on the edge.

53:44So So we always have ambitions to move forward. But my question now to you is, how has this impacted how you think about your customer's product? You know, I'm an Intercom customer. I'm a Fin customer. I interact with Intercom code and Intercom UI literally every day. My open claw has an Intercom API key. How do you think about, you know, now that you have this experience with clawed code internally, how do you think about what that customer experience is going to look like? Yeah, there's a few things going on. And one is that people are outsourcing a lot of decisions to their agents. And this is a good thing in many cases.

54:21But there was good research done recently about what does Cloud Code pick? And certainly I've had the experience in the distant past where I'd ask an agent to add something, except do it behind a feature flag. And then it would start to go and implement its own feature flag system. No, no, no. in our code base which has a pretty sophisticated old school uh home rolled feature flag system so you know nowadays mostly we'll stick to whatever's in the code base and that's fine um but you know sas products they're really good at their jobs they're actually worth paying money for and getting back to the feature flag uh situation you know if you're building a new business, you're relying on your agent to make decisions.

55:10Often an agent will, when prompted, it's like, hey, how should I solve the feature flag problem? I want to make sure I'm doing all these safe deploys and that. The agent will just go, yeah, I'll do it myself. And the kind of build over by decision. And you can see why the agents do it this way, because they can achieve this. They can get it done. They don't have to rely on the human. Okay, like OpenClaw changes things here a little bit and maybe computer use does as well but still we're not really we haven't really adopted um sas businesses to be agent friendly that means well all sorts of things around how do we position our websites and content and how do you get updated in their in their knowledge and how do they discover it and but also can they actually just get it done like can you ask an agent hey could you just sign me up to intercom and get me uh fin working on my website And so this goes alongside just having to make more APIs for things.

56:09I think I'm kind of like omni-channel as such. I think there's a feature for CLIs and MCP and REST APIs. I think I'd like us to get more comfortable around things like ephemeral APIs or multi-step APIs. I think CLIs are good at wrapping these kind of things. But the whole point of all this, where I'm getting at, is you want to be able to just help agents out at the time when they're interacting, they're in discovery mode. And you want to give them clues. You want to give them hints. You want to give them help to be able to do things like sign up for something fully without having to go back to the user and say, yeah, sorry, can't help you there.

56:43You got to go away and like figure out how to sign up for something. So I've been working on something in the last few weeks, which hopefully should solve a problem. I can paste in a prompt and then see how far it gets. I also, just while we're running this, I have to go back to your feature flag example because it you know where i used to work it broke my heart that build it yourself was at the top of the feature flagging list but i do think i have i have a paranoia moment about this which is model providers and harness providers are highly incentivized to build it yourself consumes lots of tokens versus buy it maybe consumes less so i'm i'm just really interesting to see how this all shakes out you know people people are very anti sass is dead um and i'm a little bit more like yeah but like the current form factor of sass really is has something coming for it in a particular dev tools because these models are so good at writing code i think you're in a real pickle to try to figure out how to find the right value wedge at the right moment, how you can allow agents to not just sign up and set up things, but purchase it.

58:06You know, what does your trial experience look like if your first user is an agent? I think all of that is super important. And then, you know, to your point earlier where you said, you know, are we APIs, ephemeral APIs, CLIs, MCP? I think the answer is yes right now, which you cannot predict the medium by which a user is going to come to your site. They could come through a search and hit your website and download things and look through your docs. They could come through cloud code. They could come through an open cloud. You just really don't know. And so you sort of have to meet your customers and your non-human customers where they're at.

58:45And I think it's really smart for teams that have any part of their product that needs to be implemented via code to be thinking about this problem yesterday because you will be left behind, I think, if your agent experience isn't there. Yeah, I agree entirely. And I think there's a whole craft in how to make, say, a CLI agent-friendly. I think MCPs obviously get that right a lot of the time. But, you know, for example, one of the things that we do in the help is kind of just give a hint to the agent. It's almost like prompt injection to a certain extent, except it's not malicious. you're just trying to get it along to what it's trying to achieve.

59:23It's like, well, maybe you could check email. And if an agent has access to your email. That's what I was looking at. So it's just they're going, oh, yeah, I can probably get this done. Or like you can hint to them, like, I've kind of cheated with this. So this is my own personal website hosted in Vercel. And it is, I've kind of pre-populated a few articles so they can upload and Finn has some contents to answer questions with. But you can also just return in the help going like, hey, you should probably think about creating some articles if you want Finn to actually start answering questions.

59:57And that can be extracted from the code base or whatever. Yeah, I've been also thinking a lot of interfaces, like CLI interfaces. I use GOG. It's part of the OpenClaw universe. And I think it's a lot better than the official Google GWS one. but I think if you start to use it it's actually just more human as in the interface just kind of makes more sense to a human I think the Google one is like I kind of get what they're getting at and there's kind of JSON in there and stuff like that it's not that but it feels more human friendly or something things that are effective for agents can often be things that are more human friendly because they're discoverable in these verbs and words and not just kind of inscrutable weird stuff going on in command line options I think I've confused Claude here.

1:00:50I'm not sure where is his name. That's okay. I'm going to narrate for folks what's happening here, which is you basically said, like, install intercom on this site. There's an intercom CLI that's, like, cool. I can access the intercom APIs and do a lot of this. My favorite part of it, though, is signing up, getting a verification email in your email address, invoking via like this hint, basically, of like, if the user has email access set up in however you're accessing it, go check for this verification email because we have we have a code in there that we got to snag. And because you're using GOG, which is a command line tool to access Google Workspace, you can go do that, pull that code in.

1:01:34And what I think is so interesting about that particular flow is, you know, I think AI is creating sort of race conditions in shipping across the org, which is like you can YOLO a CLI probably faster than whatever team that manages email authentication can change how email verification works. And so you're like, I'm not going to let that break my product. What I'm going to do is create a flow that I can I can use that sort of sticky part of the flow, AI brains and get through it. And so, again, your product doesn't have to be perfect for an agent to traverse it. And this is one of the things I'm actually really excited about SaaS is all those things that are just so complicated to do as a human.

1:02:21Multi-step forms and like nested fields on nested fields and finding, you know, categories and just those things that I would say UX designers and product managers have written their most tedious PRDs on and done their most detailed specs on. Like you don't actually have to worry about making that quote unquote usable because you can just brute force intelligence against it and solve the problem. And so I think that's interesting because the core value proposition can get bigger and bigger without being constrained by the surface area of a website or a UI or any of those things. And so I think if you're not thinking about what does that CLI look like for you and what adjacent systems does your product butt up against, it may be email, it may be some other dependency and how an agent might traverse those systems, you're just going to get less and less adoption because this is going to be more how people install products.

1:03:19Yeah, and if I don't poke holes, and if I don't make a CLI that kind of bypasses some of the ways their product works, somebody else will. You know, they'll just put their own agents on it, and they'll burn more tokens, they might get frustrated. You may as well shortcut them and give them an interface which just works. It may not be the perfect interface, but that's the beauty of these things. You can get updated over time, agents can just pull down the latest version. And yeah, hopefully I have something to show here, though. Well, the other thing that I want to call out while you're talking about that, which is as I'm watching this and it's taking some time to build, your conversion rate drop off point is somebody pressing the escape button and just saying, forget it.

1:04:04Like, this is clearly not working. What if we built it ourselves? And so I think it's a really interesting moment for product managers who right now are not getting the visibility of the drop off. right when you were going through a website you could put telemetry in it you could say okay users going to the sign up page drop off email verification drop off going to the docs drop off you could build this nice little funnel that identifies where your users are having problems you can put some telemetry in your cli but the end of the day some of that drop off and the alternatives is very invisible to you here and the the switching cost quote quote unquote is like pressing escape and saying, do it a different way.

1:04:44And so again, how quickly you can speedrun to a zero to one installation in an agent, I think is something that everybody should be running right now. And it doesn't just have to be a code product. Like I think more and more people are doing non-technical tasks and interacting with non-technical SaaS in Cloud Code, in Cloud Cowork. And so, you know, even if you're not dev tools, if you're not thinking about how can a user do this quickly in a third party harness or system or an agent can do this quickly, you're really missing out on customer growth. Okay, how are we doing? It's on its fourth attempt.

1:05:30That's fine. And you know what? Let's press the escape because you know what? Let me tell you how cheap that exercise was. it was like five minutes and some tokens and you're going to spin up a fresh cloud code you're i don't know if you put make no mistakes that was probably what we missed make no mistakes um and it and it could have done it and again this is just learning like why why aren't why isn't every engineer every pm doing this once a week or once a month just to figure out how it can work. I think it's great. So Ryan, you've shown us everything. You've given us all all the secrets. Let's get out of the terminal and let's do some lightning round questions.

1:06:15So my first question for you is, how does it feel? Because what I observe from our conversation is it feels fun. Like culture has in fact gotten better, not worse because of this investment. And so, you know, as a company that has really put in the effort, both on the customer side and internally, how do you think it's shifted culture? Has it at all? What have you observed? Yeah, everything is just faster and more exciting. You know, I mentioned feedback loops a good few times. And, you know, you can just get stuff out there so fast now. And I've been having the most amount of fun in my career over the last three months or something like that.

1:06:58And, like, it's fun in many ways. It's fun because I can do stuff that, again, I would have had to convince other people to do, or they were just things on my wish list and I could never get around to them. I just kind of complain about them. But now they're just realizable. But also the fun aspect of, like, making other people productive, like leveling people up, like removing work. I had like intercom is pretty good culture around resisting like the kind of slow movement towards being a large company and all this process and stuff like that and we're kind of in denial that we're like a large company um I think it's a healthy way to work in many ways and uh but this has kind of got us back to our roots in in a lot that's you know you you can make fast decisions and get them delivered and get that feedback super fast.

1:07:48And I've been able to ship actual features, like not just the CLI, but I ship some webhook features. And it's been a long time since I've done that. I've been in the weeds in platform space for a long time. But it wasn't even a big deal. It was like just a couple of hours, just kind of get something done. It was like something a customer asked for. So my job has become more varied. I'm able to kind of see more and get more done and help other people get a lot more done. So you get this kind of excitement and velocity increases. And, you know, we have all those measurements and that's all kind of good stuff.

1:08:20But just the excitement of waking up in the morning going like, I'm going to get a lot done today. Like that is a fun way to go about your day. I completely agree. And I hear this over and over and over again. I certainly feel it myself, which is this is the it brings me back to why I learned to code. It's like that same moment of I didn't learn to code because I like to type code. I learned to code because of the magic of you running like hello world and it shows up somewhere and that feels so it's just a very creative experience. Which leads us to my second question, which is, I see all the time that one of the most impactful change agents inside an engineering organization can be a senior principal engineer saying, let's go ham on some AI code.

1:09:04And the single most blocking person in the organization can be a senior principal engineer going, I don't believe it. Absolutely not. Not me. Not here. Not. No way. And in fact, last week, I heard a story of somebody who had their most senior staff engineer quit. Says, and I quote, I do not believe in AI. I will not work in a place that does this. So what is your appeal, sort of engineer to engineer, of why to invest in this? Why you think it's the way that engineer organizations are moving and how you kind of come to meet skeptics where they are and hopefully see things a little bit more from where kind of Intercom is approaching them?

1:09:44I mentioned that Indicon kind of had it on easy mode. We didn't have to convince leadership that there's something to this AI stuff. Like we were pretty much had decided the direction of the company the weekend that ChatGPT came out. So we already had this expectation that this will be transformative across many parts of our work, including all of building products and engineering. We were just kind of mostly annoyed about how long it took. But I think for sure it does need strong advocates. So you need to push boundaries. Like one of the biggest things that I've been able to do successfully was kind of push through the barrier of like, should we let an agent connect to Snowflake?

1:10:27And all these things can go wrong. Or should we let our agents run real production code in our Rails console over API? And the easiest thing to answer there is like, well, you know, I'm not sure. Or like, this is risky. Or we should think about this. but we've been largely pushing through it and they're like not recklessly uh like we've lots of good controls and we're a mature business and uh we have like i've been on our security team but definitely uh not trying to do anything uh too wild but there's still even then i have apprehension just like is this like i think i i think we should do this but it seems weird or it seems hard but then i just have to give myself permission and then i realize if i have to give myself permission there's loads of people out there who just need me permission and um honestly like one of the biggest things i do at intercom is just telling people they can do things and just there's a pre-ai and post ai and or telling them like look whatever you do just blame me if it all goes wrong and i guess maybe we can blame blame claude no but but ultimately it's that like permission and just like there's a level of ambition which comes from as well it's like if you're out there saying, I'm not sure if AI is going to take or have a big role to play in all of our work.

1:11:43I keep on saying that, that kind of will permeate through the culture and people say that. But if you're very clear, you're saying that like, look, all work is going to be agent first, like at some stage in the near future. And so we're going to figure out the path there. And so we're going to break down every barrier as we come across them. And look, it's your job, it's my job. and if anything was wrong, blame me. Like that's largely been how I've been approaching it, but not just me, like this has been a very large collective effort. But giving that kind of permission thing, but also the kind of like freedom to like explore or push things or whatever, it's kind of necessary.

1:12:17And look, it might be a less stressful way to go about it to like just take a nap for a few years and come back and then with all the problems have been solved and we've got these perfect agents running amok in our environments, then that would avoid some of this. But I think all places have to get through that kind of apprehension and initial kind of issues that some of the introduction of agents in the environment can have. And I think our job as leaders, whether it's as an engineer or as a manager or whatever, just has to be on that enablement and giving people space to go deep on the work, enjoy it and have that moment where things click and you start realizing like, oh my god this is uh something that will transform how much i can get done say it again for the people in the back i love i was like oh my gosh i love this so much and you know i it is absolutely those two things which is like give permission you you can please just go please by all means go ahead designer hit me with the pr no one's gonna get mad at you like go ahead and then the second thing of just accountability can roll to the top and not in a scary way.

1:13:28Let's not do irresponsible things. But, you know, we've seen a couple incidents in the past months, some big ones. And what you see is CEOs or big leaders coming out and saying, like, the team's shipping and we want to keep shipping and we're going to be careful with our customer data and we care for the customer experience. And stuff happens. We've learned from it. It's ultimately on me. I'm going to call the customers and we're going to we're going to move on and deliver great innovation for you. And you know what I tell people to, you know, to get them over that hump, which is like, you really got to know what your existential problem is.

1:14:04And I love what you said is the second that ChatGPT came out, Intercom changed because that is an existential problem. Who writes the code in your code base? Agents or humans? Not an existential problem. Like, will you be fundamentally disrupted by a new technology? That is the real problem in your business. So I always tell people like, let's differentiate the real problems in our business from problems that we can tolerate and then go, go, go use the problems we can tolerate to move fast. And so it sounds like you have a really good call. I mean, I think at the end of the day, the results speak for themselves.

1:14:38And again, you all are not asking me to say this. Intercom has meant the moment. You went all in on AI assisted, you know, customer support and experience. You're now building models. And so it's not just a one and done chat GPT is here. We need to change how our product works or AI assistive coding's here. So we need to change how our engineering team works. It's, you know, models are going to be how people differentiate. We need to go there. CLIs are going to be how people use products. We need to go there. And so I think this sort of like fearlessness and what I would suspect is like just a fun, nice, high trust culture, good people.

1:15:14you actually see the business results on on the other side so I'm gonna hype you up I see a lot of teams I see a lot of leaders and I think people can take a lot of inspiration from this but let's uninspire them really quickly before I get you out of here which is my last question which is when Finn takes 15 solid minutes on a live podcast to do a very basic task that you know it can do or not Finn when when clog code what do you do do you yell are you a yeller what does your what does your meta-analysis on this internal dashboard say the human needs don't prove on i i i do lapse into giving claw code like just like smiley faces or unhappy faces or you know not over the top i certainly haven't cursed at it very polite that's kind of not my smile but i do like the odd kind of like at a boy kind of smiley face and i don't know if it knows like that i'm deeply thinking about this and like these little subtle kind of hints or whatever but um yeah no i think like professional with a few emojis is my style uh with claude and you know hopefully that'll come back to me someday with an emoji same i waste the tokens on telling it it did a good job i somehow in my mind i'm like that's going into into its own sense of itself and it's gonna know what good looks like so i i am there I am there with you.

1:16:38All right, Brian, this has been one of my favorites. Y 'all, if you have gotten to the end, there is so much alpha in this episode. I cannot believe it. This is a cheat code to winning friends and influencing SaaS through AI engineering. Brian, where can we find you? And how can we be helpful? I can be found on the internet at a nice vanity URL, which is brian.scanlon.ie. And I got a few links here to some other talks and some other writing. and different bits and bobs. As you can tell, I'm not a designer. I asked Claude to design this as if I was a Unix systems administrator writing a little webpage.

1:17:14And it kind of shows. I'm active on ex-Twitter. I'm Brian underscore Scanlon. I'm on LinkedIn, Scanlon B or something like that. I think I'm the most famous Brian Scanlon on the internet. So generally, Brian Scanlon, that tends to work. And I tend to be active and like showing up to different conferences and just like getting good word out about what we do at Intercom, mostly these days AI, but I've also given lots of talks about many other different topics. And yeah, I'm also a big believer in just saying yes to a lot of things. So if you look me up, you've got a good idea, you want to get in touch, you want to run stuff past me or whatever, chances are I'll say yes.

1:17:52And I'll just keep on doing this until things break and then I start saying no. But I'm still not there yet. So bring it on. Great. So search for Brian and ask him to do something for you. That's it. Well, thank you. Thank you truly for sharing all this information. People are going to get tons of value out of this. It's going to be a hit for sure. And I just really appreciate you joining Howie AI. Of course. This is so much fun. Thanks so much for watching. If you enjoyed this show, please like and subscribe here on YouTube or even better, leave us a comment with your thoughts. You can also find this podcast on Apple Podcasts, Spotify, or your favorite podcast app.

1:18:30Please consider leaving us a rating and review, which will help others find the show. You can see all our episodes and learn more about the show at howiaipod.com. See you next time.