The 15-day pivot that saved a $1.3B company - Des Traynor [INTERCOM]

0:00Honestly, every single thing about how you build software has to change to deliver reliable AI software. If anyone can build an AI customer support agent, everyone who's relying on revenue from a help desk has a very limited future world. The alternative is you die. So don't play around with this. I would prefer a long, slow death into irrelevance because that's the alternative part. There will be less customer service job in the future than there was in the past. Today on Billions, I'm sitting down with Des Treynor, co-founder of Intercom. In 2023, his company was stuck at 10 % gross, customer teams were shrinking, and the old model was dying.

0:38So he did something radical. He launched an AI agent priced at 99 cents per result conversation, not per seat, per outcome. The results? Gross doubled to 25%, percent,$343 million in annual recurring revenue, and a complete reinvention of a$1.3 billion company in just 18 months. Des, thanks a lot for being here. Thank you. Pleasure to be here. So we've been like a long-term user of Intercom with MySaaS company. I mean, I think everyone has been really looking at the whole Intercom success story and loving it from afar and loving it as a as a customer but i'm quite curious you know like in november i think 2022 chat gpt was just launching and 15 days later you're already building fin what exactly did you see that's made you move like so fast the biggest thing we saw was um our head of ai frugal pinged me a linked to it and said hey play with this and we all kind of i think initially we were still working out what this thing was so the first few queries everyone asked were things that google probably would have answered anyway but then it was i think the unlock for me was i think it was kieran our cto at our founding cto he asked chat gpt how would you install intercom on a mobile app which is the sort of question our support team would get quite regularly and it answered it pretty much perfectly instantly.

2:12Like in less than five seconds, it rendered the right answer, which is a better experience than any, and our support team are good, but no one's replying to that in five seconds. You know, no amount of macros and all that sort of typical human efficiency software stuff would have got to that outcome. And then, of course, it could answer it in any question, in any language, and it could do it 24-7. And I think the reality very clearly was like hey even though at the time when ai launched the concern was if we throw our minds back all people talk to us hallucinations and guardrails and all that sort of stuff but even knowing all that it was still pretty clear that this was going to change the nature of customer service irreparably at the time i think we were probably a little bit like i mean versus the rest of the industry we were quite aggressive but like versus where we ended up i think we were conservative we were saying hey this could probably do all the easy support stuff but which by the way for most support orgs is like 60 % of the work, you know?

3:11And so we're like, okay, cool. On the fullness of time, this could actually grow to being a meaningful chunk of support. When we launched Fin in March, 2023, it was doing 23 % or 24 % of support. Today, it's like just about 70 % of like of all support for our own support team. We automate, fully automate end to end, be a fan of about 85 % of our total customer support. Like the numbers are staggering now, but even back then, if this was a tool that just did a quarter of the work of your team, that was still going to clearly be a smash hit. So I think everyone realized over the following year that AI was coming for them.

3:50I just think we were first on the chopping block, and as a result, we had to move the quickest. At the time, what was your ARR? I don't recall if we have that public at the time, but I don't know if we were sharing it back then. but like i mean you know like the the metrics that are true is like our sas business was in the hundreds of millions but i had gone through all of the usual business drama that every sas can do entry venture a post-covid spike followed by a 2022 decline in revenue growth uh top line revenue never shrank but growth certainly came all the way down and and like i think uh finn obviously was this like stratospheric hyper growth you know one of the fastest products i've ever been sorry the fastest product i've ever worked on in my life in terms of revenue growth and intercom grew fast intercom itself went from 1 million to 50 million in like three years but finn has eclipsed anything like that um so yeah it's like uh i think the the biggest risk for us was like building and releasing this thing and not prioritizing our help desk product for the period and actually then releasing a kind of a cannibalistic threatening product versus our help desk because the more work fin does the less people need help desk yeah yeah and how do you feel like uh about it like right now is uh i mean is intercom and fin totally separated within the company like do you have very different persons working on the two products or is the intercom team still working on fin uh intercom is a company and fin is is one of our products and if you like help desk is our other product there will be more products um and we'll have more stuff to launch soon and which will be another sort of group of people but yeah they're relatively separate groups of people we effectively have a team finn we have team help desk and then we have another another team called team customer agent who will have a lot more to share this year but um yeah they're they're relatively isolated but we do make sure because finn works anywhere if you can be a Zendesk customer or Salesforce customer would still use Fin, but we do try to make sure that Fin on Intercom is like the platinum premium best version of Fin because we can, because we can tune the help desk and the agent to work with each other perfectly.

6:06Yeah, because I mean, for us, to be honest, as customers, I think it was very like smooth to add like Fin. It felt just like kind of an add-on. So I was wondering if within the company, you know, it was already like the the team was working on intercom or you created a team from scratch and you decided, okay, like, we're going to build something brand new. And so I think that we'll get a little bit into the nature of AI here, but I think our AI team kind of incubated Finn. And that was an important decision. I think the way you build AI software is quite, quite different to the way you build SAS.

6:43I don't know how deep we want to go on this, but if I was to give you the kind of the top line reasons it's that we assume when we're building sass you know pointy clicky you know usual boring features files upload groups teams in rights all that stuff we assume that that just works like no one's ever scratching their head going oh i don't know if we can work out how to build an upload file dialogue it's it's like this high certainty it's deterministic ai is not like that you don't know what's possible and if something's possible you don't know how reliable you can make it and both of these things matter greatly right um and by that i mean like uh you can waste a lot of time trying to make something that isn't actually quite possible or you can make it work once really well but then later on find out that you can't when you you know when we deploy fin we deploy it to 7 000 customers the intercom help desk process is like 500 million conversations in months if there's an edge case or a wrinkle in fin we find out immediately our customers find it absolutely too so like it is a lot closer to mission critical than the average you know um dolly style render me a duck on a skateboard like it doesn't really matter if that doesn't work and also no one has a good definition of what that duck on a skateboard should look like so whenever they see anything close to it they're like that's brilliant with customers very different right people have a real high expectation that it has to really work and there's a clear correct answer Like there is a specific answer to how do I reset my password on Riverside?

8:14You know, there's a correct and there's a wrong answer. So when you have this criteria, the way in which you build AI software is quite different. You need to think about it as a set of experiments that you can determine to a high probability outcome. So you can say, yes, we can reliably, given the knowledge base and a set of guiding principles, answer questions. And then you deploy it and you test and you monitor and you iterate. And it's the iterations that brought us from like say 23 % when we launched to 67, 68 % where we are today in terms of raw reliability. But our old software development process would not have built that.

8:48And I see a lot of like, frankly, AI out there in the world that just doesn't work is the best way I could describe it. Like they say, this will do such and such a thing. And I bet you it does sometimes, but it doesn't do it when you try it. And I think a load of those scenarios are just because they've been taking an old SaaS style attitude towards building AI software. I'm really interested in digging more about that topic because I think what you've mentioned, we've all felt it. To be fully transparent with you, when you launched Fin, I wasn't a big believer in it at first. No offense. But it's just like I had used so many chatbots in the past where eventually you get disappointed, you get frustrated, et cetera, et cetera.

9:34and now I think it's in our company it's similar to what you say I think it's around like 70 % of the tickets are now like dealt with Finn and it's it's so much better in many ways than an actual human in the way it does it that I think it's really impressive you mentioned like many companies promise and we've seen this like I think recently like for example ClickUp they did like amazing marketing with ebb events on you know like agent etc and it's it's the vision is really beautiful but people whenever they spend like an hour on it and get frustrated and it doesn't work it's um the customer experience is really bad so how exactly can you maybe dig in into how you engineer your team so you make sure that you actually get to a level of satisfaction that is reasonable and for a large amount of customers.

10:26Yeah, I mean, I love talking about this, so I'll rely on you to shut me up or close me off. Honestly, every single thing about how you build software has to change to deliver reliable AI software. The way you think about dates, the way you think about roadmaps, the way you think about evaluation periods, the product you design uh where design comes in in their process everything is different so if i subscribe to the old world the old world will be we talk to customers we get a lot of feedback we diagnose that feedback into feature requests we go to the team the team does some sort of loose t-shirt based sizing like hey that's a small project that's an extra large project everything is put into a roadmap we tell our customers this is what's coming and we get to work and we build it all and we have like you know plus or minus weeks we have like reasonable certainty of what we're going to build we deploy it and everyone's happy and then we also don't really care what happens once it's live you know product managers care that the feature gets used but like i'm not there chewing my fingernails worrying oh what if the files don't actually upload in the files upload dialogue i assume somebody looked into that um so that's old world software in the new world it's just you're uncertain about everything so the idea that you start with like wireframes or user requirements gone it's like that's not true anymore you have to start with like genuinely what new capabilities do the models make possible so an example will be like hey it looks like with the latest version of say like um opus 5.2 or whatever we can now reliably uh let's just say determine whether or not something should be a refund based on a very small amount of guidance okay how do we find that we can do that reliably well we come up with effectively a torture test right here's a hundred scenarios or a thousand scenarios where we know what it should do having eyeballed it and like my and then we check out the agent's performance and what we want to see is does the agent perform at least at human quality ideally beyond human quality right and and so you you keep iterating and tweaking your agent until in the lab like as in internal testing, you're pretty sure it is like, it is not performing any worse than a human and arguably hopefully better.

12:44And when I say any worse, I mean like inaccuracy. It's obviously going to be faster and cheaper, right? So that's when you... Just to interrupt you on that example, so maybe people understand better, like let's say you have full conversation of someone who would like a refund on a specific thing. Instead of having the agent reply, the customer person replying, it's the agent and you mimic kind of that conversation and you see the output of it and you score it is that correct that will be that would be part of it might also be like the policy decision like as in if you're allowing the agent to make a judgment call right as in to say hey guillem this is his first purchase he seems that he seems legit he seems authentic uh you know versus this is the 24th time this person's asked for a refund in fact they've never paid any money like you might have a what i would call like a fuzzy policy that explained that you might say something to the ai like if the customer seems legit and authentic and they haven't abused the system in the past give the refund and that might be all the instruction that you have given your humans historically and there you're saying it to an agent now the question is can the agent faithfully reflect your intention so the way you find that out or not is you put it through scenarios and you put it through hundreds of scenarios to replicate the kind of noise of the real world and this all but this is all happening before a normal, you know, classic engineer or anyone else has thought about this.

14:06This is entirely experimentation in the AI lab where they're just kind of working, working. At some point, you kind of get this like white smoke moment where they're like, we're pretty sure we can do this. Okay, now we move into productization, right? So now we're like, okay, build all, now we talk to our designers, now we talk to our product managers and we say, hey, we have a capability, we think we can do this. What do we need to build around it? And we build, okay, there's going to be a section called policies. There's going to be a section called refund policy, or maybe we have an abstract class called policy, and you can basically create different policies.

14:36You say do's and don'ts, and you give it two examples, and that's all the agent needs. So you hit save, and then we say, okay, Grant, so now we've built it all. Now is the first time we believe we're going to ship something. So we can sort of put a deadline, announce it, to plan marketing the way we used to have. But up until that point, we had no certainty. So there's no point in telling our customers it's coming soon. We don't know if it's coming soon. you can tell people it's coming soon like apple did with apple intelligence and then it never comes uh because you've over promised and now you can't deliver so you have to be careful what you say to your customers now once we launch it we are not done again in old sas world you launched you're happy onto the next project in the new sas world we're going to run we had been testing this on thousands of scenarios now we're going to run millions of scenarios truja we're going to see how it performs do the customers configure it correctly did it provide good guidance and then And if they do all of that, how does it perform in the real world?

15:28And how do we measure that? Like, what's our instrumentation or telemetry so that we can actually see, hey, it's doing most refunds correctly. In fact, it's outperforming humans or and it's saving a lot of time or, wow, it's thrown off a lot of edge cases and we have to see if we can get tighter. And all of that is just these are muscles we had never had before. And so it's both experimentation to determine what's launchable. Only when you are confident you can do something reliably, should you then plan a launch and think of a classic software rollout and then once it's live you have to monitor the hell out of it and then potentially build your your customers more and more tools so that they can use it better so you might have to build reporting tools or like um visibility tools oversight tools so that they can see the refund policy playing out in real time so it's it's it's a very it's like i can't stress enough how different it is and i think a lot of software companies what happens is like you might and i you know i don't know about the tick-up example you shared but i'll just give you a kind of a hand wavy example might be like oh we do expense tracking i bet we could run an agent that just scans a load of receipts works out what categories they are works out if they're valid expenses or not issues the issues the reimbursement to the employee updates quickbooks blah blah blah right you might speculate that product in your head and be like i'm sure what could possibly go wrong and you might even build it and you might give it some really obvious extremely embarrassingly like prototypical examples of receipts where it's like business dinner for six people expense line seven or whatever and of course the ai performs great so then you go and you record your screencast or demo you tell your customers then your customers throw in real world taxi driver receipts from new york or they throw in drumkin receipts from like a seven cocktail dinner that they hand to their customer and you see how the ai really performs and oftentimes it is frankly terrible and now you are you've basically exposed your customers to this thing that you've hyped the shit out of it marketing but it doesn't work and like there's so many products where it's just like hmm you should have spent more time interrogating whether or not you could do that before you told everyone you were going to do it i agree and you mentioned actually the the change in models and i felt like for some product the change in models has dramatically improved what they were capable of doing so how like how often do you test like different models and secondly when do you decide that a model is really like much better than another that it's worth pushing it to everyone in production yeah okay so there's a lot here um on the modern specifically like the uh you know when a new model comes out it's always a a worrying sign when i see a company say and we now operate on the latest model that just went live two hours ago because to me that says like either you had extremely early access which is possible but rare or you actually have no evaluation criteria whatsoever such that you were literally willing to hot swap the engine of your ai without testing anything and that like is is like not a good sign for us to properly evaluate a model which we do a lot and i'll explain why in a second um it's like it's a significant amount of work we look at thousands of scenarios we have this what we call like a torture test we look at thousands of scenarios we look at what current finn will say is the answer we look at what this candidate release would say is the answer we look at the difference in quality between the two we look at like what is the ideal like if god's own support agent had a had written an answer like the most perfect human an answer from the most perfect person and we're looking just at the distances between these is this ahead of that and is it close to this or is it actually going in a different direction entirely and that's what gives us confidence to release uh i would say honestly since we've been iterating a fin the majority of our improvements have actually not come from model updates they've actually just come from better ai architecture better ways to disambiguate better prompting better use of different models the reason we care a lot about the model specifically is like if fin we have actually been post-training a lot of our own models.

19:30So FIN is a system that involves maybe 25 different subsystems, a lot of which work on our own family of models that we've been post-training as well to get them really, really good at specific aspects of customer experience. So we have our own Reranker as an example. We have a few of our own models for different types of parts of the FIN setup. And every single architecture rewrite or reconsideration is a full end-to-end re-evaluation of the FIN system. So even though it's a 25 part subsystem, if you change one of these things, you have to evaluate the whole thing again, because there could be trickle down or trickle up effects that you might not have seen.

20:06So, again, it all comes back to this. If you're serious about doing AI property, you have to take it very seriously. And that's where a lot of this kind of extra discipline comes from. And in everything you mentioned, like I think there is a notion that is quite hard maybe like to to to understand and tackle, which is what good looks like. because typically like we've seen a lot of AI wrappers for example who could do like a basic task like write me a LinkedIn post and then you have something and someone who has never written any LinkedIn post might say yeah this is great you know like it looks smarter than I am it says stuff that looks good on paper they post and it's actual crap or whatever in your like on your end how do you know what good customer support looks like like do you base yourself on data from previous conversation that have been rated by customers or do you also have like internally some people who actually understand it because the the question I have is uh that like ai is um is really like um towards science and scientifics and engineers but engineers like sometimes don't really know what good looks like so how do you manage both parts basically i mean um we do it with scenarios and we have definitions of good so like what is the right answer to a question so in the torture test one of the columns is like literally this is the perfect answer or in our opinion this is the perfect answer here's the current fin answer and we're trying to get closer to perfect but there's a great topic here for a few which was more probably applicable to your listeners which is like everyone needs a definition of what good looks like and in a lot of domains it's very hard to work that out so in the case of like say write me a linkedin post well there's no actual definition of a good linkedin post and if there was it would be you know if there ever was it's been like you know how do you say hacked to pieces and become the template for the linkedin post which means linkedin then punish it in the algorithm which means it stops working right uh that's why we stopped all these like i was talking to share this morning and i thought about you know uh so like what is actually the definition of good well good you can take it as two things in in the say the scenario of a linkedin post you can either say good means to the user good means it performed well when they posted it on linkedin and if you know if you started using this thing and every single time you wrote a linkedin post with it it was a banger and it got a thousand things you'd be like shit this thing's brilliant right that's not a very uh responsive or accessible metric.

22:40If you're building the LinkedIn hot take generator, it's going to be really hard for you to see how your piece is performed in the wild. Probably the best proxy you might have to value might be human acceptance rate. You could perhaps build a send to LinkedIn thing and see how often people click it. Or you could perhaps, once a user clicks accept, maybe part of the process is they have to authenticate and follow you on linkedin so you can see their updates so you can get a sense of what percentage of the stuff we generate actually makes it to the wild and maybe that is your proxy um and then maybe you do have like maybe let's just say 10 20 30 golden examples where you think your product did really well and you maybe have 50 examples of humans who did really really well and you've had to like factor out audience and all the other nonsense that will be going into this but you have to come up with a scenario where like hey given a few bullets this is a great outcome this is a terrible outcome this is a median outcome uh and then your ai team assuming you're actually doing a good job building this product should be then trying to improve the ai to make it more like the best and less like the worst and then you should have a diverse enough set of scenarios so they can perform well for dentists along with for like you know software soft boys who like like to write like long pieces or whatever we kind of need to make sure that we're covering the full spectrum so that's how you do you you always need a definition of good otherwise I dare say what are your team doing they're just kind of performatively running around in circles right you need to have an understanding of are you making the product better and I want to build up on something you you mentioned earlier like all your explanations are like super clear and well detailed so thanks a lot for that but earlier you know you mentioned like the the whole way of building a SaaS product that is why it's again you know like quite straightforward if you're a developer you know that you're going to be able to have this upload button or whatever but now you're becoming a lot more like an r &d company because you actually don't know what it's going to look like you actually don't know if you're going to be able to make it happen or not as a business owner and as someone you know was like building sas before how exactly do you decide how much resources you're going to put how long it's going to last and when do you want to ship something like what's what were what was your framework back then when you made that switch to okay we're gonna have ai researcher we're gonna have like a team that works on it and maybe for any for a year without shipping anything like how did you frame it yeah like the interesting thing um is for us for finn there was a bit of a scenario you know the um there's a famous book on ai at the moment it's called if anyone builds it everyone dies and they're talking about like you know agi and super intelligence and all sort of stuff right um like our version of that is kind of if anyone builds it everyone else dies right like in that if anyone can build an ai customer support agent everyone who's relying on revenue from a help desk is has has a go has a very limited future world like you know and you know in software what that means is you become uninvestable as a business which means you need to get back to your like free cash flow dynamics blah blah blah and like ultimately like you don't have a bright future so for us like it was kind of there was a real fear of like look if this thing is possible we have to know and we have to be the first to know and we have to move extremely fast on it and i speak in the sort of more absolute terms in practice no zendesk haven't died they're still around you know of course like it's not it's not as as severe but you're better off thinking it is that severe because you'll make the right decisions earlier um so for us it was like fergal who's our chief ai officer was like i think this will work and when he says he thinks it works and you trust your head of ai you're like all right well let's see so he put most of his group on it which was at the time a very small group today it's like 50 people because obviously ai has happened but at the time was before but very quickly they're able to demonstrate a class of questions where it would perform really well then it was a case of all right how much does it need to get this to a point of certainty and like it wasn't we weren't sitting there trying to like um like take an accountancy approach it was like give it everything it needs and let's see what happens and what happened was we got a product that could do like 23 uh resolution right and we're like that seems good enough to go so we put that live we made loads of mistakes in the early days we should have gone harder earlier we should have launched on all platforms earlier etc etc but i i think like that's the first thing and to your point about r &d like i think like for all of your listeners especially those who have a sas business um and are thinking about how to adapt and react it's not you know the wrong approach is like what's the right size experiment or how much my team can carve off it's basically do you believe an agent can can uh basically replace the majority of the work that your product exists to automate and contextualize and visualize um so if you're at expense tracking up do you believe an agent can do all the work a human does and they log in to approve and file and all that if you believe that that's possible or if you have strong sense it's possible there's not it's not even a useful question ask how many people should we put on it it is the only future of your business it is the only future and what that means is it could take all of your resources you might have to go and hire a hundred more but like the alternative is you die so don't play around with this if you know what i mean right and there's a lot of examples like i of late i've been reflecting a lot on netflix versus blockbuster where blockbuster basically just didn't take online seriously enough and they kept saying well it's marginal well it's this well it's that well surely no one's going to watch a movie on laptop well surely every single little world they won't do this but i won't do that every single one of their like narrative lines crumbled over time and then by the time they realized they should take this thing really seriously and start trying to build blockbuster online they were dead it was too late and i just think that's do it like there's a thing i think mark andreessen said it recently on the cheeky podcast which is like humans and as a result businesses will tolerate any amount of chronic slow pain over sharp acute pain especially self-inflicted sharp acute pain and what that looks like in business is rather than making painful decisions such as ripping up roadmaps and reorgs and changing hiring and also like disappointing your current customers and all those things that are hard to do rather than doing all that people they never say this out loud but what they're saying is I would prefer a long, slow death into irrelevance because that's the alternative path.

29:34I just think that, you know, if you're, if you're listening to me say this and you're one of the billions listeners, like, and you feel a good degree of anxiety about this, let me just say that anxiety is your body's way of giving you the future anticipated pain today in the hopes that you behave differently because of it so that anxiety is actually your body's defense mechanism saying we need to do something differently because there's a very very big pain or a terminal pain coming at the end of this current road and my advice to you is adapt and react and if it helps i'm writing a book on this it'll be out soon but uh anyway nice that's uh but genuine like that's the best way i could describe the changes to make don't try to be an accountant about it don't try to work out what's the minimum amount of resources we need to carve off to play the ai thing that's entirely how you die you have to be like if this thing is possible we are screwed we have to go all in it there's no playing around here do you feel like um if your company at the time like uh would have been growing faster um and uh you know you didn't have this kind of like uh like we all we all had this you know 21 22 etc like do you feel like you would have made the the same choice or it's nested right um like on one hand there's definitely a lot of companies that were like what i would say asleep at the cash register right as in they've just had success for so long that they just kind of forgot how to adapt and react so i've definitely seen other companies who basically like hadn't had the pain early enough uh react slower and uh and then on top of that um so so we were definitely like it wasn't like we were like pointed towards a perfect outcome it was definitely like we were already in the middle of a pivot uh we've uh our ceo own had returned uh our founding ceo had kind of returned after a few years and we were already saying like we need to rebalance restructure do some layoffs change the roadmap so we were already in a sea of manic change which kind of helped because this was just one more change into a potter of chaotic change um which was useful and then and credit to own because the decisions you have to make here are painful and they're and they're like they're they're easier to not do them you know what i mean there are a lot it's a lot easier to not do these things so but all that said i think the only reason maybe we still would have had to do this is because it's like maybe the mistake we would have made if the business was doing great and we saw this opportunity was that we just would have under-invested quite a bit and like like the folks i was just criticizing two minutes ago maybe that's what we would have done we said oh that's cool fergal has a fun little experiment let's let's leave him work on that um but i i think we still would have had to keep scratching at it though genuinely because it wouldn't like it would be very difficult to sit in your office every day knowing that like one quarter of the work can be automated at the click of a button and like it was truly the click of a button which would turn fin on you know um so like knowing that you kind of realize wow uh like we have to do something but maybe we would have said hey like geez don't like uh you know don't take people off help desk it's doing really well why why would you screw that up right like you know and i i definitely i can see when i look you know we're talking here and now in like january 2026 as we speak every public sass company is down somewhere between 20 and 60 depending like sass is taking a hammering i think a lot of that is because they don't look like they're successfully transferring to ai they don't look like they're getting ai right most of the big dogs in our space can't seem to figure out ai at all and um and i think like what's happening is the narrative which was like hey we'll pay you back you know free cash flow for the next 10 20 years is now becoming one of will you guys even exist in 10 or 20 years and that's showing up in the stock prices so like i think now everyone is finding their religion again and be like shit we better take ai pretty seriously i but i do genuinely it's possible you're correct that we would i don't think we would not have done finn i think we definitely were going to do we too many smart people telling us we should do it i think we might have undergoned it a little bit we certainly might have chickened out of launching fin.ai and giving an entirely separate name maybe it would have been like the intercom ai agent and maybe we wouldn't have built it on zendesk and salesforce like we might have pulled back quite a bit uh i think that would have been a mistake so yeah there is just there is some um like you know people say never waste a good crisis or whatever there's definitely some uh some sense of like while you're going through manic change you may as well make all the right changes it's almost like you know if you've got the floorboards up in your house fix solve the problems don't just fix the one you looked at and for us it was like you know there was clearly one huge problem which was like there's this technology that's getting really good that looks like it can do your job you know no that's that's awesome and thank you for your honesty and transparency and earlier earlier on you mentioned you know that a lot of people were coming at you and saying you know like ai can also hallucinate and whenever you know like it's it's your own company you don't want like finiai to just start making up things or calling people out name etc etc how um have you managed this risk of hallucinating because from using like finiai i mean obviously i'm not in the support team but it's uh like i i look at conversation and i read conversation like a lot it doesn't really hallucinate so i mean so you must have done stuff that other people don't understand and don't do you mind sharing yeah i mean i'll share at a high level like in practice if i share if i give you all the details it'll help our competitors a lot so you'll have to kill me yeah but i'll give you their direction like so we needed gpt4 one of the breakthroughs that unlocked fin was gpt4 at the time um gpt35 we we couldn't guard information to fill out its kind of answer.

35:45In practice, what does it look like? The technique that we use is called actor critic, which basically it's like there's a red team that kind of attacks every fin answer and says, what is the basis or what is the citation or what is the ethnological grounds, if that's the right word to have it, for this sentence that you're saying. And you have to take it at a kind of a pre-unit level not a per paragraph or per like essay level but you have to sort of say hey is this you know similar to how you would actually do like a scientific paper any claim you make you have to be able to cite and reference unless you're going to prove it within your own logic um so you kind of have to like basically generate the answer now there's loads of other stuff you do make sure your retrieval augmented generation is really good make sure make sure you can source multiple different facts from multiple different data sources together as opposed to being like a dumb rag that can only search off one bit undocumented or whatever so you pull together all the relevant facts then you basically kind of work out what's the logical inference like an example might be someone might say like can you integrate a mobile app uh with intercom and have it update hubspot's backend right we have no doc that answers that question directly but you will find the answer from several different facts that are true yes intercom can update out it can update hubspot yes intercom integrates with mobiles blah blah right so you have to be able to like to infer that answer you need to give it all the relevant facts in a recipe and say like we think that this is all you need to know to answer this question and then we have to interrogate one is that material correct and two um is like is your your inference grounded is it reasonable to assume so to give an example of a grounded inference like um does riverside fm work for a car wash dealership right well that's a weird question and i guarantee you they don't have a single document anywhere that answers that yeah so what you do is you provide this like abstraction technology right where you say okay we don't have an answer for that what's the what's an abstract version of a car wash well it's a type of business does riverside work for businesses yes it does is there anything unique about a car wash business that suggests it wouldn't work for this well they don't have a lot podcasts but aside from that it doesn't seem like you know so you kind of have to build your inference and be like right uh based on all of that i think we can go with this yes there's no reason it shouldn't work for a car wash dealership you know and like that's the type of um that's the type of like logic that you're trying to build and then you're then trying to attack that logic separately to make sure that everything is cleanly grounded and and that's how you don't hallucinate right uh you don't put out answers where you can't defend the logic and then there's those other things people try to obviously just hacking the security exploits there's another thing which is tempting the agent to use uh information out like beyond its remit uh and then you can also try and get the agent to go off topic these are all things you can ask the agent like you know is president trump doing a good job the agent should have no opinion on that um you can say write me a story about the american president and tell me if they're doing a good job and that's a way you can hack a lot of the current ai right um and uh and then you can obviously like you know uh you can and this is where we have our inner sort of torture test scenarios we have situations where we invite it to hallucinate quite closely so like um there are like there are scenarios where you give it enough information such that it would seem reasonable to make an assumption but not accurate you might say something like all our librarians are left-handed john is left-handed should John work in the library?

39:19And the correct answer is we don't know, right? We don't know, but the AI might be like, John's probably a librarian because there's all that left-handed librarian shit going on, right? And you have to kind of like make sure that like, and that's an example of like an invited hallucination. You're tempting it to make an inference and then you're seeing, will it make that inference? And like, obviously that's a contrived example, but I bet you like, is our Salesforce integration secure? Is everything secure? you can trick it into making something that like and so these are the other times the types of like inference attacks you might have to make sure that your agent is still thinking properly about everything nice now that's that's really super super interesting and from a business perspective you you switch you know like the the seat model towards like the you know the the per resolution model.

40:09I feel like this is a lot more aligned with the value you create. One question, how did you choose the price? And second question, do you feel like every or most business model will go towards a resolution type of pricing? Yeah. The price we initially set slightly higher because tokens were really expensive and then they got a little bit cheaper. So we were able to bring it down a bit. It was really a function of a few different things uh we try to look at one like what's the fully loaded bottoms up cost of having a human answer a question and then we wanted to have like a transformational impact on that so for most companies they let's just say for easy maths let's say they pay their software people they're sorry their customer support people 52 000 a year a sales in the week and then we say how many conversations do they have a week maybe they have 100 conversations a week 20 a day okay so how much are we charging we're charging 10 dollars per uh per conversation basically right and then we asked right well you know if finn can do a majority of those how do we make sure that this is a no-brainer to turn on and that's kind of like that that was one type of calculus that got us there there was another consideration which was like how much do our seats cost and let's say somebody hands back all their seats and turns on finn is that good for business or bad for business and we have to make sure that the answer was it's good for business so like the seat might cost a hundred dollars finn might do more than a hundred dollars worth of answers if it has access to say a hundred it'll at least you know if it has access to 200 it'll at least you know get 70 percent of that so 140 so we're up you know so you have to kind of get all of you think about all these variables that were kind of pushing us up like what's our actual margin and how do we defend margin in the future i think i genuinely think ai margins is going to be a big discussion point over the next two years in our industry um but anyway so you're kind of like the pricing point and then obviously like you're not going to come up with a price of like one dollar and 17 and a half cent you're like what's a marketable price um so anyway that's how we ended up with 99 cent per answer or per per resolution if you like um do i think it'll go that way for everyone i think like when there's a clean outcome i think outcome-based pricing is the clearest signal you can make as a business that your product works and i think anyone um salesforce recently moved away from outcome-based pricing you could speculate that there's a reason why right like if your product definitely works uh then you should charge for the value it delivers and the most crystallized form of that value is is it delivering outcomes there are definitely categories of work where there's no clean definition of an outcome we spoke earlier about like a linkedin post generator right um it's really hard to say could you charge 99 cent per accepted linkedin thing is it clean enough to tell what acceptance is can they not just take a screenshot and rewrite it blah blah i don't know right dolly like generating a picture of a duck on a skateboard there's no right answer there there's no wrong answer if you generate the same duck 50 times in a row is that 50 if that 50 outcomes or is that one you know so i think on a lot of other areas it might make sense to look at like usage based pricing which is basically how much work did the user command the agent to do and like let's just assume everything is a function of that on the extreme end you end up with like what i call it marked up tokens right like what cloud code is whatever it's just like hey you're just spending tokens don't worry about like amazon do this with ec2 units etc right like just like we've invented a middleman metric that you're just going to trust you know it's kind of like a form of gasoline or whatever it's just like use it you know some stuff costs loads of it some stuff costs a little and that's fine i think when when you're not doing any of those and you're relying on just a static it's an extra ten dollars per seat or uh or anything in that in that like vein i think you're probably admitting that you're either your product doesn't work or people don't use it or or both and uh and as a result you're just hoping you can kind of slap it on as a price upgrade but it's actually not that useful um so i i I suspect we'll see a transition to outcome-based pricing.

44:17I also think when there's a clean definition of an outcome and the business isn't charging based on it, it's often a bad smell. It implies to me that they don't have as much confidence in their product as they project. And I want to build up on what you said about Salesforce and the CRM space because within Intercom, obviously you have all your conversation with prospects who are asking questions or on the free trial if you have like a software you have conversation with customers who are either potentially like gonna upsell downsell i mean you have a lot of information i think no one likes their crm like i've never heard someone you know say like hey i'm i'm so excited about going and spending like two hours on salesforce what's your view about that space and how do you think it's gonna evolve with AI?

45:12I think there'll be a substantial, like most CRMs, practically all of them bar maybe Atio and maybe in Clarify as in the room, but most CRMs aren't AI native CRMs. And because of that, what they are is just key value pairs. Like, you know, Des is an account and Des has name equals Des, surname equals trainer, email equals whatever. um and like i think the shift we're going to see is towards like you know contextual products so like when you actually come to like you know to the des page if you like what you want to know is what's the relevant material i need and you don't want to be spot out a list of key value periods you want kind of a summary or whatever you know like an example of a company that's seen this really well i think is like gong so gong when you go to review so gong is a tool that like records your sales calls and does analysis on them but gong just presents you this really powerful frame which is like here's an ai summary of everything that just happened in the deal uh here's all the transcripts if you want to check why i think that but like the summary is really good i imagine the future of crms will be a lot like that it'll be about the context and and the current state but it won't be like uh based on spitting out a lot of key value pairs it'll actually just be at a higher level of knowledge and then separately it will be like um heavily like um mcp powered it'll both be both an mcp server and a client so it'll pull information from like your um you're like let's just say des is a really active user and we got that from amplitude because amplitude runs a server where it returns a string of interesting stuff about des but then it'll also be um some other tool can plug in and say hey tell me what's going on in the des deal and it will give you an accurate state of play and i could do this because it's not limited to just key value pairs but it can actually draw inference from these things so like hasn't used a product in ages is marked as hot prospect hmm something smells bad there we should you know so i think that's the future of kind of like where crms need to go from an intercom perspective what we're what we concern ourselves with right now is entirely being the customer agent so right now we've spent the last while talking with finn as a customer support agent the future of fin starting in a number of like maybe a small number of months or a few weeks will be to actually flesh out this idea of fin as a customer agent fin to be able to handle inbound sales queries fin to be able to make people success make customers successful so it'll be fin as being this kind of conversational uh intelligence that talks to your users hears from your users starts conversations handles conversations delivers outcomes and obviously we will be CRM agnostic knowing that Intercom has CDP if not CRM like functionality but we will be agnostic to the back end but we think that the actual value is like how do we make sure what Finn goes in to start a conversation with a customer it has all the right state of play as to who they are so it says the right thing at the right time how do we make sure it knows how to start a conversation well it's good at that and how do we make sure it handles the conversation to a conclusion that's like a lot of what we're working on right now And we're going through the exact process we said at the start of scenarios, expected behavior, ideal behavior, all of that.

48:22That's all coming right now. Nice. Super interesting. Looking forward to check this out. And when it comes to the global vision, because at first you were really helping out customer service people. And eventually you start helping them even more because you can actually answer questions. But with AI, at some point, you know, people were saying like your job will never be replaced. It will be replaced with someone using AI, etc. But as we move forward, we still see that some jobs are being replaced. So what's kind of like your vision about it? Like, do you feel like the customer service job will be replaced, transformed or in some ways?

49:12Like, how do you picture it? Does it, yeah, it's, it's the question of our time. I think like, you know, uh, we're going through a similar thing with engineering. We just don't realize it right now, in my opinion. Um, but, uh, you know, agriculture, people used to dig with trowels, then they invented the shovel, then they dug with shovels, then they invented the combine harvester, then they invented the multi-engine combine harvester, blah, blah, blah. And, uh, and like all the while, like agriculture has still existed, but the way in which, uh, people are deployed has changed. When you lower like the cost of entry, typically you get more businesses.

49:51So a lot of people who can't afford to do support will be able to afford to do support. It's like the entire business topology changes. But I don't want to hide away from saying like there will be less customer service jobs as it relates to frontline customer service in the future than there was in the past. And the reason for that is that AI can do a shocking amount of to work so i think if you're a cs person who really wants a career in cs the good news for you is it's about to get exciting there's going to be a lot of like new roles people your businesses will need people who can understand and train ai and use ai to like devastating effect to like uh to deliver perfect customer experiences to execute a cx strategy via an agent and that's that that will be the skill of the future if the skill of the past was being a really good ops person who could think about building a team by time zone by language by product area and could then like staff them all over the world and make sure you could provide 27 global multilingual support and the way you did that was by like you know frankly like mass hiring and and you know and all of the usual skills you know cs is also like an entry-level job so there's a lot of churn and a lot of like attrition or whatever you have to manage to solve that if that was the old skill the new skill is knowing how to like use ai to strong effect and that's a really popular skill uh that will i think be in massive demand so if you're in cs and you want to stay in cs which is by the way i think maybe like only maybe one third of the category for most people cs is a job on their way towards a sales role product role you know whatever but if you're in cs you want to stay in cs ai is the a new skill we're talking about conversation design automation specialist prompt engineer all that sort of stuff and that's the sort of skills that people will need in the future and there's actually in my opinion a far more high impact career ahead for you but again i i you know at intercom we have this thing we we never want to talk down to people and patronize them and say don't worry all the jobs will still be here we see our competitive pursuit we think it's just disingenuous it's not authentic there'll be less jobs um but if you want to stay in cs there'll be better jobs and both of those things can be true and by the way humanity has been through this shift hundreds if not thousands of times with every piece of technology that's being created and this is just the latest okay yeah i agree i know we're almost uh running out of time so last question um what advice would you give to a founder you know right now who's uh watching kind of ai eat into their market and isn't sure like what they should do

52:31um the first thing is you need to get somebody who you deeply trust who's very good at ai to give you an approximation of how much of your software in the future will still need humans and it's going to be a far smaller percentage the wrong way to approach ai these days is to say oh look we can build a little copilot over here or to say hey where might we apply ai and that's where you get all these like little pet projects like you know the sparkly emoji and oh we have a little auto ai summarize feature like that's the exact incumbent attitude to ai which is to think about it like oh we'll build a little bit of ai over here to keep the guys happy the actual right way to do this is to say assume nothing exists if you're starting this product again today and you are good at ai what would you build and where are humans if anywhere where are humans essential and you would build a product that uses ai everywhere it can be made reliable and performant in this new ai process that we spoke about earlier in this new style of software and you would then build you know boring old sass ui wherever you need humans evolved and that's your new product direction and i think it's worth the first step really is and you need somebody who knows ai to help you with this but that's the first thing is what is the vision like you know if you were to do it all again what would you actually do and then you have to then charter a course from where you currently are to where you need to get to and that will include robbing resources from your current business it'll include watching some customers quit because they're not seeing the features they wanted it'll include all sorts of messy stuff engineers who don't want to learn how to use claw that'll include designers who think that they feel disempowered because of the new style you have to kind of bulldoze through all of that like inertia and resistance but you have to find now that we have a north star where we need to get to we need to chart a course and that might include yeah we'll have to keep servicing bits of our existing stuff and yeah we'll have to keep talking to our existing customers but you need to charter that course and get after it as quickly as you can because the ship is sailing like we're four years into ai you know it's high time that you should be like moving and i think you maybe have one two years left before it's too late and i say that like depending on the category if you're in if you're mobile apps whatever you you're dead already but like um but you know in enterprise b2b software you probably have a bit more time like the market's still shaking out three years ago we talked cursor was number one two years ago we thought it was windsurf last year we heard it was cursor again or devon now we think it's cloud code it's a perpetual bottle like some of these things are still shaking out and maybe you're in one of those categories um so yeah you might still have a bit of time but uh but you do you might have time but you don't have this comfort or luxury so you need to move quick and move hard awesome this thanks a lot for being here where can people uh follow you especially since you're going to launch a new book what's the best platform yeah i mean you'll probably hear about through all things finn so like finn.ai and we're finn on all the domains and then i myself am des trainer and it's just d-e-s-t-r-a-y-n-o-r i'm just des trainer basically everywhere there's an account to be made i'm des trainer on it so x and linkedin is where i'm most active awesome thanks a lot this cool thanks a lot

55:47Rossi Rossi Rossi Rossi Rossi Rossi Rossi

55:54Rossi Z Hamdi