1027: Building an Always-On AI Agent for Busy Parents, with Dr. Dilani Kahawala

28:44This is like an always on Claude Max. So the price to cost is enormous. So you're right that we are constantly like whether you message Anna frequently or not, we are constantly just like spending tokens, figuring out what's going on behind the scenes for you. So the answer to which models we choose varies a lot. We started very simply, you know, like everybody else, building with anthropic models because they were just so much better. Like eight months ago, I would say there was no, like no one could get close. but over time that's evolved and we use a host of different models for different reasons so our voice model is different our we have fast models that very quickly like classify things we have fast models that like respond to you and carry on a conversation but then we have like our strong models like strong thinking models that are doing a lot of the figuring out you know have I already handled this do I need to put on your calendar but is there a conflict or do I need to go do research to figure out like which restaurants are nearby so we we use a mix of models um but recent and that eval suite is really critical because as new models come out and we and we want to switch it like having that eval suite makes it really easy to be like, okay, did we do something bad or did our performance degrade by shifting to a different model?

30:33But we do now, actually, we have shifted towards using open source models just because it has become quite expensive to serve our current user base. um so we are trying to be smart about where we like how we control the cost because for us what we don't want to do is like cap your usage and be like okay well you've run out of usage of anna that's like a horrible experience um you know and cold quote you can like turn on extra usage or whatever but for a consumer you don't want to do that um so we want to give you the most premium experience of Anna possible for the$20 a month. And that means behind the scenes, we're just trying to figure out how to get that, like the most powerful models for you without basically burning through all of our runway.

31:32Jon Krohn:For all you listeners who want to level up your AI career through hands-on learning, ODSC AI West, October 27th to 29th in San Francisco is the place to be. ODSC AI West is my favorite conference and what sets it apart is it's all about doing. You'll gain practical skills by working directly with the latest AI tools and frameworks in immersive hands-on workshops and tutorials led by experts who are actually building and shipping AI. I myself will even be doing a keynote at ODSC AI West this year on how individuals and organizations can thrive in the agentic era. The full program covers where AI is moving now, including AI engineering, AI-powered software development, physical AI, robotics and data science.

32:11Jon Krohn:Beyond the training, ODSC AI West brings the AI community together with networking events, meetups, the AI Expo, and more, giving you the chance to learn, practice, and connect all in one place. Super Data Science listeners can use the code super at checkout on odsc.ai for an additional 15 % off your pass. See you there. ODSC AI West, October 27th to 29th in San Francisco.

33:04Jon Krohn:on a topic that I think is really important for us to get into. And I do research and write a script. And then, and two of those episodes in the past month have been about open source models that are so, yeah, so Quinn and Kimmy models that are just so useful and so close to the frontier that the American labs are paving. Yeah, they're not exactly as good, I would say, yet. But they're so close. And if you have a good eval suite, I think they're just too good. And we've tried a whole host like DeepSeek, Minimax, Kimmy. And they're like every week something's changing. So just keeping up and testing that has been actually like our new challenge to being like, oh, should we try this model for this?

33:56It's changed dramatically in the last six weeks, I would say. and so it's a it's an interesting world to to play in right now for sure how do you ensure

34:10Jon Krohn:that your whole system is scalable like how do you I mean maybe there's a bit of a chicken and egg with you know you finding alpha users and then beta users and making sure it kind of each of those stages that your infrastructure can support that growing user base I guess there's two answers to it. One is managing that cost side that we talked about. As you grow, how do you make sure we can sustainably support people? And I think we're beginning to figure that out quite well. The infrastructure side, I mean, there's the standard sort of like, how do you make sure that you can have a chat service that can serve thousands of customers versus tens of customers?

34:55So that kind of thing, I think, is a more like a standard software engineering problem that's been solved before um but you kind of you know how to work through it i think the bigger um and then you have like interesting infra issues like can you there there are like rates of like how many messages you can send by sms through like twilio for example and so those are like interesting problems that we like okay I guess we better figure that out. We have to like get approval from a team level. But I think the biggest problem we've had to solve when we're scaling is because you're an agent and you can, in theory, do an infinite number of things compared to like a to-do app, say, five years ago where, yeah, maybe you have like 20 features, 25 features.

35:48You can ask Anna to do anything. and you and it might get some of that wrong and the user might get frustrated i think the biggest thing for scaling for us is how do you maintain that quality and accuracy bar as people push the boundaries there's like hundreds and thousands of people push the boundaries of what anna can do like we've uh we have a way for you to log into pretty much any like tool out there and so at some point we were seeing people like log into their notion and like what what are you doing with with notion but we hadn't necessarily optimized for that so um we've had to build this loop which is constantly picking up people's frustrations and then fixing those frustrations which i think like a cold code for example doesn't necessarily have to do because we have a much higher bar for like you can't get it wrong and that's i think that's where we've like spent all of our like code code accounts trying to like get under control this like constant frustration fixing loop which is very powerful but also like how do you scale that as you as your customer base

37:06Jon Krohn:scales it's been interesting that sounds like a part of the ip moat that you were developing for For sure. I think so. Yeah. How do you think about adding in new features? I mean, you just said that, you know, Anna could kind of do whatever, but I mean, not, you know, capability. Yeah, there's a lot of capabilities, obviously, when you have an LLM in the back end, kind of interpreting and assigning tasks. There's, you know, infinite flexibility in what could happen. But when I say features, I mean, you know, being able to support iMessage or, you know, deciding some kind of new product decision.

37:40Jon Krohn:How do you decide where to go? What feature to prioritize next? I think there's two things. One is like old school product management and the other is latent demand. So we try to make a bet on where do we think, like what's the vision? What's the dream for this? And so we made an early bet that people were going to be talking to Anna. and and and that's quite a different there were no products and people pretty uncomfortable actually talking to you know like whisper flow has kind of set the stage for this but we made the bet that actually like in the long term we think people are going to be in the conversation live because that is so much more efficient uh and so we we figure out what our vision is and we're like okay this is the bet we're making so we're going to make voice mode a priority even if like in the early days we probably had like a couple of people try it out.

38:41So that's one side of things. But I would say the primary way, once we've figured out like what's the vision here, which is like, and I should feel like talking to a human and I should be proactive behind the scenes and not, you're not having to have a term-based conversation for everything. The second thing I think, and one of the things I've kind of learned at Meta really is this idea of latent demand. We see what people are trying to do. And when we see a lot of people trying to do that one thing, we're like, okay, we need to go make that feature much better. So for example, people are asking to collaborate with their partners.

39:23And so we didn't have that early on. And when enough people asked for that, we added that in. or we see people trying to connect Outlook. Then we'll ask Anna, like, can you please, like, I want to connect my Outlook. And we were initially like, who uses Outlook? But a lot of people do.

39:45Jon Krohn:Ew. A lot of people do. And then you're like, oh, well, okay, well, that flow for adding Outlook was not very smooth. Now we go and make that smoother. So I think we pay very close attention to where is there already demand for that. And we kind of go after that. Kind of zooming out further beyond, you know, we were just speaking about individual features. Let's now talk about a product idea. So you alluded to earlier in the episode that the company that's now building, Anna, was doing something else. And so, yeah, I mean, maybe give us a bit more context on what was happening. Like how long were you and a team of people developing product ideas and how many kinds of pivots or, you know, yeah.

40:39Jon Krohn:What was the journey like to finally get to a point where you're like, cool. Now we're confident we have something that works. It sounds like you kind of, I guess you told us a little bit of this story, kind of at the end of the story where you, it seemed like there was a lot of demand. You were like, okay, let's invest a month in trying to build something relatively robust and see how that goes. But what was the journey up to that point? When we started, we were called Braid, and that's still like the parent company name. And we're building project management agents for software teams. And that's kind of what I was doing at Atlassian.

41:13We're doing a lot of project management. for, you know, with Jira. And when kind of the AI wave came about, I was like, there is a very different way to do this than, you know, with like a Kanban board. So we started out doing that, like an enterprise software project management agent product. what happened was that like when we talked to customers it seemed like there was a lot of demand for it and I think it's an obvious problem but it's a combination of I think the timing of when we brought this to market and where the technology was we found out honestly that probably didn't have the kind of product market fit that would that we were looking for and so So a few months in, after we had raised funding and we had hired a very small team, we decided, it was a very difficult decision, but we were like, we're going to be decisive and we're going to pivot away rather than sort of like continue to kind of butt our head against this problem.

42:26And we spent a good like three months exploring this like idea base of like, where are we, what is our next thing? And we were very deliberate about experimenting quickly and trying to see when there is like a spark of demand. and we were on this journey with primarily like enterprise ideas of like we would see where there was a problem and we would try to build a prototype quickly, we would try to get to customers quickly and see how the response was. That's when we came, we were doing that when we kind of stumbled on Anna basically.

43:08Jon Krohn:Yeah, stumbled on a direct-to-consumer product instead. And you mentioned enterprise, of course, you had experience in that, But I also happen to know that, you know, building enterprise products, you can get bigger valuations. You have stickier contracts, typically big, juicy ones with nice logos. There's all kinds of reasons to be focused on enterprise. But there's something really cool about when you go to the consumer route and something clicks and you can kind of because then, you know, you're in a you're in a cool situation socially, I feel as well. Like not only does that work and you have network effects, like you were saying, how you can exploit the data that you're collecting from your users.

43:50Jon Krohn:So that allows you to develop a moat relative to other people who could be like, oh, yeah, I'm also going to create an AI for parenting. Another cool thing about a direct-to-consumer product is in a social situation, the old cocktail party, it's so easy to explain, which is nice too. It's really fun. It's extremely fun to work on a consumer product, especially because it's incredibly satisfying to understand what that person is struggling with and helping solve their problems. I think in an enterprise context can be quite hard because you're sometimes quite removed from what these people are doing day to day.

44:29um it it has um consumer in ai has been very underserved i think um because all of the effort and all the companies that you see are today are the big ones are really all like enterprise or prosumer in some way um so it's an exciting time to be building in consumer ai it is challenging because i think we're still trying to figure out like 10 years ago when you're building consumer you just try to acquire like you have to grow quickly try to acquire as many users as possible but that's quite costly to serve them um within in the ai world so i think we're still figuring out like what how does the startup scale in consumer space um and so that's like the next challenge yeah scaling a sas business was way easier

45:24Jon Krohn:to have an extensive freemium tier, for example. Or how Google, Facebook, these products that are still free today because it was so inexpensive for them to... Some of those users aren't providing them any value. Someone comes in on an incognito window, it's hard to serve ads to them that are well-targeted, but it doesn't really matter because it's so cheap and there's so many people out there that are able to get hit with targeted ads or whatever. Yeah, it is a trickier thing that I think all AI businesses face today, where it is so much more expensive to serve your customers than it is for a SaaS business.

46:05I mean, it helps if you have like a mega wallet behind you to back that, like, you know, if you're like a Meta or Google and you can kind of do this. but especially as a startup you really have to start thinking about your unit economics much earlier than than I think previously and I still the funny thing is you always think like well the token cost is going to come down but you're always like you always want to be building in the frontier because you always want like you're still just barely getting by with what is possible with the best models today so you're like I always want the best model and the best model and the best model So it's going to be so interesting to see the open source models have helped.

46:53It's going to be really interesting to see how these like, because I don't think a big freemium tier can play out without at least some limits. And so we've seen other similar companies have all sorts of interesting pricing models, but they might like cap their free tier. or there was one that you had to negotiate the price with the agent, which was like fascinating, like poke. So no one has quite landed on this yet. And I think we'll be doing a lot of experimentation.

47:26Jon Krohn:Regular listeners will already be aware that I'm obsessed with Anthropik's Fable 5 model and it has taken over my working life. I'm writing a technical book that includes LaTeX files, mathematical notation, Python code examples, and Fable 5 and Cloud Code handles requests I make across whole chapters with accompanying Jupyter Notebooks end-to-end, work that a few short months ago would have been dozens of separate requests with way more manual fiddling required. With Fable 5, it just works, essentially like magic, first time. Cloud is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you.

48:03Jon Krohn:Whether you're debugging code at midnight, building a financial model or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai slash superdata. That's claude.ai slash superdata. And check out Claude Pro, which includes access to all of the features mentioned in today's episode, claude.ai slash superdata. I love it. You have an exciting trajectory. Where do you hope to be with Anna in a few years time? It's moving so fast. My dream is that if you're a family that in like a year or two years, you actually just have a team of agents who are just handling your admin and you just have like 10 more hours in your week that you're not doing like horrible admin work.

48:58That's kind of the dream. Like you just have more free time. And I think under the hood, that means there's like a team of agents who are constantly just figuring out what needs to be done to make your household run but potentially even like talking to the agents that belong to other families and coordinating for you and figuring out oh like who's doing the soccer pickup this weekend and kind of like figuring all that out like like agent to agent interaction in in family to family is like where I see this going and it's all gonna be voice and SMS or voice and WhatsApp and that kind of human interface rather than someone tapping buttons.

49:44Jon Krohn:I think so, definitely. All right, I have alluded to the fact that you have a very interesting background and we've kind of done a cursory glance over it, but I'd like to double click on a few things from your past just to give people, I think we've covered Anna pretty comprehensively now, but it'd be interesting to learn a bit about you, Delani, and your journey to now being the CEO and co-founder of Anna. So you did a PhD at a little known university called Harvard in physics, and then you jumped McKinsey. And I think that one I can understand because I also thought about Bain, BCG, McKinsey when I was getting near the end of my PhD, because it gives you the opportunity to showcase to the world that you've now developed, you know, not only do you have this great technical background, but you can be commercially savvy as well.

50:39Jon Krohn:And you get amazing experience at one of those big consulting firms. So that one, I feel like I can understand the transition, but maybe you can add a bit of color or tell me what I got wrong. I had always thought I was going to be a physicist, like an academic. And, and I was doing like very theoretical, I was like particle physics. Um, so like really large Hadron Collider thinking about like dark matter, bunch of that stuff. Um, very, very fun. I get thinking about dark matter all the time. Some dark stuff. Yeah. Especially with like AI these days. I'm like, oh man. But I was in Cambridge and you know, you're surrounded by like MIT and Harvard and there's this like, it's not quite Silicon Valley, but like I got there.

51:26I was like, there's all this like people trying to start what are startups? What is like, what is this? And I, um, I came from a show, which at the time didn't have like a huge tech scene. Um, and I was like, oh my God, this stuff moves like so fast. And like people do really exciting things in a very short period of time, whereas like takes like a year and a half to write a paper and like get it published best case um and I towards the end of my PhD was like I think that's what I want to do like it's just so much more exciting um you can see all these like startup competitions and the crazy stuff that comes out um so I actually knew that I wanted to do tech and potentially even like be a founder but I couldn't easily make that jump from like a physics PhD, like I, like my options at the time were like be an academic or go join like a quant hedge fund.

52:24Those were like the two things that people kind of do. And I was like, well, I want to be, do, do like be in a tech company and build. I didn't even know how to articulate that. And so I was like, Hey, I think the way to do that is to get a little bit more commercially savvy, like you said, which was like a McKinsey jump. And as I was doing that, I kind of figured out, okay, product management is this interesting space where you still get to work with technical people and you can leverage a little bit of your technical background, but you're still thinking about the business. And it was very exciting all the time and not even very well defined.

53:00And I happened to have, like my manager at Etsy kind of took a chance on me, even though I had no product management experience. and I was like, this is it. And Etsy's like the most fun company in the world. So it was in New York at the time and that's kind of how that transition happened.

53:19Jon Krohn:In Brooklyn? Yes. It was like the most fun I've ever had, honestly. Wow, that's cool. I didn't know it was that fun. It was like, you know, it was like pre-IPO when I joined and it's such an interesting, unique work culture. And then Facebook, Atlassian, I mean, I think those are kind of understandable transitions. I don't know if you have anything to add onto that, and you definitely can, but my question that I already kind of said I was going to ask is, then what prompted you to go from being this very senior role at Atlassian, head of product management for the shared experiences platform, you know, leading a team, leading an organization of 150 people across North America and the Asia Pacific region?

54:04Jon Krohn:And like, sounds like an amazing opportunity, but you left almost two years ago now to co-found Anna. Well, I guess it was called Parade at the time. Yeah. I've been, like I said, when I left my PhD, I was like, I want to found something. And I had like actually dabbled with a few things while I was doing my PhD. And there were a bunch of barriers. Like I was on an F1, like a student visa. I didn't have any money to do this um so that was not the right time and then um kind of like when I joined the tech companies things were growing quickly so it was always like hectic um so but I in the back of my head I was like I always wanted to found something um and then like Like 2020 to 2023 came around and I could see pretty clearly that AI was going to change things.

55:03Like, yeah, it was like GPT-3 at the time and people were like writing things with it. But you could see that this was going to be very different from every other sort of like tech revolution we've had. And I was like, well, I think this is the time. like if you don't do it now and like I at the time was had just had my third kid so I was like not sleeping very much and like struck like struggling to manage like three kids and work and everything um but I was like this is the time if you don't catch this way like this is the way we don't want to miss and I think that's what kind of prompted the okay it's time to make the call and it's too exciting.

55:51Cool.

55:52Jon Krohn:Yeah, I agree. I say a lot in my, when I do those Friday episodes where it's me just deep diving into a topic, my final sentences in that episode will often be about, this is an unprecedented time for you to be building things, whether it's in an organization or as an entrepreneur, as a hobby, there's never been a time like it. And I I don't know if there ever will be again. We'll see. Hopefully it unleashes even more waves of creativity and possibility. But fantastic. Thank you so much for taking this time out of your schedule. I mean, I guess you're lucky and it's taken so much off your plate that now you could do a podcast episode with me.

56:29Exactly. It's like I have like so many hours of like free time now that I don't have to troll through my email.

56:36Jon Krohn:Yeah, exactly. I'm sure. So before I let you go, something that I was supposed to tell you before we started recording is that I ask all of my guests for a book recommendation. I don't suppose you have one for us. It doesn't need to be a technical book. It can be like a favorite novel or whatever. Sapiens by... Oh, yeah, sure. Yuval Noah Harari. Yeah, that's one of my favorite books of all time, for sure. I feel like it's drawn so much controversy, but it was such a fascinating read. And I think about it because I actually recently saw a documentary about like human evolution. It was like this BBC series.

57:14And then it took me back. And this book was, I don't read a lot of nonfiction, but this was, this was one that stands out for me.

57:24Jon Krohn:It's a, it's a pretty easy read for nonfiction. Like you, you will know Harari has a pretty unusual gift for making it like for making the narrative compelling, even though he gets it to some pretty thorny stuff. It's a page turner, right? Yeah, it's a page turner. I didn't know he was controversial even. What I've heard is like it oversimplifies maybe like evolution in some sense. But like that's kind of what I need because I'm not like an expert. Like I need the story, the juicy story. For sure. I mean, that's like, it's such an easy way to critique something, you know, it's like, oh, well, obviously if you make something that's one of the most popular books in the world, it's probably going to have to skim over some of the detail to make that work for everyone.

58:11Jon Krohn:It's a funny trade-off. Yeah. It gives a lesser selling authors something to feel good about.

58:21Jon Krohn:nice and my final question that I always ask my guests is how we should follow you after the show your your business whatever you want you can give us tell our listeners how people should be following you for your brilliant thoughts or Anna the business for its brilliant advances of course we already know that we can go to highanna.ai to sign up for a free two-week trial of Anna so So get to it, parents. Get way more podcast listening time in your life. I'm always on LinkedIn and I post on X. And I post about both like how the product is going, but also how we're building. So that's where you can find me.

59:06Or you can just shoot me an email, delani at hianna.ai, anytime.

59:10Jon Krohn:Nice. Thank you so much, Delani, for taking the time to share your brilliance with the audience. and something else that is crazy, you might not even be aware how unusual this is. We did this episode without any breaks or retakes. And usually there's at least a few. Sometimes there's a lot. And this was just one continuous flow of conversation, which our editors have got to love. That's got to be the dream for a media editor. It's just like, that's great. I'm, well, there you go. Yeah, really easy chatting with you. Maybe we can have you on again in a few years when Anna is a household name. I'd love to.

59:53Thank you for having me. It's been a really interesting conversation.

59:57Jon Krohn:What a great episode. In it, Delani Kahawala detailed how Anna watches everything flowing into a parent's life, email, school apps, WhatsApp, calendars, and proactively services what matters. She described that the hardest three problems in building Anna are creating a magical experience with no app UI to lean on, teaching everyday consumers to just ask, to have this just ask mental model of working with an agent, and hitting a reliability bar far above what coding agents get away with. She talked about how Anna differs from turn-based tools like Claude Code by being a long-running agent that works every minute behind the scenes, why the eval and improvement loop is the heart of her product, and the brutal economics of consumer AI where a flat$20 a month subscription has to cover always-on token spend, pushing the team toward a mix of fast models, strong thinking models, and increasingly open-source models like DeepSeek, Kimi, and Minimax that have closed most of the gap with the Frontier.

1:00:58Jon Krohn:As always, you can get all the show notes, including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for Delaney's social media profiles, as well as my own social media profiles at superdatascience.com slash 1027 for, of course, episode number 1027. thanks to everyone on the super data science podcast team our podcast manager sonja brevich media editor mario pombo partnerships manager natalie zzajski researcher serge masseas and our founder kirill aromenko thanks to all of them for producing another super episode for us today for enabling that super team to create this free podcast for you we are deeply grateful to our sponsors you can support the show by checking out our sponsors links which are in the show notes and And if you'd ever like to sponsor an episode yourself, you can find out how to do that by navigating to johnkrone.com slash podcast.

1:01:53Jon Krohn:Otherwise, please help us out by sharing this episode with other folks that are struggling with parenting or trying to build a AI product in this day and age. Review the show on your favorite podcasting app or on YouTube. if you write a review on Apple Podcasts about the show. I will read that on air in a future episode. Subscribe to the show if you're not already a subscriber. But most importantly, just keep on tuning in. I'm so grateful to have you listening and I hope I can continue to make episodes you love for years and years to come. Until next time, keep on rocking it out there and I'm looking forward to enjoying another round of the Super Data Science Podcast with you very soon.

1:02:40Bye.