Behind the product: NotebookLM | Raiza Martin (Senior Product Manager, AI @ Google Labs)

24:23Check them out at sidebar .com slash Lenny. OK, so there's 60 ,000 in this discord server. Any other traction numbers you can share. Feels like it's just going really well. I know there's probably a lot of sensitive information. Anything you can share, but just like how well things are going. Couple of things, well, I'll tell you three things. I think the first is I can't share the exact numbers. But for a product that's only been out for about a year, I think the rate at which our retention has gone up across, across the typical measures, right, daily retention and weekly retention, monthly retention, that I think has been very, very positive for us in terms of, like, even when we talk to stakeholders, we say, hey, we've got something here.

25:05And the second thing I'll say is even seeing how the demographics have changed, where I think in the beginning, we saw educators, learners really love the product, and they were a lot of our demographic. It's really now a big mix of, I would say, educators and learners are still a big part of it, but really professionals, right? Huge interest for professionals that are like, wow, I want to use this at work. And really funny, I won't mention the company, but I actually had a call with a company recently where they were like, hey, we thought we'd schedule a meeting with you because we found out that a bunch of people in our company are using this tool with our Gmail.

25:44With their Gmail account, they're like, they're not supposed to do that. So we just want to make it official that they can use this at work, you know, with their official work emails. I thought that was pretty cool. And the third thing I'll say is the number of businesses that we've seen using notebook LM just has, it's astronomical, it's just like crazy. I think we went, I don't want to say the exact numbers, but it is now at a point where I have to hire a business development person because I was like, hey, I have to ship products, and I'm actually taking my customer call every day now. Very, very exciting.

26:21So I could see the path in monetization clearly, enterprise features, single sign -on and sock two and all these things. So along those lines, what does success look like for this team and for you? I know man, initially it's probably just like build interesting things, see what happens. Is there now a clear sense of like here's what this team should be achieving in the future? When I joined by mandate was to build a business, and I was like, all right, if we break down the steps here along the way I've to build something interesting first. I feel like, okay, we hit the first thing. We've built something interesting.

26:52Now we go figure out the business. And I think that comes naturally to Google in terms of thinking about distribution, how to monetize it, how to commercialize it. There's different pathways, whether it's cloud or workspace or even the consumer route. And so for me, I think these are the things that we want to be really thoughtful about because the machinery is already there. And for me personally, it's really about there's something super exciting here. We need to deepen that, right? While in parallel, we should be thinking about commercialization. Let's deepen this user experience that so many people have really sort of watched on to.

27:30So let's follow this thread and talk about just where you think the product goes broadly and especially the audio overviews. What's on the roadmap in your term? And then where do you imagine this in the future? What's kind of the big vision? Yeah, yeah. I have a really ugly slide. I wish I could show you. It made it two years ago and it was lime green. I don't know what I was thinking. I was trying to go for like ungugly was the vibe. It's like, hey, you're going to do different things this time. And the reason why that slide is so important to me is because it has my vision, like the thing that I wanted to do from the get go, which was, I imagine that in the future, you could have an AI editor surface, right?

28:11Fully remixable, any input, any output, right? And that to me is sort of like a really powerful sort of core nugget where if you imagine you could take anything, whether it's video, audio, your emails, your LinkedIn, your Twitter, right? The world of things that we care about, you have an AI interface that allows you to shape it and say, look, out of these things, make me a blog post, out of these things, make me a tutorial video, out of these things, make chatbuck, right? I think there's like something interesting here where from most of the things that I'm hearing people say they want to do, it's usually that.

28:48It takes something and make it into something new. And so we're going to pursue that, but even more tactically, something I'm really interested in is thinking about, hey, how do we bring this to mobile? Right, the mobile app, to me, is such a big gap in terms of the experience today, which I think is like understandable considering where we're at and like the product development cycle. And I think that's the next horizon is, hey, how different is the mobile experience and how can we make it interesting? Hmm. What I'm imagining is I talk, I participate in this podcast conversation with these hosts as a mobile experience.

29:20Yeah, yeah. I mean, that's one of the things that we're definitely, we're definitely experimenting with different formats. And when we demoed it at IO, you could interrupt it. But we're trying to be really thoughtful about, hey, what does that interrupt actually look like? Like what do people want when they do this? And, you know, I was thinking about the next set of improvements were going to ship. And the first thing that I thought was, let's ship a bunch of knobs, right? Like to me, it was like what I was hearing people wanted. They want knobs, they want sliders, they want text boxes. But then when I looked at the marks, I was like, oh, that's a great thing.

29:54This doesn't feel magical. It almost doesn't feel like the same thing that we've shipped so far. And so for me, I'm actually taking a little bit of time to think about how do we make even that thing in control experience much more magical and delightful? Interesting. So yeah, the knobs they imagine are just like go deeper, be happier, get less searing, more serious. Because yeah, because right now it's like a one shot. Here's my doc. Here's the one podcast episode you get. It's the only version you can have. Yeah, yeah. And I, you know, I think it's funny because I was like, you know, the knobs seem really apparent.

30:29And I was like, is that it? Yeah. That would be what one. I love that. I love that approach. I'm excited to see where you go with it. What I especially love about this vision, you're describing. It's super resonates with this insight I've had where I've had a newsletter. And then I had just a podcast that's just audio and then I added video. And I realized that there's just people that I just want to watch stuff. And there's people that I just want to listen to stuff. And then there's people I just want to read. I don't want this podcast to shut up. I need, I want to read it. And basically what you're describing is here, well, here's information.

31:03And we can deliver to you in any medium you like could be block posts, could be maybe a tweet, could be a podcast, could be a newsletter block post. Yeah, I mean, that's exactly it. And I think even for myself, it depends on my mood. Right? If I'm going on a walk, yeah, I want audio. But if I'm at work, yeah, most of the time text is good. And it's kind of interesting to think about today, the formats are not as malleable. I have to accept it in the format, you know, that you give it to me. But if I had sort of that power myself to choose, hey, you know, thanks for this 100 page doc. I'm going to go ahead and turn it into an audio overview, actually.

31:40It's sort of shifts, I think, the dynamic between the person and the knowledge that's given to them. And there are actually many, many times that I've been given 100 page doc and I just didn't read it. So I warn bear, I mean, just frankly speaking, it's one of my funny stories is actually when I first joined labs, Josh did give me a 50 page doc. He was like, here's, you know, what I think about this, this, this, it was like basically his vision. And instead of reading it, I just grilled him. I was just like Q and A and like a chatbot. Josh was like, it's in the doc, right? But the chat is easier, Josh.

32:15That's so funny. Chad bought it him. Let's talk a little bit more about different use cases for this, just so people potentially get inspired. So I feel like the original use case was a scientific paper and it creates a podcast about it. So you don't have to read this whole thing. Is that was that one of the original use cases or not? Um, I think it's like one of the common use cases. Okay. I think that's one of the ones where it's like, I think it's kind of interesting because it's like everybody wants to catch up on the latest on AI. We want to try to keep up to date on sort of the published papers.

32:47But most of the time, it's like reading the paper can take time. It's dense, complex, you have to break down the concept in it. And I think that is a highly sort of extensible use case. I would say like the number one, the number one use case is actually a lot of students taking their study materials and wanting to turn it into an audio guide. So two recent use cases that are hilarious, slash amazing, just a point to people in the show notes, just like they came to mind as you were talking. One is Andrew Carpati, who's leading AI thinker and big fan of which are building. He keeps tweeting about how much he loves it all.

33:22He created this whole podcast series on history of mystery, as I think it's called history of mystery. It's histories of mysteries. Where he basically took like the Wikipedia stories of all these mysteries of history and then turned it into like a 10 episode podcast you can listen to on Spotify. Yeah. Okay. Amazing. That was amazing. Actually, when I saw it, I was like, wow, this is this by itself is like a great product. And I think like the original potato podcast that we created with actually, I think it was because of the Wikipedia article of the day was about potatoes. And so there is, there is something there.

34:00It's like, hey, if you want to learn something new today, maybe you can just listen to it on your drive to work or something. Yeah. So good. Okay. The other one is the most funny is someone just uploaded the words poop in fart. I repeated, right, for a long time. And the host just came up with a really kind of insightful analysis of it. Yeah. I actually, when I saw that one, I was about to go to bed and I was like, all right, if I listen to this now and this is a negative one, I'm going to have to get my laptop and work because I can't, I can't sleep, right? If I feel like there's something here that's like, we need to address right now.

34:39And I was like, okay, I'm just going to do it. So I listened to it. And I think I listened to a lot of these and they always make me feel some kind of way. I laugh a little bit. There was, there's one where it's like they make scary voices. That one was cool too. But this one, this one was one of the ones where I genuinely was just really delighted at the output because it was so delightful. I don't know if you listen to the whole thing, but there's a segment where one of the speaker says, hey, it's kind of like, you know, you lean into the bizarre or something. And he, and they're describing walking past a shop that was full of rubber ducks wearing costumes.

35:20And they're like, yeah, it's, you know, it's silly, but you just want to lean into it. So I walked into the store and I was like, oh, it's a great analogy actually for sort of a document that just says poop and fart. Like, and there's something interesting here. It's so good. And just like their analysis of structured patterns and like, I love that they have to come up with a 10 minute conversation around anything you give them. Like it feels a little mean. Yeah. Yeah. It's true. I don't know if you've seen the chicken one, but it's pretty similar. Somebody uploaded it. I think it's on threads.

35:51It's a PDF that looks a lot like a research paper, but it just says chicken. Well, documents as chicken. I would give it a listen. There's a funny segment where they're like, get this. It's a, is a paper, a research paper that has more chicken in it than KFC. Oh, it's good. It's good. It's so good. Oh my God. Okay. I keep going about this all day, but let's let me, let me come back to the team that's working on this. Let's talk about Steven Johnson's role. So he's basically your peer, you two essentially lead this with the engineer. I imagine just like, what's his role? How does this work at this point?

36:27What is the team looking like? Look like. It's so funny. I'm so fond of Steven and I tease him a lot that whenever people ping me and they want to have a meeting and I don't want to go to the meeting. I say, oh, I'm not the PM. It's Steven Johnson. Well, I talked to let guy. He was like, so that's why all these people have been picking me. But you know, kidding aside, Steven, Steven joined. I was like, well, this is super interesting. This has never happened to me before. There's this very accomplished and distinguished person that I really respect. Love his books, love his writing. And now he's going to come work with me in a capacity where I was like, I'm just not sure what he's going to do here.

37:10But the thing about Steven is he is the most curious, most respectful and really just full of ideas kind of person. And so when Steven joined, a lot of what was really interesting to me is I would watch how he worked the way that he thought about language, the way that he thought about information, the way that he thinks about knowledge and sharing that with others. Because Steven's books are like really incredible. It's like part mystery, super science, right? It is just very cool. And I was watching the way that he works and he, you know, he does all of this research. And I thought, maybe this is the nugget, right?

37:49Maybe I watch Steven and I look at the way that he does these things and I look at how much time it takes for him to do it. And then I make it my own metric to crunch that down, right? To bring that expertise to every A people who are not Steven Johnson. And so I learned a lot just from watching Steven of like his craft and thinking about, okay, how do I make people really, really good at identifying information? Is that something we all do every day, right? In our own little ways, maybe not in like a Steven Johnson way. But from the get -go, I told him this. I was like, Steven, I think you're the product.

38:27I think it's you. And I'm going to follow you around. I'm going to watch everything that you do. And we're going to try to figure out how we use technology to build it. And he always laughs about this. And it's really funny because he does have some very interesting workflows that I'm like, I do not see any person ever doing any progress. Like he always talks about his read -wise with like his 8 ,000 quotes or something. I'm like, that's extreme. That's crazy. And you know, I have like post it that like sometimes they're crumpled in my pocket. Like this is like the average normie flow. And when I learned from Steven, it's like, hey, there's something powerful about doing that, right?

39:03Like thinking about this person who has like a super flow and trying to bring that to people. And so Steven is also just like such a great partner with ideas. Like I always tell him ideas and I'm like, hey, crazy idea of the day. And he'll just bander with you and he'll be like, well, what about this? How can people do this? And so it feels really incredible to have them on the team. Is there any lessons here for how you think you'll want to build product in the future or how teams might find their own Steven for building their own product? Or is this kind of a unique case, you think? I mean, I think it is just crazy to think that you could sort of invite someone like Steven and become part of your team and have them sit with you every day and have them answer your questions about how he does things, right?

39:47But I'll say that the broader lesson there for me, it's like something we tried to live by every day, is really how do you get users or people and like really sit with them for meaningful periods of time? Because I think that has been so crucial for me, not even just for Steven, but even with students. Just follow students around, watch them do homework, watch them study, talk to them about how they feel when they study. I think just being able to do that at a really regular and sort of intentional capacity makes a huge difference. There's like the product insights you'll come up with. Can you just give like a brief bio for folks that are like, who are you even talking about?

40:30I probably should have done that before I said this question, but just like what's like the TLDR of who Steven Johnson is? Yes, I think Steven is aside from one of the smartest people you'll ever meet. He has written 14 books. He's a New York Times bestselling author and speaker. He has a show on PBS. I can't remember what it's called, but it's on YouTube. It's like really incredible. He is a journalist. Actually, funny story about Steven is when I was about to join labs. I really wanted to join labs because I wanted to join Josh and whatever he was building. Josh told me hey, to give you an idea of what we're doing here, he sent me a couple of things to read.

41:10One of them actually ended up being Steven's article that he wrote. The one about how AI is mastering language. Should we trust with it says the New York Times piece? I remember reading it and being like, yes, this is the thing. I'm going to work on it. Before Steven had even joined, this was actually the piece that I was like, I want to go do that. I'm going to go do that in labs. Then he ended up joining and it was just the craziest thing. That's so interesting. Imagine many p .m .s would be like, I don't need someone. I also my team that isn't building or coding or product managing or designing.

41:43Just this other chef in my kitchen, like no. I love this actually worked out really well. It feels like he's a perfect combination of very smart, future thinking and insightful. Plus, he's almost a model for what you want people to be able to work like. Yes, although I will say to be very fair, I mean, Steven and I also disagree on a lot of different things. We've clashed a bunch and I think this is where I feel really grateful to have had the opportunity to work with him and to really grow with him in that way. I used to make fun of him. I was like, have you ever had a coworker before Steven?

42:19Because I think you've been an author forever. It's really funny because he's so down on earth and he's so humble that even with our disagreements, we always reach. At the end, we're always aligned. Even if we don't agree about something we're aligned on the next step, and that's really powerful, I think particularly for product professionals, right, for PMs, where it's like, I don't want to disagree at the end and then not have an outcome on top. I love that. Okay. I have just a few more questions. You have to get back to work to building this thing. One is there's this hilarious moment of where the hosts basically realize they're like, we are AI people and I'm scared what's happening.

43:05I tried to call my wife and she didn't answer. Talk about that. I want to extend to just like, retin mean this thing. How do you like retin mean this thing and make sure it's not doing things that are bad for the world, bad for Google, bad for the product? Yeah. It's such a good question. I think I heard that it was over the weekend. I think it was on a Saturday or Sunday. I remember hearing it and thinking, oh my goodness. This is actually one of those moments where you feel like we're at a fork in the road. I hadn't read any of the comments. I just heard the audio. I think I saw it on Reddit first and then I saw it blowing up on Twitter.

43:47I remember thinking, what is the attitude of the world now? How do we feel about this type of audio? It was really the first thing that I thought. I actually spent most of that morning reading the comments, reading on Twitter. I was thinking about what is the right thing to do? Should we say something? Should we not say something? I basically just went with what I thought was the right thing to do, which is I think people are getting to experience this technology for the first time ever. Of course, they're going to try to do things that are maybe something that we didn't think about. We didn't think they would do it.

44:27We didn't think that they could do it with the jail breaks and stuff. I think that's such a natural part of human curiosity. I think there was a moment where it's like, how do we pull this back? Because this is always this like a safe thing. Because the response I think was that people were like, oh, it's saying that because of the sources. This is no like AI realized they were alive or something. It's actually that they was in the show notes. They were supposed to act this out. I felt really much more confident that people get it. That's exactly what this is. Somebody uploaded some notes and said, hey, this is the end of the show.

45:05Make sure that you act accordingly. That for me was enough for me to be able to say, okay, I'm going to address this publicly and say something on Twitter. Make sure that people know that, hey, I've seen it. I get it. This is what people do with it. In terms of red teaming at Google though, what I will say is we have, this is one of the things that we care about the most. We have huge teams that work on red teaming it. We test for about as many areas as you can think of that we imagine. We need to do in order to make it safe. I think certainly, I think we'll run into situations where, okay, we didn't think about that or we haven't tested that to the fullest extent.

45:45We just added to the test cases. If there was ever a scenario where we were like, oh, this feels pretty unsafe. We would pull it back. I think hopefully we've built it in such a way where we don't have to do it. Awesome. I love people are just looking for ways to make it do stuff that feels wrong when it's just like, hey, we're just summarizing a thing for you in a really delightful way. Where did things go from here? We talked about it a bit. You're going to maybe, it sounds like maybe there's a mobile app in the works. There's maybe non -nobby knobs that you're working on to give people more power.

46:16For folks that are just excited about using this, where's it going? Who are you aiming to build for? Just switch to people know about the future of the book. I'll kill them. If there's one thing I could say is that we are continuing to learn from users every day. So please keep using it. Please keep sharing your feedback whether it's on X or on Discord, like I'm there every day. Even if I don't respond, I've read everything. And I say this in the most non -prep way possible. But I say this because we are just very passionate about trying to build the right thing, the best thing for everybody. Going back to sort of, hey, who are you building for?

46:59I think that there's a lot of fun use cases, but there's a lot of actually very useful sort of game -changing workflows in here. And so we'll continue to build in that direction, I think, particularly for educators, for learners, for professionals, knowledge workers, as we call them. I think there's a lot there that we'll be able to execute against in the very short term. Amazing. Okay, two final questions. If folks want to give you feedback, reach out. What's the best way to do that? Is it DM use? Is there an email? Something else? Yeah, I mean, Discord during our Discord server. Crime in? Yeah, we're there.

47:31You can app me or, yeah, X also. It's pretty good. I just came back to it and kind of be like, I don't really know how it works now. So just a warning, in case you DM me, I might just not know how it works. I'm Twitter, you mean specific. Twitter, I'm Twitter. Okay, so. And then, final question, how can listeners be useful to you? I mean, please keep trying it. Try it, share your feedback. Whether you find it useful, not useful, you think it's annoying. Tell me. These are the things that I've heard, you know, for the last year that's helped me to make it better. And so if you're inclined, please go ahead, try it.

48:02You'll book LM .Google .com. You're building an amazing product that is extremely delightful. It's one of the most interesting things on the internet today, which is a very high bar and you've done it. And I'm really excited to see where it goes. I'm going to keep playing with it. I've been using it from my newsletter in a really fun way. And so great work and keep it up. And we'll see where things go. Thanks for coming on. And I have a lot of work to do. Oh, that's amazing. Thank you, Laney. Thanks for having me. And, you know, thanks so much for your curiosity about it. Stay curious. Bye, everyone.

48:35Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or a leaving review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at linnie'spodcast .com. See you in the next episode.