A Big Week in Tech: NotebookLM, OpenAI’s Speech API, & Custom Audio

0:00There's elements of it that are almost similar to early chat GPT. Anyone who's now building a conversational voice product can have access to that level of conversational performance. The majority of people may experience AI for the first time is actually going to be via the phone call. We're taking the oldest and most information dense of all of our mediums of communication and finally making it a most programmable. Phone calls are kind of this API to the world. Within a couple weeks of deploying their voice model, they'd had three million users do 20 million calls. Last week was yet another big week in technology.

0:42For one, notebook LM, Google's latest sensation, has been making its way across the Twitterverse, with its new audio overview feature. The feature uses end -user customizable RADG, which basically means that people can create their own context window. for generating surprisingly good podcasts across 35 languages. And to add to the Voice Mix, OpenAI held their developer day and announced their real -time speech -to -speech API, enabling any developer to add real -time speech functionality to their own apps. Plus, they noted a whopping 3 million active developers on the platform. Finally, we saw one video model company, Pika, break through the AI noise with their 1 .5 model.

1:21giving us fodder to discuss what is really required to capture attention in 2024 and beyond. Today we discuss all that and more with A16Z consumer partners, Olivia Moore, Brian Kim, and General Partner, Anish Acharya. This was also recorded in two segments, one with Olivia and another with all three partners, so you'll hear as pivot between the two. Plus, Anish actually predicted that this would be the year of voice, despite it never historically working as an interface. In fact, Microsoft CEO Satya Nadella, even previously called the past decades generation of assistance, quote, dumb as a rock.

1:58Well, it certainly seems like returning a corner. Let's get started. As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the company's disgust in this podcast. For more details including a link to our investments, please see A16Z .com slash Disclosures.

2:34Another big rig in tech, I think the biggest thing I've seen is notebook LM. So just quick recap after the audience, Google is kind of known for the side quest becoming main quest. And this product actually has been around for a while. It originated in 2023. But its new audio overview feature has been taking over Twitter with these AI -generated podcast hosts, which are surprisingly good. And I'm saying that as a podcast host, as this job. And so basically, what people can do is they can drop in their own information in a context window. And then it'll use that to spin up these podcasts. Olivia, you've actually tried these out, right?

3:13Yeah, so I think it originated as something for researchers or academics. The idea was that you would store all of your notes, all of your papers, all of your information within this Google workspace. And then this new feature that they've added is these two AI agents essentially that play the role of podcast hosts. And they go back and forth talking about the data, asking questions, getting into examples. The thing that's really interesting to me about it going viral in the past week or so has been there's actually nothing that feels Incredibly new or even incredibly in some ways cutting edge about it like it's not the open AI Brand -new real -time model that cuts voice latency down to almost nothing in fact with notebook LM You have to wait three to five to sometimes ten minutes for them to generate the episode once you click the button I think what's really striking about it is the realism and the humanness of the voices and then also how they interact with each other.

4:12Yes. Is it forward? Is it forward? Is the intonation? Yes. They disagree with each other, they interrupt each other. Like this is not just upload a script and get a read out. It does feel like two human beings talking. And to that point, the other kind of striking thing about it is it's not just repeating or summarizing the points that you upload in whatever data sources. They're actually answering and asking you really interesting and deep questions. They're making comparisons, they're making analogies, they're taking it as step deeper of almost like how would you teach someone about this topic.

4:46I uploaded basically a bunch of true crime court case filings and it did a podcast about the case and then it spent the last two minutes diving into the ethics of why are we entertained by true crime should we be using this information to create media, things like that. So it's really kind of like a next level interpretation of the content, I would say. Totally. I've seen so many examples of this, someone uploaded just their credit card statement and they were able to grill them on that. And even that, I don't think the grilling was prompted, per se. It was like just talk about this, find something interesting within this.

5:20Yeah. There has to be some sort of very creative LLM or something behind the scenes. One of the other use cases I loved was someone uploaded their resume and their LinkedIn profile. And it made like an eight -minute podcast describing them as this incredible, legendary, mythic figure and going over all the high points of their careers. I really like that because I see some people using some of the music albums and they're using them for, let's say, a really nice birthday song. And so when you played with notebook LM, was it the kind of thing where sometimes you're on, let's say, Dolly or Midgernie and you're like, like, oh, it's not quite what I want.

5:53And you're just playing the AI slot machine. Was it like that or was it first shot? I'm getting exactly the kind of podcast I was hoping for. It's a little bit slot machine in that the output is different every time. But I would say it's a lot more reliable in that almost every generation that I would do. Something would be interesting, it would be on topic, it would be usable. One example I did, I got very into it. At first I was sticking to uploading academic papers, I was like, I'm gonna use this for its intended purpose. And then one of my generations, I was like, the hosts, they sound like they're flirting with each other.

6:27Right? They have such good chemistry. And so I was like, what would happen if I upload literally a one sentence document that's like, I think you guys are in a secret relationship? And they went off on like a two to three minute podcast that sounds I swear, like the meat, cutin' a romantic comedy or something. It's incredibly emotionally compelling, I would say. And so now my vision I have to do like a full audio drama than we have to It'll be like the first fully AI avatar movie using the voices inspired by the notebook LM characters This one's about AI but like AI in relationships really yes specifically AI that are like Hosting a show like us interesting and Google's notebook LM environment.

7:15Oh wow So like, could we be secretly dating? Exactly. That's wild. That's what the document asks. Someone thinks we're giving away like secret love notes to each other through our banter. Well, what was the end? Did they agree? I mean, you have to listen to it and get your take. What if those AIs, you know, actually developed feelings for each other? Like real feelings? Yeah, exactly. So it's like you're saying two lines of code could fall in love over a spreadsheet or something. That's the idea. Yeah. It's kind of wild, but also kind of, I don't know. I know, right? Intriguing. And so given that you have played around with it, and that a lot of the feedback is really good, and people are pleasantly surprised by this, what's your reaction?

7:59Because like you said, there are products like this out there. I mean, with AI, there are so many trends, as we've seen, like products that get really hot one week and then something more interesting comes along. Could be me just being optimistic. It feels like there's something here, and I hate to make this comparison, but there's elements of it that are almost similar to early chat GBT. And that one, it's really usable, even for people who are in academics, people who don't know that much about prompting anyone can upload a paper and kind of generate a podcast. The other thing that feels chat GBT -esque is like, people are already pulling it, quote -unquote, off label.

8:35And maybe it's not notebook LM itself that becomes the winning product. We'll see. I think there's a lot Google could do to extend this more. they could make it a mobile app, you could customize the voices, I could see it being used for kid bedtime stories if they tweaked it a little bit. But I think something about the format of personalized podcasts or personalized audio is going to happen. Some of the experiences or the podcast being generated are no doubt impressive, but also feel a little maybe gimmicky or like cool ones. But is this really something that you can see evolving into something practical, useful?

9:11I for one can see it actually becoming a real product because right now it's doing podcasts for example But over time it may be easier to add avatars or videos as backdrop of what they're talking about and that becomes basically short from YouTube video That is very personalized. So one of the fun examples was like kids love Minecraft I love Minecraft when there's like a new bad rock edition that drops and there's like a release notes that are pages and pages long and and kids rely on YouTube to figure out what's new, like what changed. If you drop the release note into notebook's LM and just say, tell me what's new.

9:45And tell it in a way that kids love. And then it generates this 20 minute or 10 minute back and force of a, can you believe this new update? It allows this character to fly. But those are the type of things that actually becomes really interesting in a everyday use case. It makes me wanna have like a digital diary or something where you can upload it and then it gives you like a podcast of like how the last month of your life is bad. Oh my God. Yeah, that's so hard. Because the innovation is less like a new medium and more how they've really unlocked something to your point around how to make any topic exciting and generate insights and make it something that you really want to listen to and spend time on.

10:22That's true. Potentially unlimited outputs. I totally agree. It could be videos. It could be avatars. The interesting thing about that is I'd always thought of it as you can read something, watch something or listen to something, but maybe a nuance of listening is listening to it in conversation format. I do think there's something really magical about the two hosts going back and forth. Yes. Yes. There was a TikTok I saw yesterday that had two million likes, completely organic, and it was a law school student who was studying for her midterm, and she had uploaded like, I don't know, 60 pages of lecture notes, and then a generated 12 -minute podcasts for her to review before the exam.

10:57If you even hear another human being telling a story around an example or a case, it makes it so much easier to remember and understand. You're basically opening up another lane, right? Because you can read something as you're listening to something, as you're immersed in something else in the real world. Maybe another thing to talk about is OpenAI's Dev Day. They released a lot, but maybe the highlight point was this real -time speech to speech API. And I know you've thought a lot about this idea that real -time really matters for speech, and that latency is almost like a metric that we're going to hear a lot more about.

11:29Yeah, there's a threshold above which voice doesn't really work as a modality to interact with the technology because it doesn't feel real and Below that threshold, which is maybe three or four hundred milliseconds sort of holds the illusion of talking to a person Phone calls are kind of this API to the world So it feels like the way that the majority of people may experience AI for the first time is actually going to be via the phone call And that is unlocked by this real -time technology And the crazy thing is like so much still runs on the phone system Absolutely. Even if you just think about one vertical, like health care, it's like taking incoming calls from patients.

12:04It's like doctors calling other doctors calling pharmacies, insurers. So if we think about how this becomes more real time, are there different applications that you think are unlocked, like a, let's say, music education? How does real time voice maybe change some of those industries? Most of the edtech products we've seen so far have been like, you attempt to home more problem maybe than you take a screenshot, you upload it to an AI product that tells you if it's right or not. And now with real time, both voice and some of the video and vision model stuff, it's actually almost like having a tutor sitting next to you, going through it with you, even with some of the vision stuff, show it your piece of paper.

12:38So now it's like AI is moving towards actually helping you learn versus a lot of the use cases so far have been maybe cheating adjacent in like, how do I just get to the answer? Now it's what is your process? That's actually really interesting. You're basically saying that in a way the lack of latency allows for people to integrate in that moment. Yeah. And in the past maybe because there was more latency people took shortcuts because they didn't want to wait. Or if it's with you, it can say, here's the way you're doing it. Here's another way actually that might make more intuitive sense for you to solve this math problem.

13:13It's going along the journey of understanding with you versus just being kind of answers or outcome base, which a lot of the AI -type products have been historically. What's really interesting about that is that there's a sort of design language or design cues that are already built into conversations. So interrupting is one or the sort of uh -huh is another. So that actually should unlock much more interesting product experiences as well. And of course the latency is necessary for that. But so is the ability to even understand these parts of sort of, I don't know, they're not quite nonverbal, but they're not also part of the explicitly spoken language.

13:48A lot of products, especially in consumer, it's not just about being optimal, per se, or perfect, right? In fact, what a lot of people are commenting on when you see the notebook LM examples, it is the filler words, it is the interrupting, it is the imperfections that people are drawn to. This is a big step forward, and for anyone who tried to use the chat GBT voice mode before, essentially, you would press a button, you would say something, the LLM would pause, it would interpret it, it would generate something to say back and then it would return an answer, but it take at least a couple seconds.

14:19It was very buggy. It was very glitchy. It was more like sending a voice memo, having someone hear it and send back a voice memo than having an actual live conversation with the human. And so the new model is truly more like almost zero latency, full live conversation. This has been available through Chad GBT's own advanced voice mode, which people are using in loving. But what happened this week at developer day was there essentially making that available via API for every other company. So anyone who's now building a conversational voice product can have access to that level of conversational performance, which is huge and really exciting because it brings a lot of AI conversation products from barely workable, not really workable, to suddenly extremely good and very human like.

15:07Yeah, totally. You had a tweet that said, this is a massive unlock for AI voice agents I'm expecting to see a lot more magical products in the next few months. We're quickly leaving the era of latency and conversational experience being a blocker. Can you speak just a little more to that in particular? Yeah, absolutely. Many of their AI voice products didn't really feel even SMB caliber in terms of quality let alone maybe like an enterprise could actually deploy this. Yeah. So now it is I think arguably enterprise quality in terms of real companies being able to replace humans on the phone with an AI on the phone.

15:42We're seeing this for all sorts of use cases. The most obvious is maybe having someone answer the phone at a pizza shop to take orders or at a small business to book nail appointments, all the way to things that are a lot more complicated, like even doing interviews, first round interviews with AI, which is crazy to think about, but it's happening. Or even more kind of vertical specific use cases, is like freight brokers spend all day on the phone, calling carriers, calling truckers, and trying to find someone to haul a load in a certain price range. Now you could do that with an AI that can call 100 carriers at once and negotiate the price instead of having a human being do those calls sequentially all day.

16:25This new API and there's other open source attempts at the same type of model is really gonna allow those products to shine. Yeah, and some of the products you're describing are kind of voice first. But many of the apps that we've had to date are typically not voice first. Perhaps because we actually haven't had the technology. And so I want to refer to Anisha's big idea at the end of 2023, which right now feels very accurate. Yes, he was right on. Yeah, it said that voice first apps will become integral to our lives. And he basically says that despite voice specifically being the oldest and most common form of human communication, it's never really worked as an interface for engaging with technology.

17:03It feels like voice is one of the biggest things that's being unlocked by AI. Voice is the easiest content to create and we're all creating audio all day every day essentially. But that content has never really been captured or used or automated in some ways. Like now, even outside of real time, there are so many products that will listen to your meeting and will hear you say something and can automatically slack someone with a follow -up or use it to trigger a commit in GitHub or a task on a sauna that your team has to follow up on. And so I think what we're seeing now, both real -time voice and non -real -time voice, is we're taking the oldest and most information dense of all of our mediums of communication and finally making it almost programmable and usable in a really powerful way.

17:49The one thing I think we didn't quite predict when we were forecasting voice for this year was that it's really, really been working for B2B and not as much on consumer quite yet. We're getting there. I think on B2B, even thinking about the voice agents, a lot of businesses are struggling to find people to answer the phones for all sorts of roles. They're struggling to retain them. It's expensive. And so it's supernatural to plug in an AI that can perform at similar quality. The consumer use cases are a little bit less obvious. It's probably worked the most in companion so far. So again, chat, LGBT advance voice mode or character AI.

18:27I think they announced within a couple of weeks of deploying their voice model. They'd had three million users do 20 million calls. Yes. Wow. Because if you're spending hours each day anyway talking to this companion, giving it a voice and making it more real makes a lot of sense. So that to me was like the shining star of voice so far. OpenAI did highlight two other use cases on developer day in consumer. And both of them were actually these kind of high touch expensive human services almost that are now democratized with AI. So one of them is a company called Speak that does language learning.

19:07This might be controversial. I love DuoLingo as a product. I love it as a brand, but I think it's hard to use it to learn a language end -to -end because it's just limited as an interface. So if you really want to learn a language, you might have to pay someone, I don't know, $50 to $100 an hour to tutor you. And so the idea of speak is you have an AI voice agent that is essentially your language tutor, and it's much more accessible and affordable. So that was one. And then the second one they highlighted was, what if you had a nutritionist via AI? So this is a product called Healthify, where you can send in photos and then talk live about what you're eating every day and your diet.

19:42So I think we'll see more of those use cases unlocked with better voice model. Yeah, I need that. I've been saying for a while. I didn't think of it specific to voice, but that I need an AI to just call me out on my BS. Yes. To be like, yes. These are your goals. You said you were going to run. You didn't do the things that you said you were going to do. But also what you're describing. You use the dualingo versus speak example. But in a niches prediction, he also talks about how, yes, some of these big companies are going to integrate these APIs or integrate this technology. But Gmail probably still going to look like Gmail.

20:13Yeah. And so how do you think about that balance between the incumbents utilizing this technology and then what's going to sprout that's completely new? It's really interesting and something that we watch really closely and consumer in particular because you would think that the Google's, the Microsoft's have all of your data, they have all of your permissioning. There's a lot that they could do. I think what we've seen is they're structurally in some ways disadvantaged and building towards this AI shift in a really native way. One, it's like these are big companies now. They have a lot of people.

20:43They have a lot of competing priorities. And then the second thing would be in some ways they would cannibalize their own products. Like RV was been Google is likely to maybe add AI to augment Gmail, but are they likely to create the AI native version of Gmail that you could only conceptualize in the past three to six months, probably not just because again of how big of a company they are are in the fact that they have so much writing on the continued success of the existing product. A good example of this is actually Zoom added transcriptions. Are people using that? Yes, but there's also been a ton of products that are independently successful in doing AI meeting notes.

21:24And those largely are building towards more specific and opinionated workflows for different types of jobs or tasks. And it's just something that Zoom is never going to do because they're such a broad -based platform. Talk about a completely new platform. like imagine zoom, but it's asynchronous. Yes. Right. They're never going to build that point. Yes. Exactly. Because they're inherently synchronous. Clearly, open AI is investing in voice. Right. And that's not necessarily a given, right? If you think about they also do imagery, they haven't really talked about Dalie in a while. Right. They also do video.

21:55Sora came out a little while ago. But there really seems to be this voice push, despite them operating across modalities. Is that a signal people should be paying attention to? I think so. I think we've seen already almost even though it's still so, so early, like, eras of AI so far. Creative tools was the first era and still a massive era. And I think we saw a ton of investment in image generation, video generation, music generation, much of which is still happening, especially it feels like as AI moves from pure consumer use cases into more kind of controllable, highly monetizable enterprise use cases.

22:33It does feel like voice is kind of a unique unlock in that it's a real game changer for companies in particular to be able to capture and utilize this audio data that they never have before. Maybe another thing worth talking about here from DevDay is that they announced that they have three million active developers in the ecosystem and they tripled the number of active apps in the last year. Since you've been studying consumer for so long, maybe ground the audience in how much quicker is this happening per se than let's say the app era when Apple releases App Store. How long did it take for three million active developers to be building on it and just how big is that kind of number today?

23:17Yeah, that's a great question. I have a great idea. I have a question. You were asking the question. I was like, do I know that for App Store? That's right. I do not. Well, it took, I assume, years. Three million developers, that's incredible. Like, my math was like, look, like, I don't know the App Store number, but let's say each developer has a ability to, I don't know, like, maybe reach out to hundreds or a thousand unique users. That's sort of how I think about, basically the reachability of what they're building. So I think the other question is, like, what is the revenue per developer in the App Store and is that a proxy for an AI?

23:46Yeah, that's super interesting. There is a data that I think, I'll try to put out where you look at, it's not necessarily the absur ones, but it's the SaaS. Like, historical SaaS companies versus Genai companies. And how the Genai companies are reaching a scale of revenue way faster than their SaaS counterparts. Cool. Very interesting. Yeah, I think a big part of that though is because Genai is so well set up for consumption revenue. And so many SaaS businesses are SaaS. Yeah. They're like, you pay a fixed fee for the service monthly. And with a lot of these new businesses, you're paying on a consumption basis.

24:21You're also pricing it as a subset of labor costs, which are traditionally priced for higher than software costs. I think that's like a far more compelling argument for why the revenue ramp is much faster versus I think the reason why the report said was because the Gen AI companies require training costs upfront, therefore they're imperative to make money as higher than SaaS, which Maybe, but we know the ones that are making money aren't necessarily incurring a huge training cost upfront. Much more likely as they're replacing labor costs, or as just so useful or so unique that willingness to pay is just higher.