AI-Powered "Tools for Thought": Exploring NotebookLM with Steven Berlin Johnson | E1869

0:00So I wrote this book called The Ghost Map. And so I would often start off just to figure out where Bard was today. I'd be like, hey, let's talk about Stephen Johnson's book, The Ghost Map. And so one time I did it and Bard came back and was like, oh, I would love to talk about that. That's a thrilling medical mystery set in 1854s London about the Dr. John Snow and his investigation. And it's a tale that weaves together a number of different themes, blah, blah, blah. And I finally got to the end of it. I was like, oh, well, thank you very much. I'm actually the author of that book. And I said, oh, I am so excited to meet you, Mr.

0:30Johnson. if i had any idea i'm so sorry i didn't recognize you and i was like that's fine you there's no way you could have recognized me right it's just one of those moments where i was like what is even going on this week in startups is brought to you by dev squad most dev agencies only offer developers why because product management is hard get an entire product team for the cost of one u.s developer plus 10 % off at devsquad.com slash twist. LinkedIn marketing. To redeem a$100 LinkedIn ad credit and launch your first campaign, go to linkedin.com slash thisweekinstartups. And FitBot. Tired of doing the same workouts at the gym?

1:15FitBot will build you personalized workouts that help you progress with every set. Get 25 % off your subscription or try out the app for free when you sign up now at fitbod.me slash twist. All right, everybody, welcome back to this week in startups. Got a really special guest today. I have known Stephen Berlin Johnson for a long time. We met in the 90s, which was pretty much the best decade of the last century, I think, maybe the 60s. Some people might make an argument. But when we were kids, we were running around New York City. It was called Silicon Alley back then. And he was running a website called Feed, a zine and i had a zine my zine was print his was online i was doing silicon on the reporter and we were all trying to figure out what would happen with the internet uh steven went on to write 13 books uh where good ideas come from actually i read that one really good and i think he reads his own books on audible so you get to hear his voice um he's got a great podcast called american innovations uh and now he's working at google and apparently steven uh you're a developer it's so nice to see you after many years i mean really long time it's been a really like but you're looking good i'm very impressed you too you too uh life is good right i mean but here we are you and i started on the internet before the web existed we were doing online services we were hanging out with people making cd-roms voyager blender whatever it was like a really interesting time the internet happened and somehow you wound up at google building uh notebook lm so So I guess let's just start with that.

2:50What is Google Notebook LM and why are you building it and why did you take a gig at Google? Yeah. So in a weird way, it's like a 40-year story because I had this long obsession with tools for thought. I mean, when we first met, I was thinking about this stuff. It really started when I was in college when HyperCard came out for the Mac in 1988, which was kind of proto, almost web-like thing that you could organize information. You create these little stacks of cards and you could kind of link between the cards. Yes, it had the first hyperlinks. People forget this. It had the beginning of HyperText, although it was not a network thing initially.

3:27It wasn't really connected to the internet at all. I think maybe a later version finally got wired up to the outer world. But I just had this glimpse of, oh, I could use software to help me think and create and have more interesting ideas and make connections and not just use it to kind of format my papers. You know, there was this little hint of that. I think a lot of us who get interested in technology get this little taste of an idea. Yes. At some formative point. And the tech isn't there yet, but you're like, I know someday I'll be able to do this. So, that was always in the back of my mind.

4:02know some of the things that we did at feed trying to experiment with different ways to use hypertext that was one of our kind of calling cards in the early days creating new ways of kind of connecting ideas and print and you know through text um and then i was always you know i wrote a lot about the tools that i was using to write the books you know i wrote about the stool devon think that i used for a long time i was i am a big scrivener fan i've written about that and in in my book where good ideas come from i talk a lot about how you create environments that allow you to think more creatively and and and and so i talked about software in that context as well so there's this kind of long history of this and then of course language models came out and you know we had you know in the early days of you know kind of behind the scenes google had palm and and lambda and then of course um gpt3 had come out and then uh and so in the spring of 2022 um i got a kind of a cold email um from a guy named clay before who had started kind of rebooted labs at google and he had read a bunch of my books and had followed this kind of train of thought uh in this interest in tools for thought and he reached out and he said hey do you i want to talk to you and so we met but this is actually the craziest story so we've met in um project starline which is google's new uh kind of hologram technology oh yes this is like a holodeck you you go into a phone booth i think and the other person's in a phone booth in new york and i'm in san francisco and it kind of projects the a full 3d model of the person you're not wearing glasses you're just sitting there and you know and so what's it like it was one of the you know i mean this is i say this i said this before i joined google it was top five impressive technology demos i've ever seen in my life it's it's it's uncanny um and uh and during the the conversation he you know clay kind of persuaded me to come be a part.

5:57You said, we've got a team, just a small little team, but they're ready to kind of build something in this mode with language models. Like we can finally build the thing you've been dreaming of your whole life. And why don't you come, you know, initially part time to Google and just be in the room with us and help us create, this was a big part of the lab's ethos was to like bring people from the outside, the early stage of these products and help develop them. And I thought, you know, that's a pretty cool, interesting idea. You know, you don't get an opportunity like this very often and then i got out of the hologram the project starlight meeting and i thought he literally put me in a reality distortion field yes this is not a term of art this is a reality and just one more thing on starline so you sit and the other person is sitting across from like and it feels like a table and there's depth to it but it's some sort of uh is it a television screen or a projector i mean here's like an image of it But how does it get the depth feeling?

6:56I wonder. It's tracking the location of both of your retinas. And it's sending the screen. It looks like a regular television screen, but the screen is sending a different set of pixels to your left eye and to your right eye. And so, it only works currently, I believe, with just one person on one side, one person on the other, because you have to send those pixels directly to it. But it creates a very powerful thing. My favorite example of it is that I, when I was testing it another time, I wanted to kind of lean forward to see if the illusion goes away if you lean forward. And so I leaned forward and I had this very strong visceral feeling of, oh, I'm invading the space of this other person.

7:38I'm like close talking, all my breath, all this stuff. And then I was like, wait a second, this person is like, yeah, yeah. You're like leaning in for a kiss or like bad breath. I mean, it's like all the things you're not supposed to do and you can kind of get too close. That's wild. Is it going to come out as a product ever? Where's it at? I shouldn't speak to. Yeah. Okay. Right. Yeah. It's in labs. That's enough. It's in labs. Yeah. All right. Going from an idea sketched on the back of a napkin to a robust, stable product requires a wide range of skills. You can spend ages looking for a one in a million developer who can do it all, or you can quickly ramp up an entire product team to help you build and launch your product with our partner, DevSquad.

8:18DevSquad provides an entire development team packed with top talent from Latin America. Your elite squad will include two to six full stack developers, a technical product manager, plus experts in product strategy, UI and UX design, DevOps and QA, all working together to make your SaaS product a success. You can wrap up an entire product team fast in your time zone and at rates 75 % cheaper than a comparable US based team. And with DevSquad, you pay month to month with no long term contracts. Take the hassle out of assembling and managing a sprawling team of freelancers and work with a group that's ready to hit the ground running.

8:53Visit devsquad.com slash twist and get 10 % off your engagement. That's devsquad.com slash twist. So you mentioned Scribner. I remember when that came out. You're an author of many books. I am the author of one. But when you're writing a book, especially if you're writing a book in the veins of like yourself or, you know, Malcolm Gladwell, Well, a lot of times my understanding is you're trying to take disparate ideas, stories, and kind of pull them together. And hey, this has been your life's work feed. It was a feed before RSS existed. And hyperlinks, you were trying to tell stories. So when you click the link, you went somewhere.

9:27And that was an epiphany or an emotion or, you know, a change something about your perspective when you went to that destination. It was part of the fun of clicking on hyperlinks, which seems incredibly basic now, but it was incredibly mind-blowing at the time. But the process of writing these books takes years. It takes trying to put all this information together in some way. And that's what you've built with notebooks. So maybe you could just show it to us. And I remember Scribner was like this too, because you had kind of like post-it notes kind of all over the place. And, you know, architecture, you do a table of contents.

10:00There's been software like the brain that Jerry Mikowski was obsessed with. If you remember where you would just kind of make little neurons and still is apparently. i don't even know the company exists but i know his jerry's brain exists somewhere on the internet watching it run that thing it's incredible yeah i mean i think one of the things is before we kind of dive into it in detail like one of the things that's important maybe wasn't even fully clear to me when i started this is that what i do with the books is is just a more exaggerated version of something that i think a lot of people do anybody who works as kind of a knowledge worker in any discipline, which is that feeling where you're sitting down, you're trying to generate your ideas about something or kind of build the first draft of something.

10:44And the information you need to pull off that job is scattered across multiple documents. And in my case, it might be scattered across like hundreds of documents, you know, that I just finished. It has, you know, literally, I think something like 400 newspaper articles are part of the research for it. But, you know, even if it's 10 documents, that's a lot of information. And, you know, we just are constantly in this state where we're like, oh, I've got 10 tabs open. I'm trying to figure out this thing. I'm trying to put together this blog post. I'm trying to put together this marketing plan. I'm trying to synthesize these ideas for a legal brief, whatever it is.

11:20And the information is kind of scattered everywhere. And we've never really, you know, so many people talk about the experience of like, you go through your tabs and you command F to try and find the thing you're looking for then you find it finally and then you copy and paste it back into the other doc which is in the other tab and it's it's incredibly laborious work yes and it completely it it's so disruptive of your flow state like your your creative state you're not thinking you're just doing this kind of menial you know archival labor trying to like find the thing you're looking for and not just kind of thinking and writing and there was a reason for that it's just there wasn't any software just wasn't smart enough to kind of find and summarize and make sense of the meaning of documents until that's the key part right i mean you could store documents you can retrieve documents you could edit cut and paste them and search google's you know wheelhouse was incredible and then you know organizing stuff but it didn't understand the entire corpus and whether you're doing something like i don't know you're doing m &a and you got a document library of all the documents to close this M &A deal or an investment, or you're writing a book and you've got 100 different articles and you're writing a biography of somebody and you've got all the articles written about them, you can't keep it all in your head.

12:39That's just not how the human brain works, right? And you meet an interesting human, right? If you meet somebody super interesting, Jared Diamond, or I don't know, pick somebody who's got a big brain who has synthesized a lot of this stuff for 30, 40, 50 years, when you talk to them and they put it together that's kind of like the great interview right when we find a great interview and that's the one that blows everybody's brain like wow this person is doing that in real time the documents are their brain yeah actually i'm glad you put it that way because that's that's the thing i've been trying to explain to people a little bit which is one of the things that the notebook lm lets you do and we can explain it more detail in a second but it it gives you all these tools for basically having conversations with documents and you know having an open-ended conversation with a book or a chapter of a book or a collection of quotes from books that you've assembled over the years is just something that was not possible before.

13:32Like you could, if you were lucky enough to meet the author of a book, you could have a, you know, if you met Jared Diamond, you could have a conversation with him. And that would be kind of like having a conversation with this book. If you could meet an expert who had studied Jared Diamond or a tutor or a great teacher, maybe have a conversation. But now like you can just upload these dockets and you can let's show people we got to show people this all right so i'm going to show you what i i literally got so excited about this because i was i'm working on a project and i can't wait to show you and get your feedback but um this is a very mind-blowing uh i want to show you so fittingly okay here let me share if you're listening in the audience and you want to go find this just go to youtube and type in this we can start up so you'll find it um and i'll try and describe and we'll sports cast it yeah we'll sports cast it okay so i thought i'd start appropriately enough where I've loaded up like five chapters from that book where good ideas come from as sources.

14:24And so when you begin in notebook LM, you know, one of the first things you do is you kind of define the documents that you want to work with in a particular notebook. So as we said, those could be, you know, a bunch of marketing documents, it could be research for a book, it could be legal briefs, whatever it is that's relevant to the job you're working on. In each notebook right now, you can have 20 documents, they can be 200 ,000 words each, They can be PDFs. They can be docs. They can be copied text. We're going to add a bunch of formats, as you can imagine. Those are your sources. Those are listed on the left here with these little document previews.

15:00Yeah. And that's an important word. The source has a very specific meaning here. And so once you load them into Notebook LM, the AI is grounded in the information that is contained in those sources. So it takes about, I don't know, for loading these five chapters, I think it takes about like 12 seconds or something like that. And then at that point, it's like the AI, it's like Notebook LM has become an instant expert in those five chapters, which is just astonishing to me that a computer can do this. And one of the things we've done that we've spent a lot of time with inside of Notebook LM is like setting up guardrails.

15:33So, if you try to ask questions that are outside the boundary of the information that is containing those documents, generally, it's not always perfect, but generally, Notebook LM will decline to answer. It will say, I'm sorry, I can't answer that question. It's not containing the sources. Okay. So you can see how that's a great use case in the classroom, right? So you can see a teacher loads up a shared notebook with a bunch of documents that they're using for the syllabus for the class. And the student can use those, but they can't kind of go outside the boundaries of that. So I've loaded up here, I think it's five chapters from where Good Ideas come from.

16:06And I actually just did a question before just to preload it. So there's a whole riff about 9-11 and the kind of intelligence failures in 9-11 in that book. And so I ask a question, you know, this is not keyword searching, right? This is the age of language models. You can ask these very sophisticated questions. So I ask what happened with the FBI and 9-11 and what was the significance of it? and it's going to answer that question based on not the general significance of it it's going to answer it based on the significance that i you know kind of endowed it with in writing this particular book so it's going to be based on the the subtleties of the interpretation of that that i wrote in the original book so this isn't going out to the open web and finding a new yorker story or the wikipedia and then hallucinating or a bunch of conspiracy theorists this is you know you've got a narrow corpus here to really um return a tight summary yeah and and what it ends up doing is it greatly reduces the hallucination risk you i mean you will still see you know occasional mistakes or sometimes it will just it's more like it gets confused sometimes if the information is a little bit confusing but reduces it significantly um and you get these i mean this is just amazing this is by the way this part of it we're kind of rolling over to having everything be powered by gemini pro which has been fantastic in our early testing of it and so this is an answer that gemini pro returned um it it formats it in this really nice way like it has kind of bold text for the key terms gives you nice little bullet points it does this you know really elegant kind of overview of of the situation it's nice kind of presented in just kind of a nice way we spent a lot of time trying to get that style right like it's it's it's very much like editorial product you know this would be the equivalent of if you had a great writer and you said hey summarize this very long story uh or series of books you know in our magazine or zine so that a casual user can you know within five or ten minutes of reading 400 500 words really understand it uh and not have to do a lot of work a lot of heavy lifting so it says here hey you got the phoenix memo you've got something that's overlooked so you're doing headings you're putting them in bullet points You're making it easy for people to digest knowledge.

18:19And so somehow you've trained the model that this is an academic synthesizing world in which you're working. Yeah. Yeah. And so then, so that's really useful. But then with every answer, we also give you citations. And so you can always, in a sense, check one. You can just double check that, you know, what did the model use to come up with this thing? So, if I mouse over these citations, I can see these are the original quotes from my book that it used to piece this together. And sometimes you use that just to fact check, but sometimes that's actually what you want to do. You want to actually read the original text, and this is just a faster way to get to it.

18:57And so, you can click on any of these, and it immediately opens the original source and takes you to the point in the original document. So, you can actually, like, read it originally. So, your ability to just kind of navigate through, I don't have it on this computer, but on my main Google computer, I have a collection of quotes that I've been assembling, like digital quotes from books that I've read for the last 20 years. It's like 1.3 million words of quotes from my reading history of the last 20 years. And I can literally just sit down and be like, hey, what are some interesting things about dolphins?

19:33Tell me about love. And it'll find all these things. And then I can zip immediately to kind of the original passage and see it. And that's the speed with which you can move through information and have it summarized for you is just incredibly powerful. Now, the other thing that's, you know, we were talking about having a conversation with a document. So, one of the things we found when we started doing early testing this summer at a few schools around the country, a few colleges around the country, was that people didn't know really how to ask questions. you know they would sit down they were like what do i say you know you and i are trained as journalists and so we know we know how to ask questions you know we know how to kind of think in that and you know uh dialogue kind of mode but there doesn't it's not something that's actually taught very very often and so what we started to do is actually to use the model to constantly give you suggested questions based on the content of the source and based on what you've just talked about and so you can see right below here there's a set of suggested questions here like what was the central claim of the phoenix memo and why did it fail to prevent the attacks how did the automated case support system contribute to the failure of the fbi to connect the phoenix memo uh so that's a pretty good one let's see what happens i'm going to click on this we're doing we're now in live demo mode here uh we can ask it why didn't anybody pick up that the terrorists yeah i want to learn how to land the planes it was like a little bit of a red flag like you sure you don't want to learn how to land a plane nope just how do i get it up there and keep it straight and this is such a good answer man i i'm still astonished at this that this is possible so yeah it's talking about basically there was this automated case support system at the fbi and they were just it was designed it was basically designed it was the opposite of notebook i love actually now that i'm thinking about it this is a really good example because they had all this information and they were they were unable to connect the dots so the software here has has summarized all that stuff it's figured out you know it gives me a bunch of different details from that story and then it gives me a nice little summary at the end and then again i go look at the citations like that that's the the kind of the core uh kind of kind of very beginning of this um project but anything you find that is interesting so you're you're engaged in this and you're exploring you're asking questions you're following you're clicking on the follow-up questions um you're following citations you find something interesting you just pin it and then you've got this kind of note board space this looks great on a on a big screen by the way it's incredible yeah i have a wide screen and i was doing this on my example it's really nice so then you have all these notes that you can go and refer to so you're able to just constantly like grab things that are interesting and what's coming uh like any day now um you're going to be able to select a note or a set of notes and we're going to automatically give you a set of options when you select those notes and those options will be things like create a study guide or convert into a thematic outline or suggest related ideas for my sources and so what sources am i missing yeah finding new sources is the one i was looking for that's that is that is not short term coming out yet but that is definitely where we're headed i mean for sure i mean it's google here you know your work is great um if it said you know hey i found three other sources would you like to add the actual 9-11 commission report which is a giant document to this would you like to add this amazon series uh a fictional series to your to or the or the script from it you know now all of a sudden it's this research assistant on either side of you answering the questions and this is how documentary films are made right or or a book like yours or malcolm's where you're trying to pull together themes that you know only a human previously could make these connections and when you make those connections and they and they spark something in you as an author we want to share those right and we want to have that spark happen in your brain right absolutely and that's really the magic of being an author in my mind is when you can get into when you can and stephen king wrote this you ever read on writing his book his autobiography did yeah it's incredible i mean i mean there's many reasons it's incredible perhaps one of the best is that he wrote cujo while doing copious amounts of cocaine with his nose bleeding and just that kind of tracks right yeah having read that book i think it's probably makes a lot of sense probably feels like it was yeah powered by cocaine but he said you know teleportation what writing great writing is is you can basically teleport through time and space whatever you write it just magically appears in the other person's brain so if you if you were to describe your desk right now and the microphone etc and somebody read it a hundred years from now all of a sudden you would have manifested in their brain this image and that is just once you look at writing that way that you're literally creating a vision in somebody else's brain you really understand the power of it and why it's important in the world and then this it just takes it to a whole nother level that most people are not even thinking about right now well that's it's a great point and it reminds me of something um you know tiago forte the second brain i wrote how to build a second brain really interesting guy we've been talking to him about notebook because it's right up in his alley and uh he has the thing about like taking notes and capturing things that are interesting to you and kind of storing them is this way of you know you're trying to do it to like send a message to some future version of yourself who's going to need this thing in five years they don't know why but like this piece of information is not and so you know i i haven't kind of fully done that just the kind of cleaning up of my documents to do this but you know by the end of the year early next year i'm gonna have one notebook that's effectively like all the important things i've ever read and all the important things that i've ever written are going to be live in this one notebook and my ability then to find that idea that i jotted down you know in 2007 that yeah you know is so long gone you know to my actual physical brain's memory but that you know the combination of this software um and my curating all the information initially is gonna is gonna enable me to find those things again and it really evokes you know like where you all always wind up in these discussions is consciousness right like what is our consciousness but a collection of the things we've read we've written we've spoken we've experienced and so now we start looking at what is actually being created with these llms and i think we're it's either explaining to us um how basic our brains are in some ways or how incredibly complex they are because if you did have all of your writing every instant message every email and then you start thinking well every song and movie i've consumed every book i've consumed everything i've written you've now got almost your consciousness you know in in a book here and that gets really trippy because it's a perfect consciousness as opposed to ours which has bias or something when we actually become conscious of something we know this because in different scientific studies if they play some music or you know give you a scent or take you to a location in your high school all kinds of other memories come out yeah but but this is like becoming perfect memory uh to the point about Yeah.

26:56We're trying to, it was interesting. I think, I think you saw that Stephen Levy piece that he wrote about it in, in Wired. And, and he, I think he had gone into it thinking it was just going to be a really good search engine for his own stuff. And he was like, it turns out it has a little bit of its own opinion about things. Like it's steering you towards certain ideas. And, and, you know, we've spent a lot of time trying to figure out like, what is the right balance here? because you want your kind of assistant to be smart and help you develop your ideas, but you also want it to be, you know, to play second fiddle to your ideas, your own thinking.

27:32One of the things that's interesting about it is, in general, we have tried to create a voice for the AI that doesn't have a first-person voice. So, it's not trying to be your buddy, right? It's just living you the information. It's not, we're like, we don't want it to pretend to be a person. It just should be somehow just an incredible service that finds the information you're looking for and summarizes it and doesn't. So every now and then it'll sneak in, but it won't generally say things like, sure, I can help you with that. Yeah. That kind of stuff. You got it, buddy. Business to business marketing is not an easy job.

28:09It's much different than business to consumer advertising. Why? Well, the enterprise buying cycles are very long and they're filled with decision makers And those decision makers are going to kill your deal if you can't get to them. That's why you need to check out LinkedIn ads. LinkedIn has amazingly, but not unexpectedly, passed a billion users. This includes 180 million senior executives. There's also 10 million C-suite executives. Those are the CEOs, CFOs, CTOs, the chief strategy officers, chief finance officers. This means 18 % of those users are the ones who are the decision makers. How do you get to them?

28:44You get to them through LinkedIn in a respectful business environment. They're ready to accept a business message as opposed to, you know, another platform where they might be consuming cooking videos or podcasts or political discourse. No, LinkedIn is about business. You want to get people when they're in that cognitive mindset and they're willing to accept a business to business message. 79 % of B2B content marketers said LinkedIn ads produces the best results for paid media. This is obvious. I can tell you this is true. When you think about business, you think about LinkedIn. It's just exactly what comes to mind.

29:16So here's your call to action. Make business-to-business marketing everything it can be and get$100 credit towards your next campaign by going to linkedin.com slash thisweekinstartups to claim your credit. LinkedIn.com slash thisweekinstartups. No spaces, no dashes. LinkedIn.com slash thisweekinstartups for a hundy,$100 in credit. Terms and conditions do apply. I'm going to show you one other thing and then I want to see your human doing with it. But I just want to make sure that, you know, it's clear that this also has, it doesn't always have to be such a highbrow use case. So this is something that occurred to us very late in the process where we were like getting ready to launch.

29:53And we were like, wait, we can, we've written all these help documents for using notebook.lm. What happens if you just load those up as sources into notebook and it becomes an instant expert in how to use notebook, which, you know, it turns out to be a really, just as a, as a kind of onboarding guide. it as a help desk it's so powerful so anybody whatever kind of service you are a company you are you can just load up the documents that describe how your system works and then you can share notebooks with other people and so then they can ask questions and engage and one of the things i like so you can ask these kind of prosaic questions like how do you upload a pdf to notebook lm it'll be good but generally you know this is um uh i'm a lawyer um how could i use notebook lm so we don't have any language in there about like lawyers using notebook lm all we just have is a description of the product so generally this is generally this generates a pretty interesting answer like it kind of understands yeah this is great so it understands how the software works and it understands generally how lawyers work and so it's able to kind of synthesize those two things and come up with this actual kind of precise you can actually dive in deeper and say actually can you give me a step-by-step um you know set of examples for how i could use it to draft legal documents or something like that right you just go down the rabbit hole from there yeah it's really interesting like um if you were to think about instances where you need a large corpus of documents and and in fact when we were talking about the phoenix memo and like the failings of that computer system kind of alluded to the fact that hey you know if you're trying to do a case you're trying to solve some puzzle in the world which is what law enforcement does or mathematicians etc you know there are going to be these examples of the language models being able to if given the right corpus of information and given the right prompts and somebody actually reading the result and checking the work we're going to solve a lot of mysteries you know like just like these crazy people who get obsessed with um i don't know if you've seen these lunatics like on subreddits and etc like and they make documentaries about them i think there was one like something about cats or something but there are these people who don't mess with cats i think it's the name of the documentary but there are these people who are like stay-at-home crime hunters like detectives from their keyboards and they find a case of a missing person or the you know somebody who was murdered and they are inspired by other podcasts about people being murdered to then go solve it and they become online salutes and it's usually they don't find anything and they just are twisting uh you know reality um yeah it creates sometimes they do figure stuff out well you know it reminds me of what you said at the beginning of the conversation about how mesmerizing it was to click on a link in 1994 right you're like i clicked on this blue word and i was taken from one server to another to a completely different document on the other side of the world, and it just seemed mind-blowing.

32:52And it was just fun to, in those early days, just clicking around the web was kind of just fun as an adventure. And I think that there's something that is like that, that I'm seeing again in Notebook LM, in part because of the suggested questions. I mean, that has that same feel of like you jump into a big archive of information, you ask an opening question, or we'll actually recommend topics that you can explore right out of the gate. And then you just kind of ride those suggested questions for a while and you get, it's a great way to kind of engage with, you know, kind of initially figure out what's in this text and discover new things.

33:29And I could imagine somebody who was going down a rabbit hole of some, you know, complex crime, you know, investigation and had whatever, 30 PDFs of evidence there. Yeah. That that could be particularly intoxicating. Let me show you what I did. And this is related to the pod. So, you know, I've been doing this pod for like 1800 episodes and like sometimes these things have to remain interesting to the host, right? Like the art has to be interesting to the person producing it or else it's like it just becomes a chore. And I was like, you know, I just love movies and I love business stories. So let's just do like this business breakdowns where we find, you know, a great movie about business.

34:06And then we kind of do like a bullet, you know, either timeline kind of approach or the key lessons from a movie or from a success story. and so your mind is already racing with the possibilities of movies to do because you're uh well read i assume so ray crock who created mcdonald's uh wrote a great book called the founder and he was an insane founder like pretty sharp balboad insane and i know about this because i'm a big fan of mark knoffler the lead singer of dire straits and when he went into his solo career he wrote a great song called boom like that it's probably one of my favorite songs of his um solo career and he had read ray croc's grinding it out biography and wrote a song based on it the guy who made the movie the founder with michael keaton had heard mark knoffler's song found the source material of the book and then convinced michael keaton to make a movie the founder is an incredible movie that nobody remembers have you ever seen it never seen it and you know i assume you haven't read ray croc's i have not for grinding it out right so this is like obscure shit but i read it and it spoke to me now i got to it mark knoffler song the movie then i found out the movie was based on a book okay here we go right because i went down the rabbit hole so then i was like okay i'm i'm doing this episode will come out after this episode comes out where we're going to break down the story of ray crock through the song by mark knoffler and the movie and the book three interesting pieces of source material so then i went out and you know and i paid for all this stuff so anybody who wants to give me a hard time about copyright i found what i assume is the public domain version of a pdf of the script of the founder i found a public pdf of the book grinding it out which i have three different versions of i've paid for so please do not sue me uh if this is your book i assume i'm legally allowed to do this um and then i found a youtube video now the youtube video i thought i could just drop it in there since it's google but you guys are version 1.0 not yet but i know there's transcripts so i i opened the transcript and i very awkwardly had to you know drag and drop and cut that transcript in here so i got my three and then i started asking things hey what were the key moments uh you know in the history of mcdonald's and then i said well what page is that in the screenplay right and so i haven't gotten too down things but you know i also asked well what is uh uh ray crock consider his the key to his success now i don't know where this is coming in from citation wise if but you know he said it's perseverance and determination yada yada like you would expect and so i'm just started my adventure but there's probably 20 other documents i don't want to find the harvard case study on this and the penn case study on it and you have to put the song lyrics in great what are the let's see what are the suggested questions actually so what were the three decisions the mcdonald brothers made regarding the design of restaurants that made it stand out from other drive-ins oh i know this it didn't have seating right it was a counter so i know it didn't have seating um the limited menu that's right and the streamlined process uh and the unique building design is there the brothers restaurant wasn't a building with a red and white in here the sign was eye-catching i guess they didn't the restaurant was a standout from other drive-ins interesting yeah so look at the citations too um so just coming from the script you'll see so when you roll over yeah you can see it's coming from the grinding it out which is his book yeah yeah so i mean i you know what i i've only been in this for like a half hour i was like most of my if you do a listening lab with me like the hardest thing i had was getting information in um and i tried to do this before i found out about notebook lm when i saw steven levy's story on you and said hey i know i know steve i know you steve and i said get them on the pod i want to talk about this i just couldn't figure out how to get stuff into it that was my blocker and then i was like oh wait a second and i then i took the same pdf and i tried to put it in claude and claude was like yeah that's way too big and i was like god damn it um right now by the way for folks is is either a simply formatted pdf or it's just kind of a straight text thing if you if you have a pdf of a book you know you have all those complicated things where like it'll grab the like headers and the page numbers and footnotes and going to be a little confused um anything that's in doc format is going to be good um but we're gonna you know we're gonna get better and better and you know the other thing is that gemini is is natively multimodal so there's there's all this incredible stuff that we're going to be able to do with images as well i mean right now it's all text but we we were just doing a bunch of experiments the other night with images and it's it's astonishingly good all right you know I've been on a health kick over the past year.

39:00And you know, I care about data driven solutions. And if you listen to this podcast, I bet you do too. So let me tell you about FitBot. This is a data driven workout app that blends machine learning with exercise science. FitBot creates custom dynamic workouts programs based on your fitness goals, your experience. And most interestingly to me, the available equipment, let's say you got a bunch of kettlebells, or let's say you're at some, you know, sparse gym at a hotel, or you're on vacation, you got nothing. Well, FitBot will maximize your fitness gains by varying the intensity and the volume between your sessions and leverage the equipment you have or don't have, as the case may be.

39:38You can customize the length of your workout, what muscles you want to target and so much more. So let's say you want to get a 30 minute workout in and I want to do chest triceps and abs, but I'm staying at an Airbnb, there's no equipment. FitBot can create a perfectly optimized workout for me based on these parameters and it will do it for you to check it out. It's amazing. The design of this app is extraordinary. I was able to invest in it. That's how impressed I was with it. Fitbod takes the guesswork out of fitness. Just open the app and start making progress. You deserve it get 25 % off your Fitbod subscription or try out the app for free.

40:12When you sign up now at fitbod.me slash twist. That's F-I-T-B-O-D dot M-E slash T-W-I-S-T for 25 % off. Have you looked into the rights and you're an author? If I asked it, if I had purchased your book to allow me to talk to your book as an author, you'd be cool with that, I think? Yeah, well, I certainly would be. I would be as well. I would want people to do that. Yeah. You know, one of the things that we like to recommend people do is, you know, you can use, if you buy an e-book on the Kindle or the Playbook store, you're allowed to save quotations from those books. So as you read, you can save quotations, and they're wonderful services like Readwise that will allow you to export those quotes to a doc, and then you can just bring those in.

41:06So you wouldn't have the whole book there, but actually, sometimes you don't want the whole book. You want the passages, the most important ones for you and then and so that's what basically i've been doing with my my set of quotations so that's a great way to do it but i but i agree i think you know there's a logical place we could end up where if you buy an ebook you could read it in an ebook reader or you could read it inside of notebook lm i would love i would love that future yeah i mean i think we i wonder how the our publishers are you know how harper's or whoever your publisher is is like thinking about this because i feel like you could charge an extra 10 bucks right for a digital book to allow it to be to talk to it you know and to query it um or it could just be like it's almost like a new format like i wonder if apple they must apple and amazon must be thinking about this as a feature you are you are thinking very much along the lines that i've been thinking so sorry i would hope that we would make some progress on that front next year but the other thing that that's really worth pointing out to people, and this is important whether you're an author or not, is in terms of privacy and security, we are not training the model on the information you upload in those sources.

42:13So the model has been pre-trained. What we are doing, the easiest way to put it is we're putting, if you know the AI languages you do, we're putting the information from your sources briefly into the context window of the model and asking you questions based on that. for people who don't know what that means it means we're basically showing the model we're giving your information to the model's short-term memory and the second you end your conversation it remembers nothing and right and we do we do no training based on the on the information in the document so that means you can use it with you know private documents corporate documents or a rights holder can feel confident that if somebody is taking some quotes from a book that they've read that they purchased it that information is not somehow getting into the training data for the models is also key issue for yeah i think it's a key issue for authors i i was um really flabbergasted by open ai's approach and you work at google so i'm not going to have you comment on i'll tell you my opinion that they just took open crawl and some other you know corpuses and train their model on it and they know full well that they're that's just taking the open web and that the open web has all kinds of stuff that hasn't been cleared and then they train the model on it now it's six six six of my books were in there right now as far as i'm concerned they owe you a licensing fee and you know the books are in there because when they you ask very specific questions it's going to give you the answers and in the earlier versions of chat gpt you could ask it is you know steven berlin johnson's books in here and it would actually tell you yes i've got them right here um have in your case did they subsequently take them out and what do you think of these um just broadly speaking the rights of authors in terms of model training because it feels profoundly unfair to me yeah i i probably should be delicate about this oh right because of where you work you know but um but i do think um it does feel like one as an author i want people to be using language models with my work i think that's a way that people are going to be exploring information so i'm very comfortable with people if they do it in a proper way they're paid for the book they should be able to interact with the model and making sense of it and i do think that yeah if models are being trained on copy you know written material that there's there needs to be something that's going on with the rights holders there but um i honestly i haven't spent that much time thinking about that side of it because i've just been so immersed in the product side of it um i mean in some cases it's super obvious right like i was asking chat gbt to make me like a jedi knight bulldog and it was like sure here's a jedi and i was like okay now make me darth vader as a bulldog and it was like sorry i can't do that because our content policy and i was like okay i get it you understand like a jedi is a category but not a character so they must have taken the entire disney corpus and said for dolly let's not make images of marvel characters let's not kick the number one rights holder who is the most litigious and thoughtful and you know about this and so it declines to make that and so then i just said i'll make a sith lord and it literally made me Darth Vader as a bulldog.

45:15Okay, here we go. I talk about searching authors' work. So, in the early days when Bard first came out internally, I was kind of at home where the rest of my family was skiing. This was like a year ago. And I was just like, okay, this is our new model. I've got to test it. I've got to figure out everything. So, one of the things I would do, like Bard's kind of intelligence in those days would go up and down as they were retraining and things like that. Variable, variable. I have like a standard kind of question that I've asked. So, I would often ask, so I wrote this book called the ghost map and so i would often start off just to figure out where bard was today i'd be like hey let's talk about stephen johnson's book the ghost map and so one time i did it and bard came back i was like oh i would love to talk about that that's a thrilling medical mystery set in 1854 london uh about the dr john snow and his investigation and it's a tale that weaves together a number of different themes and i finally got to the end of it i was like oh well thank you very much i'm actually the author of that book and and said oh i am so excited to meet you mr johnson wow any idea i'm so sorry i didn't recognize you and i was like that's fine you there's no way you could have recognized me right it's just one of those moments where i was like what is even going on uh so this feels to me like part of google docs eventually what how do how does a lab product where does it go from here because it feels like a product and listen i'm tip of the spear for you and and use cases so i i'm wanting to pay 199 bucks a year for it to get like the feature set or whatever i pay 20 bucks a month for it because i would use it and my production team would use it for this podcast and things we do because we frequently have a guest and i would have taken interviews with you i would and i would have said hey what are the most interesting things he's been asked in other interviews right and and i would want to pull in podcasts etc so for me it's a it's a great paid product but how do you think about taking it from a laboratory experiment and productizing it what how does that work at google or how do you think about it i i mean genuinely it's not a cop-out to say we don't really know because this iteration of labs is is a new one right it's a new uh new labs um now run by a wonderful guy named josh woodward who was also instrumental in bringing me in um and we have kind of graduated up to a you know a public launch in the u.s um we're still built as an experiment although bard is still built as an experiment too i believe um and we're trying to figure out you know what's working what's not um what actually the path is if if people like it as much as we think people will like it particularly as we expand it we make it easier to bring in sources make it easier to discover sources all the things we talked about.

47:58I, you know, I don't know what becomes of it. It's in a nice spot, I think, where it does something different than what docs does. Yes. And, or slides does, you know, we're for the, you know, almost all of our features are like helping you think and understand. And there's almost no, like our formatting is like bold and italics. Like that's, you know, write a note, that's all you have, you know, enjoy it. Um, so it's not at all about creating the final product at all, but it is a place where you can synthesize across lots of different docs. And so I think it, you know, it compliments Google's existing offerings, whether it graduates up into some more elevated points.