Inside the AI memory machine

0:00When will AI finally make work easier? How about today? Say hello to Gemini Enterprise from Google Cloud, a simple, easy-to-use platform letting any business tap the best of Google AI. Retailers are already using AI agents to help customers reschedule deliveries all on their own. Bankers are automating millions of customer requests so they can focus on more personal service. And nurses are getting automated reports, freeing them up for patient care. It's a new way to work. Learn more about Gemini Enterprise at cloud.google.com.

0:39Welcome to The Vergecast, the flagship podcast of Total Recall. Both the concept and the movie, but only the first movie, not so much the con-feral one. Anyway, I'm your friend David Pierce, and this is the first episode in our latest mini-series, all about AI. We talk so much about AI these days, and so much of it is so abstract. AI is this big, messy concept, and it's easy to get lost in all the theories and consequences inside of it. Those things matter, of course, a lot. But we also want to try and get a bit of an on-the-ground view of how AI is actually being used and whether it's actually useful.

1:20Today, I want to talk about a use of AI that is suddenly everywhere. It goes by many names. Microsoft calls it Recall. With Recall, we're going to leverage the power of AI and the new system performance to make it possible to access virtually anything you have ever seen on your PC. Apple calls it personal intelligence. It understands who's in your photo library, so you can just ask photos to create a movie about Leo learning to fish. For Google, it's one of the most important features of Gemini. Do you remember where you saw my glasses? Yes, I do. Your glasses were on the desk near a red apple.

1:56There are also lots of startups, enterprise software companies, and everybody else you can think of working on something similar. Call this feature whatever you want, but the goal of them is all the same. To remember everything you need to remember and give it back to you at exactly the right moment and in exactly the right way. As an idea, I find this incredibly compelling. I think I find it more compelling than most, actually. See, I have this thing called aphantasia, which essentially means I don't see pictures in my head. Like, if I tell you right now to picture an apple, there's a wide range of things that you might be picturing.

2:32Maybe you see a super crisp, detailed apple with colors and leaves on the stem, and there's a whole surrounding universe around it. Or maybe you just see sort of a cartoony-looking apple, like the apple-iest apple you've ever seen. Maybe you just see the rough shape of an apple in your mind's eye. If you're like me, you don't see anything. Nothing. Blackness. I can tell you what an apple looks like. I know what an apple looks like, but I can't see it there. I honestly always thought that when people talked about visualizing stuff and their mind's eye, they didn't mean it literally. It breaks my brain still to this day.

3:11Every time I think about the fact that people can just conjure images in their brain and they see them. I don't get it. And people who see it don't get that I don't. Brains are weird. One thing that seems to be true about people with aphantasia, like me, is that we don't necessarily have particularly great visual memories. Like, I can't just conjure up a photo of my childhood home, you know? I remember lots of things. And to be clear, I don't think aphantasia is like a disability so much as it is just a different way your brain works. But I just don't have that kind of visual memory. So over the years, I've leaned pretty hard on technology to help me fix that.

3:47I take a lot of photos, and I love all the on-this-day and automatically-generated albums that I get on my phone and in Google Photos. I've tried to journal much more over the years because going back to words and pictures and videos does put me back in a place better than just closing my eyes and trying to remember something. And anyone who knows me will tell you that I am an obsessive note-taker and list-maker because it just helps me keep my world in order. It's how I remember things and go back to things. All of this is to say that a tool like the one these companies are describing is kind of my dream.

4:21Something that can remember every article I read, every message I receive, everything I need to do, person I meet, place I go, everything else that happens in my life, and then both store all of that for me to access and then actually present it to me when I need it. That would literally change my life. There are obviously huge, complex privacy and data security questions associated with how that all works, and I want to get to that. But this is one of the things AI is actually pretty good at. Just storing, sorting, and categorizing vast amounts of data. What if that data could be my life? I wanted to get some perspective on what it takes to actually pull this off, so I called up one of the people who's been working on it the longest.

5:03I'm Dan Soroker, the co-founder and CEO of Limitless. Here's Dan's story in a nutshell. He worked at Google out of school, then started a company called Optimizely, which he sold in 2020. And along the way, he started to lose his hearing. The way he described it to me, he had this really visceral experience the first time he tried on a hearing aid. He told me it was like gaining a superpower. And he started to look for other ways to give himself and other people technological superpowers like that. And he landed on memory. We as humans are terrible at remembering things. We forget things over time.

5:38We forget things immediately because we just weren't paying attention when we learned them. We remember things but misremember where they came from. We remember things selectively because we come to everything with specific biases. Some people have better memories than others, but nobody's memory is perfect. And Dan had this thought that maybe he could help fix that. It started with this idea, well, if there's a hearing aid for hearing and glasses for vision, what's the equivalent for memory? The first version of Limitless was actually an app called Rewind, which you'd install onto your Mac, and it would immediately begin to capture audio through your microphone and speakers, and it would also take almost constant screenshots of whatever you were doing on your screen.

6:15That basic technology is pretty simple, right? Capture a bunch of audio, take a lot of screenshots, and it gets you pretty far in terms of trying to figure out what you're doing on your computer. The question then is what to do with all of that. What does it actually mean to give someone a better memory. At the beginning, that was the big question for Rewind. It's good to think about it as an analogy. There are many things today that you do not remember, and you're very happy to have offloaded technology. A good example is phone numbers. I'm not sure the last time you had to type an actual phone number in your phone, pretty rare.

6:46Now you just open up your phone and tap the person's name. There was a period of time in my youth where I had to remember everyone's phone number, and that's the way I would call them. Similarly, with getting from point A to point B, there was a period of time in our life when we had to remember cross streets, and we had to remember how to navigate the world. Now we just tell the computer, this is where we want to go. And it gives us in time, real time, the most optimal route we can take. And like the memory of those things, that sort of, I'm using those as very basic examples. We don't miss at all.

7:12There are other things in our lives that are like that, that we don't even realize we're forgetting because our memory is flawed. You know, a good example is how I know a person or, you know, when you and I maybe reconnected in a few months, how old was your young one? And, you know, those details that actually help build connection and rapport, make us feel more human to one another. They may seem robotic to have your computer remember for you, but they also create this amazing connection that if you imagine, just imagine a world where if you had perfect memory, what you could do both in terms of connection, in terms of productivity, in the same way, imagine, could you imagine if you wore glasses today or contact lenses, could you imagine going a day without them?

7:47Like, why would you live your life blurry? And that same thing could be said of memory. Why would you live your life with a blurry memory? That's such an interesting philosophy question, right? Because I feel like the way you think about that as opposed to something like hearing aids is an interesting one because I think what you're saying is less kind of let's solve for an issue that you're having right and then kind of get something back to the level at which you had it or would like it to be and more saying like how do we limitless pill this right and it's like what if instead of using 10 % of your brain you use 100 % of your brain and that's such an interesting like what does it mean to be a person when you have all of that stuff it's like have you ever seen the the reaction people have to like dating spreadsheets when people have these really intense databases with all the people they're going out with.

8:32On the one hand, makes total sense. But on the other hand, I think it gives people this sort of like human ick in a way that I've never really been able to describe. This is such a technological question we're having to reckon with right now in so many ways that it's like, if all of this stuff is just available to me, how do I use it in a way that makes it still feel like me and feel human and feel honest and feel real? And I don't know, it's not even really a question. It just opens up these really complicated, like philosophical wanderings in a way that so much AI stuff does right now for me.

9:03Yeah, I often imagine how will the future think of us today. And one way to think about that question is to think about how we think about people 50 or 100 or 150 years ago. And that feeling you described, that ick of using computers to do a thing that you thought should just be, want to just go to a bar and meet somebody. That ick actually, in hindsight, seems silly. You know, it's like the same way that people had an ick toward cars. Well, I love my horse. And they had an Ictor telephones. You know, that feeling of newness, of novelty, of difference. People just as human beings are not wired well to respond to change.

9:35We are just not. There are some people who are very excited about new. They buy the newest gadget. I'm one of them. Like no matter what it is, no matter how bad it is, just that it's new is good enough for me. There's others for that would be the worst way to live life. They just want to, they're happy with the way things are. And the world is changing faster than they like. And it's changing faster than ever. and you know the way i think about it is five fifty a hundred years from now people look back at today and be shocked that we accepted the lives that we live that we would forget 90 of what happens after a week and that we thought that was okay sure yeah this this brilliant device we have of our mind that's so precious this incredible machine we just let it forget things and that's okay no big deal that's just life i think people will laugh at that you know my grandkids will ask me really daddy you just went through life forgetting most of it and i was okay with you I don't know that I buy that theory of the future completely for whatever it's worth, but it is a theory that I hear a lot in talking to AI people.

10:26And it's largely true over time that as we found more ways to augment our own capabilities, things have gotten better. I mean, look, what is a computer if not a way to just do math faster than writing it down on paper, right? Offloading things that we don't do well to technology that does do it well tends to be a pretty good outcome in a lot of ways. We have to take a really quick break, and then we're going to talk about what an app actually does when it tries to give you a better memory. We'll be right back.

10:58Support for this show comes from LinkedIn. When you're a small business owner, your business is on your mind 24-7. So when you're hiring, you need a partner that works just as hard as you do. That hiring partner is LinkedIn Jobs. When you clock out, LinkedIn clocks in. LinkedIn makes it easy to post your job for free, share it with your network, and get qualified candidates that you can manage all in one place. LinkedIn's new features can help you write job descriptions and then quickly get your job in front of the right people with deep candidate insights. Either post your job for free or pay to promote.

11:33Promoted jobs get three times more qualified applicants. At the end of the day, the most important thing to your small business is the quality of candidates. And with LinkedIn, you can feel confident that you're getting the best. Based on LinkedIn data, 72 % of SMBs using LinkedIn say that it helps them find high quality candidates. Find out why more than two and a half million small businesses use LinkedIn for hiring today. Find your next great hire on LinkedIn. Post your job for free at linkedin.com slash track. That's linkedin.com slash track to post your job for free. Terms and conditions apply.

12:11All right, we're back. So when Dan starts really working on his memory superpower app in earnest, it's 2020, early COVID times, and everyone's suddenly working from home and on Zoom calls all day. This is pre-ChatGPT, by the way, and before everybody decided that AI was going to immediately change everything forever. Zoom, by the way, was like the hottest company. Do you remember these times? It was weird times. Dan has this big, huge idea about memory superpowers, but he ends up deciding to start his product on a much narrower path. We started with a meeting bot. Everyone was doing Zoom meetings.

12:46It was a bot that would join your meeting, capture, record it, transcribe it, and make that shareable. When we started, that was novel. And then every week, a new startup came and did it. And now there's dozens of companies doing that. You just described my whole news writing life of 2020 is that startup over and over again. Yeah, exactly. So that became very crowded. And what we realized from our users is, and this is where any of our best ideas came from the seed in a problem that our users described, they loved the idea of comprehensiveness. They really wanted to capture more of their lives, not just these Zoom meetings they're having.

13:16And that led us to ideate around, how do you capture more than just meetings? And around that time, the enabling technology of Apple Silicon came around. Apple M1 came, a chip that allowed you to do a lot more things locally, to offload a lot of the things that would otherwise have to be done in the cloud. And that's when we moved from Scribe, which is MeetingBot, to Rewind, the Mac app. That was, again, long before ChatGPT, GPT-3.0, 3.35, which is the model that powered it. And I actually say we got more lucky than good there, where the data that we were actually already capturing with Rewind just really lended itself well to RAG, retrieval augmented generation.

13:50Like the ability to use that data, you know, to ask a model like GPT-3.5 or 4.0 now, if you asked it to draft me an email to Sam Altman, if you just ask it today without any context, it does an okay job. It knows who Sam is. It certainly doesn't know who I am. It doesn't know our relationship. But if you augment it with the context of our relationship, the fact that he invested four years ago and that, you know, we've been working tirelessly in this migration revolution to Limitless, all of that context, you provide a large language model, it drafts a perfect email. In fact, an email I sent to Sam that an AI could have drafted worked great and we met and we connected.

14:22So like these things that, you know, otherwise you have a blank piece of paper can easily be made easier by having AI augmented with your context of your past. What would you have done had this not happened? Like in this sort of parallel universe where like chat GPT doesn't happen and we don't get this kind of incredible run of stuff we've been on the last few years. It doesn't sound like you were counting on that. You weren't saying like we're going to bet on this foundational technology to be the thing. What is parallel universe, Dan, building to make that stuff work? Yeah, I would have been doing more of the same.

14:51I mean, I would have. It was already valuable enough to do search over the things you've seen said or heard in the past. The large links model just makes that search more useful and more actionable and more. It sort of takes that, you know, the task you would have had to do, like, let me search through all my emails with Sam. Now we figure, OK, when did we meet? Who introduced us? Instead of doing that manual task, now you do it in an automatic way. And so it just made me analogy draws like before we were on this evolution from horse and buggies to self-driving cars. Our evolution before ChatGPT was maybe we added a car with manual transmission way better than a horse and buggy.

15:22But what ChatGPT and the underlying API that we use enabled us to do is go to automatic transmission. So we're not quite at self-driving cars where the autonomous AI simulations of your mind doing things for you. But we are able to actually save you time and give you a lot of value through this sort of evolution. So that's what we would have been doing. And by the way, now our focus is basically just banking on the models getting better. Like everything we do is under the premise of just that the models are going to get better. So like just collect all the data in the best possible way to ride that wave.

15:50So our mindset has definitely shifted. We're not sort of doggedly just pursuing what we've done before. Now we realize, OK, there's this amazing wave of change or these models just going to get better and better inexorably. Why not just ride that wave and build a product that just gets better on its own, gets cheaper, gets better on its own as the models get better? The car analogy there is a little messy, but it's an interesting way to think about where we are with AI. We spent two decades or so with products like Google and Facebook, which built very smart and sophisticated systems for looking at a huge amount of stuff and ranking it a million different ways.

16:22Those systems are by and large very good. But the promise of AI is that it can take all that stuff and actually come to understand it. Not just find you the thing you're looking for by putting it at the top of the list, but by finding the perfect thing that you're not looking for, or by using everything that you already know to help you do the next thing. We don't need AI to do Google searches. We really don't. Even Google is currently showing us how much we don't need AI as it tries to put more AI in to Google search. But if all the AI boosters are right, we can use AI to not just find things, but build new things on top of them.

16:58In a personal context, that brings up the central problem with all of these AI systems. In order for an app like Rewind to know everything about you, it needs to know everything about you. Do you want your computer to store and save everything you do, everything you click on, all the words you type, all the TikToks you scroll through, all the pictures you look at, every single thing that you do? while you're at your computer? Forget about the data security risks behind that for a second. Just like as a human, how does it feel to know that all of that stuff is being recorded and stored in perpetuity?

17:32And how useful does your computer need to become? What does it need to do with that data in order for it to be worth the trade? Dan has been thinking about this for a long time, and he calls it the personalized AI privacy paradox. And it goes something like this. In order to build a more useful personalized AI, you want more context recorded. But that raises more privacy concerns, which requires more need for data protection, which makes it harder to build a more personalized AI. So the desire of a personalized AI, part of it is this, okay, we want to collect more things because that context is going to be useful.

18:06But inexorably, you go down this path that at some point that's going to make the original goal harder to do. And you can see this paradox playing out in the world in two very different ways. With Microsoft, they were pretty cavalier on privacy when they launched Windows Recall. Lots of things to say on that, but let's say they launched a product called Windows Recall, which looked very familiar. But they took a very, very cavalier approach to privacy, and that really hurt them. On the flip side, Apple intelligence seems to respect privacy, but it actually is limiting what Apple can actually do in terms of usefulness.

18:34So they're kind of on both sides of this paradox. We've actually made many mistakes in this space. I think the most recent evolution of our thinking has really set us up well for the future. But it's not obvious. It's not straightforward. It's one of those things you have to kind of, you're like tight roping between landmines. You know, you're trying to find the right path that respects privacy and doesn't make it a choice between privacy and convenience. At the same time, it makes the product useful enough because you're able to use the data in a way to offer personalized AI. I agree with that.

19:00But then you have to put that in front of users, right? And I think even there's something about the idea of just like, here is an app that shows a timeline of every web page I've ever been to feels instinctively weird to some people. Right. And I think it's been very funny watching a lot of this because like, yeah, of course, your browser knows all the web pages you've been to. Like that's your it's your web. Like, yes, that's how it works. But I think a lot of what has been happening in these recent months is people are slowly starting to understand kind of how much awareness their technology has of what they're doing with it in a way that everybody probably should have had before, but didn't.

19:37And I feel like what you did, especially with Rewind in the early days, is like really speed that process up. You're like this app knows everything and it's actually its job to know everything. And I wonder if to some extent it helps because it's like a thing you have to download. So by definition, you're going to get people who are more comfortable with it rather than like building it into the operating system. But you have to do this thing, I would think, right away where you're like, OK, this is asking a lot. It's going to know a lot about you. I have to sort of immediately telegraph to you why it's worth it.

20:06And I feel like that's a that's a pretty big hurdle to clear right away. Yeah, I think we certainly paved the way there, but I wouldn't say we didn't make any mistakes. I think showing a timeline of everything you've seen is interesting. It's a cool party trick. Like it helps create, you know, this magical moment of like, wow, I didn't realize I could. And then the people love that. And that's partly why we named the first product in this space Rewind. But if I really had to be honest, like people don't care about technology. They just care about their problems being solved. The technology is the means to the end.

20:36And this was too much of a cool, let's see what the technology can do part of the product unless here's the problem we're trying to solve for you. So I do think the right user experience around personalized AI is to be very opinionated on the use cases. What are the problems in your life every day that we're trying to solve for you? How do we give you time back? How do we make it so that you're in time for dinner with your young kids at night? Those are the kinds of problems people care about. The technology and how sophisticated it is, they could care less. Like that data, the architecture, that is important to us because it's how we build the product.

21:08But to a user, that's just implementation detail. So I do think Limitless does a much better job of this. This is why we've kind of evolved and even partly renamed the product because the core experience of rewinding time isn't the core thing people want. The job to be done is give me back more time, make me more productive, take things off my plate. What are things I do every day that, frankly, I just don't need to do? And a machine can do a better job of be more reliable and just give me back more time so I can do the things that I'm uniquely well-suited to do. So those are the kinds of things and the kinds of experiences that I think will ultimately win in this world of personalized AI.

21:40What are some of those things that you identified early on as solutions to those problems that you're describing? Yeah, I mean, big one is a blank piece of paper. Very often as knowledge workers, you start with a blank piece of paper. Maybe it's an article you're writing or an email you're sending or even a simple text message you're sending to somebody. Writer's block is a big version of this problem, but it's this idea that starting from zero is much harder. And part of what a machine can do uniquely well is capture the context you might want when you're starting from zero. A good example is drafting emails.

22:09I gave an example of sort of drafting emails, Sam Altman. Starting from scratch is a much – why spend all that time and energy when a machine can surface to you perfectly the thing you might need and want? The way you can think about it is autocomplete for your life. Why so many times you have to start with an empty line where a machine can provide an option? It doesn't do it autonomously. It just writes a draft, something you can edit and tweak and change and delete. That's like a perfect win between, like I said before, horse and buggy and self-driving cars. There's going to be a day when AI can do things autonomously.