Inflection AI CEO Mustafa Suleyman on building modern AI, DeepMind origins, and much more | E1794

0:00apparently if i fancy getting married anytime soon you're available for that too right so Currently the world's greatest officiant is available. If you can find a woman who will marry you, Mustafa. But you've got a startup. Oh, you already got that accomplished? No, no, I'm struggling with that. I'm very much single. So if you want to marry me to my startup inflection. You're not single. You are married to your startup. You raise a billion dollars. I can tell you who you're married to for the next 10 years. Absolutely. Inflection AI and your 40 people over there. This Week in Startups is brought to you by open phone brings your team's business calls texts and contacts into one delightful app that works anywhere get 20 off your first six months at openphone.com slash twist crowdbotics great ideas can change the world and crowdbotics is the fastest way to turn those ideas into code get a free scoping session for your next big app idea at crowdbotics.com slash twist and carta now lets you launch and administer spvs for your syndicate share your knowledge capital and network to launch your syndicate spvs through carta get 10 % off your first spv at carta.com slash twist with promo code twist all right we got a big treat for you today on this week in stardust mustafa soleman is here he's with inflection ai but uh very famous for having been the co-founder of deep mind welcome to the program mustafa Great to be here.

1:30Thank you, JC. Thanks for having me. Of course, of course. You know, I wanted to start with the origins of DeepMind because it seems like so much of what we're seeing in AI stands on the shoulders of that organization. And I don't think most people know the history of it. I happen to know a little bit of the history of it because I remember when Peter Thiel and Elon, I think, were two of the early funders of it and were talking about it. And we met, I think, at a couple of different industry events over time. Tell me, what was the origin of DeepMind? And then how did it, you know, originate and start to tackle AI, general AI, vertical AI, all these different things that are coming to fruition?

2:14And I guess that was 2010, right? 2011? It was 2010 that we started the company. Yeah, exactly. Which seems kind of insane. like almost 15 years ago and it's just quite surreal to see because in the last sort of what is it nine to 12 months it feels like the kind of large language model revolution has come out of nowhere um and exploded onto the scene but in fact it has been the kind of steady march of many many years and a huge amount of failure and a lot of risk and a lot of persistence that I often think gets slightly neglected in the story of the perfect explosion of a new technology. In fact, for most of the last decade, we didn't have language models.

3:00I mean, the Transformer was really only popularized in 2017. I mean, people often say that it was invented then. It was certainly not invented then. It was invented a good 15 or 20 years earlier by Osho Benjo. And then many other people developed the ideas. But it was really only four or five years ago that the idea started to get traction again. And then it wasn't until GPT-3 that people started to get a glimpse of what it looked like at scale instead of just in a test environment. So, yeah, it's been a crazy journey. How do we start the company? So 2010, I was actually playing poker with Demis Hassabis, who is my longtime friend since we were quite a bit younger.

3:48In London, I assume, at those high-rate casinos? That's right. It was at the Victoria Casino in London, which is on Edgeware Road. Not the biggest game in the world. I seem to remember it was probably a 250 pound tournament, only 120 people, but you know, um, so we would play at these things regularly. Um, both of us were very passionate about poker. I was playing, I was, I was, I was one of these people that was doing like eight table poker stars. Yeah. Back in the day. Um, my friends were doing 16 table, but I didn't have the actions per minute speed to be able to manage that. you're you're on a clock yeah it's it's not easy to multi-table uh although it's something about multi-tabling becomes like a flow experience and you start to see patterns right because you're playing so fast that you have no choice but to kind of play instinct right um and now it seems like gto and all these theories are people are able to really deploy it very quickly i hate online poker i like in person because i think the only edge i have is my ability to read people which is such a critical part of the game and it's so hard for me to read people online it's also the fun part of the game right like pushing people off pots and teasing people for their losses i mean that's that's the fun part like so yeah but i mean getting through a lot of hands is also a very great way to practice i mean because you end up developing heuristics and so you just see that's the problem is it's such a high variance game in your career if you only ever play live you never get to see the volume which gives you the range of experiences so the good thing about really having a short stint of abusing online poker is that you just get to see depth and breadth which which is which is cool but you can pick up bad habits because it can make you too cautious ah interesting uh i haven't heard that before so you don't is that is the reason you get too cautious is because everybody's reading each other's like statistics and you're just like, I'm going to be too easy to read here.

5:53I can't make a non-traditional play. I'm going to get caught. Yeah, because everybody sees so much more volume, then they play in a much more predictable and structured way. So you learn to predict everybody else's moves. And also, they end up being, because they see more volume, they are more deliberate with their hands and more cautious. Whereas in a home game, you may only see a couple hundred hands, even in a six to eight hour game, right? And so your range is clearly much lower. you're playing cards that you would otherwise leave behind because you're seeing more throughput online right so you know the classic is the knit you know we used to call them the knits when they would come to the the live tables and they'd like clearly just playing this robotic game and driving themselves nuts because they weren't seeing enough volume just pretty funny yeah it's it's it's it's such a fascinating game and playing live in a casino you get to see like a real broad spectrum of humanity i was just talking to somebody about my friend sky date and i used to play at hollywood park and commerce uh in la and we would play at the lowest tables and at one point i was trying to figure out how to read people better and i came up with the idea of jedi poker uh where i would pull my cards up and put my thumb on it but i'd make a bit of a show of looking at my cards but i would have them covered so i didn't know what cards i had mustafa that's the best way i would only play the person and i'd be like this person seems very strong this person seems pretty scared let me see i can get this person off the hand let me do it and then i get to the river and i would literally if somebody called me down i would turn over my cards and be embarrassed like oh i have a set i didn't know it or i had bottom pair and they'd be like how would you bet like that it makes no sense and making no sense is part of poker because you have to break the ability for people to be able to read you a hundred percent this is my one of my favorite ways of doing it the other way i like to do it is to represent a hand off the flop that i don't have assuming that it is the opposite hand or a better hand and the whatever i place that person on so you know that that's actually a very good way of doing it because then you bet consistently across the three you know uh steps streets but you um but you know so you're not being ridiculous and wacky but you're telling a story you're representing a narrative yourself that you have 10 jack and when the board comes down you know nine king queen you're like i've i'm playing 10 jack and i'm gonna play it like 10 jack would play this yeah you just got to make sure you know how to lay down if your opponent actually ends up having the hand that you're trying to represent that can get pretty sticky but yeah are you still using your personal phone number for your startup it's 2023 it's time to stop.

8:39It is a huge mistake that founders make. Why? You're just getting started with your company and you don't think about phone numbers as being an important part of the IP collection of your startup. With open phone, you can totally solve this problem. They've rethought everything about a modern business phone and how it should work. It's super easy. You just download the app on your phone or your desktop and you pick a number and you're done. And you do it for just such a low price. It's so affordable. And think about it. If you have your sales team using their personal phone numbers, a salesperson leaves and goes to a competitor, you don't have any insight into what phone calls occurred, what people's phone numbers are, that's your company's database.

9:19And if you allow the sales team to run them up, or the customer support team, it's just unprofessional, be professional, use open phone. And we use it for things like event communication. So we get one phone number, but it can go to multiple people like a round robin thing, we have a shared phone number. Do that for customer support. And Open Phone is rated number one on G2 for customer satisfaction. And I trust G2's ratings. Open Phone, it's ready. It's affordable. Starts at just 13 bucks a month. But Twist listeners can get 20 % off any plan for the first six months at openphone.com slash twist.

9:52And if you have existing numbers with another service, no problem. Easy peasy lemon squeezy. Open Phone will port them over at no cost. Head to openphone.com slash twist to start your free trial and get 20 % off. so you're playing cards and uh you get bounced out of this tournament and you're you're sitting there uh doing your post bounce uh or did you make it to the final table and you're just early you nailed it so now you're trying to explain your bad luck and how bad everybody else is to each other right we've we've gone over the whinging about our bad beats right that took up the first half an hour running through our knockout hands and we're sitting there eating chocolate cake and vanilla ice cream and diet cokes because obviously we're super cool and we you know we're not getting pissed we're talking about the future of the world and you know both of us have always been interested in like how do we impact the world how you know what does the future look like we've both been very very long-term thinkers and just instinctively that is just one of our kind of gifts i think and i i was particularly interested in you know how you do good in the world and how you know politics shapes our future and stuff like that we were both talking about robotics and is now the time for robots to come on and automate everything.

11:09And I think we both agreed that actually that was way further away than people realized. But the thing that was likely to be more prescient is teaching machines to learn their own representations of what is valuable in a space. Surely a machine could learn to play poker, a machine could learn a set of heuristics and then reproduce those patterns. And at the time, Demis was just finishing up his PhD and postdoctoral work in neuroscience at UCL at the computational neuroscience unit. And so he invited me to join the Lunch and Learns, which I did for almost six months, I think, pretty much every day.

11:47Went down to, you know, basically smuggled in the back door of the Gatsby Computational Neuroscience Unit and just listened to the Lunch and Learns. And that's where we met Shane Legg, our third co-founder. And then we all went for lunch. What is this lunch and learn? I mean, there's lunch and then somebody speaks and you learn? Yeah, it's like a brown bag lunch, you know, like where it'll be like at the lab. So there's 40 or 50 people at the lab and people invite different people and so on. And so there'll be speakers or there'll be postdocs or every lunch, basically, someone gives a talk about their work and takes questions.

12:19And it's a bit of a bear pit. I mean, you know, they don't take prisoners. If you're not on your toes, then you get some pretty rough questions. And it was just an amazing way to learn and be thrown in the deep end and really experience it firsthand. I was only 24 at the time. So then basically a few months after that, Shane Legg got invited to the Singularity Summit in 2010 to be a speaker because he was on the Less Wrong forums back in the day. and was a bit of a transhumanist, to be honest with you, at that time. And then we decided to go because Peter was one of the sponsors. I think he was the main sponsor of the summit.

13:01And then we got invited to the drinks afterwards, and we used that as an opportunity to pitch Peter on AGI. He was the only person in the Valley, to his credit, talking about AGI or even AI in any form. To everybody else, AI was a weird, taboo word, and everyone was sort of talking about machine learning, but even not really. It was mostly in the labs, in the academic labs that people talk about machine learning. Yeah, and then we went to his office in the big park, the Presidio. Yeah, the Presidium. Presidio, yeah. We went there to the Founders Fund office and yeah, he made a decision on the spot.

13:39It was pretty easy. I think he gave us like$2 million. Yeah,$10 million valuation. Not even, dude. It was like half that. We were like randoms from London. I mean, he joked that it might as well be Somalia. That was his view. It was literally what he said. You might as well be investing in Somalia. I was like, London's a serious place. But apparently not to Peter. Well, you've got to also put in context, he had done the Facebook investments, probably feeling pretty good about himself. That was going well. and uh you know five to eight million dollars was what a seed round would evaluation would be um and ai at the time to be honest as you said nobody thought there was a commercial application or or that it was going to work right like that was kind of the big question is this actually going to come up with an answer that is going to have some application in the real world because you had deep blue right we had kasparov got beat and so narrow ai had proven itself but ibm had spent hundreds of millions of dollars and they had no product i mean that was the playing field right it was like this is a money pit right and of course that was a decade before us as well you know so that that that had proven to not have serious commercial applications so that was that That was actually a kind of non-goal in our pitching is to not bring up.

15:05Yeah, don't bring up Deep Blue. Don't bring it up because it was like a cool research thing, but never quite had the impact that we hoped. And yeah, for the first two or three years, it was very tough going because deep learning just didn't seem to be catching on. And then all of a sudden, we had the CAT classification paper from Alex Krzyzewski, AlexNet in 2012. And then in 2013, we had the Atari game player, DQN, which we published. And that was really the thing that changed everything for us. Because, you know, Larry Page had seen the demo and just emailed us cold page at Google.com and was like, you know, you guys should come and come and be part of us.

15:49I've spent my entire career building the infrastructure to enable a company like you guys to come and work on on AGI. why stepping back what was the pitch to peter we're going to build reinforcement learning we don't know if there's an application it's a science project your two million is going to be gone in three years like was there any path to commercialization that you pitched him on or was it let's see what we can do in the lab there was yeah so i mean we actually didn't pitch him on reinforcement learning because at that point that was really early we pitched him on deep learning and um what we were working on was a visual image search uh tool for uh fashion and furniture and and clothing and so on and we actually i actually i held the first pattern um for deep learning in this area which actually takes the shape and the texture and the color of one item of clothing like ideally a more affordable high street version and then uses that to find the more expensive uh you know equivalent that you could then you know you know go on you know find find a comparison for and um you know that that was a big moment actually because it was it was the beginnings of you know the generative ai movement i mean you know it's now called gen ai but it was never called that at the time it was really just deep learning classification yeah at that time forget about generating something you were trying to identify something this is a hot dog this is a dog these are two different things and that hadn't that framework hadn't actually happened yet what google was doing at the time would they put two or three low wage people on a group of images and they would say there is it you know describe five tags for this image and then whichever three or four came you know in common with two different people that was what the image was about right that was the state of the google index at the time right spot on and they would have them draw bounding boxes around certain parts of the image so this area of the image contains a penguin this one contains an iceberg and you know uh it turned out that was exactly the kind of thing that this hierarchical neural network representation was pretty good at doing like it would certainly it would essentially cluster together pixels which were correlated around a particular region and And then where there was a sharp distinction, like an edge or a line or a break in a cluster, then that would end up being a sub-representation.

18:21And then the next layer would absorb that sub-representation and increasingly build more and more symbolically representative ideas. Like it would go from basically a tiny little area of the iris to a wider eye to an eyebrow to the side of the face to the full face to the background. And you could kind of think of that as a way of understanding how the hierarchical neural network representation was formed. And obviously, now that we had made so much progress over the last 10 years on the classification side, you then use those classifications to generate novel predictions. And that's basically what image generation is doing is saying, given this sentence, find the sort of optimal representation of all the competing points in this big space that best represents this long sentence as a new image.

19:13And that's the transform a model that we hear about in that 2017 paper from Google. Yeah, exactly. Yeah, yeah. Yeah, that's deep learning. But then there's lots of other generative AI components that were, you know, pushing it that direction. But they did it. They made it work first for the language side of things. That was really the big deal. So you get a couple of years into this. You figured a couple of things out and you start getting into reinforcement learning. So and that's when Larry Page was Larry on the board or just Elon on the board at that time and Peter. No. So we had first Peter Invest, then Elon, then we were the third check out of the first fund of Mark Stad's Dragoneer in 2012, I think it was.

20:06It was such a great time period to be an investor because only lunatics were starting companies after the great financial crisis. it was like this five-year period where if you started a company you you had no choice because you were a lunatic who had to start that company because everything in the world was telling you don't start a company right it's going to be pain and suffering so you raised this money what was the first project that you guys started to work on how did you pick it and then you know what clicked because there were i remember alpha go was one and then there was this clock that became sentient there were just all these like little projects that we would hear about inside of deep mind but deep mind kind of kept a lot close to the vest i think we i mean we operated in stealth for most of our entire period and we actually didn't even announce our investors i mean there was a bunch of other like we we had selena chow as another investor from horizons lee cushing's fund and you know there was a we had a very good group of people we're lucky i think we raised 45 million dollars in the end so we every each year we went back i think we raised like we raised two or three and then 10 and then 30 um and what did you show each time to keep people investing in the vision during a time when people didn't believe in the vision yeah most people didn't yeah yeah i mean so we showed in the second time that we raised we showed Flatland, which was our little agent-based environment, like a 2D grid world, where the model had kind of learned a way to navigate through the environment using purely the pixels.

21:41And we then said, okay, for our next milestone, we're going to basically teach the model to learn um arbitrary games of atari and in the end we we played 56 games um at you know which is pretty incredible this is 24 frames per second and it's learning to basically correlate actions where it basically go up down left right or shoot um the original atari 2600 controller yeah exactly which had five actions exactly exactly and and you know so it's basically got to figure out which of those actions, it's randomly kind of moving them around at the beginning. And then it stumbles on a rewarding moment.

22:26It luckily gets some score. And then it realizes, okay, that's a useful thing to do. Next time I see the ball bouncing towards me in that position, I'll move the paddle left or right. And it's just kind of incredible that purely through self-play and reinforcement learning, just very simple heuristic exploration and then exploit the strategy that turns out to be useful for generating score and suddenly you can learn to play all the games to basically superhuman performance i mean that was mind-blowing to me and that was all done in an atari 2600 emulator obviously you're not taking a physical joystick and putting a robot on it it's able to run very quickly in the cloud right you figured out a way to accelerate it so that it could just be playing whatever pong or tank whatever those early games or adventure play them you know millions of them right how many how many runs did it have to do to be to perfect them did you did you track that like how many how many how many quarters until you perfect the game and you get the high score i mean it's interesting that you mentioned cloud right because this was 2012 13 so there wasn't really any cloud to speak of we actually ran it on prem uh we had our own little cluster in the office and it used to train atari dqn we it used um two petaflops of computation so so a flop is a floating point operation this is a unit of computation it's like one calculation think of it and obviously peta is a million billion so it's two million billion calculations to train the entire model over the course of about two weeks um so put that into perspective like and then obviously at the time that was you know one of the largest i mean we don't know for sure but there weren't any other big training runs of those kinds of things at that time so it's fair to say it was probably the largest um that was a decade ago and you know you roll forward the models that we train today at inflection uh and you know the other frontier model companies use 10 billion petaflops wow 10 billion million billion operations which is insane it's you a human brain cannot even conceive of what that is uh it's kind of like when we start talking about there's a billion suns in our galaxy and right that's and there's billions of galaxies the human mind is not designed to even comprehend millions of billions of millions billions of millions of billions it's just not even possible all right we all know the one thing that separates great startups from the good ones is product velocity what does it mean product velocity fancy term right here you got your product and your velocity speed the speed in which your product improves so can you ship updates?

25:14Can you release new features? Can you do bug fixes? Can you iterate on the interface? Can you solve problems for your customers? And can you do it quickly? Because you're not alone. You have competitors and your customers have choices. They may solve their problems by writing their own custom code, or they might use your solution. This is what startups are about. How fast can you get that product velocity going? And so, how do you supercharge it? Everybody says, okay, yeah, we want to go faster, but you got to go faster intelligently. And Crowdbotics is going to help you do that. They're your CTO as a service.

25:47Basically, they provide you with the most optimal architecture to get your product to market as fast as possible. You'll have access to an on-demand product manager and developer talent, and they will help get your app into production 10 times faster than conventional development. Crowdbotics can work with your in-house dev team, or you can just have them work independently. And you own all the IP. You own all the source code. Let the folks at Crowdbotics supercharge your product velocity today. No more waiting. Get a free build plan at crowdbotics.com slash twist. That's a$4.99 value just for the twist listeners.

26:17You get that for free. That's C-R-O-W-D-B-O-T-I-C-S dot com slash twist for a free build plan. The hardware did start to catch up here and hardware seems to have been part of the enabling here. Maybe you could talk a little bit about what the infrastructure looked like at that time, the hardware footprint versus what we see today and what you're doing at inflection and the hardware footprint it's a great point i mean it's it's really the hardware revolution rather than the ai revolution i mean the it's funny because people fixate on the algorithms um obviously the algorithms are critical but they they really have not evolved at the exponential rate that computing has evolved at right so those 10 billion million billion petaflops i described that's a that that is the equivalent of one order of magnitude so 10x increase in the total amount of compute used for the cutting edge models every year for 10 years 10 to the power of 10 i mean it's insane um it's truly insane so so yeah that is basically about hardware and i that's why i think actually this revolution is has been easier to predict than I think people realize.

27:40I mean, this trajectory has been continuing for a long time, and we can look out at what the next three, four, five doublings look like. Sorry, three or four, five, 10Xs look like. They're not doublings anymore like Moore's Law. They're orders of magnitude increasing compute, and that's a very predictable trajectory. I mean, obviously, it's unclear exactly what capabilities emerged from that, but you can certainly predict what we're going to be able to build. Steve Jervison has a lot of charts on this where he's been tracking. I don't know if you've seen Steve's charts on just, you know, the amount of computing power and, you know, the sort of tipping point is somewhat predictable.

28:22And now we've got heat and power friction, I guess, is the limit right now or how much we can connect these supercomputers together. What's the gating factor now? Yeah, it's a good point. that is going to become the constraint. So the A100 uses 700 watt per chip. The H100 is twice that, like 1 ,200 watt. Wow. So the next per chip, right? So obviously you have eight of these on a node, then the chassis and the node itself has some additional power constraints. So they're actually, it's a different data center design to what it was two or three years ago where you know there's actually spaces in between racks they're not like completely stacked up they have to be like really large gaps in between and i and some of the designs i've seen for for new cooling systems are that there will actually be fans in between the node layers um so and all fiber optics all glass photonic computing to to transfer data from one to the other because the amount of data being moved now can't be moved over copper it can't be moved over ethernet cables it's just too much right being moved around for sure for sure all of it is is a fiber optic cable it's actually called infiniband the um melanox um nvidia cabling and that that's like 900 gigabyte a second um which is pretty nuts for you know direct chip to chip connections um you know so it is really driven by all the hardware innovations and those hardware innovations are very predictable because you know they're they're actually laid out three years in advance uh yeah because they're planning on building they're building those schematics and getting the fabs and the factories ready to actually build them right um so there's starts to be a little controversy inside of deep mind i guess at a certain point uh larry page is like we need this team inside of google maybe peter tl elon wants you to stay independent Maybe you could explain that moment in time and the decision making there.

30:32Yeah, I mean, I think this was way back in 2014 that we were acquired. And, you know, I think that Elon and Peter, all of our investors, you know, wanted us to stay independent. And I think that the challenging decision for us was just the scale of investment that we could see that would be required going forward. I mean, we'd raised$40 million and we could see a path to spending$500 million in three to five years. And in fact, that's what we ended up doing exactly that. DeepMind now has 1 ,200, 1 ,300 people and spends over a billion dollars on compute a year. So, I mean, that's public information.

31:19So, you know, it's pretty remarkable, the trajectory. And so one of the things that we were focused on, you know, Larry made us an incredible offer to be able to do that. We were acquired for$650 million, free revenue, obviously. Yeah, it's a pretty great deal. It was a pretty good deal. Especially at the time. I mean, the world has changed dramatically in the last decade. But at the time, people were shaking their heads like, what did they buy? i mean in fact the conversation was i think you had maybe a hundred people at the time less yeah yeah yeah exactly the conversation was is larry lost his mind he just paid 10 million dollars per engineer and then that became well engineers in silicon valley are worth 10 million each it's like well these are different types of engineers you hired a very elite group of people maybe you talk about the recruiting of bringing together the deep mind team at the time because it was a lot of phds a lot of people who had some you had a pretty deep bench there yeah we were extremely focused on hiring the best PhDs and postdocs actually and I've carried that through to how I hire at Inflection I mean you know you talent is the differentiator at the end of the day I mean you could be first to get access to compute you can have the most amount of capital but selecting a very very high quality team is really the only thing that makes the real difference and that means you have to be very deliberate about who you don't hire um you know it was actually amazing at that time how many people who were fundamental to the deep learning revolution we had around us right so you know um jeff hinton was uh one of our consultants uh for two years before he set up his company that he then sold to google so it was ilia satskiva the chief scientist of um open ai now um voychek was an intern at deep mind who was one of the co-founders of uh open ai it's gonna be like the paypal mafia it's gonna be the deep mind it's already turned out to be the deep mind mafia basically you got a whole group of alumni who are just creating the future here yeah was it a looking back on it was it a mistake to sell you regret selling to google should you have taken elon's advice and stayed independent or i mean about it elon was certainly keen for us to come and be do the tesla thing be part of his ecosystem yeah but you know i'll be honest i was a bit i mean back then you know he's an incredible person but But, I mean, it was a very uncertain bet in 2014.

33:45Tesla 2014 would be the definition of uncertain. I mean, Model 3 almost killed him. Almost killed the company. I mean, that company's had a near-death experience with each launch of a product. I mean, you want to talk about hardware plus software and manufacturing at scale and building a public brand. I mean, the degree of difficulty is absurd. Inside of Google, to the extent you can talk about it. uh you guys worked on a lot of theoretical things but you also worked on a lot of practical stuff what were the big wins inside of google that you can talk about that deep mind participated in yeah i mean we uh deployed deep mind technologies on all of the main products other than search actually and youtube um so i think we did seven pas in the end on everything from um data centers to healthcare to play store to android battery optimization to android operating system i mean we we reduced the amount of energy needed to call the google data center fleet by 30 percent um that was a three-year collaboration it's a huge project we made the google wind turbines 20 percent more efficient which google has the largest winter the largest wind turbine farm in the world which is pretty crazy um yeah we designed the activity classification uh algorithms for uh all the wearable devices that would basically tell whether you're sleeping or running the two biggest franchises they wouldn't let you touch search they wouldn't let you touch youtube why would they you got this incredible thousand folks and you don't let them touch the two biggest franchises why well we politics no i mean we tried and we actually tried youtube in 2015 and we failed it was too early and it was it was just super hard we were trying to optimize watch next time actually yeah um and we were trying to use reinforcement learning for it and it was just too it was too early we we didn't succeed um search is a different story i mean search is just so difficult to ship anything and they're super conservative they also they like the fact that all of the rules are very transparent so they can see exactly why a page is being recommended and really have much more transparency on the algorithm which is very understandable so in fact there were some you know deployments of deep learning systems which ended up causing regressions over time because of drift um you know over over a six-month period and in other words quality would go down well it would go up initially at the beginning and then come down and then come down exactly why why does that why does that drift happen people are talking about that with chat gpt4 that results have deprecated well i i didn't understand why that would occur is it's garbage in garbage out kind of situation what's what's happening well and something like that happens i think there's slightly different problems i think with the chat gpt thing it's probably that they basically serve their best model which is expensive to serve right because the biggest and best and uses the most number of gpus and then once people are coming back frequently they'll use they'll serve a smaller model which is cheaper it'd be a less well trained model quality is basically as as always the case quality is cost right so we can serve a cheaper model for uh you know quicker um but it won't be as good so that that's probably what's going on i think ah yeah i've never heard that theory but that would track and make sense and it's more people use it they they may have no choice but to give everybody a little bit of an easier model to use or a more basic model because they don't they don't have a choice when the other variable would be speed so if you want it really fast then you have to get a smaller model or you have to use more chips to serve a super large model so you you can't have all three and so if you want a super high quality one you could have it really slow and cheap but that would be really slow 20 seconds or something for a response listen if you're in the tech industry you know about carta carta is the leading venture capital and equity management platform and they have huge news to share here on this week in startups carta now lets you syndicate an spv you know what an spv is a special purpose vehicle so you create an spv on carta why would you do that hey listen you're an angel investor and you're putting 25k in a company like i did with calm.com.

38:14But you got about 20 friends who also want to put in five or 10k. Now you put them all into an SPV, you tell the team over at calm.com or whatever company you're investing in, it's going to be one line item, I'll sign for all 25 of those angels. And they say, Oh, great. Can I put 10 other angels in your SPV? And then you hey, if you want to take carry on it, because you syndicated the deal, great. Now you got a business model going, huh? They are used by more than 4500 funds representing over$120 billion in assets under administration, they're going to support you, at every stage of your fundraising journey.

38:46From doing your first syndicate to building a global venture capital firm, you can raise and deploy from anywhere in the world because Carta offers US and international SPVs. Also, Carta provides an automated back-off solution for you so you can focus on what matters, finding great startups, building relationships, and supporting the heck out of those founders. Here's your call to action. Go to carta.com slash twist and use the code twist to get 10 % off your first SPV. What a deal. carta c-a-r-t-a dot com slash t-w-i-s-t make sure you use the promo code twist for 10 off just wrapping up your time at google they never launched any of this stuff until open ai did but they clearly had it sitting there um it makes sense that google has more responsibility with their brand name and they can't put stuff out there that's schlocky or confusing under the Google brand name.

39:45But eventually, I guess, OpenAI and Microsoft forced their hands. Right. Why did it go down that way? Yeah. I mean, people say Google was asleep at the wheel and all the rest of it, but it's not quite true. I think... So I was there at Google and working on the Lambda team, right so that i spent a year and a half working on that team and we um you know basically had chat gpt before chat gpt it was incredible i mean summer of 2020 and we had it it was working it was amazing and we were actually was that what we saw on the gmail autocomplete was that model it wasn't gmail autocomplete but it was featured by sundar in may at io the annual developer conference at um yeah in 2020 so and he actually it was actually featured as lambda you can see it up there now and yeah he actually had a conversation we designed it was so stupid we he had a conversation with a paper airplane about what what it's like to be a paper airplane and then he had a conversation with pluto and then the language model pretended it was pluto and like you know talked about the weather and stuff well you know what we always say examples matter and they pick terrible examples literally you know when you're when you're pitching your startup you're pitching a new product you want the most evocative interesting applicable example well it's two inane ones and and i can tell you it was deliberate because we didn't want it to look like a person or sound like a person who wanted it to be kind of like a you know sharing the cool technology but it was just you know the first small step in that direction don't be scared it's not taking your job it's just pluto i mean if you make it a doctor or you make it a librarian or you make it a copy editor or you know all of a sudden it's like hmm and and that's what's happened today which i think is a good pivot point here so anyway suffice it to say google is a large organization they're conservative and so they just took a measured approach and they have the goods right i think there was just a confidence that you know we don't have to go first on this and we could take more time to get it right and that you know search is just this phenomenal lock-in in distribution and data and i i think that's going to pay dividends because i i you know i think google's going to be just fine i mean google google's going to be fine i agree i bought google shares when i saw this going down because i was like i looked at bard me too and i'm watching bard and i'm like you've got so much clickstream data and you got so much local data that it's all of a sudden doing links tables it's putting in photos i mean i've seen this movie before i watched google go from 10 blue links to you know comprehensive search content shopping maps everything and that happened over a decade or two and it's obviously going to happen there and i also think the ad model you know there is a theory like the more confusing it is the more you click on ads but if you do a search for travel there's no reason that links inside the barred result cannot be monetizable in fact they will right i think that's true i think i think where google is going to struggle is that google has developed an incredible expertise for getting in its own way right it's just almost like the master of like internal chaos and so So there's loads of amazing teams and projects which just block each other because there's huge amounts of duplication.

43:14It's a very chaotic place. It really is. And so I think that's going to be challenging for them. I think the second thing is the ad model may not be the model of the future, right? It may be the case that people cannot tolerate having an AI in your pocket that is funded by whoever is the highest bidder trying to sell you something. Because these models are so persuasive, because they're so personal, because they'll get to know you, because you end up having conversations with them and sharing information that you wouldn't normally type in a regular search query where it's just like, you might say something sensitive about your cancer.

43:55or your, you know, whatever, your heartbreak, you know, but it's not the same as having a fluent, continuous natural language conversation as though you and just like you and I are now. Right. And so I think people are not going to want, you know, your AI to suddenly turn around and say, by the way, ta-da, like I'm, you know, so we'll see how that turns out. And I think Google's going to struggle with that one. Yeah, it could be affiliate links. you know if i if i was taught i was talking to my ai and i'm suffering i'm melancholy i got depression i'm feeling sad and it knows my my ai knows i'm sad uh it could be like you know maybe exercise cold plunge bath go see a psychiatrist all of those things are monetizable links in some way um and so you know if it gives you the perfect answer the question is is is it possible to monetize if you just got the answer and larry always said like eventually we're going to give you the answer we're just going to give you the answer and so it the mind does wonder if that screws up the ad auction in a major way well and that's precisely the problem number three for google which is that if google always gives you the answer then what is the you know future for the open web because google is going to disintermediate the third car the third party content creator.

45:16Like if you're a regular mom and pop shop with your bakery on a website, or you have a blog post and you rely on that display ad income, well, Google's just going to give you the perfect recipe. So why would you ever go to that kind of third party blog post? And that's actually a problem for Google and the regulator because Google has been telling the regulator for the best part of 15 years that the reason it can crawl all of these websites is because it's only indexing so that it can redirect the user to the third party page not so it feels fair it feels fair right it's a yellow pages they always used to say it's a look up table whereas if it's now cutting out that source of information and giving you the perfect answer that's a big problem with the regulated certainly in the european context because many google execs have been on the witness stand claiming that they will never do that right right so it's now the models are doing that they've been trained on the web it's obvious it's been proven you can you used to be able to ask open ai chat gpt like hey what where's this answer trained from it would actually tell you um some of the training data i think it doesn't do that now what's the fair outcome here for pools of data lakes oceans of data and who gets to leverage them to build these models but what do you think is the outcome here because we're starting to see the lawsuits pile up we're starting to see you know elon say hey twitter data is not available reddit saying it's available at a price core saying it's available at a price or maybe with a link back stack overflow built their own language model this new ceo just emailed me to say like look i know keeps stack overflow keeps coming up we're building our own co-pilot nobody else can use our data set so talk to me a little bit about what you think will happen in the industry because i feel like it's tremendously unfair to take gourmet or whatever recipe database and then just give the answer and not give a citation at least what's going to happen here's the tricky thing i mean the reality is that the information was placed on the open web and the open source crawling engines have gathered up their information um under perfectly legal uh you know acceptable terms and that crawler you know the common crawl crawler you know collects the information and clearly says that it'll be used for um you know research and development purposes and you know used for experimentation by other you know people trying to build other products off the on top of the open source search engine so the crawler the crawling data that everyone's collected is is just a well-established status quo so i don't think that is going to be undone or there's going to be any compensation you know people sometimes talk about this data trusts idea where you know each individual data contributor gets like one cent or something i mean this is not going to happen i think why not too hard to execute on i think it's impossible to generate sufficient revenues to make the payment to the end you know producer of data material right so maybe in the case of a very large data owner like you know the open ai just didn't deal with associated press right but that's actually not for historic data that's actually for fresh uh real-time news see that's where i think there is a possibility of this if we think as an industry collectively that this could actually be a benefit you remember minitel in france used to charge a certain amount per hour and they would share that with the data sources aol used to charge three four five bucks an hour compu serve and they would share that with the data provider so if you were on some data site that had to do with weddings or whatever they would just give them 50 cents of the hour right you actually had a model there i think if we took robots.txt and we put in a license and said hey listen these are my recipes i'm gordon ramsay if you want them in your index mazel tov there's a thousand recipes it's a minimum payment each year of ten dollars a recipe it's ten thousand dollars a year to put it into your index plus i want something on top of it whatever it is and that might be enough to incentivize people to start putting more recipes online it's possible it's possible i mean i think the the challenge of these things is that the creative tools are now going to be so widely available that the models are going to be you know better at generating new recipes um so the cat's out of the bag i mean in a way it's true yeah so tell me you uh you leave google uh and you start inflection with uh reed hoffman who's just on the pod um you raised a bunch of money what is inflection ai what what is the goal here well you obviously uh got to um see everything up close and personal that's happened uh with open ai and with deep mind uh where do you sit uh in that sort of pantheon of you know elite uh ai offerings you got bard over here you got open ai over here where where are you going to sit and what market are you going to try to carve out so we're developing a personal ai um i believe there are going to be lots of different types of ais there'll be business ais you know there'll be medical legal you know every digital influencer will be an ai every brand and big platform that's trying to sell stuff will have their own ai that you know that is more than marketing ai i think you know wherever you see a website or an app expect that in the next five years that's going to become a conversational interface that you might as well just call an ai right it's it'll be able to produce video and text and audio and talk to you just as i'm talking to you now.

50:59In that world where everything becomes an AI, I think you as an individual consumer want to have a personal AI that is on your team, right? It is fiduciary aligned to your interests in your corner, helping you find information, identify credible sources, negotiate with other AIs for the best bargains, plan and prioritize your day and your thoughts, your ideas, follow up on your research interests, find you entertaining information. And it is super important that it's personalized to you because you're going to end up sharing a lot of sensitive, personal, intimate information in order that it can then go out and be your representative, right?

51:46Whether it's in gaming environment and it's kind of in the metaverse or whether it is looking for sports news on your behalf and coming back to you and talking about it. The way I think about it is, imagine if everybody had a chief of staff, a digital chief of staff that was a coordinator, scheduler, prioritizer, summarizer. You wake up in the morning and it gives you the perfect briefing of everything you've got on in your day, what's happened with the news what's happening with the sports the companies that you're tracking um that is what i think it reminds me of uh remember general magic uh and there yeah this is we're dating ourselves but sony and a company called general magic made a pda personal digital assistant device uh long before palm i think there was a documentary on it um but they had a concept of agents and the this is before search really on the internet uh search engines even existed and the agent would go on your behalf and go find your flights or go find your reservations or go do tasks for you.

52:47And so you see this AI as being autonomous in some ways and being able to put it on repeat tasks. Hey, I'm trying to lose weight. I want to be 165 pounds. What should I be doing? And then it's going to counsel me every day about that. Right. And if you're going to put an AI in that kind of position, which it will be incredibly effective at doing, because it's not going to nag you and moan it's gonna you know be inspiring it's gonna be reassuring it's gonna be gentle and polite and respectful i mean it's it's not gonna be an arsehole about it unless that's obviously what you want whatever you're into please fat shame me i am not worthy it could get weird it could get weird but to your point it's going to be personalized and it's going to be your agent and so this framework is critically important in terms of your vision is that it's your agent it's working on your behalf not the corporation's behalf not open ai's not bing search results or google search results this is your ai and whatever you talk to it about we don't have any insights into and if you put data into it we're not sharing that with advertisers or anybody else so that means i have to pay you 100 bucks a year for this Yeah.

54:01I mean, at the end of the day, if you want to have full trust, you need to not be the product. And if you're not paying for it, somebody else is paying for it. And if you're putting that amount of attention and sensitive information into a place, the only way to make sure that it's on your team is for you to pay for it in some way. You wouldn't rock up and be like, oh, my accountant is actually being funded by this insurance company. Yeah. And so I'm going to go and speak to my accountant and you're trying to get, you know, your tax return done or, you know, you're trying to decide on how to make some investment.

54:38And you're like, well, are you working for me or are you trying to sell some insurance product or whatever, right? Yeah, I think understanding the intent and the business model is so critical and consumers are super savvy now. Like they understand you can't get over on customers now. they they expect that like alexa's listening to them even you know in serving them up ads even when that's not what's actually happening but you know they they are pretty empowered and and they understand this concept of you are the um if you're not paying you're the product so i've used it a bit quite delightful beautifully designed um what can we expect to be the beachhead markets or tasks that you think it's going to delight people with in the early days here?