20VC: Is More Compute the Answer to Model Performance | Why OpenAI Abandons Products, The Biggest Opportunities They Have Not Taken & Analysing Their Race for AGI | What Companies, AI Labs and Startups Get Wrong About AI with Ethan Mollick

23:03The issue is that that's where the connections are and it turns out Zoom only gets you so far. The average distance between a VC and a company invest in is about 40 miles. That's because when you look at where VC spend their time, it's networking and it's monitoring. It's networking with other, you know, and learning about companies and then it's monitoring the portfolio companies And that's much easier when you're local Zoom doesn't let you do monitoring the same way in fact way direct flight is added between SFO and another city VC investment that city goes up because it's just get it's easier to fly there and help do and do monitoring there It's a local business right everyone's like oh, it's global.

23:38It's connected. It's a local business Can I dive into a couple of different market participants? We've already touched on some of them but I want to start on AI labs. We mentioned Lama and Model progression earlier. In terms of the AI labs, what do the big AI labs not understand about companies themselves do you think? I think that's such an important question. There is just the products being released are just super weird, right? Like I think there's very little considered as you have use cases. I mean, you look at the number of people inside these organizations who've worked at large companies.

24:07I often joke when I go to the West Coast, there's like, you know, it's all like cold plunges and how do you live forever and really, you know, And then in the East Coast, it's like we drink a coffee till we die. And the goal is like get our work done, get home. Like, you know, in a large company, it's very different. There's a lot of like contempt, I think, for large companies. That's where most smart people are, right? Our large organizations doing, you know, other work that is not still like in value work. For every coder, there's 16 managers. I think that there's not a sense of what this stuff does for them.

24:35And as a result, there's a lot of half -built products that are brilliant and then get walked away from code interpreter is a huge world -changing product for data analysts that got partially abandoned by OpenAI, that haven't moved the needle in that sense. Chatbots and APIs remain the main area. Almost all the documentation is technical documentation. And almost all the interesting use cases are not being discovered by technologists who are actually quite bad at using AI off. And because it doesn't work in normal technology, they're being discovered by end users, managers, and the systems are built for those things.

25:05So there's just this huge gap between technology and use. I'm so sorry to be so naive. Why if code interpreter is such a generational defining product for analysts? Why would they walk away from it or not walk away from it? But you know not progress in the same manner as they started. I think Open AI abundance products like crazy. I think these products are passion projects from various people again They want to build the machine God if you have any talent of people you're going to have them building the next Technology for a GI if you have staff that's what you throw it if you compute that's what you throw it at I mean, they're incidentally making $3 billion run rate this year, I think, by like just accident.

25:42But there isn't like a, there isn't really a product there right now. It's the chatbot and the API and the system gets smarter, solve more problems. I think a lot of people in this space are just assuming scale solves issues. So why would I buy or develop solve issues? So why would I bother spending some time thinking about, you know, how to productize this when the products will be obsolete any year anyway? Can I see on the flip side, we have the companies themselves. What are companies getting wrong about AI that they should know more about? I mean, I speak to orders age as a whole time. First of all, almost nobody uses these systems.

26:14I mean, they all try chat TPPT. When I ask my hand, everybody's trying chat TPPT. Five to 10 % of people in any room, whether or not by the way, Silicon Valley, actual people, who aren't in a lab, whether that's at a large bank, whether that's at a conference of innovation, professionals, maybe five to 10 % have used those models, and maybe two or three percent have used ten hours, which has been my guideline, you know, minimum number. Again, there's no onboarding. You are faced with a chatbot and when people are faced with the tyranny of the blank page, they panic. What do you talk to the system about?

26:44There's no information, there's no instructions, and so people aren't really using it. So the issue is that partially it's them, they need to adopt because when people start using it, they find uses, right? So a new study just kind of Denmark of people are using chat, CBT, and you know, in knowledge -intensive work environments, and you know, they're estimating that over 30 % of their tasks, they're saving 5 % of their time. So once people use it, they find productive uses. So then the question becomes, how are you harnessing those uses? What policies do you have? I mean, there's so much we can talk about, about what companies are getting wrong.

27:13In terms of how we harnessing the uses, what would you like to see change there? Because that's where we can fundamentally drive productivity, which is arguably the most important thing. I mean, so first of all, it just starts with policies. When you look at companies, a lot of them don't even allow access to GPG -4, because the regulatory environment's unclear. So one thing that would be great for a regulator perspective is not just we've talked about the negative side of regulation There's a reason why banks are regulated or Farmer companies are regulated it would be useful for clear guidance about how to possibly use AI And I think there's been some movement towards that that movie so that I would want the EU to be doing a lot more of too It's like okay, what are the ethical use cases that we should be pushing and opening regulation for so That that extends the company policy side company policies are often very vague You know don't use this or use it, but don't get in a way that doesn't get you fired and then there's a whole bunch of like uncertainty over how you get rewarded What happens if you figure out a solution to your work?

28:05So what I find is inside organizations when I finish with the top all these people come up to me and reveal that there were secret Cyborgs all along so they've been using this for all of their work But they're not telling anyone they're not telling one because they're worried they get fired They're worried that people stop respecting their work because they realize it's AI written right now Red it's full of people saying I'm a people think I'm a wizard of work They're worried that people will realize you don't need as many staff members So you fire them or you fire their their colleagues or you won't reward them for it So everyone's hiding AI use I just spoke to a winning band chat to be T and a major bank She is chat to be T on our phone to write the band because like why do it by hand?

28:37So once people start using it they're all using it secretly and I so we need clarity around how do you get rewarded for this? So like what happens when I automate my job and to go back to our industrial revolution analogy if you are brewery in the early 70s and you are serving your local community because everything was kind of local and you have steam power. You have a choice. Do I want to fire a lot of people and make the same amount of beer for less money and have a higher margin or do I want to begin this and expand my production around the world and hire another 100 ,000 people. We're used to IT solutions being a cost saving measure.

29:14If I get 30 % of primary boost, I 30 % of people. Your people are never going to show you how they use AI at that rate. And you're never going to win in a world if we really believe is industrial revolution happening. So policies are really at the heart of the problem. And I always feel that in the majority of cases, we just redistribute talent. As you said, they're more effectively for new projects, for new initiatives, for expansion. What I find worrying is here, especially in the early days, it is killing the lower classes if you're being horrible and blunt, which is like, Clon, I have like 70 % improvements with AI in in terms of customer service and cut so many of their workforce.

29:50You're seeing especially customer service be the core kind of Trojan horse, which is replacing 90 % now of customer service teams. To what extent do you think I'm overwiring and actually we'll see the continuing redistribution of talent, not the removal of talent? So this is the case where I think we're being a little sanguine about this. I think every technological revolution, people lose jobs and then your jobs are created. But there's, you know, and we talk about this all the time. There are two big caveats to that. Caveat number one is not always, right? When the telephone switchboards went from sort of manual to digital and starting our not digital demo, but mechanical and then starting the 1930s, that went I think one out of every 16 women had spent time at a telephone operator.

30:31It was like a job. And then if you got fired from that, if you were young, you found other jobs. If you were older, you never found another job as good, right? Because you were really good at telephone operator. So not every job ends up with a new category replacing it and the other thing is living through the industrial revolution kind of sucks Right like you can look backwards and say like oh great everyone got better jobs They were much richer now, but there were also people you know smashing machines because they didn't want their jobs replaced There was a lot of unrest. There's a reason why that there was that's when the great debates between capitalism and Marxism arose because There was unrest during this period that was serious Yes.

31:06I think part of what I worry about is a little bit of even if you have a sanguine view that everything is going to be fine, that doesn't just happen automatically. You can't say the market makes everything great. Let's not worry about it. Or else, everyone's going to lose their job and we need UBI. There has to be something much more specific about, yes, there's going to be waves of disruption heading through the economy. How do you do things that we are very bad at? Retraining. It's a problem that actually be solved. It's not something that it has to be made as a science fiction. What I worry more about is actually the distribution of knowledge and productivity versus the distribution of wealth.

31:36And what I mean by that is there is this like 1 % of like Silicon Valley and tech elite, I think, who are using AI and the surrounding products incredibly well to do 10X, the work that we used to do and to be way more efficient and way more cost effective. And then there's the rest of the world. You know, we joke about Europe, but I live in the UK, you go to place in the UK, you've got no idea what chat GPT is. Let alone how to use it to create marketing campaigns that are 99 % cheaper in 10 % of the time. I think it's just creating even more knowledge and productivity to 1 % and the world could get left behind.

32:12Am I right to fear that too? Yes, I'll say yes, and. Okay, so Denmark study I told I was talked about did find that the people are using this skewed mostly male and mostly wealthy, right? There were people finding use cases because that tends to be a fairly common tech adoption curve. The thing that is unusual about AI though is first it's ubiquity. Normally getting your new tech installed means I've got to have, you know, know how to use a computer really well and be really in with like, you know, how do I get a, you know, a distro from GitHub and like, you know, there's work involved that is, that is a narrow, excited work that requires time, effort, money.

32:46That isn't the case here, right? The chatbot is accessible through a phone in, you know, a hundred, six down countries around the world of access to the world's best -day eye systems. That's one thing, right? And chat is a fairly normal interface, especially when you have voice. The second is, early evidence is that coders are not particularly good at working with AI, right? Because it doesn't do the things you expected to do. My favorite example is Simon Williamson, if you don't follow his terrific and really rated the stuff, but he works on data journalism. And he was used in Claude for OCR, political campaign donations.

33:16And when he checked back, he just found that Claude refused to do the work because they were named in addresses and even though they were public, he was, Claude was like, I don't want to violate anyone's privacy. Like we're not used to systems that object to the task that they're given or like sometimes argue with you or give you different answers every time. So, coders are often not the best users. Often the best users are people who are actually really good at working with humans. I mean, my wife is probably one of the best prompt engineers in the planet. She's got a doctor who worked together with code directors, the AI lab.

33:43She's never coded a day in her life. But regularly, just stuff that open AI and anthropic are like, wow, that's a really amazing prompt. We didn't know that Google used her prompt as the gold standard to measure their fine -tune models against, right? But what she has is a, you know, doctorate education. It would be building educational, teach the games for a long time. And she has good theory of mind for other people. If you go write instructions, if you can manage, you can use this. So that's what I'm hopeful for. It looks like a tech adoption curve, but tech people shouldn't have the advantage they have in other spaces and we're just has to get out.

34:11I do have to ask you mentioned that kind of the quality of prompts and how amazing your wife is with her quality of prompts. You also mentioned kind of the white screen of death and when you have kind of that blank template not knowing what to do with it. You've said before about bluntly the kind of challenges of the chat bot interface and what a weird interface it is. What do you think will be the interface of the consumer between the power of AI and consumers? I think multimodal is really the answer here. All the pieces are in play. Some of the most interesting people I talk to are really using at, or just having conversations with it.

34:42I think about Ali Miller, who is a really great person who is thinking about a lot about AIX, Amazon person. She has conversations with the AI every morning while she's doing her hair, just the limited chat interface. Once these things have full visual, which they do, they have a lot of latent capabilities that people haven't recognized yet in multimodal, and you can chat with them. Then it starts being more like having a human on cold. I think when you start adding agency into that, where they can take action in the world, I wonder if we just sort of skip the step of, you know, how do you use these things?

35:12So like, oh yeah, you talked to your phone and your assistant does the thing that you wanted to do. There is this narrow window, I think we're prompting style really matters, we're being really up to the end of these systems matter, but then they come to your phone, and also by the way, if they save you time and work, If they really do do that, humans are exquisitely designed to figure out how to minimize the effort they put into things. There's a reason why adoption rates are over 70 % in universities for chat GPT, and while they're like a few percent elsewhere in the world, we figure stuff out like this.

35:42And I think that that's the other piece that's missing. One element that we haven't discussed is start -ups themselves, actually. And so on that, you've said before that you don't think start -ups are being ambitious just enough in the face of AI, what should they be doing ether? And why are they not doing it? I think the problems of the lean method are coming home to roost. What every VC wants to see is they want to see product market fit. There's a method we have, right? You come up with like a rough business model canvas and then you go out and you do talk to people and then you test it the world.

36:16That is not a good model for breakthrough innovation. That's a really good model for incremental innovation or you find market need. So part of this is that we're incentivizing startups to find solutions right now for a moving technology and they're just going to get lapsed. And they're not trained to be imaginative. They're not like, they're trained to think money first. And how do I get a product market fit? Which is fine in normal technological regimes? Not a great idea in radical regimes. What is a good model for a radical regime then? Because I've been brought up quite rightly, as you mentioned there, in the incremental innovation kind of economy, where it's like, test, iterate, find, product market fit, someone pays for it, good, well done.

36:54So what is the right model in this new age of kind of radical innovation shift? So, I mean, we, we, we see have funded this model, right? And it's like deep tech medical things where you're making larger bets in the future, where they're, you know, where there's payoff is it where, where when it's revealed to the world, the other succeed or not, right? And where you're making a better technology itself, that's where BC got it start. It sort of became, you know, perverted a little bit to this like, how do I get, you know, big money fast machine? I mean, not that fast, right? Still years till exit.

37:24But there's the idea of like, it's all about prerada rights and the idea of like, I make a lot of small bets initially and then I can double down in the people doing well and not do double down in others and it's about finding the diamond in the rough. Like, all of that stuff is like a great model for funding incremental innovation. If the market is changing, like, and we're using market changing slowly enough that like, that's not a problem. I think it's an issue here. I think you need to be imaginative. I think you need to be subject specific I need to think you need to do it soon model. I mean it is very strange from one hand for all these people It's like I'd be like yeah, HGI is coming and then the applications that are building are like these very narrow like hey I slapped something atop of llama.

37:59That's not gonna do it What should I and what should my fellow venture investors change than about the way that we invest do you think? I think that But what you should be thinking about is have a position on the future. And start off, you talk to have a position on the future of AI. How good does it get and how does your model work? The second thing I think people need to be thinking about is how actual adoption happens again. It used to be that if you were large enough market to play with, we just go after all of it. And you know, some part of it starts to respond and we double down in that section.

38:30You're going to be much more opinionated about how you imagine your technology being spread or adopted. How does it spread throughout our organization? How does it fit with the organizational structure and approach? Requires people to have more plan and strategy they did before, rather than just letting the market tell me the answer. Do they not go in constiction? You said they're about, hey, people work on small, kind of minute things on top of Lama, say. And then it's like, well, you need to be opinionated about who you're going after and who you're not going after. You need to be more targeted.

38:59It's not kind of one and the same, which is like the verticalization of approach and the targeted approach being the core. Well, I think it's not about verbalization as much as opinionated, right? I think you need to have a strong opinion of what the future looks like and where the gaps are going to remain. This is a jagged technology. Figure out where you think there's going to be Jaggedness and that can be organizational jaggedness, interface jaggedness, but I mean you're also basically the real problem right now is every startup in the world is betting against AGI, which I find really funny because all the funders are like, AGI is coming in the next five years.

39:30If it is Why are you funding these startup companies? None of them are surviving an AGI world. for those that don't understand, why will none of them survive in an AGI world, Ethan? So, AGI is a common definition of AGI is a machine that's smarter than humans at every task. The machine will decide what to do. You're not going to, like, who cares about your stupid product, right? Like, you've been making this for humans to get a product market fit, and the humans will say, you know, optimize by training strategy, or the AGI will just decide to optimize your trade strategy. I mean, no one knows what AGI looks like.

39:58So, I'm not going to try and paint a science -fiction future, but I will say there's a huge contradiction between a Margaret Grayson saying AGI soon, and we're funding a bunch of companies that are helping you. Already, I don't know if you play with them, not that we're in a new area, in your AGI with this, but you could tell Claude, come up with 30 ideas for a product, serve MarketX, then rate them all on quality and feasibility level. Then create, this is one prompt, by the way, then create a playable prototype of the interface for the application, then interview me as a user about how to change it and adapt it as we go.

40:28And it does it. I get a little playable interface for a game, and I can then edit the game and say, like, oh, I wish it was more, it's just that fun enough in some way. And it's like, okay, great. I'll make it more fun for you. If that cycle's really there, what is your stance and what an HCI world looks like becomes very relevant? It's all I can say. Do you find it interesting to see the different people's opinions on a, especially on the founder side, different people's opinions on the time to AGI and their requirements for funding? And so what I mean by that is like, Damis and Zuck, a very long term minded in terms of how long it will actually take to achieve AGI and they also don't need any money.

41:04And then there are other founders who I will remain nameless, who are pumping it as being much sooner, but they need to present that future because they need the money. I don't trust anything. People are very self -motivated, right? I think the signal you should be attention to is that people are betting their careers to a large extent, right? On this being possible. And I think there are people who care about the reputations. That is a signal to me. They don't have to be right. I mean, look, I work with Marvin Minsk, like I said, he was there in the 57 coffers where they out in Dartmouth, where they outline the concept of AI.

41:34I mean, we're in a world where like, you know, AGI is always soon. So I think you would take everything with a grain of salt, but I think you need some coherence about your own viewpoint on this set of stuff. Now, the large companies, I mean, we're seeing a lot of, you know, people warning that this is coming soon. I mean, in the meta paper, the paper outlining the release of Lama 3 .1, the keynote yesterday, it says we see exponentials continuing for the, we don't see any reason why exponentials we're gonna stop. What does that mean for you? As a startup, it feels like a relevant question. You're betting for a future world.

42:02So what does that future world look like? And you can't both say everything is changing, but also I'm doing this minor thing. I also think crypto did a sturdy in this kind of front, which is like it made all technology feel like hype. And it emphasized again, short buck return. If you just believe something will happen and that's really a great way to think. I actually liked Sam Altman when he said on the show, the simple kind of heuristic of like whether you're gonna get steamrolled by open AI is, would you be excited or scared by a 100 -axe improvement in our model? If yes, then you're going to get steamrolled, if no, then great.

42:33But I like to do the heuristic. But I don't think it's useful as a heuristic. What does that mean? What is 100 times better, GPD? It's a baffling heuristic to say better. What does that mean? It's an uneven system. It has gaps in the world. That mean 100 times better reason. How are you supposed to? This is what I mean when you start looking at these things, it's like, what the heck am I supposed to do with that? It's 100 times better. It's a machine god. Like what? And so I don't like this heuristic because I like I don't have any way to operate within that 100 times better. Does that mean it will be able to process an entire legal document and do a very good legal review of a document on its own?

43:10Great. That's disrupts a huge industry But that is a actual question about hallucination rates It's ability to handle words, you know to think about words instead of tokens to understand precedent to be okay across different languages is to hold a huge amount in its context window, that feels like a useful question to ask. Could it write an academic paper on its own, right? Where you'll give it a data set, it generates hypotheses, tests, them writes a really good paper, formats and latex, writes the letter to the editor, and handles reviewer responses. We're getting close, but there's a lot of gaps there.

43:40Give me a concrete example of what this thing does, and then we could talk about a heuristic, but like 100 times better is a really hard one. No, those use cases are 28 times better. They're not 100 times better, specifically 28 times. I'm joking. But maybe, but I mean, I think that's a valid question. Some of those things are huge gaps. Some of those things are small gaps. Are you asking about that? Reviewing the legal document, not really a problem. Review it. Like, we're close to that. But like, if it does that out of the box, that also implies a lot lowering of hallucination rates below a threshold, but they're not currently at.

44:12There's no benchmarks on hallucination. So we have no idea how good we're getting on the hallucination right side. It also though implies the ability to seamlessly move between different perspectives. We can do that with agents today is that an agent -based model of thinking action like there's so many questions I would love some specificity and that's why I'm saying fields specific is great. If you are a lawyer who knows law field really well you probably might have some interesting things to think about it where the real gaps are or not. And I don't think a lot of the AI firms know that. I know this because we're deep working with all of them on things like education and like they don't really understand education There's no educators there So they don't really understand what teachers do and they don't really understand what classrooms are for and so it's all the AI work place Everyone and I think we're a lot and we're not there.

44:55That was a trigger. Wasn't it? I just give you a some Auckland heuristic even I was like oh Tell me you said they're about kind of the importance of being opinionated and for startups to have strong opinions about where you know AGI will be how they fit into it, organizational design. If I would ask you and flip that on you, where are you most opinionated in your views around it? Why would you suggest or point to first? So I think education is a good starting point, but an education, Tudorang is the gold standard for interventions based on the research we have. And AI is an incredible one -on -one Tudor.

45:28It's transformative. But when I find Silicon Valley people and AI and education people often think is like, once we have a really good Tudor, we don't need teachers. or like I hated this subject in school, or people would be self -motivated to learn. Absolutely untrue, people are not self -motivated to learn. And even all the computer scientists out there were like, I was like, yeah, you're an auto -diadjacked at some narrow area, but you would have learned nothing about very important topics because you only cared about one topic, right? People need extrinsic motivation to learn. It turns out that there's a value in having an instructor guiding the direction of a class, that there's a value in putting things into practice.

46:01So even an incredible AI tutor that knows you and loves you really well, doesn't sub in for teachers. And also forget all of them. Let's talk about systems. Schools are in a complex system of society and where they are for, you know, providing daycare services to how they fit into education that works, how we do credentialing to the teacher unions, to like, there's a billion things about schools that don't get replaced by AI, by having a magical button you push to make stuff happen. There's gaps and opportunities that are very different than a naive view of how education changes. So one of the biggest problems in UK education today, I'm not sure if it's the same in the US so you can tell me but it's the exponential increase in class sizes that we've seen particularly in public schools, which is the schools provided by the state and The quality of education has gone way down when we look at AI's ability to increase education standards Will we see the ability to maintain high education standards with increasing class sizes?

46:58How do you think about that? I mean, I hope so, but let's just tell you the first Renderized Petrol trial we have, my some of my colleagues at Warden was giving a GPD 4 people from Math 2D ring in Turkey. Now they didn't do a huge amount of like, you know, it was a signed class and they used the system, but it turns out that everybody who used it used GPD 4 without a special prompting or anything else had much higher homework scores and then did much worse in the test because basically the AI just did the work for them. And once you have better, once you have better prompts that that effect disappeared, though we didn't see an educational gain from it.

47:29But I think it's an early sign of not being naive about how these systems operate. We need to put the work into building scalpelier in them. I absolutely believe that we can't be naive about the work that needs to be done here to make this stuff operate. So you can't just drop these systems in, but a good tutor will make a difference. I think in the long term, we'll have flipped classrooms where that giant classroom is actually fine because a lot of your learning is done outside of class, AI tutor help. And then inside a class will be activity exercises application where large class size doesn't matter as much.

48:02But there is a road to get from here to there. I think this shows done well because I'm not scared to admit my own flaws and stupidity. Everyone talks about kind of the incredible optimism that AI brings for education and talks about tutoring great. But I don't really understand what that picture of the future of education looks like then. Does that look like when When kids come home from school, they just have a plastic steel opening up and they have another tutor with them. As you said, in most cases, they end up doing the work for them and so they don't learn, is it a crutch that I don't understand actually intangible reality?

48:40What does the future of education look like with AI and why is it optimistic? We actually have a lot of research on this. It turns out that, first of all, there's a couple things you need to learn about learning that people don't tend to think about, which is learning is hard and sucks. What makes you feel like your learning is in what's learning. Like, you have to do grinding work. There's no solution to it. It's just like any other thing like exercise or anything else. You have to be pushed to desirable difficulties where you're having trouble through not failing at a thing you're not working hard enough.

49:04Like, that's, which why you often need to shrink motivation. And the second thing is we actually have some research. We know things like active learning where you're in a classroom doing activities, beats the idea of passive just receiving a lecture. When we have those sets of pieces, there's been a move that kind of fizzled called flip classrooms that has some early evidence in your class. Which suggests this idea of like classroom should be about doing stuff and outside of class should be about getting the basics because we can get you to do stuff in the classroom setting. So that would mean that outside of class that what that practically meant is you watch videos outside of class to your teacher talking to the lecture stuff is all outside of class.

49:40That's your homework. Read the book. Do that. And your homework is in class where you can mess up in front of people and work in teams and that you learn by kind of watching other people and how you're doing it. The teacher can help you solve problems. I think slip classrooms are a very natural fit and active learning with AI based approaches. So instead of having a passive video you watch, you'll have an AI tutor outside class, you'll log into the school's website and that tutor will be amazing. It'll be adapted to you and then it'll pass that information on into the classroom setting where you actually the teacher gets advanced stuff.

50:09By the way, we've actually built a version of this already at the Journal of AI Lab at Wharton. We'll be open -source to all of that, like it does this kind of stuff. It's not that hard to imagine. We just have ways to go still. So is that really an order of magnitude improvement if we compare that post -crashment? You could give me incredible high quality videos of you talking lecturing, giving examples that you give to your students now, very easy to do, versus that AI tutor. Is it 20 % better? Sure, maybe it's personalized, but is it really an order of magnitude better? Education is a complex system.

50:41So I think order of magnitude is a very weird thing to talk about because every student has their own talents, abilities, interests, and gaps. The early work on in one -on -one tutoring, we don't talk about order -maggin improvement because that doesn't really work in the educational world It's very hard to say what our magnitude is, but we can talk about grades a lot. The classic study that is probably would not be replicable But it sets up our model is that one -on -one tutoring creates a two -sigma increase in classroom outcomes. That's two standard deviations Which is a fairly huge improvement.

51:10You go from the 50th percentile to the 97th percentile class We have no idea if that's going to hold up with AI tutoring. But if we could do that, that is amazing and improvement as you can possibly ask for. I mean, a 10 % improvement is amazing. I kind of feel like aiming for order -managed education if we can get improvement in a system were a great shape. I also think that doesn't include a lot of different elements that you mentioned, the extrinsic motivation being a big part of it. I think a big part of having a tutor means you actually have a bond with them. You want to impress them. You want them to feel proud of you.

51:40Does that extend to an AI tutor way you don't have that human? Maybe. It's not clear that that is the key to tutoring is the bond with the human being. Forcing people to come from what they don't know turns out to be a lot of the value of tutoring. So tutoring is often reflective back. So when we build the tutor chat bot, right, what that tutor chat bot, like the way we test by the way, education technology, chat bots, our rule of thumb is that if it asks you, if you understand a topic or you rate a move on, it's a bad tutor because humans don't know when they're ready to move on or not. But the AI should be doing is asking you questions, probing what you know, and making you expand on what you don't understand that helping you fill those gaps.

52:17It's not the one -on -one bond. There is a method to teaching that we actually know make a difference. Self -reflection makes a difference. Repeated practice makes a difference. Low stakes testing makes a difference. Like, to zoom back out to what we were talking about before, subject matter expertise is going to be absolutely critical in making AI work. It's a system that experts, I can look at a prompt in entrepreneurship and education, and instantly tell you about whether that's gonna work or not, or whether it's a stupid idea or a good idea, whether the subtes that the system is missing are problem or not, because I'm an expert.

52:47And if you're not an expert, you're gonna be like, that looks really good. So expertise actually matters. I'm sure in the same way, it's one of the things I actually, when I talk to my students and teach them how to pitch, right, one of the things I talk about is there's this really interesting research that shows that venture capitalists are not swayed at all by the quality of the speaker. Their ability to be a good speaker and that is absolutely relevant amateur and angel of essence are swayed by that why because you're an expert you've seen so many pitches that you instantly see through all that stuff and you're like you know what the core issues are right away because you've seen 10 ,000 pitches you've seen how they play out in the world you know you have to be a really amazing speaker to pull off I'm persuasive on top of that and so in the same way I think expertise is going to matter a lot here.

53:30When you look at the pervasiveness of AI and specifically chat GPT in homework and in coursework and in the answers that many students give today. Is there any point in university or educational facilities doing homework or coursework when it's largely done by AI today? Of course there is. Everybody was already cheating. There's this great study at a repeated university that found that homework improved when you did the homework and improved on 80 % of people's test scores in 2008 and by 2020 it only helped 20 % of people. That's that because homework stopped helping, it's because everyone was cheating.

54:06And so we have ways around this. There's really two options in how to use AI and education. One of them is to ban it cautiously, right? People are still gonna use this explainers and stuff like that, but you have in -class tests and blue book writing, like we've solved this problem in math. And like you make people do exercises and do work, nobody likes it, but there's no shortcut to learning. It sounds dumb, it's like what your teacher said, out there that's true. You need a grinding amount of work to understand something. You need to do interleaved practice. You need to like, there's a lot of stuff you need to do to learn something.

54:33And so we absolutely can make you do blue book work in class. We absolutely can install terrible monitoring systems. I don't like this approach, but like a couple of companies already have this. Watch what you're typing and make sure you're not pasting stuff in from AI. Again, I don't necessarily recommend it, but like these are possibilities. I think you're underestimating how much you can do those kind of things. Homework is valuable. Cheating is bad. What is AI cheating? We have to define that. I'm a big, but the other options transformation. My classes are 100 % AI based at this point. The students have AI mentors and tutors they talk to.

55:03They have AI -based assignments. When they learn how to do hiring, we build a simulator that actually makes them have to fake hire somebody and the AI plays the person they're interviewing and gives them multiple choice answers and they have to reflect on the assignment. There's one of the other assignments is they have to teach the AI to do something. You could do really exciting stuff. It's just not gonna happen right away. All of the different avenues, functionalities that we've spoken about require a lot of intense compute. considering there was such a trigger when I gave you a last quote from Sam thought I'll give you another one.

55:34Compute is the currency of the future, what Sam Altman said. An energy is a concern when looking at the energy requirements this next generation of AI will bring. How do you think about the energy requirements required for this next generation of AI, usage and society and why the Sam is right that computers the currency of the future? That's what Sam believes. I mean, Sam believes in AGI and he believes that it's going to be achievable in the near term, right? And when you talk to open -air insiders, they feel the same way. Invest the case. If intelligence of demand is the case, intelligence of demand is power hungry.

56:04And there's infinite demand for intelligence on demand because there will be, right? If you have an AGI, I want that to be looking over all my medical records and monitoring our air space and finding scientific ideas and helping me with a project I have to do and also booking tickets for the ultimate trip. Like there is infinite demand for intelligence, right? So then compute becomes the currency and energy becomes the big deal. That will make a big deal in that case and we're gonna build a lot of nuclear power plants I guess and you know, in relatively short order. It seems like that's a pro or aGI figures How to diffusion and it doesn't matter or we like it turned into batteries all our matrix although we don't produce enough Waterage, you know, so I don't think that's really the issue.

56:42I'm training data That's what the a guys will use us for but anyway mostly joking but right now I think the energy debates an interesting one because again, it's one where tumors and Optimists sort of like to talk about because on the downside risk when I meet people who are skeptical at AI the first thing they talk about is energy use. And the truth is that AI uses a lot more energy per query. We don't know exactly probably two orders of magnitude than a Google search, but a lot less orders of magnitude to energy than a human doing the same amount of work, right, with a laptop. How do we balance those kind of things becomes an issue?

57:12Right now, 1 % of US power goes to data centers, and 10 % of that goes to AI at most. So we have a lot of room left at the top before this becomes an issue. So again, we're assuming AGI is available, and slowly useful, and in which case, absolutely compute becomes, and energy becomes the issue. But then that becomes the reverse salient. And there's a lot of money to be made that if the currency of the future is compute, a computer's energy, then there's a hell of a lot of money to be made in building your own nuclear power plants. The final one before you do a quick fire, a friend of mine who's also a quite a well -known bunch of capitalist, Jeff Lewis said that when it comes to democracy in the future, we will vote for algorithms, not for people.

57:50to what extent do you think AI pervades into electoral systems, electoral voting, the political fabric of us as well? When something feels like a dystopia, to most people, it probably is something that's not going to happen very quickly. Human systems are complicated. I just keep seeing this technological view, which is like in a rational world, the machines will rule us all. It's just people don't want that, right? So we already have algorithms ruling lots of what we do. Your FICO score determines a huge amount of, you know, things that happen in your life. and that's an algorithm. We have these kind of systems in place, but the idea of an overall all -seeing kind of approach, it's hard.

58:27Now, on the other hand, we do find that AI is hyper persuasive already, right? In a controlled experiment where you do your apps to be, you talk to a normal person versus the AI, your 81 .7 % more likely change of use the AI's view to a human's view. That is going to change marketing in very big ways, which is going to change politics, right? Deep fakes are going to be our big deal already, although it's been funny how little big deal they are because it just turns out all you need to do is show a video of politician X talking and say, I can't believe he said he's going to eat babies in minute three and everybody shares it online who should know better and without actually watching the video at all.

59:01But I post on my viral tweet, nobody clicks the link. I feel like we weigh over, estimate people and therefore how much this stuff will get a matter. But in a world where AI is hyper persuasive, this does change things. In a world where AI gives really good advice on everything, people should have an AI second advisor happening in every role including in politics, right? That make things better. But people are going to listen to a politics change as much more slowly as much more human than people think. By the way, it plugs into larger issues of like when we can produce all this stuff on demand, what is actually valuable or not.

59:29I mean, everything is going to change. It's very unbreak predictions how a general purpose technology rolls out. But I do think people overestimate how quickly the short term change is going to be. And as usual, from Mara's law, underestimate the long term. I completely agree with you that. My biggest concern actually, where as a content creator in many respects is with the infinite supply of content, the value goes down and discovery becomes much more challenging. That is a big concern. I mean, that problem has already happened, right? I mean, to me, the really interesting thing is, like, Suno and Odio, and they're getting pretty good.

1:00:02At what point does having an AI -generated song playlist, you know, there has a couple real musicians, but also makes up songs based? Like, that doesn't feel as far off, in terms of people enjoying it. Like, what happens to content creation is a very big deal, right? I mean, like, I've been author of my books in New York Times best seller, that's amazing. I don't think people realize how few copies you need to be a New York Times best seller. Like, you're selling like 6 ,000 hardcover copies in a week. That's getting in the New York Times best seller list. The attention's already scattered across a huge amount of content.

1:00:31The one thing you'd hope for is maybe AI creates better connections, right? And, you know, in some ways. I'm not being rude, but could you not just buy, I know you haven't, but I could, could I not just do it? Do a book and spend $75 ,000 and being new at times best seller then. So people do that all the time. The newer times has a small cabal of people who refuse to talk about how they do this. So they use the number ranking, but then they also try and exclude both buys. They actually try and cut that out. So you'll notice there's a little dagger next to the name of companies the best seller list, they think that they're cluting but they still had potential both buys.

1:01:05I actually got the little dagger on mine because a company bought 500 copies, which wasn't the main reason for the list but they were found that suspicious. They're trying to filter that out by hand, but yes, you can often buy your way into listening people do that all the time. Yeah, no, I've seen many of my friends who have VCs who have books and are like, really? There are ways of doing this. You scatter buyers across multiple locations and they all do, but like it is very much true that a lot of your unnamed VCs do seem to have asked a lot of friends to buy book copies of their book. Amazing.

1:01:34I love that. Listen, Ethan, I can talk to you all day. I want to do a quick fire round. So I say a short statement, you give me your immediate thoughts. Does that sound okay? Sounds great. What do you believe that most around you disbelieve? I feel like the very simple idea that AI is very profoundly much better than people think and as it keeps betting better is something that I think most people don't actually believe. What's the most concerning future that AI could bring? The most concerning future I think is one where we lose agency and not necessarily to the AI systems but to the systems that incorporate AI.

1:02:05What I mean by that is we have a chance to make AIB use for human thriving, that's not an automatic process, right? That means not firing people when you have AI in your company, but it means figuring out other uses for them that are valuable. It means building systems that help people feel like they're accomplishing more as a result of using these things. And I worry we're not seeing enough people modeling that kind of behavior. It's all about just the technology itself and then how do we get cost savings? What have you changed your mind on most in the last 12 months? I have gone back and forth on how much juice the technology has left and now I'm back to the it has lots of juice left, like the exponential continues for a while.

1:02:41And I think I was not clear on that for a long time. What caused that shift backwards? A cumulative evidence, right? So we dug great Kevin Scott saying, like there's a bunch of people who weren't talking about scaling, solving everything six months ago or eight months ago, who are now more competent, which indicates to me another generation of models came out, and everyone at all the labs are getting that hard to look at their eye again. I don't know when we'll see these models, but they're clearly people are seeing things that indicate to me that there's more more left in the curb, and they're all talking about it.

1:03:07Do we see all large players and incumbents move into the chip player? We've seen Apple move into the chip player internalized margins, remove reliance away from Nvidia. Do we see that as a large shift in all providers? Your requirement in any supply chain pipeline is to eat the value. If you're like, that's the whole idea of like how, how, you know, how those things work. So like if you're spending a lot of money on chips, you can go into chip making. Just like if you're spending lots of money on, you know, the warehouse saying you figure out a way to reduce your warehouse costs. They're going to figure something out.

1:03:41What element, sorry, of AI development has most positively surprised you? How clever these things are. If you haven't seen my Twitter feed where I asked the AI to remove references about squid from the novel Alquain and Western Front, I just look for that. These systems are really clever. They're joyful to use. I think that's surprising. Final one for you, Ethan. What question are you not ever asked that you think you should be asked more? The question that I think people should be asking and that I don't have an answer to yet is Why are some of you about some off -based systems? Like why are they not you know?

1:04:18Why are so many people using them a little bit and not forever? Because I don't think it's just simple like it didn't work people are getting kind of freaked out by these things in ways that we don't really Understand how humans are relating to these tools like we always talk about the systems the technology how industry is going to change. And I don't mean just like the dating thing which tends to be like, well, you have a relationship with that. But like, how are we relating to these kind of tools? That's one of the questions. Let me do a second take on this. The thing that I would be thinking about since related is meaning.

1:04:44People don't ask me enough about meaning of work. That matters a lot. Graber's bullshit jobs, I think, was mostly not correct based on survey data and other stuff we saw. But it's real. People do feel alienated from work. People do, like, most employees say they're bored, at least 10 % of the time at work. But they're doing work that they feel is meaningful. When you survey people, most people think their jobs matter in the world. And what's going to happen that I'm very worried about is when you realize as a middle manager that AI does your work and nobody cares, what does that mean for the nature of work?

1:05:11How does that matter if people don't care? Like AI subs in and does stuff. If your boss is responding with the AI answer to the emails you send them. And I think that that meaning crisis is when we're not talking about it. It's one thing to be replaced in a job is another to semi replace yourself and realize why am I doing this. And I think that's going to be a bigger issue that we're not talking about. Ethan, listen, I apologize for going in so many different directions. I so appreciate you putting up with some of my base questions, but this has been fantastic. And for me, as a lover of your writing, it's been a huge pleasure, my friend.

1:05:42This has been a wonderful two. Please don't tell Samo when I disagreed with him, because he's building a machine god, and I don't want to be angry with me.

1:05:51As I said at the beginning, Ethan's writing is some of my favorite writing on AI. So if you haven't checked that out, you can check that out by checking out Ethan Mollick on Substack and if you want to watch the full interview with video, you can find it on YouTube by searching for 20VC. We always love to see you there. But before we leave you today, when a promising startup files for an IPO or a venture capital firm loses its marquee partner, being the first to know gives you an advantage and time to plan your strategic response. Chances are, the information reported it first. The information is the trusted source for that That important first look at actionable news across technology and finance, driving decisions with breaking stories, proprietary data tools and a spotlight on industry trends.

1:06:33With a subscription, you will join an elite community that includes leaders from the top VC firms, CEOs from Fortune 500 companies and esteemed banking and investment professionals. In addition to mastery journalism in your inbox every day, you'll engage with fellow leaders in the active discussions or in -person ad -exclusive events. Learn more and access a special offer for 20 VCs listeners at www .theeinformation .com slash deals slash 20 VC and Speaking of incredible products that allows your team to do more, we need to talk about secure frame. Secure frame provides incredible levels of trust to your customers through automation.

1:07:11Secure frame empowers businesses to build trust with customers by simplifying information security and compliance through AI and automation. Thousands of fast -growing businesses, including NASDAQ, ANGEL, LIST, DUDELE and CODA, trust secure frame, to expedite their compliance journey for global security and privacy standards, such as SOC2, ISO 2701, HIPER, GDPR and more. Backed by top tier investors and corporations, such as Google, Client, and Perkins, the company is among the Forbes list of top 100 startup employers for 2023 and Business Insiders list of the 34 most promising AI startups of 2023.

1:07:49Learn more today at SecureFrame .com, it really is a must. And finally, a company is nothing without its people. The global law firm built around startups and venture capital. Since forming the first venture fund in Silicon Valley, Coole has formed more venture capital funds than any other law firm in the world, with 60 plus years working with VCs. They help VCs form and manage funds, make investments and handle the myriad issues that arise through a fund's lifetime. We use them at 20 VC and have loved working with their teams in the US, London and Asia over the last few years. So to learn more about the number one most active law firm representing VC backed companies going public, head over to coole .com and also coolego .com, coolees award -winning free legal resource for entrepreneurs.

1:08:35As always I so appreciate all your support and stay tuned for an incredible episode coming Friday.