20VC: The Biggest AI Leaders on What Matters More; Model Size or Data Size & Where Does The Value in AI Accrue; to Startups or to Incumbents

0:00This is 20VC with me Harry Stubbings and I've now done 15 episodes with the world's best experts in AI and two questions in particular, rise more than any others. 1. What matters more? Is it model size or is it data size? And then 2. Where does the value accrue in the next wave to start ups or to incumbents? Today we hear from 8 of the best, bringing their thoughts and ideas together, breaking down these two questions. It's an incredible compilation. Let me know what you think of this style of show. I always love to hear your thoughts and you can let me know on Twitter at Harry's Debbings. But before we dive into the show's day, you know all those mind -numbing tedious tasks that seemingly take up half your day will code is here with their new AI powered work assistant that helps you and your team not just finish tasks but make progress so your product team can bring a feature to market faster by using code AI to tag customer feedback draft PRDs, suggest target audiences, summarizing product discussions and more.

0:57Your sales team can engage more customers by bringing in data from sources like salesforce, and then use Coda AI to suggest action items meeting agendas or lead scores, and your marketing team can drive an impactful launch. With Coda AI summarizing user insights, creating briefs generated from notes, and writing mock press releases to visualize taglines. With Coda AI, you can reimagine your stu -list, and how you collaborate, So you're not only finishing toss, but really making progress. If you want to work a system that lets you get back to work, you can get started with Coda AI's day for free, head over to coder .io, slash 20VC, that's coder .io and get started for free.

1:36And to be here of amazing tools we cannot live without, travel and expense are never associated with cost savings, but now you can reduce costs up to 30 % and actually reward your employees. How do you do this? or Navan will want to be your employees with personal travel credit every time they save their company money when booking business travel under company policy. Does that sound too good to be true? Navan is so confident you'll move to their game changing all in one travel, corporate card and he spends super app that they'll give you a $250 in personal travel credit just for taking a quick demo.

2:10Check them out now at navan .com forward slash 20V scene and Last but not least, we need to talk cash. As of 29th June, you can get 5 .5 % yield on your cash, with 26 -week treasury bills. But buying treasury bills is not that easy and you have to navigate a website that looks like it was made before I was born. Enter public .com. Their treasury accounts make it simple to earn a high yield on your cash and it takes 20 seconds. Here's how it works. sign up at public .com, easily purchased 26 -week treasury bills that automatically roll over at maturity for a compounding yield. Plus, there are no minimum hold periods.

2:49You can access your cash at any time, with the flexibility of a bank account, of course, to receive the full -guaranteed yield. Of course, to receive the full -guaranteed yield, you do have to hold a maturity. But here's the thing, these are tea bills, which means your investment has the complete back of the US government, making one of the safest places to park your cash, go to public .com forward slash 20 VC to lock in a historic 5 .4 % yield on your cash. You have now arrived at your destination. So today we're going to kick off with the core question, what matters more? Is it model size or is it data size?

3:26Kicking it off today is Nume Shazier, found and CEO at character .ai. Yeah, probably the size of the model is the bigger challenge. We can get a lot of data, but actually the number one thing that's important is how much computation you do to train it. So you want to train a bigger model and you want to train it for longer. So the two things are both important, but the real constraining factor has been how many operations of computation it takes to train it. Because if you make it bigger and you train it for longer, both of those multiply into to how long the thing takes to train, so that people have been building better and better, essentially supercomputers to train these models.

4:07What are the biggest constraints on your models today, do you think? Computation. So, you know, the model we're serving now, we train last summer and spent about $2 million worth of compute cycles doing it. We will do a lot better in the near future, but if we get a lot more better, hard to act, which we are getting, and spend longer training the thing, we can train something smarter. Back in say 2016, you could train something that was like smart enough to translate languages, but not smart enough to answer questions or be fun. Nice for moving to another of the hottest AI companies of the last year.

4:46We had courage .ai there, and now we're joined by Chris at RunwayML on What Matters More. Is it data size or model size? I think size of bottle motors in the sense that we've seen that larger models parameter -wise are going to get better at doing more things in multimodalities, but at the same time, it depends. It depends where you're trying to do. And there's opportunities to get models to be smarter, but has more specific to the type of thing that are trying to accomplish. And we're already seeing approaches like this in the language domain. We'll start to see this as well in the image and video domain.

5:18You think we'll see the verticalization of models, or light -riches -sulture -with -you .com, very much pronounces the importance of a single model to rule the rule, which is much more horizontal. Yeah, I don't think there's going to be a single model to rule them all. That's like saying that the internet would only have one e -commerce site. There's many. You have something like Shopify where everyone can build their own website and their own e -commerce site and you can sell anything. If you think this is a tool and it's a general purpose tool, there's so many opportunities to build different type of models and different ways of working with those models.

5:50and it's too very early to be so specific to say, oh, we're gonna only use that thing or that other thing. How do you think about model lifespan? Do you have to continue to see update models? How do you think about that? I've found myself hearing a lot about models as the mode and mode has been something that Silicon Valley has been discussing for some months now. I think models are not a mode. Models eventually don't matter. What matters most is the people building those models and how fast can you change your learn from those models. So I don't think that's why I go back to the snow one similar model that's going to rule them all.

6:25Speaking of not having one single model to rule the more we did an incredible discussion with RichardSortcher at U .com, I've had guests on the show before and they say the model size isn't so important, but it's the data size that is. Is that wrong? It's totally not wrong, it's just not mutually exclusive. You need a large model and you need a lot of training data for that model. Either of them in isolation, like just like imagine the simplest neural network will just predict a single one -dimensional output line. Let's like a regression analysis. You have some input X, some output Y, and you try to model where it goes.

7:01You can model that with a handful of neurons. And the simplest one is just like a line, a linear regression. And that model has even fewer parameters. Along story short, if you now give this linear regression model billions and billions of training data, it's not going to learn magically anything but a simple linear line. But if you give the model billions and billions of parameters, it can learn all kinds of very complex predictive functions and abilities. So concretely the big breakthrough on top of this idea of prompt engineering of being able to have a single model was to also use language modeling as one of those tasks.

7:39And the idea of language modeling is you just predict the next word, which is very easy to get a lot of data for because you can use anything on the internet and so on. But it's actually incredibly hard. And to do it really really well, you have to learn so much about the world. If I'm just, I have the sentence like, I'm a New York City and I'm driving North too. And now you want to predict what's the next word? Maybe it's Boston, maybe it's Montreal, maybe it's Yale. But you probably want to put more probability on the word being Boston than Yale because there are more people probably driving to Boston because it's a bigger city.

8:13And so not only do you learn about geography, but you also learn just to predict that one word really, really accurately from that one sentence. You need to learn everything about the geography of the Northeastern United States. And now if you do this billions of times across the internet on all the chemistry articles and biology articles and so on, you'll learn world knowledge, you're in few world knowledge into that large language model. But only if you have enough parameters to learn it all. I need your help. I think they're straight away about the access to this training data. And some people say, ah, this is where incumbents really thrive.

8:46They have the consumer data. They have jobs. Actually, they have all this data. They can utilize that. And others say, no, there's so much open data today that actually access to data is highly democratized. Which side would you sit on and how would you think about that question? Because I don't know the answer. It's so funny because it's another question where they're actually both right So it's complicated in the sense that unsupervised data just raw internet text is easily accessible But there's still a lot of data sets out there that are not out there. They're actually stored in private databases and indeed if you want to to answer customer emails automatically.

9:29It's very helpful if you're Salesforce and you already have all those emails. You already have labels that people, this knowledge -based article, answered this email or answered this question, right? If you have that data, you can entrain that particular AI for that company to answer its questions from its customers automatically. You can do that much more easily than if you're a small startup and you need to just get access to that data and you need to get permissions and so on. And so that part is true. At the same time, the state before large language models, and we call them foundational models, because you can often build on top of them very easily, before that, it was even harder.

10:08Like, it would have been impossible for us as a small company to build a search engine that understands all of these different things in many different languages. It would have been unthinkable, but now we can actually because of these large language models and foundational models, we can have a general sense of understanding natural language, and you can at a small start up nowadays it builds in like 80 % solution, very quickly. You layer more and more specific data for your task on top as after you have an MVP, a minimum viable product, and then you make it really, really good. Now the big and common they can make it really good much more quickly, but you could get to at least an 80 % solution thanks to large LMS like more quickly also.

10:48A lot of people suggest in the venture community So you can poo poo this one, but like when they're denigrating kind of a lot of AI's so up to say they say oh It's a thin value layer on top of foundational models and actually really the value accrues to the foundational model layer Because it's this thin line on top. Is that fair? What do you think that's total bullshit? You're asking very good question that often have a more subtle answer than what would sit in a tweet I think there are some very thin wrapper companies out there that probably have very little mode but there are also companies that people don't realize the complexity to make it really work and they underestimate all the sudden all the other stuff that companies need to get right to build a viable business.

11:33So concretely, you know, you could think about Instagram, right? Instagram is like, what's the mode of Instagram? It's certainly not their AI and their backend and the brilliance of engineering. It's just like a fairly simple photo sharing app of some fun filters back in a day. None of that was rocket science. Turns out you can have more other than your backend AI model, right? It's distribution, it's partnerships, your sales funnel, processes, and so on. So there's a lot. And then there are also areas where the default large language model will not do as well. So for instance, if you want it to be more factual, more up to date, and have citations for the facts that it tells you, you need to have a search packet.

12:16And so at U .Com for instance, we've had to build this very complex search backend with a ton of data in it and knowing when to retrieve what facts from the internet so that your model you can ask about Messys, Miami, Switch or something. And it'll just talk to you about that even though the LM itself couldn't be trained. And if anything, you can think of these LMs as like reasoning engines, but you still need to feed them with the right facts and information so that they can reason over the right things rather than just sort of reason with what they remember. And their memory is a little bit like maybe your uncle who sometimes exaggerates some idea of the stories from the past and doesn't remember all the details exactly.

12:58He often still gets it right but not always and so you want to infuse the fact into the LM and then reason over it and that whole retrieval back end is also highly non -trivial, but But it's something that some VCs don't appreciate and understand the complexity of, and then they say, oh, like, you don't come to just a thin wrapper around a large language model, which is very far from the truth. Now we move on. I'm thrilled to welcome Duake. He learned found and CEO at contextual .ai to share his thoughts on more matters more, data model or data size. Yeah, great question. I think Sam Aldmann at this interesting quote where he was saying that he thought models would stop growing in size.

13:35GPT -4 kind of hit this ceiling. I think that's probably right, but not really because size doesn't matter It's just that data size matters even more than model size And I think the llama paper out of meta really brilliantly showed this if you train a smaller model on more data for longer Then you get a better model So you get more bang for your buck if you train it on more data rather than having more parameters But in an ideal world if you had infinite compute budget and infinite data then you would train the biggest possible model because that's the most likely to give you emergent capabilities as we call it an end of field.

14:11Does that take more time then? If you have solar models with more data, given you need to feed more data through the model, does it not take more time than if you needed less data to go through the model? It depends, so it's a trade -off here, right? But these big models also need a lot of data, so it really is a function of the number of GPUs that you have available. Let's say you have With 1000 GPUs you can choose to train a huge model on relatively little data and it will be okay but it will be under -trained. So you have some sort of optimal point where you can train the model to perfection.

14:42In the field we were underestimating where that optimal point is and it seems that data is much more important than model size when it comes to what's optimal. Final opinion to end this question of what matters more data model or data size, I'm thrilled to welcome e -mad at stability. We need to feed these models better data and other stuff. There should be no more web script data near. There should be national data sets, the good quality to feed these three range organic models and national and proprietary models and others. Now that's why when the research I signed that letter, I think there's six month pause to get all of our shit together.

15:12Before things go completely insane and make sure this is everywhere and everyone's investing in everything and it's just absolute chaos. There is no doubt this is more important than 5G. These models are like really talented grants that occasionally go off their meds, and you want to have the ones from Oxford, Imperial and Edinburgh, as well as the ones from Stanford, because they understand the local context, so they understand you better and they'll be better for that. As part of that, every nation will need their own data sets, which again have from broad cluster data, they will need their own open models that can stimulate innovation internally as well.

15:42Who owns the international data sets? Is that youngness? I think it should be the people, I think it should be open and public domain. There is no such thing as an unbiased model. Dali too, when OpenAI had that, they introduced the BIOS filter. Any non -gendered word, they'd rather random gender and a random ethnicity. So you type in Sumer -Slo and you get Indian female Sumer -Slo. That was a good feature, I gotta save somewhere. This is where you need national data sets, you need cultural data sets, you need personal data sets. They can interact with these base models and customize to you and your stories because you and I both have our stories that make up our psyche.

16:15Sure, and understand that context is so important to have AIs that can work for us, not on us. The reality is no models there are today will be used in a year. I have to, I so love these compilation episodes for the variants of thoughts and opinions if we move on to the next question now of where does value accrue in the next wave of AI? Does it accrue to the startups with innovation at their bones or does it accrue to the incumbents who have the powers of distribution in their hands? We're going to kick off today with Sarah Grove found and general Paul Ned conviction capital. Yeah, this is maybe a very discouraging answer, but I believe in like intellectual honesty.

16:53Like, classically, the only real advantage startups have is speed. And speed actually might matter more than ever when the environment seems to be moving at warp speed. Right? What's the quote? Some decades, nothing happens, and some years, a decade happens. I feel like that is happening right now. It's hard to make a large organization move at that speed. on the incumbent and vantage side, much ado has been made about this idea of a data mode. But honestly, there's a lot of data out there, and entrepreneurs are incredibly creative about collecting it and increasingly about generating it. And I don't think it's, oh, the incumbents are going to warn this one or the startups are going to win this one.

17:27Now we're going to join one of the spaces leading founders, Clem at hugging face to share his thoughts on where value accrues to start -ups or to incumbents. I think if you were thinking about AI as API, I agree. If you're thinking of AI as a more radical paradigm switch to build technology, right? And if you think about an AI startup as a company that is actually training models, creating new architectures, optimizing models themselves, I think it's a different story because this is really hard to do for the incumbents. And so I think the new startups have like an opportunity there to do things 10 times 50 times 100 times better than the incumbents.

18:12I want to hear again from Duet Healer and Contactual .ai on who wins startups or incumbents. It depends on where the data comes from. I think incumbents definitely have an advantage there, but only some of them. And a lot of the data is just freely available on the internet. And so the Lama model was not trained on any proprietary data. It was just trained on open data on the web. And there's a lot more data to be had there. And as the society were generating a ton of data every day to add to that big pile of data. So you can really train very high quality language models just on public data on the web.

18:46But I think if you look at the secret sauce to a lot of these other models, like Y is GPT -4 so awesome, a part of that is that they went through in order to get like special data that nobody else has. So it's allegedly that this whisper project where they're very good at transcribing audio because that would allow them to transcribe like all of the podcasts in the world which gives you very high quality language. If you can train on that language but nobody else has it that puts you in a position of advantage. How important is proprietary data? The main reason I would say YVC is that turning down start -up AI companies is because they do not have a proprietary data set to operate against, and they are defined as like a thin layer of generative AI on top of a foundational model.

19:30How important is proprietary data do you think for startups innovating in space? If you want to build a deep tech AI startup, then you really want to get a big data flywheel going. You want to start with a lot of data and then have a way to generate lots more data and that data is going to be your mode. But I think one of the interesting things about these large language models is that they're incredibly simple efficient or data efficient. So you can do cool things with them with relatively little data that just previously just wasn't possible. That unlocks all kinds of possibilities that just didn't exist even a couple of years ago.

20:04So on the one hand, yes, you need lots of data if you want to build like big AI first things, but at the same time, if you want to do a startup that builds on top of this technology, you need very little data to get started, bit of attention. But But one of the use cases I've been seeing now for GPT -4 is actually that people are using it to generate data and then they're training on that data with cheaper models. So GPT -4 might end up disrupting not knowledge workers necessarily, but it might just disrupt like mechanical Turk and is just an annotator on steroids. And you can use all of that data to get much more custom models that you can then deploy very cheaply on specialized use cases.

20:44That's a quite interesting development. We're going to rejoin Richard Sarchat at U .com to his thoughts on who wins Startups or Incumbence. Will the Incumbence acquire innovation before the start -up requires distribution? How do you think about that? The truth is distribution will be ever fully solved and it's a constant up -to -battle. Where do you think value most occurs if you were to bet on one in the next five years? Does the next wave of A create more value for Incumbence or more value for Startups? I think it'll be a mix. I think there are companies that have been fully activated. I see Salesforce having launched a bunch of really incredible features and made incredible announcements in their AI day that will make it harder for AI for service automation, for instance, because it's so core to their business.

21:28We've also seen Bing and Google copy what we have launched late last year with you chat. They've copied us in the sense that we've launched it earlier and then they launched something very similar three to four months after, at the same time, we don't see Google change and become a chat for a search engine. They have some features somewhere else, but the main Google experience is the same. That kind of big change will be hard for Google too because they make $500 million a day with privacy invading advertisements on that page. And so you don't just really nearly change most of that page and you get rid of the five six ads that are on top of that page followed by a bunch of SEO and microsites that are not as good as the ads so people click more on the ads.

22:10Like, you don't just replace all of that with a chat, right? Because you just lose hundreds of millions of dollars a day. And so there is still some innovator dilemma that will not make them change their main experience overnight. That it's been very carefully tuned. Every shade of blue has been abtested to death. It's very hard to make a massive change and improve that entire revenue overnight. So they'll slowly move carefully in their default experience and that's sort of one opening that we have But it's not gonna be easy you have to keep innovating and you have to keep working on partnerships too and try to stay clever and And a little bit paranoid as the startup CEO and in a space where you have these huge incumbents Trillion dollar something companies that would love to crush you.

22:52God. I mean the amount of views we get coming into this episode is awesome Now we're gonna join Alex at Nabla on who wins startups or incumbents So incumbents have a huge advantage through road distribution. That's one thing that's hard for us startups. But incumbents have many disadvantages. I think first they are very slow. You mentioned Adobe I was sure you would because this is the one example that everybody has, but then who else is the either was as fast as Adobe. So I think most of the incumbents won't move fast. And it been many cases. Do you not think they're moving faster? I mean, I don't know if you're called notion in incumbent, but no issue of move very fast.

23:31Adobe move very fast, Nivan, travel, experience management company now. It's solely on the AI. I think actually they have been very fast. That's fair. Some of them can move fast and it's not fair to say they are slow and that's it. I think the incumbent suffer from problems because in many cases they will do AI -enhanced features and this is what all the one you mentioned, did the notion you still have a document, you said, edit, you have a cursor, or the right LL, by the way, you can call chat GPT and to summarize a paragraph is what I can spread in, a little bit of AI dust on the magic dust on your existing product.

24:07Who knows how differently you can think about building knowledge for your company, probably something will come and destroy notion and Google Docs and all of them with a totally new paradigm that is made possible by AI. Certainly this incumbents won't do it. So I think the best incumbents can benefit from AI to be competitive in their existing markets, but I don't think they will invent totally destructive things that we killed, that's a, you know, Kodak invented a digital camera, and of course they never released it because it would kill their main revenue, which was the films. Yeah, I think that's why Google didn't innovate in the way that they could have done, because actually the cost per query that ChatGPT does is so much more expensive than the way that they do it today, and actually it would have cannibalized the whole business if they were like, Hey, let's embrace this new cross -stretcher, which is so much more expensive.

24:58I think the reason is simpler than that. The reason is nobody could predict that LLNs would be so useful and powerful before you train one at this scale. And who in the Google All Chart had the incentive to invest $500 million and just to see this without any business benefit for the company? If you add to that legal department, we're not that happy that you are releasing random childbugs, we can say anything. then nothing happens. And this is what happened to Google and probably also at Mid -Aware, nobody has the incentive to do that. And so, OpenAI has a private company that needs to show something to build products and they have this scarcity and the financial power and they did it.