20VC: Why Data Size Matters More Than Model Size, Why The Google Employee Was Wrong; OpenAI and Google Have the Advantage & Why Open Source is Not Going to Win with Douwe Kiela, Co-Founder @ Contextual AI

0:00Data size matters even more than model size. If you train a smaller model on more data for longer, then you get a better model. There was this Google memo from an internal Google employee who had written that open AI and Google have no mode. For me as an AI researcher, when I read that memo, I was like, this person has no idea what they're talking about. Welcome back to 20VC with me, Harry Stabbins. And stay with Delphur, the Into the World of AI with an expert who's been in the industry for the last 15 years. Dal Keeler, co -found and CEO of Contextual AI, building the Contextual Language Model to power the future of businesses.

0:37Last month, Contextual closed a $20 million funding round, including Bain Capital, Sarah Gwo, Eli Gil and 20VC. He's also on a junk professor in symbolic systems at Stanford University. Previously, he was the head of research at Hugging Face and before that a research scientist at Facebook AI Research. But before we dive into the show's day, did you know that over 50 % of your day is filled with tedious tasks? What would you do if you got half a day back? Well, you can, with Coda, the all -in -one platform that changes the way your team works. Together, and Coda just introduced an AI -powered work assistant to take the busy out of work.

1:13With Coda, your team's important workflows and content already live in one place, and Coda AI helps each team member focus on the highest priority work, even as priority shift across the team. By taking over a curry, and there's be honest, very tedious work often, Coda AI empowers team members to prioritize longer -term and strategic work to accomplish goals. Coda AI not only makes space for collaboration in each person's workday, but can make it easier to stay in the loop and share informed opinions. So if you want to work assistant that lets you get back to work, you can get started with Coda AI's day for free, head over to coder .io -2 -0 -VC, that's c -o -d -a -dot -io, and get started for free, coder .io -2 -VC.

1:59And speaking of tools we cannot live without like coder, we have to talk about Brexit. The all -in -one financial stack, trusted by founders. Founders have to think globally in order to open new markets, unlock cost savings, and gain access to new talent. That's what we're having the right financial stack is more important than ever, and that's where Brexit comes in. With Bracess you get a high -limit corporate card, a high -year business account, with up to $6 million in FDIC protection and Bill Pay all build with a global first mindset. Bracess enables you to operate in more countries and cancies than any other provider, so you can pay vendors, run payroll and make international payments faster.

2:35We all know that's a must have for start -ups at any stage of growth, so are you ready to learn more about Bracess' global first solution? Visit Bracess .com forward slash 20VC. that's b -r -e -x .com slash 2 -0 -e -c, and finally Angel List is fast becoming the center of the venture ecosystem. The startup world is just buzzing about how fast they've been shipping products that meaningfully improve the lives of both startups and fund managers and their investors. For startups, Angel List reduces the friction of camp table management, bank and fund raising, all in one place. Teams can focus on scaling and let Angel List handle the rest.

3:11Thousands of startups have moved their camp tables to Angel List in the past year. Anjelist also supports large ranch funds and their teams with an automated software first approach and the best customer service in the industry. Fun managers can focus on making great deals while Anjelist handles reporting, taxes, compliance and more. So if you're ready to scale your startup or fund with the platform at the centre of it all, visit AngelList .com for slash 20VC to get started. It's a bit... You are now arrived at your destination. Now I am excited for this and we've chanted before we've known each other for a while, but thank you so much for joining me today.

3:51Yeah, thanks very much for having me on the show, I have a big fan. Oh, it's very, very kind of you. I literally paid you $25 ,000 to say that. But my question to you is, it's such a hot space, and there's very few people who've actually been in it for a while. You are one of them. How did you first make your way into the world of ML and NLP first? My journey has been a little bit unusual actually. So when I was in high school in the Netherlands, I wanted to be a cool kid during the day, but at night I was secretly fascinated by computers. So I taught myself to code. So then by the time I had to go to college and go study something, I thought I already knew everything about computer science.

4:29So I decided to study philosophy instead, radical departure from what I had been interested in at the time. but it was fascinating. I use it still every day, I think. But then at some point in my career, it became clear that I had to start making money. So I needed a real job. Then philosophy is not really a real job. And I did some logic in between foundations of math, which is also not really a real job. So I decided to study computer science after all. So I went to Cambridge, in the UK. And so that's really where I started doing NLP, natural language processing. One of my internships was done at Microsoft Research in New York with a very famous researcher called Leonbo -2, who is Young Lecunse kind of, I wouldn't say sidekick because that doesn't really do justice to what he's done.

5:11So one of the Godfathers of Deep Learning and I had the opportunity to work with him that was really an amazing time. So afterwards when Jan and Leon started fair, Facebook and I researched, I joined that out of my PhD and that really kicked off my career. I actually supposedly was part of the prep for my interview with Jan. amazing amazing person but I do want to talk about that five years that you spend at Facebook say I research team it's such a transformational team as you said with incredible individuals like Jan and Neon what are your biggest takeaways from that experience and how did it impact how you think today I'd learned so much there mostly around how to focus your research direction so I think initially I was doing all kinds of weird stuff and it took me a while to figure out that a very clear real -world application for the research that you're doing makes it much more valuable than going off on the tangent and maybe being a bit too far ahead of the rest of the field.

6:02A special place and I think actually they don't get enough credit for the impact that they've had on the world. Facebook or meta in general actually. So almost every web app in the world runs a react which is an open source project coming out of meta. When does contactual come from? What was that aha? I've got to do this. This is now the idea and the time. Yeah, so this really started at the beginning of the year. We just saw This great need after Chatchy PT have gone viral. We saw this great excitement in the world that at the same time a lot of disappointment about it not being quite ready yet for real world adaption in enterprises where you actually want to use this technology.

6:38So we decided that now really is the right time to build a company to try to tackle that and we think it's still very early innings in the game. So I think a lot of people sometimes think that the game has been played but is just getting started. Okay, so you said that about not being ready for like traditional adoption. I think a lot of the general public would say, absolutely, it's cool. What makes it not ready for general adoption do you think? So there are a couple of just really big issues. Hallucination, these models make things up with very high confidence. Attribution, we don't know why they're saying what they're saying.

7:09We can't really trace it back to anything. There's compliance issues, so we can't really remove information from them. Tricky from a GDPR perspective, for example, we can't revise information, we can't keep it up to date. There's massive data privacy issues where you have to send your very valuable Company data if you're an enterprise you have to send that to somebody else's servers These models are also quite inefficient still so you can make them much faster What we are building at contextual is a different kind of language model We're really thinking about this as the next generation of language models where we think about it from first principles for Enterprise use cases and what that means is that we want to solve all of these problems by being a bit smarter about the architecture and the architecture where specifically basing it on this retrieval augmented generation which is something that me and my colleagues at Fair came up with in 2020 and what you do there is you decouple the memory from the generative capacity of the large language model and this allows you to ground the generations from the language model in the things you've retrieved in your memory essentially.

8:13So you get much less hallucination, you get attribution for free, you can always update the memory so you can remove information on the fly, you can add things, you can revise them, you can have a stream of memory. It's much more efficient because you're compressing a lot of the compute inside the memory, and you have a very clean separation between the data plane and the model plane as they call it, which means that you can have better data privacy guarantees. I have so many questions for you. First off, you said that about an attribution of knowing where it comes from and being a bit of a black box.

8:43I had someone on the show the other day and they said, open it all closed. You still don't really know what's going on in the core foundational model layer. The open or closed is not really the point, you still don't know. Is that true? And how do we think about actual true transparency of knowing what's going on and why it's producing what it is? So we're not going to be able to know why a neural net what does what it does at the scale that neural networks operate. So this is kind of like your own brain, right? I think your behavior is relatively predictable. So that goes for every human, right?

9:13We all like a predicted shutter's actions, but I have no idea what's going on in your brain. And I will never know. There's no way I can. The only way I can kind of find out is by asking you. But if you train the architecture the way we are training it right now, then at least you make sure that the model has learned to rely on the information that it finds. And that gives you much stronger attribution than if it's just predicting the next word based on what it has seen before. because it doesn't have this ability from birth, basically, to find relevant information and ground its generation on that thing it found.

9:47Another thing that I have to ask, you mentioned the word hallucinations that I had e -mad at stability on the show, and he said, hallucinations are a feature, not a bug, which I thought was a very tweetable statement. You agree hallucinations are a feature, not a bug. It's a great quote, but as always, it's a bit more nuanced than that. I think in some cases it is a feature if you want to use a language model for creative writing and if you want it to be really really creative Then you probably want it to hallucinate. So in a way it's a spectrum of ground it and hallucination If you really care about the language model doing the writing and you want to deploy it in an enterprise critical situation Then you really don't want it to be creative.

10:25You don't want it to hallucinate. You just wanted to do what it has to do But if you want to use it for a creative writing exercise then sure you can have it to hallucinate because you're going to revise whatever it gives you anyway. You also mentioned the multi -model aspect. Another guest on the show said that the winners in Startup Land will be determined by those who can bluntly switch models faster than anyone else, and that today's models will be unusable in the year. Do you agree with them? At this particular point in time, probably yes, but just because the field is moving so incredibly quickly.

10:56And so I think that in the next year we're going to see lots of other models coming out and if you can have a language model, agnostic AI company that relies on language models then that would give you a competitive advantage. But at the same time it also runs the risk of you relying on other people's language models. It's a bit of a tricky situation. Do you think there will be many more startup language model companies? This technology is really going to change the world than every aspect of it. So there are definitely of the couple of incumbents, but they are also focused on very specific parts of the market.

11:30If you look at Anthropical Open AI, I think they're really chasing for this idea of AGI and they're relatively consumer facing if you focus less on this idea of artificial general intelligence and you want to have something a bit more like artificial specialized intelligence where you just need the model to do what it needs to do. You don't need it to know about Shakespeare or quantum mechanics or things like that. You just need it to solve your business's problem. In that case, I think there's still a lot of room for innovation and that's where we are trying to innovate. We've seen size of model matter less and less it would seem.

12:03How do you think about the importance of size of model today? And does it matter as much that you used to and will it matter even less with every year, month, day? Yeah, great question. I think Sam Aldmann at this interesting quote where he was saying that he thought models would stop growing in size. GPD4 kind of hit this ceiling. I think that's probably right, but not really because size doesn't matter It's just that data size matters even more than model size And I think the llama paper out of meta really brilliantly showed this if you train a smaller model on more data for longer Then you get a better model So you get more bang for your buck if you train it on more data rather than having more parameters But in an ideal world if you had infinite compute budget and infinite data then you would train the biggest possible model because that's the most likely to give you emergent capabilities as we call it an in the field.

12:55Does that take more time then? If you have smaller models with more data, given you need to feed more data through the model, does it not take more time than if you needed less data to get through the model? It depends, so it's a trade -off here, right? But these big models also need a lot of data, so it really is a function of the number of GPUs that you have available. Let's say you have a 1000 GPUs you can choose to train a huge model on relatively little data and it will be okay but it will be under -trained. So you have some sort of optimal point where you can train the model to perfection.

13:26In the field we were underestimating where that optimal point is and it seems that data is much more important than model size when it comes to what's optimal. If we take this to a net it's like layer deeper. Data more important than model size. What does that mean then in terms of who's Vontaged. Does that mean that startups are more advantage that she than incumbents? I don't understand who's more advantage than that case. Is it incumbents? Because they already have existing massive data modes? It depends on where the data comes from. I think incumbents definitely have an advantage there, but only some of them.

13:58And a lot of the data is just freely available on the internet. And so the Lama model was not trained on any proprietary data. It was just trained on open data on the web. And there's a lot more data to be had there. And as the society were generating a ton of data every day to add to that big pile of data. So you can really train very high quality language models just on public data on the web. But I think if you look at the secret sauce to a lot of these other models, like Y is GPT -4 so awesome. A part of that is that they went through enormous lengths to get like special data that nobody else has.

14:34So allegedly they did this whisper project where they're very good at transcribing audio because that would allow them to transcribe like all of the podcasts in the world which gives you very high quality language. If you can train on that language but nobody else has it that puts you in the position of advantage. How important is proprietary data? The main reason I would say YVC is turning down startup AI companies is because they do not have a proprietary data set to operate against and they are defined as like a thin layer of generative AI on top of a foundational model. How important is proprietary How much we dated do you think four startups in a rating in space?

15:09If you want to build a deep tech AI startup, then you really want to get a big data flywheel going. You want to start with a lot of data and then have a way to generate lots more data and that data is going to be your mode. But I think one of the interesting things about these large language models is that they're incredibly simple efficient or data efficient. So you can do cool things with them with relatively little data that just previously just wasn't impossible that unlocks all kinds of possibilities that just didn't exist even a couple of years ago. So on the one hand, yes, you need lots of data if you want to build like big AI first things, but at the same time, if you want to do a startup that builds on top of this technology, you need very little data to get started, a bit of attention.

15:53But one of the use cases I've been seeing now for GPT -4 is actually that people are using it to generate data and then they're training on that data with cheaper models. So GPT -4 might end up disrupting not knowledge workers necessarily, but it might just disrupt mechanical Turk and is just an annotator on steroids. And you can use all of that data to get much more custom models that you can then deploy very cheaply on specialized use cases. That's a quite interesting development. Pre -trained data changes a lot. Can you just help anyone who doesn't know on Sam? What is pre -trained data? How does it change the game for a lot of companies that don't have existing data most.

16:31Maybe it's useful to kind of go through the steps if you want to build your own chat GPT like what do you need. So the first thing you need is a core pre -trained model and this tends to be just trained on the web, the task you're training it on is just next word prediction. Then once you have that core model, then you want to do supervised fine tuning. So essentially you want to fix the user interface to that model because the model doesn't really know how to follow instructions for example. So you want the model to listen to you, but it has only been trained on predicting the next word So it doesn't really know how to do that.

17:03So that supervised fine tuning. That's also proprietary data You can get a much better model out of that and then the final step is RLHF Reinforcement learning from human feedback where you get this feedback loop to make the model even better for your specific use case Even if you don't have signal at the word level you just have signal at the sequence level So you can tell it like okay, that was a good response or that wasn't a good response but you can't tell it like what did you do wrong necessarily. If you go through those three steps, then you get a chat GPT. It's as easy as that. Which company do you think has the best data acquisition flywheel?

17:36When you look at them today, who do you admire and respect most? Open AI. They haven't even really trained as far as I know on the data that comes out of chat GPT going viral. And so they had chat GPT. When viral, this led to just giant, giant data mode that they haven't even really used yet. So in terms of data modes and maybe you've seen this come by actually there was this Google memo from an internal Google employee who was written that open AI and Google have no mode for me as a AI researcher when I read that memo I was like this person has no idea what they're talking about. Why are they rolling down on the stand?

18:13These places have a giant mode because as I said this is really all about data and And OpenAI has this very deep understanding of how people want to use language models. Basically nobody else has. And they have this giant economy of skill where they can serve up language models very cheaply because they get so many requests coming in at the same time. So they have a giant mode. So I'm a big fan of open source, right? I would like it to be true that with open source we could just keep up with all of that, but I think that's just incredibly naive. What do you think of the biggest challenges that they face?

18:46because I think we will dismiss Google quite significantly if I'm honest. And then Bard came out and was pretty impressive. How do you evaluate Bard and Google's display actually? Language model evaluation is a whole separate topic. It's a super interesting question. Actually, I've been fascinated by AI evaluation for a really long time. And the answer is we don't really know how to evaluate the quality of these models anymore. So what we've seen people do in the field now is they're using GPT -4 to evaluate the quality of other language models. That just feels wrong. So I think there's a giant opportunity in the market actually for a startup or several startups becoming like the Moody's or the S &P The folks who evaluate the quality of AI for specific use cases because nobody really knows It's really the Wild West out there and one of the big problems for example is data contamination where a bunch of these language models are Trained on the things that they are being evaluated on so GPD4 looks like it's an amazing coder but it might also just be trained on the data that is evaluated on, which means that it's not actually that great of a coder.

19:50How do we know a yard state for progress or measurement? What is the right way to approach AI, measurement and effectiveness? So there's the Stanford Helm project, the holistic evaluation of language models and these benchmarks where you look on static test sets, how good language models are, how good they are on this static test set. But I've been arguing for a long time that that's just completely wrong anyway and we need to do something that's much more dynamic. So ideally what you want is to see how easy is it for an adversarial person to mess with your model. The harder that is, the better your model is.

20:27And so it used to be very easy to come up with adversarial attacks where these models would just completely mess up and it's getting harder and harder over time. So success rate of an adversarial attacker, that's something that can keep evaluating over time. So we need humans to evaluate these models by trying to break them. When you say about kind of the adversarial entrance and abilities, does this mean we'll have like an entirely net -esgeneration wave of cybersecurity companies around model protection? Oh, absolutely. Yeah. That's completely going to change everything. So these models can also get contaminated with data itself, right?

21:02So we need the security layer on the generations of the models. You can do all kinds of prompt injection attacks. These models right now, they're still mostly just producing language, right? But they're starting to produce code and instructions and actions. And when that happens, then you can mess with a model to get it to produce actions that you really don't want it to produce like removing your entire database and things like that. So that definitely is something that you want to think about. My question there is, OK, but is that built by contextual? Is that built by OpenAI in terms of a data contamination checker in terms of a health checker?

21:35Or is that a next generation semantek? You name it security company external that you provision internally. There are lots of startups looking at this opportunity. Standard security companies are also looking at this right now. So it's very obviously a very big new threat surface. But I don't think that the actual foundation model builders, like open AI and contextual, are going to build that technology in -house. It's probably going to be an external audit. How much of a concern is data contamination at this stage do you think? If you're sitting in open AI's day, how do you think they discuss data contamination?

22:11I think they're aware of it. They're starting to invest more in evaluation. They probably should have done that sooner. They have an army of annotators now, human annotators who are checking their models, so they probably have a pretty good sense of how good their model actually is. But obviously they're not going to share that with the world. I had Jan McEun on the show who we discussed earlier. Yeah, obviously a very big proponent of kind of open models. Where do you sit in terms of the model that rules for the next five, 10 years? And is it different for the model that rules for the next five years versus that that rules for the next 10?

22:43So the way I think about the language model space is kind of as a pyramid. So at the top of the pyramid, we have these frontier models. So these are GPD4 and entropic models and things like that that are much better than everything else, but also much more expensive and much bigger than everything else. And then at the bottom of the pyramid you have open source models, anybody can train on them, anybody can fine tune them on their data. That's a very fruitful area for research. But I think the most interesting part is kind of the middle piece of that pyramid, where you have the most bang for your buck.

23:18So that's from a business perspective, the most interesting part where you have mid -size models that have capabilities that you don't really see at this bottom of the pyramid, that you can monetize it in various ways. It's not going to be the case that there's just one model that wins everything. It's going to be lots of models at different layers of this pyramid being used for different kinds of applications. So if you have very strong AGI requirements, you probably want to have a frontier model. If you care about it a bit less, maybe you want to have artificial specialized intelligence. If you care about it even less, then you can just take an off -the -shelf open source model.

23:52So there is always going to be a place for open source. But why I don't think that open source models will move up that pyramid to the frontier is because they're just too expensive and this whole flourishing that you see right now of open source models that basically comes from meta's generosity in giving Lama away for free and if they had that then you wouldn't see that. We saw Elon's petition, he mad at stability very much was in favor of it on the show. Where do you sit in terms of Elon's petition and how did you read it? This is the petition where he asks everybody to stop working on AI so that he can catch up.

24:25A lot of the narrative in the media right now is really driven by self -interest from a bunch of folks in the field. So the whole kind of existential risk debate, I think it actually comes from a very good place and a lot of people are worried about this and I think there is a non -zero probability of AI extinction risk. So it's something we need to think about and a lot of smart people are thinking about this like Benjo and Hinton and all of these folks. For me, she said there's a non -series charm on stuff. So there is just a non -zero, but very, very, very small chance. There will be some sort of paperclip maximizer scenario.