E69: Lost Tapes: Elad Gil on the Eve of the AI Boom

0:04Welcome to a special Lost episode of Turpentine VC from September 2022, featuring Elad Gil, an investor, entrepreneur, and podcaster who likely needs no introduction. Elad sat down to record this with me at a pivotal moment in AI's trajectory to talk about how he expected LLMs to impact technology. This was before ChatGPT, Perplexity, Claude, and many other AI applications we take for granted today launched. Elad's insights into the foundational shifts happening in AI prove remarkably prescient. Our conversation covers his first-hand experiences at Google, working on early ML systems, closed versus open source models, early NVIDIA takes, labor predictions, AI alignment, and some behind the scenes anecdotes.

0:47Please enjoy. I'm here today joined by a very special guest, Elad Gil. Elad, welcome to the show. Thanks so much for including me. So Elad, you've been in the tech space for over a couple of decades now, and people have been talking about AI for a long time. Why don't you give a little bit more of the history of how AI has been foreseen or how, you know, when you were coming up as an entrepreneur, people predicted what AI was going to look like. Yeah. To your point, I think the bar for what AI or machine learning actually means keeps going up. So in the eighties, it was things like, oh, maybe we can have like a plane that can land itself or things like that.

1:22And obviously those sorts of things have happened since then. And then I think in the, you know, the first real AI first company that really worked at scale to some extent is Google, where artificial intelligence kind of pervaded everything that it did early on in terms of thinking about how to rank search results or how to target ads. And so I actually started off between 2004 and 2007, I was at Google. And one of the things I worked on was the ads targeted machinery, which at the time was sort of the world's biggest semantic machine learning systems alongside search. And I think, in the early 2010s, there was a big wave of discoveries around things like AlexNet, CNNs, RNNs, deep learning, GANs, all these different innovations, which fundamentally really focused on things like machine vision and voice.

2:07And the interesting thing is that a lot of these, this sort of, you could almost call it like second wave of AI, if Google was the first wave, largely translated into incumbent value. In other words, companies that already existed, it's captured all the value of these models. So for example, you know, Netflix launched a really strong recommendation engine, Amazon launched Echo or Alexa. So you had this voice interface suddenly, Facebook had the newsfeed. But the reality is very few startups that were AI first actually turned into anything interesting over the last 10 years. And so this first wave of really compelling AI models translated to enormous incumbent value and very little startup value, which I think is a really interesting question in terms of why is that?

2:48And now it's happening because of transformers and these large language models, we're seeing a whole slew of new startups, and some of them are actually getting really interesting traction. And so that will be platforms like OpenAI and maybe tooling, like Hugging Face, or specific application areas like Jasper, Copy.ai, what GitHub has done with Copilot, or Replit has been doing in terms of its AI assistant, or AI mode, I think are really cool examples where you're starting to see people really scale using it in ways that you didn't see in the first wave of AI first. And again, incumbents work, but for some reason startups didn't in the prior wave.

3:26And do you see that changing or what needs to be true in order for that to change? You know, I think there's an interesting question of whether people are choosing the wrong application areas or whether the technology simply wasn't far enough along. And I'm guessing it's a little bit of both. I think some aspects of the technology have really been advancing in leaps and bounds. And you can clearly see that because of transformers, which again, were only invented about four or five years ago, and only really got started being used two, three years ago, right? And so it's a very recent technology innovation.

3:56I think some of the things that these models can do are really dramatically superior to what was done in the past. And the things in the past were already quite impressive. I mean, you talk to Alexa, or you look at some of the results that you get through Google or other things. It's actually, these things work very well in terms of large data sets. But for startups, finally, it seems like suddenly there's this opening, particularly if you start building like a tool or a series of workflows around a core use case. And so I think one of the areas where these things will be really useful or performant from a startup perspective is applications.

4:28We have white collar workers who do highly repetitive tasks and don't have a strong workflow tool overall. And so that'd be things like copy.ai for marketing copy. You know, and maybe obviously it's the coding side for various different tools. And it may be all sorts of other types of work. Like you could imagine a sort of copilot or companion or helper for tools around, you know, physician practices, legal work, summarization of documents, data entry, like there's all sorts of automation you can imagine. And then in parallel, you could imagine all sorts of really interesting applications on the consumer side.

5:01You know, do you have a better interface to search if it's, you know, a chatbot or do you have, Can you create a virtual companion? So I think there's lots of really interesting stuff that's still coming. Well, we'll get back to use cases in a bit. This is a great foreshadowing, but I want to talk about this Transformers paper that you've written about, the All You Need Is Attention post and why that is so significant. And maybe it's about the authors as well. Yeah. So Attention Is All You Need is a paper that came out in 2017. And there were eight authors on the paper and six of them have gone on to start companies.

5:37And of those companies, four are in the AI space or natural language space. And then one actually started off that way and turned into the crypto protocol near. And I'd say four of the five companies started, I believe in the last 24 months. And so it's this really interesting wave where the people who first came up with and developed this technology, and it was all Google authors at the time have since left to start some really exciting companies in this area. And I think Google almost had like a Xerox PARC style moment where, you know, just like Steve Jobs went and visited Xerox PARC and he saw the GUI and he saw the mouse and saw all the early things that then turned into core parts of what Apple launched later.

6:20And Xerox PARC made nothing off of it. You know, they didn't commercialize the technology. Google came up with this amazing breakthrough. And then they kind of sat on it a little bit. The people inside were trying to work on it and launch things. But I I think a lot of them got blocked by Google and sort of a degree of AI safetyism and a degree of fear of brand impact and things like that. And OpenAI, I think, saw this technology and they were really smart about saying this is the future. And then they went and they built GPT-1 based off of it. And so I think it's kind of a parallel moment to what happened with Xerox PARC where all the innovation happened in one place, but other people ended up commercializing it.

6:56And maybe the big difference here is instead of all the researchers from Xerox Park leaving and doing things, my sense is a lot of them kind of stayed there or joined Apple. I think in this case, a lot of the authors off of this paper decided, you know what, there's a great opportunity. We're building all these really interesting things inside of Google that could be useful outside. And so let's go and start building companies that are really useful here. And they're not doing direct analogs. It's more just they understand the technology is so fundamental that they can build new things that weren't available or possible before.

7:22So it's a really seminal kind of group of people. And it reminds me almost a little bit of like the Trader's, was it the Trader's 8 or something? I can't remember the number. The people who all left Shockley Semiconductor to start Fairchild. And then from Fairchild, there was people who started Intel and a variety of other things. Sequoia Capital came out of there. A bunch of really seminal places did. Totally. That's a great analogy. Let's fast forward to today. We've already mentioned some of the big companies and projects. Why don't you do kind of a deeper dive as to the differences between some of them, maybe in terms of design, trade-offs that they made, practical trade-offs, or more even philosophical.

8:00There's kind of this open source, closed source, or closed ecosystem conversation that's happening. Why don't you give a bit of an overview of the landscape today? Sure. So with a lot of these transformer models or large language models, which are a subset, a lot of the emphasis has been on how large these models are and how many billions of parameters do they scale to. And so a lot of the early thinking in the industry is the only way to win is to have these massive models, which means you need a massive amount of compute, which means you need to spend a lot of money on it just for the compute, right?

8:30You don't need that many people, but you do need a lot of hardware to train the models. And I think one of the really exciting things that happened more on the image gen side is what Stable Diffusion did or Stability.ai, where Imad, the founder, tweeted that they spent $600 ,000 to train their model, which is nothing, right? In comparison to the hundreds of millions of dollars that were going into some of these companies or billions in some cases now. Then they went ahead and open source. Once you open source something, it's really easy to modify it. That means people were very rapidly modifying things like the safety filters.

9:04There's this whole field of, I don't know what to call it, liberal arts or something, apply to technology where there's a strong emphasis on not wanting to launch things because they could be offensive in different ways. Offensive is very broadly defined in terms of anything that offends a specific subgroup of people who are focused on safety versus everybody in the world. There's obviously some things that are really offensive that most people would really want to avoid, but there's lots of other things that may just depend on your political orientation or your specific point of view. There's a very specific point of view that's represented in all these companies in terms of what safety actually means.

9:37And so they made it very easy to strip out that safety layer, which means that suddenly you had an open source model that anybody could use that could be modified in all sorts of interesting ways. So it could take any type of prompt. And that's created this explosion online of different people trying different things simultaneously. And the real breakthrough that I'm waiting for now on the open source side is something that can compete or reproduce the level of fidelity that the GPT models have on language side because these are all image-based things that have been open source now. And there's some open source models that are pretty cool on the language side.

10:10Luther and a few others are working on these things. Hugging Face has a bunch of these models. But fundamentally, nothing is as performing as GPT-3 or GPT-3.5. And then of course, GPT-4 is supposedly coming soon. So the question is, can you build that in an open source way? And what's the bottleneck? And is it data quality and cleansing? Is it some model structure? Is it something else. And so I think once that happens, you know, it really changes things, I think, in terms of the accessibility of this technology to everyone. Totally. It is fascinating. Ahmad, the founder of Stability, is tweeting quite a bit, and he's writing about how when people say AI ethics, as you were alluding to, they sort of have, you know, one definition of ethics in mind, but there's all over the world, very different kind of philosophies when it comes to ethics.

11:01And so he's really broadening that AI ethics conversation to kind of a global way where it's previously been among just a very few set of people who all think alike. So it'd be interesting to see how that evolves. Yeah, absolutely. I think he points out, and I'm probably going to get at this wrong, but he comes from a very specific religious background himself. And he's like, well, why isn't it reflective of my specific ethical framework versus a very specific framework that all these people who went to the same Ivy League institutions seem to have. And so it seems like a very narrow sliver of quote unquote ethics is represented in a lot of these things.

11:34And it'll be interesting to see how these things end up globalizing over time and incorporating multiple cultures. Yeah. Hey, we'll continue our interview in a moment after a word from our sponsors. How deep do you go to seek out an answer to a question? Maybe you've spent hours clicking the source links on an obscure Wikipedia page, or maybe you're even the type of person who checked out the entire shelf on the topic at your library. If you're nodding along, then check out GiveWell, an organization that researches questions about global health and philanthropy, even if a satisfying answer might require years of reviewing studies, talking to experts, and over 300 footnotes.

12:07GiveWell has now spent over 17 years researching charitable organizations and only directs funding to a few of the highest impact opportunities they've found. Over 125 ,000 donors have used GiveWell to donate more than$2 billion. Rigorous evidence suggests that these donations will save over 200 ,000 lives and improve the lives of millions more. GiveWell wants as many donors as possible to make informed decisions about high-impact giving. You can find all of their research and recommendations on their site for free. You can make tax-deductible donations to their recommended funds or charities, and GiveWell doesn't take a cut.

12:40If you've never used GiveWell to donate, you can have your donation matched up to$100 before the end of the year, or as long as matching funds last. To claim your match, Go to givewell.org and pick podcast and enter econ102 with Noah Smith and Eric Torenberg at checkout. Make sure they know that you heard about GiveWell from econ102 with Noah Smith and Eric Torenberg to get your donation matched. Again, that's givewell.org to donate or find out more. Let's talk about OpenAI for a second. Of course, we all have friends who work there and we respect what they're doing tremendously. I'm curious, just from a business model perspective, the trade-off of...

13:19If you imagine in another world where they didn't raise money for Microsoft, what sort of new... How they might have evolved or what sort of trade-offs they made by partnering with Microsoft, but what also Microsoft enabled in a way that maybe they couldn't have today. Do you see another world where OpenAI didn't do that and what could have been? Yeah, absolutely. I think OpenAI really feels like the juggernaut now in this sort of area. They recently launched a really interesting API called Whisper as well, which deals with voice and text transcription. And so I think they've really been building a lot of really interesting things from GPD to Dolly to whatever.

13:56You know, I think Microsoft provides capital and other aspects of a relationship to them. And obviously some of those things can be found elsewhere, particularly over the last few years where we had such a large capital bubble, right, where so much money was flowing into tech. And I think their choice of Microsoft as a long-term stable partner makes a lot of sense. I mean, Facebook did the same thing in the prior wave when they were really thinking of themselves as battling Twitter and Google and sort of everybody else. And they raised a lot of money from Microsoft. And I think that created a stable partner for a while there as well.

14:27And so I think there's a pretty good precedent in terms of Microsoft being a good partner now to a variety of companies as you're scaling in a way that didn't exist 20 years ago when they were kind of viewed as the evil empire that would quash anything that they viewed is competitive, you know? And so it feels like they've taken a really smart ecosystem abrasive approach now. How about business model? Like how do these companies become, you know, they're creating a lot of value. How do they capture value? How does, how does stability, for example, become, become, you know, how does the amount become massively rich?

14:57Yeah. How do these play out? I think it's going to really depend on the type of company you have and the area you're focused on. So if you're a API first company, like an open AI, you could imagine that you're charging off of some API usage fee. And then the other thing that platforms tend to do, although it may not happen here, is they tend to forward integrate into the apps that are working the best on their platform. And I'm not saying OpenAI has any plans to do anything like that. I'm just saying that's a generic pattern that happens often, but not always. And so you look at Microsoft and its OS, it forward integrated into the things that were used most on Microsoft PCs, which was Excel, Word, PowerPoint, and Access.

15:31And they basically bought or cloned multiple companies on their platform that were succeeding there and then subsumed them eventually. And Microsoft apparently was doing really shady things like they'd changed their OS to break any app that wasn't theirs that was competing with their apps, right? So that's one approach. The other approach is you truly have something that is open and you're just charging on the API side. And to some extent, you could argue AWS is more like that. It's kind of like a services layer that anybody can use. Obviously, with things like Mongo and others, they have in some cases kind of integrated a little bit and competed.

16:02But in other cases, they've left things reasonably open, just build on top of us. They never built Dropbox or whatever it is, even if they had a cost advantage there. It's just possible that OpenAI ends up more like AWS. I actually have no idea. I don't know their inner workings or plans. If you're a vertical application like a Jasper or a Copy.ai or a GitHub Copilot, you can obviously just charge for your service like any other SaaS company. And then I think a lot of the value, again, in this industry is going to get captured by incumbents. I think a good analogy is mobile where, you know, when the first real smartphones launched, like the iPhone and Android, there are a lot of really interesting mobile first companies like Instagram and WhatsApp and Uber.

16:42And then at the time, people were saying things like, oh, we're building the mobile CRM company. And you're like, well, mobile CRM is Salesforce on your iPhone. You know, it's not a new standalone startup. And I think the same thing will happen here. And so I think there'll be a lot of startups that will either be a technology hammer looking for a nail or they'll just be things where the incumbent will just win because it's a pure distribution and it kind of doesn't matter if your product is better or not in that case. Preston Pysh, M.D.: And maybe riffing on the open versus closed, Peter Thiel famously said AI is centralizing and crypto is decentralized, AI is communist, crypto is libertarian.

17:16How do you view that? Or can you kind of explain what you think you meant by that and how do you make sense of that claim going forward? I don't know exactly what the context was in terms of that statement. I think Google and a few other companies were perceived as having a really strong position around AI and machine learning-based technologies. And to some extent, the transformer world was one that Google effectively lost, right? They gave it away in some sense. And that doesn't mean they can't come back from it. They're an incredibly talented company with great people, but they created an opening for OpenAI and Cohere and A21 and all these other companies to emerge using technology that was invented in Google to impact the world in a big way.

18:00That's going to be true of applications as well. There's lots of things that Google could be doing with this. I think fundamentally though, if you do assume that at some point there will be really strong open source models of what stable diffusion or stability is doing in the image gen world for everything in this world, and OpenAI just open source whisper, for example, then you really change the nature of the business models. And there are lots of open source business models. And often what you end up doing is you end up with a hosted version or a SaaS tool or some equivalent, you know, a lot of the software stack that Amazon eventually ended up providing to people was something that consisted of a lot of open source components that everybody had to stitch together.

18:39And they'd rather just go on Amazon, right? You could use a Linux server for free effectively in terms of the software side, but then you had to do a bunch of stuff to make sure it was running all the time. And the first versions of AWS just simplified really basic aspects of their platform. So you can imagine something similar happening in the AI world or the machine learning world, where eventually maybe a bunch of the stuff is open source and then there's other services layered on top of it or tooling that you pay for or other things like that. But so far, that isn't the case. It really seems like early in a technology curve, there's often a technology advantage to being centralized, it's not always true.

19:15Crypto is a counterexample to that in some cases. But even the most quote unquote decentralized aspects of crypto end up being centralized if you look at them more closely. If you look at mining pools for Bitcoin, it aggregates, I think, up into five to 10 players. And you look at the number of code contributions that are accepted from who, it's a small number of developers. And you look at the number of exchanges and it's pretty centralized. And so I'm not saying Bitcoin is centralized, I'm just saying many of these things end up being a little bit more centralized than you'd think simply because there's scale effects to every business.

19:47And on the AI side, I think it's an open question in terms of open versus closed source, but I think stable diffusion gives real hope that there will be an open source version of all this stuff. How could you raise as another example of that, actually? Totally. And the question is how good will it be relatively? I mean, if you were Ahmad, what would your strategy be or what would the biggest thing you'd be trying to figure out be? or like, what would you be trying to do if you were going against the big guys and you were running stability or even just open source? Yeah, I can't say about stability specifically simply because I've thought about it much more deeply than I ever will.

20:22And so I think everything I'll say will probably be wrong, but I think in general, there's going to be two different aspects to it. One is, what are the areas that I want to impact from a platform perspective most? Is it image gen? Is it language models? Is it both? And then secondly, how do I create long-term sustainability? Because I may be able to keep raising money indefinitely up to a point. And then once that point is reached, what do I do? And so I start thinking about the SaaS services that I can wrap around the open source models, or are there some things that are proprietary that I somehow hold back?

20:51But again, that doesn't seem to be too much of a direction philosophically. And so maybe it becomes more just, what are the SaaS pieces that I build around this or other aspects of open source models that tend to work these days? I think I heard Balaji in some group chat talking about the intersection of open source AI and the token business model, do you see this juxtaposition potentially happening between the AI and crypto or what three worlds? You could imagine that a lot of the algorithmic trading and bots and behavior on the blockchain could be governed by autonomous agents that are programmed.

21:26You could imagine these machine learning systems adjudicating different aspects of DeFi or smart contract adjudication or finalization or settlement because fundamentally, to some extent, a lot of what's happening in crypto up to a point, I mean, there's lots of different things happening. But one of the things that's happening is you're wrapping money in code and contracts in code. And then you're having them execute on a large scale in parallel or in a way that's interactive. And so I sometimes wonder if the first place that a true artificial general intelligence or AGI, the AI that's actually smart and sentient emerges or sentient, I should say, emerges maybe on something like a blockchain simply because you have a utility function that's selecting for economic behavior.

22:08And if you look at game theory and you look at the evolution of human species and all the rest of it, you have a lot of things come out of that in terms of cooperation, in terms of competitive games, and in terms of a lot of pretty deep behavior is driven by how do you interact effectively economically with other groups? And economically may mean fighting for food, right? It doesn't mean necessarily the monetary system. And so I think it's a very fascinating utility function against which to involve an artificial organism as well as intelligence, or sentient, I should say. Yeah. And we'll get to the AGI question in a bit too.

22:44I wanted to return back to the use cases because you gave some examples that were both on the creative side, i.e. what we're seeing with ImageGen, but also on sort of just the language stuff more broadly, but then also So the caring side, you mentioned virtual companions, stuff like that. And it's funny because, or it's interesting because in the last decade, there were these tropes that, hey, don't worry, or the last to be affected by AI, the last fields would be the creative fields and the caring fields. And it was more self-driving cars and just other kind of more blue collar stuff that was at risk.

23:22But it seems that it's almost the opposite to somebody where self-driving cars has significantly slowed down and some of these creative and caring fields were really sped up. Is that accurate? And how do you see that playing out? I kind of view it as asking where is there highly repetitive, highly paid work that can be automated? And wherever that is, that's where I think machine learning will have its first impact. And I still think self-driving is very promising and exciting. But to your point, they used to really focus on building a robot that would replace farm workers, right? Versus saying which classes of our society either have a big labor shortage and you need a lot more and it's very valuable and expensive labor like doctors and nurses, or where are there other types of things that you keep doing over and over again that are workflow tool that automates that can actually help a lot.