Listener Q&A: AI Investment Hype, Foundation Models, Regulation, Opportunity Areas, and More

0:05Hey, everyone. Welcome to KnowPriors. Today, we're going to switch things up a bit and just hang out and answer listener questions about tech and AI. The topics people want us to talk about include everything from the evolution of open source models to the balkanization of AI, Elon AI, which I think will be super interesting to cover, regulating AI and AI hype in the investing environment. Let's start with the march of progress for open source models. I guess, Sarah, what have you been paying attention to and what are some of the more interesting things that you view happening right now? So there's nothing out there today in open source that is like GPT-435 or anthropic cloud quality, right?

0:38So there's one player out in front, and that's OpenAI. But I think the landscape has changed a lot over the last couple months. Like, Facebook Llam is quite good. Many startups are just using it despite its licensing issues, assuming Mark won't come after them. And then you have a number of other releases that have happened, right? Tomorrow just released a pre-training data set, which seems quite good. Stability just released Diffusion XL in the image gen space. and so I think the like larger dynamic is that there's been an increasing number of people and teams that now know how to train large models the cost of a flop is only going to go down there's a lot of investment in like distilling models and a lot of researchers would claim that you and I know that it's going to be 5x cheaper to train the same size model the second time around like once you've made your mistakes and know what you're doing and then you have these other accelerants, like you can use these models to annotate your data sets and increasingly do advanced like self-supervision.

1:35So if VCs are going to continue to fund foundation model efforts, including open source foundation model efforts, like if I were a betting woman and I am, I'd bet there's a three, five level model in the open source ecosystem within a year. And I didn't personally believe that would be true like a few months ago. I guess that puts it about two to three years behind when GPT 3.5 came out, though. And so do you think that's going to be the ongoing trend that there'll be a handful of companies that are ahead of open source by, you know, one or two generations? Yeah, I think that's like the status quo.

2:09So if we just straight line project, I imagine that will continue to happen. And the real question is like, can you stay in the lead if you are open AI and, and like get, get paid for that? Or is that the is that the objective of the organization anyway. Like I think, you know, if you have a great leader and a lot of resources and a lot of really talented people, that's not something I want to bet against. Is there anything you think is coming in terms of other big shifts in the model world, either on the open source side or more generally? Yeah. I mean, we should also talk about just like stuff that you're interested in investing in and generally paying attention to.

2:48But I think the big idea that's been very popular over the last few weeks are autonomous agents, right? And I don't think that that's like a, I want to hear what you think about this too. I don't think that's necessarily an architectural change, but for our listeners, the basic idea is to orchestrate LLMs in this like iterative loop towards some high level goal where they're doing planning and if memory and prioritization, reflection. And so you're not necessarily changing the architecture of the LLM itself, but this orchestration allows you to like do many new things, possibly the classic example being like make money on the internet for me.

3:24And there's a good number of hackers trying to figure out how to make agents that, for example, like analyze demand, find a supplier, set up a drop ship Shopify store, generate ads, then promote that store on social, right? The whole loop being like one call to an agent with this high level goal of make money on the internet for me. Do you think this stuff is interesting around autonomous agents? I think it's super interesting. And I think there's the old saying that the future is here, it's just not equally distributed. And I feel like that's one of those things that people in the AI community have been talking about for a while, and there have been very clear ways to do it.

3:59And then I think there's one or two people that went and implemented interesting things there in terms of auto GPT or other things. And then everybody's like, oh my gosh, this can happen. And I think a lot of people in the community are like, this is really cool. But at the same time, yeah, of course it can happen. You know, because effectively you have some form of context as an AI agent is acting and then you use that context to inform the next motion and sort of update, you know, the prompt or what the model is going to do. I think there's other forms of memory that people have been talking about that are super interesting.

4:29How do you make that a bit more of a cohesive part of how an LLM or AI agent functions? Because right now, effectively, every time you start a new instance of ChatGPT, a new chat, you've lost the context on all the other sessions you've had. And so a lot of what people are thinking about is how do I create ongoing context so that whatever chatbot or whatever API I'm using remembers everything else I've done with it over time or perhaps everything it's done with every other user over time. And then that becomes really powerful because you're effectively crowdsourcing an understanding of the world and then integrating it into an AI system and agent.

5:06And so suddenly you have global context. Like imagine if you as a person understood the life of every other person who's lived, and then you had all the context around what that means in terms of just how you operate in the world, right? And so I think those sorts of notions. I wouldn't operate anymore, Elad. We'd be hive mind. Yeah, exactly. It's just the hive mind. So I think that's where we're all heading. So you heard it here first. Okay, fine. And while we're on this topic of directionally AGI, there's been a lot of call for regulation of AI from, you know, Sam Altman to Satya to Elon Musk.

5:37Do you think AI should be regulated? You know, I think the first question is why do people want to regulate to begin with? And I think there's, you know, two or three reasons. One is if you're an incumbent, it actually really benefits you to get lock-in. And one of the best ways you can get lock-in as an industry is to have regulators get involved because they start blocking innovation and creativity and new efforts. And if you basically, there's that famous chart of where prices have gone up by industry and where they've gone down. And they've largely gone down in areas that have been unregulated traditionally.

6:12That's things like software or certain types of food products or other things. And then there's areas where prices have gone up dramatically. And that's education, it's healthcare, it's housing, it's the most regulated industries. So regulation tends to lock in incumbents. It means you have fewer people making drugs and you have fewer people doing all sorts of things that could actually be quite useful. So that's one thing. The second is, I think, some people are just scared. And in some cases, you could say, well, there's reasons to be scared, right? Like, what if the AI is used to unleash a virus or what if the AI is used to cause war?

6:44And if you look at the history of the 20th century, humans have done that pretty well on their own already, right? It's not a new concept that bad things will happen. And often they're driven by other people versus technology. And of course, technology can have accidents or can be misused, but fundamentally, usually people have driven a lot of the really bad things that have happened over time. And there's a really long history of doomers who are wrong. And I should say, by the way, on AI, I'm a short-term optimist, long-term doomer. I actually think eventually there may be an existential threat from AI, but I think in the next 10 years, everything will be okay.

7:16There may be accidents or maybe terrible things that happen, but fundamentally it won't be different from any other period. but if you look at the doomerism in the past it's things like you know public intellectuals worried about swine flu and nothing happened or pop you know a lot of people worried in the 70s about population collapse we're gonna have too many people in the world will starve and we're gonna have global famine and that didn't happen and so we have a lot of examples of people in the past kind of predicting doom and nothing happened and we also had that during covid where a lot of people said covid's the worst thing that ever happened to the world and then they would be hosting dinner parties unmasked inside with large groups, you know, later that evening.

7:53And so I just think you kind of have to look at people's actions versus their words. And fundamentally, you know, my view would be, let's not regulate right now, at least most things. I think the things that maybe should be regulated are things related to export controls. So there may be advanced chip technology that we don't want to get out of the country. And we already have those sort of export controls on other capabilities. We may want to limit the use of AI for certain defense applications. Do we really want a really smart, hyper-intelligent AI agent driving swarms of offensive drones or weaponry?

8:29And so there may be some need to do some sort of global regulation for things like that, or at least something like what we've done for chemical weapons or the like. And then I think in the long run, we may want to think twice about advanced robotics and their implications as AI becomes more of an existential threat to humanity. But overall, if I had to choose right now, I'd say don't regulate in the short run, except for those areas that I mentioned. And then I think that the big pivot point for regulation may actually come during the 2024 election. Because I think that's the moment that people will show examples of AI being used to influence the election or influence voting behavior, just like ads influence voting behavior, but AI could write better ad copy or do other things.

9:11I worry a bit about that becoming the reason that people claim that they should not regulate things, just like they got really aggressive about social networking. So I don't know. What do you think? I largely agree with that. I feel like it's worth describing what I think are the two more rational cases. By the way, I don't think, I think it's too early to regulate. I just want to make sure that's very clear. But I think the two rational cases I've heard, because I keep asking smart people that I don't think are taking cynical actions, why they're afraid or short-term afraid or why they think this makes sense.

9:45And the two things I've heard are, one, you know, this is unlike the past because of the speed of progression, like this hard takeoff idea of like, especially if you and, you know, I see you nodding and smiling. But when very, very smart people who are working at the state of the art tell me that they're concerned within a 10-year ban for humanity because of the ability of this current generation of models to be used to train the next generation of models and we're all very bad at thinking about compounding. I'm like, okay, that's not a completely unreasonable point of view. I think the other is more of a tactical thing for the industry, which is, as you said, for whether it be the election or some other trigger, there's a version of the reaction to this from people who are afraid or from political opportunists to go in two directions.

10:40One is mass surveillance, right? Or one is It's like complete lockdown, right? So I think the tactical thing is to like try to create a democratic process that gets ahead of it with something that's a reasonable path forward. But largely I'm just, I feel as far as very early to be figuring that out. And then you also have the problem of like, if you're talking about the more existential risks or sort of the AGI risk, like alignment research is very tied to capability research, right? And so it's sort of impossible to be like, we're going to stop making any progress on research, but figure out how to control this stuff.

11:18Yeah, absolutely. And I think related to that, I think it's really important to your point to separate out almost what I consider technology risk from species risk. Technology risk is there are some bad things that can happen due to technology being abused. And that could be a nuclear disaster or that could be an AI being used to shut down a pipeline or to crash a flight or to do something really bad. And those sorts of things already happen, but you could imagine it could accelerate it. In that case, you could literally turn off a bunch of servers. You could turn off every machine on the planet if you really needed to, and humanity would keep going, and it'd be a reset, but we'd reset fine.

11:52I mean, separate from that, there's species level risk. Like, is there an existential threat to humanity? And that's like an asteroid hits the planet and kills everybody. And I think a lot of the people who talk about these things mix those two things. And I think the true Doomer view is, well, AGI eventually becomes a species and we compete with it. And then it wipes out all humans. And in order for an AI to rationally want to kill everybody, you'd need some replacement for the physical world because eventually all the hard drives would burn out and the AI would die. if it existed as a species or a life form.

12:24So you need physical form for the AI in order for it to truly be an existential threat. And that's why if I were to focus on an area, it'd probably be robotics or something like that, because that's where you suddenly give physical form to something. And if you're like, oh, isn't it great if AI can build my house and AI can now build a data center and now build a solar farm and you're eventually now build a factory, you've basically created an external system that no longer needs people. And that's when I think there's real risk. And that's why on the 10-year time horizon, I'm not that worried because robotics and atoms in the real world takes a lot of time.

12:57So even if you have this hyper-intelligent thing running, the reality is if you really needed to, you could turn off every server on the planet. Yeah, I agree with the embodiment being like a key piece in this theory of the AIs are going to kill us. And we're pretty far away from that. Okay, so one question we got from listeners, and then I'm sure you get all the time is there's a ton of hype in the AI investing in startup world right now. What do you what do you think of it? Is it justified? Is it appropriate? Yeah, I think we've both lived through a couple of different hype cycles now, right?

13:28There was hype cycles around social and mobile, and then the cloud, and then, you know, multiple crypto hype cycles. And the reality is out of all those hype waves, interesting things emerged, right? And maybe in the standard hype cycle, 95 or 99 % of things fail. But there's still like the 1 % that work, or maybe it's 5 % work and 1 % end up being spectacular. And I think the hard part, usually is to know what's actually going to work because so many things seem so overlapping and similar. And so I remember when the mobile wave happened or mobile and social at the same time, a bunch of different people I know started mobile photo apps.

14:04And each one of those things took off. And so you'd suddenly see something go from zero to a million users in a week, just literally, it just spread virally. And none of them stuck, right? They all burnt out as cycles. And the only one that really stuck was Instagram. And in part, that's because Instagram emphasized filters, which things like Camera Plus already had. And then a part of it emphasized a network. It's like, let's have a follow model like Twitter. And that's the thing that really worked. And so it feels to me like if you'd gotten excited about the overall cycle, you were right. But if you got involved with the wrong set of photo apps or you built the wrong thing, then you were wrong in some sense.

14:36I guess you were right about the trend, but wrong about the specific substantiation of it. And it seems like the same thing here. And so I think often it's that question of, you know, Peter Thiel has a good saying, which is you don't want to be the first to market, you want to be the last standing. And so I think it's a similar thing here. How do you end up being the last person standing in that or last company? And it may be the same thing as being the first mover, right? It's Amazon and books or things like that. But sometimes it means you actually do something a bit smarter and you come later in the cycle and it's fine.

15:04So, I mean, are there specific areas you're most excited about in this wave or cycle or opportunities that you think are these things that are obviously going to happen or important to happen? Absolutely. I mean, well, and I want your ideas. Some of them are shared ideas, to be fair. But I would agree. Actually, at a data point, I was just over at OpenAI yesterday, and they're biased, perhaps, in a way that I'm also definitely biased. But a friend was saying they actually think that investors are being somewhat wary at the application level right now because they can't figure out what's going to be standing.

15:39Right. It's a very different competitive dynamic. but the market is extreme for researcher-led foundation model companies because everybody is pretty sure OpenAI is going to be around, right? And I agree the applications are going to be non-obvious, but as one example, like any investor that claims they knew image generation from text was a killer use case like a year or two ago, besides you, is just empirically wrong, given David's completely investor-free cap table and amazing business, right? So, David, in case you're listening, I still love MidJourney and want to invest. But that's why this podcast exists.

16:18But, you know, in terms of like specific things that I'm interested in now, I'd say like, I think there are a lot of things on the application side that are exciting. So to start with some of those, I think, you know, voiced into this and dubbing are going to be just a huge unlock for like content providers and publishers. Like, I'd like to back something in that space. I was just talking to some people at a very large financial, and they said the, like, biggest potential cost savings, you know, on order of tens of millions of dollars a year for us is in turning every line of code we have into explanations for a regulator.

16:55And that's at once, like, pretty specific to them, but also not, right? I think the areas of audit, tax, compliance, accounting, reconciliation, like, there's a lot of natural language understanding that could be better served by semantic understanding. And so I think that's an obvious area. I think annotation is changing again, right? And we can use, this is like a very specific idea, but we can use LLMs too much more here. We talked about agents. And then this isn't necessarily a specific company idea, but I think architecturally, retrieval is a field of active research, but the idea of personalizing LLMs with enterprise data is an important but very tricky one.

17:44You have to do data management. You have issues in scalability, sync, access control. You likely want to apply traditional IR if you own both retrieval and the model. You can do very magical things. And so I think the chat GPT retrieval plugin is super cool, but it doesn't just serve a whole host of use cases. And I think this entire half of the stack is still missing. So those are like a couple of the things that we're sort of explicitly like hunting around. But what are you paying attention to? Yeah, I mean, I think we have a lot of overlap, as you know. So I'm super interested in sort of voice synthesis, dubbing, and related, both in terms of infrastructure, but then also in terms of application areas.

18:26And so I think that's going to be a really big C change that perhaps people aren't paying enough attention to. I'm actually quite long on compliance in general. Like I've done a bunch of things like AgentSync and Medallion and other compliance related companies in sort of the old world. And so I think that's just an area that there's always going to be, you know, converting spreadsheets and offline processes and, you know, random checks and docs into code is, you know, really powerful. I think there's a lot to do on the upside. I actually am maybe on the other side of people who think that it's impossible to tell what's good and, you know, nothing's defensible and everything's just a wrapper on GPT or whatever.

19:02And I actually think there's tons and tons to do there. I mean, Harvey AI, which we're both involved with, I think is a great example on the legal side. But I think there's two dozen things like that to build over time. And it probably takes five years for all those things to get discovered and built and substantiated. So I don't think it's like this year, there'll be 12 of them. But I think like every year, there'll be a couple of really interesting ones. And then there's probably a lot to do on the tooling side, right? Obviously, Langchain is sort of a hot one in the area. but there's everything from people exploring vector DBs like Chroma on through to other forms of infrastructure.

19:38And so, you know, Lama and DAX and other things. So I just think there's a lot to be done at every level of the stack. It'll be interesting to ask what happens on the foundation model side, because to some extent the question is if we locked in a few of the leaders or is more to come. And I think the Elon Musk startup that's rumored to exist is sort of an interesting example of a new entrant. And back to regulation, you know, Musk was asking for a six-month moratorium on progress, which seems to me very self-serving if you're simultaneously starting an LLM company, you know. Just hold off until I catch up, right?

20:11Yeah. And if I was in that position, I'd do the same thing. Don't get me wrong, you know. So it's not meant as a diss. It's just meant as a, you know, remember people's incentives. but I do think there may be some interesting things to do on the foundation side and I do think some people are doing that in a vertical specific way they're saying hey we're going to build a healthcare specific model and we're going to build a you know Bloomberg did like their their Bloomberg GPT or whatever it was called on the financial side and so I think you can clearly see these verticals emerge and a lot of people obviously are debating will a general purpose model just cover all those use cases are you going to have bespoke sort of vertical models and what what parts of the actual logic and synthesis and sort of magic of these AI models comes from the fact that you've trained on a massive amount of data and language, and then you're applying it to a specific area with potentially unique data sets overlaid?

21:00Or is it something that's just, you know, that can be dealt with vertically specific and you don't need that broad-based understanding of the world? So I think that's a really interesting area of like exploration. And I have no idea what to predict there. I don't know if you have any thoughts on that? Well, I would agree. I think there is real opportunity for vertical-specific models where you can imagine that control for either a compliance or a safety or a just performance, like reliability of input data makes sense, right? As well as like if there are architectural differences because, For example, you have multimodal data in healthcare and pharma, right?

21:43If you are looking at protein structures and radiology and healthcare records, it's not clear that you would want to do that, train that in exactly the same way as a general web text model, right? Right. So I think that makes sense on the broader foundation model question. You know, we were talking about open source at the beginning. I think that OpenAI will continue to be a leader. Anthropic is very dangerous here, like really talented team. But the number of people who know how to train large models and the cost of a flop goes down. Right. And so I think there's like just a lot of incentive in the ecosystem for additional players to compete.

22:30what do you think is the opportunity for incumbents? Or how do you, how should they react to all this? Yeah, I think, you know, obviously with every technology wave, there's a differential split in terms of where market cap, revenue, employees, innovation, et cetera, goes in terms of incumbents versus startups. And, you know, every wave is a little bit different, right? The internet wave was almost, you know, it's probably 80 % startups in terms of value and 20 % incumbents. And then mobile was sort of the other way around. It was 80 % incumbents and 20 % startups, right? The big platforms for mobile were Google and Apple.

23:08But then you had a lot of interesting apps like Instagram and Uber and others emerge. For crypto, it was like 100 % startup value, right? And it feels like in this wave, it's probably 80-20 again, right? It's probably... Google will probably become a player, right? OpenAI is closely aligned with Microsoft. And then, you know, Salesforce with AI is probably Salesforce, right? It probably isn't a new company. It might be, right? I actually think certain companies are vulnerable for the first time because of these capabilities. And that includes everything from ERP providers where there's like a defensive moat through integrations.

23:36And obviously this could make integrating your data into multiple things really easy and fast. Instead of six months to roll out SAP, maybe you could have a next-gen approach where it takes a day or two on a new product, right, to do all the integrations that you would have spent six months on consulting fees for. and so there may be certain types of companies that are vulnerable but the reality is I think in most cases you know if an incumbent is already doing something and they're quick to integrate it then it works great the one area that may be really interesting is almost like there's probably room for a new private equity approach where if you think about how private equity companies bid on things they basically look at cash flows and costs and all the rest of it and if you can radically decrease cost for people heavy businesses by using LLMs as like a replacement for certain types of work, or at least an augmentation, then you can differentially bid on companies as a private equity shop.

24:27And so I think like people who do buyouts could have this as a strategy. I don't know that any of them will, because most of them tend not to be very technology savvy, but I think there's really interesting alternative things to do at scale there that tend to be kind of under discussed. The healthcare side that you mentioned earlier, I think is kind of fascinating because if you look at the cost of developing a drug, for example, say it's a billion or$2 billion to develop a drug, whatever it is, most of that early stage development is in the tens of millions of dollars at most. And so I think a lot of the default focus of people who don't understand healthcare very well is to say, I want to use this for drug development.