AI Revolution: Top Lessons from OpenAI, Anthropic, CharacterAI, & More

0:00Look, this is the beginning of something amazing because there's no limit. This is right now an inflection point where we're sort of, you know, redefining how we interact with digital information. These are the fastest -going open -source projects. These are the fastest -going products. Some of the fastest -going companies we've seen in the history of the industry. We, for a long time, really folks are building our own infrastructure. We have hundreds of thousands of servers. He said, well, I think we can get by with like 500. And I said, OK, I think we can find 500 cases somewhere. And I remember you dead band thing.

0:32Do it, I'm talking about $500 million. The internet was the dawn of universally accessible information. And we're now entering the dawn of universally accessible intelligence. The AI revolution is here. But as we collectively try to navigate this game -changing technology, there are still many questions that even the top builders in the world are grappling to answer. And that is why A16z recently brought together some of the most influential founders from OpenAI and Thropic, Character AI, Roblox, and more, to an exclusive event called AI Revolution in San Francisco recently. And in today's episode, we share the most important themes from this event Starting with the economics of AI, but we also touch on broad versus specialized models, and which ultimately may win, the importance of UX, and also whether we can expect scaling laws to continue.

1:29By the way, several founders comment on what they're seeing there, including Nome Shazir, lead author of the preeminent Transformer Paper from back in 2017. Now I won't delay us any longer, other than saying we've got a lot more coverage of the spent coming. including how AI is disrupting everything from games to design, how two important waves in machine learning and genomics are colliding, and what we can expect from the enterprise. But in the meantime, if you would like to listen to all the talks in full today, you can head on over to a60Z .com slash AI Revolution. As a reminder, the content here is for informational purposes only.

2:11should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the company's discussed in this podcast. For more details including a link to our investments, please see A16Z .com slash Discochars.

2:38Alright, let's start with Martín Casado, General Partner at A16Z giving the Y now and also how the economics of the space may finally be coming together. I will give you the punchline up front. The punchline is if you've ever wanted to start a startup or join a startup, now is a great time to do it. But how early are we in the trajectory of this technology? For example, the microtrip was invented in the late 50s, but it wasn't until the turn of the century when Steve Jobs famously put a thousand songs in your pocket. So just how much opportunity is still in the table? Okay, so what is the narrative been for AI over the last 50 years?

3:20The narrative is this episodic thing with summers and winters and all of these false promises. I remember when I joined PhD in 2003, for my cohort that joined, I would say 50 % of the people were doing AI. This was like when Beijing stuff was super popular. And then within three years, everybody's like, AI is dead, right? And so it's kind of been with this love -hate relationship with it for a very long time. But if you look at all of the graphs, we've made tremendous amount of progress in the last 70 years. And along the way, we've solved a lot of very real problems, right? Like way back in the 60s, we're doing expert systems that you're still used for diagnosis, right?

3:56Like we're very good at beating Russians at chess. You know, we're doing self -driving cars, we're doing vision, there's just a lot of stuff that we've solved. And so much so has become a cliche that every time we solve a problem, we're like, oh, well, that wasn't really AI, right? So we just keep moving the goalposts. So we've had steady progress, we've solved real problems. And not only that, it's been a while now that we've been better at humans for some very important things. For example, like perception or handwriting detection. It's been about 10 years since we've been better than humans at entity identification.

4:25And not only that, we've actually gotten very good at monetizing this, particularly for large companies. And so, as we all know, there's been a ton of market cap that's been added to companies like Meta and Google and Netflix by using AI. So I think the question we should all ask ourselves is, why hasn't this resulted in an actual platform shift? And by platform shift, I mean, why has the value accrued to the incumbents? and why hasn't there been a new set of AI native companies that have come up and displaced them, which we've seen in many other areas, right? We saw that in mobile, obviously we saw with the microtrypt, etc.

5:00But I'm going to argue is that the capabilities have all been there, but the economics just haven't for startups. So if you step back and you look at the standalone case for AI academics, not like what a big company can extract from it, but to startup, it's actually not been great. I mean, to begin with, a lot of the sexier use cases are pretty niche markets, right? Like, you know, it's great to beat Russians at chess. Like, maybe it's a useful tool that you can apply to solving bigger problems, but that not itself as a market. I actually think the second point is the most important point and it's pretty subtle.

5:32Many of the traditional use cases of AI require correctness in the tail of the solution space. And that's a very hard thing for a startup to do. For a couple of reasons, one of the reasons is if you have to be correct and you've got a very long and fat tail. Either you do all of the work technically or you hire people. So often we hire people, right? And for start, start hiring people to provide solutions as a variable cost. And the second one is because the tails of these solutions tend to be so long, think something like self -driving, where there's so many exceptions that could possibly happen, the amount of investment to stay ahead increases and the value decreases, right?

6:06You have this perverse economy of scale. So we've actually done studies on this, and it turns out many companies that try to do this as startups end up with non -soft for like margins. They're lower margins and they're just much harder to scale. Of course, with robotics comes the curse of hardware, classically a very difficult thing for startups to do. And if you really think like, what is the competition of most use cases of AI, it tends to be the human and traditionally it's stuff like the human brain is really good at, like perception, right? Like the brains that we have have evolved over a hundred million years to do things like whatever pick berries and evade lions or whatever it is.

6:36And it's incredibly efficient at doing that. So this leads to something that most investors know, which we call the dreaded AI mediocrity spiral. And what is it? It's very simple, which is, let's say a founder comes in and they want to do an AI company, and they're going to use AI to automate bunch of stuff. Of course, correctness is really important, and they wanted to look at it first, so they hire people to do it instead of the AI. Then they come to us, we invest in them, and I join the board. Then I say, listen, this is great. You need to grow. And they're like, oh, man, we need to grow this AI's hard.

7:06Like, the tail's very long. I'm going to hire more people. And now you're on this treadmill of continuing hiring people, and this is one of the reasons why so many startups that have tried to do this just haven't had kind of this breakaway economics, and the value accrues to large companies that can actually seek these perverse economies of scale. But you know, market transformations aren't created when the economics get 10 times better, they get created when they're 10 ,000 times better. So what is the learning from the last, say, 70 years? It's not that the technology doesn't work. It's not that we can't solve the problems.

7:37It's not even that we can't monetize it. Big companies are great at monetizing it. It's that it's very, very hard for startups to break away. And if startups can't break away, you don't get a transformation. But what about the current wave, where the everyday consumer can prompt LLMs with natural language and have it output of variety of things, from conversations to images to even 3D models? So this wave is very, very different and we're already seeing productive viable businesses, right? Like I like to call them the three C's, there's creativity, like any component of a video game, you can automatically generate, there's companionship, which kind of more of emotional connections, and then there's a class that we call Copilot, which will help you with tasks.

8:20These are already emerging as independent classes. So remember the properties of AI previously that made it difficult to build a startup company. So none of these really apply to this current model. The first one obviously, these are large markets that this is being applied to. It's like arguably all of white, color, work, even just like video games and movies is like $300 billion in market. These are massive, massive markets. The second one, again, I think is, you know, the most important point and maybe the most subtle. In this domain, correctness isn't as much of an issue for two reasons. One of them is when you're talking about creativity, the first C, there is no formal notion of correctness, really.

8:58And I think what does it mean to be incorrect for like a fiction story or a video game? I mean, for sure, like you want to make sure like they have all their fingers, but even then do you really, in sci -fi? And so we have absolutely adapted to use cases where, you know, correctus is not a huge issue. The second one is a little more subtle, but I just think it's so important, which is the behavior that's developed around these things is iterative. And so that human and the loop that used to be in a central company is now the user, so it's not a variable cost to the business anymore. The human and the loop has moved out.

9:27And as a result, you can do things where correctus is important. Like, for example, developing code, because it's iterative, so you're constantly getting feedback and correction from the user. And I want to talk about this brain portion, because I think it's so interesting. I'm not a neuroscientist, but for these types of tasks, the silicon stack is way better than the carbon stack. So if you think about it, traditional AI, a lot of it is doing stuff like the 100 million -year -old brain is doing, right? The one that's been fleeing predators or picking strawberries or whatever it is. And that's very, very hard to compete with.

9:58Like, remember, if you have the CPU GPU set up a self -driving car, some of these kits are like 1 .3 kilowatts, where the human brain is 15 watts. So economically, that's very tough to compete with. The new Gen I wave, it's kind of competing with like the creative language center of the brain, it's like 50 K years old, like it's much less evolved. And it turns out it's incredibly competitive, so much so that you actually have the economic inflection we look for for a market transformation. So let's just go ahead and break down the numbers very quickly. So let's say that I'm our team wanted to create an image of myself as a Pixar character, right?

10:30So if I'm using one of these image models, the inference cost let's call it, you know, a tenth of a penny, it's probably less than that actually. Let's say it takes one second. If you compare that to hiring a graphic artist, let's say that it was a hundred bucks in an hour. I've actually hired graphic artists to do things like this. It tends to be a lot more money than that, but let's conservatively say that. You've got four to five orders of magnitude difference in cost and time. So these are the type of infections you look for certainly as an economist when it's like there's going to be actually a massive market dislocation.

10:58I'll give you another example from insta -base. So let's assume that you have a legal brief. It's in a PDF. You throw it into this kind of unstructured document LLM and then you ask questions for that legal brief. Again, the inference cost, say, a tenth of a penny, maybe it's a little more, maybe it'll last. Time to complete maybe one second, maybe a little more, maybe a little less. But as someone who has actually spent a lot of money on lawyers hours, I want to point out a couple of things. The first one is, it takes more than one hour to iterate on this for sure. And the second one is, they're not always correct.

11:25In fact, built -in for any interaction I have with a lawyer is cross -checking their work and double -checking their work. So again, we have four to five orders of magnitude difference in cost and time. And if you want to have an example of how highly extremely nutty that this can get, I see no reason why you can't generate an entire game. There are companies today working on it, the 3D models, the characters, the voices, the music, the stories, etc. They're companies that are doing all of these things. And if you compare the cost of like hundreds of millions of dollars and years versus, you know, a few dollars of inferences, now we have like current, like internet and microchip level asymmetries and economics.

12:02Now listen, I'm not saying this happens soon. We're not there yet. What I'm saying is this is the path that we're on. And these types of paths are what you look for with big transformations. So it's little wonder why we're seeing so much takeoff the way that we have. And these are the fastest -going open -source projects. These are the fastest -going products and some of the fastest -going companies we've seen in the history of the industry. And it's because, again, it's less the capabilities and it's much more that the economics work. So listen, this may sound hyperbolic, but I really think that we could be entering a 30 -pock of compute.

12:34I think that the first epoch, of course, is the microchip. Before the advent of the computer, you actually had people calculating logarithm tables by hands. Like, that's where the word comes from. They were computers. They would compute. Then we created any act along with other machines, but let's look at any act. So any act was 5 ,000 times faster than a human being doing it. There's your three to four orders of magnitude, and that kind of ushered in the compute revolution. Then this gave us a number of companies that were either totally transformed like IBM or totally net new. So the micrature brought the marginal cost of compute to zero.

13:06The internet brought the marginal cost of distribution to zero. So listen, in the 90s, when I wanted to get a new video game, I would go to a store and buy a box. And again, I don't have the math appear, but if you actually calculate the price per bit relative to DSL in the late 90s, is about four or five orders of magnitude again, relative to actually shipping it. So I think it's a pretty good analog where you say these large models actually bring the marginal cost of creation, is there some very fuzzy vague notion what creation means? But for sure, we could talk about it of like content, conversation, whatever it is.

13:35And like the previous epochs, when those epochs happen, you had no idea what new companies were gonna be created. Nobody predicted Amazon, nobody predicted Yahoo, like I remember when this happened. So I think listen, I think we should all get ready for a new way of iconic companies. I don't think we know what they're going to look like, but forget the capabilities, economics are just too compelling. We'll hear more from Martin at the end of this episode. But speaking of economics and the scale of top models today, here is our new general partner, Anjani Midha. Reminiscing about an early call he had with Dario Amade, co -founder of Anthropic, who you'll also hear from shortly.

14:12I'm going to take you all back in time, two about three years ago. You and Tom gave me a call, one of your co -founders, and said, hey, I think we're going to go start on topic. And I asked you, great, okay, like, what do you think we need to get going? And you said, well, I think we can get by with like 500. And I said, okay, I think we can find 500k somewhere. And I remember you deadpan thing. Dude, I'm going to $500 million. And that's when I realized things were going to be a little bit different. Dario was one of the first employees at OpenAI and spent five years there before co -founding and throbbing.

14:46And the last year of AI has absolutely captured the masses, but people like Dario were early in recognizing just how far these technologies could scale. What was it that at that moment when you've been the team at OpenAI had started publishing your first experiments on scaling laws that gave you so much confidence that this was going to hold when everybody else just thought that was crazy talk. Yeah, so for me the moment was actually GPT2 in 2019 where there were two different perspectives on it, right? When we put out GPT2, you know, some of the stuff that was considered most impressive at the time was, oh my god, you give this five examples, just offer it straight into the language model, five examples of English to French translation, and then you put a six sentence in English, and it actually translates and then to French like, oh my God, it actually understands the pattern.

15:39That was crazy to us, even though the translation was terrible, it was almost worse than if you were to just take a dictionary and substitute words for word. But, you know, our view was that, look, this is the beginning of something amazing because there's no limit and you can continue to scale it up. And there's no reason why the patterns we've seen before won't continue to hold. The objective of predicting the next word is so rich and there's so much you can push against that it just absolutely has to work. And then some people looked at it and they're like, you made a bot, the trends rates really badly.

16:10It was just, I think, two very different perspectives on the same thing, and we just like, really, really believed in the first perspective. What happened then was you saw a reason to continue down that line of inquiry, which resulted in GPT -3. And what would you think was the most dramatic difference between GPT -3 and the previous efforts? Yeah, I mean, I think it was much larger and scaled up to a substantial extent. I think the thing that really surprised me was the Python programming, where the conventional wisdom was that these models couldn't reason at all. And when I saw the Python programming, even though it was very simple stuff, even though a lot of it was stuff, you could memorize, you know, you could put it in kind of new situations, come up with something that isn't going to be anywhere in GitHub, and it was just showing the beginning of being able to do it.

16:58And so I felt that that ultimately meant that we could keep scaling the models and they would get very good at reasoning. What was the moment at which you realized, well, okay, we think this is actually going to generalize much broader than we expect. What were some of the signals there that gave you that conviction? I think one of the signals was that we hadn't actually done any work. We had just scraped the web and there was enough Python data in the web to get these good results. When we looked through it, it was like maybe 0 .1 % to 1 % of the data that we scraped was Python data. So, the conclusion was, well, if it does so well with so little of our data and so little effort to curate it on our part, it must be that we can enormously amplify this.

17:40And so, that just made me think, well, okay, we're getting more compute, we can scale up the models more, and we can greatly increase the amount of data. So we have so many ways that we can amplify this. And so, of course, it's going to work. It's just a matter of time. Another person optimistic about scaling laws at the time was Nome Shazir. Nome was one of the researchers and lead author behind the transformative 2017 transformer paper and has since co -founded Character AI. I knew that you know you can make this technology better in a lot of ways we can improve it with model architecture and distributed algorithms and quantization and all of these things.

18:16So I was working on that but then struck me hey the biggest thing is just scale, can you throw like a billion dollars or a trillion dollars at this thing? What would happen if we did massively scale compute? Well, many companies chose to find out, and we, the consumer, are the beneficiaries of that. But can this realistically continue? Can the industry just continue to throw more computer to the problem and get better solutions? Or will a more fundamental unlock be required? This theme was top of mind for many at the event and here is OpenAI's co -founder and CTO, Mira Murati, tackling that question head on.

18:54Do you think the scaling laws are going to hold and we're going to continue to see advancements or do you think we're heading diminishing returns? So the reason any evidence that we will not get much better and much more capable models is we continue to scale them across the access of data and compute. Whether that takes you all the way to AGI or not, that's a different question. There are probably some other breakthroughs and advancements needed along the way. But I think there's still a long way to go in the scaling laws and to really gather a lot of benefits from this larger models. We'll hear more from Mira and touch on AGI in part two.

19:35But first, here's no again, in conversation with ACCCNZ General Partner Sarah Wing. On just how much compute we expect to soon be available, but also how much innovation is on deck, even if there aren't additional fundamental breakthroughs. And for those listening on audio, yes, no fully did this competition in its head. I see this stuff massively scaling up. It's just like not that expensive. So I think I saw an article yesterday like Nvidia is going to build another 1 .5 million H100s next year. So that's 2 million H100s. So that's two times 10 to the 6 times they can do about 10 to the 15th operations per second.

20:16So two times 10 to the 21 divide by eight times 10 to the 9 people on earth. So that's roughly a quarter of a trillion operations per second per person, which means that could be processing on the order of like one word per second on like a hundred billion parameter model for everyone on earth, but like really it's not gonna be everyone on the earth because like some people are blocked in China and some people are sleeping. Like it's not that expensive, you know like this thing is massively scalable if you do it right and you know we're working on that. You said this once that the internet was the dawn of universally accessible information and we're now entering the dawn of universally accessible intelligence.

20:58Maybe building off your last answer, what did you mean by that? Do you think we're there yet? Yeah, I mean, I think it's like we're really a right brother's first airplane kind of moment, right? Like we've got something that works and is useful for now some large number of use cases and looks like it's scaling very, very well. And without any breakthroughs, it's going to get massively better as everyone just kind of scales up to use it. And there will be more breakthroughs because now, you know, like all the scientists in the world They're like working on making the stuff better. It's great that all this stuff is accessible.

21:30It's open source. We're going to see a huge amount of innovation and what's possible in the largest companies now can be possible in somebody's academic lab or garage in a few years. And then, yeah, the technology gets better. There are just going to be all kinds of great use cases that emerge in pushing technology forward, pushing science, pushing the ability to help people in various ways that love to get to the point where you can just ask it how to cure cancer or something. I mean, it seems a few years away for now. Do you think we need another fundamental breakthrough like the transformer technology to get there?

22:08Or do you think we actually have everything that we need? I mean, it's impossible to predict the future, but I don't think anyone's seen these scaling laws stop, I think, as far as anybody has experimented, stop just keeps getting smarter, so we'll be able to unlock lots and lots of new stuff. I don't know if there's an end to it, but at least everybody in the world should be able to talk to something like really brilliant and have incredible tools all the time. And I can't imagine that that will not be able to build on itself. At the core, the computation isn't that expensive, like operations cost like 10 to the negative $18.

22:45These days, and like, you know, if you can do this stuff efficiently, even talking to the biggest models ever trained, the cost of that should be way, way lower than the value of your time or most anybody's time, and really we should, you know, the capacity there to scale, these things up by orders of magnitude. As the industry does pursue scale, here's Stario's take on what bottlenecks maybe along the way. But the next 24, 36 months, what do you think the biggest bottlenecks are in demonstrating that the scaling laws continue holding? Yeah, so I think there's three elements. There's data. there's compute and there's algorithmic improvements.

23:26So I think we are on track, even if there were no algorithmic improvements from here, even if we just scaled up what we had so far. I think the scaling laws are gonna continue, and I think that's gonna lead to amazing improvements. I think the biggest factor is simply that more money is being poured into it. The most expensive models made today cost about a hundred million dollars, say plus or minus a factor of two. So I think that next year we're probably going to see from multiple players models on the order of $1 billion. And in 2025, we're going to see models on the order of several billion, I don't know, perhaps even $10 billion.