DeepSeek: America’s Sputnik Moment for AI?

0:00R1 comes out and it looks pretty good. That's not the best layer to monetize it. In fact, there might not be any money in that layer. I have yet to see the GPT wrapper. Internet is such a great example because there's no way this doesn't play out like the Internet. It's actually a very big step when it comes to the proliferation of this model. It's a good reminder that there are always pockets of people innovating. World Common AT &T did not predict the Internet was going to come out of universities. 2 words have caught the internet by storm. Deep, seek. Specifically, a Chinese reasoning model that seems to rival others at the frontier.

0:40But, that's not all. Alongside there are one model that dropped in late January, came, a fully open source MIT license, a paper outlining its methods that some claim maybe 45 times more efficient than other methods, and a legit $5 .6 million dollar cost, the release of reasoning traces, a follow -on image model and the fact that all of this was released by a hedge fund in China. Since then, there have been so many claims and claims about those claims that many are already referring to this as a Sputnik moment. But if you think about it, the reason that Sputnik, the first satellite launched into lower Earth orbit by Russia in 1957, the reason that Sputnik still matters in 2022 -25 is because America took all the actions that it did in 58, 59, 60, and moon landing speech in 62, all the way up to 1969 when we reached the moon.

1:35Those are the actions that made Sputnik, Sputnik. A wake -up call was responded to. So now that we're here, how should we weather your a casual listener, a founder, a researcher at a top AI lab, or a policymaker, not just react to this message, but act. Joining us to discuss this and tease out the signal from the noise are A16Z General Partner and Pioneer of Software -Defined Networking, Martijn Kassato, plus Steven Sinovsky, longtime Microsoft exec, including being the president of the Windows Division between 2006 and 2012. Steven, by the way, has also been a board partner at A16Z for over a decade and shares his learnings online at Hardcore Software, where he recently wrote a viral article called DeepSeek has been inevitable, and here's why.

2:27Of course, we'll link to that in the show notes. Both Martin and Stephen have been on the front lines of prior computing cycles, from the switching wars to the fiber build out, and have even witnessed the trajectory of companies like Cisco, AOL, AT &T, even WorldComb. So what really drove this DeepSeek frenzy? and more importantly, what should we take away? Have bigger and better frontier models been optimizing for the wrong thing? And where does value and stack a group? Today, we address those questions through the lens of Internet history. I hope you enjoy. As a reminder, the content here is for informational purposes only.

3:06Should not be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the company's discussed in this podcast. For more details including a link to our investments, please see A16Z .com slash Disclosures.

3:33It's been a busy few weeks. I don't know about you guys, my Twitter feed, podcasts, everything deep -seek everywhere, maybe unsurprisingly, but what's your TLDR in terms of what came out and maybe also your take on why it blew up in the way it did because we've seen lots of releases in the last let's say two years since chat GPT. The quick overview of course is out of essentially nowhere a small hedge fund quasi computer science research organization in China releases a whole model. Now those in the no no it didn't just appear there's a year and a half or so build up and they're really good and they're really good.

4:13Nothing was an act, but it appeared to take the whole rest of the world by surprise. And I think there were two big things about it that really caught everybody's attention. One was how did they go from nothing to this thing? And it seems to be a constant factor of compatibility and capabilities with everybody else. And this number got thrown around that it only cost $5 million. Yeah, $6 million. The number is irrelevant. Because it turns out they wrote a paper and they said, hey, we innovated in this particular set of things on training, which even here was like, oh, well, that was pretty clever.

4:53And then because of the weirdness that we don't need to get into of the financial public markets and how this whole thing happened on a Friday, the whole weekend was like everybody whipping themselves into a frenzy so they could wake up money morning and trade away a trillion dollars of market cap, which seems to be a complete overreaction and craziness, but that's not what we're here to talk about. To your point, there's a lot of moving parts here and there's a lot to consider. It's actually a fairly complicated situation. So there has been this view that the traditional one -shot LLMs were starting to maybe ask some talk, like GPT -4.

5:28There hadn't been a big advancement, but then there's this can be this new breath of life and open and release the reasoning model, which is a one and everybody's very excited about that. And so in this grand tapestry we're considering you have all this excitement about a one and of how that's going to drive compute costs and Nvidia and then R1 comes out and it looks pretty good. And then all of a sudden they're saying, well, if you can do it just as cheap as this can actually drive the next wave and so forth. And so there's a lot of build up to O1, which led to the R1 hype. And then I think to your point, people didn't know really what to think about it.

6:01And I agree with you as a total market over correction. By the way, it's also worth pointing out that, in addition to people saying, wow, this is a great model, there's a lot of like theories and rumor around, oh, well, maybe this is the CCP doing a siaop. Maybe it costs a lot more. Maybe this is very intentional. It was right by Chinese New Year. There's just a ton of rumors. Maybe we'll do our best to dissect everything going on. Yeah, maybe let's just do that because two of your points, There was a lot here, right? There was the performance element. There was these quotes around costs. There's the China element.

6:31There's the virality at hit number one in the App Store. There's also shipping speed. I think Martini shared that they released an image model shortly after and then it was released on a Friday. So there's this huge mixture of people reacting, some people who know what they're talking about and some people who don't quite frankly. And so we're like 10 days or so out from this release, which by the way, as both of you said, that was the R1 release. There was the V3 release what two months ago, which was the base model. So now that we're a little bit further out, what's the signal from the noise?

6:59So maybe I'll give you the lens of Chinese people or smart. There's one lens, the lens that I hold, which is trying to have great researchers. DeepSeek is actually released a number of soda models, including V3, which is actually probably a more impressive fetus. It's almost like a chat GPD4. And old by the way, to create one of these chain of thought models, these reasoning models, you need to have a model like that, which they had done and we had known about. All of the contributions that they've done have been in the public literature somewhere, just nobody had really aggregated. So there's a thought that I hold, which is this is a very smart team that has been executing very, very well for a long time in AI.

7:37They are some of the top researchers. The fact that they spent $6 million just in the chain of thought is actually not out of whack, which what Anthropic has now said, they've spent it open. I said that that they spent. And so this is a meaningful contribution from a good team in China. And so it means something and we should respond to it. So some of the outcry is warranted. I do think that we respond to it, but I don't think for the reasons a lot of people are saying. I completely agree with that. And in fact, you also saw the people outside of that team in China sort of piling on to try to make it more intergalactic than it was.

8:08I mean, my favorite old friend of mine, Kai Fu Li comes out on X and says something about this is why I said two years ago, Chinese engineers are better than American engineers. But the truth is, to your point about reaching some asymptotic level of progress. Yeah, like the previous base models, like the GPT lineage, seem to have asymptoted around GPT -4. Right. But what's super interesting about that is that asymptote was true if you looked at it through the lens of the function that everybody was optimized for. is to my view, this kind of crazy, hyper -scaler view of the world, which is we need more compute and more data, more compute, more data, and we're just on that loop.

8:48And a lot of people from the outside were like, well, you are gonna run out of data. And I was just as, you know, a micro -computer person was like, well, at some point, you're gonna end up breaking the problem up to the seven billion endpoints of the world, which will have vastly more compute than you can ever squeeze into one giant nuclear power data center. And so a lot of what they did was sort of a step function change, not necessarily improvement, just a change in the trajectory. Yes, and that to me is the part where the hyperscalers needed to take a deep breath and say, okay, why did we get to where we were?

9:22Well, because you were Google and Meta and OpenAI funded by Microsoft, which all had like billions and billions of dollars. So you obviously saw the problem through the lens of capital and data. And of course, you had English language data, which there's more of than anybody else, so you could keep going. The way I thought of it is, when Microsoft was small, we used to just decide, is it a small problem, a medium problem, or a large problem? And I remember, at one point, we started joking that we lost the ability to understand small and medium problems and solutions. And we only had, like, large, which was just trivial.

9:57And then huge and like ginormous. And our default was ginormous. because we thought, well, we could do it and no one else could, and that's a strategic advantage. And I feel like that's where the AI community in the West, if you will, got just a little carried away. And it was just like every startup that has too much money, the snacks get a little too good. So I've heard two theories of why they were able to do this. One of them is this constraint one that you've said, I think which is actually very true, which is we've just been using this blunt intraminative compute and blunt instrument of all data.

10:30And we just haven't thought about a lot of engineering under constraints. To the second theory I heard, I don't know if it's true, but it's tantalizing, which is the reason B3 is so good is actually because it has access to the Chinese internet as well as the public internet, which is actually an isolated thing. We don't really have access to the internal Chinese internet and we certainly don't train from it as far as I know which they do. So it could be the case both things are true. They could have had a data advantage. They definitely have the internet constraint. Well, even on the data, their starting point is the Chinese internet per se, that has much more structure to it.

11:03It's a much better training set. That's a great point. And in as much as human annotated data is important here. And for chain of thought, you do want experts saying, here's how I would reason about a problem. I mean, this is what this whole chain of thought is. It's basically, what are the reasoning steps? If you want to look at a place to arbitrage, really smart educated people and relatively low cost, it's hard to be China globally. Right? And so they definitely have access to a bunch of, potentially highly educated annotated data, which is very relevant here. And so I happen to be of the belief that this did not come out of nowhere.

11:32It's not a siop. This is a great team taking advantage of what it has. But there are still things that are very significant about it that are worth talking about. For example, the license is very significant. The fact that they decided to release the reasoning steps is very significant. Those are two things that you're not seeing headlines about, right? You're seeing headlines about all the other things that we just talked about. You said the reasoning traces, those were released, which using the comparable O1 were previously not. Right. And then the open source license. So there's two things that are pretty remarkable about deep C -Gar one that have implications on adoption.

12:06We haven't seen a license this permissive recently for a soda model. It's basically MIT license, which is like one page. You can do anything, right? It's like free isn't free beer for real. Yeah, for real for real. And I think at A60C we have one of the large portfolio of AI companies, both at the model layer and at the app layer. And I will say any company at the app layer is using many models. like I have yet to see the GPT wrapper. They're all using a lot of models. They do use open source models in licenses really matter. And so this is definitely gonna result in a lot of proliferation. The second thing is, so a reasoning model actually thinks through the steps of the problem.

12:41And it uses that chain of reasoning or chain of thought to come up with deeper answers. And when OpenAI released a one, they did not release that chain of thought. Now we don't know why they didn't do it. But it just turns out that that chain of thought, If you have access, it allows you to train smaller models very quickly and very cheaply. And that's called distilling. And so it turns out that you can get very, very high quality smaller models by distilling these public models. And the implications are both that this is just more useful for somebody using R1. But also, you get a lot more models that can run on a lot smaller devices.

13:15So you just get more proliferation that way. So it's actually a very big step when it comes to the proliferation of this model. Absolutely. And I think that there's this tendency to peg yourself at, oh, it should just be open, but without really defining it, which I think is important in this case. And I think because of where they came from and that they don't have a business model, that was part of what was unique about this, was it was a hedge fund, like almost a side project, but not really a side project, it has this effect that like, well, we're just going to give the whole thing away. And the rest of the companies are still trying to figure out their revenue models, which I would argue was probably premature, and it starts to look a little to me like, hey, let's charge for a web server.

13:56And it's like the business of serving HTTP, not a great business. And I think everybody just got focused on the first breakthrough, which was the LLM, which if you look back at the internet, what exactly happened was everybody got very focused on monetizing the first part of the internet with through HTML and HTTP. And then along came, I don't know, Microsoft and a bunch of other companies to say that's not the best layer to monetize it. In fact, there might not be any money in that layer. And the real money is going to be in shopping and in plane tickets and in television. And even other companies, AT &T got wound up trying to monetize even lower layer.

14:33But that's not how you're going to get to 7 billion endpoints. And I think that the licensing model really matters because what's going to happen is that there's going to end up being some level of standardization. Now, I don't know where in this stack or in what level, but there is going to be some level of standardization. And the licensing model for the different layers is going to start to matter a lot. Anyone who was around during the internet remembers the battles over the different canoe, the three, the four, the open this license. And it was very well. Right, well, between a dissertation and turns out, even your dissertation, which part of it, and how you released it was a huge issue because it could make or break a whole approach.

15:15And I think that the US industry lost sight of that importance because they got so used to this model of like open just means we're a business and we pick and choose what we throw out there is evidence that we're an open company. Yeah, totally. And I think that view isn't an aligned with how technology has just shown to evolve in an era where there's no cost for distribution. Before when there was a cost for distribution, it turns out the free model was irrelevant because you still couldn't figure out how to get it to anybody. It's got people to use it. Yeah, totally. I do want to take the other side of this because I actually tend to agree with you.

15:48And so what you just said is it could be the case that the models, the wrong place to focus and everybody thinks there's a lot of value in there and so they're playing all these cute games with openness as opposed to distribution. And that could very well be true, but there's another view which is actually the models really are pretty valuable and in particular, the model itself isn't an app. But it could be the case that if you're building an app you need to vertically integrate into the model. It could be the case. And therefore, like if I'm building the next version of chat, GPT, or we just had today deep research, it could be that the apps actually require you to own the model.

16:21And in that case, DeepSeek is less relevant because they're not building apps. And then this means that the impact to the opening of eyes or anthropics are not as great, right? And so I do think that there's this fork that we don't know the answer. Fork number one is, maybe the models do get come on, it's as you need to focus at the app layer. and then the license doesn't matter or the model's really matter up the stack, in which case the whole deep -seek phenomenon really isn't as impactful an event as people are making it. So I'm going to go on that just because I want to say you're right both times.

16:51And the variable is time. And the internet is such a great example because there's no way this doesn't play out like the internet. Like it just has to. And what we saw was for a while building one app seemed like a crazy thing because you had to own windows and you had to own office. But then a new app came along that didn't own any of those and it was search. And so that's why I think a lot of people also because of age and what they lived through immediately jumped to, oh, these LLMs are gonna replace search. But it turns out that's actually gonna be really, really hard because there's a lot of things that search does that the models are bad at, really bad.

17:30And so what's going to happen is a new app is going to emerge. And then when the new app emerges, that's going to get vertically integrated. And the research app is a super good example of that. And then all of a sudden, other apps are going to spring up. Oh, there's Google Maps. And there's Search. And then there's Chrome. And then it goes back and eats the things that it couldn't do before. And I really feel like that's what the trajectory were on. Now, it's still a matter of where and what integrates. But the thing is is that the apps that ended up mattering on the internet literally didn't exist before the internet.

18:05And I think that's what people are losing side of it. Same with mobile, like you were. Everybody is completely right. There were no social apps. Okay fine, I get it. There was geo cities and a bunch of others. But people get so caught up on new thing, it's gonna replace something. There's something so different. There's zero sum and you can think of everything as this spectrum. And when something new comes along, the whole spectrum gets divided up differently, which is what Google said when they bought it rightly. They said, you know what people would do in the internet? They're going to type stuff.

18:35And what are they going to type? They're going to type it, but they're going to type it with other people. Okay, so this is great. So we're actually seeing this happening now, which was someone will come up with a model that does something like in a consumer space. Let's say like text to image. And then it turns out that over time people are like, oh, it's kind of like Canva. Exactly. It's like slowly do the AI native version. Just like the cloud native version of Word. the AI -native version of these kind of existing apps. The reason it's important is because it looks like Canva or it looked like Word or it looked like PowerPoint or it looked like Excel, but what's important is that they're actually different.

19:07Nothing is gonna ever be PowerPoint again. Why? Because PowerPoint, the whole reason for existing was to be able to render something that couldn't ever be rendered before. And so all of the whole product, it's 3 ,000 different formatting commands. Like literally, that's not an unbridled up. It's 3 ,000 ways to earn and nudge and squiggle and color. And actually, it turns out you don't need to do any of that in AI. So the whole product isn't gonna have any of those things. Yeah, exactly. And then it turns out all those things make it really hard to make it multi -user. And so then when Google comes along and starts to bundle up their competitor that's gonna replace it, they're focused on sharing.

19:47So Steven, let me ask you this. You said something really interesting. I'm good, I got it, I did. which is this has to pin out like the internet. And you guys have used examples of different companies, the mobile wave, cloud era, those are things we can learn from. But I just want to probe you, is there something different here? To bring it back to deep seek. This is very important to realize the capabilities of China. It's a very credible player. But I don't think that R1 itself is a standalone is going to have that deep of an impact. But on the internet, so there's actually these parallels when it comes to capital build out that you see in the AI, which is it takes a lot of investment.

20:22And there's a special parallel that Mark and Drisnext would remind me of it, which people don't tend to see as well, which is in the early days of the internet, like the mid to late 90s, a lot of investors, a lot of big money, think banks or sovereigns, they wanted exposure to the internet, but they had no idea how to invest in software companies. Like what are these new software companies? Who are these people? Like they're all private companies. Like, so what did all of them do? They all invested in fiber infrastructure. So we're starting to see this thing again, right? we see a lot of banks and big investors, let's say we want to build that data centers, because they don't know how to invest in startups, like we know how to invest in startups, right?

20:56So on one hand, you can be like, oh, we're going to see all of this kind of capital expenditure and all this capital expenditure is going to go into physical infrastructure, and therefore we're going to have another fiber glut equivalent by the data center glut. So the counter to that point, where I think is different is, at the time of the fiber build out, you've had one company would happen to be cooking its numbers, where it had a ton of debt to build all of this out. When the price of fiber dropped, that company went out of business and that caused the huge issue. You have a much better foundation for the AI wave.

21:27The primary investors are the big three cloud companies that got hundreds of billions of dollars on the balance sheet. Even if all of this goes away, they'll be fine. In video, I can take a price dip, and video will be fine. So I don't think we're heading to the same type of glut and crash that other people have, which is very appealing to draw parallels to the internet for that I don't think is it? Oh, I am completely with you on that. That part of it is gonna look like the amount that Google invested in the, or the 2000s, or the amount that Facebook invested five years later, or people forget that Microsoft poured, I don't know, $30, $40 billion into Bing.

22:00And it's still number three or whatever, but it still doesn't matter. Yeah, I would bet, I don't know this is a fact. I'll bet Meta's spending more money on VR than it is on AI right now. Yeah, not just to show. It's an Apple too, right? Well, Apple, also because Apple, whatever is bigger than Gargantuan, is how much they're spending. And so it really isn't about the investing profile. And I think that is a super important point that you made to really just hammer home. There's a certainty that nobody's going to come out of this unscathed, but the scathing is not going to be at all when anybody thinks.

22:31And then not like what it was. Oh, yeah. Well, come on. I believe he had $40 billion in debt, right? I mean, it was just one of these things where structurally it was. Oh, and there were companies that we've all forgotten about. That went bankrupt over that era. Actually, there was one in Seattle whose name I'm forgetting, but that was like $20 billion, just poor God. To your point, these companies have had so much cash on their balance sheet. They've been waiting for a moment to invest in the next generation, which also contributes to their willingness to scale up as much as they did. So let's talk about that.

22:57In your article, you talk about the difference between scale up and scale out, and the natural tendency in these early parts of the wave to scale up, when really there tends to be a shift towards software basically going to zero, So, Coss, so, Stephen, what do you mean by that? And are we at that change in directory? Now we'll just switch to make sure we're really talking about the technology now, not the finances. But when you're big, you want to double down on being big. And so you start building bigger and bigger and bigger computers that don't distribute the computation elsewhere. So if your IBM, you just say the next mainframe is another mainframe that's even bigger.