The Four Wars of the AI Stack - An Update with Swyx and Alessio of Latent Space

0:01Today on the AI Breakdown, part one of my conversation with Alessio and Swix from Layton Space. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to Breakdown.network for more information about our YouTube, our Discord, and our newsletter.

0:24Hello, friends. Right now, I am doing a bit of travel with the family, and so we are doing interviews this week. And today and tomorrow, I am bringing you two parts of a great conversation with my friends from Leight in Space, Alessio and Sean, better known as Swix. You guys might remember a show we did together back six months ago or so, and this follows a similar format where we are talking about some of the biggest trends shaping the AI space right now. Hello, friends. Quick note before we get to the rest of the episode. You have probably heard me talk about the AI education beta over the past few months.

0:56We've had a ton of you participate, which has been amazing. and now we're almost ready to announce something big and something new. If you want to be one of the first to hear about our new approach to learning AI that is hyper-practical, hands-on, immediately relevant, continuously upgrading, and anchored by community, go to besuper.ai and sign up to be notified when the project goes live. We're getting there in just a few weeks, and I want all of you along for the journey. Once again, that's besuper.ai. In this half of the show, we talk about the four wars of the AI stack, what we've learned about the power of being GPU rich, and some of the realigning battles around big tech.

1:35I don't want to waste any more time before we dive in, so let's listen. All right, fellas, welcome back to the AI breakdown. How are you doing? I'm good. Very good. Yeah. The last time we did the show, we were like, oh, yeah, let's do check-ins monthly about all the things that are going on. And then, of course, six months later, and, you know, the world has changed in a thousand ways. It's just it's too busy to even to even think about podcasting sometimes. But I'm super excited to be chatting with you again. I think there's there's a lot to catch up on just to tap in, I think, in the beginning of 2024.

2:09And so, you know, we're going to talk today about just kind of a broad sense of where things are in some of the key battles in the space. space. And then one of the big things that I'm really excited to have you guys on here for us to talk about what patterns you're seeing and what people are actually trying to build, where developers are spending their time and energy and any sort of trends there. But maybe let's start, I guess, by checking in on a framework that you guys actually introduced, which I've loved and I've cribbed a couple of times now, which is this sort of four wars of the AI stack.

2:44Because first, since I have you here, I'd love to hear sort of like where that started gelling. And then maybe we can get into, I think, a couple of them that are particularly interesting in light of some recent news. Yeah. So maybe I'll take this one. So The Four Wars is a framework that I came up around trying to recap all of 2023. I tried to write sort of monthly recap pieces. And I was trying to figure out what makes one piece of news last longer than another or more significant than another. And I think it's basically always around battlegrounds. Wars are fought around limited resources. And I think probably the most limited resource is talent, but the talent expresses itself in a number of areas.

3:27And so I kind of focus on those areas first. So the four wars that we cover are the data wars, the GPU rich, poor war, the multimodal war, and the Rag and Ops War. And I think you actually did a dedicated episode to that. So thanks for covering that. Yeah, yeah. Not only did I do a dedicated episode, I actually used that. I can't remember if I told you guys. I did give you big shout outs, but I used it as a framework for a presentation at Intel's big AI event that they hold each year where they have all their folks who are working on AI internally. And it totally resonated. That's amazing. Yeah, so what got me thinking about it again, is specifically this inflection news that we recently had, this sort of, you know, basically, I can't imagine that anyone who's listening wouldn't have thought about it.

4:16But, you know, inflection is one of the big contenders, right? I think probably most folks would have put them, you know, just a half step behind the anthropics and open AIs of the world in terms of labs. But it's a company that raised$1.3 billion last year, less than a year ago. Reid Hoffman's a co-founder. Mustafa Sullyman, who's a co-founder of DeepMind, And, you know, so it's like this is not a small startup, let's say, at least in terms of perception. And then we get the news that basically most of the team, it appears, is heading over to Microsoft and they're bringing in a new CEO. And, you know, I'm interested in kind of your take on how much that reflects the cold aside, I guess, you know, all the other things that it might be about, how much it reflects this sort of the stark, brutal reality of competing in the frontier model space right now.

5:04and just the access to compute? There are a lot of things to say. So first of all, there's always somebody who's more GPU rich than you. So Inflection is GPU rich by startup standard. I think they bought 22 ,000 H100s, but obviously that pales compared to Microsoft. The other thing is that this is probably good news maybe for the startups. It's like being GPU rich, it's not enough. You know, like I think they were building something pretty interesting in Pi, their own model, their own kind of experience. But at the end of the day, the interface that people consume as end users is really similar to a lot of the others.

5:46And we'll talk about GPT-4 and Cloud3 and all this stuff. Sometimes when you're a startup, you can have a lot of success being GPU poor, doing something that the GPU rich are not interested in. And, you know, we just had our AI center of excellence at Decibel. And one of the AI leads at one of the big companies was like, oh, we just saved$10 million and we use these models to do a translation, you know, and that's it. It's not it's not a GI. It's just translation. So I think like the inflection part is maybe a calling and awaking to a lot of startups and say, hey, you know, trying to get as much capital as possible, try and get as many GPUs as possible.

6:26it's good, but at the end of the day, it doesn't build a business, you know, and maybe what inflection. Again, I don't know the reasons behind the inflection choice, but if you say, I don't want to build my own company that has 1.3 billion and I want to go do it at Microsoft, it's probably not a resources problem. It's more strategic decisions that you're making as a company. So yeah, that was kind of my take on it. Yeah. And I guess on my end, two things actually happened yesterday. There was a little bit quieter news, but Stability AI had some pretty major departures as well. And you may not be considering it, but Stability is actually also a GPU-rich company in the sense that they were the first new startup in this AI wave to brag about how many GPUs that they have, and you should join them.

7:11And, you know, Imadis is definitely a GPU trader in some sense from his hedge fund days. So Robin Rombach and like most of the Stable Diffusion 3 people left stability yesterday as well. So yesterday was kind of like a big news day for the GPU-rich companies, both inflection and stability having sort of wind taken out of their sails. I think, yes, it's a data point in the favor of, like, just because you have the GPUs doesn't mean you automatically win. And I think, you know, kind of I'll echo what Alessio says there. But in general, also, like, I wonder if this is like the start of a major consolidation wave, just in terms of, you know, I think that there was a lot of funding last year.

7:49And, you know, the business models have not been worked out very well. Even inflection couldn't do it. And so I think maybe that's the start of a small consolidation wave. I don't think like that's like a sign of AI winter. I keep looking for AI winter coming. I think this is kind of like a brief cold front. Yeah, it's super interesting. So I think a bunch of stuff here. One is, I think to both of your points, in some ways, there had already been this very clear demarcation between these two sides where, like, the GPU pores, to use the terminology, like, just weren't trying to compete on the same level, right?

8:27You know, the vast majority of people who have started something over the last year, year and a half, call it, were racing in a different direction. They're trying to find some edge somewhere else. They're trying to build something different. If they're really trying to innovate, it's in different areas. And so it's really just this very small handful of companies that are in this like very, you know, it's like the coheres and jaspers of the world that like this sort of, you know, that are that are just sort of a little bit less resourced than, you know, than the other set that I think that this potentially even applies to, you know, everyone else that could clearly demarcated into these two two sides.

9:00And there's only a small handful kind of sitting uncomfortably in the middle, perhaps. Let's come back to the idea of the sort of AI winter or, you know, a cold front or anything like that. So this is something that I spent a lot of time kind of thinking about and noticing. And my perception is that the vast majority of the folks who are trying to call for sort of, you know, a trough of disillusionment or, you know, a shifting of the phase to that are people who either, A, just don't like AI for some other reason. There's plenty of that. You know, people who are saying, look, they're doing way worse than they ever thought.

9:37You know, there's a lot of sort of confirmation bias kind of thing going on. Or two, media that just needs a different narrative, right? Because they're sort of sick of, you know, telling the same story. Same thing happened last summer when every outlet jumped on the chat GPT at its first down month story to try to really like kind of hammer this idea that the hype was too much. um meanwhile you have you know just ridiculous levels of investment from enterprises you know coming in um you have you know huge huge volumes of you know individual behavior change happening but i do think that there's nothing incoherent sort of to your points twix about that and the consolidation period like you know if you look right now for example there are i don't know probably 25 or 30 credible, like, build-your-own-chatbot platforms that, you know, a lot of which have, you know, raised funding.

10:32There's just no universe in which all of those are successful across, you know, even with a total addressable market of every enterprise in the world. You know, you're just inevitably going to see some amount of consolidation. Same with, you know, image generators. There are, if you look at A16Z's top 50 consumer AI apps, just based on, you know, web traffic or whatever, there's still like, I don't know, a half dozen or 10 or something, like some ridiculous number of like basically things like MidJourney or Dolly 3. And it just seems impossible that we're going to have that many, you know, ultimately as sort of, you know, going concerned.

11:09And so I don't know, I think that there will be inevitable consolidation because, you know, just it's also what kind of like venture rounds are supposed to do. You're not everyone who gets a seed round is supposed to get to series A and not everyone who gets a series A is supposed to get to series B. That's sort of the natural process. I think it will be tempting for a lot of people to try to infer from that something about AI not being as sort of big or as sort of relevant as it was hyped up to be. But I kind of think that's the wrong conclusion to come to. I would say the experimentation surface is a little smaller for image generation.

11:45So if you go back maybe six, nine months, most people will tell you, why would you build a coding assistant when like Copilot and GitHub are just going to win everything because they have the data and they have all the stuff. If you fast forward today, a lot of people use cursor. Everybody was excited about the Devin release on Twitter. There are a lot of different ways of attacking the market that are not completion of code, the ID. And even cursors, they evolve beyond single line to chat, to do multi-line edits and all that stuff. Image generation, I would say, yeah, just from what I've seen, like maybe the product innovation has slowed down at the ux level and people are improving the models so the race is like how do i make better images it's not like how do i make the user interact with the generation process better and that gets tough you know it's hard to like really differentiate yourselves so yeah that's that's kind of how i look at it and when we think about multi-modality maybe the people why the reason why people got so excited about sora it's like oh this is like a completely it's not a better image model this is like a completely different thing you know and i think the the creative mind is always looking for something that impacts the the viewer in a different way you know like uh they really want something different versus the developer mind is like oh i i just i have this like very annoying thing i want better i have this like very specific use cases that i want to go after so it's just different and that's why you see a lot more companies in image generation, but I agree with you that if you fast forward, there's not going to be 10 of them.

13:23It's probably going to be one or two. Yeah. I mean, to me, that's why I call it a war. Individually, all these companies can make a story that kind of makes sense, but collectively, they can all be true. Therefore, there is some kind of fight over limited resources here. Yeah. So it's interesting. We wandered very naturally into sort of another one of these wars, which is the multimodality kind of idea, which is basically a question of whether it's going to be these sort of big everything models that end up winning or whether you're going to have really specific things, like something Dolly 3 inside of sort of OpenAI's larger models versus a mid-journey or something like that.

14:04And at first, I was kind of thinking like for most of the last, call it six months or whatever, it feels pretty definitively both and in some ways, and that you're seeing just great innovation on sort of the everything models, but you're also seeing lots and lots happen at sort of the level of kind of individual use cases. But then Sora comes along and just obliterates what I think anyone thought where we were when it comes to video generation. So how are you guys thinking about this particular battle or war at the moment? Yeah, this was definitely a both-and story and Sora tipped things one way for me in terms of scale being all you need.

14:48And the benefit, I think, of having multiple models being developed under one roof. I think a lot of people aren't aware that Sora was developed in a similar fashion to Dolly 3. And Dolly 3 had a very interesting paper out where they talked about how they sort of bootstrapped their synthetic data based on GPT-4 vision and gpc4 um and and it was just all like really interesting like if you work on one modality it enables you to work on other modalities and all that is more is is more beneficial if it's all in the same house whereas the individual startups who don't who sort of carve out a single modality and work on that definitely you know won't have the state-of-the-art stuff on on helping them out on synthetic data so um i i do think like the balance is tilted a little bit towards the god model companies, which is challenging for the dedicated modality companies.

15:44But everyone's carving out different niches. We just interviewed Suno AI, the music model company. And I don't see OpenAI pursuing music anytime soon. Yeah, Suno has been phenomenal to play with. Suno has done that rare thing where, which I think a number of different AI product categories have done, where people who don't consider themselves particularly interested in doing the thing that the AI enables find themselves doing a lot more of that thing, right? Like it'd be one thing if just musicians were excited about Suno and using it, but what you're seeing is tons of people who just like music all of a sudden like playing around with it and finding themselves kind of down that rabbit hole, which I think is kind of like the highest compliment that you can give one of these startups at the early days of it.

16:27Yeah. I asked them directly in the interview about whether they consider themselves mid-journey for music and he had a more sort of nuanced response there but i think that probably the business model is going to be very similar because he's focused on the b2c element of that so yeah i mean you know just to just to tie back to the question about you know large multi-modality companies versus small dedicated modality companies um yeah i highly recommend people to read the sora blog posts and then read through to the dolly blog post because they they strongly correlated themselves with the same synthetic data bootstrapping methods as dolly um and i think once you make those connections you're like oh It is beneficial to have multiple state-of-the-art models in-house that all help each other.

17:10And that's the one thing that a dedicated modality company cannot do. Today's podcast is brought to you by Plum. You've probably noticed by now that many of the AI features that are embedded in your favorite products kind of suck. They're cool the first time, but pretty soon you're underwhelmed. That's because truly great AI features require complex pipelines and rigorous testing that most startups simply don't have time or tooling to get right. That's why Plum created a collaborative AI app builder that's purpose-built for product teams. Your users deserve better than a glorified GPT wrapper.

17:42Blow their minds with Plum. Check out useplum.com. That's Plum with a B. Send me a note to get early access. So I want to jump, I want to kind of build off that and move into the sort of like updated GPT-4 class landscape, because that's obviously been another big change over the last couple months. But for the sake of completeness, is there anything that's worth touching on with sort of the quality data or sort of rag ops wars, just in terms of anything that's changed, I guess, for you fundamentally in the last couple of months about where those things stand? So I think we're going to talk about rag for the Gemini and Clouds discussion later.

18:18And so maybe briefly discuss the data piece. I think maybe the only new thing was this Reddit deal with Google for like a$60 million deal just ahead of their IPO, very conveniently turning Reddit into an AI data company. Also, very interestingly, a non-exclusive deal, meaning that Reddit can resell that data to someone else. And it probably does become table stakes. A lot of people don't know, but a lot of the webtext data set that originally started for GPT 1, 2, and 3 was actually scraped from Reddit, at least the vote scores. And I think that's a very valuable piece of information. So like, yeah, I think people are figuring out how to pay for data.

19:01People are suing each other over data. This war is definitely very, very much heating up. And I don't think I don't see it getting any less intense. I, you know, next to GPUs, data is going to be the most expensive thing in a model stack company. And, you know, a lot of people are resorting to synthetic versions of it, which may or may not be kosher based on how far along or how commercially blessed the forms of creating that synthetic data are. I don't know if, Alessio, you have any other interactions with like data source companies, but that's my two cents. Yeah. Yeah, I actually saw Quentin Anthony from Allutari at GTC this week.

19:45He's also been working on this. I saw Technium. He's also been working on the data side. I think especially in open source, people are like, okay, if everybody is putting the gates up, so to speak, to the data, we need to make it easier for people that don't have 50 million a year to get access to good data sets. And Jensen at his keynote, he did talk about synthetic data a little bit. So I think that's something that we'll definitely hear more and more of in the enterprise, which never bodes well, because then all the people with the data like, oh, the enterprises want to pay now. Let me let me put a pay here Stripe link so that they can give me 50 million dollars.

20:23But it worked for Reddit. I think the stock is up 40 percent today after opening. So, yeah, I don't know if it's all about the Google deal, but it's obviously Reddit as being one of those companies where, hey, you got all this like great community, but like, how are you going to make money? And like they try to sell the avatars. I don't know if that it's a great business for them. The data part sounds as an investor, you know, the data part sounds a lot more interesting than than consumer cosmetics. Yeah. Yeah. So I think, you know, there's more questions around data. I think a lot of people are talking about the interview that Mira Murati did with the Wall Street Journal, where she just basically had no good answer for where they got the data for Sora.

21:06I think this is where, you know, it's in nobody's interest to be transparent about data. And it's kind of sad for the state of ML and state of AI research. But it is what it is. We have to figure this out as a society, just like we did for music and music sharing, you know, in sort of the Napster to Spotify transition. and that might take us a decade. Yeah, I agree. I think that you're right to identify it not just as a sort of technical problem, but as one where society has to have a debate with itself. Because I think that there's, if you sit rationally within it, there's great kind of points on all side, not to be the sort of person who sits in the middle constantly, but it's why I think a lot of these legal decisions are gonna be really important because the job of judges is to listen to all this stuff and try to come to things and then have other judges disagree and have the rest of us all debate at the same time.

21:55By the way, as a total aside, I feel like the synthetic data right now is like eggs in the 80s and 90s, whether they're good for you or bad for you. We get one study that's like synthetic data, there's model collapse, and then we have a hint that Lama 2, the most high-performant version of it, which was one they didn't release, was trained on synthetic data, so maybe it's good. I just feel like every other week I'm seeing something sort of different about whether it's good or bad for these models? Yeah, the branding of this is pretty poor. I'll kind of tell people to think about it like cholesterol.

22:29There's good cholesterol, bad cholesterol, and you can have good amounts of both. But at this point, it is absolutely without a doubt that most large models from here on out will all be trained as some kind of synthetic data, and that is not a bad thing. There are ways in which you can do it poorly, whether it's commercial sourcing or in terms of the model performance. But it's without a doubt that good synthetic data is going to help your model. And it's just a question of where to obtain it and what kinds of synthetic data are valuable. Even alpha geometry was a really good example from earlier this year.

23:12If you're using the cholesterol analogy, then my egg thing can't be that far off. Yeah, exactly.

23:45maybe very broadly speaking uh which of these do you think have made a bigger impact uh probably the the one you can use right so well i'm sure gemini is gonna be great once they let me let me in but um so far i haven't been able to i use i so i have this small podcaster thing that i built for our podcast which does um chapters creation like named entity recognition summarization and all of that um cloud 3 is better than gpd4 cloud 2 was unusable so i used gpd4 for everything uh and then when opus came out i tried them again side by side and i posted it on on twitter as well um cloud is is very good you know it's much better it seems to me it's much better than gpd4 at doing writing that is more you know i don't know it just got good vibes you know like the gpd4 text, you can tell it's like Judy for, you know, it's like it always uses certain types of words and phrases.

24:44And, you know, maybe just me because I've now done it for 50 podcast episodes. So I've read like 75, 80 generations of these things next to each other. But Clutter is really good. I know everybody is freaking out on Twitter about it. My only experience of this is much better has been on the podcast use case. But I know that, you know, Quran from from news research is a very big opus uh pro opus person so i think that's also it's great to have people that actually care about other models you know i think so far to a lot of people maybe entropic has been the sibling in the corner you know it's like cloud releases a new model and then open ai releases sora and like you know there are like all these different things but yeah the new models are good it's interesting My perception is definitely that, just observationally, Claude 3 is certainly the first thing that I've seen where lots of people, no one's debating evals or anything like that.

25:46They're talking about the specific use cases that they have, that they used to use ChatGPT for every day, day in, day out, that they've now just switched over. And that has, I think, shifted a lot of the sort of like vibe and sentiment in the space, too. And I don't necessarily think that it's sort of a like full, you know, sort of full knock. Let's put it this way. I think it's less bad for open AI than it is good for Anthropic. I think that because GPT-5 isn't there, people are not quite willing to sort of like, you know, get overly critical of OpenAI, except insofar as they're wondering where GPT-5 is.

26:24But I do think that it makes Anthropic look way more credible as a player, you know, as a credible sort of player, you know, as opposed to where they were. Yeah. And I would say the benchmarks veil is probably getting lifted this year. I think last year people were like, okay, this is better than this on this benchmark, blah, blah, blah, because maybe they did not have a lot of use cases that they did frequently. So it's hard to like compare yourself. So you defer to the benchmarks. I think now as we go into 2024, a lot of people have started to use these models from, you know, from very sophisticated things that they run in production to some utility that they have on their own.

27:06Now they can just run them side by side. and it's like, hey, I don't care that the MMLU score of Opus is slightly lower than GPT-4. It just works for me. And I think that's the same way that traditional software has been used by people. You just strive for yourself. And which one works best for you? Nobody looks at benchmarks outside of sales white papers. And I think it's great that we're going more in that direction. We have an episode with Adapt coming out this weekend. And in some of their model releases, they specifically say, we do not care about benchmarks, so we didn't put them in, you know, because we don't want to look good on them.

27:46We just want the product to work. And I think more and more people will go that way. Yeah, I would say like it does take the wind out of the sails for GPT-5, which I know we're curious about later on. I think anytime you put out a new state of the art model, you have to break through in some way. and what Claude and Gemini have done is effectively take away any advantage to saying that you have a million token context window. Now everyone's just going to be like, oh, okay, now you just matched the other two guys. And so that puts an insane amount of pressure on what GPT-5 is going to be because it's just going to have, like the only option it has now because all the other models are multimodal, all the other models are long context, all the other models have perfect recall.