How open-source & distributed models can win AI with MosaicML’s Naveen Rao | E1754

0:00AI, in my view, is the next evolution of what humans can do. You know, language was a big technology that humans used to pass knowledge that that exploded what humans could do and the influence we could have on the world. AI is going to be that next inflection point. But how are we going to make sure that everyone has a place in that world? How are we going to make sure that the demand that's created by the increase in efficiency is commensurate or does it collapse? This Week in Startups is brought to you by Vanta. Compliance and security shouldn't be a deal breaker for startups to win new business.

0:34Vanta makes it easy for companies to get a SOC 2 report fast. Twist listeners can get$1 ,000 off for a limited time at vanta.com slash twist. Trovata. Starting up is hard. Trovata makes managing cash easy. Start automating your cash management at trovata.io slash twist. Use code TWIST for 30 % off one full year of premium features like AI forecasting. And the Microsoft for Startups Founders Hub helps all founders build a better startup at a lower cost from day one. Startups get up to$150 ,000 in Azure credits, access to free OpenAI credits, free dev tools like GitHub, technical advisory, access to mentors and experts, and so much more.

1:23There is no funding requirement and it only takes minutes to join. Sign up today at aka.ms slash This Week in Startups. All right, everybody, we are really focused on AI and this crazy revolution that started really with GPT-3 and 4 having a moment for open AI and people starting to realize, hmm, this stuff is going to impact everything. Since that time, Dolly and Stable Diffusion showed what's possible, and now people are incorporating generative AI, the ability to generate some type of content intelligently from a prompt into every single product. Whether it's Notion or Microsoft Office or Gmail, we're going to have AI companions, co-pilots in every piece of software.

2:18but you're left with a big question as an entrepreneur and enterprise do you do this on your own and do you own the ip and do you control your destiny or do you partner with existing platforms that are out there with me today is naveen rao of mosaic ml he's the co-founder and ceo correct naveen that's correct got it so you heard me sort of teeing this up uh you've been in machine learning for quite a time you had a company before your current company that you sold to uh intel i believe and so that's correct that was nirvana maybe you could explain what you're doing today uh with mosaic and why yeah so what we're doing today is really bringing these capabilities of large-scale machine learning which is generative ai in my in my mind to many organizations um i i think one of the things we've done even with my previous company was trying to really bring these capabilities to more people to create the world we want.

3:20I see success as people that disagree with me being able to build models equally as good as me. I think that's how we're going to make this world work. And it's become front and center now with debates around regulation of AI and putting some sort of government licenses and this and that. But I think really this is solved more in a market as a market solution where many people can build the stuff. Many people can imbue these models with the biases that they see fit. And, you know, we'll let the market decide where where things should be, not some sort of centralized regulatory agency. So your company allows my organization to take our data, put it into a language model.

4:06This is called Mosaic ML. Yeah. This is the software that will let me train my own model and then host it easily on AWS or whatever cloud I choose, I suppose. That's correct. Yeah. In fact, we even collapse the experience across different clouds. I mean, you can start training on 512 GPUs and AWS and then move it to 1000 GPUs and Azure. We actually make it very easy to move things around and be modular. And really that enables people to use their resources more effectively, but also not have a lock in from the provider and really just kind of own their IP. I mean, I think it all stems from the fact that respecting data privacy, I think, is important.

4:51Data is, you know, arguably an expression of your company, of your IP and building solutions that respect those bounds and enables people to build on top of that data and own that thing that is built. i.e. the model is very important and i think that's what we we aim to do at mosaic so this uh harkens back to the i don't know the thorn in my paw that i have been screaming about since the beginning of this which is hey what did you train these things on and are those people being compensated we now have a handful of lawsuits and letters that have been either filed or sent twitter to um microsoft about the training use of their data reddit coming out and saying hey this is our data if you want to use it we're going to need a fee and of course people trained on it without uh their those two sources permission and then of course you have getty images uh versus stable diffusion you have open source uh those the community uh or open source contributors versus co-pilot my github's uh composer essentially or um you know co-pilot for developers so if i was disney and i own marvel every comic book ever written by marvel every film every piece of dialogue written every treatment ever written things that made it onto uh the screen things that didn't make it onto the screen things that were spec scripts that were written that were never done you can be sure they've got 10 scripts for every one they actually produced disney could take that marvel or star wars corpus put it into mosaic compose a library never have to worry about having put that train a day training data into a public entity that would then go use it for future or maybe even claim some ip ownership of it and then they could let their writers on the marvel series ask questions hey tell me about this character dazzler was she ever part of the x-men what is she known for what does her dialogue sound like can i get some backstory whatever something the writers are fighting against but putting that issue aside this is a pretty compelling case for a company like disney to start this work now and to keep open ai and google's hands off this data, correct?

7:13That's right. Yeah. I mean, I think that's, that's one sort of really flagrant example, like kind of, uh, incentivizing content creators to keep creating content. Uh, there's even more, I'll call it mundane things where, Hey, I'm a, I'm a company that has, you know, huge data sets on the behavior of my customers. And I want to build something that gives me a competitive advantage in my space. Right. And I don't want to share that with my competitors. I want to build a model and express that competitive advantage directly, but I don't know how to build models, right? I'm not an expert at doing that.

7:48I can't hire a team to do it. They can use our tools to go and do that and leverage their data for a competitive advantage. But yeah, I think it all comes down to this sort of similar way of thinking is that we have data that gives us a kind of an economic incentive to keep gathering data, to keep creating new content. and then we have some way of expressing an advantage in the market and uh you need to be that could be proprietary data about consumer behavior or it could be you know i picked the the most iconic ip of all time star wars and marvel or at least in recent history that have you know generated massive massive profits for those companies if you're a sass or services company that stores customer data in the cloud, then you need to be SOC 2 compliant.

8:37You knew that from a third party and you need that third party to close big deals. And if you want to get compliant easier and faster, you need to use Vanta, V-A-N-T-A. Vanta makes it so easy for you to get and renew your SOC 2. On average, Vanta customers are SOC 2 compliant in just two to four weeks. Prepare that to three to five months without Vanta. And Vanta can save you hundreds of hours of manual work and up to 85 % of compliance costs. This is a total no-brainer. And Vanta does more than just SOC 2 compliance. They also automate up to 90 % compliance for GDPR, HIPAA, and more. You can't afford to lose out on major customers.

9:15We all know that. Listen, it's a hard year. Last year was hard. You can't lose those major customers because you don't have your compliance dialed in. Just work with Vanta. Get your compliance automated and tight, and tight is right. Lock down those big deals. Here's the best part. Vanta is going to give you$1 ,000 off. That's 10 hundies get$1 ,000 off at Vanta.com slash twist. That's Vanta.com slash twist for$1 ,000 off your sock too. Walk us through for non-technical people who are listening, you know, maybe founders of companies or capital allocators, exactly how I would start this process of taking my data.

9:50And I'm going to use myself as an example. It'd be easy for people who listen to this show to understand. We have thousands of meetings and notes from thousands of meetings. we have thousands of applications of startups asking us for funding we now have those in notion we have zooms calls with transcripts and summaries that we've done with ai and there are external data sources like crunchbase or linkedin that have signals that a startup has done well i.e downstream funding it's not perfect data sets but there's pitchbook there's crunchbase there's how many employees they have listed on linkedin not perfect but directionally correct so i want to see if i can moneyball startup investing you know i don't know if it's necessary for us since we have incubators and accelerators that let us place a lot of bets but i do want to do this uh at some point just for the giggles and to see what comes out of it yeah how would i start that process i have a non-technical team let's say of investors and a you know technical and that we can talk about technology but not developers how do i start this process of taking a corpus of data let's call it a thousand meeting notes or five that five thousand meeting notes uh and applications of startups how would i what would i do walk me through it yeah so i could first preface this with uh our tools are meant for technical audiences like ml engineers data engineers uh but there there are sort of conceptually three major ways to modify the behavior of a language model right so we'll start with language because we're talking about text.

11:25So when you're in a smaller data regime, when I say smaller, I mean that which was fits in a book, like less than a hundred thousand words say, uh, then we can use things like prompts, prompt injection, um, to, to actually change the behavior. So actually the model we released, uh, less than a month ago called MPT seven B, uh, has essentially an infinite prompt window. So we, we tuned it to have 64 K tokens. Uh, the token is about three quarters of a word. of a prompt. Explain what tokens are for folks and prompts just so we really can explain what's happening here with machine learning in this process.

12:05Yeah, token is really the important part of a word. So if I said evergreen, we think of that as one word, but it's ever and green are tokens. The vast majority of words, the token and word are the same thing. So that, boy, girl, those are all single token words. So that's why we kind of give this ratio of about 0.75 of, you know, tokens to words. Yeah, words to tokens rather, sorry. So when we, the way these models work is that we tokenize the language into these, into these blocks that are meaningful, a word or some piece of the word. And then that becomes the input to the model, which we call a prompt.

12:43That prompt sets what we call a context. context it's like uh i'm telling you hey jason we're going to talk about uh evergreens so now if i say something like naked seed that makes sense to you right so because i set the context really this is what a prompt does is it sets a context for a model and then it can sort of recall knowledge that has been trained upon from that context this is why prompt length is actually quite important so uh what we enabled was a very long context window that you could actually feed a whole book we in fact fed it the entire great gatsby and we asked it to ask it to write the uh the epilogue uh made up epilogue and it did so and it can do that across the entire context of the book it's imagine imagine like reading the whole book keeping it all in your mind and then writing writing it out that's what the model is doing so i could take smaller i could take the i could take take a successful company like amazon or netflix have some research on that company like a research report that was written plug that in and say of these thousand meetings i've done do you see any companies that would correlate with this company in some way and i could use the prompt of a 50 000 word gartner report or goldman sachs report on amazon or something from 1999 or 2000 i could take bill gurley's reporting on you know amazon from the 90s or 2000s mary meekers and start using that as prompt engineering for looking for patterns in startups huh something like that correct that's right uh prompting and context windows are i'll call them the weakest form of learning in a sense where uh you can take information put it in the prompt and have the model you know do some analysis on that the problem is sometimes there are weird conditions where let's say there's conflicting evidence from where the model was trained versus what was inserted in the prompt, you might get, you know, kind of undesirable behavior in those cases.

14:41So that backs us up to one more version of how we modify the outputs is what we call fine tuning. Fine tuning allows us to kind of condition the model to act in certain ways. Like if I ask a model, you know, racist questions, maybe we want to say, Hey, I don't, I don't, I don't want to talk about that subject. So we can condition it to do that. It's very similar to a human. Like if I put a human in a call center, I know they're talking about customers. I'm like, hey, don't talk about our competitors' customers, right? Or don't talk about our competitors. Just talk about our products. Don't talk about politics, right?

15:15Don't use swear language. These are things that I might tell a human, right? And so we can actually use fine tuning to condition the model to give us outputs that are like that. You can even imbue new knowledge through fine tuning as well, but I would argue that doesn't work quite as well. The real way I think to describe or modify the behavior of a model in a very profound way is using pre-training and data mix. So pre-training is where we take a model that doesn't know anything and we train it on a bunch of data. And the way we do this is actually an optimal mix of maybe some domain-specific data along with some general data.

15:51It's actually kind of similar to education, right? I have a child, I'm going to put them in school. I'm going to teach them about history and politics and math and science. And at some point later in life, you start to specialize, right? It's actually a similar process with, uh, with LLMs. And so in this example where I'm dumping in my startup data, um, what would be then the next steps, uh, for me to get value from it? What would I do once I've got the model set up? Yeah. So I think the first thing is to analyze how much data you're throwing at it. So if you're under that 100 ,000 word limit, then you're probably in the regime of tokens, sorry, prompts into tokens.

16:31If you're in the call it 100 million, so range, we can start talking about fine tuning. If we're now in the billion range, we can talk about pre-training and layering in this data. So that's really the analysis that we kind of walk our customers through. Typically, it's like which method you want to use depends on how much data you have to throw at it. Typically. So in your example, you said 5 ,000 transcripts, something like that. That's probably in the prompt regime. We're probably not doing anything beyond that. We may be able to do some light fine-tuning to actually condition the model to act in certain ways.

17:08Like, hey, I want this kind of information pulled out. Like, I want to know something about the quality of the founders. I want you to focus on that as an output. Right? Got it. I can condition the model to do that with fine-tuning. Got it. so i could say hey what's the problem you know because typically if you backed out of a deck the deck structure was architected to convince investors investors were optimizing for big problems solving big problems with high margin businesses with people who had great backgrounds who could execute so you could actually like the deck having a competitive landscape or the total addressable market or the problem and the solution those are the things that investors would go to first we know this because when you send a doc you send or some of these tracking software is a little bit creepy but it will tell you how long people spent on each page which pages they zipped right over like advisors who cares you know uh you know right there's a lot of stuff is uh you know thrown into decks just for performative reasons but the problem and solution and the background of the founders are paramount the number of customers and the pricing paramount the business model so you fine-tuning would be essentially that process of trying to tell the model this is important that's not as important um that's right and and you can even link it to outputs right we call this process reinforcement learning with human feedback rlhf and uh actually what you do then is you say well the inputs are all this all this deck material say and um then And these companies did really well and those companies didn't.

18:44Right. You could actually start to link it to an output and you can start saying, hey, show me companies that you think are going to do well. Right. And it can actually kind of pick up some patterns. And it would be different for a C stage investor. It would be they got to a billion dollar valuation. For a late stage investor, it might be this company went public or got bought for over a billion dollars. And so you could actually have two different outcomes could be defined as success. Totally. you know for y combinator or r accelerator at launch like you know success might be the company gets past 200 million dollars because we're investing in low single digit valuations when companies are just starting out and they're just ideas so you have a totally different uh approach there um so you're in competition with some of these open source projects is your solution open source and you know maybe you could speak to who's going to win ultimately uh having the great language models is it going to be the person with the greatest data the person with the largest open source community fine-tuning the open source projects to analyze that data who in your estimation is going to win the day or will it be parity where you know having a web a cdn a content delivery network sure there's on the margin some that are faster than others and you can probably have debates with the sys admin all day long but the fact is in 2023 you throw up any any of the top 10 cdns your site's going to work really well there's parity right right yeah yeah i mean uh so our models are open source we open sourced our 7b model a little over a month ago and or a little want to give people great starting points to get going.

20:32I think for us, what we're learning is our customers are on a journey here. This is all very new, right? It's new for every company right now, and they all want to do it. It just comes, people come at it from different starting points. Some people are like, okay, I'm a hundred percent in, I'm going to budget$10 million. I'm going to go do this. Okay, great. We can help you with that pre-training side of things. Some are like, well, we're dabbling. We want to understand how we can add value to our customers. Can we start with a smaller byte. So we want to meet them where they are. And open source models are a great way to do that.

21:01They can start with the open source model. They can fine tune it. It's relatively cheap. And then eventually start integrating to the application and then customizing even further. So we want to, we have the whole breadth here and open source is a very big part of that. I think who's going to win out of this is the one who can serve their customers the best. I don't think those principles are going out the window. Everything that you've talked about over many years, those things are still real, right? I mean, at the end of the day, you've got to give your customer something they want. If that means taking an open source model and fine-tuning it, and that's good enough, great.

21:36If it means that you need to pre-train something, that's fine too. If you're ChatGPT and OpenAI, like, yeah, they got to go build their own thing because that's their competitive advantage. I don't think that's true everywhere. But what we are seeing now is that the game is ratcheting up pretty fast. Like if I have something that I put in front of customers that interacts with specific kinds of data. Getting really good at interacting with that data probably means you need to own how that model works. If you don't, your competitors can buy that thing too. If I'm integrating an OpenAI API, maybe that's a great way to get started, but I don't have much of a competitive advantage because my competitors can go do exactly the same thing.

22:16You basically have decided to be on par with everybody. uh and everybody will get to the same place whereas if you have your own proprietary model and you're tweaking it and tweaking it everything you do past that open source moment where you use the open source software you own and those are accrete to your product or solution not to and this is distinctly different than just hosting picking where to host your server when you pick to host on amazon or google or rackspace azure whatever your the act of hosting on azure doesn't make azure or google cloud or aws better but the act of hosting on chat gpt or bar does make those models better correct and that is a subtle point where it can i guess yeah yeah yeah yeah so the physical infrastructure it's it's interesting i mean nvidia clearly had a huge bump recently they are the backbone of all of this both training and inference right now i mean we we we're actually encouraged we encourage many different types of hardware vendors to come to us and we want to run on their stuff nvidia is great they build really good products and we're running on top of them uh then the clouds are sort of the channel through which you get gpus right uh they also have some types of differentiation i mean network interconnect and you know reliability failover all this kind of stuff.

23:41You know, we find that it comes largely down to availability and price is the biggest differentiator along with some of these other more minor things like network capabilities. And really customers want choice right now. They want their cloud as a relationship that they do a lot of things on because they run their business on it. And they want some choice here. It's like, hey, you know what? I don't want to be like in a vice with one vendor because of this relationship. I want to have some choice. And so that's where the multi-cloud thing actually became a pretty good value prop from their perspective.

24:14Not every customer, but some of them. In that case, I'm building, AWS is giving me a great price, but Azure just dropped the prices massively. They're trying to win our business. I've got to keep running this model, growing it. It's not cheap to run these models. Can you give us an idea of what my, the job I gave you of my 5 ,000 meetings, what do you think this all costs to you know run these models at scale to add you know a thousand you know uh new startups a month to it and and really keep growing what is this going to cost well i think there's some misnomers out there that some people believe it's like you need to be at 30 billion to to build a model that even matters that's not true and uh but i think like in the level you're talking about let's start with your you know thousand documents when you're talking about hundreds of thousands or even maybe tens of millions of words it's really pretty cheap this is on the order of 100 bucks 100 bucks we could get we could do a lot in terms of fine-tuning um a thousand bucks you can do a really a lot so it's really not that hard uh but when we start talking about pre-training building models from scratch i'll give you the numbers our seven billion parameter model was trained on one trillion tokens, 1 trillion with a T.

25:29So that's approximately 750 billion words. A very long book has 100 ,000 words. So you can kind of do the math. It's a lot of content. That model took nine and a half days on 440 NVIDIA A100 GPUs, and it costs about$200 ,000 to build from scratch. Just to build that one model one time. Correct. And you run it again. You have to run it again. And you don't own all those. You rent those. You timeshare them. Correct. On other platforms. In the cloud. And these platforms in the cloud. Trovata is a cash management platform that helps you keep tabs on your runway, which is super important when you have to answer investor questions.

26:16And this is just going to gain control over all your financial data for you. Trovada scales from seed rounds all the way up to your IPO. And it makes it easier than ever to manage multi-bank liquidity with a single source of truth. You know, everybody now is putting their accounts and your money, you're splitting it across multiple banks. So you're protected with that FDIC insurance. Startups shouldn't be managing their lifeblood, aka your cash position in a spreadsheet. No. don't opt for bulky solutions that take months to implement when the banks now have super fast API connectivity. With investors like Wells Fargo and JP Morgan, Travada has pioneered the largest library of corporate bank APIs.

26:58Tons of unicorns like Carta and Fanatics trust Travada to gain visibility into their multi-bank data. It's the cash command center that helps you analyze, report, and forecast cash like a pro. Recently, they also launched Travada AI. This is the first genitive AI for fintech that uses GPT to automate cash reporting and business intelligence, while keeping your data private and secure, of course. So here's your call to action, go to travada.io slash twist to get started for free. Use the code twist for 30 % off premium features for one year, like AI forecasting and reconciliation. Educate the audience on the utilization rates of these in the cloud right now because we're hearing hey a100s h100s whatever there's a line around the corner we saw nvidia have this huge spike a couple billion dollars and yeah um you know unexpected orders came in uh so they're doing fantastic but uh if you want to run one of these models for your company are you waiting in line to get access to them do you have to reserve them is there like a line out the door to just use them or can you just use them anytime you need to well yeah it we are in a gpu crunch no doubt about it and that's not going to alleviate for a while i'm happy to talk about why that is as well it's a lot of time in that in the semiconductor industry um but right now um if you're willing to sign longer term contracts you can generally get them so we we as a company actually have blocks of gpus that we buy and we can bring to customers we call that a 1p deployment a first party deployment where we basically create a tenancy for our customer with GPUs that we already have contracted.

28:38So that actually works great. They basically pay us for a block of time and we can run those things and get them access and do it very efficiently and effectively. The other way we deploy things is within the tenancy of a customer. So a customer has a relationship with AWS. They believe they can get AWS to give them GPUs. We can run our software stack inside of their tenancy without ever seeing their data. That's something that people like because of the security and privacy. But as he said, the shortage of GPUs starts dictating how people go here. And so we actually do have a large number of GPUs.

29:12I don't necessarily want to comment on how many, but in the several thousands range that we can bring to bear. Now, the reason this is an an issue is that we're seeing scale scaling up these neural networks matters right for a 7 billion parameter model i needed on the order of four to five hundred gpus uh that wasn't true two years ago people weren't doing this and all of a sudden it's like what you would do on four or eight or 16 gpus now you're thinking i need you know 400 and so the the demand just went through the roof the new h100 that's the latest gpu from uh from nvidia uh is going to help a bit in the sense that each one is faster than the previous generation.

29:52So you don't need as many. But I think what will happen is it's sort of like goldfish. You grow the pond you have. As the capabilities of the hardware gets better, people just want to use more of it. And I anticipate us using routinely 1 ,000 GPUs for customer workloads. So that crunch is going to continue. Now, why do we have a crunch? I mean, can't we just crank out more silicon, right? uh what i think the the world doesn't realize is that there's really three places in the world that can build state-of-the-art silicon uh tsmc taiwan semiconductor that's the biggest one and that's where nvidia has a deep relationship samsung is another one that nvidia also fans on and then intel uh as a fab and intel primarily focuses on cpus for their um for their fabrication capabilities.

30:42And then beyond that, there's something called high bandwidth memory, HBM memory. It's packaged within the same physical package as the GPU. The process of packaging and getting memory together and making it all yield is actually the biggest bottleneck. There are two places in the world that make HBM memory, Samsung and SK Hynix. So this is your supply chain for these things. And there just ain't a whole lot more left of it. And to build out capacity means you got to build a whole new building you know and that's part of what the chips act was trying to do here in the united states is to create some redundancy have some of these on the north american continent and maybe have less dependency on regions that could be impacted by uh geopolitical events taiwan fill in the blinds right yeah um and so ramping those fabs up is underway but it this is a non-de minimis task it is a significant task to put one of these to stand one of these up this is a couple year process yeah i mean two to three year process and you know on the order of 10 billion dollars of investment to build a state of the art fab if not more these days so it's not small uh and it takes time i think that's the other part is that like if you want to react to a change in demand which there has been a big spike in demand the reaction time is a minimum two two and a half years just to build the capacity then you got to deliver that capacity so it's another year beyond that it's like a three-year minimum kind of thing the software and the models are getting so good um hugging face has like uh um a leaderboard of the models you're in the top 50 models and you have all these different players trying to make language models open source them and make them better so is it not true that these base models are going to be built and a lot of the demand to use them is not going to require they're going to get so good that maybe you're just not going to require to do as many new models or is it just induction where people like well i can make a new model i should run a model for my vertical etc yeah i think what's happening now is where one can't get the resources they're just going to take the other approach of I'm going to use an existing model.

33:00Great. It's a practical approach. They are giving up performance knowingly. If they had the capability to build their model, they can and will. So I think the demand is not going to tap out because of this. Like if I want to build a better model to be competitive in my space, I will. And if I can get the resources, I'm going to go do it. I might be strapped by resources, not be able to get them. We've taken a fundamentally different approach to a lot of companies in that we focused on efficiency of compute from the get-go. So meaning that, can I do more with less? Can I build a big model and make it cheap?

33:36So the reason that our model is state of the art and only$200 ,000 is that we put a lot of engineering time and research time into making it very efficient. We use that GPU completely. It's like, when I kill that animal, I'm going to eat everything, kind of a thing. And we're going to continue to do that. So we're getting more and more efficient with it but honestly the demand is going so fast that even with our efficiencies which bring nearly an order of magnitude of efficiency compared to what it was a couple years ago uh there's still not enough gpu compute and i think we are an absolute requirement to make this happen still not enough uh people are going to be seeing that they can build a better model and get an advantage and there's going to be an economic incentive to do it it's just they can't buy the gpu talk to me about the difference between specialized models and the general models and how this is going to play out because you know reddit bloomberg twitter quora these are very unique data sets not only are they unique data sets they've already have built into them some amount of categorization i.e a subreddit i.e a topic on quora that this is a legal topic versus a health topic i know it's the model can figure that out itself but the fact is these are very structured sets of data that have been built for decades that are really unique in the intent in building them stack overflow would probably fall into this so how does this i guess balkanize uh or manifest itself in the next two three four years is reddit just going to have a reddit gpt and quora already has their own gpt basically an interface maybe twitter has their own bloomberg created their own a small investment i have a small company i have skipped skift.com created their own based on their reports of travel companies they're kind of like a verticalized b2b travel publisher plus the transcripts of all their interviews plus all the research and the companies that they cover and they've made their own narrow language model uh how does this all pan out in the coming years so i'll give an intuition first uh before i go into the answer here i think the way to look at it is you know if i want to be of i want my kid to be a famous violinist what do you do you don't start them at 20 years old you start them at four right um arguably if they're if they're going to be a virtuoso and violin they're probably not going to be a finance virtuoso because they're going to put a lot of time and effort into making their brain very specialized toward that task.

36:14Even with everything that biology has given us in our brains, we still need to specialize to be really good. So right now we're at the very beginnings of this. Yes, you can talk about, I can build a model that can do a lot of things. It's going to be a jack of all trades and actually not a master of anything specifically. And that's okay. There are tasks where I want something general, right? If I want to take over multiple tasks that maybe people do or find mundane, I maybe want something general for that. And those general models will work for that. But when I really start getting into healthcare, being a co-pilot for a doctor or a nurse or a co-pilot for an investor, then I need some really kind of specific knowledge.

36:57And it's very difficult to make a general model do really well in specific tasks. That's one. The economics of building a general model that could potentially be very good at every task start getting kind of out of hand. I mean, just the training of itself gets very expensive. Then, because that model itself has to be so large, serving that model just has very unfavorable economics, especially when we're talking about compute being so scarce, right? Training and inference compute is basically the same kind of chip. So now I got to start thinking about, well, all right, if I want to actually serve this model to my customers, I need to think about the economics of serving that model.

37:33So I think we're going to be in a world where there are going to be some large general models and they serve some set of use cases and the cost to serve them is justified. There's going to be a whole tale of multiple expert models that are much smaller, that have much more favorable economics, maybe are very good at particular tasks and less good at other tasks. If I'm building something that's going to do customer support, I really don't want it to philosophize by why Rome fell. It just doesn't need to do that. It needs to talk about my products. It needs to get the user, you know, to fix their problem ASAP.

38:07That's it. Right. I don't want it to do anything general. So I think this is what we're going to see is this world where everything's kind of, um, uh, coexisting and solving different problems. We're already seeing that, uh, now, I mean, I talked to the founder of a company called perplexity AI, which is doing like kind of, you know, a search and, um, you know, uh, finding knowledge across different sources using LLMs and they're, they're doing a whole bunch of different things. They use every possible model they can to best serve the task that they have. So sometimes they use a general model to do filtering and they use specific models to condition the output the way they want them.