Microsoft CTO Kevin Scott on How Far Scaling Laws Will Extend

0:00The things that are riddled right now where you're like, oh my god, like this is a little too expensive or it's a little too fragile for me to use. Like all of that gets better. Like it'll get cheaper and like, you know, things will become less fragile. And then like more complicated things will become possible. Like that is the story of each generation in these models as we've scaled up.

0:37On any given day, Microsoft may be the most valuable company in the world, and arguably no one has been more ambitious, more strategic, or more effective in its AI strategy than Microsoft. The key architect behind that strategy? Kevin Scott, CTO of Microsoft. We've had the pleasure of knowing Kevin for a couple of decades now dating back to his time at Google when he overlapped with our partner Bill Court. Bill will join us today for a very special episode of training data. We hope you enjoy.

1:14Kevin, thank you for being here on training data. Now, glad to be here. So just to start, I know you've talked about this before, but for our listeners who might to be familiar with your story. How does a kid from rural Virginia end up becoming the CTO of Microsoft? Who knows?

1:37Certainly not a not a repeatable plan, I don't think. I don't know. It is the thing when I reflect back on my story is it's just a lot of being at the right place at the right time. So I'm 52 years old. So I was, you know, 10, 11, 12 years old when the personal computing revolution started to hit full steam. And so like right at that moment when when you're a kid trying to figure out what you're about and what you got to latch onto. I had this really convenient thing that captured my interest and was a good place for me to ground my curiosity on. And I think that's one of the object lessons in general is if you happen to be interested and really motivated to learn more and do more with something that is at the same time growing really, really quickly.

2:44Like you probably are going to end up in a reasonable place. And so, you know, I was interested in computers. I, yeah, I was the first kid in my, or the first person in my family, not that my mom, nor my dad went to university. So I was the first one to graduate with a bachelor's degree. I majored in computer science and minored in English literature. I had this moment when I was trying to decide what I was going to go do after I got my undergraduate degree where my two advisors were arguing about whether it should be PhD in computer science or PhD and literature and I was very seriously considering both but I was so broke and just so tired of being busted all the time that I picked the pragmatic path.

3:45Not that, like I still imagine what my life would have been like as person with a PhD in English literature and I think it would have been just fine, but like I chose one of my two equal interests. And then, you know, for a while, I thought it was going to be a computer science professor. And at the last minute, this is where Bill and I intersected, like I decided I was a compiler optimization and programming languages person through years and years in grad school. And I got almost all the way to the end. I was like, I don't think I want to be a professor anymore. I would work on these things where it was six months of effort to write a paper and you make some synthetic benchmark 3 % better.

4:37I was like, this doesn't feel to me like the way to have a lot of impact in the world. I don't want to do this over and over and over again for the next 25 years of my life. And so I sent my resume cold into Google in 2003 and I got an email from this guy Craig Neville Manning who had just gone off to New York to open up Google's first just remote engineering office. And like I had an amazing interview at Google. I don't know whether this was on purpose or not or like I just got luck of the draw, but like it seemed like every compiler person who was working at Google was on my interview slate.

5:28And I was like, this is amazing. Like all these people, know all of this stuff that I know and you know, we can have easy conversations I worked on nothing that was even remotely close to compilers at Google and I was confused why all of these people were there. But it was a great interview and I was super stoked and I joined Google. And Google has yet another one of those things, just like the PC and just like the internet, like it was a phenomenon that was growing crazy fast with a bunch of smart people working there and that resulted in this opportunity I had to go join this startup ad mob when it was very early on, like right at this pivotal moment when mobile was taking off and you needed things in mobile like advertising infrastructure and I helped build the seminal company I think in mobile advertising.

6:25thing. And then was back at Google and then I helped LinkedIn go public running its engineering operations team and then I was at LinkedIn when we got acquired by Microsoft. So like, none of that I think you can plant. It's just like a lot of right place at right time and yeah, trying at every point you can to do the most interesting thing you can do on the thing that's growing really fast. You know, when you talk about your personal history, you come in, I guess, you know, the focus nowadays is on AI and machine learning. A lot of the practitioners are people with PhDs. How do you think about sort of practical teams for AI?

7:17Since you're obviously doing a lot of that work at Microsoft and involved with partnerships with OpenAI and others. Yeah, I mean, I think if you are building the really complicated platform pieces of AI, so like the big distributed systems for training and inference, the big networking and silicon and you know system software components or the algorithms that you're using to do training and inference. I think a PhD is super helpful like there's just a huge amount of prior knowledge that you need to have in order to jump into the problem space and be able to like go quickly and like, you know, you need to be clever, but like, you don't, PhD is, I like, yep, I know you have a PhD, Bill and a far cleverer than I am, but like, usually folks with PhDs are clever, but like, they're not the only people in the universe who are clever.

8:26So like, I think it's mostly helpful in the sense that you've gone through like a pretty rigorous training regimen where you get a whole bunch of prior art stuff into your skull and like you, you know, demonstrably can do a very complicated project. And, you know, the PhD projects, you know, look kind of like AI platform systems projects, except the AI platform and systems projects are lots and lots of people working together, whereas, you know, when you're getting your PhD, you often are like working in relative isolation a non -particular thing. So that's one of the things people have to learn is how to get yourself docked into a group and to be able to collaborate effectively with a bunch of other people like yourself.

9:16So useful. But there's so much else in AI that needs to be done other than building the platform. For those things, PhD is helpful, but certainly not necessary. Yeah, like figuring out how do I apply this to education? How do I apply this to healthcare? How do I, like how do I build developer tools around this? How do I do all of the million things that happen when new platforms emerge that you sort of complete the whole platform into like a portfolio products and a portfolio of middleware and a portfolio of like all of the other stuff you need. Well, speaking of which Microsoft seems like it has about the most sort of far reaching or ambitious AI strategy of anybody out there.

10:03Can you just kind of say in a couple words, what is the AI strategy for Microsoft and then just for fun? If you're going to grade yourselves, what have you done particularly well? What have you done? Maybe not as well as you could. Yeah, so I mean, we've been sort of talking about the strategy. Microsoft is a platform company like we, I think have participated or like helped drive a handful of the big platform waves and computing like we were certainly one of the pillar companies in the personal computing revolution like we had a important part to play in the internet revolution although I think that one was a far more or diversely contributed to revolution, then, yeah, then personal computing.

10:51Yeah, we kind of miss the mobile computing revolution. But like each one of those things, like we have thought about, how do you go build a technology platform for this particular era of technology that allows other people to go build on top of that platform to make useful things for other people. And so that is AI strategy. It is like how do you from frontier models to small language models to highly optimized inference infrastructure, you know, like hyper scale on both training and inference, like economies of scale, like making the entire platform more accessible because it's cheaper and more powerful with every turn of the crank.

11:46And like all of the developer tools and safety infrastructure and testing and everything that has to be there in order to have robustly built AI applications, like go build that and like listen to developers and listen to people building AI is as intently as you possibly can so that you are filling in all of the gaps that you can for them as they are encountering problems deploying this technology to users. So that is that is our strategy. And so yeah, I think we're doing a reasonable job of it.

12:33I don't know, like I hate to grade myself. It seems a little bit disingenuous. Right. I like to look like. Well, so maybe before I do that, like, you know, let me describe something about my own psychology. So I, like I am an engineer, and I think most engineers are like short term pessimists, long term optimists. And so the short -term pessimism is like you come in every day and you're like, oh my god, this is like a back of crap Like I just don't like any of this and like everything's broken and like I gotta Like I got I got so much stuff to fix and I'm so frustrated But you work on all of those things anyway because you're Optimistic that all of the problems can be fixed and that they're gonna be worth fixing at the end of the day And so yeah, I mean the There's a bunch of stuff that I think we're doing really well.

13:28I think we have absolutely, along with OpenAI, made a very powerful AI, dramatically more accessible than it otherwise would have been to a larger group of people. I think you, because of that work that we've been doing alongside OpenAI, we're just seeing lots and lots of customers who otherwise wouldn't be building powerful AI applications. And so I feel like we're doing a good job and the way that we're partnering, I think we're doing a good job and having a really particular point of view. And it's not an immutable point of view, but it's a point of view about what an AI platform to look like and we're trying to like make it as complete as we can.

14:18You know, low lights is I think we were a little bit late to like some of the basic AI stuff. So it wasn't that we were not investing in AI at all. And like you can sort of look at some of the work that Microsoft research had done over the years. And like MSR was an early AI leader. And I think, you know, the, you know, bit Bill knows this just as well as I did just from his time at Google and, you know, where we overlap for a number of years. But many, you know, maybe most of the really important advancements in AI over the past 20 years have been a function of some kind of scale. And it's usually you got data scale and compute scaling combination.

15:17Let you do things that weren't possible at lower scale points. And at some point, that scaling of data and compute is so exponential that you get past the point where you can have fragmented bets, where you can literally just become economically impossible to bet on 10 different things that are all exponentially scaling or have the ambition of the need to exponentially scale simultaneously. And so I think one of the things that we were a little bit late to is like we didn't put all of our eggs into the right basket soon enough. Like we just, yeah, we were spending a lot on AI, but it was fragmented across a whole bunch of different things.

16:03And because we didn't want to hurt any of the feelings of smart people or, yeah, whatever, I don't even know what the diagnosis was because a lot of that was before I was at Microsoft. We just weren't as quick as we should have been at like saying, no scale is what matters. And like here's how we're going to focus our investments on scale in a principal way. When did you get religion that scales what matters? Was there a particular event or a moment that really crystallized that for you? So, yeah, I mean, I was, so I've been at Microsoft for about seven and a half years now and like, my, when I, when I became CTO, my job was like take a scan left to right across both Microsoft and the entire industry and try to see where, yeah, we just had holes in execution where we were not doing things at that point in time, which was, I guess, 2017 early, where, all right, what are we not doing today that we're gonna deeply regret in 2019 or 2020?

17:23So like two, three years out. And like the biggest thing on the list was like, you know, our rate of progress on AI was not fast enough. So I'd say mid 2017, like I had religion that that was gonna be a big part of my job was like helping us figure out what the strategy was gonna be. And then in 2018, like if anything, the of the publication of the birth paper from Google was like a real crystallization of that belief. So like everything that I had that was in my analysis, I was like, this is as fine an example as anything of like why we have to really, really accelerate on getting more serious here.

18:15And so like, very shortly after that, like I restructured a whole bunch of stuff inside of Microsoft to get us more focused on AI. And then about a year later, we did that first deal with OpenAI. And yeah, we have been accelerating our investments and like trying to get more focused, more crisp, more purposeful since them. When you were very early to appreciating the potential of OpenAI, what did you see in and then at that time when that first partnership was true. Well, we had, like, or at least I had, this, like, real belief that what was happening with these models as they scaled, as they actually became a basis for building a platform that, like, the big shift that was happening is it wasn't just, you know, I ran one of these teams at Google where you had a pool of data and a bunch of machines and an algorithm and you were like training a model and like the model was for a specific thing like in the other case of the thing that I was doing at Google it was like click through rate prediction for advertising and you know, like a handful of other things and like just outrageously effective, right?

19:35But most of the work before this, before GPT was about those sort of narrow use cases. Like you were purpose building models for narrow things, and it was just tough to scale. Like you couldn't, you'd invest a bunch of compute and like you couldn't amortize the cost of the compute across anything more than just the narrow thing that you were building the model for. And yeah, I'd have a lot of expertise that you know, if you wanted to replicate all of this, It's like you had to have different data and different AI PhDs and different processes every time you wanted to go build AI into an application.

20:21What was happening was, yeah, you had these big, large language models that were useful for lots of different things. So you didn't have to have a separate model for machine translation and sentiment analysis. and all of the different text things that you were doing. And I was like, okay, this is extraordinary. And they were also becoming more platform like as a function of scale. So transfer learning was working better as things scaled up. And this is still the general pattern. So like everything that we understand that large language models can do plus or minus, like we'll get better when you get to the next scale point.

21:08And on top of that, they will become slightly or maybe dramatically more general in the sense that their capability set broadens. And OpenAI had that same belief, and they also had a very principled analysis So like how the those platform characteristics emerged over time is a function of scale at a bunch of experimental validation that said that their forecast were right and so it's like it just you know You sort of like look at what the forecast says and you like this is how much money it costs to run the experiment to see if you're gonna be on forecast for the next turn of the crank and like, you know, it was felt like a big number at the time, like it was billion dollars.

21:59But like relative to what was happening, it just wasn't a large amount of money. And then, yeah, GPD -3 was on forecast and GPD -4 was on, so like it just was, like finding a partner that had the same platform belief that you did and like a track record of being able to execute through these scale points. Like it was... It didn't... Like I've done a bunch of things before that I have way more reservation about in the past, like just in terms of investments. Like this one didn't... You know, like there were a bunch of people who didn't agree with me, but like I had pretty high conviction. You touch on investment, I guess.

22:44You know, there's a lot of trade publications now speculating about the cost of doing training and so forth and you know, rumors of billions and billions of dollars being spent and so forth. And I guess based on my own background, I think training is going to get dwarfed by inference here very soon. How do you see the nope so? Well, yeah, otherwise we're building models that nobody knows what to do with. That might not be a great investment. How do you see kind of computing landscape involved and where's it going? I think people are joking that all the money is going to Nvidia at the moment. Well, look, I think Nvidia is doing a good job.

23:31So like the two interesting things that are happening with these models just in terms of the efficiency of the scale up is each hardware generation's better price performance wise, usually by an extent greater than Moore's law used to work for general purpose computing. So, A100 was about three and a half times better price performance than V100, H100, like not quite that much, but close on paper. The next generation and looks very good as well. And so like you've got hardware for a variety of reasons, like you know, part of its process technology but part of its architecture and like a lot of it is like being able to leverage narrower word size in the computation.

24:34And so like, you know, instead of needing 64 bit arithmetic, like you're, you know, you're doing arithmetic with much less precision right now. And so like that, you know, there's just an embarrassing amount of parallelism there, then like we're getting better and better at extracting that architecturally in the hardware. And, um, yeah, there's a bunch of innovative stuff happening with networking as well. Like, we're well past the point for the frontier models at least where you can do anything interesting on a single GPU. So for years and years now, like both training and inference have been multi -GPU, multi -compute node problems.

25:13And so like there's a bunch of innovation happening on the network side as well, which allows you to strap all of the compute together, like at the chassis level, the rack level, the row level, the data center level, more effectively, which is great because, you know, for the nerds listening, like we haven't had effective power scaling or denards scaling since 2012 or so. So, yeah, we're not, we're getting more transistors but like they are not, they're not getting cooler. Yeah, and like we just, we have a lot of density issues just with power dissipation that we have to go deal with. Do you see inferences driving different data center architecture?

26:11Yeah, I mean, look, we already architect our training environments and our inference environments differently. they just need different things. And like I think, you know, all the way down to, you know, Silicon and like through the network hierarchy, you need different things for inference. And like, inference is, inference is kind of easier than training, like training the way that we're doing it now is like we go build big environments that take, you know, a few years to build.

26:45And yeah, with inference, if somebody came along with a better Silicon architecture, a better network architecture, like a better cooling technology, it's a much easier experiment to go run. You just go swap some racks out. I mean, like my data center people would yell at me, like it's not quite that easy, but it is easier than having to go do a big capital project. like a training environment looks like. And so intuitively you would think that that is going to result in more diversity in the inferencing environments and more competition and more like a faster rate of improvement. Like that's, and on the software side that's certainly what we see.

27:35Like the inference stack just because it's such a large fraction of the overall compute print and it's constrained because we have more demand and supply at the moment. Like you just have very, very powerful incentives to go optimize the software stack to squeeze more performance out of it. You think we'll be in an environment anytime soon where that demand supply balance changes? Not necessarily at Microsoft, but it feels like we're seeing that at the market level as well. Yeah, I don't know. I mean, if we continue to see the platform continue to expand capability -wise, and it just becomes more useful, like I think demand increases if anything.

28:26Now, the shape of the demand is probably going to move around. Like, I think you're already seeing a little bit of that. building a frontier model is like a very, very resource intensive thing. And as long as people are building frontier models and making them accessible, and maybe they're not accessible quite the way that people want. like they're only API accessible and like there isn't an open source thing, you can go instantiate and muck around with. But it's way more accessible than it was six or seven years ago where the only way to access some of this stuff is you had to go work for two or three tech companies.

29:11And so anyway, but I think you do have to ask yourself, and somebody else should do the asking right, because I'm like all kinds of bias, right? But I don't know how many frontier models you actually need. If they're all roughly speaking in the same tier of capability. Yeah, that's an awful lot of money to spend for things that are roughly equivalent. It's sort of like, if you're starting a company right now and you believe that you have to build your very own frontier model in order to go deliver an application to someone, That's almost the same thing is saying like I gotta go build my own smartphone Hardware and operating system in order to deliver this mobile app Like maybe you need it, but like probably you don't and so I mean to build point like I think the thing that Make sense, you know for the market is like you just you you're gonna want to see lots of people doing lots of inference because that means you've got lots of products that have found product market fit and those things are scaling.

30:32But like lots of speculative dollars flowing into infrastructure R &D, like probably ends the same way that the, you know, many speculative infrastructure blooms who ended. on this, excuse me, on the scaling front, Microsoft published a paper sometime ago pointing out that the quality of training data is maybe at least as important as volume. And I think one of the speculations you see now in the industry is that we're running out of sources of high quality training data and you're reading at least some articles claiming that various partnerships are being struck to get access to training data and they might be behind paywalls and so forth.

31:30How do you see that evolving? Because it feels like we have more and more computation but we may not have more and more training data. Yeah. Yeah, I mean, I think that was almost inevitable. It is, in my opinion, a good thing that quality of data matters more than quantity of data because of give or you an economic framework to go do the partnerships that you need to go do to make sure that you're feeding your AI training algorithm a curriculum that is going to result in smarter models. And like, honestly, not wasting a whole bunch of compute feeding at a bunch of things that are not. And I think from an infrastructure perspective, one of the things people have been very confused about is like a large language model is not a database, it's not a repository of facts.

32:28It's like important for it to quote unquote, well, knows some factual things, but it is like the world's crappiest database, honestly, like if you need it to be your retrieval engine. And so like you just shouldn't think about it as, like, hey, I got this thing and like it has to have everything baked into it, like into the model weights themselves so that you can recall a bunch of stuff. We got, you know, like as you've seen, like the recall is imprecise in the same way that human recall is imprecise. So they're just much more efficient ways to do recall. Yeah, so look, I think you are, I mean, the way that we see things developing is like you have data that is valuable for training models and then you have data that you need to have access us to for an application in order for the model reason over.

33:31And like those are two different things. And I think they're probably two different business models around those things. So at the end of the day, this is all just, this is about business models, right? Like people who produce data want to be compensated for use of that data. And so, yeah, we have all of this data sitting inside a search engines right now, like not in randomized weights, but quite explicitly it's like sitting in indices and being in Google and whatnot, just waiting to be retrieved. And plus or minus, everybody's okay with that because there's a business model there that makes sense.

34:22So you enter a query and you're either sending traffic or there's SEO and advertising a whole bunch of business model that surrounds that. I think we'll figure out a business model for that referral data so that when an agent or an AI application needs to retrieve some information from someone so that it can reason over it and give the user, an answer, like we will figure out the business model for that. Like it'll either be subscription revshare, it'll be licensing, it'll be like some new flavor of advertising. Yeah, I was just telling someone the other day, like if I was in my 20s right now, like, you know, for all of your entrepreneur, like somebody ought to be out right now figuring out what the new ad unit is for agents and like just building the company.