Going Full Send on AI, and the (Positive) Impact of AI on Jobs, with Kevin Scott, CTO of Microsoft

0:05Microsoft, the BMF productivity cloud and gaming company, has taken a massive bet on AI. Everyone's paying close attention to its partnership with OpenAI, and the technical community has been amazed by its release of some of the first truly useful and broadly deployed AI products such as GitHub Copilot. Its full-on attack on web search with the new LLM-powered Bing chat is making its incumbent competitors dance. Today on NoPriors, we're thrilled to speak with Kevin Scott, CTO of Microsoft, and the driving force behind their AI strategy. Kevin's leadership, both at Microsoft and prior at LinkedIn, Google, and AdMob as a technologist is especially inspiring to me, given his distance traveled from his childhood home in rural central Virginia.

0:46In 2020, he published a book, Reprogramming the American Dream, about making AI serve a saw. Kevin, welcome to KnowPriors. Thanks so much for joining us. Thanks for having me, guys. Can you start by sharing with us some of your story? How does one go from a farming community in Virginia where your parents didn't attend college to CTO of Microsoft? I don't know. I think it is a very unlikely journey. It's certainly not a thing that I ever could have imagined. I think part of it is I was just super fortunate to be wired like a nerd and growing up when I grew up. So, you know, when I was a teenager in the early 80s, personal computing was happening.

1:29And, like, that was the thing that I happened to fixate on. And even though we were relatively poor, I managed to, you know, scrape together enough bucks to get myself a personal computer that I could have and just tinker with all the time. And it was like it was a Radio Shack color computer to like one of these things with chiclet keys that you you actually connected to a television. Like I had it hooked up to a 13 inch TV and it had a cassette recorder that you stored and loaded your programs on. And, you know, and it was just the thing that I was obsessed with. And I stayed obsessed with computers from then on.

2:10And it was just me trying to find a path at each step where I could work on the most interesting thing that someone was dumb enough to give me permission to go work on. And again, it's a lot of luck. Like, there's no way you can plan a path from rural central Virginia to CTO of Microsoft. But, you know, I think it does help to have a high-level vision in your head for what it is that you want to do. Like, just knowing what you're aiming for always helps. What was that vision for you besides, like, you know, obsessed with computers, wanted to work on them? Yeah, I more or less had two of them. So, the first vision I had when I was a teenager was I wanted to be a computer science professor.

2:55So I just looked at what computer scientists did and thought this is the most amazing stuff I've ever seen. And I went to a science and technology high school. And the way that it worked where I lived is like a really rural area. And so the science and technology, it was a governor's school. So it was centrally located in each high school in these four or five counties that surrounded the governor's school got to send two students each. And so I was one of the two students that got selected from my high school to go to this thing. And my computer science professor there was this guy, Dr. Tom Morgan.

3:37And I just sort of felt like he'd opened up this entire new world to me. Like it was just thrilling to learn all of this stuff. And I was like, yeah, I want to be like Dr. Morgan. And a lot of this stuff for me is about who those influential role models have been in your life. And so as soon as I met Dr. Morgan, I was like, oh, I should just go be a computer science professor. And that was the path I was on until I was about 30 years old when I was a compiler optimization and computer architecture programming languages person. And I got pretty disillusioned with what being a computer science professor actually was relative to what I wanted to do.

4:17Like, I just wanted to have a lot of impact. And my perception at the time when I was making these decisions was that you can have a lot of impact as a computer science professor. And the impact was actually great, but it wasn't the impact that the system appreciated. So the impact that you can actually have is inspire students to go pursue these careers and they will go on to do much greater things than you've done yourself. And that to me was the greatest impact, but it was the least appreciated part of being a computer science professor back in the 2000s when I was making these big decisions.

4:58And so I decided to leave, and I didn't at the time know what Next actually was going to be. Like, it had been my mission for almost 15 years at that point, and, like, I was a little bit lost. And I saw that a bunch of my academic buddies were all working at this startup called Google. And I didn't understand why they were working at Google. Like, Google was, you know, like some little box, and you typed keywords in, and it gave you 10 things. How is that hard? But, you know, Urs Holzler, who was a compiler person, and Jeff Dean, who was a compiler person, and Alan Eustis, who was a compiler person, like all of these people who, you know, who I went to conferences with and whose papers I read.

5:43And I was like, all right, well, maybe I should send my resume in. And I sent my resume in and got called to do a bunch of interviews. And it was the best interviewing experience I've ever had because they took what must have been every compiler person in the company at the time and put them on my interview panel. And I was like, oh, my God, this is amazing. I had the best day interviewing there. And I got this job offer. And I got this choice. They just started Google New York. which was the first office outside of Mountain View. And they were like, you can come to Mountain View or you can go be the 10th person in this New York office.

6:22And my wife and I wanted to live in New York more than we wanted to live in Mountain View. And so that's what we did. And after I got there, this is where the new mission came in. So we were hiring these brilliant, brilliant people at the time. And the way that we did hiring was kind of crazy. It's like, all right, well, if you're smart, just come work here. And, like, we have no idea, like, what exactly it is you're going to do. And you, like, came in and you sort of sorted yourself out. And we had these people who were so accomplished and so brilliant. And they would come in and choose to work on things that just were going to have no impact at all.

7:01Like, they were intellectually very interesting, but they were just sort of silly in that they were never going to connect with anything that moved the needle for the company. which was exactly the problem I was trying to get away from, you know, in being like a research computer scientist. And so I sorted myself out. Like I found like a pragmatic thing to go work on. Like, you know, I won't go into the details of what it is. But, you know, like the whole team won a Google Founders Award, which was a big deal for like solving this like very sort of unsexy problem with a bunch of very fancy computer science.

7:38which was one of the things I think Google did really well. And then I was like, okay, well, I should just go help more people sort themselves out as well. And that's when I became a manager. And then from that point on, it was all about like, hey, I want to help as many engineers as I possibly can, like make sure that their work lines up with something that's both interesting and meaningful. I think that it's actually pretty under-discussed the degree to which early Google had so many academics actually running important parts of the company. Yeah. I think ERS is a great example, and I think there's others.

8:13And so I haven't actually seen anything like that since until maybe now, more recently at OpenAI, there's more academics. So you feel like the research community is popping back up again. But it's been maybe a decade or two since that's happened. Yeah. I mean, I think that's actually a really, really great observation. So when I go sit in OpenAI, it really reminds me of early Google days. And it's about the same size Google was when I joined. And so I couldn't figure it out for a while. And I was like, wow, this is really giving me early Google nostalgia. And the conclusion to draw from that is not that they're the same companies or they're trying to solve the same problem.

8:52It's just sort of the energy of the place and who they've chosen to hire. Yeah, it's the first time I've seen string theorists getting hired again into computer science roles. Yeah, 100%. Since Google days. You and I probably both worked with Jonathan Zunger, who works at Microsoft right now. I remember, it's like, Jonathan's working on this big distributed file system stuff. And what's his degree? Oh, yeah, he's a string theory guy. Yeah. So a big part of your mission for the last decades has been helping string theorists and other engineers figure out how to be useful in their orgs. The other part seems to be, of course, like actual technical direction, right?

9:34Deciding like what's worth investing in. And you've worked on machine learning products for a really long time, like ads auctions at Google, recommendations at LinkedIn, et cetera, et cetera. Was there a moment when you decided or you realized personally that AI should be a key technical bet for Microsoft? Yeah, I mean, I've been at Microsoft a little over six years now, so almost six and a half years. And pretty quickly, it was obvious that AI was going to be very, very, very important to the future of the company. I think Microsoft already understood that before I got there. And then it was just how do you focus all of the energy on the company on the right thing?

10:12Because we had a lot of AI investment and a lot of AI energy, and it was sort of very diffuse when I got there. So no lack of IQ and actually no lack of capital spending and everything else. But it was just kind of getting peanut buttered across a whole bunch of stuff. And so the thing that really catalyzed what we were doing is, I mean, maybe this is a little bit too technical, but before I got there, the technical thing that had been happening with some of these AI systems that to me was very interesting is transfer learning was starting to work. So, like, you were going from this mode of, you know, the flavor of statistical machine learning that I cut my teeth on, like, in my first projects at Google, which was, you know, you have a particular domain of data and, like, you have a particular machine learning model architecture that you are, you know, you're training in, like, a particular way that you're going to go do the deployment and measurement and whatnot.

11:18and it's all siloed to a use case or a domain or an application, to seeing AI systems that you could train on one set of data and use for multiple purposes. And you saw a little bit of that with some of the cool stuff that DeepMind was doing with reinforcement learning, with play transfer across some of the gaming applications that they were building. But the really exciting thing was when it started working for language with Elmo and then Bert and then Roberta and Turing and a bunch of things that we were doing. And that was the point where there were so many language-based applications that you could imagine building on top of these things if it continued to get better and better.

12:05And so we were just sort of looking for evidence that it was going to continue to get better and better. And as soon as we found it, we just started all in. That was everything from doing a partnership with OpenAI to, you know, like at one point I seized the entire GPU budgets for the whole company. And I was like, we will no longer peanut butter these resources around. Like we will focus them because it's all capital intensive. It's like we will just allocate these things to things where we have really, really strong evidence-based conviction that like a particular path is going to benefit from adding more capital scale.

12:39I remember, it must have been like five years back now, we were at dinner and now GPU capacity is the talk of the technical town, right? But you were like, I asked you what your like most pressing issue was. And you're like, how am I going to spend on GPUs this year? And how I'm going to distribute those GPUs? Yeah, and it was, and it has been. It certainly hasn't gotten any easier. But I mean, so Eli, I think the question you were asking is how we decided to do the OpenAI partnership. And so the reason that we did the partnership was twofold. So, one is with transfer learning actually working, you can imagine building a platform for all of this stuff so that you're building single things where you're amortizing the cost of the things across a whole bunch of different applications.

13:27And because we have a hyperscale cloud, like one of the things that I was really, really interested in and like beyond interested, like it felt just like an existential thing is how do you make sure that the way that you're building your cloud all the way from your computing infrastructure, your networks, your software frameworks and whatnot, how can it really serve a whole bunch of interests beyond your own? And so, like, we felt like in addition to the high ambition things that we were doing inside of the company that we needed, like, high ambition partners. And when we looked around, like, OpenAI was clearly the highest ambition partner that was in the field.

14:07You know, and I think still their ambition is just breathtaking in what it is that they're trying to accomplish. And so, that was one thing. And then the second thing was like, you know, they really had a very similar vision to the one that I had about like these things were evolving into platforms. And like we were able to, because we were so aligned on vision for the future, like we could figure out how to do a partnership where like even though like there's just a ton of difficult things. And like, you know, I think there's probably some conservation law of, you know, the stress from difficulty.

14:40So it's not like it ever goes away, but it's stress in service of a common goal. And that's the thing that make good partnerships work. I think one of the stunning things about the partnership in some sense was the timing. Because if I remember correctly, Microsoft made its first investment or its first significant investment in OpenAI right after GPT-2 launched. They were right around GPT-2, and this is before GPT-3, and there was such a big step function between the two of them that I think it was less obvious in the GPT-2 days that this was going to be as important as it was. And so I'm a little bit curious, what were the signs that made you decide that this was a good partnership to have versus building it internally versus, you know, usually as a larger company, there's the old buy-build partner kind of thinking.

15:22And so I'm just sort of curious how you all decided to partner in this moment in time where it's very non-obvious and you invested a large sum of money behind that. Yeah, and I don't want to have revisionist history and paint a rosier picture than there actually was. So there was a huge diversity of opinions inside of the company on the wisdom of doing this. And so Satya has this thing that he talks about, no regrets investing. So, like, there are things where you do the investment and, like, there are multiple ways to win. And, like, you even win a little bit when you lose. And so, this was one of those no regrets things in that, like, the very, very worst thing that could happen is we would go spend a bunch of capital on computing infrastructure.

16:11And we would learn, like, what to do at very high scale for building these AI training environments. And, you know, you'd have to believe something very strange about the world of AI that you wouldn't need advanced computing infrastructure. And then there were just multiple ways where, you know, like, and we had a bunch of evidence that, you know, we had gathered ourselves and that OpenAI had that gave us, you know, which unfortunately I can't talk about, but like that gave us, you know, pretty reasonable confidence that scale up was actually working. You've probably seen the famous OpenAI compute scale paper where they sort of plot on the log scale, like how many petaflop days or whatever the unit of total compute they were using on that graph that shows from 2012 when we first figured out how to train models with GPUs through, I think, the plot ends sometime in 2018.

17:08that we're basically consuming 10 times compute, more compute every year for training state-of-the-art models. And so I just had super, super high confidence that we were never going to get to the point where we're like, all right, we got enough compute. It was a very bold move. I think it's very striking all the amazing things Microsoft has done over the last few years in terms of just incredibly smart strategic moves that at the time didn't seem obvious and now are just in hindsight. really brilliant. I guess a more recent move is you announced a collaboration with NVIDIA to build a supercomputer powered by Azure infrastructure combined with NVIDIA GPUs.

17:46Could you tell us a little bit more about your supercomputing efforts in general, and then maybe a little bit more about those collaborations, both NVIDIA and OpenAI on the supercomputing side? Yeah, so we built the first thing that we called an AI supercomputer. I think we started working on it in 2019 and we deployed it at the end of that year. And it was the computing environment that GPT-3 was trained on. And we had been building a progressively more powerful set of these supercomputing environments. We built them in a way where the biggest environment is just because they're very capital-intensive things, tend to get used for one purpose.

18:34But the designs of these systems, we can build smaller stamps of them, and they get used by lots of people. So we have tons of people who are training very big models on Azure compute infrastructure, both folks inside the company and partners who can come in. And it was a thing that was not possible to do before where you could sort of say, like, hey, I would like a compute grid of this size with this powerful network to do my thing on. And so, you know, NVIDIA has been, you know, our compute and network partners since they bought Mellanox, you know, for years now. And the thing that makes that work is generation over generation, like you're just getting better, you know, price performance from the systems.

19:27and we work super closely with them, like defining what the hardware requirements need to be in the coming generations of GPUs because we have a pretty clear sense of where models are going and what model architectures are evolving towards. And so, yeah, I mean, it's just been a super good partnership. You know, like we're deploying Hopper now at scale and, you know, like a bunch of the features of Hopper, like, you know, 8-bit floating point, you know, arithmetic and a bunch of other things are like things that, you know, like we've been planning for for a while. Yeah, I guess one last question on sort of this, both supercomputers as well as platform side of things is I'm a little bit curious how you view the world shifting in terms of closed source and open source models and, you know, the mix that'll exist.

20:19Because obviously, from an Azure perspective, lots of people are running open source models on top of Azure right now. Yeah, I mean, it is an interesting thing that people are framing it as some kind of binary thing. Like, I think you're going to have a lot of both. Like we still don't see any reason to believe that you're going to want to not build bigger models. But we just know in our own deployments, if you look at things like Bing Chat or Microsoft 365 Copilot or GitHub Copilot, you end up using a portfolio of models to do the work. And you use it for performance and cost optimization reasons, and you use it for just sort of precision and quality reasons sometimes.

21:08And so there's always this melange of things that you're doing, and it's never either or. I'm actually really excited by what's going on with the open source community. I think my biggest question mark there is how you go deal with all of the REI and safety things. But if you look at the technical innovation inside of the open source community, it's really thrilling. And we're doing some cool stuff right now. I was just playing around yesterday with that 12 billion parameter DALI 2.0 model from Databricks, which runs quite nicely on a single machine. And I'm still enough of a dork to love playing around with things that run on single machines.

21:54It's really, really impressive work. Yeah, it's super cool. How do you think about that from the context of enabling AI for your business customers outside of your core products? So is there a specific sort of B2B AI stack that's coming? Are there specific tools coming? To your point, there's safety, there's analytics, there's fine tuning that, you know, there's so much stuff that you could potentially provide. I'm just sort of curious how you think about that. Yeah, I mean, I don't want to turn this into some kind of weird marketing spiel, but we have this point of view that we started with this assumption that AI is going to be a platform and the way that people are going to most use of the platform is by building tools that assist people with jobs.

22:38So it's like less about these fully autonomous scenarios and more about assistive tech. And so the first thing that we built was GitHub Copilot, which is a coding tool. It's a thing where you can sort of say in natural language what you would like a piece of code to do, and it emits the code. And then you, as the developer, like the same way that you would take a suggestion from a pair programmer, like you scrutinize it and code review it and decide whether or not it makes sense for your application. And, you know, like that was the first version of GitHub Copilot that does a bunch of other things now.

23:14And so the thing that we have observed is this Copilot pattern is actually pretty, you know, pretty generic. And we built a bunch of Copilots since then. And the way that we built them, like there's a Copilot stack that looks almost like one of these OSI, you know, networking diagrams. And it starts with a bunch of user interface patterns that you have, like they're now an emerging plugin ecosystem for how you extend the capabilities of a copilot for things that you can't natively get out of the model. And then it is a whole stack of things, sort of an orchestration mechanism, like Langchain is one of the popular open source orchestrators.

24:03orchestrators, but there are a bunch of open source orchestrators. We have one that we've developed called Semantic Kernel that we've also open sourced. There is this whole fascinating world right now that didn't exist nine months ago around prompt construction and prompt engineering. So there's an entire art form and a set of tools that people have access to to design a meta prompt, which is sort of the standing instructions to the model to get it to conform itself to the application context that it's in. You have these new things, new software development patterns like retrieval augmented generation, or RAG.

24:45We were doing this before. It had a name on it. So it's basically a way to take the prompt that's flowing from the application and to inject context into the prompt that will help the model better respond. And then there's a whole bunch of safety apparatus that you have. So, it looks a lot like filtering on both the way down as the prompt flows through the stack all the way down to the model as well as, you know, as it flows back up. So, what things are you not going to let the application or the user send all the way down to the prompt because it's going to get a bad response back? Or what things are you going to filter out at the last minute because it is a bad response that has gotten all the way through?

25:33And sometimes you have multiple round trips through this cycle before you bubble the thing all the way back up to the user to get them the response that they need. And so we have a point of view about what this stack looks like, Like, you know, which Microsoft tools exist that will help people build these things? And, like, what special things you have to go do in the context of an enterprise to, like, answer the actual direct question. Where, you know, safety and data privacy and, like, understanding, you know, where the flows of data are and, like, which plugins can be enabled and, like, which can't.

26:10Like, all of those things, like, I think are getting built out right now. So, and like the other thing too, I'll say is like, we'll build some of this stuff and like the community is going to build a tremendous amount of it because like there's never been a platform or ecosystem where one company builds all of the useful things. Like that's just nonsense. Like it's just never happened. And to me, it's the sort of super exciting thing to just see all of the energy that's happening right now. Like I just like immediately before this call, I was doing a review with Microsoft Research. And it's just amazing to watch MSR, which is so many researchers there have pivoted what they're doing research on to like these AI adjacent or AI like on point things.

27:02And it feels a little bit like what MSR was like when I was an intern there in 2001, where you had all of these super bright people who had the tiniest little glimpse of what the future must look like that no one else had because it was the point where the PC was racing to ubiquity. And they were just all orienting their research around what that little glimpse was that maybe they had the earliest peek at. And it just feels magical. That's massive realignment of the research community right now, sort of in real time. It's very exciting to watch. I mean, and it's awe-inspiring. I mean, it's just crazy.

27:41It's hard to keep up. Like, super hard. Like, we went from, I mean, this has been the biggest surprise for me is, like, I just didn't realize that GPT-4 and chat GPT were going to catalyze as much of this as they have. We'd sort of kind of been expecting a bunch of this stuff. ChatGPT was a 10-month-old model with a little bit of RLHF on top of it. And by admission, not a beautiful user interface. It was just sort of a way to get something out there because you needed some practice with a handful of things before the big GPT-4 launch was coming. And like no one really knew that it was going to blow up this way.

28:29And it's only five months old. That was only five months ago, which is shocking. I think everybody forgets how little time has passed. Yeah, just shocking. But it is the open source community and like the, you know, big tech community, I think at its best is like, you know, everybody is sort of realigning to like what I think is, you know, unlike some of the other, you know, faddish things that have happened over the past handful of years. Like, I don't think this is a fad. Like, this is real. Yeah, I launched my new fund about six months ago with this AI focus. And a few weeks later, ChatGPD comes out.