#144 Matt Hicks: Red Hat’s CEO on Open Source, Linux, and the AI Revolution

0:00If you go back to the early 2000s, the debate was operating systems. They may sound boring, but like these are the things that run our hospitals, our airlines, our defense systems. How on earth could you have every line of code visible to the world because all the bad actors could just find their exploits in it and then risk health and travel and national security and all the things. AI with the right structure and creation to it can have a lot more good than bad, but you always have to think through the bad and the malicious uses in technology. Hi, I'm Craig Smith, and this is Eye on AI. In this episode, we're joined by Matt Hicks, CEO of Red Hat, to explore open source's pivotal role in technology innovation, particularly focusing on large language models.

0:57Red Hat, known for its proficiency in infrastructure software, bridges the gap between model development and deployment. Matt, with his 18-year journey at Red Hat, shares his profound experience. Whether an open source enthusiast or curious about the future of generative AI, this episode is packed with riveting discussions and enlightening perspectives. Let me mention our sponsor, NetSuite by Oracle. If you're a business owner, having a single source of truth is critical to running your operations. So remember these three numbers. 36 ,025 ,1. 36 ,000 because that's the number of businesses that have upgraded to NetSuite by Oracle.

1:46NetSuite is the number one cloud financial system for streamlining, accounting, financial management, inventory, HR, and more. The number 25, because NetSuite turns 25 this year, that's 25 years of helping businesses do more with less, close their books in days, not weeks, and drive down costs. One, because your business is one of a kind. So you get a customized solution for all of your KPIs in one efficient system with one source of truth. You manage risk, get the most reliable forecasts, and improve margins. Everything you need all in one place. I'm not the most organized person in the world, and there's real power to having all the information you need in one place to make better decisions.

2:42This is an unprecedented offer by NetSuite to make that possible right now. Download NetSuite's popular KPI checklist designed to give you consistently excellent performance. Download it for free at netsuite.com slash IonAI. They support us, so let's support them. I'm Matt Hicks. I'm the CEO of Red Hat right now, and I have been at Red Hat for almost 18 years now. So I actually joined the company when Red Hat was pretty much in startup mode. We just had Red Hat Enterprise Linux. My background, I went to school for hardware for computer engineering, but at the time I learned software really well and that introduced me to open source software which drew me to red hat i loved that model and i started in i.t actually in red hat sort of applying software and worked about every role you could imagine in i.t and in the software groups um and moved up through the ranks to where now i get the privilege of getting to uh to help try to lead the uh the company so So I enjoyed every minute of it for the last almost 18 years here.

3:59Yeah, yeah, that's incredible. Just for listeners who aren't that familiar with Red Hat, can you take us back to Linus Torval? And was he involved in the creation of Red Hat or was that spun out of the foundation? Anyway, sort of how did Red Hat come to be and how does it relate to Linux? Yeah, so Red Hat at the time in the early 2000s, even a little before that, Linus and a lot of the software around there, there's this belief in new model of creating open source software and that turned into a Linux distribution of how do you collect all of these things, the GNU technologies, the Linux kernel and put it together into something usable.

4:49I remember at the time there were lots of different distribution choices there from VA Linux and Cabrera, Yellow Dog, all sorts of them. And Red Hat was one of those distributions. And the angle that Red Hat took was that they were going to serve enterprises with Red Hat Enterprise Linux. They weren't. Red Hat actually started selling boxes of Linux and Best Buys and Circuit City at the time. But they made that pivot to provide predictable life cycles and long term support for enterprises to be able to adopt Linux. And that's really where Red Hat got their start was that enterprise focus on it.

5:32And, you know, since then, the rest is sort of history. I think the model of open source and software development has really taken off Linux, I would say, is the operating system that sort of underpins a whole lot of modern enterprises. these days and and red hat's grown from you know 800 or so people when i started to over 20 000 worldwide we do linux you know products like open shift and containers automation sort of the full gamut of uh of software technology all based on an open source model right and and um you're still primarily it's it's like a service company wrapped around an open source operating system is that right so you you help take uh the linux operating system and implement it and and refine it or or deepen it or add features or whatever but it's it's built around the core open source uh model or system.

6:36Is that right? That's right. So all the software we produce is under open source licenses. So it's freely available to the world. What we do is we focus on the support, BraithFix support. We obviously know the software. We write a lot of it. And then also things like security updates when most enterprises can't stay on the latest versions of things. Open source moves so fast. And so we make sure that security bug fixes, those things are also applied to older versions so that even if you're running in some cases, Linux versions that are 10 years old, you can still be secure and on a stable platform.

7:17So we do the break fix support. And we like to describe it as a, it's innovation at a rate that enterprises can consume from open source the open source innovation is tremendous and we give companies all of those choices on that life cycle of how they want to consume it yeah uh just generally on on open source i remember when linux uh first came out or first sort of hit the public consciousness uh it was considered and you'll forgive me, kind of a poor cousin to commercial operating systems that were bundled with hardware largely. But over time, it's grown. And I think at one point, I had the impression that open source was sort of the future because you have this hive mind effect of all these people improving, improving, and how could a commercial enterprise hiring expensive engineers and developers keep up with that?

8:33And to some extent, that's borne out. So can you talk, first of all, about the promise of open source and how it stacks up against proprietary software and then we can talk about generative ai which is i said is what i'm interested in and how open source maybe breaks down uh in that arena because of the the uh hardware constraints and cost and all those things yeah so the uh i'll give you an example of open source from sort of my moment in the field. And I actually, before Red Hat, I was in consulting. And as a consultant, I was literally deploying Unix systems right next to Red Hat Enterprise Linux.

9:22It was actually Red Hat Enterprise Linux 2.1. It was the first version that Red Hat released. And the difference to me that really brought the promise home to me as a consultant was if I hit an issue with the Unix deployment, they were at the time much more polished. They worked probably more consistently, but I was paralyzed with any issue. I was dependent on a vendor at that point to put in a support ticket. I was in the queue and you're a consultant. Your whole job is to make progress and get deployments finished. On Linux with all of RHEL's rough edges at the time, it was totally transparent to me.

10:05I could go understand every line of code. I could debug anything myself. I could put in a support request to Red Hat, but I could actually work a lot of that issue myself. There was nothing hidden from me on it. And I fell in love with that model because I felt like this really lets me do my best work with it. And I think it wasn't just me. it was millions of people around the world fell in love with that model of just being able to not just be given something that they use like a black box but to be able to understand it and learn from it and iterate on it and and improve it it really is it's pretty captivating and so i think that that's why you've seen that model applied to so many different forms of software at this point.

10:54But yeah, that's sort of the promise. And that's why, you know, at Red Hat, we often say like passion runs deep at Red Hat. We have associates that have our logo tattooed on their body and we have changed our logo and they have both of them tattooed on them. So it's a different level of belief in sort of what we show up and do every day versus is just produce software it is you know we're trying to change the world for the better in the process right okay so um and we can talk about all the different kinds of open source software that have come along since on of and the successes and failures but what i'm interested in And is that model now there's, you know, Meta has open sourced the LAMA models.

11:48And, you know, I started talking when GPT-2 came out with, I'm sure you're familiar with them, Connor Leahy, who was at Luther AI, who they built the first open source large language model. And at the time, that seemed kind of scary that this decentralized group of hackers could build a large language model to rival OpenAI, or at least come, you know, they didn't have the money to get as large as OpenAI, but they could build the model. Now you have meta open sourcing models. And I was just reading Yanlokun's testimony at the tech hearing in Washington. I can't remember, frankly, which one it was.

12:50but he was talking about how open source will play a big role in the spread of large language models but i've since had conversations with people that argue yeah there will be small commoditized llms that people can deploy for specific use cases. But the future of generative AI remains with big companies and the proprietary model because they're so costly to build and deploy, and particularly everyone knows about the training. So something like an operating system, which I could put on my laptop, it's very different from a large language model where the cutting edge is going to be beyond the reach of the open source community because they can't marshal those kinds of resources.

14:05So, yeah, what are your thoughts on that whole debate, whether Yanlich Kuhn's right that open source is a future for generative AI or these proprietary proponents argue that the cost is beyond the reach of open source? Yeah, you know, I think I've certainly heard these arguments before. Yeah, sure. Take relational databases. I think it's a classic example of in the days I was coming up in software, it's like no company's ever going to be able to put in the R &D effort to build a relational database. It's too complex. The query optimizations of those things won't be achievable within this model.

14:51And yet, if you look today, I think there's always a use for both models in it. But I would argue that you take relational databases and there are different scaling patterns that have come up, not just vertical scaling, but horizontal scaling. Much of that gap has been closed with it. When you look at large language models today, I do think there's always an element of truth to build that first foundation model on it. It is an incredibly resource and time intensive process. We work pretty closely with a lot of different companies, but with IBM in this space. And it does take a sizable amount of resources to build those first models.

15:38What I think is pretty amazing that's happening right now, though, is the specialization of these models and the resources that it takes to refine them is a laptop operation. It's not a hundred or a thousand GPU operation. And areas like that is where we see the open source contribution and tuning and specialization and application exploding in it. Because there is a lot of IP that goes into that original model. But I would say I probably see as much innovation that goes into the refinement of a general model into specialized use cases. And that is pretty within reach of a really, really wide base of developers at this point.

16:32And then I would say, as technology is going to change a lot in the next five to 10 years from a hardware aspect of what does it take to build those foundation models, I do think it might be a today statement that that's out of reach for a lot of people on it, not the refinement aspect, but I don't know how many years that actually holds on it. Just putting more billions of parameters into the models hasn't necessarily shown to drive the same accuracy rate. So I do think we're going to learn a lot in the next couple of years of what drives the best outcomes from it. Is it in the refinement? Is it in the core model and when do you sort of hit diminishing returns in that but it is certainly moving fast i think in in both sides of it the original model creation and then what we see people doing with it afterwards yeah that's interesting and and and that actually uh not contradicts but challenges one of the arguments that's been made to me and i'm a journalist so uh this stuff gets very confusing you know is whoever has the the bigger officer the the most prestigious phd you know you tend to listen to them but uh the uh so a foundation model right takes i don't know tens of millions hundreds of millions to build and if it's open source you don't need to to build and train that model you download the weights but you still need infrastructure and and a lot of uh power to to put those weights on a model on uh in a in the cloud or something i mean i these larger models are not going to fit on your laptop yeah so uh i mean one of the arguments is that yeah open sourcing it allows people uh to to do that to take a model and and then fine-tune it but that's that's still the domain of companies that are extremely well resourced.

18:58It's because, to back up a little bit, one of the concerns when, you know, Eleuther AI open-sourced GPT-J, that first open-source large language model, and now Connelly, he has kind of switched sides and he's running an alignment startup and is very freaked out by the power of generative AI and how if it falls in the wrong hands and how open sourcing is a terrible idea because people can then take that model and fine-tune it or refine it to do terrible things.

19:45So just talk a little bit from your understanding, and I realize that you're in a different part of the business, But downloading the weights of Llama 2 and how realistic is it that a small company or a group of people could take that model and deploy it and then fine tune it? Yeah, well, I'll go through an example. um i know because i like playing around this but llama two to the side for a second look at something like the stable diffusion or for those might not be steeped in it image generation and manipulation in this it was not that long ago i mean we're talking probably a year where that lived in the land of big well-resourced companies that could do right hugging face which is an aggregate of all these refined models it is stunning to me to see how many people with their own custom data have been able to specialize that image generation into domains that they know whether it's animation focused areas or you see people that are you know specializing image generation on their animals or their pets so that they can create it with their dog and cat there is a tremendous amount of creativity that's being applied to these where i'm sure that the creators that train stable diffusion in these um they know art but they're not going to know anime as well as someone who lives in that domain and yet we're seeing creators that live in those domains that are able to download these sizable models.

21:47They're not at the same parameter counts that you're talking about, but then be able to apply unique data to them, get a pretty impressive refined result, and then share it back. And where Hugging Face comes in is someone else can pick up that refined model and refine it again. And it's been interesting to me. With code, I understand code. It's a very linear progression. I can get work. I can make an incremental change. I sort of know the outcome with it. with AI, it's not quite as linear to it. You really, the outcomes and refinements you might do might have really unexpected or pretty amazing results on it.

22:33So it is a very incremental process to get there. But that aspect, I think it's amazing how accurate it's gotten on it. And this is really, for most people, it's constrained by their laptop or their desktop on it. They're not going into Amazon and renting a bunch of NVIDIA GPUs to do that. But I think the same thing is we will get there on smaller, large language model parameter counts for it, where you get to a point where you can load it on modern hardware and you can refine it. And one of the reasons I remind people that I think this will happen is in our world. And we're in the world of, you know, we connect this modern hardware to the applications you run.

23:20So we're in the, it's called inference. Like when your model is trained, how do we run it efficiently? One of the most common places that we see these models that desire to run them is at what we call the edge. So you can think of the edge as in your car or in your factory. having a 20 billion parameter model that needs really sophisticated hardware to just to load into memory to run is not that practical in a modern day factory with it. And so there is a there's a boundary constraint, I think, that just will be put in place for the cost of inference. How much does it cost to ask that thing a question?

24:03And we're going to want to ask it a lot of questions in autonomous driving or lane correction or those things but there's a cost constraint there so i think we'll always see that pressure to get these really powerful models small enough to be able to run on edge-based hardware and the minute that happens it is usually within reach of hardware in a laptop form a desktop form that most developers can get so i do think there's a there's a self-correcting aspect they'll always be the chat gpt's where you can have a big model you can run it in the cloud with pretty impressive resources behind it but i think that's just going to be one vein where we'll see ai applied outside of the data center as much as inside of the data center yeah makes sense uh yeah and that that brings up a couple a couple of other questions so the this discussion i've had now with several people uh on open sourcing these large models uh one sort of view is that it's going to fall into commoditized models uh based on open source as you say hugging face uh and then very large foundation models that are going to eventually develop agency and be able to use tools, connect to other tools, and that the real value creation is in that higher form, That, yes, there will be commoditized smaller models and there will be a lot of economic benefit to that.

25:53But the real value creation is going to be at the frontier where these models that, again, are out of reach of open source developers because of the resource constraints are the ones that will be marching forward towards AGI or whether you believe in that. I mean, does that sound like a reasonable scenario to you? And maybe I'm an incrementalist on this, but I think there's always going to be the general purpose AI debate and risk. Like, at what point can it learn by itself and do all the things, and is that good or bad on it? And I read a good analogy. that's sort of like you're traversing a mountain range and the only thing you can see in front of you is the next mountain for it.

26:50And so it's very hard to tell how far ahead that is. But I think most people would agree, we're not there right now on it. We have the, my experience in it, it's not great at novel work. It is very good at reproducing and analyzing things that have been created not the best at creating something net new that's not there so we have some work to do there but if you look at like for me where will enterprises get the most value right now will it be in the race for a general ai knowledge or winner takes all or will it be in the application of pretty impressive specialized models, like whether we call them commoditized or not, it's sort of like the application of Linux.

27:43You could certainly say an operating system at this point is a commodity. You have a lot of different choices, but an operating system that can run on ARM hardware that enables you at the edge or run on the latest Intel gear to give you a better inference and capability or ties into the next generation GPUs for it, there is real value in that. And so for us, I think most of the enterprise value, which is going to change how we interact with things every day is probably for a while going to be applied at that specialized model where openly, I think open source is going to have much more influence and application there.

28:27We will have large resource efforts at government levels and large companies on general purpose AI and how does quantum affect that and all of these things in parallel. But I think there's a tremendous amount of value unlocked in terms of how can we do some of this rote work that we're constrained on really, really well. We know how to do it. And we have shown with large language models and refinement that the AI we have at our fingertips today is good, like impressively good at doing this work. But it can't be high level, I get it right 90 % of the time. That's specialization to get you to the 99, 99.9 accuracy.

29:15It's needed for most businesses in the industry who are, right? If you're in the medical profession, telecommunications, you need that precision and accuracy. And that's where I think we're going to see just a lot of, I mean, the thing I love with open source, it's inspiring to me to see how much creativity is applied into software. In the early days, it was the Linux from every corner on this planet. And it's not always from the people that you would expect of like the classically trained backgrounds. and i i feel like we're seeing the same thing i'll give you a good example a moment that i loved we are we have a big event each year rat hat summit and i was talking to our our director who coordinates all of the work at summit it's a large you know it's director like a movie director type role on it.

30:14And he was telling me how he has video of all of us and we have to do animated work every once in a while. And he was playing with a hugging face model and refining it with data of us presenting on stage to help drive like immediate animation view, vector graphics of us as he needed them. And Dave, in this case, he's not from engineering. He's from a creative background. And the fact that he could apply that and was empowered to do it, it's a pretty neat moment. I think that's where we're going to see a ton of value in parallel to some really big companies and you know it's like the moon race type things to get to larger more capable ai platforms on it but i don't think that's the only place that value will come from there yeah uh okay then then to to another uh part of the open source debate uh with regards to generative ai there's a lot of concern that just in the way that this fellow could take an open source model and train it with his data and do something interesting with it a malicious group of actors could take an open source model and refine it to i i was it Mustafa Silliman or somebody recently I saw described a test where they asked a model to create bio weapons or something.

32:06And overnight, it created some massive amount of super toxic molecules that then someone could synthesize. uh yeah what do you think of that argument that that open sourcing that sort of takes the guard rails off and everything that people are worried about is suddenly possible yeah i think um this is tough being in technology i think this is has for as long as technology has been around it's been that double-edged sword with technology if you go back to the early 2000s the debate was you know operating systems they may sound boring but like these are the things that run our hospitals our airlines our defense systems how on earth could you have every line of code visible to the world because all the bad actors could just find their exploits in it and then risk health and travel and national security and all the things on it was a very similar argument at the time.

33:24What played out, I think, was that that that visibility, the accessibility to understand the creativity and the contribution to it. Close those holes faster than proprietary models for because like bugs exist in software on it. But the open source models, there's the statement, you know uh with enough eyes all bugs are shallow i think linus might have said that that really started to play out in that model now the thing that becomes uncomfortable with that is you know i think as a nation or a lot of nations we get comfortable with controlling access to things like that's your security basis as you control who sees what who can get what But with open source, it becomes the application of technology more than the control of it.

34:21I think there are always risks with these, but it is just a different use of that model where you're not going to control on who sees the code because there can always be bad actors that see code that you don't want to. But I think we've shown that that model of everyone having access to it, being able to contribute produces more secure code with it. In the case of artificial intelligence, the challenge I think is different, but it's similar if you look at it in another lens, which is how do you control these models with it? How do you make sure that it won't answer questions on your example of creating bio weapons, but it will help you optimize your business for it.

35:13Just as much creativity will have to go into that as avoiding time or hardware loop, attack errors and operating systems on it. And I think we have to pick an approach. It can either be control access to it, where you only have a few people that can see how the model is created and we all hope they get it right. Or there are a lot of people that control how those models are created, how controls are put in place, and you're tapping into the creativity of many with the downside being if everybody has access to it, the controlling the thing, the access to software isn't your control point anymore.

36:02And I don't think one system is right or wrong, we certainly have seen a better outcome using open source models with software on it. AI, it is still so early right now, it is slightly different in application, but any technology, I think we've seen people do incredibly good things with it. And I think you will always have bad actors that will try to use it for other purposes or find ways around it. And we just have to find the model that amplifies the good while constraining the bad. I don't think it's, unfortunately, I don't think it's that new with AI. I think this has been going on for a long time.

36:42No, that's right. The one thing, and maybe I don't understand how open source works, but if you open source a model and someone downloads the weights and the code base and sets it up on their own servers, it's no longer what they do with that code is no longer visible to anyone if they don't want it to be is that right yeah i think it'd be the question of um if you download an operating system you can always take out code but it only affects you and i think with ai it's you download a model and weights but can refine your way out of those constraints, can you use it to apply to other things? It's new.

37:36There's certainly a risk of that on it because you're using that to apply to other things. But at the same point, if you look at hackers in the world today, they have their own scripts and kits that aggregate all the known issues that run and apply those to it. Does what I speed that up? Absolutely. But it wouldn't be like it doesn't exist today. What I would argue is as long as we can use AI to help defend against it more so than you're speeding it up, you probably end up in a better spot. But but it is with any technology that amplifies how fast innovation works and plays out. You know, I think there's always that double edged sword.

38:24I saw it in the in the early 2000s with the onset of things like Linux. I think we're see we saw it in our space with the application explosion of Linux containers, how portable that made software in all of these different environments. I think we're seeing that again with AI. I have luckily in the past, I think it's been on the more good than bad side. I'm an optimist. So I think AI with the right structure and creation to it can have a lot more good than bad. But you always have to think through the bad and the malicious uses in technology. Yeah. The other thing on open source, you know, the open source community, of course, is open.

39:11and researchers and academics are by nature collegial. But in a national security setting or geopolitical context, it may matter which country has the strongest models or, you know, what they do with models. And by open sourcing these large models, very large models, and certainly not on the scale of GPT-4, but, you know, every government in the world can now build a pretty powerful LLM and then take it wherever they want it to go. there was a rumor that that ernie bought you know baidu's uh they have a 70 billion parameter and i think 130 billion parameter version uh that that was built that they started with llama uh that you know they didn't have to build this from scratch is that a concern at all or uh yeah You know, I think at any point we sort of rely on who can access technologies through some regulatory aspects, export control or those areas.