#137 Raul Martynek: How AI Will Change Data Centers Forever

0:00What a data center ultimately does, it underpins the ability for companies, organizations, for society as a whole to deploy technology. and all that technology has to go into a data center, which is why the sector, you know, for the last 15 years has had, you know, call it double digit growth. And this is a real estate business at the end of the day, right? So that's a pretty attractive dynamic for a real estate type of business. If you think about Google as a search engine, right? Because that's really what it is, right? It's an advertising platform that delivers search results, generate billions of dollars of revenue, right?

0:36It absolutely used AI to generate those results to figure out what to present to people and all that stuff, right? Hi, I'm Craig Smith, and this is Eye on AI. A lot has been written about the enormous and increasing costs of large language models. But one aspect to that story that's gone unreported is the impact it's having on data centers, the physical server farms where all of the computation takes place. This week, I speak to Raoul Martinek, the CEO of Databank, one of the industry's leading data center providers about the intricacies of data storage and the challenges of GPU-intensive workloads for AI.

1:23Raoul's perspective illuminates the foundational layers of our digital world. Join us as we delve deep into the world of data centers and their pivotal role in shaping the future. But before we begin, let me mention our sponsor, NetSuite by Oracle. If you're a business owner, having a single source of truth is critical to running your operations. So remember these three numbers. 36 ,025 ,1. 36 ,000 because that's the number of businesses that have upgraded to NetSuite by Oracle. NetSuite is the number one cloud financial system for streamlining, accounting, financial management, inventory, HR, and more.

2:13The number 25, because NetSuite turns 25 this year, that's 25 years of helping businesses do more with less, close their books in days, not weeks, and drive down costs. One, because your business is one of a kind. So you get a customized solution for all of your KPIs in one efficient system with one source of truth. You manage risk, get the most reliable forecasts, and improve margins. Everything you need all in one place. I'm not the most organized person in the world, and there's real power to having all the information you need in one place to make better decisions. This is an unprecedented offer by NetSuite to make that possible right now.

3:06Download NetSuite's popular KPI checklist designed to give you consistently excellent performance. Download it for free at netsuite.com. That's E-Y-E-O-N-A-I. All run together. Go to NetSuite.com slash IonAI to get your KPI checklist. That's NetSuite.com slash IonAI, E-Y-E-O-N-A-I. All run together. Again, that's NetSuite.com slash IonAI. They support us, so let's support them. It's great to be here, Craig. Thanks for inviting me. So, Raul Lortnick, I'm the CEO of Data Bank. I've been in the internet infrastructure space for over 27 years, kind of since the advent of the commercial internet, you know, working with sponsors to build kind of the foundational businesses that underpin the modern internet.

4:11When you think about your cell phone or your desktop, and you think about the physical infrastructure that's required to move those bits to you, it typically starts with, on the wireless side, some type of cell tower or a small cell, as we call it, all hits of fiber optic cables, and then ultimately everything comes back to a data center. So those kind of three links in the chain are what we consider digital infrastructure. And DataBank is a U.S.-based data center operator and developer. We own and operate about 65 data centers in 26 U.S. markets, giving us the largest geographic footprint of any data center operator in the U.S., public or private.

4:54We're backed by a whole group of investors, including DigitalBridge, which is a leading infrastructure manager in this digital infrastructure space. Yeah. And a lot of people with the migration to the cloud and all of the talk about the cloud, I think a lot of people have forgotten that there are independent data centers out there. I mean, the cloud is a network of data centers itself, but that there are data centers outside of the big cloud providers that are not necessarily cloud data centers. Yeah. Yeah. Can you talk about how? And then, of course, there's on premise data centers that that people use.

5:46They are very concerned about their data security and that sort of thing. But can you talk about how the existing independent data center market has evolved as the cloud has taken over and then how that plays into what's happening with AI and generative AI in particular? So, look, just a quick little rewind, right? So, if you think about it, about 15, 20 years ago, you know, you'd walk into any office in America and there'd be a data center. It was called the computer room, right? And what the, you know, the internet came along and really kind of made applications portable and data portable, so to speak, right?

6:28So, that's what kind of created this kind of need for, as we call them, a multi-tenant or carrier-neutral data center, right? You know, so in the mid 2000s is what people's mentality really switched in enterprise that said to themselves, hey, you know what? We don't want to build and operate our own data centers. It's like an individual building and operating their own house. So we're going to outsource that physical aspect to a third party data center like a data bank, right? And that's where you look at some of the early companies in the space, Equinix, Digital Realty, which are still the two public guys, that's kind of how they started, right?

7:02By meeting the requirements of enterprises that wanted to outsource their data center requirements. In the early 2010, you had the rise of public cloud, right? Obviously, at that time, there was a lot of different companies competing for that, right? It was looked at as an alternative to on-prem or to third-party co-location. You have companies like Rackspace, and if you remember Softlayer and Joian and GoGrid. I was working for a cloud company called Voxel at the time. And ultimately, we were still kind of trying to tell customers, hey, this is a better way to manage your IT infrastructure. Well, you know, the industry kept growing because what happened is enterprises just kept outsourcing.

7:43And there was so much kind of on-prem data center demand that it just kind of propelled the sector forward. I'll tell you, the mid-2010s, 2012, 13, 14, you know, people would ask people like us, hey, how's the cloud going to impact your business, right? Because it was perceived this way where it's either you're doing cloud or you're doing on-prem or third-party co-location. And fast forward to today, what's actually happened is that those are different things. And ultimately, the cloud has been a huge accelerator of multi-tenant third-party data centers, right? And it's happened in a number of ways.

8:22Number one, enterprises have gotten more savvy about where they want to put their workloads. And it turns out that the cloud is really good for certain things, but for other types of workloads, especially what we call persistent workloads, it's not that great from a cost perspective. You don't get any of the benefit of that ability to burst up and burst down, right? So companies have become much more savvy about where they want to put their workloads. And that's this term that we use in IT. It's called a hybrid approach, right? Do some public cloud, do some private cloud, do some colo. But more importantly, what happened is that the public cloud companies themselves grew so much and so quickly that they couldn't even satisfy their own demand.

9:04And today, if you look at the big four, Microsoft, Google, Amazon, and Oracle, they probably outsource, because no one knows the exact number, but they outsource about 50 % of their data center capacity and build the other 50%. They just can't be everywhere. And even their balance sheets can't support the tens of billions of dollars required to build out all this infrastructure. And as importantly as that, you know, when everyone talks about the cloud, we like to think about it with those four players, right? Like it's some monolithic or oligopoly, right? The reality is, as I like to say, there's this long tail of cloud companies.

9:43And if you think about businesses like NVIDIA or Akamai or Pure Storage or PayPal, any company that is a technology company, a lot of these companies have said to themselves, we're going to build our technology inside co-located facility, and we're not going to build our own facility. So the reality is, is that what a data center ultimately does, it underpins the ability for companies, organizations, for society as a whole to deploy technology. And all that technology has to go into a data center, which is why the sector, you know, for the last 15 years has had, you know, call it double digit growth.

10:23And this is a real estate business at the end of the day, right? So that's a pretty attractive dynamic for a real estate type of business. And what's happened most recently is, you know, AI has appeared, right? That is now the latest kind of technology that is highly, highly dependent on data center capacity. And what we've seen over the last six months is just really an unprecedented amount of demand to deploy AI in these data centers. And you've taken an industry that already had some pretty good tailwinds as a function of public cloud growth, as a function of enterprise outsourcing, as a function of increased technology adoption across the board, now has this now, this new vector of growth, which is AI adoption.

11:10And for people like us, it's obviously a really exciting thing. Yeah. And the AI and the cloud, they both kind of, from my understanding, and grew up together, it's kind of chicken and egg. But what I've heard, I heard this from Michael Jordan, who's a very famous computer scientist, not the basketball player, who is an Amazon scholar, so he has some insight into Amazon. I also heard it from Ilya Sitzker, who was one of the researchers on the original Transformer algorithm paper, The way Jordan put it is that Amazon was building these data centers to carry the computational load of their logistics calculations, which were AI-based.

12:06And they eventually realized, hey, a lot of other people need this. We can make it available publicly. And that was AWS. And the way Ilya said it to me is that the cloud existed, but there was really no demand for it until people started doing AI. And then there was a demand for massive parallel computing. So in the early days, and that was GPU-based. So in the early days, it was all CPU, I imagine, or mostly CPU. Can you talk about the economics of the GPU-based data centers to handle these AI loads and particularly AI? And I love the comment a little bit about what you just said there about the AI in the public cloud, right?

13:08If you think about it, yeah, I mean, you know, some people that refer to the age that we were in as kind of the internet AI, right? If you think about Google as a search engine, right? Because that's really what it is, right? It's an advertising platform that delivers search results, generate billions of dollars of revenue, right? It absolutely uses AI to generate those results to figure out what to present to people and all that stuff, right? You think about Meta, another big platform, right? They used AI, obviously, with their social algorithms, right, into a recommendation. Amazon's a little bit different, right?

13:42Amazon, at the end of the day, Amazon was a computing platform, right? We call it IaaS, Internet Infrastructure as a Service, right? Amazon's proposition to enterprises is don't go buy your own servers and your own storage. Just rent it from us because we can do it better, cheaper, faster. They really didn't need AI for that. what they really need was just great software engineers who knew how to deliver this physical infrastructure like it was software, right? And then you had Oracle, which obviously with their database is a little different, right? But I think what, you know, so they were building out all their capacity to basically meet those business needs.

14:21And along comes ChatGPT, right? Which is a new form of AI, right? I would argue it's kind of like, it's a general purpose AI. Maybe it's, So we're entering the business AI phase where that is a chat box, obviously, that's built on a large language model that gives out these early human-like responses. And now people are envisioning all kinds of instances to use that technology. For example, Bloomberg just recently announced that they're building a large language model called Bloomberg GPT, which is not all Bloomberg's financial data and financial insight, to be able to output, I assume, responses around financial decisions.

15:05How do you balance your portfolio and what's the risk reward and things like that? So what we think is happening in the space today, and DataBank, you know, we have kind of a unique perch that we sit in, right? Because we operate these 65 data centers with 3 ,000 customers, and we get to see what equipment and what applications they're bringing into our data centers. I mean, they own all that equipment, but we get to work with them on that, right? So what we're seeing today, to your question, is that people are starting to say, okay, I want to deploy some type of large language model or generative AI in our data center, and that's going to require a new type of processing.

15:45It's no longer going to be CPU-based. It's going to be based on GPUs. And obviously, we all know from reading the press about NVIDIA and how they are now in kind of the crux of this massive movement because their platform is now the platform of, you know, the first platform that people want to use to develop these large language models. So those are new types of workloads. I like to say they're net new workload. It's something that has nothing to do with how the world operates today. Because again, every data center in the world today is, you know, somewhere between 70 and 100 % utilized. And think about that.

16:25It's running the world today. So these are all new things that are coming into the data center because there's a new application that is, there's a useful. Yeah. And those GPUs are much, much more expensive than CPUs. So how does that impact the, ultimately the cost of inference that, that end user is paying? I mean, All of these costs are built into that end cost. Is that right? That's correct. That's correct. Yeah, there's a couple layers of cost that are pretty significant, right? As you just pointed out, the boxes themselves are just more expensive. The networking is more expensive. This doesn't use Ethernet.

17:10This uses InfiniBand, which is a very high-end networking technology that makes these clusters look like they're on the same wafer, the same suitor wafer, right? All that stuff is a lot more expensive. A rack of GPUs is a lot more expensive than a rack of CPUs. In addition to that, these clusters consume an inordinate amount of power. A CPU rack might consume 5, 10, 15 kilowatts of power per rack. We're seeing GPU racks consume 10, 20, 50, up to 100 kilowatts per rack. So you're talking about maybe 10 times more power consumption just in that same data center space, right? Ultimately, that all outputs to, you know, these queries are very intensive, right?

17:58If you think about like how much data this model is holding in RAM and its ability to take a prompt and then spit out a highly intelligent answer, there's a lot of computations, you know, happening to just output that, you know, that simple sentence that is a prompt, right? So multiply that times millions and millions or tens of millions of prompts, and you've got a significant amount of new data center capacity that will need to be deployed if businesses, you know, want to, you know, adopt this technology. And all indications are that, you know, people are doing this, right? We know Microsoft, because of their public announcements, have, you know, identified a number of products where they're going to read ChatGPT into their product set, one of them being a product called Copilot, which allows Microsoft Office 365 users to basically have ChatGPT help them write Word documents and PowerPoints and Excel files and things like that.

19:02Again, that's going to require, in our estimation, hundreds of megawatts of incremental data center capacity, and that's just one application. And that pressure, that demand, presumably, I haven't tracked GPU prices, but presumably that puts up upward pressure on GPU prices. And so maybe you can talk about that, But also, is there enough power generation capacity to handle this growth, let's say, in the United States? Well, you know, there's probably enough available power. The question is, where is it available, right? Because, you know, data centers are, as you said, they're a collection, right?

19:53And they need to be located, in many cases, in proximity to other data centers. So, well, look, our view is that, you know, in the short term, the data center development timeline, we measure things in years, right? Because we're building these very, you know, technologically advanced buildings. The parts that go into them, what we call the MEP, the lead times on that are 50 to 60 to 70 weeks, right? So you just don't pop up a data center like you pop up a bar or something that you might be able to construct by doing the whole depot and putting stuff together. So our cycle is measured in years and you have this demand that has hit like a tsunami over the last six months.

20:39And we believe that, no, there isn't enough, there isn't going to be enough data center capacity next year or the year after to meet all the demand that we think is going to come from businesses wanting to adopt this generative AI technology and incorporating it into their business. So, yes, the GPUs, you know, NVIDIA is doing a really, really good job of trying to prevent a secondary market in GPUs and creating like a stub hub for GPUs where, you know, they go to the highest bidders because obviously that doesn't help, you know, them maintain a good supply chain. and once you got GPUs though, you got to put them in a data center, right?

21:26These are not boxes that can sit in the house, can sit in an office, can sit in a warehouse. They need to go into a data center. They need to be powered. They need to be cooled. They need to be secured because they're very expensive and they need to be connected up to the global internet, right? So yeah, I think, Craig, that's going to be a really interesting dynamic to see what happens over the next six to 12 to 18 months when if this demand continues at the pace that we're seeing, but data center development just can't accelerate off a dime because it's more of a multi-year type of timeline.

22:01Yeah. Are there other GPU producers that are filling in, backfilling where NVIDIA maybe can't meet demand? And is there any question that NVIDIA is going to fall short of demand? them yeah uh two good questions i mean yeah there are look there are other players out there right there's uh obviously uh hp is uh is a big player they they bought cray supercomputing if you remember them so they have uh and they recently announced i think was last month they announced their kind of own internal uh machine learning uh type of offering right based on uh some chips from intel and some chips from the Cray that were more specialized.

22:47There's specialized companies like Graphcore. There's a company called Cerebris that's built. The largest chip that has yet been built and they claim is, you know, from certain types of applications is more cost-effective than GPUs. I mean, look, I think it's like any new technological wave, right? When the internet came out, right, it was like there were, I think there was Cisco routers and that was it, right? I mean, it was our wealth fleet, I think, at the time, if I'm not mistaken. So, I mean, so what did all that demand do? It brought in more entrance, right? And it brought in more supply.

23:23Obviously, we're talking about, you know, highly, highly advanced chips that, again, not something that can be started on the dime. But look, I think there's probably enough of an ecosystem where these other players are going to benefit, right, from this demand because probably NVIDIA isn't going to be able to deliver enough GPUs to satisfy the demand that is going to happen so that people are going to turn somewhere rather than not do these projects because, you know, this is considered obviously a technology that can create massive competitive differentiation. Yeah. And this is a bit of a dogleg on the conversation, but I'm interested in China.

24:03I've spent a lot of time in China and China is scrambling because of the sanctions and certain entities from buying GPUs or advanced semiconductors. How can China compete? How can it keep up with data center demand if it doesn't have access to NVIDIA GPUs? That's an excellent question. And in fact, there was a book written called AI Superpowers, which kind of talked about AI war between US and China. And I think the previous administration and the current administration's efforts to kind of red fence China away from advanced chips, number one. And then as you know, even advanced manufacturers like the AMSL in Amsterdam, right?

25:02So, I mean, they're going to, you know, so obviously what is China's reaction is to try to accelerate their own super, their chip semiconductor industry. They have, you know, programs where they're investing tens of billions of dollars to try to do that. But I, listen, right now, I would say the U.S. is in the Western powers are in a really good position that they want to press their advantage, right? Because there really aren't any substitutes for the most advanced chips, which are really the ones that you kind of need to drive, you know, these types of technologies. Yeah. On the cost side of you're filling new data centers with GPUs.

25:50Has the price changed? I mean, and are you passing price increases on? Yeah, yeah, pricing is going up. I mean, look, you already had, you had the beginnings of that last year because the data center space, because as I mentioned, we're basically technical real estate. Our pricing is based, broadly speaking, on two major things. One is our cost of capital, right? Number two is construction costs. And last year, what we saw is because of inflation, because of interest rates, because of supply chain, not only cost of capital went up, we used to be able to borrow at two and a half cents using securitized debt.

26:29Now we got to do it at 7%. So the cost of capital's gone up and the cost of construction has gone up. So last year, you already started seeing an increase in costs that were being driven by inputs. But now you're seeing even more dramatic increase in costs being driven by demand, right? That folks are, again, there's just so much demand. We have been pricing deals at 30 % to 35 % more in Q2 than we did in Q4 of last year, so six months ago. And we're getting 30 % to 35 % more per kilowatt because of the input cost here because of the land. Wow. Yeah. And just to clarify, you're not buying GPUs.

27:18You're providing the rock space for GPUs. Yeah. Or do you also provide GPUs? We do not. We do not. No, we're agnostic to, you know, what technology people deploy. That's one of the beauties that we like about our business model. Right. Where do you think the power consumption is headed and there's a lot of concern. I mean, there's certainly a lot of research being done on making these systems more efficient, making training more efficient. But ultimately, there is a concern that as this works its way into the economy, I'm talking about generative AI, that the percentage of national or global power output that's consumed by these data centers is going to grow.

28:17Have you guys looked at that analysis, or are we very far away from that being a real issue? Yeah, no, I think that is going to be an issue down the road. And look, this is good. We need to have a dialogue in this country about how we're going to face that, right? Because we're obviously not consuming power because we're just deciding to run innate things. We're doing it because other consumers say, hey, we want this capacity, right? So I think there's a lot of questions about AI. I mean, look, the US power grid, as you know, is going through just like power grids all over the world through a dramatic transformation around fossil fuel to renewable.

28:57I'm based here in the state of Texas, and the ERCOT power system, which is the Texas power system, has been extremely progressive at delivering new generation. A lot of that, people might be amazed that Texas is the number one state in the country for solar and renewable, because there's wide swaths of land in Texas, and then the business climate is very, you know, promoting those types of investments, right? So, you know, there are, and, you know, the main utility here in Texas, Encore, is part of that. So there are utilities that are very progressive about, okay, working with data center operators to, you know, figure out how to bring more power to locations that make sense from a data center development perspective.

29:47I mean, what I think is happening, you saw it in Ashburn, in the Washington, D.C. area last year, where Dominion Power ended up in a shortage, right? And basically, I had to tell data center developers, we're not going to be able to deliver the power that we told you we thought we could. And what that has done is it's spawned, it's pushed that demand into other markets, like Dallas, like Atlanta, like Chicago, like Phoenix, like Reno, right? So it's moved them into other markets around the country. So there's going to be a lot. I think what we'll see is that, you know, there's going to be new, as we call them, data center clusters that appear.