Yotta CEO Sunil Gupta on Supercharging India’s Fast-Growing AI Market - Ep. 225

0:10Hello, and welcome to the NVIDIA AI Podcast. I'm your host, Noah Kravitz. We're recording at GTC24 in San Jose, California, and we're here to talk about the fast-growing AI market in India. My guest is Sunil Gupta. Sunil is the co-founder, managing director, and CEO of Yada Data Services. Yada is the first Indian cloud service provider member of the NVIDIA Partner Network Program, and the company Shakti Cloud Offering is India's fastest AI supercomputing infrastructure, featuring 16 exaflops of AI compute capacity supported by more than 16 ,000 NVIDIA H100 GPUs. Sunil has been a busy man this week, but he's taken some time to stop by the podcast studio here at GTC and to tell us all about Yada and its role in India's fast-growing AI sector.

1:01So let's get right to it. Sunil Gupta, thank you so much for stopping by. Welcome to the NVIDIA AI podcast. Thank you for having me. Thank you, No. So let's start with the basics. For listeners who maybe just heard of Yotta for the first time this week, you've been in the news, a lot to congratulate you on. What is Yotta? What does the company do? Great. So Yotta is a managed data center and cloud service provider operating in India for the last five years. We have been running our own, you know, self-designed and engineered and constructed data centers. We, that way, offer, you know, traditional data center, co-location, managed hosting, cloud, you know, and a variety of managed services there.

1:38We have been offering, you know, GPU services for the last four years ever since we started, but those were the smaller GPUs, the A40s, B40s, and T4s, and the use cases used to be, you know, creating content and you know game creations and things like that and possibly those were the credentials that I have so much of large data center campuses and I have got you know the experience of handling GPUs and delivering it to enterprise customers for different use cases and I you know from an India scale point of view if I'm having 700 GPUs today in my data center that still possibly is the largest deployment of GPUs in India and those were the credentials which possibly attracted the eye of NVIDIA India and they suggested to Jensen that possibly these got the right guys having data centers and have power and the right skill sets and exposure to GPUs that, you know, and they are there, as I said in one of my interviews just recently, that I'm hungry and ambitious and I want to take a plunge of, you know, in this market which possibly can become the largest or one of the largest market in the world.

2:35And it can also become a garage, you know, as a service provider to the rest of the world for developing AI models. So we took the jump, you know, and today, you know, We have got our first set of deliveries already. And possibly by end of March, I'll have all the deliveries completed. And by around 15th May, we are targeting to go live, giving customers our GPU-based services. Fantastic. 15th of May, you said? Yeah. Is that Shakti Cloud to go live? Yes, that's Shakti Cloud, right. So can you tell us about Shakti Cloud? Yes. So first, yeah. So we are running data centers and other cloud and managed services for like four years.

3:11We have been giving GPUs for four years. But yes, Shakti Cloud is something which is essentially a GPU-based cloud. It has got about 16 ,384, to be precise, H100 GPUs. It has got a couple of thousands of L40s GPUs, which are mainly for inferencing purposes. We have put up this entire GPU cloud on NVIDIA's reference architecture to the T. We are not deviating from that even for any single element of that. It is not deviating from that. So essentially there's an infini band layer, which is actually in a, you know, leaf and spine architecture. And so we are creating pods of 2000 GPUs, you know, 256 nodes putting, put together, create one pod.

3:54And there's a core layer on the top of that, which essentially means that I can connect eight such pods of 2000 GPUs into one super pod of 16 ,800 GPUs. GPUs. And that essentially means that on one hand, I can give to a small startup, you know, a single GPU or a single node or even a partial element of the GPU. That is one end of the capabilities for some use cases. But on the other hand, if some large scale customer comes and he says, my model requires 16 ,000 H100s working parallelly to train a very large language model, we can give her even that as well. So that is essentially one element in terms of the underlying processing capability.

4:35Then, you know, you need as much high-speed special type of storage. You know, the data need to be trained on the GPU. So it has to be on a high-speed storage. So that is what we have put in from Weka. Then what we have done, we have put a complete NVIDIA AI enterprise stack, software stack, on the top of this. We have, because we have been working to develop our own sovereign cloud in India for last two years. So that experience came in very handy. A lot of that coding, actually, we sort of repurposed to develop our own orchestration layer, our own, you know, self-service portal. And so today my self-service portal has got entire NVIDIA AI software stack, you know, bundled into that.

5:16I'm putting a whole lot of open source software libraries. I'm putting in a capability for people to invoke the, you know, the models from, let's say, hugging face and then bring it into my orchestration layer. And then people to bring their own data, you know, annotate the data, clean the data, maybe create more synthetic data do all that stuff it's essentially MLOps operations and then use an existing model in my marketplace which is there in the portal to fine tune their own model and once they have fine tuned their model then they can put that model directly for an influencing purpose in my marketplace or they can take it to wherever they want to take it or they can build that model through APIs into one of their enterprise applications so essentially what I have tried to do in Shakti Cloud is that on one side you are creating a very I would say a reasonably large size underlying infrastructure layer of compute and storage and network.

6:07And on the top of that, you have tried to create a complete, I would say, you know, self-contained orchestration layer where users can come online. They can consume GPUs from one GPU to a partial GPU, thousands of GPUs. They can decide to create, you know, their own clusters. So I give a capability to them to make online Kubernetes clusters or they can go for slum as a cluster. And once they have done that, And then, essentially, they bring in the data and train the models. Either they make a model ground up or they use one of the pre-existing models in the marketplace and then fine-tune that with their own data.

6:41So that is essentially what we're trying to do. Amazing. Can you tell us a little bit, and for the audience, for myself, what the digital economy, what the AI scene is like in India? and what, you know, you talked about building a sovereign cloud. And clearly, you know, Shakti Cloud is offering, as you said, services ranging from a partial GPU all the way up. What does it mean to the Indian economy, to the tech sector in India? You know, what is this presence going to do? Well, see, as you know, I just delivered a speech in the conference today. And the topic of my speech was that it is AI from India, for India, and also for the world.

7:22And for the world, right. So essentially there were two elements to my speech. One was that India itself is potentially going to be the world's first or the second or third largest market. And there are some reasons why I'm saying so. And second part of this was that just like India has dominated the world scene as the garage for delivering IT software and services for the last three decades, there's no reason to believe that India cannot be also an AI garage for the world. What India has already If I just give you some dynamics In the first part Why India has the potential To become one of the largest market in the world See, you just see the scale of digital adoption Today, India has got 900 million internet users Possibly more than the population of some continents Out of this 900 million 600 million are smartphone users Essentially the users who are consuming videos Who are consuming images Who are generating videos Generating images And actually pushing it out India today is far ahead in terms of adopting digital payments, you are doing 100 billion payment transactions per month, which is like 10 times bigger than any other economies in the world.

8:27India has got more than 5 million trained IT professionals. This count keeps on increasing and out of which I think 420 ,000 plus are actually AI trained professionals. This is the skillset side and the economic side. By the way, Indian economy is growing, which is the bright spot in the world scene. It's 7.5 % CAGR and as per all the various reports for the next few years. because the demographic dividend which India is enjoying, the population which is earning is more than the population which is dependent. India is expected to grow with the same speed for the next couple of years. So when you combine all these factors together and also couple that with the factor that on the infrastructure side, where typically India was seen as maybe a laggard economy, in the last couple of years, India is building infrastructure like anything in terms of digital infrastructure, not only the network and the fiber going to the last mile and the 5G going to the last mile.

9:13Indian data center scene which was just about 200 megawatt you know just by 2015 today India is boasting of around 1 gig of data center capacity which is ready and into production and there's another about 700 megawatt which is going to be going live in the next couple of months and this is something which is growing at a 40 % CHG and because all these data centers came up just in the last 7-8 years because of the hyperscalers coming and putting up the shops in India so you can just imagine that these data centers have been built as per the specification of the hyperscalers. So these are world-class, latest generation data centers as good as anywhere else in the world.

9:50And, you know, with some retrofitting and extra engineering, many of these data centers can be customized to handle the GPU load. So if you see, combine the skill sets, growing economy, the digital adoption, and also that India also is now building up on its infrastructure. And I now couple that with that with Shakti Cloud getting launched, India will also have a very large supercomputer for the purpose of training AI models and putting them for inferencing. There's no reason why India will not, first of all, become a very, very large AI market itself. We can become a consumer of AI, but it will also become a very, very big service provider for the rest of the world.

10:34You don't have to consume AI from India. One of the things I noticed in reading about Shakti Cloud and listening to some of the media coverage was the importance of self-service. Can you talk a little bit about that, about being able to offer a complete end-to-end high performance and AI-tuned platform, especially considering the diversity of customers that you're going to be serving, you know, startups, academics, researchers, and obviously industry at scale? Yes. So this is one of the things which we focused on big time, and I can see, you know, possibly we are following NVIDIA's footsteps. So if you see Jensen's keynote also two days back, you know, while he was announcing the launch of GB200, which was the hardware part, I can say, but 70 % of his talk actually was focusing on software.

11:18And as much capabilities you are giving to the end users, which is making their life easy to develop their AI model and put them for influencing for different industrial use cases, and as much you make it easy for them to do this, and that is where the software comes in. So, as I said, we have been working to develop our own sovereign cloud in India for the last two years. This is something which is in need of are for India. We actually use that capability to also very, very quickly, possibly just in about three months, we actually developed our own self-service portal, an orchestration layer, completely homegrown, which essentially means that one end users can come in.

11:56It can be a small startup or it can be a very, very large organization wanting thousands of GPUs. For everybody, you can just come online, create your own account, authenticate yourself. If you're end user, you can authenticate using MyAD. If you are a large enterprise, We can plug in your AD and you can authenticate with that. And once you have done that, starting from, for example, you want to have a bare metal node. So whether you want to have a single bare metal node or whether you want to have thousands of nodes, you can subscribe it online. You know, it will show you a dropdown. You'll show you the types of GPUs available and you can just subscribe it.

12:29You want to put operating system on there. So we have got images of various operating systems on that. Now you want to create a Kubernetes cluster on that. We want to create a slum cluster on that. We are giving those capabilities. is you select your master nodes, how much work you want, and then, you know, the cluster gets made right then and there, and then you can monitor the various health parameters of your clusters. The other part is serverless, for example, that especially during inferencing, how I'm seeing the requirement coming in from our customers that during training time, possibly they need, you know, maybe a couple of hundreds, a couple of thousands of GPUs dedicated for themselves to train their model.

13:01But once they have done that over a, let's say, period of 60 days or 90 days or whatever the time it takes, depending on the size of their model, after that they're putting the model for inferencing and inferencing traffic is like any other application put on the internet where users are coming and consuming your model of the application. Now that traffic is going to be very very bursty. There will be peaks where there's a huge user traffic and there'll be troughs where basically the traffic is not there. So essentially the users are saying that instead of we paying for a dedicated committed capacity of GPUs you rather give it to your GPU on demand essentially what we call a serverless function you know where multiple users are actually vying for the same GPU capacity and everybody gets satisfied because there's enough capacity and not everybody is peaking at the same time.

13:42So that's for serverless functionality also we have put in on the fly you can develop containers and put the same GPU capacity for inferencing function as well. Then what we did is that you know while there'll be certain people who would like to make grounds up large language models, they don't need any of my software other than access to GPUs and clustering but majority of the people possibly will be coming to consume one of the foundational models which is available in my marketplace. So we have a big marketplace in the organization layer and they would like to bring in their own data, possibly to clean the data, annotate the data, maybe create some more synthetic data and then they will put that data in one of the pre-existing models and actually create their own fine-tuned model.

14:20So that I think will be a much, much bigger market going forward. Most of the enterprises may not be creating their own foundational models. They would rather be using some of the foundational models, but putting in their own company specific or industry specific data and create their own, you know, use case specific model. So that is the whole environment we have created in this. So there are a whole lot of pre-trained models available, whether it is something which is a part of the NVIDIA AI stack, which I'm integrating into my this, or whether you invoke some of the pre-trained models from HuggingFace or any of the public platforms.

14:54In fact, many of my startup customers who are creating their own foundational models. By default, they also are going to put those models also in my marketplace so that their end customers can come to marketplace and start consuming their, you know, their models to build their own fine-tuned models. So, if you see this entire, entire capability of this end-to-one software, there's no end to that. I mean, I myself don't know what all we'll end up doing in the next one year because every day is a new discovery as to what all is possible. Right. What essentially we are trying to do is that, yes, right now, majority of the GPU usage seems to be the domain of the tech companies, the startups who are actually trying to train the models, either a foundational horizontal model, a language model, or an industry-specific LLM.

15:39But gradually, I will see if the market will start moving more and more towards fine-tuned models, especially to the needs of a company or an enterprise or an industry. And that will be put for inferencing. So how we are able to meet the demand of this wider audience, the enterprises and the wider startups instead of just focusing on some of the bulk customer upfront. And that is where the importance of the software that comes in. Multiple people coming in without any manual intervention of me trying to do something for them, which delays the whole thing. They just come and start consuming whatever they want.

16:09Does your approach to security and reliability change going from data centers sort of of the past, if you will, to rolling out this large powerful GPU cloud that's serving so many diverse, you know, types of users and use cases. Are there specific things for kind of the AI age that you have to take into account when it comes to security? Mostly not. I mean, and again, I can keep on saying that we are also ourselves into a learning phase and we are every day discovering what are the new dynamics coming in. But as far as my understanding as in today's concern, there's not a major difference. One is that, yes, you know that when you are training the model or even when you're putting the model for fine tuning or or for inferencing you know there was a specific concern of the customers that when i'm putting my data you know it is going to be sometime a very very secretive you know very very sensitive ip data or sensitive data for a company they'll they want to be doubly sure that even the operators that is ourselves are not able to have access to the data so that is where nvidia in their HGX H100 boxes have come out with a confidential VM feature.

17:16That is something which gives the confidence to the end users that, you know, my data is not exposed. And that was very, very important. You know, this is one key thing which is specific to AI, specific to GPUs, which NVIDIA also brought in and which is very, very helpful. And many of my customers actually asked for this feature. But if you go beyond this, once you have trained the model and you put the model for inferencing, either the model directly goes to inferencing with some console layer which users can consume the model into, or you plug it into one of the enterprise application. I think after that, it becomes like any other enterprise application, which end users are coming and consuming.

17:47So you will have good users who are genuine users, and you have bad users who are trying to attack it and who are trying to make it, you know, go bad. So from that point of view, all the security services, and I've got a big practice of cybersecurity, you know, as a service for my end customers who are today actually hosting their maybe SAP ERP or Oracle or some of the intranets and portals. So for that, you know, many of my existing services like I give you know a SOC as a service there's a SIEM and SO tools so there are firewalls as a service, there's an IDS, IPS as a service you know users want their end users will be coming and they should be authenticated so there will be a sort of a PIM layer or a PAM layer so there are a whole lot of security layers, there's a DDoS layer for example so yes for while I have sized up my internet backbone which connects my data center to the rest of the world through internet for a particular use cases which were present till date But now with AI coming in and once some good, serious, good models are put to inferencing use, I presume the traffic which will be coming in into my data center to use this model will be humongous.

18:48So from that point of view, expanding and increasing the size of my internet backing, making it much more robust. And also then also protecting it with the layer of firewalls and the DDoS protection layer is something which we are enhancing. But if you ask me, fundamentally, you know, the approach to security remains the same as you would put for enterprise applications. Certain specific things which are specific to AI, like I talked about confidential VM feature in H100, is something which is new. I'm speaking with Sunil Gupta. Sunil is the co-founder, managing director, and CEO of Yada Data Services.

19:20Yada Data Services is the first Indian cloud service provider member of the NVIDIA Partner Network Program. And we've been talking about Yara Shakti Cloud, which is launching, well, by the time you hear this, it may be launched already, offering India's fastest AI supercomputing infrastructure, an end-to-end environment for basically everything you want to do, whether you're a startup, huge player already. In India, and as you said, Sunil, beyond providing AI services out to the world, you've been working with data centers for a while. As AI has started to explode into the mainstream consciousness, and since generative AI in particular has really captured people's imagination, there's been more of a light shined on the sustainability of data centers.

20:04The power, you know, as compute grows, power needs grow, and these data centers grow physically. how does that sort of coexist with a world where we're talking about climate change and we're talking about sustainability and where is the power coming from and clean power and unclean power and all that kinds of things? What are your thoughts? What's your approach to building a giant data center and thinking about how sustainability factors in? No, no, absolutely. I think it is a very, very fine balance between your need for growth as well as the need for sustainability and how you actually balance it is the real key.

20:39I often say that sometime when you are starting late in industry, it is like a boom sometimes. There can be disadvantages to that, but there are like advantages to that. So you are able to go to the latest technology right in one shot. You are able to take that jump up front. And similarly, if there are certain concerns about industry, like for data center industry, data centers are known as power guzzlers. Even today, I think 3 % of the word energy is used by data center to cross two digits. if the growth of cloud and AI just keeps on happening the way it's happening. So in India, because we started the digital industry late, we started scaling it up late and now AI also has come up.

21:15So by default, we are baking in the technologies or, you know, the frameworks where not only we build larger data centers to handle these type of workloads, but we also take care of the concerns for that you use lesser amount of power and also you use the right type of power, which is the green power. So those type of things are getting baked into our design right from day one. just to give you an idea. So number one, when I designed my data centers, I designed it at a PUE level which was less than 1.5. Now, for an Indian tropical environment, you know, having a PUE of 1.5, you know, compared to what India used to have traditionally as 1.8 is something which was good.

21:50It was saving you lots of power. And now the attempt is by way of adopting latest technologies like a direct liquid cooling or immersion cooling, you can potentially bring this PUE down to 1.2 or possibly 1 also. So that will be a big, big, big contribution to environment. when you start using lesser power itself, right? And that's what we are trying to do even for Shakti Cloud that we have, we right now because the power per rack from a traditional 6 kilowatt per rack, now you're handling a 48 to 60 kilowatt per rack. There's so much power being put into the same rack. So we have actually taken the liquid, the water right up to behind the rack, which is called a rear door heat exchanger, you know, which reduces the PUE and also make sure that I'm able to handle that type of workload.

22:32That's what I've done for the Shakti Cloud environment in one of the floors in my data center. In my second phase of deployment, I'm already getting the direct liquid cooling, which is potentially bringing the PoE to 1.2. And once NVIDIA certifies, and that is something which I hope NVIDIA will, I potentially would like to use immersion cooling because immersion cooling is something where you are just dipping the chips directly into a liquid or water. And there are no fans. And practically the PoE becomes zero, one, which essentially means that you are not spending any extra power for cooling the equipment, right?

23:07So this is something which is in terms of you trying to reduce as low a power as possible. The second part is the type of power you're using. Are you using more of a coal or thermal-based power or you are using green power? So the good part is that even today in my data centers, I'm using more than 50 % of the power which I'm using in my data centers actually are green sources. It's lots of hydro sources, a lot of solar and wind sources. and because our approach since starting when we started Yorta was to build large-scale data center campuses you know which can have multiple buildings and I can serve customers with megawatts and megawatts of power and racks so we took some fundamental steps we actually took power distribution licenses in India if you want to have your own power distribution you have to take some licenses from the government so I ended up taking those distribution licenses for both my Mumbai and Delhi campuses now that is giving me a leverage to decide my sources of power you know at any moment of time I can decide which type of power I'll consume.

24:00So we actually decided to have more and more element of green into our power consumption. And yeah, so I would say that as we have more and more GPUs into our data center, the amount of power we'll be consuming in the buildings is just going to scale up much faster than I had imagined earlier. So my race to put my source of power from a 50 % to hopefully 100 % green is something going to become faster. So possibly in the very short period of time, I would ideally love to have my power source to be completely 100 % great. Great. You've spoken about it a little bit in this conversation already, but kind of just to put a point on it, what does Yada's partnership with NVIDIA mean, kind of both to the company, but also to Yada's position sort of now and going forward in the global tech landscape?

24:50Well, today, if you're talking AI, you're talking NVIDIA, right? NVIDIA is practically holding, I think 88 % of the whole market share, right? And with the type of announcement which Jensen is making, both on the hardware front, the GB200 is just taking it so many notches against competition. And of course, the bigger focus is clearly that how do you make the GPUs to the practical use cases of the industry? And you guys are doing so much in terms of the software libraries which you're bringing in. So that capability that I'm aligned with NVIDIA and you are becoming a NCP, NVIDIA, you know, the cloud partner, and you're following the reference architecture to the T, that is something which is giving a huge confidence to the end customer that these guys are bringing in exactly what NVIDIA is doing in their own DJX cloud.

25:39I'm practically a replica of that. In fact, it is NVIDIA's professional services team. You know, so first I engaged the NVIDIA design team to design my entire GPU cloud. Now it's the NVIDIA professional services team who will be landing in India and the first week of April and till 15th of May, they'll be there. Right. Who will actually will be doing the commissioning also and then they'll be putting all the CUDA layer and all the software layer on the top of that and then they'll be taking this whole cloud through a huge, you know, I would say sequence of testing it and then we'll be publishing the performance benchmarks on MLPUF.

26:10Right. And we are expecting because we have followed the exact, you know, reference design of NVIDIA with the same software as well. So my performance benchmarks of Shakti Cloud ideally should be almost same as what NVIDIA published on DJ Cloud also. So this is something which in all my conversations with end customers, the moment I tell them this, coupled with my software layer and coupled with my managed services which I'm delivering through the end customer in my data center, there's something with a huge confidence that yes, these guys are bringing in absolutely top-notch something which is there, available in any part of the world.

26:41So that is helping it very, very big time. I feel this is from my side, but I think if you see the reverse way also for NVIDIA, it's like a marriage of two consenting partners. For NVIDIA also, I think, you know, the Indian market is a huge market, you know, not to be ignored. After US, India has the highest potential to become the market for AI. NVIDIA also needed a partner who knows the local situations and who has the right set of capabilities and the infrastructure to take the NVIDIA capabilities to the end customers. So I think it is a great situation to be in. we are talking to a whole lot of customers across segments whether it's government customers or startups or enterprises or educational institutions like IITs in India or for that matter a whole lot of customers from other parts of the world because scarcity is everywhere so whether it's Europe or Middle East or some of the APEC countries you know I'm getting requirements from everywhere and yes because it is NVIDIA because it is NVIDIA reference architecture and because I've got all the software stack out there.

27:41And NVIDIA, the company, is fully behind Yotta in our success. That is something which is coming in very, very handy. That's tremendous. Kind of to wrap up, and again, everything you've said so far sort of leads into this, but to ask the question, what's the vision? What's the vision for Yotta in India, globally, once Shakti Cloud is deployed? What comes next? Where do you see this going? And if the answer is, like you said, I don't know, because everything's changing so fast, that's okay too. Yeah, somehow I believe that you can never be a fixed strategy. You know, the days of having a fixed strategy and just keep on working on that strategy are gone.