With Groq, Jonathan Ross is taking AI inference to new speeds

0:00Starting a business comes with its share of ups and downs, which is why staying true to your vision is essential. a non-negotiable for Romeo and Milka Bregali, Capital One business customers and co-owners of Ra's plant-based restaurant in New York. Romeo and Milka took a leap of faith when starting their own restaurant, gutting an empty space and building it from the ground up, every pipe, every wall, every detail. But building from scratch came with a heavy financial burden, which is when they turned to their Capital One business card. With the flexibility of the card's no preset spending limit, they were able to spend more and earn more rewards while bringing their vision to life.

0:36Today, Roz's success is proof that with passion and the right support, it's possible to make your dreams a reality. Learn more at CapitalOne.com slash business cards. I want you to imagine walking up to a computer and typing in a Google search and waiting a minute for an answer. You would just give up and go on and do other things, right? Once you get speed, you can never go back. It's sort of like getting broadband. You never want to go back to dial up. That's Jonathan Ross, founder of Grok, one of the rising AI chip makers out there. And Jonathan says that the reason why people want fast internet is the same reason why people want fast AI.

1:16And so right now, everyone is fighting for market share in AI. And if you have a high latency version of a product, you are competing with one hand tied behind your back. Grok, with a Q not a K, so not to be confused with X's AI chatbot, is not one of those companies. They make low latency a priority. Their AI chips, called LPUs, focus on inference. This is when AI takes all the information it's been trained on to make a decision. So basically, that answer ChatGPT just gave you on what to cook for dinner, that's inference. Grok is taking their claim on fast inference, and as a multi-billion dollar company, their bets are paying off.

2:01Today I'm talking with Jonathan about how his chips provide faster inference while saving energy, why AI will create more jobs, and the importance of democratizing access to AI.

2:17I'm Rana El-Khalyubi, and this is Pioneers of AI. A podcast taking you behind the scenes of the AI revolution.

2:33Jonathan, welcome back to Pioneers of AI. We last saw each other in December at the Fortune Brainstorm AI conference, which was an awesome conversation. So I am so glad to have you back on. Thanks for having me. I appreciate it. So to kick us off, I want to take us all the way back to when you started Grok. Before that, you were at Google. And I'm just always curious to hear people's kind of founding stories. And what was the tipping point that led you to start Grok? So you want to know what my radioactive spider bite is? Yeah. So I started the Google TPU. That's the AI chip that Google uses. And I did it as a side project.

3:12It started to become known that Google had made an AI chip. We actually had it in production for about a year before it was ever announced. And so I started getting calls from other companies that also wanted to build AI chips. And one of them made a pitch to me, which was, come join them because Google had a tremendous advantage in having their own AI chip. At the time, it was 10 times faster than any GPU. And make sure that there isn't a concentration of AI in one place. Help us make sure that there are two players. And I was thinking, gosh, that's a pretty good argument. But one versus two, that's not a huge improvement.

3:48I want to make sure that everyone gets access. So that was in the back of my mind, but I still didn't intend to do a hardware startup because, frankly, it's hard. And then I realized, well, gosh, people are giving away the models for free. They're giving away the frameworks like TensorFlow for free. I didn't feel that software was going to be a great moat. So as I was going out and talking to investors, one of them asked, well, what would you do differently? And I said, two things. Number one, I would start with the software. The software is a mess on all of these AI chips, even TPU. It's very difficult.

4:25As in it's like not optimized for these chips? It's optimized for it, but you have to optimize it by hand. It takes a long time. But the other realization was everyone knows about Moore's law and that the number of transistors doubles every 18 to 24 months. What we realized was actually the number of chips was also doubling every 18 to 24 months. And so if you were doubling the number of chips, that meant that effectively you could act as if the number of chips was infinite. And if the number of chips was infinite, how would you design differently? And so we came up with a very different architecture that doesn't even use external memory.

5:05We just use a larger number of chips, and we lay out the models and the computational problems across those chips. And it reduces the cost. It improves the speed. It's a very different architecture than anything else that exists. Yeah, so the AI chip market looks a lot different today than it did when you started Grok. And so much has happened, including, of course, the generative AI explosion. And there are giants like NVIDIA, but there's also competitors like you and other companies. We actually had Gavin Oberti, the CEO of Etched, on our show. They do specialized chips that are very focused on transformer models.

5:44I just think the whole AI chip space is very ripe for disruption. And I'm curious, like, what do you make of the current kind of AI chip space? And where does Grok fit into all of that? Well, history doesn't repeat itself, but it rhymes. And our inspiration came from AlphaGo. AlphaGo is an AI model that was developed at Google DeepMind to play the game Go. At the time, Jonathan was still at Google, and he says that the AlphaGo team reached out to him during a critical stage in its development, a five-game match with top Go player Lee Sedol. The test game against Lee Sedol didn't go very well. And so we got an email saying, is your chip as fast as we've heard?

6:33This was the TPU at Google. And the answer was, yes, it is. We didn't know how fast they had heard, but that's the appropriate answer, right? So we spent the next 30 days furiously recompiling their model to run on the TPU. Even though the model didn't change, it performed dramatically better. It went from losing to winning dramatically. And that was the realization that compute was going to influence the quality. Now we're seeing this with the reasoning models. Why would that be the case, by the way? Why would it change its answer? Well, let's think of it this way. Imagine that you are writing an essay.

7:09And I tell you that when you write that essay, you have to do it without hitting the backspace or delete key even once. How good is that essay going to be? Right. Not so good. Now imagine you ask 10 different people to write essays in parallel, not seeing what the others are writing, without the backspace or delete key. You pick the best one. It's better, but it's not going to be great. Now you let one person iterate 10 times. There's a good chance that it's going to be a lot better. Right. That's especially true with coding and technical problems. So by being able to iterate, you actually improve the answer.

7:45This is one of the reasons we focus so much on latency, because as long as it takes to get these answers from LLMs today, as you add reasoning, it gets worse. The AlphaGo trials unfolded in 2016, but that realization of the importance of compute speed eventually led to Grok and its chip specifically designed for generative AI. Now, nearly a decade later, Grok is a multi-billion dollar company. They actually just landed a$1.5 billion deal with Saudi Arabia. So the way that we deploy around the world is we partner up. We've been working with Saudi Arabia. We've been talking to them for about a year, but this moved very quickly.

8:28We decided to do this about four months before we completed it. We signed the contract, and 51 days later, we had almost 20 ,000 chips up and running and serving traffic in the country. Yeah, amazing. What are they doing with all of these chips? What are the applications? It's mostly commercial, and it's not all Saudi Arabia, actually. So we're serving traffic from all around the world. And when you think about it, training is a bit of a local game. You do it in a data center somewhere in the world. Inference is a global game. So this is one of the reasons why you see so many people trying to build out quickly, build all their infrastructure.

9:05But one of the advantages of being fast is even from halfway around the world, we can complete a query faster than anyone else. We did those 20 ,000 chips, and that worked out so well, we've signed this deal to expand it quite significantly this year. Yes, Grok is going full speed ahead with their LPUs. But Jonathan doesn't see his company in competition with the biggest AI chipmaker. NVIDIA. We'll get to why after a short break. Stay with us.

9:45If you've spent any time building AI products or leading technical teams, you know this. Transformation doesn't fail because of ideas. It fails because teams can't move together. Enter Atlassian's Teamwork Collection. It has planning in JIRA, documentation in Confluence, video updates in Loom, and now AI agents in Rovo, which connects the dots across your work so nothing gets lost. It's one AI-powered teamwork platform designed for how modern teams actually build. Learn more at Atlassian.com slash TeamChanger. That's A-T-L-A-S-S-I-A-N dot com slash teamchanger. Do you see Grok as an NVIDIA challenger?

10:35Not really. I think when you look at what NVIDIA does well, they do training amazingly well. And with training, all you need is brute force compute. It doesn't matter how long it takes. It doesn't even matter if it's particularly expensive because you're going to amortize it across all of the usage. When you're talking about inference, typically you're going to have about a 10 to 20 times larger inference deployment than training. That's where it starts to become very cost sensitive. And that's also where you just need more scale. And also it's latency sensitive. And so our expectation is that NVIDIA is going to continue doing very well in training.

11:13In fact, we've had customers ask us when we show them the speed of our demo, should we just not buy NVIDIA GPUs? Should we just buy LPUs? And my answer to that is always, no, get every GPU you can. First of all, it's hard to get them. Second of all, you need them for training. And third of all, the more inference you do, the more you're going to want to train your models to optimize them more because then you get more out of each inference. So the more inference, the more training, more training, the higher the quality, the more the demand goes up. Every time a new model's released, our usage spikes.

11:45So it's a virtuous cycle. Now, of course, every large incumbent wants every part of the market. But success is usually based on an element of focus. And GPUs are not designed for inference. They're expensive, and they're just not low latency. Yeah. What is your competitive moat? Like, what is stopping an NVIDIA from, like, designing chips that are optimized for inference? One of our moats is that not only did we start early, not only did we build something general that seems to work no matter where the market is moving on different model architectures. But also, if we need to adapt to what's happening, we're going to be able to do that faster than almost anyone else because we just don't have to rewrite the software.

12:29We're actually moving to a much faster chip cadence. We've hinted that maybe there's a new chip coming, but there might be another one coming soon after that. And if you can get into a very fast cadence, because most of it's the software, if your software is automatic, then you can iterate. And the speed of iteration is the speed of innovation. Okay. Let's talk about energy consumption and just sustainability of AI, both on the training side and the inference side. LPUs are a lot more energy efficient than GPUs. So why is this important? Well, you're only going to be able to have as much AI as you have energy to power it.

13:07There's a stack up in civilization, right? So first you have materials, then you have energy, then you have information. and then you have compute. And compute is different. It's about creating something contextual in the moment creatively. That requires compute, but it requires everything else down that stack. The materials to build the chips, then you can't have the chips. If you don't have the energy, you can't power the chips. If you don't have the information, you can't train the models. And if you don't have the compute, you can't run them. And so it's absolutely fundamental. And then what we did that was quite unique, rather than retrieving data from external memory we actually just have a large number of chips and each chip does a little bit of the computation and then passes it on to the next set of chips to sort of like an assembly line right and the reason that that's so efficient is when you're reading from external memory there's a wire and that wire gets charged and discharged and the longer that wire is the more energy the wider that wire is the more energy that memory is not inside the chip It's actually quite far away.

14:12And so all of that data that you're reading is requiring a large amount of energy to move from that memory over into this chip. And so when we look at the amount of energy used by GPUs, just the memory reads alone are more energy than we use. Yeah, that's amazing. I love how you explained it to one. I love the stack explanation too. I had not heard that before. It's very powerful. So many people think of Grok primarily as an AI chip company. but you're a vertically integrated solution. You know, again, more like an NVIDIA from that perspective. And the Grok cloud platform, I don't know the latest number, but I don't know, it's probably over 500 ,000 developers at this point.

14:53So last week we crossed over a million. That is amazing, congratulations. So why was it so important to be a vertically integrated platform, both hardware and software? And I would love to hear some of what are some of the developers doing on the Grok cloud? So we always intended to do this. But when we started, capital wasn't as easy to get. So our initial plan was, let's sell hardware. And eventually, we can use that money to build our own cloud. What ended up happening was, it turns out, when you develop your own chip, and you try and get others to adopt it, you're actually not solving a problem.

15:34You're creating a problem. The problem is, they had something that worked. and now you're trying to give them something that requires a lot of work to get working. When we were trying to explain what we could do to people who were serving their own models and that it would be much faster, the feedback that we got was, why would you ever need an LLM to be faster than you could read? Like just like a teletype thing, like dial up, whatever. And we're like, but that's like not how the internet works. No one wants web pages to load that slowly. Use your intuition. And they couldn't. once we put an LLM on our website and showed how fast it was, all of a sudden we went viral.

16:13And we immediately started making it available as an API. We had to develop the software for that so that people could build on it. So if you've got a real amazing product, rather than trying to sell it to people on how, you know, what they're doing will get better, just put it out there yourself and go viral yourself. Like bring the possibility to life and then people see it. Now, as for what people are doing on it, you name it. I mean, a million developers, we've seen it all. Everything from people searching documents to people doing legal work. What we're seeing is most customers are either in the startup bucket or actually in the very large, like Fortune 50 bucket.

16:54There's not a lot in the middle. So we've analyzed that. And what we've come to a conclusion on. We could be wrong. But startups, when you're going from having no customer support to I've got a chatbot that has access to a little bit of information and it sometimes gives wrong answers, but it works. It's better than nothing. So it's an improvement. It solves a problem. So they're not looking for perfect. They're looking for solving a problem. When it's the larger companies, what we've seen is they're willing to try a very large number of initiatives. Now, most medium-sized companies can't afford to do that.

17:28Yeah, that's super fascinating. I am very passionate about democratizing access to AI. And, you know, in our previous conversations, I know you are too. And with my investor hat on, I'm always thinking about how these startups get access to compute, right? Because that's going to unlock a lot of innovation. And I sometimes worry that, you know, a lot of the compute is hogged by the bigger players. Who do you think is currently being left out? And I guess, how do we fix that? As always, the way that you're left out is either it's just too expensive and you can't launch what you're doing, right?

18:09Or you can't get enough to launch what you're doing. Like what we hear from customers all the time is they have requested a large number of GPUs and the wait time is just too long. Others who have paid for GPUs are waiting sometimes as long as a year to get them. But what it comes down to is, can you launch the application that you've developed? So very often people will develop on the most capable model and then they will attempt to deploy it. It's too expensive. So then they look for open source models, smaller models. Can they port their application? It's not like people are typically saying, oh, I just don't have enough compute.

18:50They're typically saying it as it's too expensive, but they mean the same thing. Can you be very specific around what we mean when we say too expensive? Yeah. So it used to be that your largest line item in running a business was your employees. and nowadays your cloud budgets are getting pretty close to to your people budgets and that's only going to get worse if you replace your customer service with an llm you're effectively replacing what some people did right now people are growing sort of hybrid they hire people but they also hire LPUs and GPUs, right? Now, it's not too surprising that people like to do stuff with AI because it's repeatable.

19:40Once they get it to a certain capability, they can just scale that up. It's not like I have to train another person to do that thing, or then they want to scale up really quickly, and they can't. They can't get access to enough compute. And so what you're seeing is we're getting to a point where we're almost able cost-wise to get the same capabilities out of the hardware as hiring sort of minimum wage workers and so on. And then you're just not gonna be able to scale if you can't get that. Yeah, as we think about the cost equation, it's back to this cost of every call you make. That factors in too, right?

20:19So if you have a customer service bot and this bot serves, I don't know, a thousand customers, right? and these customers are prompting the bot all sorts of questions, you got to kind of put into consideration the cost of each of these calls to the LLM or to the underlying model. Is that also how it gets expensive? Well, there's also another thing. I actually believe that AI is going to cause labor shortages. I don't think we're going to have enough workers. I'll give an example. I was talking to someone recently. they were explaining why they thought all customer support jobs were going to go to AI.

20:57And then what they explained was they just had had this amazing experience calling into a call center. And it was all AI up until the final step where it went to a human being to verify the correctness of the actions. So I asked them, how often do you typically call support? He's like, never. It's a terrible experience. I'm like, but that was a good experience, right? Yeah. Would you call more often if you got that same experience? Absolutely. So now that you're making that experience good, people are going to want it. If you know that you can call an airline and change your flight easily, if you know that you can call a hotel and get any issues sorted, you're going to do it more.

21:43And so the demand for that service is going to increase. And even if the proportion of time that's spent with a human being on the other side doing it, is lower, overall, the value to that business because of the better experience is going to mean that they're going to spend more on it. But there's other reasons why I think it's going to cause labor shortages. The other one is, think about jobs that exist today that didn't exist 100 years ago. Software engineer, right? It's not like those people just sat at home and did nothing. we invented new jobs for them to do. And then the last part is if we end up in this sort of deflationary economy because things are so inexpensive to build, a lot of people are going to opt out of the workforce because they're going to be able to work either part-time, they're going to be able to work fewer days a week, or they'll be able to work fewer years of their life before they retire and then go on and do whatever they want to do.

22:42And so that's also going to lead to labor shortages. So the fact that businesses can exist with fewer employees means there'll be more businesses doing things that weren't even profitable before, that many people will opt out sooner, and that there will be greater demand for services because the experiences will be better. And I think all of those will come together to cause enormous labor shortages. Yeah. What are some new jobs that you're seeing AI create? The most obvious is prompt engineer, right? Yeah, right. We're actually looking to hire our first head of prompt engineering. How do you hire for that position?

23:18So we're trying to figure that out. But here's some of the initial intuition that we have. Prompt engineering is more about communication and leadership than it is about doing sort of, you know, grindy work where you're just like iterating and tweaking. And so can you communicate clearly? Can you visualize what you're to get at the end. The way that we look at it is it used to be that you would have hardware engineers that would wire things up. And then this wacky thing called software engineers, which at first people were like, that's not a real job, right? You just sit there and you type stuff like what?

23:54And then eventually people became full-time software engineers. But at first they had to do a little bit of hardware engineering in order to make the computers work. And then And what we see happening next is right now, prompt engineering involves a little bit of software writing. But eventually, it's going to be purely through language. Now imagine anyone who can speak can now create their own business. It's going to unlock billions of people. There's, what, 1.4 billion people in Africa, about the same number in India, right? All of a sudden, they will be unlocked. Yeah, it's very exciting. AI can unlock human potential.

24:35But with more access, how can we make sure that our data is safe? Jonathan thinks the answer is open source. That and more after a short break.

24:56Meet Nicole Nicholas, Capital One business customer and co-owner of Ansett Uncles, a plant-based restaurant and community space in Brooklyn, New York, that got its start from a need for unity. The inspiration, it was born from the desire to create a space that felt like home, where we can connect community culture, good food, and come together. with family and friends. That's how we birthed aunts and uncles. Nicole and her husband, Mike, were fulfilling their dream of bringing people together out of their home kitchen, but they soon learned that the demand for community was greater than they knew.

25:27It became overwhelming and we were like, we need home, but not in our actual home. We realized that there was also a need in our community for something bigger and in our neighborhood, so we had to find a place. Moving from a home operation into a storefront was a huge next step, but Nicole and Mike were able to take it on with the help of Capital One Business. It's not for the weak. As a small business, finding resources is super important because that's the way you'll be able to manage and scale. We would have never done that without having Capital One to be able to help us along the way. The cashback rewards are very helpful.