Why Star Google AI Researcher Joined OpenAI, OpenClaw Competitor Arrival, Amazon’s AI Chip Advantage

23:03So even if their chips cost more, you know, over the long run, maybe they would, because each NVIDIA chip is more efficient and does more work.

23:13Stephanie Palazzolo:And I might be actually conflating the model efficiency here with the chip efficiency, because the chip efficiency ultimately comes down to how much power they consume as well. So I might be conflating two separate things there. But one of the interesting points in your story is that you talked about the prospect of customers' enterprises wanting to build their own data centers potentially with these chips. What do we know about that? So, yeah, I mean, that's something that's really interesting. So first of all, Andy Jassy in his shareholder letter in April talked about, you know, we might start selling these R chips outside of, you know, AWS.

23:53Which is a big deal because, you know, like AWS like basically invented the public cloud. They got all these people to move off of, you know, their own data centers. And now they're seeing an opportunity to sort of work with customers' data centers, especially in, you know, regulated industries where they can't do all their work on the cloud. And then because of the problems with accessing and paying for NVIDIA chips that we talked about, I talked to some customers that are interested in opportunities to use Amazon's chips in their own data centers, either through sort of hybrid cloud environments where Amazon does some of the equipment and some of the management for the data center, but the company does own some of the footprint.

24:38or, you know, totally just buying the hardware themselves. I think there's sort of interest and then exploratory conversations happening on both fronts between Amazon and, you know, enterprises. So starting certainly with hybrid cloud environments like AWS Outpost, which is kind of a system where AWS provides you with some equipment, but, you know, the company owns the data center.

25:03Stephanie Palazzolo:How big is the chip business for Amazon now? They say it is a$20 billion run rate. So, and, you know, to be fair, we've been talking right now about AI chips, Tranium and Inferentia. Amazon also has a CPU, so that's like your kind of traditional chip, and that's called Graviton. So the$20 billion run rate includes Graviton. But it notably, you know, so it's just sort of, I think that we have, we break this down in other stories I've written. But that$20 billion run rate is sort of just for companies renting the chips versus, like, Amazon using the chips in, like, other kind of services or capacities.

25:43Now, what's interesting is Jassy has said that the business would be a$50 billion run rate business if they started selling chips outside of their data centers in the way that NVIDIA does. I mean, that's what he says. I don't know if we can fully take his word for it. Like, how did they get that number? You know, it sounds good. But right now, yeah,$20 billion business, including Tranium, including Graviton. And, you know, we'll kind of have. And so that's why they kind of see an opportunity of like, how can we grow this outside of, you know, just the traditional way people have been renting Amazon's AI chips, which is including in like, you know, Amazon's AWS servers.

26:22Stephanie Palazzolo:So what's interesting here is, I mean, Amazon and Google are very much developing these chip franchises pretty quickly, I'd say. Google, of course, had that deal with Blackstone. They established this joint venture where they're going to create a lot of compute capacity to rent out with the TPUs. Could you see Amazon pursuing maybe a similar kind of joint venture with a financial institution like that? Yeah, I think it's definitely like something that could be on the table. It seems like they are really seeing their chips as sort of a standalone business for them and not just sort of like a means to an end for AWS.

27:03I think actually speaking to that strategy, at the end of last year, they took the chip business and reorganized it outside of AWS into a separate unit of Amazon, which also includes the development of their own AI models and also quantum computing. So they kind of made a whole new organization separate from AWS. So I think, you know, they want to sort of be leaders in chips. And I could totally see them trying to get a partner to help, you know, both through that business, especially when a lot of the neoclouds where a lot of this sort of chip activity is coming from are supported by NVIDIA. And it's, you know, you're trying to figure out ways to get in and support sort of new entrants to the market that might want to use Amazon's chips.

27:48I mean, though, having said that, you know, it is a good sign that I've talked to some companies that seem interested in, you know, using the chips and they're kind of getting traction. So I think that kind of would give support to, you know, outside investment. You know, I think before the Blackstone deal, we reported that, like, you know, Google was talking to Meta about using their TPU. So once you have customer interest, I think that can get financial institutions interested in sort of making a bigger bet.

28:17Stephanie Palazzolo:Great. Well, Catherine, I want to thank you for coming on. That is Catherine Perloff, our Amazon reporter here at The Information. Kindred Ventures recently raised$355 million in new funds. The company has invested in Coinbase, Perplexity, a whole host of other companies, my favorite, Blue Bottle Coffee. Joining me now to discuss the new fund and some of what is on his radar is Steve Jang, founder and managing partner at Kindred Ventures. Steve, welcome back to the show. It's great to have you here. Thanks for having me. Do you have a favorite Blue Bottle order? You know, black pour over. That's it.

28:52Stephanie Palazzolo:Black pour over. Okay. I'm an ice NOLA guy myself with a little bit of oatmeal. The sugar, right? Right. Okay. So you got a new fund. What do you want to invest in with the new money? Yeah. You know, I think there's a few things that are happening right now. You know, one, we're a focused AI and deep tech investor, and we have been for the last couple of funds. We made that pivot back in 2022, 2023. That's played out well for our founders and for our LPs. And what that means is that we've been focused in a few areas. One is Frontier Labs. We continue to believe that domain-specific frontier labs, research labs, have an opportunity to create real progress where large, dense foundational language models are not making the progress in deterministic outputs that a lot of different industries and specific use cases need.

29:45The second area is AI infrastructure, which continues to be a huge opportunity and a pain point for the overall AI industry. And the reason why that is, is that the widening gap between demand and supply is ever widening. You know, we have right now 20 gigs, essentially gigawatts of compute power at AI data centers today. We will probably get to, if we're lucky, we'll get to 40 gigawatts within three years. But by our estimates, we see the demand for compute, AI compute, to actually reach 80 to 100 gigawatts by 2030. And so we have a potential gap of 60 gigawatts of compute that we need to race to build in various different oblique ways to shore that gap.

30:36And so AI infrastructure continues to not only be an area for early stage investment for us, but also ever widening and ever growing opportunity. And the third area would be physical AI, which includes autonomy, self-driving vehicles, all the way to humanoid robotics and specialized.

30:54Stephanie Palazzolo:Your old stomping ground, Uber, is the Uber flavor of investment. So those are the three areas that we really. And you announced a deal today. Architect Labs is probably in that second bucket, AI infrastructure. They are a custom AI chip lab. Is that the idea? Correct. So it was interesting to listen to your last discussion with one of your reporters in that I think what people are realizing is that controlling their own stack all the way, not only from applications down into models, right? of fine-tuning open-source models for lower token costs across their operation and product, but also going down into chips, right?

31:39And to be able to have different types of custom ASICs, different edge computing combinations for their particular needs. So if you think about a robotics company that has a physical robot as well as a cloud operation, if you think about autonomous vehicles, If you think about software companies and hardware device companies that may need a custom chip that a current NVIDIA GPU is overpowered for, you want to be able to rapidly develop new chips as well as new models. Architect Labs is offering a way for a lot of these companies to co-design with Architect Labs research team. And they come from Anthropic, Intel, Meta, Tesla, and Apple.

32:23And this team has come together to be able to rapidly use AI models to co-design new variations of chips very quickly in the same way that you develop a new model. Why wouldn't they just go to Broadcom? Broadcom does the back end. This company accomplishes their work on the front end of chip design. And so you have a whole process of front end chip design. Then it goes into back end chip design. and then it would go into tape out and fabs from there. So there's a whole process and workflow that takes many years for a novel chip design to become live and public and shipped to customers. And front-end and back-end, walk us through the difference.

33:04Stephanie Palazzolo:So what is front-end? What does that mean? Right. So in chip design, you'll go and create novel chip designs. It'll take a fair amount of research, maybe a year, maybe two years. In a large company, especially with many different interests, And then they'll go through that design engineering progress and then run simulations, run data through that. And then it'll go into a back-end engineering process. And then that back-end engineering process, they'll go through further design work, further engineering testing work. And then they'll come up with a complete chip design and then take that to, you know, there'll be tape outs with a company like TSMC Global Foundries.

33:44Samsung is a big fab and foundry partner there. Intel is another one. And so there's not many of these companies. And so the backlog there is tremendous. At every juncture that you see here, there is a stacked rank of partners that each of these leading companies will work with and prioritize. And so it's very, very difficult to do anything on the chip level under three, four years. And so what this company is doing is very, very focused on that first part, that front-end chip design based on the functional use cases and needs of that particular customer, which have historically been locked out because of cost, talent, and then also the available tools and models.

34:26And there's some companies in the space like Synopsys that have been there for a while. But again, the ability to use AI models to arrive at that novel chip design that's suitable for the customer is something that just hasn't been possible to date.

34:40Stephanie Palazzolo:Right. You sent me a note earlier this week talking about the difference between high-quality tokens and mid-quality tokens. What did you mean by that? What's the difference between these two classes of tokens? Sure. If you look at a reasoning model, a large reasoning model today, a lot of the Opus models and GPT models that you see out there that people are talking about, the high cost of it but the high value that you get from using it, a lot of those reasoning tokens are very expensive. you are, you know, there's a huge amount of long context operation that's happening with that. And then there are more instruction and tokens that are used basically for sub-agents in an agent system to keep the workflow going.

35:26And so I think in terms of inference right now, if you look at, you know, there are a lot of inference platforms like FAL, Fireworks, base 10, what you see is there are a lot of very, very high ROI tokens that differentiate someone's product or agents or robot. And then there are fine-tuned open source models that provide tokens that are great for other use cases. So in coding agents today, in knowledge work, there's a lot of very expensive token usage happening. When you look at a lot of sort of workflows and sub-agents within an agent system and applications, you will see a lot of tokens that are perfectly fine for open source models, right?

36:15Getting serverless inference at a much lower rate per millions of tokens than you would from one of these large closed models. And I think that's the argument that you're seeing out there today about model routing. I think over the last month, people have been talking about token costs and token budgets. One of our former portfolio companies, Uber, was notorious for saying,

36:33Stephanie Palazzolo:Yeah, they said, we've capped out our budget. We have no more. Yeah, and I think so. It's good. I think this concept of token maxing, it's nuanced, right? Using a lot of tokens is a good sign, but are they high quality? Is there... So if I understand you correctly, is what you're saying, high quality tokens are the ones that basically they actually get you a better product, low quality or mid quality tokens, I guess, as you put it. those are the ones that are sort of running in the background. You might not actually need, you know, some of these agents may actually be wasting tokens and maybe those are lower quality tokens.

37:14Stephanie Palazzolo:And maybe that's what's jumping costs. Is that the way I understand it? Well, there's many different parameters to how to assess this. But think of it as like high quality inference, right? High value, high cost inference versus lower value, lower cost inference. And it doesn't mean that it's sort of maybe like poor word choice to say low quality or lower quality. But really what it is is saying there's a different type of model that's useful for a different type of functional use case within an agent workflow, for example. And so reasoning models are quite expensive, as we've seen. Right. That would be a high quality token.

37:51So you wouldn't want to use that for certain types of workloads, certain types of jobs. And that's where open source models and most of the high quality open source models, KimiK 2.6 and 7, have kind of emerged as the most powerful open source model for coding agents. Cursor uses it. Perplexity, which I'm on the board of, also uses that model heavily as well as Quinn. And they're fine tuning with their own proprietary data. They're creating reinforcement learning loops that help them improve the quality of the outputs of that model. And so that makes a lot more sense than using a closed large language model that's very expensive today from the opening.

38:30Stephanie Palazzolo:But in your definition, that would be sort of what you call a mid-quality token, a token that is being consumed by not one of the most expensive frontier cutting-edge models, but maybe an open -source model that does the job very well. They're getting better. But that's the mid-quality we're talking about. Yeah, and this inference level, these inference outputs, what we're seeing historically, they were sort of six months behind a lot of these. Yeah, yeah, yeah. And I think some of the recent data from some of our portfolio companies, but also what we're seeing out there from others is that that gap is actually shortening.

39:13It's closing.

39:14Stephanie Palazzolo:Can I ask, the token maxing pivot, You know, the shifted mentality, the realization that we have capped our budgets off at the big tech companies. Are you seeing that affect the revenue profile at all of the AI startups that you are working with? Maybe even companies saying, well, we thought we could bank on$10 million from, you know, Uber, let's just say, right? And we now are being told they can only spend$4 million, making it up. But is that affecting the revenue profile for these startups? I think that there are, there's sort of three tiers of where inference is coming from today. Hyperscalers, which are the most expensive inference that you can get today.

40:05And really like the hyperscalers, the AWSs, the Azures, the GCPs, they're prioritizing frontier labs, right? The large frontier labs that you know that are going public soon. And then from there, the NeoClouds have essentially been ticked out of any hyperscaler data centers at this point. The NeoClouds are now on their own. They're building out their own data centers. They're taking on a lot of capex, and they're trying to convert existing data centers out there. They are specializing, obviously, in AI workloads alone. And so from there, they are prioritizing down into Frontier Labs as well, just like the hyperscalers.

40:44And the inference platforms that we're providing to AI startups that you described, those inference platforms like FAL and Fireworks and Base 10, they are now somewhat on their own too. They're still using some of these native labs, but they're getting prioritized over for the large frontier labs. And so what that means is there's a trickle down. There's sort of a pyramid of compute happening on inference. And so as demand increases from a lot of these AI native applications and agent platforms and robotics companies now, they're coming to inference platforms and to neoclows and hyperscalers and looking for GPU compute.

41:21And this shortage is creating not only maximal revenue for a lot of these companies all the way down that pyramid, but it's also creating a shift of where you're getting compute. And so we just announced earlier this week, and I think it was on your show, we led a Series A into Hydrohost. Hydrohost is an asset light neocloud. Right. And we did this alongside NVIDIA. And the reason why we were so excited about what they're doing is they're taking this long, fat tail of data centers out there around the world that are getting – there's new greenfield sites. there's brownfield conversions there's crypto miners that are getting converted into ai data centers there's this long fat tail of telcos reits um and and uh these existing data center sites and how do you take them and provide a high sla service to perform like a neocloud but for all of these ai native startups that you're you're mentioning now and so the question the question

42:20Stephanie Palazzolo:maybe i'm trying to get at steve is so i hear that from the the you know from the infrastructure companies. I was more talking about sort of the application layer companies that these big tech companies are using, let's say, for coding or other applications, stuff like that. I mean, I have to imagine that if a big tech company is saying, we've already capped out on our budget, that would have implications on the revenue forecast or the revenue potential for any one of these AI application layer companies or maybe even the model companies. Do you not see that to be the case? No, we're not seeing any sort of negative or dampening effects on revenue.

43:01The demand, so there's like, there's demand per customer, right? Per end user, whether it's a company or it's a startup enterprise. And then there's just the expansion, the number of customers entering the market and saying, we want to use coding agents. We want to empower our knowledge workers with research like perplexity computer and APIs from them. Or we want to go into legal and we're a law firm and we want to use Harvey and Lagora, right? And medical use cases and healthcare use cases are exploding right now in terms of consumption. And that's a lot of very complex compute workloads that are coming from new biotech areas as well as synthetic biology.

43:45And so, and we haven't even really touched physical AI yet. The really like, we've been talking a lot about physical AI for the last year. We're not even in the first inning. We're like in the top of the first inning. When that hits, the sheer number of flops and tokens per watt that are going to be happening on edge as well, but on cloud around the world. Think about the always on 247 aspect of what's happening, whether it's in China, Southeast Asia, in the US, Europe. And so even today, where I think our estimates and forecasts are weakest, we have the least confidence on whether or not this is accurate, is going to be in the physical AI sector, because we're not quite sure how quickly that's going to be pervasive.

44:28And if that is, the numbers that I gave at the top of this interview about reaching 80 to 100 gigawatts by 2030 in terms of demand for compute, we may far exceed that based upon the adoption rate of physical AI. And so I think right now we're not seeing any sort of dampening effect on early stage AI native startups, even mid stage, late stage inference platforms. And I will tell you, just looking at Anthropic and OpenAI revenue and some of the NeoCloud revenue, you're seeing just a logarithmic curve that is like, right? It's actually quite an interesting time to be an investor because you have to really dig down deep two, three layers down because everyone's revenues are growing.

45:11Stephanie Palazzolo:Yeah, I mean, just like you assessed sort of the quality of token, quality of revenue, quality of earnings is very much something to think about. Steve, I want to thank you for coming on. Congrats on the new fund. Congrats on the investment today. That is Steve Jang, founder and managing partner of Kindred Ventures here on TITV. That does it for today's show. A quick note, we are off tomorrow, but otherwise, we are on this stream Monday through Friday at 10 a.m. Pacific, 1 p.m. Eastern. If you can't make it then, episodes are available on theinformation.com, on our YouTube channel, or wherever you get your podcasts.

45:48Stephanie Palazzolo:Make sure to follow us on social media, on X, on Instagram, on TikTok, and on LinkedIn. I am already excited for our next show on Monday. Have a great rest of your Thursday. Have a great weekend. Bye-bye for now.