Tensordyne's R K Anand: HPE Juniper Fabric, Logarithmic Math, MoE Inference, Air Cooling, 3nm

0:04Hey, everyone. Welcome back to another semi dope podcast. Today, we've got a special guest, R K Anand, co founder and chief product officer at Tensor Dine. If you haven't heard of Tensor Dine, Tensor Dine is building an inference chip and inference rack that does its math in a completely different number system and has other interesting architectural tricks that we'll talk about. So R K, I'm pleased to have you here to talk more about Tensor Dine. Austin, it's a pleasure to meet you. And I'm very glad to be here. I watch a lot of your podcasts and I'm a big fan. So I'm glad to be here today. Oh, right on.

0:39Awesome. Awesome. I love that. Thank you. Okay, so let's start with you. You have a fascinating background in networking, which is really interesting because I think now that we're at rack scale, obviously networking is a big part of the AI inference story. So maybe tell our listeners more about you. Take us way back. You were at Sun. You were a founding engineer at Juniper. Tell us about your career arc that led you all the way here. Yeah. So this is my 37th year in Silicon Valley. I started my career at Sun in 1990, actually, designing microprocessors. I was part of a microprocessor team. And Sun was building high-end servers for the market.

1:22So it was a really fascinating time. There's a lot of competition. There was risk versus CISC. And you got exposed to building really large multiprocessing systems. Fast forward a few years, so 1996, I was fortunate to join Juniper Networks as one of the founding engineers and led the first silicon efforts there. And the reason I was able to do that was because Pradeep Sindhu, who's the founder of Juniper, was one of my early mentors at Sun. He was part of a team from Xerox Palo Alto Research Center helping us build these big machines. And he gave me the opportunity to join Juniper. And I spent almost 17 years there with a little bit of a break and saw the company from zero to slightly over$4.25 billion of revenue.

2:12And the most fascinating part was that we were in the early days of the internet. The internet was doubling every six to nine months. And I think Juniper, I think, transformed networking pretty significantly. So it's a very good learning opportunity to start from scratch and grow with the company and eventually be very successful in building the biggest routers in the world. But most importantly, I think building great relationships and great friendships and being able to mentor a lot of the talent there. So I think it was a great learning experience for me. Oh, man, it sounds like it. Okay, so interesting.

2:47So you got to work on compute back at Sun, and then you got to work on networking for a long time at Juniper. And then here we are in this era, which is both compute and networking. So it seems like you have learned all the right things that you need for this era. Yeah, it's fascinating. You know, I think the Silicon Valley, if you've been here long enough, goes through cycles, right? And we're in one more of those really great innovation cycles. And yeah, you're right. You know, computing, you think about computing and you say, okay, general purpose computing or high-end client server computing.

3:22And how was that built? You know, because we started from PCs and, you know, mainframes and PCs and then went to client server. And Sun was a premier company, a great innovation, a great place for innovating and building great technology. and then Juniper ends up being the premier company in networking, building high-end core and edge routers, right? The fact that we have the internet today that works so flawlessly is because of all the work that companies like Juniper, Cisco, and others did. And then we come to the world of AI, and it's, again, back to computing, but then we start hearing for the last couple, three years, like networking is so central to AI.

4:01So it's a coming together of all of these things, And, you know, you can then reflect back and see if you can take those learnings and apply them to what you're doing today, I would say. Sure. Yeah, totally. So what I think is interesting, you know, correct me anywhere I'm wrong, but when you guys started at Juniper, I mean, I assume that it was Cisco was probably the dominant player in the space. There were. Cisco was started in the early 80s, and by probably mid-80s, I don't remember the exact time, but when Juniper started in 1996, the browser had just appeared, right, out of Urbano Champagne and Netscape, the Mosaic browser and the Netscape had appeared, and we were trying to put a lot of our data on the internet, and we were trying to start sharing our, think of it almost like sharing your hard drives, basically.

4:52and there were these internet service providers like UUNet was an example of them and they were using Cisco routers and Cisco was the dominant player. I think Cisco had like 93, 94 % of the core router market share and fortunately for us, Pradeep, who was the founder of Juniper, looked at the way the routers were getting built and he was trying to wonder why are they not getting built like high-end servers? So he had a general thesis that he could actually build routers far more efficiently and then we get to hear from people like Mike O'Dell who was CTO at UNED that the internet was doubling every six to nine months and he was not able to keep up and so Juniper fundamentally re-architected how routers were built and that completely transformed how the growth of the internet happened Cisco by the year, sometime in 2000 was the number one market cap company in the world you know what they were like And John Chambers was on the cover of every magazine in the world.

5:54And Juniper appeared in the landscape, and we built the highest performing router. And then I think within the span of a couple of years, we had 30 % market share. So there is proof positive that when new markets evolve, there are opportunities for startups to come in. And if they have technology that's differentiated, they can have an impact on the market. Nice. Awesome. Yes, you took it exactly where I was thinking in my head, which is like, wow, this actually feels very similar to back then where there's a new technological revolution happening that's impacting the end users, which was the internet at the time and it's LLMs today.

6:34And there's an incumbent that, of course, is the largest market cap company in the world and is quite dominant. And I'm sure when you guys started Juniper, there's probably a lot of people that just said, like, what are you doing? How could you ever possibly compete with Cisco? But then what I heard you say was the interesting insight is like, well, if you think that there's a different way to tackle the technology, that ultimately you do think you can compete on technical performance and then probably TCO and other things. yeah and you know of course the parallels only uh quickly end there because the size and scale of the market is kind of way way different now uh and and the competitive landscape you know i mean when in those days the internet you know information was more constrained everybody know we live in a different world today information is instantaneous everybody knows everybody else but in that period between say 96 and 99 there were many other companies that were funded to build routers.

7:35I mean, all those names are etched in my mind, right? There's Avicii, and there's Flouris, and Nexabit, and there's a bunch of companies. So there's an opportunity that a landscape opens up, an opportunity opens up. I think if you look at today, the scale is orders of magnitude larger. The market capitalization of some of these companies is quite significant. And the technology power that is there, especially with Google or a lot of hyperscalers also building silicon systems, is that there's a lot more information. But I think there's a, you know, just like anything else, there are waves and there are changes that happen.

8:16I think that if you look at the last five, six years, the initial phase was all training, AI training, right? And then we have slightly shifted from training to inference. And inference was initially we were doing monolithic kind of models. And now we're in the world of 2 trillion plus MOEs. So the architecture demands for inference are different from training. And then for monolithic models to MOEs with 2 plus trillion models or 2.8 like Kimi K3, the pressures on the system are different. So there's a window of opportunity that opens up on if you think about it and how can you build a system that addresses demand for today's market that looks different from something that's four or five years back.

9:03Yes, perfect. Okay, so lead us from there then into Tensor Dine. And we have this window here. It's inference at just kind of massive scale. It's mixture of experts, 2 trillion parameters. We want high interactivity these days. It's agentic AI, right? So agents want to run at thousands of tokens per second, ideally. And so we've got this gap. We've got this window. There is definitely a way that everyone's doing it now, which is mostly like running this inference on racks and racks of GPUs. And Tensor 9 is taking a different approach. So tell our listeners, what is the technical differentiation that you guys are pursuing that you think is a different bet?

9:46If one were to break down the problem,

9:51you know, the models are immensely large. They, you know, so they don't fit on single devices anymore. So now you have to have many devices that somehow operate in a kind of a synchronous or orchestrated mode. Well, how do they do that? Well, you have to connect them together. So there's a networking problem that's in front of you that makes it apparent that the model ain't fitting on a single device. You've got to spread it across many devices. then there are power and other constraints that are in front of us, right? We can't get power quickly enough. These data centers are large.

10:27And, you know, we didn't, I mean, we didn't start at this point, right? We, if you look at, if you want to, you know, rewind a little bit, TensorDine got its, was prior to this, was named as Recogni, and we had our Series A funding in 2019. And we were more, at that time, the world was about image and image recognition and ResNet and ResNet 54, ResNet 101. And we started as an inference company. But because it was image recognition, we had to think, okay, how are we going to build a piece of silicon and a system that can do really exceptional, deliver high performance for AI, for inference, for image recognition?

11:10And the market was automotive that we focused on. Automotive has a power constraint. The car can only give you so much energy. So there's a power constraint and you have to deliver inference and you have to process, let's say, incoming frames from cameras that are at eight megapixels. So we set that up as a problem statement. And then the realization was that if you took traditional math and tried to do it in the car, you would blow the power budget of a car. You know, rather than using it to move the car more and more miles, you would basically, you know, shrink the range of the vehicle. So our approach to inference at that time was like, let's think about mathematically it being different.

11:50And let's approach math from a first principle math perspective. And that's how we converge on using logarithmic math for AI. So that set up, if you want us to be fortuitous, that set up the foundation for TensorFlow today. We didn't build a perfect device, but we validated our math. We validated that it really worked for AI. and the analogies were that, you know, first of all, logmat's been around since John Napier invented it over 400 years back. So it's not something that we dreamt of. And logmat has been used. You know, if you think about like the Apollo program and all these programs, people used to use slide rules and logmat to determine escape velocity and re-entry and all these things.

12:40And then over the last two, three decades, we've been using log math in DSPs. So it's not like we were the inventors of log math, but our realization was that can we use log math in AI? And so that helped us, right, and set us up for today. Gotcha. Okay, okay, interesting. So for listeners who haven't heard of Tensor9 before, Tensor9 actually started as a different company. What did you say the name was? Recogni, R-E-C-O-G-N-I. Recognize. Okay. So probably image recognition. Maybe it came from that. Yeah, yeah, yeah. Exactly. So you were looking at like convolutional neural networks, that kind of stuff, pre-LLM, pre-transformers.

13:21And you were asking, okay, how can we do this at the edge, thinking automotive, so self-driving cars, that kind of thing. It makes a ton of sense. And obviously a power constraint there, which I'm sure we'll get to, same power constraints in the data center, same but different, but like thinking about power. But you were asking, how can we do this incredibly efficiently? And the idea was to use log math instead of the traditional matrix, multiplication, multiplies and adds. Tell our listeners a little bit more about what does log math unlock as the benefit? So you can't avoid matrix algebra. Matrix algebra is a fundamental for all of AI.

14:05So it's not like we could avoid that. But if you take a number and convert it to a log number, then, and most of the operations are what are called mat-muls, right? Which is basically, if you think about it, it's a multiplication followed by an accumulate, an addition. So we're doing trillions of multiplications and additions in AI. Whether we're doing it in convolutional neural networks for image recognition or in LNMs, it's exactly the same. So when you take a number and make it a log number, then log base to A times B is log A plus log B. So multiplier becomes an adder. And, you know, let's think of ourselves as young children.

14:43Let's say we are in elementary school and we're trying to multiply two numbers or add two numbers. Adding is easy, right? You just take the numbers and add them. But when you want to multiply, you know, you have to do the long multiplication and add a bunch of after that. It's the same thing in silicon. If you think about it, you have to have a lot more gates. You're going to take a lot more area, a lot more power to multiply two numbers. But you want to add two numbers, it's more easy. Adders are, you know, like a building block in silicon. So you solve the first part of the problem. The second part of the problem is really hard because now you have to take the result of the multiplication and add it.

15:21And the minute you are in a logarithmic domain, adding becomes a problem. And if you were to take a traditional approach of a lookup table or Taylor series, all the gains that you have in going to addition in place of multiplication are lost. So our fundamental innovations that we patented, it was to figure out how to go back from a log to linear space and perform the additions without losing the gains in area and space and power that we got from multiplying the numbers, actually adding them, right? And so we've spent inaudited amounts of time in mathematics. So almost half the company spent almost half a decade in math, right?

16:04Because that's how you can be a revolutionary startup rather than an evolutionary startup. Because if you do the same thing like everybody else does, and you have access to the same, say, silicon geometry, 7 nanometer, 5 nanometer, you're going to have no advantages. You can't compete against the biggest guys. So you actually have to think about this differently. And that goes back to how, you know, in my thinking, reflecting back in 96 and 97, how Pradeep and the other co-founders of Juniper thought about routers, right, and do it differently, forwarding in silicon versus forwarding with, you know, microprocessors.

16:42So, similar analogies applied here, and so we spend a lot of time thinking about that. Nice. Okay, awesome. So you reached for log math because you realized, okay, we have a bunch of math moles to do. And multiplication is expensive from a power and a dice area perspective and ultimately probably latency as well. And therefore, if we can convert it to the logarithmic domain, we can turn these into ads. But then eventually it has to get turned back, converted back. And you guys thought really hard about how to do that efficiently, effectively, cleverly. And so, okay, so then you said, okay, we figured this out.

17:22This is great. This is more power efficient, small, fast. So then you took it to market ultimately, or at least built chips for automotive to kind of prove out essentially the R &D side of things. Okay, now connect me from there to getting into data center. All right, let's do that. Let's do that. So think of a convolution neural network as like a three by three matrix. So you have to perform nine operations, nine multiplies, followed by all the additions that may add up those nine numbers. So we had the building block for it. And we had functioning silicon in seven nanometer that was fully functional.

18:06And we were able to demonstrate taking camera streams and doing that. we hadn't incorporated a lot of the safety features that we needed for automotive and so at that time in you know late 22 early 23 chat gpt happened and we realized quickly that our building blocks were really set up perfectly for this moment the market was 50 or 100 times larger the ability to have a product from power on to getting to revenue is much, much faster in the data center, in networking, rather than in automotive, because automotive has a long call cycle, right? It takes a long time to build a car. You got to certify it.

18:49You got to make sure it's safe. There's a number of things that need to happen. So we coalesced as a team and said, let us look at our technology and see how applicable is it for LLMs. An LLM, you do one by one multiplication, not three by three. Great. uh well llms so if you look at resnets and all think of 100 million parameter systems and now you're going for 100 million to multiple hundred billion to trillion parameter so the memory hierarchy the memory architecture has to change you have to think of memory slightly differently yeah uh but your fundamental building blocks for math you can build upon and so that's what we did And so we started architecting in 2023.

19:34So we had the good fortune of retrospective view on math and 100 million parameters. But we also had a view that what was a problem statement in front of us? That is that you could go from multiple hundreds of billions to maybe trillions of parameters. And so how do we build and architect a chip for that? We still hadn't solved the networking problem, though. So we said, okay, let's understand how to build first an accelerator that could be world-class and deliver world-class performance for LLMs. And then the multi-hundred billion parameter problem now became a multi-trillion parameter problem.

20:13Yeah. And so now you have to think of the networking problem next. Yes, yes. Okay, so you guys said, oh, we've got all the building blocks. The opportunity is enormous. the path to production is shorter. I'm sure it was hard decision, but in retrospect, it's kind of a no brainer. But of course you had to make new memory hierarchy decisions. And then to your point, that leads to like, oh, the model weights are not going to fit on one chip. Therefore, we have to connect lots of chips. Therefore, networking has to be like a first class design priority. It does. Yeah, so tell us more then about your networking and where you guys ended up.

20:57Yeah, so now let's for a moment step back and think about the first principles of how do you build JNI inference systems, right? So there's compute. There's constraints that are at their place, right? What is your compute? Whether you compute bond, whether you're like memory bandwidth bond, how much of bandwidth do you have coming in from external memory? Memory capacity bond, How much memory can you put? And then when you think about the networking problem, then you say, okay, how much bandwidth do I have facing in and out of, let's say, scale-up network? And then what is my scale-up network latency?

21:36Because that also might have an effect. So now you think about silicon and you say, okay, how much of my real estate do I allocate for all of these things? How much do I allocate for compute? How much do I allocate for memory interfaces and memory capacity? How much do I allocate for on-chip memory? There's SRAM. And then how much do I allocate in terms of interfaces and bandwidth and for my fabric interfaces? So when we sat back and looked at it, and this was like, I think, I believe in the summer of 2023, the realization from me and some of my colleagues were, gosh darn it, we'd been building these networking systems for 15, 20 years at Juniper Networks.

22:21And I had the good fortune of leading some of that effort at Juniper in Silicon for almost 17 years. And in the backside of a router, in the rear of a router, is a scale-up fabric. Because if you think of a router, it's got lots of front-side ports. So think of like a high-end router. It could have, I don't know, almost hundreds of 800-gig ports. So traffic comes in from one port and then goes into the back, the rear of a router, and somehow that traffic has to be routed to an outbound port. Yeah. So a router typically is, you know, in a large point of presence, AT &T or Verizon, or connects big data centers to Google or Amazon.

23:06So in the rear of a router is a scale-up fabric. And it's designed to deal with any-to-any traffic, any packet size. So typically, if you look at TCP IP, you could have a small 40-byte, 20-byte packet. You could have a jumbo frame, 9K. And the fabric has to deal with this random traffic. It might have email, which is SMTP traffic, or it might have real-time video, like this call. So the router has to be robust under all conditions. And so the realization internally was there is a ready-built scale of fabric available for us. if you can make the connection. And so we had the good fortune to be Juniper Alum and HP Juniper now, HP Juniper Alum.

23:55And so we reached out and partnered with Juniper Networks and leveraged one of their highest end routers and took the scale of fabric in the rear of it and leveraged that for our system. So suddenly we went from having an accelerator that could be world-class, highest performing, to an accelerator that is now well-connected in a scale of Fabric. So this combination of networking and compute, if you had that, if you had the background, if you thought about it, then some of these docs get connected. And we were fortunate to do that. Gotcha. Fascinating. Okay, so you had the compute and then you needed, ultimately you needed a scale of Fabric because you're like, hey, we need to connect, let's say 72 chips together into a rack.

24:38And instead of having to start from scratch there, you're able to partner with HPE Juniper and come up with a scale-up fabric that fits your needs there. And, okay, so talk to us more about the scale-up fabric because ultimately I know that this could limit the amount of tokens per second at the end of the day. But take us there and explain it. Let's think about the system broadly, right? So I've always been, I mean, I think I've had good fortune and have mentors through my career that told me like, when you design systems, think about the balance. So if you over pivot on one resource, let's say you compute, compute heavy, then you might be in points in time where the computer is sitting idle.

25:27If you have undersized something, then you might be starved of that resource. So you have to think about how you allocate this precious resource in a silicon. So the thing that happened to us was that we wanted to build a scale-up fabric that was ready for yesterday, today, and tomorrow. And the best scale-up fabric in the world was in the backside of a router. Why is that? Because it has to have some really exceptional any-to-any characteristics. So you can connect like multiple accelerators or multiple, think of routing devices, and you can connect them together. It had to have incredibly low latency because we know that, you know, on a video call like this, latency has an effect on how we experience each other.

26:23We, you know, today we use FaceTime or whatever, and we call people on the other side of the globe. Think of that latency, right, with multiple hops. It had to be congestion free. It had to be reliable. And it had to be, it had the capacity, the fabric or the scale-up network should have the capability that you can push its utilization all the way up to 99 plus percent. And it will still perform really well under those conditions, under loaded conditions. So all of these were key elements for us. If you had to look at all of these elements, that was the Juniper, HP Juniper Network cell fabric in the backside of a router had those properties.

27:05So its latency characteristics are like between one and two microseconds. Okay, so now you look at this picture and you say, wow, this is the perfect fabric for us. What happened in parallel in the world in networking, sorry, in AI models? The models got larger and the traffic patterns went from rhythmic tensor parallelism kind of patterns to random traffic. because when you do experts, the experts are random in nature. So now communication, for high tokens per second or whatever, communication now becomes a bottleneck. And if your fabric is better than anybody else's, that communication bottleneck is relieved some bit.

27:52Sure, sure. Okay, this is so interesting. So taking a step back, you guys said, hey, when you first started thinking about this, You needed to balance compute, memory, capacity, memory, bandwidth, networking bandwidth. And on the scale-up bandwidth, you said, essentially, yeah, what's like the best? Maybe you're able to look forward and think that this would be like the defining bottleneck of sorts. What's like the best possible scale-up networking technology that's out there? and you looked and you said, hey, at the back of these world-class routers, that's great. And to your point, they can handle heavy load, low latency, and so on and so forth.

28:38So you're adopting that. And now, to your point, we're in a world where it's like, yes, you actually do need a rack of 72 chips, maybe many racks even, running a 2 trillion parameter, a mixture of experts. So the experts are living in different places. And for every token, it might need to go to a different expert. So, uh, irregular communication pattern. Um, and now contrast it for listeners with maybe like what's being used today. Like when you say one microsecond, um, how should they benchmark that against like, um, you know, let's say in an NV link or, or AMD's UA link or something like, or is it like an order of magnitude or how should we think about it?

29:20Yeah, so let's reflect back in the last couple of years. So the first scale-up system for AI in the world is a GB200 NVLink NBL72. So it's the first time, because prior to that, only eight devices got connected together. So to connect 70 devices, it's Gen 1. NVLink 72 is Gen 1. We are in Gen 7 of a scale-up fabric when we leverage the HPE Juniper networks fabric. That's point number one. So there's a learning cycle, right? Because when you build scale-up systems, you have to build them, learn them, refine them, you know, this iterative process. That learning process we can leverage from HPE Juniper.

30:10So that's one part. Now, if you look at the world of AI, most of the vendors now said, okay, we need to scale up fabric. So you saw consortiums come together with UA-Link, with ESUN. Still early days. They're still in specs and then early implementation. We still don't have accelerators that can talk UA-Link and UA-Link fabric chips and all of those things. So there's a learning cycle that has to. We skip those learning cycles with Juniper because we know the architecture of that fabric. A lot of our engineers were like core chief architects on some of that in the prior generation. So we know that.

30:53That's one aspect of it. If you look at the Helios system from AMD, they needed to get to this quickly. So they are now tunneling UA-link kind of packets through a standard Ethernet kind of network. So now you think about, you know, Typically, when you move data over any link, you have to think about, okay, how much space is used by headers, addresses, and all this other stuff, and then how much is payload? Because what matters is how much payload, and how long does it take to transmit the payload. So, all of these things are important elements in our choices. The other thing is, as a startup, if you carry the burden of building silicon, but also carry the burden of building a system, then you have to qualify the system.

31:42The system has to have reliability characteristics. All of these other things that now affect your time to market. So by partnering with HPE Juniper Networks, we can leverage an existing shipping system that is shipping today from Flexis factories in Penang. So we get to take our silicon, integrate it into and transform a router into a compute box and get to market faster with a reliable scale of fabric. So I honestly believe, hand on heart, that NBL-72 is the first scale of fabric in the market. We will be the second vendor in the market with a true scale of fabric. that has exceptional latency characteristics.

32:29Now, what is the latency? It's between one to two microseconds. And this is, by the way, I don't know for a fact, but this is what I've heard anecdotally, that typically the NBLink fabric has some latency characteristics that are not amenable to, for example, decode performance. So maybe there's some refinement that needs to happen with NBL72 in future generations. So NetNet, as a result of that, we end up with a system that's really well-balanced and can do pre-fill really well, but can do decode really well, too. So you don't have to have, you know, these heterogeneous systems that have to appear to solve this high, what's called interactive kind of AI needs of today.

33:10Yes. Yes. Okay, great. Let's go into that thread, too. So there was the era where it was just like, use, you know, hoppers for inference and then grace blackwells for inference. And then some early AI ASIC startups who started well before LLMs, namely Grok and Cerebris, had SRAM-heavy designs that worked in the inference world for high interactivity, decode very fast. Ultimately, of course, they had their own tradeoffs of you don't have HBM, you have to chain together tons of these to get enough memory. um and then recently in the last you know 12 months basically we've lived in this world where it's like okay if you want really high interactivity you can use an nvidia system with grok and that lets you extend that pareto curve to the very high interactivity low throughput but high interactivity um and you know we've seen amd uh partner with cerebris um amd also bought uh talus recently which is different conversation but um so there's we're kind of in like phase two Phase one was just run everything on GPU.

34:17Phase two is disaggregation, split, pre-fill, and decode. Put pre-fill on the GPUs, put the decode on the SRAM heavy chips. And then you're kind of hinting at this next era beyond that where I think TensorLine is aiming, and so are some other AI ASIC startups, which would be actually can you just use one chip to do both workloads, pre-fill and decode? But tell us more about that because it feels a little counterintuitive because, as you said, pre-fill may be compute bound. Decode is memory bandwidth bound. So how does one chip do both? Really good question there, Austin. So let's go back maybe 10 months back.

35:09So something happened towards the end of last year, right? GPT 4.5 got really good. And then suddenly people started seeing, wow, the core is generating, it's functional. I don't have to debug it a lot. And then comes, no, GPT, I didn't 5.3, correct me wrong. And then Opus 4.5, right? And so suddenly the demand for AI goes through the roof. And what are those models? My gut says they're over 2 trillion parametre models. So there's a realization in the industry that those models cannot really work in high interactive environments where the developers are saying, listen, I need code faster. I need everything faster.

35:53And so the reaction to the industry is, well, we have great systems for training. They do pre-fill really well. And, you know, kudos to the Groks and Cerebras of the world that had only shown for smaller models like LAMA 70 or GPT OSS 120P, high tokens per second per user. So now comes the problem statement. Like I have a system that does really well in pre-fill, can be used for training, but now let's talk about inference only. Now I need to pair it with a system that has a lot of SRAM and has low latency and I can do decode faster. So I think that is an intermediate solution from these vendors, whether it's NVIDIA plus Grok or Amazon Tranium plus Cerebras or Helios plus Cerebras.

36:44Imagine, you know, nine racks or 14 racks to run a two trillion parameter model. Imagine trying to build a compiler that compiles in CUDA for a Verarubid and not CUDA for a Grok system. And then imagine connecting them with lots of Ethernet switches, because you have to hook up these chassis together. So that is an answer to the demand of the market, right? Because the demand from everybody else is that we want to run these two plus trillion parameter models. And now if you look at Kimi K3, it's 2.8 trillion parameters. But we need high token. In the In genetic world, you want high token rates, for example, tokens per second per user.

37:27Here, the user could be human or could be an agent for all you care. All right. So now let's get back to what did we do at Tensor Dyn. So when I look at silicon real estate, there's only so much space. Now, if my compute takes less space, what does it free up? It frees up space for more SRAM. So now you imagine now a chip that has more SRAM, kind of like the Groks and the Cerebras of the world, but also has HBM and has the compute capacity of an NVIDIA or AMD. So you get the best of both worlds. You get what's called, you know, you can do high throughput, but you can also do high tokens per second per user with the same system.

38:13Why? Because ultimately, what you want to do is when you have compute, what matters is model flops utilization, MFU, for example. Well, if you have a lot of SRAM adjacent to, let's say, a compute array, you can keep that compute array really busy. And you can hide the latency of getting data from HBM. So if you have a lot of SRAM, you are not penalizing yourself by stalling your compute array every time you have to go and fetch data or activations or weights from HBM. So we start getting the kind of unique properties of both worlds. But that's necessary, but not sufficient. Because now suddenly there's communication now that now these models also have.

38:58So, you know, you got to compute, then you got to communicate, then you got to compute, then you got to communicate. And this rhythm of compute, communicate, or expert selection now, the communication starts becoming a dominant effect on latency or throughput or tokens per second per user. So our good fortune was to stick with log math, have the power characteristics, not blow the silicon. You don't have to build a chip that's vertical size to achieve the performance that's in par with others. And then suddenly you get the benefits of both of these. So now you have a great machine for pre-fill and decode.

39:35Ah, yeah, yes, yes. Okay, so the key insight is by using the logarithmic math that we talked about earlier, the compute doesn't need to be reticle size. It can be smaller. So then the question is, what can you do with the rest of the silicon? And oh, we can have even more SRAM, for example. So you can have a lot of compute, a lot of flops. You can have a lot of SRAM, which is great for decode. But then you can still also have HBM for pre-fill and just having HBM in general, KV cache, that kind of thing. and then you also feel confident in the competitiveness of your scale-up network so as you're communicating between all these things you feel competitive there too so ultimately then um if i have a rack of tensordyne oh which and by the way we'll touch on power too but my first question is if i have a rack of tensordyne chips um is it you know 72 chips in your rack and if so like do you still sort of split pre-fill and decode amongst them within the rack So you have immense flexibility.

40:34So first of all, we are building, the TensorLine system is only 13 rack units. So it's a tiny, it's one-fourth of a rack. We can put four of them in a rack. At one-fourth of a rack, think of our system, it has 72 chips in one-fourth of a rack. That's point number one. Secondly, it consumes only 30 kilowatts. So if you were to do apples to apples, the comparable system is like a Blackwell GB300. So one fourth of a rack, one full rack. 30 kilowatts, 150 kilowatts. So how did we do that? So first of all, we leveraged a Juniper air-cooled system, so it's not liquid-cooled. Secondly, we have a scalar fabric that has the characteristics that are necessary for these multi-trillion-pound meter MOE models.

41:26Thirdly, our log mat allows us to have silicon that consumes much lower power. Yeah. And so you get multiple benefits that are compounding in nature that now allows us to have performance at the GB300 Blackwell levels in a one-fourth of a rack. the second thing is that because we have the scale of fabric as this elegant properties you can you can do you can put some chips on pre-fill some chips on decode or in a full rack we can basically say okay there's one of our parts is a pre-fill part and three parts are decode parts but ultimately if you look at it because of our characteristic nature of our any-to-any scale of fabric and the characteristic nature of keeping a very high MFU because SRAM keeps that, you know, I always refer to it as the compute dragon.

42:23Keep the dragon fed because the dragons are hungry. You got to keep it, right? You can't let it off. So you keep the dragon fed. Now you get a system that is so much more balanced and can perform really well. And in some of our metrics, now we are yet to power on our system, but in some of our sense, we think that we could have advantages in the order of almost a magnitude over the latest greatest systems in the world. Wow. Wow. Okay. I'm going to reflect this back. So it feels almost too good to be true. So you can fit 72 in a quarter of a rack. That's only 30 kilowatts. You can stack four of those in a rack.

43:00So 120 kilowatts or so. And then that whole rack is air cooled, right? Yes. Yes. Yes. So you could deploy it in brownfield data centers, existing data centers. Yeah. So imagine, you know, if you look at it, we have data centers around the world. You know, some of these hyperscalers have 400 data centers around the planet. They're not liquid cooled. You can't retrofit them, right? They're in some of them are in bigger cities or in countries where you don't have enough power. But you want to get AI around the globe. How are you going to do that? So our system is, because it's a telco rack, it's 19-inch wide, it fits in AT &T, Verizon, Deutsche Telekom, NTT, New York Stock Exchange.

43:42That rack is perfect for being used anywhere in the world. So suddenly you can get that benefit. The other is that you can imagine building smaller data centers, but having the performance characteristics of a large data center. So imagine, you know, with our system, you can actually get, you might need, you can get like one eighth the power. You can get, you know, you can have a data center that's similar to a very large data center with the latest systems from our biggest competitors. So now you can deploy them in smaller locations, smaller cities. You can put, you know, 20, 30 kilowatt kind of data centers, not needing a gigawatt data center, but having the performance characteristics of a 200 megawatt or 500 megawatt data center.

44:29So you get all of these benefits, too good to be true. But, you know, if you build systems, you know, you have to know the system characteristics and system performance before you build it. You don't want to discover it in the lab. So we've spent a lot of time at this, simulating everything. Totally. Okay, very interesting. So then my mind naturally goes to a couple of questions. One, who are the target customers? Because on the one hand, I think of like enterprise because it can fit into existing enterprise data centers. And maybe instead of needing to buy, you know, nine racks or 13 or however many, you know, maybe they can buy one for their needs or a few or whatever.

45:08And then two, yeah, timing. So, you know, I think you had mentioned that this isn't totally taped out or stood up or productionized or something. So, yeah, remind us like, so tell us who are your target customers? Because, of course, I can also see, yeah, hyperscalers in the model labs and stuff being just as interested. And so maybe like target customers route to market and timing. Yeah. So our target customers are certainly, number one, hyperscalers. Number two, a lot of the NeoClouds. We have significant number of letters of intent from almost all the NeoClouds in the world. Well, I want to clarify, NeoClouds in North America and Europe.

45:49and then maybe some sovereigns and some enterprises, right? So, but as a startup, we have to be laser focused, right? We don't have a very large sales team. So we have to focus on a targeted set of customers and we are doing that. The second question was timing, right? Yeah, yeah, yeah. So we've taped out our 3 nanometer chip with our awesome partner, Broadcom. And so we expect devices back late October, early November. and we expect to power our system and then probably sometime in Q1 start doing beta with customers. We have some customers who have signed up for beta and then potentially go into production, early production sometime in end of Q2, early Q3 next year.

46:34So we're not far away from there. Now, how can we do that? Because 80 % of the system is already validated, qualified, certified in like 70 countries. we are building it out of the same factory uh so uh in in penang malaysia so we can actually quickly go from power on to qualification to certification to shipping uh of our system so we have a workout out uh for the next i would say 10 months but uh we're fairly confident that we can get from here to there pretty rapidly nice nice so is that because you're riding the supply chain that Juniper was already riding? Or is this the advantage of partnering with Broadcom or both?

47:18It's both. So, you know, Broadcom is one of the largest consumers of TSMC capacity, both TSMC and HBM capacity, right? They have a deep partnership and we have an exceptional relationship with the Broadcom team. They're great friends and great supporters of us. And so we designed, Of course, our front-end design is ours, but the physical design and the partnership with TSMC is Broadcom-led. So that's one part. So that means that from a supply chain perspective, as long as we forecast well in advance, we can get parts from Broadcom, right? That's part number one, qualified tested parts. The second part of the story is that can we get to volume with our systems?

48:02And this is where the partnership with HPE Juniper Networks and the fact that things are shipping out of a Flex factory and Flex knows how to build these systems in volume gives us those advantages. The third most important part is that these systems have to be reliable because you're running business critical AI now. So Juniper's routers and HPE Juniper's routers are carrier grade. There's something called fine lines of uptime. So finance of a time, if one were to Wikipedia, is that the system cannot be down for more than five point some odd minutes in a year or 680 milliseconds in a day. So that is a reliability characteristic that we leverage from our partner.

48:47And so we are able to we will be able to deliver systems that are reliable and can run the largest models in the world. at 2.8 trillion power meter, like a Kimi K3 or a QEN, or from the Frontier Labs, the biggest models. Our systems will have adequate amount of memory. If we look at the quarter rack system, we have 10.8 terabytes of HBM capacity. We have significant, I would say, over 18 gigabytes of SRAM. So imagine we have SRAM capacity of the big SRAM guys, and we have DRAM capacity of also the largest thing. So now, whether it's KVCash and agentic workflows, we're well set up for those markets and those customers.

49:36Fascinating. Amazing. A really nice way to punch above your weight, so to speak, as a startup by partnering with people who can help bring you along. When you mentioned Flex, by the way, is that Flextronics? It's Flextronics. Yeah, yeah. Okay. And, you know, I mean, most of Silicon Valley thrives on partnerships, right? You can never go alone, right? Whether it's Silicon partners, system vendor partners, the ecosystem actually lends itself to that partnership. And we, you know, we benefit from it. We mutually benefit from it. And we really love these partners because they are helping us get to that scale, will help us get to the scale.

50:19And they're very supportive of us, right? They're supportive of a startup. Not only that, HP Juniper Networks is an investor in the company. So you think about it, they, you know, in all of these things, you know, you can never go it alone. It's like building a team, right? In the company, the team is multidisciplinary, but you can't build a product by yourself. All these functions come together. It's the same in the ecosystem of vendors and partners who come together to build these solutions. Yeah, sure. That makes sense. And customers are the final, you know, arbiter of product, right? And customer validation is going to be super important for us over the next many months.

50:59Yes, totally. Okay, so let's land the plane there. Customers. So, you know, you mentioned really ramping up in 2027 with customers, beta, and then production. the one thing we hadn't touched on yet was software. So for those model labs, those hyperscalers, what does it look like as far as how heavy of a lift is it to take your system and get their software, their frontier grade software up and running? Yeah, exceptional question, Austin. I think a few things have happened, right? One is that from the signals of these aggregated stories, we understand that the mortes of CUDA don't exist anymore for inference because we are proof positive of a number of inference companies running solutions.

51:50And then these disaggregated solutions with CUDA and non-CUDA or Neuron and non-Neuron or ROCM and non-ROCM, right? So we have this. So I think there are a couple of things that have helped us. One is obviously we have to build a robust compiler for our hardware, but also at the higher layers having a technology that can take PyTorch or Trident kind of models and then convert them quickly and compile them quickly is super and we spend a significant amount of time. So for example, in our engineering organization, almost 60 to 70, 60 plus percent is software, right? Because you have to have a software team that does that.

52:28But one of the things happened with this beauty of these new Opus 4.5 and GPT 5.6 and other models, is that then agentic workflows are now proving that you can take kernels and quickly get them ready for your hardware. So we have incredible momentum on taking the best models in the world and quickly getting them running in our simulators and then eventually in our hardware. So the walls for taking new architectures and new systems are quickly crumbling when it comes to software because you can take models and because you have all these workflows and automated kernel generation capabilities that you can actually get to there, get to that, and also get to high what's called utilization of the systems quickly.

53:17That means that you can get first proof positive that the model works on the system and then you can iterate quickly to get the model performance up in the system. So both those things seem to coincide with our systems coming on. Yeah, yeah. What fascinating timing with the rise of the agentic AI to kind of help you with that last hurdle that people are a little concerned about. Am I going to have to spend a lot of time? To your point, not only getting my code to run on this new system, but also optimizing it to get the most utilization out of it. Indeed, indeed. And that is super important because you don't want an 800 horsepower car and get like 50 horsepower out of it, right?

53:57Because then you're leaving stuff on the table, right? You're paying for it, but you're not using it. And we believe that we will have some of the best MFUs in the industry with our system. Nice. Amazing. Okay. Well, we're at time, folks. I hope you learned a lot about RK and Tensordyne and networking in logarithmic math. We covered a lot of ground here. RK, you'll just have to come back and keep us updated, you know, as you guys get closer to delivering chips and systems. And when you have customers to talk about, you know, can't wait to hear more. Austin, again, thank you for the opportunity.

54:36Thank you for, you know, the very thoughtful questions and, you know, and navigating all the different parts of the terrain on our behalf. and we'll certainly keep you in the loop. We'll keep you posted as we make progress, as we bring up our systems, as we get to beta and would love to come back and tell you about our success and our progress. Again, I want to actually give credit to my broader Tensordyne family because we won't be here without the complete dedication of a number of people and then most importantly, thank our partners Broadcom, TSMC, HP Juniper Networks and Flex and others because we can't be here without all of them helping us in this journey.

55:20So thank you, Austin. I really appreciate it. Yeah, team effort. Very cool. All right. We'll talk next time. We certainly will. And my best to you, Austin. Thank you. Thank you. Cheers.