#206 Krishna Rangasayee: Why Edge AI is the Next Big Thing in Tech

0:00The embedded world is an open source world. So the world is used to Linux. It's used to computer vision platforms like OpenCL, OpenCV. And from an ML framework, they want to retain the same open source access. The edge market is going to be the next AI gold rush. And if you really think about the edge market, it's 40 ,000, 50 ,000 customers globally. So it's diffused in its footprint. But if you aggregate that, it's a much bigger market than the cloud. By popular demand, NetSuite has extended its one-of-a-kind flexible financing program for a few more weeks. Head to netsuite.com slash IonAI. That's IonAI, E-Y-E-O-N-A-I, all run together.

0:44Again, head to netsuite.com slash IonAI. Netsuite.com slash IonAI, where it's one-of-a-kind flexible financing program. Craig, pleasure again. And so I took a long road to really come into SEMA, about 30 plus years in the industry, a background in software, silicon, and then I managed fairly large businesses. My last tenure was at a company called Xilinx. I was there for 18 long years. I was their executive vice president and general manager when I stepped out. And I was pretty passionate about watching what's happening in AI. And clearly, I think AI, even then five years ago, took off in the cloud.

1:33And obviously, in the consumer mobile experience as well. And what was fascinating for me was that I think the physical world that we live in, the industrial world we live in, robotics, industrial automation, medical, automotive, we're still fairly using archaic old techniques and embracing AI. And I wanted to start a company that would really build a purposeful platform to scale AI at what I call the embedded edge market. So that's the history and the backdrop on how we started water lead made it start sema and now we are five and a half years into it and and sema is designing chips for the edge is that right correct we built our purpose-built chips for the edge we also make the associated development software so our application developers or our customers can write their application and port that application through our software onto our chip Yeah.

2:35Just one question, which I may or may not leave in. I had Dave Patterson on the podcast quite a while ago, and he was talking about instruction sets and creating an open source instruction set for chip design. is when you talk about the programming environment, are you talking about the instruction set or how does the instruction set figure into that? Okay. It's a simple question, but a very complicated answer. And I'll do my utmost. So you should think of our chip as a heterogeneous compute platform. So we have our own proprietary ML accelerator. We have an ARM processor complex, which is our control plane processor.

3:31And we have a DSP vector engine that we have licensed from Synopsys. So those three make up the various compute elements we have on our chip. So that's number one. Number two, unlike the cloud where ML acceleration is the only problem, in our world, it's application acceleration. So you need to accelerate not just the ML workload, but also the computer vision workload, the pre-processing, the post-processing, the control plane analytics, and the decision making all needs to be on a single chip. So perhaps what Dave Patterson was referring to is that ARM has its own instruction set to program their chip.

4:11He, I think, is driving a RISC-V platform. Yeah, that's correct. which is creating an open source instruction set so that anybody could have, you're democratizing, if you will, the access for compute. And I think one day, no doubts, RISC-V will be a very, very popular, if not a parallel path or an opportunity for many designers. In the space that we are in, the embedded market, the incumbent solution provider or lead solution providers are. and what's more important here is not just the instruction set alone but the ecosystem of software vendors that surround that platform right unlike other companies which may create a closed platform where they control the entire software stack we create an open platform for anybody to be able to build applications on top of us and also leverage the software ecosystem So in that aspect, we found that finding ARM as a partner is a better way to go for who we are as a company.

5:13And I think they've been a fantastic partner. And so we use their instruction set for the control plane aspect of it. But our secret sauce is really in the ML portion of it. But what we enable is what we call a product called a ML SoC, a machine learning system on a chip. and we allow people the opportunity to partition the problem in any of the heterogeneous compute elements we have yeah and when you say uh to you know to open it up i mean when you're referring to the proprietary uh software development environment around a chip i i assume you're referring to NVIDIA's CUDA. Yeah, correct. And in what way is yours different?

6:04I mean, not different technically, but in terms of it being more flexible or more open. Correct. So really good question. So let me try to stay at the high level and try to see if I can get my point across. The embedded world is an open source world. So the world is used to Linux. It's used to computer vision platforms like OpenCL, OpenCV. And from an ML framework, they want to retain the same open source access. So they want to be able to work with PyTorch, or TensorFlow or Onyx or any variant of an ML framework. So what we have done quite different than NVIDIA from that perspective is we have built something made for the embedded market.

7:00Everything we do touches open source. And so we kind of, I mean, if I were abstract, think of us as the Ellis Island of everything. So you could give us your poor, you're tired, you're hungry. And once they take, once we ingest their application, then we convert them to a proprietary environment to deliver the performance and the power that they desire. So we are very open source and we are not limiting. CUDA was not built for the embedded market. So to a certain degree, people are having to transform their environment to fit into an NVIDIA world or CUDA world. So we do not force the transition.

7:38They can remain in the application and the environment they've been used to. So that's a large difference that we go to. but obviously the combination of the hardware and the software cuda and their silicon similarly our version of our software is called palette palette come a combination of mlsoc that lower is a world class performance and power and ease of fuse parallel i see and and what you mean is uh People can use other platforms than Palette or Palette works with Linux and other open source systems? The latter. So Palette is our entry. So anybody can bring anything and they come to work with us through Palette.

8:29Okay. And I'm sorry, I jumped into those questions that have been kind of nagging. But let's back up and talk about the evolution of chip architecture, you know, from the von Neumann architecture and CPUs and the rise of GPUs, which was a kind of accelerator. and then the kind of split in, you know, as Moore's Law slowed, then people's, instead of trying to miniaturize transistors further, they started creating specific chips called accelerators. And then now we're moving to the edge. So can you just kind of walk us through that evolution? Absolutely. Absolutely. And I actually think you did a great job already in walking through the progression of how I think compute architectures have evolved.

9:32I would still say at the underlying element of it, until we got into ML, everybody's architecture leveraged one nine minute, some simple load store instruction, you load and you store data and you compute, right? So the fundamental architecture hasn't really shifted. And combination of Moore's law slowing, but also the performance needs being exacerbated. And power being a limiter has forced an hour, has created an opportunity for innovation. And a plethora of new approaches are still being brought together. GPUs have kind of taken pole position on solving AI ML. But from our perspective, there's no one size that fits all.

10:18And no doubts, they've done a fantastic job. And in reality, NVIDIA's strength is not just their GPU architecture, but more importantly, Kuda, right? And they've done a fantastic job in building that as a moat to really navigate and protect their growth and opportunities. And I admire what they've really done. I think for the edge market, hard limiters power. So things in the edge market cannot support a GPU like power consumption. so your world is really very simple and 5 watts 10 watts or 20 watts and anything more than that is untenable from a system power performance or from a thermal heat dissipation perspective right so you got to kind of think of it the other way and then how much compute can i get for a given power budget that forces a radically different approach and an opportunity for people to think very differently like we have and what we really came up with is saying hey it's not ml acceleration alone or ai acceleration alone we need to think about application acceleration and that's why we created a heterogeneous compute platform and if you really dig into our ml architecture it's a tile array of 10 by 10 and we fundamentally stream applications to run through the tile array and efficiently utilize the compute that we have on our ml array and so that's our approach that we have taken.

11:43And from our perspective, that's the best we could figure out to solve for the needs of the edge market. Yeah. And talk about the edge market and why AI is moving to the edge. Great question. So I think I will open with this and perhaps even close with the same statement. I think the edge market is going be the next ai gold rush and if you really think about the edge market it's 40 000 50 000 customers globally so it's diffused in its footprint but if you aggregate that it's a much bigger market than the cloud so think automotive think robotics think industrial automation think medical think the government sector we use compute in very very many different facets and everything that we touch in our lives today and none of them really have significantly adopted AI yet and over the next 10 years the next 20 years that is the next big gold rush and a lot of focus and a lot of attention that duly is focused on the cloud today is probably going to shift the edge and this is the exciting opportunity I think that's ahead of all of us and so and I think I've been positioning this is the next big thing from a compute perspective and wherever you're going to be focusing and the reason for it is very simple uh everybody's data needs are going through the roof the compute needs are going through the roof and three key dynamics are forcing people to think about the edge a lot more number one locality not every application where you collect the data has the affordable opportunity to transmit the data back on a 5g or in some network back to the cloud and compute and come back and proof point automotive you're scanning the road you want to detect a pedestrian you have milliseconds to make a decision not seconds uh robotics is a similar thing so like localized compute is key the second one is security and privacy and as you particularly get into generative ai and llm elements access to data and really protecting both the security and the safety of it and the privacy of it is going to become very very key and you're going to see a lot of architectures move to localized edge compute where the data stays with you the third one is tco the cloud is pretty expensive and also there's monetization of the data when you had the data why not monetize the data for your own benefit why give it away to somebody else right So the TCO element is going to kick in.

14:25So it's not the cloud is going to go away. The cloud will continue to be a large entity. But there will be a better balance between what needs to be at the edge and what has to go to the cloud. And I believe in the next 20 years, you're going to have a hybrid mode for almost every architecture or for every compute aspect that we're going to look through. And it should be very exciting times ahead. Yeah. And I remember when we spoke last time, everyone's looking at the edge, all of the chip designers. And there are two ways that people are approaching it. One, the big companies that have an existing, you know, very capital-intensive architecture are looking to scale down the architecture to make it work at the edge.

15:17And then there are companies like SEMA that are building up from scratch with the edge in mind. Yeah. And what's the role for each of those? Or is it, in your view, is there no role for the scaling down?

15:40Yeah. More than the scaling down, what people are not paying enough attention to is the software. So scaling down the silicon, there's a lot of smarts, a lot of capability. But AI software is an entirely new paradigm. So you cannot just add AI to an existing infrastructure. It's a kind of open-heart surgery. So people need to think about the AI software architecture first before they really look at scaling down. This is where the industry's large gap is. There are very many amazing public companies that have classically served the edge, but very few of them have an industry-leading AI software and an AI roadmap.

16:21So that's one large opportunity and gap. I would submit to you that if people don't have an AI roadmap in 10 years, they may not be around as a company. So there's going to be a mad scramble and a rush to really kind of AI enable everybody's roadmap going forward because it's here to stay. And this is not a fad anymore. And if you don't have an AI story, you're probably going to be yesterday's story. So that's the big shift people are moving through. What we are building is something purpose-built. And we really targeted this market. And I would say we are one of the first few companies to really dedicate our focus to this.

17:02We are not the very first, but we are one of the very first few. And we have come a long way in five and a half years. And we are Gen 1 in production for a year and a half. We are engaged with a wide range of applications and customers and market leaders. And our Gen 2s are on the corner. And so we're pretty excited about what we're going to bring. The learning I've gone through is while AI architectures are hard, what's been the Achilles heel remains the Achilles heel for the industry, which is AI software. And this is also perhaps the reason why NVIDIA has had a great run and very little competition to date.

17:38And so AI software is probably, whoever does a great job in AI software is going to be the lead. That much is clear. and when you say ai software you're talking about cuda or yes or the equivalent right a palette in our case correct yeah yeah and maybe just uh explain that a little bit more for listeners that aren't familiar with cuda yeah our palette right and so so i think customers design their application and they come back and say hey this is what i want to go accomplish and they have an expectation on the performance they want the power that they want the accuracy that they want or the cost that they want to spend for a particular application right so they walk in and say here's my application description here's what i want to get out of it what translates the application from a customer description into a chip executable that runs on the chip and then delivers the promise of the performance the power the cost is really the software programming environment and in nvidia's case it's called coded and in our case it's called palette so it's really a translation between the customer's description of the application to what eventually runs on the chip yeah and a translation uh to what i mean what the to to it it ends up being fundamentally a microcode or an executable that is then downloaded onto the chip so that the chip can be programmed to perform as the application requested.

19:19Okay.

19:23You mentioned some of the applications. What are the kinds of industries that you're getting the most interest from? Right now, it's, I would say, across the board. And I'll give you the spectrum and then I'll amplify the areas where I believe there's going to be the highest traction. So we are engaged with the top five customers in robotics, top five customers in the government sector, the aerospace and defense sector, automotive, medical, transportation tech, agricultural technologies, and also a category of smart vision that's a Venn diagram overlap or industrial automation, right? So visual inspection, factory flow automation.

20:11So these are all the applications, and we have, as a company, the last five and a half years engaged with the top customers in each one of these market segments. Play this out five years from now, where would we see the highest volume, highest growth? I think clearly automotive, just by the nature of it, is going to be a key area of focus for us and the driver. right behind it i estimate would be robotics and i think there's a resurgence of a lot of focus on robotics uh supply chain issue covet brought a lot of the supply chain issues and the legacy architecture is coming in the way of global productivity in a big way but i also think that there's a lot of articles written up on labor shortages globally and people really want to bring in a lot of automation into the factory floor environments so you're going to see a lot of amrs or robotic environments across quite a few warehouses there's a lot of talk around humanoid robots this is a new area of development some of these may not pan out or as with everything in life may take much longer than anybody's estimate but the fact that i think robotics is going to play a very large portion of our lives that's a gift yeah uh and and with um with these edge applications uh they work with the cloud right i mean they're not intended to work independent of the cloud are they so um it depends on the customer's architecture take for example archip archip is capable of working independent of the club So you could fundamentally train an application and you could infer that at the edge.

21:55There is really no need for the application to be connected to the cloud unless you choose to do so. Right. And so the application is fully capable of being entirely independent, running on its own with no connectivity. And many applications, really, if you think through it in the real world, may not have connectivity elements and or may struggle to get connectivity. and you don't want to be dependent on the connectivity alone, right? So we built a chip that's entirely capable of being self-managed and being out there with no need for a cloud connectivity. But if the customer chooses to have a cloud connectivity, either for updating the algorithms or to really do a nightly upload of information or analytics, we enable that also.

22:41Yeah. Are you working with, I would assume you are, with many drone manufacturers? Because this sounds like what you would need on drones. Absolutely. And so we have built a platform that is the best in class in performance and power. And the programming environment, we believe, is the best ease of use in the industry compared to our peers, if you will. and drones are a poster child that needs all three. They want the highest performance, the longest flight time, lowest battery consumption. And I would say we are better poised to serve that compared to anybody else. And so absolutely, we are engaged with the market leaders on unmanned aircrafts and drones.

23:28And we see a lot of good traction for technology in that market. Yeah, and the processing that's taking place is computer vision and analytics and control, I guess. Are there others that I'm not thinking of? No, I think there's also navigation. Right. You could argue navigation fits in the context of computer vision. So there's fundamentally a camera subsystem or some kind of a LIDAR radar subsystem. So gathering location specific data and you're computing and you're making decisions. So it's a combination of computer vision, perception, localization, segmentation, and really distributing the workload, both from an AI context, but also into classic computer vision.

24:29and you also have control plane functions to watch your flight navigation and really ensuring that i think you're combining all that into a single chip and this is kind of what we do well and given that we are a heterogeneous compute platform we do more than ai workloads and so you could actually put the control plane the navigation the decision making and have it all work on a single chip Yeah. And how large are the chips? Is there a standard product that you talk about it being heterogeneous?

25:06I'm not the only one that struggles with that. Yeah, that's right. How large are the chips? Because certainly, you know, the NVIDIA packages or systems are, you know, getting, you know, you need a truck to move them. And they're certainly not going to fly in a drone. So how large are your chip packages? They are big. big. That's it. And so you can almost visualize what I see in my thumb, that's about the size of what we need to have. And that's what we deliver. And so we that's the chip. We also provide board form factors so that customers can easily integrate that into our existing structures.

25:55And one of the benefits and having spent 30 odd years is I've spent a lot of time studying not just me our entire team, what we need to be at in terms of cost power form factor and we have put all of that into really what we have built eventually and i think the opportunity we have had is we built something from grounds of purpose built i'm not a public company that's got a lot of legacy i have zero legacy so we have said hey this is our learning this is what we need to do and we have raised a good amount of capital and we said let's put the capital to good use and build something really helps for the customers The other side of what's happening in the chip world are the foundries.

26:41You're designing chips, but you need a foundry to etch the chips. And, you know, there's various companies are building foundries for various types in the United States. but for the time being uh tsmc and taiwan really is is the one place where you can get this done how do you get capacity sure with them um i i keep joking that though we're in the technology business it increasingly more and more is a relationship business yeah and and one of the side benefits of growing old is you get to know a lot of people that are all in key decision-making positions at all these companies a very very small company we are fantastically happy with the partnership we have at tsmc they've been an amazing partner to us from day one and they've given us and they've made us feel like a big public company and so i'm very lucky that our problems are all demand related and not supply related.

27:51And so we have worked very hard with them and we have done a good job in managing the inventory needs and the market demand and really planning well. And I would say, I think it's never an easy thing to do, but you've done something for 30 years, you tend to become reasonable at it. And so I think the team is a very well qualified team, but we won't be here without our partnership with TSMC. So we very proud of what we've done together with them yeah is there something different in the equipment required to to build these chips i mean i would presume in that you're trying to build them small that you are you want to use the the smallest uh i'm not sure what it's called but nanometer uh gates or whatever and and uh what what size are you at and are you looking i would presume you are looking at the new foundries being built and uh is your architecture something that could be produced at any foundry or is it a particular kind of foundry that you would need?

29:12Great set of questions. I think one of the key things we decided to do early on was we recognized that our software-centric approach gave us a lot of opportunity to get better performance and power. We decided to scale back in what you refer to as process technology. And so we created our Gen 1 silicon product at 16 nanometer, 1.6. and this is at least 10 to 12 year old technology. So we did not want to take very advanced process technologies and take a lot of risk. There's also a cost equation and a development equation and it's as you get more aggressive or more newer nodes, the cost, the mask set to reproduce the chip, but also the development costs both really go up quite a bit.

30:01So Gen 1, we said, hey, we're getting amazing performance and power compared to anybody else. And in real applications, we're 10X of the very best companies. So we said, let's stay back in process technology in Gen 1. We did 16 nanometer. And in Gen 2, which we have now gone public, we have announced that we'll be at 6 nanometer. So we'll be getting a 2X improvement over what we have done in Gen 1 on a performance per watt, right? Which is a huge leap. Typically in our business, you get 10%, 15 % improvement Gen 1 to Gen 2. For us to get a 100 % lead is pretty phenomenal. And we now feel like we have the right opportunity to do it.

30:41And our customers will hugely derive the benefits of it. And to answer your question, we are tracking very closely where we get our chips manufactured. And we have the luxury of getting them manufactured now in the U.S. as well. And at the appropriate time, we'll take full advantage of every opportunity that we have. and within tsmc i think we definitely have the choice of being manufactured any place and just one thing is and i think uh we we are our business doesn't work in that you build a design you could go to anybody and so yes at a design implementation level i think you have that opportunity but i think once you get into the chip development you're fundamentally locked into one entity.

31:29And that's typically how our companies work. And we are too small to play with too many players. And our volumes are too low to really play the game. And if you really do it, it's uneconomical or economically not viable. And so I think we really want to work with the best in class and TSMC is as good as it gets. And they've done a phenomenal job and they truly understand that this is services business. And even today, I'm very impressed that as a startup, we get the same attention, if not perhaps more than a public company. And so all of those really are huge positives. Yeah. So you're five years in.

32:11Are your chips deployed now in any edge devices or are you still working toward that? so we are um we are deployed uh i would still say it takes a long time in our business to really scale like the cloud business or the consumer business where the design in cycle is fast the embedded space things take a long time but then the other positive is that they stay for production for 15 20 years so it's a game of inertia getting in is pretty hard and And the opportunity to stay in production is also pretty significant. So we have quite a few customers that are moving to production with us globally. And over the next year or two, we are excited about scaling more and more customers to get into production with us.

32:59Yeah. And how many people are at the company right now? And how do you scale a company like this? Because, as you said, the demand is massive. uh we are about 160 of us today and with a lot of consultants i think our contractors we are on 200 so i would say wrongly we are 200 people working at the company um we are lean we um we we are pretty lean for what we are doing and um and it really is kudos to everybody that works here in that it's a startup and we do whatever it takes to make things happen and we are some ways doing what many companies use tens of thousands of people to get done um and and and i'm a big big admirer of everybody that we have here i'm biased and these are all more than employees family for us and that we do whatever it takes to go make things happen the macro thing that is a priority for us is scaling is scaling our go-to-market and how do we touch 40 000 customers in scaling our go-to-market partners that we have reps and distributors and vars and odms and isvs and and again i go back to not only myself but i think we have had a lot of experience scaling businesses in our career before and so we retain the core of the company to be small but through a partner network we expand our footprint and where we go look to the underlying thing that really enables us that Craig is really our software and more self-managed the software is the more the customers can manage themselves that's a large large component of how we can enable the scaling if every one of them needed a hand holding we don't have a company right and so so and then maybe I think are going to touch tens of thousands of customers and it'll be a very exciting scaling opportunity for us yeah and and the biggest market uh i would assume right now is is the united states but i don't know i mean this is a global phenomenon this absolutely yeah i i think by footprint north america would be the largest market uh for us no doubts um and but we are enjoying the benefit of our customer attraction in europe japan uh in korea and in India too.

35:24And about 70 of our team members are based in Bangalore in India. And we also are building out sales and go to market activities in India. And I think India is going to be a massive growth in general, a doctor of AI. And we feel like we are one of the leaders and pioneers in this category of edge market scaling. And I am quite convinced that I think in the next 10 years, this is going to be a high priority for every nation a high priority for every industry sector and people that don't have an ai enabled environment are going to be left behind and i think it's a matter of now who's wanting to gain leadership position and there's going to be a fear of missing out that's going to go through the entire customer base yeah and you were talking about how i'm going to avoid the word again but uh how you use you your chip is not specialized one particular application but is there an an application that i mean at some point it seems if not in the design of the chip in the uh in scaling the your sales and marketing and all that you've got to focus on on a few applications uh and let the others come to you which which are you guys focused on i would say um robotics industrial automation and vision systems if you really do a venn diagram of those three i think that'd be a key gravity area for us um and and And I should expand on that and saying every day our struggle is specificity versus generality.

37:17So we've got to really make this balance happen. And I've built products before that have worked for 40 ,000 customers, where you build one thing. Every one of them is doing something radically different than the other, but it all still ends up being mostly the performance, mostly the power, mostly the cost that they need. Not easy to go to. and and we have learned and built good businesses earlier in our career and we've used that experience to build it and i would say though i explained our gravity point of the highest volume where i think we're going to drive most people are going to find out that i think we are better than everybody else for their application as well and all the other market segments we've already talked about yeah um this wasn't i wasn't going to ask this but it just occurs to me on this question of scaling You know, the big boys are all chasing this market as well.

38:15What's your future? Do you think you guys will end up as part of a larger, you know, Google or NVIDIA or, you know, somebody at that scale? Or is your expectation to stay independent and grow to that scale? a really good question. And I would maybe give you two or three things that I think maybe I should mention. One is what I've learned now in five and a half years, and all my career I've been in large public companies, and this is my very first startup I started. And what I've really come to recognize and truly internalize is it's not the size of the dog, but it's the size of the fight in the dark that matters i've been in very large companies and i've always wondered my god we need more to get things done we are 200 people and we are doing things of 10 000 20 000 100 000 people companies yeah so when you really build yourself into an element of this is our team and this is what we're going to go do and this is our life and blood you operate very differently so That's one thing.