Etched is ready to disrupt the AI chip market, with Gavin Uberti

0:00Starting a business comes with its share of ups and downs, which is why staying true to your vision is essential. a non-negotiable for Romeo and Milka Bregali, Capital One business customers and co-owners of Ra's plant-based restaurant in New York. Romeo and Milka took a leap of faith when starting their own restaurant, gutting an empty space and building it from the ground up. Every pipe, every wall, every detail. But building from scratch came with a heavy financial burden, which is when they turned to their Capital One business card. With the flexibility of the card's no preset spending limit, they were able to spend more and earn more rewards while bringing their vision to life.

0:36Today, Raz's success is proof that with passion and the right support, it's possible to make your dreams a reality. Learn more at CapitalOne.com slash business cards.

0:50As we continue to see advances in AI models, whether it's DeepSeek or the latest Grok model, we're also seeing an increase in the demand for compute. As a result, the race to create the winning chips that can power cutting-edge AI innovations shows no sign of slowing down. At the forefront of this race is of course NVIDIA, which has over 80 % market share when it comes to AI chips. But the world of AI chips is ripe for disruption. And there are many other players in this world who are not making headlines every day. Gavin Oberti is carving out a space in the market with his company Etched and its specialized chips that are optimized for transformer models, the de facto AI architecture for most AI models we see today.

1:42And Gavin's very aware of who his competitors are and what he's up against. Well, our competitors are very good at what they do. I think a lot of companies have tried to go do what NVIDIA does, and they've turned out to be worse. There have been other AI chip companies who have tried to build flexible AI chip products that have, again, been able to be cheaper than NVIDIA products, but not better. So if you wanted to beat NVIDIA, you can't play the game the way they're playing it. I think specialization is one of the few solutions that can let you get an advantage. We are not going to beat NVIDIA at almost everything they do.

2:23But at one. transformers. We will dominate.

2:31In this episode, we'll talk about what domination could look like, the big bet Gavin is making on transformer models, and how AI chips could help lay the groundwork to making AI accessible to as many people as possible. I'm Rana El-Khalyubi, and this is Pioneers of AI. a podcast taking you behind the scenes of the AI revolution.

3:09Before Gavin and I dove into our conversation about chips and transformer models, I wanted to learn more about his story leading up to the founding of Etched. Following the legacy of tech founders like Bill Gates and Steve Jobs, Gavin also dropped out of college. And I had some feelings about that.

3:34Well, I have a confession to make. A confession? A confession, yes. So my daughter is a senior at Harvard. And in preparation for this interview, I was kind of simulating the scenario where she comes to me and she was like, mom, like I'm dropping out to do my own thing. And I was pretty uncomfortable. Like, I don't know if I was totally on board with that idea. You, of course, dropped out of Harvard to start Etched. And I have to say, I am totally on board with that idea because I'm a proud investor in the company. But I've never actually heard your origin story. How did that whole thing unfold while you were at Harvard?

4:14What were your parents' reactions or other people important to you in your life? Like, tell us more. Well, I actually did not drop out to start Etched. I just dropped out. So I was a world champion for mathematics in high school, went to Harvard to study mathematics, loved it there. Unfortunately, the world is very big and most of that's not at Harvard. So around a semester and a half or a year and a half into college, I got the idea of, hey, I'm going to drop out. I'm going to live as a digital nomad. I'm going to travel all around South America and Southeast Asia. For the year, I was on a flight about once every four days.

4:50That is crazy. And what was your parents' response? I paid for it myself, so I didn't really care. But I actually paid for this, which is not cheap. I got a job doing microkernel development. Microkernel development. Basically, it's developing software at the operating system level. Even back in 2022, there was a lot of demand to go run AI models very efficiently. And since AI models are powered by matrix multiplication and convolution operations, somebody has to go write the code that runs these very efficiently. So I got a job doing this for our microprocessors. And the way that you do it, the way that you get every last little bit of performance out of these programs, is that you write them not in a programming language, but in assembly.

5:37Oh my God, I remember assembly. I studied that when I was a computer science undergrad. at. It's basically the low-level language of computers. It's the opposite of prompt engineering. So I got to know the Cortex M4 and M7 very well. They are two of the cores made by ARM that are on some of the cheapest devices that you can buy. Cheap, but Gavin found ways to get more performance out of them. He then traveled the world and worked on optimizing machine learning code just months before AI development began to ramp up due to generative AI. And during his travels, he noticed a gap. I love traveling, but the thing you really realize when you write this assembly code is that only a very tiny part of each one of these chips is used for doing the math.

6:28Most of the job that you do is not program the math blocks. It is get the data to the right place at the right time. That is much, much more challenging than the math operations themselves. In fact, on a chip like the H100 from NVIDIA, only around 3 % of the chip is spent on matrix multiplication blocks. Meaning that the H100s are highly flexible chips that can be used for AI and non-AI functions, which had Gavin thinking, could a more specialized chip dedicated to AI be more energy efficient and faster than a graphics processing unit? Okay, so you're doing all that and you have this realization.

7:09Then what happens? Well, at the same time, I become convinced that we are going to see a little bit of a plateau of AI architectures. If you think back many years ago, longer than I've been alive, to CPU instruction sets, there used to be many different kinds of instruction sets. But x86, the one that powers most Intel processors, sort of reached this one critical point of popularity where it had enough adoption. That software got written for x86, which made it get more adoption, which meant there was more software, and led to this cycle where x86 became the dominant instruction set. And I thought the same thing was going to happen for Transformers.

7:53They were pretty good in 2022. to, there is not really a killer product yet. So I thought they'd get better, more hardware, more software support, better products, and it becomes the self-reinforcing cycle. Okay. So I want to pause you here. So transformer models are basically a specific type of AI architecture or an approach to building these AI models where it really leverages sequential data. And what is so transformative, no pun intended, about these transformer models is that they are able to hold a lot of information in memory and process all of this information in parallel. Transformer models are now at the core of ChatGPT and a lot of the other generative AI products, actually also multimodal, right?

8:40They power some of the vision and video processing applications as well. The reason why Transformers won, I think, is not just because they're very performant, but because they run really well on hardware. So if you want to go build a new model, that's a transformer, it's super easy. The software is there. The hardware supports it really well. If you want to do it for something else, you're on your own. You have to go write your own software. You have to go, well, accept the fact that hardware is not going to really support it all that well. And I think that means that transformers are going to give better research results, which means there'll be more investment in them, which means even better research results, which leads to this cycle of transformers staying winning.

9:23So the moat is basically the fact that it has become almost the de facto way of building machine learning models? I mean, the moat is the fact that there's a great software ecosystem, there's a great hardware ecosystem, including us, and people know how to use transformers. And the last piece here, too, is that models are getting really expensive. Rocking the boat, trying new things when they cost$100K each, that's the thing people can do. Rocking the boat at the$100 million or billion dollar scale, that's much more challenging. So I think that as models get more expensive, you can't really risk it to move away.

10:03We recorded this interview before the DeepSeek news broke. DeepSeek, among many other things, proved that it is possible to train cheaper models. It also showed that at the moment, the cost of inference is still high. Since DeepSeq's release, Etched has said that they've actually seen a lot more inbound interest from companies wanting to partner with them. Essentially, for AI usage to be commercially viable, companies need to reduce the cost of inference. Something that Etched can do faster and more cheaply than GPUs. so you clearly made the bet on transformers and you've basically kind of decided to build a specialized chip that doubles down on these models and is optimized for these models so i'll just take a step back to make sure that everybody's kind of coming along with us on this journey an ai chip is essentially a computer chip very much like the computer chip that powers your smartphone, but it's specialized for AI tasks and machine learning tasks.

11:07For example, processing large amounts of data or processing images and videos, like you said, but also generating images and video. They tend to be optimized for these kind of complex mathematical calculations that allow us to implement these kind of approaches to AI. NVIDIA is an example of an AI chip, like the NVIDIA H100s, and that's an example of a chip that is optimized for AI. But you're even doubling down even more and building a very specialized chip. We call that ASIC, or Application-Specific Integrated Circuit, that is designed to specifically solve for transformer models. Talk us through what that means.

11:48Well, it means that we can't run most kinds of AI models out there today, which is very scary. that the snapchat filters that power the dog ears or whatever those aren't transformers those are convolutional models or the recommendation engines that power ads for google and facebook are not transformers but the text models like chat gpt and the image generators and the video generators those are transformers unlike a gpu or a google tpu our chip cannot really be programmed in the same way, it is by design only able to run transformers. But in exchange for that, it can have much more compute on that same die area.

12:33And as a result, it can get much better performance. 500 ,000 tokens per second for a model like Llama 70, dramatically outperforming NVIDIA H100s and B100s. To what extent? How much more improvement in performance are you getting? Well, I think the right way to compare, based on the numbers NVIDIA gives, they have this page on their T30LM repo that says how many tokens per second do you get for Llama 7DB running on 8 H100s in FTA Precision with 2048 input tokens and 120 output tokens. And you get around 30 ,000 tokens per second for this H-Chip system. Now, that's not 30 ,000 for one user. It's 30 ,000 across a number of different users.

13:21Before a chip like ours, we will get more than 500 ,000. Meaning that these chips are faster and have better performance compared to GPUs. After a short break, we get into what this actually means and how it gives Etched a competitive edge in the market. Stay with us.

13:46If you've spent any time building AI products or leading technical teams, you know this. Transformation doesn't fail because of ideas. It fails because teams can't move together. Enter Atlassian's teamwork collection. It has planning in Jira, documentation in Confluence, video updates in Loom, and now AI agents in Rovo, which connects the dots across your work so nothing gets lost. It's one AI-powered teamwork platform designed for how modern teams actually build. Learn more at Atlassian.com slash TeamChanger. That's A-T-L-A-S-S-I-A-N dot com slash TeamChanger. So your first product is Sohu. Describe what that product does and what are the key kind of advantages of Sohu over other chips out there.

14:44Sohu is a transformer machine. It has way more compute than competing products. And as a result, it can do things like give that 500 ,000 tokens per second 16 times faster than 8 H100s, some of the fastest AI chips on the market today. Additionally, you can get much better latencies too So there are a few kinds of latency But I'll be precise I'm talking about the latency between When you ask chat GPT your question And when you get the first token of your response back That is the time to first token latency Four traditional endpoints Usually see around 300 milliseconds That's good But too slow for a lot of use cases You can't do things like real-time video generation or real-time audio generation with that kind of delay.

15:32NVIDIA's TRTLM page says that they can get Llama 7DB running at FP16 precision for a length of 2 ,000 input prompt. They can get that time-to-first token in about 100 milliseconds. With eight Sohu chips, it will be six, a factor of 20 times faster than our competitors' products. Now, again, that's not because we're clever. That's because all SoWhoo does is run transformer models. And it's not just text. We see similar speedups on video generation, image generation, image processing, or multimodal models. Speedup, or lower latency, is key. Of course, everyone wants an AI chatbot that responds faster.

16:18But there are other reasons why latency is so critical. Think about an AI co-pilot in your car that's sending data back and forth to AI models. Lower latency is so important here, especially if the car co-pilot needs to make a life-saving decision. But for Etched to implement this full vision, it needed to be more than just a chip company. Our first product is a data center product. We're competing with NVIDIA's box, which is a data center box as well. That's where most of the market is. Yeah. Talk a little bit more about the first kind of customers in this data center. I saw somewhere that you described Etched as an AI infrastructure company as opposed to an AI chip company, and I thought that was really intriguing.

17:01But I think you're getting to that in your answer around data centers. Well, one key thing is Etched does not sell chips. Etched sells solutions. Now, why do we do that? Is it because they're doing harder things? No. It is because I want to get these products in customer hands as fast as possible. And that means you have to build the chip and build the circuit board that chip sits on and build the server the circuit board goes into and build a little bit of the rack that sits around that to make sure it's cooled correctly and powered correctly. That's harder, but it's almost as important as the chip itself.

17:38When I say that we are more than an AI chip company, I mean that we sell systems. And that's where the data center comes in, right? So if I'm an enterprise customer that is building all these AI models, I'm not just going to buy the chip directly from you. I'm going to buy an entire kind of solution, which includes the servers. And yeah, tell us more. Like, what would that look like? What would the solution look like? I'll underwrite a little bit. But we saw similarly to how other people like NVIDIA or AMD or Intel do. They sell at a couple of different levels. There is what they call L11. They sell you a whole server rack.

18:17It has the servers, the cooling solution, and the top rack switch all put together. Or you can buy L10, or you get just the server. And some very technical customers buy AMD and NVIDIA products at the L6 level, or you get the circuit board, the chip, the circuit board, and the base board all together. Yeah. So I'm kind of thinking, what is the so what here? Like, why is this so exciting and really disruptive for the AI market? Is this going to help us adopt and deploy AI faster? Is it going to help us build these models quicker? Like, yeah, why does this matter? The future is here. It is just not evenly distributed.

19:01That the vast majority of people have never experienced GPT-4 level models. And when GPT-5 and GPT-6 and other next-gen models come out, they are going to be very expensive. What do you mean by expensive, by the way? Define expensive. I mean, they will cost a lot of money for a token. That as models get bigger, it costs more money to run them. And that's why for a long time, even still today, you can only get the premium GPT-4 model if you pay for the checked GPT-plus subscription. You can get smaller distilled versions like GPT-4-0 and GPT-4-turbo for the free version. But the big ones, those are behind paywalls today.

19:46And I think that specialized chips, both those made by us and others, are going to help decrease these costs by an order of magnitude and really democratize access to these huge foundation models. And while today it's not a huge issue, it is going to become a big problem in the future. If next-gen models are 100x bigger, they might be 100x more expensive. And we see solutions now with agents as well that are very performant but very expensive. One of the big coding agent companies today charges more than$10 ,000 per month just to go cover their own costs of how much it costs to run it with these tokens.

20:27And that is just unaffordable for almost everyone besides the big tech companies. For this to really be an invention for the people, the cost has to come down. Now, that said, I'm very confident it will. Many other inventions start a very expensive. Cell phones, for example. And then once purpose-built hardware came along, they got way, way cheaper. Yeah, but I love this vision of democratizing access to AI. And I love that you're optimizing on the training side of these models, but more importantly, on the inference side. So every time I make a call to a chat GPT, it's not costing, you know, it's not costing a lot of money to do this call, but also costing the planet and costing, you know, costing the environment, too.

21:17That's a big consideration as well. But how can we democratize access to AI responsibly? That's after the break. Stay with us.

21:41Meet Nicole Nicholas, Capital One business customer and co-owner of Ansett Uncles, a plant-based restaurant and community space in Brooklyn, New York, that got its start from a need for unity. The inspiration, it was born from the desire to create a space that felt like home, where we can connect community culture, good food, and come together with family and friends. That's how we birthed aunts and uncles. Nicole and her husband, Mike, were fulfilling their dream of bringing people together out of their home kitchen. But they soon learned that the demand for community was greater than they knew.

22:12It became overwhelming and we were like, we need home, but not in our actual home. We realized that there was also a need in our community for something bigger in our neighborhood. So we had to find a place. Moving from a home operation into a storefront was a huge next step. but Nicole and Mike were able to take it on with the help of Capital One Business. It's not for the weak. As a small business, finding resources is super important because that's the way you'll be able to manage and scale. We would have never done that without having Capital One to be able to help us along the way. The cashback rewards are very helpful.

22:48You know, it just gave us that runway to be able to breathe a little bit. Then you get to focus on the cooking of the food and making the experience great. To learn more, go to CapitalOne.com slash business cards. All right. So at this podcast, and also as an investor, I'm very focused on this idea of human-centric AI. AI that's good for the planet, good for people, and is ethical and responsible. How do you think about building AI responsibly? What does that mean for you and for Etched? For me, it means keeping the cost low. that I think we live in a bubble. Silicon Valley is one of the richest places on Earth, and we can afford our$20 chat TPT subscription and a$20 Anthropik subscription and a$30 mid-journey subscription.

23:38And most people in most of the world can't. That look at iPhones, for example. While iPhones are very popular here, iPhones only make about 25 % of market share worldwide because almost nobody else can afford them. So for me, building AI ethically means building products that can be used not just by the select few here in Silicon Valley, but all across the world, building models for not 10 million, but 8 billion people. All right, so you are taking on, you're disrupting the AI chip industry and the AI chip space, including taking on big competitors like an NVIDIA or an ARM or a Google. What is stopping NVIDIA from basically building a specialized chip optimized for transformers?

24:33Eventually, they will. But the fact is, as is common in deep tech, it takes a very long time to get these products to market. It doesn't matter if you have$1 or$100 billion. dollars. There are just things that have to go after other things. So there are long lead times. And we started really early. We began this company before ChatGPT came out, back when it was really risky to bet on transformers, not only because the models weren't as solidified back then, but because there was no use case. And that lead has put us in front and is going to make us first to market. And yes, if we go sit on our butts, then eventually somebody will go build a better specialized ASIC.

25:20But just like you see in CPUs or in networking chips or in Bitcoin mining chips, whoever's in front can usually keep innovating and usually stay ahead like that. You've said that Etched could be the biggest company of all time. And I have to say, I love this energy and conviction from founders. It is one of the qualities I look for when I decide whether or not to invest in a company. But also, let's be real, right? Like the journey of running a startup is, I kind of liken it to an emotional roller coaster. Some days are amazing. And, you know, you've just closed your Series A round and the world looks awesome.

25:57But there's a lot of challenging days as well. What are some of the obstacles or biggest challenges you're facing? Well, the fact is we are not just building a specialized chip. We are building a chip that has more compute than anything else ever built and more not just memory, not memory bandwidth, but just off chip IO bandwidth than any other AI chip ever produced. It leads to a lot of issues on the chip design, on the platform design, on the package design, on the circuit board design that have to be solved. Mm-hmm. So that's a lot of challenges kind of on the R &D perspective, like the actual development.

26:37Yeah, figuring out like how to do it as efficient as possible or as build these chips to be as fast as possible. That we are building a product on the cutting edge of technology. But I have such a fine team working with me. My CTO, my chief architect, and my VPs to hardware and platform are some of the most talented people I've ever met. It is a privilege to come and work with them every day. That is awesome. I love that. So you have announced your Series A$120 million Series A round. How are you using the proceeds and what are some of the milestones you're aiming for? We want to make sure that we are at a point where we're generating revenue, serious, serious revenue, before we think about, do we even need more money after this?