Next Unicorns: Unlocking the power of photonic computing with Lightmatter CEO Nick Harris | E1787

0:00I think what you're going to see from Light Matter over the next year and a half is extreme volume on groundbreaking optical products. And, you know, what I'm really aiming to do with Light Matter ultimately is to build the photonics company. So if you think of the word photonics, I want you to think of Light Matter ultimately. And the way that's going to happen is through the whole set of technologies that what we've developed can kind of be realized. This Week in Startups is brought to you by Carta now lets you launch and administer SPVs for your syndicate. Share your knowledge, capital, and network to launch your syndicate SPVs through Carta.

0:36Get 10 % off your first SPV at carta.com slash twist with promo code twist. Eight Sleep. Good sleep is the ultimate game changer. Now you can add the pod cover to any mattress. Go to eightsleep.com slash twist to check out the pod cover and get$150 off a checkout. And LinkedIn Jobs. A business is only as strong as its people and every hire matters. post your first job for free at linkedin.com slash unicorn all right listen everybody uh ai is changing everything a big part of that is what's been made possible by gpus tpus all of this is powered by electric chips basically right that's a standard for decades we took sand we made it into chips and we changed the world but electric signals we all know that produces a bunch of heat requires cooling systems but there's something called photonic computing and our guest today is at the forefront of that and his name is nick harris and the company's name is light matter nick welcome to the program hey good to meet you jason and thanks for having us on the show yeah so tell me what is photonic computing because i am a neophyte in this i'm aware of it But I don't really know the state of it and if it's actually competitive and when this will actually be reaching people's desktops or servers.

2:03Yeah, so maybe I'll start with a little bit of a story to give context of it. So for the past 60 plus years, computers have done all calculations and communications generally using electrical signals. There were a bunch of incredible physics that were enabling those transistors, the fundamental component doing a lot of this computation and communication that we all use today, enabling it to get better, faster, more energy efficient, smaller, cheaper, all these good things. And that was kind of described by this idea of Denard scaling, which is that energy usage per device would shrink with time, and also Moore's law, which was that we'd get more transistors per unit area as time progressed.

2:46So those were the two things driving all that progress. In around 2005, that trend broke. And a lot of people in the physics community, you know, at universities, but also at the big companies doing the work developing these fab processes realized that there was a fundamental challenge. I was working at Micron at the time when that wall was hit, and you could feel it. And I love computers. I don't know why. I've just always been obsessed with them. And for me, it was, wow, this thing that I love is kind of coming to an end. We're at this wall. How can we make it better? There are a lot of different ways that people have looked at trying to extend the computing roadmap.

3:27The way that I've looked at and the way that others have as well is using light. Now, there's two ways that you could use it. One would be to do the computation. And that's the oddball, really new thing. The other is to do communication where you're sending data between points. You know, a lot of the undersea cabling in the ocean, all of it actually is optical. In data centers today, if you go into the data center, there's optical cables everywhere. They're using light to communicate signals. The new thing that's happening in the communication space, which we're working on at Light Matter, is that people are sending data between chips that are actually within an inch of each other using light and between racks and all of these scales.

4:09But what's fundamentally happened is that these optical engines are getting closer and closer to the processors that we use. We're finding that there's a benefit actually to bringing them all the way into the chip itself. Wow. So if you were going to put a bunch of servers across the ocean from each other or across the data center, of course, you'd use fiber optics, you can move a ton of and you see that, right some computers have ethernet plugs on the back other ones have literal you literally you can plug into the back of a server an optical cable where yes all the data coming out comes out by light and there are other constraints like hard drives which of course have become ssds so those are faster but what you're saying here is the next step is hey between the chips we're sending the data over optical computing.

5:00That's right. So the distance that people are using optics over is getting smaller and smaller and smaller, and it's starting to make its way inside the processor. So we've actually developed two kinds of technology at Light Matter. One is actually doing the computation behind deep learning, using tiny little optical components that measure microns by microns. And it does the additions and the multiplications that describe neural network mathematics, deep learning, think about large language models, it does that math using light. That's a huge breakthrough, new field, something that we've pioneered here.

5:34The other thing we're doing is we realize that these optical compute engines we built are very fast. We need to find a way to keep them busy. One of the big pieces of feedback that I've gotten from diligence groups, famous people in the field, when they've looked at us is like, oh, it's cool. You can build that. How are you going to keep it busy? So we took that to heart and we invented our product line passage. And what it does is it allows you to feed the beast. It allows you to send bits at data rates high enough, both between the cores and memory, but also to other servers so that you can actually keep these things busy and not just sitting there waiting for work.

6:10And that is one of the keys right now as part of this massive compute intensive language model building is that we're actually underutilizing a lot of the hardware, correct? Oh, yeah, it's embarrassingly underutilized. I've heard numbers as low as 5%. So you'll have a huge number of theoretical maximum operations per second, and only 5 % of them are being used. And the reason is that if you look at a profile of these jobs, let's say a chat GPT type workload, most of it is spent shipping data between memory and the processor core or communicating between cores in the same box or distant boxes. It's not doing computation.

6:50Yeah. So just moving the data to the point at which the computation can occur is the bottleneck. And you're trying to solve for that. So have your products hit the streets yet, as it were? Or is it still theoretical? Where are you at in terms of these things being in the field? Yeah. So a little context on the company. We have about 130 employees offices in Boston and Mountain View. We've raised about$300 million from investors, including Sequoia, Spark Capital, Fidelity, Matrix, Google Ventures. So we've really got a bench of all the top VCs in the world. We have built many generations of chips, about three or four at this point.

7:32We have six customers on our compute product line, and we're going to be launching our interconnect product line at high volume next year. Got it. So pretty far along. And who are the customers for your products? Is it the open AIs of the world, the cloud computing platforms, or is it deeper in the stack, people who are building the servers and who are building the platforms? It's cloud infra. So we're building heavy-duty hardware for running these huge AI models. So it's these big clouds. And so are you then being put up against, say, NVIDIA in these cloud offerings? And how do you compete against them today?

8:13So on the compute side, yes, we would be compared against NVIDIA. On the interconnect side, not really. So on the interconnect side, we actually enable all companies. So you can think about companies like AMD, Intel, NVIDIA, and even the internal product teams for TPU, Amazon builds their own chips, and so on. They can all use our interconnect technology to help extend what they've got as far as it could possibly go. and those are generally chips that would be deployed in a data center the way that we work with them is they have a chip an accelerator or a network chip or fbga and they want to scale it out using our optical interconnect we figure out basically what the floor plan of their chip is where are the pins and then we build a passage which is a very up to a full wafer format silicon photonics chip with waveguides that interlink everything on top think about it like a chess board with light connecting all the chess pieces where the chess pieces are chips.

9:12Wow. So we basically lay their chips out in arrays. They can be processors, memories, any type of chip, and allow them to scale the solution. So in that way, we're kind of helping everybody. And what impact, if this is fully deployed to the cloud, what impact is this going to have on what we've already seen in terms of language models being built and the number of or the amount of data being processed by them. Yeah, there's an interesting story here. So we're kind of bound by power usage and cooling at this point, not light matter, but the world in terms of data center deployments. When I started the company, chips were about 300 watts.

9:55If you looked at a GPU, maybe 250, 300 watts today, they're over a kilowatt. The next gen chips are about one and a half kilowatts. So things are accelerating really fast. You could even say they're heating up. things are heating up. Yes. I mean, that is, that is the key, right? Is that this creates a bit of a heat issue? No? Absolutely. And if you look at that heat issue, part of it is that the interconnect bandwidths, how many bits per second are leaving these chips is growing very, very fast. It used to be maybe 100 gigabit per second. And the latest chips will push out seven terabits per second.

10:30So we're seeing a big acceleration in the number of bits that are shipped. The problem is when you ship those bits over electrical wires, it ends up being the case that a server, if you do a power profile, more than half of all the power used is just moving the bits between the chips, not even on the chip, just between the chips. And so if you look at what we're doing with Passage, it's about 10 times better energy efficiency. So 18 picajoules per bit for the typical type of inside the box communication that you would see with an NVIDIA server or an AMD server, take that down to two picojoules per bit.

11:05And I know this sounds quite technical picojoules per bit, but the point is 18 divided by two. It's a big difference. What is a picojoule? 10 to the minus 12 joules. Joule is a unit of energy. Right. So you can massively reduce the amount of energy, which then gets rid of the heat and the cooling. And then you can put more compute in the rack. Right. And all that real estate that you build out is now more useful. If you didn't want to add more compute, the power bill went down. So that's nice. And by the way, the power bill is about 50 % of the cost for running the data center. Listen, if you're in the tech industry, you know about Carta.

11:42Carta is the leading venture capital and equity management platform. And they have huge news to share here on This Week in Startups. Carta now lets you syndicate an SPV. You know what an SPV is, a special purpose vehicle. So you create an SPV on Carta. Why would you do that? hey, listen, you're an angel investor, and you're putting 25k in a company like I did with calm.com. But you got about 20 friends who also want to put in five or 10k. Now you put them all into an SPV, you tell the team over at calm.com or whatever company you're investing in, it's going to be one line item, I'll sign for all 25 of those angels.

12:16And they say, Oh, great. Can I put 10 other angels in your SPV? And then you hey, if you want to take carry on it, because you syndicated the deal, great. Now you got a business model going, huh? They are used by more than 4500 funds representing over$120 billion in assets under administration, they're going to support you at every stage of your fundraising journey. From doing your first syndicate to building a global venture capital firm, you can raise and deploy from anywhere in the world because Carta offers US and international SPVs. Also, Carta provides an automated back-off solution for you so you can focus on what matters, finding great startups, building relationships, and supporting the heck out of those founders.

12:56Here's your call to action. Go to carta.com slash twist and use the code twist to get 10 % off your first SPV. What a deal. carta c a r t a.com slash t w i s t. Make sure you use the promo code twist for 10 % off. Do you think we can catch up to the need? Because part of the issue here is we went from, let's say, millions of enterprises having no AI strategy and saying, yeah you know we're going to get to that at some point to in one year half of those you know million saying this is the most important thing in the building and the other half are kind of realizing and i'm sure by the end of the year they're going to say hey this is the most important thing we need to get catch up on so we're in catch-up mode right now clearly do you think we catch up or do we do we think that all of this um energy by corporations not actual energy but all of this intensity that they're putting towards this is um we're going to be able to between nvidia intel yourselves and everybody else working on this we're going to be able to catch up or do you think that this is just going to spiral where once organizations get a taste of what ai can do they're just going to be doubling and tripling down so are we going to catch up and what's the window of catch up?

14:16Is it a year or is it 10? It's going to grow like crazy. We're not going to catch up. You know, I think it's going to be a case of trying to build out enough ships, you know, companies like TSMC adding more capacity, Intel catching up on process leadership, and I think they will. It's going to take... You actually think Intel is going to catch up? Yeah. I think they'll do quite well. I actually think there's an end game for process technology for semiconductors. There is a final node. I'll put it that way. And someone will reach it first, but then everyone will get there. That's sort of my view on things, having seen the physics behind it and what it takes.

14:51yeah so i think that the demand is insane uh if you look at companies like nvidia they have more demand than they can possibly build out which gives them great pricing power and these sorts of things and the whole world doesn't like it when someone has great pricing power and so there's an opportunity for other companies to come in and bring a solution um that does seem to be the truism here which is it's very rare that everybody wants the same thing in the world but taylor swift and H100 seem to be the two, Dweller Swift tickets and H100 seem to be the two items that are that much in demand. And there's only so much Taylor Swift to go around, but NVIDIA has competitors, right?

15:31And they're going to catch up. They're going to have competing products and that will bring these prices back down, yeah? Yeah, I think that that's possible that it will happen. There's another thing to consider, which is that you can use the AI tools to build your AI chips. One of the big powerful things behind Moore's law was that you use the chips that you just made to make the chips that you're about to make. And that's something that you're seeing NVIDIA do, and hopefully the whole industry do. Google's done good work on this. Our VP of engineering led TPU at Google, and they had a paper in Nature a couple of years ago where they were using an AI to actually design the floor plan of the chip and lay out the circuits.

16:08And you can get big advancements in this. So it's going to be a big nonlinear feedback loop. you're going to see these tools being used to design themselves. Yeah. And at the speed these things are going, my Lord, that could be quite impressive. Yeah. Yeah. I mean, I have to tell you every year, one of these results like ChatGPT comes out and it's mind blowing. It makes you question like what the future is going to be like. And if that happens every year, I mean, it's hard to imagine where this ultimately goes. But I can tell you we're going to need to ship a lot more bits per second. The AI models are going to need to get bigger, you're going to need more compute hardware, you know, and you're going to see exotic things, you're going to see water cooling is standard for computer chips today.

16:55So we run water over the top of the chip to pull heat out. The next thing that people are doing, and in China, this is actually quite common, is immersion cooling, where they'll take the chips and actually put them in a in a liquid that boils. When liquids boil, their heat capacity increases. So we're able to pull out more heat at that phase transition. So it's a very interesting technology. I don't know what happens after that maybe data centers in space. Well, it's pretty cold up there. And yeah, that would, and it wouldn't, it doesn't cost you any energy, except to get up there and doesn't cost you an energy to cool it up there.

17:33But you need to radiate the heat because there's no air blowing, there are no molecules bouncing into the the chips to cool them down or the heat sink. So you have to build a special kind of radiator i think that's an interesting physics problem uh but it's something to disperse the heat off the chip because you can't blow air onto it you need to shine the heat away you need to shine photons away fascinating to think about data centers in space the pros and the cons obviously all the energy to get it up there and to maintain it up there but then you do gain a lot by having it out there i guess in terms of maybe you'd have to look at serviceability like the lifetime of the chips There are all sorts of alien things that happen to silicon.

18:12I hear they grow dendrites when they're in outer space, like silicon chips will grow these dendrites and ultimately the packages will fail. Oh, wow. It's crazy. It's crazy. But I guess the thing to look at, and this should scare you a little bit. We're at sort of the heat death with computers. With Denard scaling being broken, Denard invented DRAM at IBM and he also predicted. that theory if i'm correct is hey no matter what we do to these chips in terms of you know morse law or doubling the capacity of it whatever it's just going to be the same amount of energies being used is that like the basic point is density goes up if you happen to find a way to fit more transistors in the same unit area the amount more that you added the heat's going to go up by the same amount and this this is the real issue today computer chips have the same energy density as a nuclear reactor.

19:06Hopefully that like sets the right picture here. So ultimately what happens, there is a death spiral to the heat problem. Silicon, the resistivity, the thermal resistivity, and we're going into physics land here, it's nonlinear with temperature. The hotter it gets, the resistance changes and you generate more heat and it goes into runaway and sort of lights itself on fire. We're already close to this with chips coming out that are one and a half kilowatts. were pretty close. There's a limit to... It's just going to go up in flames and there's no way to cool it, I guess. So how do you fix that?

19:41The way that you fix that is, okay, stop scaling the chips. Like don't change the performance per chip. Just build a huge supercomputer. It's going to be very expensive. All of those like maker movements and things that were created by electronics becoming cheap, they die. Now you build gigantic supercomputers that will continue the gauntlet of performance that's it so you only have a few companies that can build them and own and maintain them and the progress is fascinating because we went you're referring to like arduino chips and like oh yeah we're going to get chips down to a dollar two dollars and we'll just make a ton of them but that doesn't work for these jobs that we want to do these jobs require much more compute therefore yeah we are going to have we're going to live in a world of supercomputers again and we're going to be time rent we're going to be renting time on a supercomputer we're literally back to the age of vaxes and mainframes you got back to perot computers where you're going to be renting time on some giant supercomputer and the these computers are going to cost how much to to build and maintain do you think in the next five they're billions of dollars if you look at uh you know anthropic and all these other uh ai startups that are getting funded like crazy to build llms flawed uh you know from them uh they're they're raising billions and it's all going straight to nvidia which is amazing for jensen and nvidia but but it's also kind of scary i mean uh if you think about that future so that's why at light matter we've looked at the compute problem too because fundamentally you can't just let the chips go on this singularity spiral of heat death you have to find new ways for computers to make progress and so for us it's both the compute piece and the interconnect you have to innovate on both to keep it going.

21:26I also suppose the software needs to be optimized massively. It seems like we are so far behind on the optimization of these LLMs. And then a lot of these jobs are just getting, they're just throwing hardware at the problem. Yeah. Oh, a hundred percent. There's a big, there's a big improvement to be had through software innovations, but just remember in 1960, you could have done software innovations. So you do need to do that, but the hardware also has to progress. That's the tricky thing. If you want to get ahead in your career, then you got to be ready to fight every day. You got to grind it out day after day.

22:05This is obvious. You're listening to This Week in Startups. You're grinding right now. But if you're a workhorse, you need to get a good night's sleep, period, full stop. We all know that getting great sleep equals better performance the next day. Sleep is a secret weapon. And your secret weapon to getting great sleep is eight sleep. You know, I love my eight sleep mattress. I use it every night. I got it in both houses, because I want to be able to get control my temperature. And I want to make sure I optimize my sleep. And you can do this with their new product, the pod cover by eight sleep fits on any bed, just like a fitted sheet, going to keep you cool all night, all the way down to 55 degrees Fahrenheit, if you like, and it will improve your sleep by automatically adjusting the temperature on each side of the bed and you get personalized sleep reports.

22:48If you're one of those quantified self people, you're going to love this feature. And when you get great sleep, you get great performance. And listen, everybody's been talking about it publicly on twitter.com. I'm sorry, x.com. But no, coast, the Paul, Graham, Zuck, myself, everybody's been talking about how great eight sleep is. Invest in the rest that you deserve by going to eight sleep.com slash twist. They're going to give you 150 bucks off the pod cover, because you're friends with this podcast. That's E-I-G-H-T-S-L-E-E-P.com slash twist for$150 off the pod cover. And they're shipping, not just in the US and Canada, but they got the UK and select countries in the EU and Australia.

23:30Knowing what you know, it's always fun to speculate about, you know, general AI and And it feels like one of the interesting tricks that ChatGPT has taught us is just how simple our brains are. Next word prediction? Yeah, it's kind of like, you know what I'm about to say. Oh, I just had this conversation with a friend. You know, when you're talking to someone you're close with, you kind of know what they're about to say. They don't even really have to. It gives you this hint that maybe there's something to that, that this next word prediction is actually fundamental. it is in terms of us speaking and us consuming the data and so we may have just figured out something very basic about how the brain works and then it does feel like we're getting to the point where yeah we forget about what test we run to figure out if it's general ai but it doesn't feel like there's much a human can do that we're not going to accomplish with this latest volley, where do you sit on when we might accomplish a general AI?

24:39That's a tricky one. I think the one that is most tractable, the way to approach this that's most tractable to me is just parameter counts. If you're looking at trillions of parameters, we're almost there already. So just from that kind of standard, let's say two or three years. practically there are going to need to be algorithmic improvements and theoretical improvements to get this done you can't schedule those unfortunately you can lock theorists in a room with a schedule and they'll laugh but let's see you know you know maybe hardware wise we could be ready for it in the next three or four years but i'll put my bet on 2030 feels like a good number to a good round number and you will have a tipping point it feels like when we look at vertical AIs, something like self-driving, who certainly watched it in games of skill and video games.

25:34And we see it with words and just being able to write and communicate that we're kind of there, right? And it feels like watching what Elon and some other folks are doing with self-driving, man, they seem like they're really close. It feels like this last AI push is going to solve it, yeah? it does feel like you're really close but remember you're really close and you're up against the wall of physics of this heat problem so you're going to see some new technologies come to bear and we'll be one of them that will allow that to happen but we're very at a we're at a scary line we've pushed the limits of everything so whether they get to that without new technologies is is the big question yeah and quantum computing in relation to all of this yeah i did my doctor at MIT on quantum computing.

26:24And I love the field. I think that it's like sort of an incredible Olympics for your brain. Great exercise. You're being in incredible shape, but maybe not super practical yet. In another 10 years, I think there'll be enough progress on the theoretical and experimental side to kind of scale quantum computers. But right now it's going to take companies like Google and IBM just pouring cash into the program and believing in the vision of what it ultimately could be to get there. It will be a useful tool, but it's not going to be super broadly applicable. And that's the thing that I think people don't understand.

26:59It's not like our savior. You need GPUs and things like GPUs to keep marching along. The quantum computer is not going to run your deep neural nets faster. And by the way, and this is very controversial, and I think I'll probably get in some trouble with my friends from grad school. But, you know, it's not clear to me that with deep learning, you need quantum computing. There are a lot of problems. You look at AlphaFold and some of these molecular challenges that people are looking at. It seems that deep learning is able to traverse the problem space in a way that reduces the amount of compute you need by huge orders.

27:37And so the question is, do you even need quantum computers at all? Maybe a sufficiently complex deep neural network could actually just do that for you. That would make sense, I think, and then let alone some optimization on the software driven by AI. So with AI writing the software, there's going to be massive gains there as well. I mean, that's the part that I think people are underestimating is exactly the role of AI in solving some of the problems around the ceilings in AI. Certainly, you're using AI to design some of these optical solutions. Yeah, definitely. When we design optical components, that is using AI algorithms.

28:19We'll use things that look like backpropagation to decide the shape of these optical components that are built out of silicon that the light travels through. So it's a big, big tool in the toolbox. I think there's a lot of work in ship design, computer-aided design for ASICs that can improve energy efficiency, the size of the circuits and things like this. And we're looking into it. You know, one thing that sort of came to mind when we were talking about where AI is going to go, there's a heat limit, but there's another one, which is a sort of operating cost expense limit for companies. Look at how much they're spending on these supercomputers.

28:57It's billions and billions of dollars. If you double that a few more times, it's going to really make a big dent in Google's bottom line, right? There's, yeah, I mean, billions of dollars are speeding tickets. Tens of billions are considered purchases. And that's kind of as far as you can take it. Hey, speaking of like scientific breakthroughs, every nerd in the world is over the last four days watching this LK99 room temperature supercomputer. I'm sure by the smile on your face, you've been on Twitter or I'm sorry, X.com. X.com. On X.com, watching people try to recreate this paper. Yes. I went straight to the prediction markets where people are placing bets because gamblers are so good at handicapping stuff.

29:42And they had it at 25 % replicable in the next two years. What do you think's going on here? and how awesome is it that it seems like a dozen teams that are in some way credible decided it's august whatever it's the summer let's try to replicate this for the of it at night i i love that i i feel like it's a new a new kind of um scientific process someone posts a paper papers have gotten to the level where people actually read them non-scientists non like deep academics are reading them and they're trying to reproduce the results that's actually an exceptional model. There aren't many things that you'd be able to go after this with, but this is very cool.