Meta's AI Leap: Unveiling Their Custom Chip for Next-Gen Data Centers

0:00With every major tech company making AI announcements right now, it's no surprise that Facebook is jumping into the game. And today I want to talk about on the podcast the fact that Facebook is going a step beyond what many companies are doing and is actually starting to build AI specific hardware. So just recently they announced a brand new device called the Meta Training and Inference Accelerator. So today on the podcast we're going to talk about what that is, what this means for the industry, and why they might be doing or going this route, right? So essentially the story here is that we have Facebook that traditionally has been a software company.

0:35it's no surprise that they have a lot of technology they have a lot of infrastructure like AI infrastructure that they've built and you know it makes sense they've been doing the their Facebook algorithm for a long time billions of people use their site and so they use AI to help serve up relevant results on their site so of course they have a background in AI and I think that the the value in that is that because they kind of have like strong use cases for AI and always have they have a really solid team and when you have a really solid team they tend to try to come up with new things so recently in their blog post announcing this new piece of hardware they said we found that gpus were not always optimal for running meta specific recommendation workloads at the levels of efficiency required at our scale our solution to this challenge was to design a family of recommendation specific meta training and inference accelerators so apparently they co-designed the first generation and in any case this is a big thing right because right now Now, it would appear that doing something like that, Facebook is really, meta, is really focusing on, you know, selling picks and shovels in this AI gold rush rather than just selling the AI.

1:41And, you know, this comes after Facebook had their, or Google had their leaked memo that said, you know, we don't have a moat and neither does OpenAI. So obviously there's this concept where the software behind this, it's kind of tricky to have a moat. Now, when you're starting to talk about, you know, custom chips and stuff that can run this AI, definitely emote, definitely IP, definitely patents, and something that's very protectable. So it's going to be left to be said whether Meta's decision here to really focus more on the hardware side versus the software side is going to pay off in the end.

2:16Now, that being said, this new chip that they've designed, it's essentially, it's really cool, but, you know, people are saying, like, don't get too excited about this thing, because this thing isn't actually slated to come out till 2025. So we still have a, you know, a year and a half or a couple years before this thing is actually gonna be on the shelves. But when you look at companies like Nvidia, and how you know, their stock price, and you know, everything that's going on with their company right now is absolutely insane. This makes a lot of sense that, you know, other people would be interested in the tech, in the hardware side of this, and not just the software.

2:50And Facebook is actually positioned better than a lot of software companies to do the hardware, ever since their purchase of Oculus, which has now been rebranded. But ever since they purchased the MetaQuest, or the Oculus, changed the name to the MetaQuest, but the VR goggles, they really have been focusing on a lot of hardware now because they have this whole AR, VR hardware arm of their company that is, you know, they're really trying to push to be quite a big thing. They're spending billions of dollars on the metaverse, yada, yada. So what I do think is important and interesting here is, you know, why are they creating this chip?

3:22obviously for their own AI needs. This appears to be something that, you know, was useful and they needed to develop. And so now they're going to obviously be selling this thing. But I think what's also interesting is they said, you know, for our specific AI workloads, blah, blah, blah, we need this. I think it's actually really smart because this comes on the back of Facebook who has open sourced their Lama model and a lot of their other AI or just other tech like Facebook actually has been doing a lot of open source stuff, not just now, but historically, right, with like React Native and all a whole host of different AI tools specifically as well.

4:00And so I think that this is actually really smart of them, well played on their part to release a model like Llama. And then to release, you know, I'm assuming their chips are going to be like hyper optimized for their own models. And so they're, they're putting out these open source models it's going to take time for everyone to really get them integrate with them build with them but there's going to come a point when if you want the you know the best if you want the absolute best experience on their llama model you're probably going to want to get their chip that is running your that is running your ai model and so it's it's honestly a really clever way to go about doing that in my opinion because other people like open ai they're never going to open source obviously never say never I guess they have open source some AI models but like chat GPT is like their big cash cow right now so until something incredible and bigger comes out I don't see them or Facebook relinquishing control of that so Facebook already has given up control of its big model which is better in a lot of ways people are saying it requires a lot less computational power to train stuff on it and in addition to that now they will have this chip that is going to be optimized for that one can assume so i think this is going to be a really big growth for them in addition to that chip they're doing they also introduced a new it's called the asic specifically to help with video transcoding and so it's called the msvp or meta scalable video processor and essentially it's designed to support support both the high quality transcoding needed for VOD, as well as like, just the low latency that you're going to need a faster processing time that live streaming requires.

5:45So obviously, these are tools that are built right into meta, like obviously, people are going to be live streaming right on their platform. And so building tools that increase, you know, the user experience for their own platform is going to be really key and really beneficial for them. And if they already go through all of the R &D research and development to create these new chips why wouldn't they just sell them in the future in addition meta said in a separate blog post to the one announcing this they said in the future it will help bring things like ai made content and ar and vr specific content to meta's apps so obviously um there's you know two sides to these this hardware where they can have it running ai model specifically but also it's going to be helping build out their AR and VR play this is really interesting from Facebook honestly I give them credit they have not abandoned their crusade of AR and VR it definitely has been stale and not highly looked upon by a lot of people who think it's a waste of money but they are still investing in there and I feel like Facebook has kind of taking this whole AI wave to, to like, they've taken this and applied it to that business where they're essentially integrating more things and building things that are good for AI and also for the AR VR play.

7:03And maybe they'll be the winners in that AI AR VR play. As we see that the space is about to get more popular when Apple is set to announce their own AR headset. So it'll be really interesting to see what happens there. I think Meta is also currently working on sort of a next generation data center they announced that is going to be AI optimized and faster and more cost effective to build. And when you think about like why they're doing that, I think that when you look at like the possibilities for in AR and VR with AI specifically, I think it's going to be really incredible to, it's going to be really powerful for them to have like very AI optimized data centers where they can have really powerful AI experiences inside your VR goggles.

7:50You could be talking to someone. And one of the hardest parts right now is kind of the latency between like if you had an AI life coach, for example, that you wanted to like literally talk to, you would speak the AI model, like Whisper would have to listen and take your input. And then it would have to send it over to something like ChatGPT to, um, to like respond to. And then it would have to turn that into audio, which takes a while to process. And then would have to send it back to you. So there's a lot of latency in that. But if we can get to really AI optimized data centers or a Facebook can, they can make processes like that and all sorts of different things really seamless and smooth.

8:25I think that is when you really see an entire new inflection point in AI. It's no longer like a tool that's like a one off use. All of a sudden, it's like a really powerful, interactive, instantaneous tool. And I think that that is, you know, whether that's Amazon's Alexa or Facebook's Google Home Mini or Apple's Siri, that is what they want to achieve. So in whatever way they can accomplish that to remove the latency and absolutely optimize for it, I think that they're going to be pushing products that push that forward. So it'll be interesting to see what Facebook continues to do in the future.

9:00So far, they are absolutely not out of the AI race, and they have entered an entirely new aspect, which is now hardware.