Andrew Bosworth on AI, wearables, and mixed reality

0:01Every person, superhuman vision, superhuman hearing, superhuman memory, superhuman cognition. That is the vision that we have for these wearables. A tremendous equalizing technology who has incredible memory, incredible vision, incredible hearing. Some people are born that just have those talents. The vagaries of the bell curve and the ranges of human capabilities are beyond just the demographic ones. They exist in terms of fundamental capabilities. And how interesting will we be as a society if everyone has the full access to those facilities?

0:38Hi, I'm Reid Hoffman. And I'm Aria Finger. We want to know how together we can use technology like AI to help us shape the best possible future. We ask technologists, ambitious builders, and deep thinkers to help us sketch out the brightest version of the future. And we learn what it'll take to get there. This is possible. It's no surprise that innovations in tech are changing the way we interact with each other. But now we're living in an era where devices and programs are changing the way we interact in the physical world. And they're doing so through advancement in augmented reality, virtual reality, and of course, AI.

1:22The future is already here, and it is quickly redistributing more evenly. And while we talk a lot about software, it's happening increasingly with wearables. namely electronic devices designed to blend seamlessly into our everyday lives, like watches, clothing, and glasses. However, these aren't just fun gadgets, but devices that could refactor how we navigate our world. Our world is soon to be one where glasses don't just help you see, but where they enhance what you see. Now, add AI to the equation. The glasses could potentially guide you, talk to you, and troubleshoot your problems while keeping you rooted in the physical world.

2:02The possibilities and applications in this space continue to grow. One of the key figures shaping the future of this technology is Andrew Boz Bosworth, Meta's chief technology officer and the head of Reality Labs. Boz has spent nearly two decades at the forefront of digital innovation. He joined Facebook back in 2006, where he developed pivotal features like the newsfeed, messenger, and groups. Those functions playing a crucial role in transforming Facebook into the tech giant that it is today. But his influence doesn't stop there. Boz's journey in tech runs deep. from his early days as a teaching fellow for an AI course at Harvard, working on Microsoft Visio, and later founded Meta's AR and VR division, which became Reality Labs.

2:52Today, as Meta builds towards the metaverse, Baz is focused on the next frontier in mixed realities. His latest project, Orion, a next-generation pair of AR glasses still in development, aims to revolutionize how we interact with the digital and physical world while striving to make this groundbreaking technology accessible to a wider audience. In this episode, we'll be diving into the intersections of Boz's career, Meta's ambitious AI hardware projects, and the broader implications of having AI and mixed reality in our day-to-day lives. So let's get into it. Here's our conversation with Boz.

3:35So you were part of the California program where you raised livestock and essentially grew up on a farm, which by the way, I went to a high school on a farm, so I'm very sympathetic. I'm curious, how did that experience influence the person you are today? People are often surprised by the fact that, you know, I grew up on a farm and my family's farming from way back and still is farming to this day. But actually, really, if you know about farmers, three things that are important to know about farmers. Number one is they're governed by time, daylight and seasons. They have X number of daylight hours to which they get their work done and they got to get it done because the seasons are moving forward.

4:12And that forces two other things. The first one is every one of them is an engineer. You got to fix that tractor now because you got to get that crop done. You got to mend that fence today. You don't have time to be dealing with getting these livestock back in the pen. So they're all engineers, not scientists. I mean, real, like get it done with what you have on hand kind of engineers. And the second one is they're entrepreneurs. You know, my family runs a horse ranch. Well, a horse ranch means you got a lot of manure. Well, you got two ways to handle that. The one is you can pay someone to pick that manure up and haul it off.

4:43And that's going to cost you money. The second one is you can market that manure as a fertilizer and you can make a little money. That's a two for one swing. And so my cousins who are still farming and my uncles who are still farming, it's either a cost or an opportunity. then the margins are slim. So you got to make those opportunities work. You know, in my experience, at least, it's not as big a stretch as people thought. But I do want to shout out National 4-H Organization and California State 4-H. I learned how to program in 4-H. It's not just cows and cooking, as we like to say. The first person who taught me to program was a fellow 4-H-er and got me into computers that way.

5:17So it's a great program, not just from an entrepreneurial and engineering standpoint, but also directly programming computers. Was there anything in addition and the kind of farming that also shaped how you think about the interface between the digital and physical worlds? Well, it's interesting. I think the point that humans are busy, they got a lot of things on their plate and the tool either has to work for them or it's just not worth it. I really think that when we think about how we build these tools, a lot of times we're beset by the tremendous value that we see, especially in our industry.

5:48We see how great it could be. And this isn't new. This goes back to Douglas Engelbart, right, who invented the mouse. To some degree, his vision at SRI failed because he had such a complex idea. He wanted humans to do a ton of work to get skilled enough to unlock the full power of the machine. And my understanding is even in his retirement, you know, late in the nineties, he lamented the fact that we took the easy exit. We took the computer mouse and we ran with it and just did this point and click stuff, whereas he wanted to replace the keyboard with the cord set. He had all these ideas. And I think that's a lesson we keep relearning, which is like, it has to be so easy.

6:25Like the big pot of value at the end is great, but it still has to be so, so easy to get at it that you can lead people down that path. If they have to take a course to learn how to do it, they're not going to do it. It doesn't matter how valuable it is. And that's not nothing to do with anything, but just how humans are. But that's like, that's true in a wood shop. That's true in an auto shop. The tool that you can pick it up and you're like, yep, I see it. I get it. I use it. It works. That's the tool people reach for time and time again, not the super elaborate complex one. I think about that all the time.

6:56That was the truth growing up on a farm is you just you had to get it done. You didn't have time to be trying to learn a new thing. You had to get it done. So I love that way to start, because when people think about technology, at least lately, they think so much about the digital world. And it feels like there's nothing more real world than like working on a farm and using tools to fix your equipment and making sure the cows get milked and sort of all that stuff that's in the real world. But we're in this new era, this new ecosystem of device wearables and smartphones and, of course, glasses.

7:26And so how do you think about sort of this new world where we're actually going to navigate and engage with the physical world through digital in a way that we used to interact with digital spaces? Yeah, this is the construct of the metaverse, which I think has been pretty broadly misunderstood or at least is understood differently by different people, is this idea of blending of the digital and the physical together. there. Actually, let's keep with the farming thing because it's great. Farmers are pioneering some amazing work. Autonomous driving is not as far along as autonomous farming is. Tractors that are able to plow the fields automatically, drones that are out there doing reconnaissance on what fields need what kind of treatment, that is actually pretty advanced technology.

8:07And it really is very much about blending the physical and the digital capabilities together. And so we had this amazing explosion with the internet and software. And I think we took software as far as it could go within the construct of a phone, a laptop. It's like we just took it to the absolute limit. And what's exciting me now is these really physical manifestations where through advanced sensors, whether it be audio, mechatronics, whether it be drones or robotics, automation. And then ultimately, I think through wearable devices, we're getting to a new plateau of hardware that can further allow software to breathe and expand.

8:49AI is such a fun example because it's so vogue right now. And if people have had a chance to ignore all the hype and really try to use these tools in their daily lives, there are some areas where it's mind-blowingly useful. You know, I'm doing a little home automation project, and I'm debugging, and there's these obscure, you know, Internet of Things devices that have unlisted APIs. And it would take me a long time to build a fuzzer to discover those, man, you can do it in minutes with these tremendously useful AIs, because there's a ton of things they're not good at. But I still find the interface to them very awkward, where I'm going to my phone, or I'm going, whether it's voice or text, it's this very transactional thing.

9:32And what's so funny, I find myself, I'm like the cut and paste machine suddenly, like I'm doing a coding project here. And I'm like, okay, I got to cut this result, the debugger output into here. And then it gives me the answer and I'm cutting and pasting back into the thing. This should all be integrated. So let's broaden out to AI generally. I mean, like I'm enough of a geek and the interface with kind of the digital and the physical world and how that transforms what it is to be human and homo technique, I could spend the entire discussion on this, which will be amazing. But there's also AI more broadly.

10:05So what's the meta view of AI and in use in products, use in the world? What is the AI philosophy? So there's three layers to this in my mind. The first one is we're building these very exciting models. And I find myself actually sometimes at war with both sides of the AI discussion. I have a profound belief that these are hugely important, meaningful things that will meaningful, advanced human capability. I kind of liken it to a word calculator, right? In the year 1960, a calculator was a person. In the year 1970, it was not a person anymore. It was a different thing. And at first they like ban them from school and get rid of them.

10:38And I grew up in a, you won't have a calculator with you at all times. I have three calculators with me at all times. My high school teachers were wrong about that. I think of the AIs that we are building as word calculators. I really mean that word image, like visually, they're really complex calculators that have move beyond the simple symbolic space of mathematics into this higher order space. I also don't think it is even the kind of thing that is human intelligence as we understand intelligence and agency and consciousness and thought. And so I have to fight both sides. Like AI is both a huge deal and also not that kind of a huge deal.

11:10So that's like my first kind of belief about it, which gives me tremendous confidence in how I use the tool. The second one is we're running into the information theoretic limits of it. If you go all the way back to like Norbert Wiener and his cybernetics of the first, you know, constructs of information theory. Hi there, I'm Pi and I'm here to add some context. Norbert Wiener's construct of information theory can be explained in simple terms as the study of how information flows and is used to control systems, including mechanical, biological and social systems. His work on feedback loops and data processing laid the foundation for modern control systems.

11:48This idea of how many bits can we pull out of something that are sufficiently generalizable bits? And we're finding out that for all the corpus of human media ever produced, it's not enough. It's not enough. We're finding that in robotics. Robotics is an effort that we picked off recently instead of meta in partnership as kind of an adjunct to our LAMA program. And no matter how many videos you have of somebody grabbing a coffee cup, you're actually not getting the data you need because you don't know the proprioception of how much force is applied and how we detected, okay, this is a plastic cup.

12:22It's going to deflect to a certain point and there's condensation on it. So I need to apply a little bit more force to counter the loss of friction that I'm experiencing. We do that autonomically. There's not a single conscious thought in our head when we're doing those things. Aria, when you're taking your phone out of your pocket, you don't know what the angle of your second digit is or how much force you're applying with your thumb to avoid getting the keys. To some degree, the things that we think of as intelligence, we're talking about the higher order functions of the human brain. That's arguably the less impressive part of intelligence.

12:54The deep brain, the mammalian, the amygdala, that lizard brain intelligence, that is wildly hard for us to capture in the modern era. So as much as I'm excited about the word calculator, I really do believe in Jan LeCun's vision that you have to do this pioneering work to break through to a world model that has common sense, that understands causality in a more substantial way, not in a statistical kind of soup way, but in a model-based way. And then my third layer of this stuff is I want it to be embodied. It almost goes back to like J.C.R. Licklider, who was one of the first computer scientists, to sit down a terminal and do a live programming.

13:31And he believed in that vision. He really was the one who then funded what would become SRI, found Doug Engelbart, funded a bunch of DARPA. it was at the ITO at DARPA. Like, I feel like we're in that era. We're in the terminal era of AI. And it doesn't want to be like that. It wants to be everywhere. It wants to be ubiquitous. It wants to be in full context of what your life is and with history of who you are and what your life is. I think what Douglas Hofstadter's book, I'm a strange loop and how we all have little mini versions of each other's consciousness simulations of each of those consciousness that's running in our brains that allow us to collaborate effectively.

14:08And my AI obviously doesn't have that. It has no idea what I want, what I'm about. It can't infer anything from context. So for me, like, I love where we are. I am a huge believer. I also want us to invest in these world models, and I want to free it from the terminal. And as the world models and the free from the terminal, are those fundamentally new different technologies from the kind of scale transformer? What's the kind of looking through a glass darkly thinking about how this would be done? I think the embodied part can do both. The embodied part will benefit a lot and maybe help a lot with world modeling.

14:40Once you have these sensors out there and you have better, richer data, when you have robotics data, which will give you proprioception, which will give you friction. Like, I think that is going to be a big unlock. It benefits the current models a huge amount to get that data and to be in that context. And also, it probably is some of the data that you need to start to understand what it takes to build a world model, which we appear to be born with. Tell me about the Orion project. I had the pleasure and honor of coming by, you know, Meta HQ and playing with it and getting some of the detailed exposures.

15:10Tell me about like kind of what you see as the significant use cases and where the metaverse is moving towards with Orion. So we've got the Orion glasses are on right now. I've also got this wristband, this neural interface wristband. It's got these little kind of metal bumps on the back of it all down the band. EMG sensors, electromyography. They're measuring electrical impulses going down my hand. And what I'm able to do, I've got a screen in front of me so I can do my email. I can do Instagram. We have little games that you can play. I have been caught playing the games in meetings before. They weren't my meetings to be fair.

15:47I was just listening in, but I have been caught. And I can do it with my wrist at rest by using a small number of gestures, but I'm using eye tracking to try to direct things. And so we had to do a lot of tough problem solving on photonics and optics. Some of the stuff we understand, we understand how to build apps. We understand how to do these things. And there's some novel interaction design here with the wristband and eye tracking, but I think we've made that pretty straightforward. Doing the neural interfaces was super hard. You're building an AI model of what the hand is doing based on these electrical impulses that you're able to observe from the surface.

16:19And so you just need a lot of people to build a generalized model that works so that anybody can put this on. And I think we had success with you, Reed. Like, I think it's, we're well into the 95th percentile, I think, of people who put this on and can right away, we know what shape the hand is in. And that allows us to do these gesture-based controls, even with your hands in your pocket, behind your back. These need to be regular glasses first and foremost. When they're powered off, I have to able to see your eyes, to see my eyes. And that has to be a human connection that we have is important.

16:44Otherwise, I wouldn't use the glasses. If you do all of this in a comfortable, all-day wearable form factor, there's a lot of challenges. This was 10 years in the making, and we thought when we started this program that we had less than 10 % chance of being able to build it. So the fact that it exists is a true testament to a vision that Mark Zuckerberg had, that the research team, Michael Abrash, our chief scientist, and his team had for a long, long period of time. Now, we always thought, to your point about the metaverse, that the first thing that would happen would be these holograms in the world.

17:13And at first, they'd be just referenced to you, so just your personal interface. And then over time, they would be attached in the world. And then eventually, you would have an AI that was doing it. What's been surprising to us is the AI came first. We have the sequencing wrong. The AI showed up earlier than expected. So what's been so exciting is there's a lot of products now that are totally valid products that are easy to use between full AR glasses, which are spectacular, but will be expensive. And the Ray-Ban Metas, which are super affordable, but like a little bit more limited in their functionality.

17:46That entire spectrum is now open to us. And it's really, really exciting. And we have demos internally and our wearables. And with the Ray-Ban metas that are in markets today through our early access program, you can use this tool called Live AI. 30 minute session until the battery runs out because it was kind of bolted on after the fact. But for 30 minutes, it can see what you're seeing. It hears what you're hearing. And the difference in how useful it is to me as I go about my day in the world, I was doing a film development project and film developing is a real fussy business with what kind of chemical at what temperature for how long.

18:19Usually I'm doing it with like a laptop next to me and I'm trying to type into the laptop, okay, what's the thing? Doing it with the live AI session was incredible. It just sees what you're doing. It's like, okay, you got 20 more minutes of doing that. And then you're going to do this other thing. And why don't you go ahead and prepare this now? It's stunning. So for me, it is about blending the physical and digital together in both directions. You have to give the AI access to the physical context in which you operate and the digital context in which you operate on your desktop or on your phone.

18:48And conversely, you want to be able to bring those digital constructs into reality through robotics, through automation. So I see really a very, very exciting kind of decade ahead of us in that synthesis. So I am someone who sees 90 % of my vision out of one eye. Will I be able to use these? Yes. So these are binocular. You'll experience the same limited depth perception that you would experience normally. That's right. That I have now. Yeah. So you'll still be able to use these. Over the future of time, I do expect one of the real option spaces here is what if you go to monocular? What if you just had the display in one eye?

19:27Yeah. And there's some challenges there. It creates some binocular rivalry. So for people who have full vision in both eyes, they might struggle sometimes to know, hey, my eyes are seeing different things looking at the same space. Where should I look? Yeah. You would actually be better off in those displays. I can't wait. I finally have a superpower. You would have a little superpower. That's right. You'd have a little advantage. The monocular displays would probably be great for you, cheaper, lighter, and probably just as good. So I think you've got maybe a little inside opportunity on the rest of us.

19:57I love it. I love it. Okay, so for some of the skeptics out there, I feel like this is especially a good question for you because you were essentially the person who created the news feed. And for those of the youngs among us, they might not remember this, that the news feed came out and everyone was like, no, this is horrible. What are we doing? And now we're like, oh, a news feed. This makes so much sense. This is perfect. We can't imagine a world without it. And so some people might be saying the same thing. They might be saying, why are you creating these things that we don't need? Like, what would you say to the skeptics?

20:28And then also, as you see people using it, what do you think some of the mainstream adoptions are going to be? Things like AI are different kinds of things. These are different types of innovations. These are truly new tools. This is a new space. the smartphone was a kind of a disruptive thing where it was like hey you have a phone and you have an ipod and you have like you like the web we're gonna put all those in one places right that was like that was the pitch this will be similar to that this will be like hey look you already have a phone that you like doing this you already like it you already like instagram this is a better way of doing a thing that you already do and then once you've established that beachhead through comfortable paths then the huge opportunity presents itself you know you talked Aria about having superpowers.

21:11That is the vision that we have for these wearables. Every person, superhuman vision, superhuman hearing, superhuman memory, superhuman cognition. That's what I'm talking about. That's what I really believe is going to happen here. A tremendously equalizing technology. We talk often in our society today about the vagaries of the birth lottery, and we're rightly doing so. We usually talk about the birth lottery in the context of whether it be race and gender or physical geography. And those are hugely important factors in how our lives play out. But we don't talk about the other parts of the birth lottery, who has incredible memory, incredible vision, incredible hearing.

21:48Some people are born that just have those talents, incredible ability to think creatively, to pattern match. And there's no reason we couldn't all be Garry Kasparov in chess if we have the wearables. You know what I'm saying? Now, it probably takes the fun out of chess, so I'm not recommending anti-competitive performance. But my point is like the vagaries of the bell curve and the ranges of human capabilities are beyond just the demographic ones. They exist in terms of fundamental capabilities. And how interesting will we be as a society if everyone has the full access to those facilities? That is the future I see.

22:19And I think that's a pretty compelling pitch to people to give them to try a new thing. Exactly. And I'm going to take this moment to ask you a question that I hate getting, so it's kind of entertaining to be the asker of it, which is, you know, if you're predicting out, call it three years, you know, and with kind of wearables and AI and everything else, what kinds of things do you see in the future that will be the kind of thing that people will be doing? Is it kind of like a, it's giving me a constant scan on my life and it's doing the proactive search of, oh, you're looking at like film developing, or you're looking at this thing.

22:53oh, let me tell you some stuff about this. What is your future prediction that might cause, call it semi or non-technologist to go, oh my God, that's coming in three years? Yeah. I think what's interesting, the hard part of this question is the timeline. I think you and I probably would do better in one year and 10 years. Three years is the most awkward time because you can kind of, I know one year, I have a good sense of 10 years, three years is tough. I don't think we'll be at the proactive place in three years quite yet. Early adopters will probably have always on systems that are capable of it.