Meta's CTO On Its Augmented Reality Dreams — With Andrew Bosworth

0:00Meta's chief technology officer joins us from Menlo Park to talk about Orion, the company's new augmented reality glasses and what they mean for the future of computing. That's coming up right after this.

0:41Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. And we're here in Menlo Park with Andrew Bosworth-Bos, as he's known, the chief technology officer of MetaBos. Great to see you. Welcome to the show. Thanks for having me. Yeah, great to be here. So I am just off a demo of Orion, your new holographic augmented reality glasses. First of all, they're crazy. That's what we're going for. First and foremost, they're crazy. I love that in Mark's, the Supercut Mark showed, the kind of sizzle reel. It was just like people say, like 10 people in a row just saying, oh, this is crazy.

1:14It's amazing because you expect it to be an experience you're having in VR goggles. I'll tell people, you know, just for our listeners. So these are glasses. They don't feel extremely heavy. You wear them just on your face like you would normal glasses, but they're augmented reality. So you can make calls on them. You can watch videos on them. You can play games on them. and you have to like remind yourself like, oh wow, like these are glasses. That's right, yeah. I'm not wearing goggles. So you've had the, and this is a 10 year project within Meta, but it looks like from what I saw with your team that this is something that has really come a long way, even in the past year.

1:49That's right. And you've had an opportunity to use them. What have you been using them for? Yeah, the hardware came up last year at this time and so the software bring up really only came together a few months ago and they just made incredible progress since then already. So for us, there's a couple of different things I would call out. The first one is actually very relatable, I suspect, for those of us who worked during the pandemic. During the pandemic, you're obviously doing a lot of video calling. And one positive silver lining from all of that was the development of the back channel. So every meeting would have a chat alongside the meeting.

2:22And somebody could ask a question, hey, I missed that. Is there a link to that? Okay, cool. And so you're not interrupting the meeting to get this extra context. And what's been fun with these glasses is these glasses kind of serve as like the in-meeting back channel. So now I'm in an in-person meeting, in a live meeting, and somebody can add context to a conversation that's happening in the glasses. So I'm like, oh, okay, somebody references a prototype. And then the next thing I know, up pops an image of the prototype that they were referencing. So now I'm fully in context again in the meetings.

2:50So these things are kind of wild. Before this, it would have been very rude to me in the middle of this meeting with Mark Zuckerberg. Take my phone out, like go look up the reference of the thing. So that's been kind of interesting, this real life back channel when you're in a live scenario. But let's also not overthink it. It's also a great way to scroll Instagram and enjoy the reels and tap on those things and to watch videos. The surprise hit, I would say, from the demos has been video calling. And there's two parts to that, right? One of which is the experience of having someone video call you while you're just walking around is really cool.

3:20And you can see them in full fidelity, but you can also see and navigate the world really effectively. You've got your hands free. and then conversely what they would see is your Kodak avatar and we've really fooled some people with the Kodak avatars when they've seen the Kodak avatar call in they don't realize it's Kodak avatar they think it's a real an image so let's just go right away to like what this does to real world experience you you mentioned it right so like it's so hard to be in the moment now right and a lot of times you're speaking with someone oh there's a buzz on their phone Hey, one second, let me check this out.

3:52And then we went, a lot of people went to smart watches. I see you're not wearing one. Not currently, no. And that also, like, you're like sitting at a table with someone and they're like, keep on looking at their watch. So does this take people out of the real world in a way that, you know, might be detrimental? I mean, what do you think about the real world implications here? No, tremendously important question. And we've tried to use a tagline, more present and connected. the idea of kind of reducing the dichotomy we often feel. And actually one of my anti-patterns that I've been telling people about for these glasses is just having to be notification services because notification services are an invitation to leave the present moment and instead go elsewhere.

4:32Like that's what a notification is intended to do. It's to let you know, hey, if you want to, you could leave this physical plane and go dive into this digital surface over there. I want people to be able to do that if that's what they want to do, but it can't just be that. Instead, though, if you think with the examples I've given, it's about adding context. And so the real difference maker between these and a phone in your pocket and all the things that are attached to a phone in your pocket is they have the ability to be aware of the world at all times. And this is where AI has been such a tremendous tailwind for our vision for these.

5:03You know, we really thought early on when we were developing these, even until maybe a year ago, it was going to be holograms and then at some point later AI. And in fact, it's going to be the opposite. The AI is going to come first. It's already come to the Meta Ray-Ban glasses. And so for me, at least, we've got this idea where, okay, what if these glasses are contextually aware? They know I'm talking to someone face-to-face. Hey, I don't need that notification about my cat is moving on the camera at home. I don't need that notification right now. We can hold off on that. Oh, someone's calling me.

5:31It's my kid's school. Okay. Even though you're talking to someone's face, we can probably pipe that through and let you know, hey, your kid's school is calling. And so that is the real missing piece for allowing these to both be more present and more connected when you're using them is the contextual awareness that they have the potential to have about the world around you and how you're engaging in it. And there's also just like a cool part of it that I saw in the demo is imagine you're just like at a restaurant waiting for somebody instead of like sitting and scrolling your phone, you can basically have a computer with you, which you just brought in with your glasses.

6:06You can do some work. You can watch a video, do whatever you want. And it's there with you. Yeah, one of the fun demos that we have in prototyping, it's not even in the demo that you tried yet, is this cool thing where you have your whole workstation set up. So you've got like a three-panel workstation set up. And then you just get up and walk around, and it goes away. And it minimizes. And now you've got like your walking around UI if you need it. And then if you sit down again, you can just recall it immediately. So like all your context comes immediately back whenever you want it there. So not just a mobile and portable workstation, but one that's literally, you know, at the twitch of a finger ready for you to use.

6:43A lot of computing, it solves a problem. Sure. And another question I have for you is why do we need this? Like what type of problem does this solve? Can't we just get along fine with our phones? And I know that's going to sound stupid because every computing phase, they've been like, well, can't we just use desktops? Why do we need this in our pocket? But so what problem do you think this solves? No, it is. It's exactly the right question. And it speaks to how we as a society so quickly normalize everything that's come before as completely normal and everything that's new is like kind of this unnecessary thing.

7:14Douglas Adams, a famous quote on this comes to mind. I'll leave that as an exercise to the listener. So for me, at least that, yeah, when people, I remember when cell phones came out and everyone was like, who is so important that they need a phone everywhere they go? And it's like, all of us are. It turns out all of us are that important. And I feel a little bit the same way. And some of these things will sound trivial, but having meta AI is actually a really good test point. I've had meta AI on my phone for a while as an employee of the company. And it's pretty useful. And I like asking it questions.

7:45Now I have meta AI on the Ray-Ban Meta Glasses. And it's like, oh, I use it 10 times more often. Because just the friction of getting my phone out of my pocket and pulling it up and pulling the app is not trivial to me being able to just ask a question. And now what if instead that AI didn't need to even be asked a question? It's anticipating. It sees me. I'm like checking my pockets. I'm checking. It's like, oh, your keys are over in the desk where you left them. So these things start to come together where because the glasses are on your face, they're not in your pocket. And they have the ability over time, not immediately, to become aware of the world and to understand what you're trying to do in the world.

8:24They can actually be very proactively helpful. Just to you, privately, you're in control. But that is a thing that phones can't do. And I do think it is a real step forward in our relationship with digital technologies from going very transactional. Like I have gone to the technology and then I come back with my answer to being very integrated. So is this basically a way to just deliver the AI platform more seamlessly? It's not just the AI platform. I think the AI platform is what changes the scope of how we interact with it from being completely reactive, which is what computers have been from time immemorial to being proactive.

9:01I think it also, of course, is a super more convenient way to go ahead and manage communication, manage all the different things that phones do. Listen, use video calling again as an example. This is just a terrible way. I'm holding my hand out in front of my face. My arm is literally tired. I've been doing this for eight seconds. And I'm like, I work out, man. Like, it's like I can't. So there's a few things that are just fundamentally better on this platform. And then there's a few things that I think, yeah, you can do it on the phone or you can do it here. But if this was already on your face, you wouldn't use your phone.

9:31Right. I mean, I've definitely had those moments where I'm like video calling in New York and just running into people. Absolutely. Maybe this will help. I've walked into a post before. Like, I'm not proud. Why is it important for Meta to do this? So Meta's mission long has been, all right, let's, you know, connect people. Well, obviously, it was the name change. Now it's metaverse or meta. Why is it important for this company to build this device? Yeah, there's two different answers to this. One that I think people will recognize and understand. Strategically, we have just been at the mercy of platforms since the start of the mobile era.

10:07And we were a web company. We were born of the web. And the web was this amazing, unowned platform that you just really had a lot of control over what you built. And you had a direct relationship with consumers. And we absolutely feel that in our application development, we're stifled by the gatekeepers, especially Apple, in terms of the policies, what they allow, what they don't allow. And there's a real value that we could have delivered to humans who love our products and would have been happy to have that value, but we weren't allowed to. For example? Well, anything that has to do with, you know, keep in mind, on the web, Facebook had a major games platform, you know, which we were not allowed to bring to iOS.

10:46App store fees and app store rules. And gosh, I mean, and they've gone even further than that. And kind of, you know, finally, we built an app install business. And then Apple came for that with ATT, you know, a tremendous amount of kind of self-preferencing for themselves there. And in doing so, they really put, you know, the brakes on what was a really rapidly growing part of the games industry. And so anyway, so I think, like, for us, like, platform control is important to us. That's a strategic reason. That's what everyone jumps on. That's a fair one. If I can be honest with you, that's like a very nice story.

11:17And I'm sure it's a part of it, but it's not the real story. The real story is that we're a technology company. And Mark is an enthusiast who wants to continue to advance what is the state of the art in how we connect people with each other and with technology. And we've known for a long time that these are the next thing. As an industry, I think we've had a sense, AR especially, is like, we don't have to convince people that someday AR is probably a thing that will exist and will be great. and this is where Michael Abrash, our chief scientist, has this kind of state. He's like, you know, this is the myth of technological eventualism, this idea that eventually the technology will exist.

11:51He's like, that's not how it works. Someone has to stop, sit down, and actually build it. And it's a tremendous credit to Mark. I'm going to sound like a kiss-ass. I don't mean to. I'm very critical of Mark in many, many areas, but on this, he deserves his credit. It's a tremendous vision and commitment that he's made to this technology in the face of incredible scrutiny the last several years to stay with these investments and deliver Orion, which I think is just a watershed moment for us to say, like, this technology is coming. It's here in some form. It's not evenly distributed, but it's here.

12:25And it's within our lifetime, certainly within the next decade, it's in our view we're going to be able to use and enjoy this technology. And I think he just wanted to do it. And we should say that these glasses are internal. So over time, they might be released. They're not released yet. That's right. But I do think we should talk a little bit more about the platform side. So, okay, for listeners. For listeners, I'm sorry. I put on a pair of clear and transparent ones. The ones that we actually use have opaque magnesium shells. The clear transparent ones are super fun to have. They only work for about half an hour at a time before the thermals shut down because this plastic does not dissipate heat fast enough.

13:01But the real ones, they don't thermally exhaust themselves. They go until the batter is exhausted two to three hours. But, yeah, so it's pretty fun. It's just, I'm sorry, if I have these cool demo props, what kind of show them? I can't put them on. But let's talk a little bit about the platform risk and the operating system. I mean, do you have to build your own operating system for these? Yeah. So a couple of different stories to this. I mean, when we talk about what is an operating system, one of my former colleagues, Ficus Kirkpatrick, had this great saying. He was like, operating systems are a set of promises made to developers and to consumers.

13:31So you can think of in iOS, the swipe gesture will get you home. No matter what's happening on the app, that's a promise that you're making to a consumer, and the apps can't control it. That's a promise that's being made. You promise to developers, this memory is yours. You're allowed to use this memory. This API will cause this thing to happen. And so we're using Android at the most underlying low level as the operating system. But of course, the operating system isn't Android. The thing that people are interacting with on, whether it be in mixed reality or in augmented reality, doesn't look or feel like Android.

14:04And there's a bunch of reasons for that. Number one is the interaction paradigms are different. And so for augmented reality, to stay there for the moment, you're not going to have a direct manipulation interface. You don't have a keyboard. You don't have a mouse. You don't have a touchscreen. Well, it's worth noting this is really the first computing platform that we've had ever that didn't have a direct manipulation interface. I mean, you're going back to Stanford Research Institute, Xerox PARC, the keyboard and mouse. Before that, it was all mainframes, the keyboard and mouse. Then you had this tremendous innovation in touchscreens and multi-touchscreens, direct manipulation, very akin to the mouse.

14:35Every single application you've ever used, you've used via a combination of direct manipulation. We don't have any of that because you're not going to walk around with those things. You're not going to have touch controllers with you as you go about your day. So what are we using? We're using hands, which can sometimes be modeled into direct manipulation with little laser pointers like we do in mixed reality. But ideally, you're using voice, which is not always appropriate, and eye tracking in EMG, which is, again, approximating what you had with direct manipulation with your hands. And so for me, at least, like when you change the interaction paradigm, that is a half of what an operating system is.

15:08It's like, how do you, the consumer, affect the machine? The other half is how does the machine bring information back to you? And here we're more similar. You've got a display, you've got audio, except for the fact that the display is additive light. So there's light coming through from the world around you. So the display has to be somewhat aware of what is happening in the world around you, ideally, to be effective. And so anytime you're building a new interaction paradigm, especially when pairing it with novel display technology, you're basically building a new platform. There's not really another way to do it.

15:39And so then how do you think about like who might who else might build this? You were just speaking with Ben Thompson. You talked about I don't know if you were saying the potential for Apple to build something like this, but you talk about I'm just going to quote from you. You look at what are Orion glasses. They're full AR glasses. You have custom silicon in the glasses, custom silicon in the puck. But Apple could build all this and just be like, oh, it only works with us. So does it fully insulate yourself from the platform risk? No, nothing fully insulates us, especially in a world where mobile is the dominant platform.

16:10Do you expect them to build something like this as well? I have no idea. You know, they keep it pretty tight down in Cupertino. They don't. They're a pretty tight ship and still don't have great information on that. I think they would love to build something like this. I think we've been investing longer. I think we've believed in it more. And as a consequence of, frankly, Mark Zuckerberg's faith and tremendous execution by a team of thousands of people over a long period of years, I think we have a meaningful lead in a bunch of the technologies here. But you'd be foolish to count out a great technology company, one of the great all-time companies in general at Apple.

16:46But my point is, like, I think that the fact that they already have this phone that's in your pocket has some advantages because you've now got this device that has a bunch of compute. It has a bunch of silicon. They can do great custom silicon. They can add more to that. It's also a disadvantage, I think, in a way. Are they going to be willing to walk away from, you know, one tremendously outsized profit center in pursuit of something that is fundamentally disruptive to that profit center? And if you're centering more of the experiences on the phone as opposed to on the glasses, do you make the leap that you need to make in developer platform, in interaction design?

17:18Because a lot of those things that we've discovered, if you really were to center them on just the phone and the way phone apps want to work, they don't work as well. So I think it's going to be a dynamic landscape for a while. And I think we take some risk sharing this with the world early to demonstrate what's possible. But we do want to ignite the passion of the developer community, the engineering community around us. And I think it's also something that we kind of owe to people so they can see what this 10-year journey we've been on in Reality Labs is all about. Okay. So let's talk a little bit about the reason to have an app.

17:52It's just the normal way that you use a phone. But you're talking about these glasses. They're sensing the world. You know, you now have AI voice that you can speak with. I think Mark yesterday said, or we're talking a week before this is coming out, but he said, I think that voice is going to be a more natural way of interacting with AI than text, which is basically interacting with computing. If you think that's the way this is all moving, do we need apps anymore? It's a great question. I love this. And actually, it goes back, you know, applications didn't always exist, right? In the beginning, it was just like the system was the software it was running at the moment and you would kind of program it.

18:26And I think often about JCR Licklider sitting down for the first time at a terminal, which is reprogramming itself on the fly. And it's really one of the first times in history that people are doing human-in-the-loop computing, as opposed to what would have before that been through mainframes and timeshares. Like, you put your job in, you come back later, get the results of your job. If it was wrong, you change your software and try again. This was human-in-the-loop computing. And it was this tremendous watershed moment that ultimately led through his work at DARPA, but the IPTO with Doug Engelbart with Xerox PARC to the Alto.

19:00And that's where you start to see, oh, there's like a windowing SRI. Like there's a windowing model and each application is contained. And the application was born as a consequence of us being in a human loop with the computing. And what's crazy to me is the analog that I draw to chat GPT, in fact, which was this human in the loop AI. Whereas before this, you would like create an AI and it would go do a job and come back. And if the job wasn't right, you'd like reprogram your AI. Whereas this is like, no, no, like I can just modify my queries on the fly and the AI is like adapting to it. And so one does wonder, like, does this get into a more, if not an app free world, a more web like world again, where things are deeply linked and interconnected through services to one another?

19:41I think it's one of the challenges that the app model is going to face, like things like Apple intelligence, for example, will face on the phone. is what I really want is for something that actually intermediates the apps and can go to my Amazon app and grab my purchase history and then go to a different app and drop that in there and add that to my spreadsheet for my budget, which is in this third app that I have. And that is not something that Apple wants some other developer to be able to do because that would disintermediate their position, their privileged position on the phone. And so if the AI can't work across all the breadth of my applications, well, it's not the most helpful AI it could be.

20:22One thing that's really fun about the desktop and the web still is that it can still work that way. And I love those platforms for that, including OSX. And so I think often about, okay, once I have this AI, I want it to work at basically user level. Like the AI is actually my agent in the machine. I think you probably end up still having applications. There's a lot of really valuable constructs we've built up as an industry over time that have to do with the safety and and security of data, and the model of how these things interact with both the computer itself, with the network, and with other users.

20:52So there's actually still quite a bit of value in having these apps, and not to mention just how literally would we do it and have these systems not constantly walk on each other. So I think the application model doesn't go away. But I do think you end up injecting this potential new layer on top of it, which is an agentic layer. And so wait, how do developers then think about building for this type of technology? It becomes much more about what APIs you're providing, and the unique value that you provide through the software below the API. And if it's about the API you're providing, does that mean the main interaction layer is going to be through some sort of voice AI?

21:23It's hard to say at this point. I think voice is a great modality for some things. It also has a lot of vagueness to it, even amongst humans, a lot of context that's required even among humans that obviously even a really great agent wouldn't have for a long time if it was with you. And so I think voice is probably not the only piece of it. Voice and text are part of it. I'm sure there will still be a lot of very direct manipulation interfaces for a long time to come. But you can start to imagine, yes, as voice gets more developed, my ability as a developer to create value in how I manage process data or how I, you know, manufacture new artifacts is actually the more important thing I'm doing rather than having to also extend that to the consumer through some kind of interface.

22:04So in this world, do the AR glasses replace the phone or do they supersede it in some way? I can't imagine it goes away. Yeah, it's a good question. We think about this a lot. It doesn't go away for a long time, if it goes away at all. There is a real strong argument that when the AI glasses become super prevalent, and that is your interface to a device, and you have, especially if you have a display, whether it be on AR glasses, or it be on your phone, or your watch, or whatever the thing is, that you can kind of, when you need to directly manipulate something or provide very precise entry you can.