Chief of Staff for the Masses: How Meta’s Joshua To Designs Wearables with AI

0:00The number one use case that I'm excited about when it comes to AR and AI is fixing notifications. like literally when I interviewed Bob everyone asked me what's the killer AR use case and I literally looked at them dead in the eye and said notifications I just want to like get notifications in a way that is helpful to me don't give me notifications that aren't useful in a certain time and like everyone's got this vision to like summarize notifications do these all these things But I think like, really, I talk about this all the time. I really think that AI with AR is going to like allow everyone to have like a president of the United States sort of level chief of staff kind of doing work, doing life with you.

0:54Hey, I'm Josh. I am the VP of product design at Meta, and I get to work on wearables and AI. and I'm super excited to be with you all to talk about AI and how we use it at work and how we think about the computing products that people are going to wear and how all this soup of things are going to sort of create and deliver a profound impact on people's lives. I am thrilled to invite my friend Josh to the conversation. Josh is a proper designer, like capital D. I always joke at the D school that I'm a lowercase D designer. Josh is a capital D designer, I feel. Josh, how do you react to that framing of your professional credentials?

1:44I mean, I don't know. That gives me a bit of imposter syndrome. I'm a property designer, but capital D, that's a big statement. Now, Josh, okay, so when you and I met, call it 10 or even 15 years ago, maybe somewhere around there. I think if I'm not mistaking, you were doing hardware at Google. Is that right? That's right. Okay. So how did you get from kind of hoodies to hardware? And then how'd you go from Google to meta just to, just to finish tracing your hero's journey? Yeah. I mean, it's a pretty long and winding journey, but anyways, um, I do hoodies and then, And, you know, on a personal level, like I, you know, like the impetus behind hoodies was like sort of who fold at least.

2:33Right. One was I just thought it was cool to run a clothing brand and to see people that I really looked up to, like vibe with our brand, if you will, and wear it around. So, you know, it's like, whoa. The Rock is wearing our hoodie like that's cool. But really, like the thing that really drove me back then was like we thought that clothing, T-shirts, hoodies were like a medium for people to communicate things that they really care about. And so that was kind of like our whole thing. It was like like I think what was our tagline? Clothing people love messages society needs was like our tagline.

3:12and uh so much of what we did was about like raising money for causes raising awareness about issues that were going on but doing it in like a very graphic forward like design forward way where it wasn't just the message that mattered it was like it looked really cool too but uh i guess like two years out of college i'm like grinding um on hoodies and it's like it's really hard it's like doing as well as I could have dreamed of doing. And yet I can like barely pay myself more than minimum wage. And I'm like, oh man, I ain't going to live at my parents' house for the rest of my life. And the Rock's already wearing my hoodie.

3:51So like, where else? Yeah. And so I was lucky enough to get my first job at Google, but it wasn't, you know, sunshine and daisies. Like I kind of went through the school of hard knocks at Google where it was like pretty hard to kind to go from being top 15 college student in America and having rock stars wear our clothes to being like a contractor at Google. My first job was like, I was basically like a customer service associate for Google AdWords. And it was a really hard job to get. And, you know, like, so the first year of Google was me like memorizing this like five finder of policies for the text ads on Google.

4:39And then my job was to like approve ads and to look for spelling mistakes and grammatical things like that. And there was like a fear of getting fired every Thursday. Someone can tap you on the shoulder, do like a quality review with you, and then you can just get fired. Like be like crying, vomiting, like really worried before these quality reviews um and you kind of had to go through this gauntlet but anyways like for me it's like man i was like a i studied design in college and so what i would do is like work really hard approving ads for 45 minutes of every hour and then i'd spend 15 minutes coming up with design ideas product ideas for google uh kind of how i stayed sane just from your own personal kind of time management it's not that somebody's asking you to do it you just said hey i'm carving out.

5:30In fact, it was like Bryce sort of like frowned upon. It's like, you just say it's my 20 % time, right? There was no 20 % time for contractors doing the approval then. Let's just put it that way. But yeah, like from there, I like managed to get through that. And then the progression was you go from doing ad approvals to doing email support to doing chat support. And then you get the final kind of thing is you do phone support with Google customers. And that's kind of how I got my start in tech. And, you know, credit to Google. It was just like a really crazy, fun time and things were growing like crazy.

6:13And it's sort of like once you got your foot in the door and once you became a full time employee, it was sort of like there's no problem in Google that you can't at least attempt to try to solve. or at least that's sort of just the feeling of the time. And so even when I was a contractor at Google, I was able to suggest things that made it to google.com. But these are like super subtle things. Like I noticed that the spacing and the I'm feeling lucky in Google search bar were not aligned correctly and the padding around the search box was wrong. And it was like pretty thrilling to make these small tweaks and then see millions, if not eventually billions of people see these like small tweaks you you made um and so that was like pretty cool um even though like you know looking back it's like oh you know that's not that big of a deal but for me really big deal would you say your kind of 45 15 split would you say it kind of prepared you is almost like exercise for looking for opportunities because i'm just looking at my notes here you said once i got my foot in the door there's no problem you can't at least attempt to solve but i think there's always this tension because like I've got a job and it's usually not my job to solve this other problem, but it sounds like, I mean, if I'm kind of narrating over your character art, you had started practicing being deliberate, making the most of your time, carving out time to do stuff beyond kind of your scope.

7:38Do you resonate with that characterization? Yeah, I would say even today, that's sort of like what makes my job most exciting and what maybe gets me in trouble too, kind of at the same time, is that the easy version, not easy, but like the safer version of doing my job is to focus on purely just design things. But I also think like design in general is kind of about problem solving. And I think our work gets most interesting when it kind of butts up against like all the other things, whether it be partnerships and new technologies and different brand collaborations we could be doing. And but, you know, when you do that, your job sort of involves a lot more people and you have to be a lot more thoughtful and just empathetic about how to sort of connect the dots and work with the right people to make things happen.

8:32But that definitely resonates. Like, you know, I think just getting your hands dirty and putting your hands in different cookie jars just helps like the cross-pollination of things for sure. Love that. But anyways, continuing the story arc, I get to phone support, which is like the final boss. And then you're supposed to kind of slowly become like the sales executive over time. And you help big companies advertise on Google and use Google's latest technologies to help your company grow. And that's when I kind of like broke off from that and just like had really cool opportunities that I got to jump on.

9:11And so then got into like marketing and program management and then eventually became a product manager. But at the time, Google didn't really hire designers unless you studied HCI at Stanford or, you know, Carnegie Mellon. And I just, you know, I studied design media arts at UCLA and I was like, sorry, we don't understand how that maps to user experience. And so I eventually left Google, started my own company, which was like kind of a crazy concept, but it was like a VC fund attached to a co-working space attached to a design consultancy design studio. And we did a dozen investments over a number of years, had like a pretty big design team that both serviced our portfolio companies as well as some selective outside clients.

10:00and kind of unexpectedly a couple years in Google came knocking and acquired the entire design team including me back into Google and so it took kind of leaving Google to come back to do the job that I actually wanted but it kind of all worked out and so that's when I kind of like came back led a bunch of like enterprise software so like email docs and spreadsheets Google Drive and those type of things. And then that kind of led me to like Google Cardboard and AR and VR and hardware. And so it's been, it's been really fun. Did you ever get to work on the original glasses over at Google? I worked on it.

10:42I was not on the original Google Glass team, but I did get to work on it some kind of the years before I left Google, when they were starting to look at how the Google class could be um well applied in enterprise for folks working in factories or folks out in the field and so what do you think i think sometimes when people like yourself have invented these kind of early versions of things that then becomes kind of like more mainstream but it takes like a few years there's like a few things that kind of like is obvious now but wasn't that obvious back then do you have any of those that you kind of remember from like the early google class days where because i think a lot of people now go oh yeah i was too early and like but i think a lot of us had them and tried them and started to understand how visual language would be when suddenly you had to you like touch your side of your head instead of a mouse and stuff like that were there any of those kind of like uh things that kind of comes to mind yeah i mean the most obvious thing is that like you know coming kind of a bit from like a street you wear fashion standpoint um you got to make sure people want to wear it and you got to make sure that people around you want you to wear it right and i think that's probably the miss if you will because like i think in many ways like the experience on google glass for the time was like way ahead of its time and like it was actually pretty well stitched together um and uh you know you can do like turn by turn navigation with the thing but it just looked crazy you know like if this was not this thing you you kind of want to be seen wearing and people around you were really uncomfortable yeah i want i wanted to hit that point you said people around you because i get the the two parts the first part of you got to make sure you want to wear it that makes sense but then the part around you want people around you to wear it how do you feel like the latest kind of you know iteration of technology addresses the do people around you want you to wear it problem differently than the original google class did yeah well i think we take privacy pretty damn seriously and so that's why early on it was It's just like, hey, when you're taking the photo or when your video is on, it should be really obvious.

13:11And so when you take a photo with the glasses, the LED turns on. So, you know, folks know when the camera is on. Do you think that will continue? And I'm asking because after COVID, we all did Zoom. And then now, then everybody started to transcript and record their Zoom. and now obviously, you know, the classes, but also like all these new devices that records everything. And it seemed to be kind of like just slowly becoming kind of a thing where people, I guess, assume that at least all Zoom calls are recorded. And I guess the question is, do you think that will ever just become commonplace?

13:48We assume that all our conversations, when we talk to other people, are recorded? I mean, I think we're probably like in the most recorded era of, time right in human history and so i really do think that like over time there there will like continue to be more things recorded but like these things take time right even early on when people started getting you know smartphones um i mean it's been a like you know even like the smartphone arc has like even like the most successful consumer electronic in history um there was sort of like a period of time right and cultures matter a lot too right like in japan to this day if you sell a phone in japan um the camera needs to have like an audible like shutter button sound right like privacy um is really really important i think what aids in in sort of these uh things too is that as new technologies become available they also often like help groups of people um so like specifically with our glasses, probably one of our like most fanatic product users are sort of the, the, um, low vision blind community.

15:04And I think like kind of early on in kind of product design, product adoption curves, right? Like folks who find like a really unique spiky way to use it and has like a sort of pretty like meaningful and profound impact on their lives kind of helps pave the way for other people to accept using it. So because I think in general, if there were folks around me like recording me, I'd be like a bit uncomfortable. But if one of those folks was someone who was using that recording and technology to help them get through the day or help them get things done, it would be like, hey, like you could have 10 cameras for them.

15:49It helps you do your thing. But I think in general, right, it's really important that there's trust and the social cues kind of all exist. Right. I do think that as like the utility and the usefulness that you get outweighs some of the privacy concerns and you do a good job of the social contracting and And, you know, when someone's recording, you know, when someone's not, you know, there's going to be a bit of a shorthand for like, hey, Jeremy, do you mind turning it off or like having a more personal conversation? And, you know, I just don't want that locked. And I think that should be totally OK.

16:28So can we get to I would love to hear how you think, Josh, about the aesthetics, the want to wear and people want you to wear kind of now that you are building what you describe to me, at least as one of the most complicated, if not maybe the most complicated consumer electronic of all time. Right. How do you think about preserving the, call it street culture, you know, aesthetic mentality when there's now so much, you know, a hoodie is one thing. The most complicated consumer electronic of all time is something totally different. How do you think about kind of merging those worlds? Yeah. I mean, I think to be clear, to clarify, the thing I showed you, Orion, is like our internal prototype for air glasses.

17:13Um, that's, uh, many clicks more complex, um, than the Ray-Ban meta though. I think the Ray-Ban meta glasses is like also an incredible feat of design and engineering, right? Because it's like, literally it's the Ray-Ban Wayfarers or the headliners. And we managed to put as much tech as we put into it. And we're really proud of that. But, um, it's really the Orion's that, you know, I sort of make this claim that it's the most complicated consumer electronic in history. but um i think the the question is a good one and i really do think that um it's kind of what i shared it's like does the utility and the usefulness you get out of it outweigh sort of the fashion hit if you will for most people right it's like you give a ride to a supermodel and they put it on like hey this looks good it's kind of look like my uh dior sunglasses but the average person uh wouldn't want to get caught dead maybe wearing that like you know unless folks knew like oh my god when you put it on you got this like wide field of view display you've got ar and ar i like you know popping off at the same time and doing really good things for you and so these things will sort of take time and like right now i think we're very very proud of Orion because it's like under 100 grams.

18:36It looks like a bulky pair of glasses. It's clearly not your normal pair of glasses because, you know, at certain angles, you see kind of some cool light leakage things coming through. And so it's got a bit of like a cyberpunk vibe to it. And, you know, you can maybe kind of own it. But I think the form factor that you particularly saw is like, you know, like we're aggressively working on getting the form factor to be lighter so that more people will be willing to work. In your career arc, one thing we haven't touched on, but is obviously relevant to our audience and our conversations is AI. Can you weave AI into your journey?

19:14When did it first kind of hit your radar and specifically generative AI? And then are there any kind of moments of early moments of delight you remember going, oh, this is something special? Wow. I mean, I think when I was at Google, I got like some really early glimmerings of it where, I don't know, maybe like three out of every 10 times you prompted it, you'd get like a pretty incredible answer. And it would almost feel like, oh, my God, like, is it like a real being or a real thing? But then seven out of 10 times, it would just like hallucinate and give you like an absurd answer. And so it's like, oh, it's not quite there yet.

19:57But I would say like probably, you know, GPT is probably when I was like, oh, wow, like it's really progressed and a really big leap has happened. And, you know, like I think we've all kind of been a bit on this journey and it's pretty wild how many things are changing in the short period of time. And so I think the other thing beyond just like large language models and text is like the stable diffusion and the image generation stuff. Like I think as a designer, as like a creative, seeing like the quality of visuals coming out of Dali and MidJourney kind of a couple years ago was like mind blowing to me.

20:44And then now the most recent QPT4, like you basically have like comfy UI embedded at your fingertips without having to do the crazy workflows is also just like kind of unbelievable. Why are you in trying to build like the new kind of worlds that we will live in? So when you have, you know, VR and AR built into the glasses, let me tell you why I ask. One of the things that I found profound when using the Oculus for meetings were not necessarily that the graphics at the time were kind of like, I didn't feel I was there. but what I noticed was that I had also like the depletion of all other inputs and so the meetings were actually pretty intense because nobody had their phone nobody brought their computer like their email up in front of you so it was like going to the cinema like I was having this conversation with the other person and there's the only thing that happened because the only thing I could do was to be there yeah and so as you're thinking about these creating kind of like the world that we will most likely live in in the near future right these worlds that have hot reality and part like augmentation on top of it what are some of the things that you kind of like are most excited about that will emerge and how do you think about maybe even like creating kind of like a yeah environment that is cool for people to be within because that you're in many ways now creating the imagery that we all will see through you know like you know being awake every day yeah i mean that's a good question i mean i use this to go back to jeremy and the demo you had of orion i think the biggest difference between um a quest headset and an orion is the fact that you're present in the world around you right and so sort of early signs of how the physical world and the digital world kind of come together.

22:53And so I think that's like a big part of our vision is that like, there'll be moments where you want to be like fully immersed, like you were mentioning, and there's like nothing you are doing, but the thing you're doing. Um, but I really do see the world's blending together. Um, and I think the trick for, for me, our team, the company, the industry is to like, sort of work alongside people's sort of like attention stream, right? Because we don't want to be doing this chat and then I'm doing nine other things at the same time. Right. But at the same time, it's like very nice for me to like have my phone to my left and be able to be aware of like the people walking by in the office or whatnot.

23:39And so I think a big part of the design challenge is figuring out sort of like, well, one, we want to give people control, right? Over what's the right kind of like flow of visuals, information, interruptions, and coming up with really good defaults, but also like helping people be able to kind of tune these things. So that, you know, I think the last thing you want is for people to feel overwhelmed, but you know, you're also not going to go very far if like people are just really bored. And so I want to help people kind of find the happy, happy medium. What's that phrase? The devil is in the default.

24:22Yeah, I mean, I think I have not heard that phrase, but that that definitely does ring true.

24:31But also, like, you know, like, I think there's like a pretty massive spectrum of AR, right? like AR could be as small as you've got one AirPod in your ear while you're at a museum. And it's like you have your private docent in your ear kind of telling you about a history of a work. It knows a bit about your preferences and the things you've seen in the past. You get this like really discreet and awesome kind of experience. And then there's like sort of like full AR where it's like you're in, I don't know, Van Gogh's studio and you're painting a painting right alongside him. And your entire sort of 360 world is like, you know, you're in a studio.

25:24And and so we think a lot about that. Right. And so many of these things are. in like getting products to people. There's like cost considerations, there's weight considerations, there's social acceptability considerations, there's power, thermals, silicon. And so we think about all these experiences we sort of want to have for ourselves or we want to deliver to people. But kind of along the spectrum, like we have to sort of like intersect direct technology availability and what people are sort of ready for. And so, um, that, that, I mean, that's what makes this work pretty fun and interesting.

26:08You know, how do you think about attention? I mean, I want to come back to, you said, um, we want to work alongside someone's attention stream. There's a real, I would say, I feel a tension in regards to my attention because I, I, I'm sure others have had this experience, but you know, where time kind of evaporates where you go, I did not mean to be watching the YouTube, you know, shorts for 45 minutes. And now it's like, I actually wanted to go to bed, but then like my amygdala got hijacked right or whatever. Um, and so there's, there's this weird thing of like, what do people want? What do they need?

26:43And I think in a more immersive, more blended world, I, I think I fear in some way, uh, my inability to control, you know, what I want or, and what might be good for me. You know, I have to do increasingly kind of savage things like leave my phone outside of the room to be able to go to bed at night, things like that. And I just wonder about in a world that's increasingly integrated, how we, uh, give users or customers the control that I, you know, you use that word control, like the control they need versus the control they want. I don't exactly even know what the question is, but I think there's probably lessons from the, call it the social revolution about unintended consequences of, you know, things that we thought people wanted, but actually resulted poorly.

27:32How do you think about all that? Yeah, it's a really good question. I mean, I think like a person, a human's attention is actually like pretty narrow. And I also think the attention spans of each individual person is also pretty different right just kind of like depending on how you're wired and i don't know like i think in many ways like that's the job like for us right like i think as designers our job is to sort of like choreograph attention almost um and it's sort of a a skill to like be empathetic to how a person's going to be experiencing the things that we're doing and how that like sort of intersects with like the life they're doing, the specific task that they have in and that they're trying to get done.

28:34And so I don't know, like we take that responsibility really seriously right like in in many ways like i don't know if any of you have watched the hyper reality video it it's like this it says youtube video that a group of designers made called hyper reality i think there's this guy named um keiichi suda um you can look it up But it's like really scary. Basically, like it's Black Mirror kind of vibes. You put on glasses and your wearables. And it's sort of like what, you know, 2030 is going to be like where you're just getting bombarded with digital things. And in order to unlock a, you know, 50 cent coupon to buy the broccoli in the grocery store, you got to like collect seven coins.

29:30by watching ad, you know, it's like everything is just like, you're like, you know, is there worse possible version of the future? Worse possible version. But it's like in a, I think it's been like a powerful cautionary tale for technologists. And like, I think especially designers, like we, we get caught with that. And so like, I don't know, I like whenever I'm reviewing design, whenever we're like looking at what features we should launch. Like, I kind of have like hyper-reality video. You're kind of going, that's part of the criteria. Would this make hyper-reality? If so, please reject. On that, I feel a lot of us on the entrepreneurial side, when maybe specifically when the EarPods came out, we were like, that's going to start a whole revolution in new applications and new use cases of basically having these always on kind of like things in our ears.

30:29And I think we haven't seen as many as we've imagined. And some of the, I would say, successes that has been has been this very weird kind of success, like podcast, like who would have known that like long form spoken word content would suddenly be the thing that, or the outcome of like really good headsets. When you are looking at, you know, building glasses, is obviously what you can see is a big thing but what you can hear i would imagine is incredibly important also yeah and meanwhile the tone of the voice the type of voice how they talk when they talk what they talk about all those things seem to be like an incredibly unexplored world because at least i haven't seen too many of those like it's very reactive right now like hey meta what do I always see?

31:19And you know, and it tells me what I see and all like you have a notification, which is basically a readout from whatever I'm used to have my phone. What do you see kind of like the next few steps are in the, in the audio side of all these amazing devices that you're building? I mean, to take a step back, right. I think in many ways, like designing for screens have been a lot easier because it's like you assume that that's the thing that they're paying attention to and so you kind of have their undivided attention i think designing on a ai and and wearables is like you kind of want to like slip stream into people's attention kind of moments right and so to your point like if you can see what someone sees and hears what they hear you should never give them like a group on coupon when they're in the middle of like giving a big talk right it's just like hey not a good time well you're saying is you know it actually that's interesting if you when you say if you can see what they see in here they hear what you mean is as the designer of the product if you know their context then you can you as the designer can be contextually aware about what you're serving up is that is that what you're saying yeah totally Yeah.

32:36And like, that's what the person wants. Right. It's like I actually am like a huge fan of advertising, like when it helps me find a product that I really want or is helpful to me. But like, please tell me about the product at a at a good time. Is there an interruption there that's like, hey, man, like in the middle of the talk, slow down, pace yourself. Right. That's actually a pretty interesting, contextually aware suggestion. Right. And speaking of audio stuff, right, I'm I'm the most annoyed these days where I'm saying like, hey, thank you so much for having us for dinner. Exclamation mark.

33:16Can't it? No, I want an exclamation mark at the end because of the tone in which I'm sort of dictating. Right. And so if you think about like sort of how high the ceiling is for how much better these things can be and how much less friction it'll be to help people communicate and get things done. It's like super exciting. It's super fun. So I want to turn harder towards AI just because I think maybe some people are going, wait, hang on. Isn't this beyond the profit? Because we've been like design. We've been in lots of lots of great places. Can you talk for a second, Josh, about how your team or how you even personally leverage generative AI in your work?

33:57And one kind of seed I would just put there, if it's interesting, feel free to run with it. If not, feel free to take it however you want. But I heard Mark talking recently on a podcast about how the Lama team in developing AI is really using a lot of internal AI products, which are advancing their development, which is great. And so he said there that the llama team's general kind of use case is not the world at large and kind of any general use case, but really for almost, you could put a better word, dog fooding or dog AI, right? Yeah. Do you find your own experience, you know, somebody who's not coding necessarily or primarily, do you find that you're able to kind of leverage AI models in unique and differentiated ways?

34:43or are you kind of doing more off the street, off the shelf retail kind of stuff? Just tell us how you're in the design team are leveraging AI. Yeah. I mean, I think just to give you all like a little bit of a view under the hood, we're sort of just like super curious about it all. And for sure, I think if nothing else, like I don't think AI replaces taste just yet or AI isn't going to sort of do our job for us. But boy, is it like an incredible sort of like brainstorming partner imagination machine, right? And don't get me wrong. I do think like AI also does like quite a bit of like speeding up of some of the more repetitive tasks or even some of like the early steps of design tasks, right?