What OpenAI taught me to unlearn — Blaine Billingsley (OpenAI, Slack, YouTube, Airbnb)

0:00Blaine Billingsley: Is it a goal of yours to make more money in 2026? My name is Travis Chappell and I'm the host of the Travis Makes Money podcast. It's the only podcast on Spotify that brings you two episodes every single day on how to make more money. From side hustlers to real estate investors, brick and mortar owners to online creators, beginners to billionaires, every conversation will give you the ideas, inspiration and practical tools you need to make more money and earn your dream life today. Subscribe right now to the Travis Makes Money podcast on Spotify and I'll catch you over there.

0:32Randy Silver: How do you rebuild after a setback? Find your next direction? Or understand someone whose life looks nothing like yours? On Let's Have This Conversation, journalist and motivational speaker Kevin McShan meets the people behind those questions. From resilience and well-being to careers, leadership, and human connection, discover personal stories and expert perspectives that could change how you see your own life. there's a lot we can learn from each other. Let's have this conversation. Search for Let's Have This Conversation wherever you listen to podcasts or click the link in the show notes.

1:21Blaine Billingsley: That's pretty cool to come back from lunch and have like 80 ideas mocked up for you. I've worked at Gmail, Airbnb, YouTube, Slack, and now here at OpenAI. Where we're focusing now is like going beyond building what you already want and helping you actually like think through and ideate new ideas and make better ideas. What I think is really cool is all of the ideas on earth are available to an LLM. What makes ChatGPT for work really useful is you can have it cover a much wider range of techniques. We focus a lot more on the present moment because the kind of fear of time has gone away a little bit.

1:57Blaine Billingsley: Like we know we can whip stuff up fast. We know what the company's goal is. And everybody, no matter where you're connected to that goal, it might be really far away from that goal. You're all contributing to that. The thing I've been working on is product design plugin. I just asked ChatGPT to go browse the web and find some cool ideas. And in about 10 minutes, you know, it pulled a bunch of ideas, a bunch of references.

2:18Randy Silver: What kind of advice would you give us on day one about what do we need to unlearn? What do we need to learn?

2:23Blaine Billingsley: The titles don't make sense anymore. Things are shifting. We're in a moment of transition. Just focus on who you're working with and what you're doing and good things will happen for you.

2:35Randy Silver: Blaine, how are you doing today? It's great to see you.

2:38Blaine Billingsley: It's good to see you too. I'm doing great. So thank you for having me.

2:42Randy Silver: Yeah, thank you so much for joining us today. So we've had a brief chat before this, so we got to know each other a little bit. But for anyone who doesn't know you already, do you mind just giving us a quick introduction? How did you get into this space? What are you do in these days? And what was your first design slash product gig?

3:00Blaine Billingsley: Yeah, so I'm Blaine. I'm currently a designer at OpenAI. And I didn't like study design or engineering formally. I have a degree in music composition. But my grandfather was a technologist. He worked at NASA on all the Apollo missions. So we had like computers around a lot. I was really lucky that I had a lot of technology at my disposal as a kid. So in high school, I would just like make websites and posters for my bands mostly. And then in college, I made websites for spare change around campus. And then I moved to San Francisco and got a job working data entry at a startup and then just kind of learned on the job from there from engineers and designers.

3:42Blaine Billingsley: So since then, I've worked at, you know, Gmail, Airbnb, worked at a couple of startups, been through two acquisitions, worked at YouTube, Slack, and now here at OpenAI.

3:54Randy Silver: Fantastic. And we're going to talk a bit about a new feature that you've helped develop. But before we get into that, I'm just curious, coming from a design background, you've worked at all these jobs in places where design theoretically is a more traditional pixel and interface-based thing. But OpenAI and all of the the LLMs that we're working with, the interface is mostly a text box. So what is the role of a designer when you're moving from Slack and Gmail and things like that to OpenAI? What is the job that you're doing?

4:31Blaine Billingsley: Yeah, totally. I know a lot of my family is like, what do you do? Isn't it like finished? You know, there's not that much to it. To some degree, there's truth to that. There's a lot of stuff that's similar. You know, like Slack is an open-ended text box with some non-deterministic inputs and outputs. Also, just comes from humans. And Gmail, same thing. So there's a lot of familiarity with conversational experiences that I've had in the past. I think the biggest thing that changes is like a lot of my job is more about what the output of the content should be. What evals are we looking at to grade success in the first place, actually kind of deciding how we measure ourselves?

5:12Blaine Billingsley: And then doing that heuristic evaluation of like, do we actually editorially think, you know, this is the format or structure that we think is a good output? That's a huge part of my job that's kind of categorically different than it's been in the past. but a lot of design has always gone beyond just like the interface and the pixels and figuring out what features we need and how to how big should they be how small they should be how should they feel and all that still you know that all still totally applies so let's let's talk

5:43Randy Silver: about the new thing that you've just launched so i'll ask you introduce it and then let's talk about what it was like to work on it and what does a team look like what is it what did you actually do on a day-to-day basis and not just you but the whole team how did you all work together but let's set the scene first. Tell us about the new launch.

6:00Blaine Billingsley: Originally, I was hired at OpenAI to work on a different product that actually just got announced called Presence. But the last couple of months, I got moved to work on how to make this new ChatGPT work work well for designers and make it a daily driver for designers. So the whole premise behind this merge is that Codex is really great at writing code, but that actually means it's really good at doing kind of anything with a computer. And ChatGPT, with that new harness and model underpinning it, can do a lot for workers, whether you're an engineer or a designer or in marketing or in any profession.

6:37Blaine Billingsley: And so I was the quote-unquote subject matter expert to sort of figure out how to fill the gaps to make it really useful on a day-to-day basis for designers or for non-designers who need design.

6:50Randy Silver: okay so uh chow chp t being really good at doing anything on a computer that is given the news of the last week and with what happened with hugging face it's kind of kind of a short selling it but the reality is obviously that was all safety measures turned off it was not the nor the consumer product by any stretch of the imagination but it just goes to show there's a it's infinitely malleable the the way that these things work but and also hugely unexpected. So what's it like when you're sitting there and you're trying to determine, you know, create an experience for people of this is good content.

7:27Randy Silver: This is going to be a good experience. This is useful and safe and good. How do you approach that job?

7:33Blaine Billingsley: Yeah, it was really interesting. I mean, I had the luxury of working on the job that I've had for, you know, 15 or 16 years. So I've had this, you know, just background of what it's like being a designer at small companies, what it's like being a freelancer, what it's like being at a huge company. I've been a manager, been an IC. So, you know, at first we were just kind of relying on my experience of what it's like working at companies as a designer, what's hard, what are the things every designer wishes they could do. And we talked about it. It was, it was just me and one other guy, an engineer and myself working on this.

8:09Blaine Billingsley: So we were just like chatting about, you know, what is, what do designers do every day. You know, we're in Figma 90 % of the time making mock-ups. You know, we're trying to make prototypes all the time. And we're gathering requirements. We're listening to research. Sometimes we're running research. And we just started kind of thinking about like, well, where do we want to, where do we want to focus? Do we want to help Codex get a lot better at using Figma? You know, do we want to focus on what it's already kind of like great at, which is writing code? Do we want to focus on all the minutiae and day-to-day around being a designer, like gathering context and writing briefs or clarifying briefs.

8:45Blaine Billingsley: And we explored a lot of stuff. Like we tried a little of that and some more over the first few weeks. But we ended up kind of always coming back to leaning into what Codex is already really great at and what was coming to ChatGPT in the near future at the time, which was computer use. You can kind of use your computer on your desktop on your behalf. It's really great at making images. The new image models are absolutely incredible at interface design. It's like a zero to one shift from where it was when I first joined. And if you kind of lean into that stuff and of course writing code. So if you lean into those three things, you're taking advantage of what it's already been trained on and what it's naturally going to be really good at.

9:29Blaine Billingsley: And it turns out there's a ton of stuff that you can do with those capabilities at its core. So from there, thinking about it through that lens where we ended up landing after a lot of exploring a bunch of different kinds of workflows that some were really hard, some were really easy. We landed on first and foremost, just helping designers do what kind of codecs traditionally was already good at, which is just make code prototypes and how to do that faster, make more prototype-y decisions and give it a little bit of bootstrapping so that it could do stuff really, really nicer than it would just one-shotting it, how to help it kind of find your design system, et cetera.

10:09Blaine Billingsley: So we explored that for a long time. I'll show you a little later. We spent a lot of time trying to build a prototype starter kit builder, which you can kind of do in Codex on your own. We kind of realized we don't need to do too much extra, and you can kind of do that. And then we did spend a lot. That kind of brought out a lot of different smaller workflows, like being able to send ChatGPT off to take a bunch of screenshots of your app and audit the entire product surface was kind of an experiment that came out of that process of building these baby apps, that actually has a ton of uses. If you're a design system manager, you're trying to audit a certain component, or if you don't know how a certain flow works and you want to have it recorded.

10:53Blaine Billingsley: So yeah, like a lot of little mechanics sort of came out of just like, how do we get people building code prototypes easier and faster? And where we're focusing now is like going beyond just building what you already want and helping you actually like think through and ideate new ideas and make better ideas, hopefully. So that's where we're focused now.

11:13Randy Silver: Let me ask you about those three stages, prototyping, auditing, and production. And I'm not sure if auditing and production go in that order or the other way around, or auditing is kind of a continuing thing. But let's start with prototyping. So traditionally, when we are doing a new product and we want to go really wide, we've got the double diamond approach, We want to go as wide as possible and explore. And LOMs, you know, they're all trained on prior art. And granted, so are we, but we're trying to break out of it at the early stage. So do you find that when you're doing this, that it is more of a constraint or is it an enabler?

11:54Randy Silver: How do you maintain activity? And when you're working with an LOM, which I find, generally, I find that it's trying to conform.

12:04Blaine Billingsley: totally right right yeah yeah I know what you mean um you know I hope it's okay to go on this like personal tangent for a minute but you know when I first joined the the world of software design I was so um like uh I just felt like I I didn't have the skills you know I came from a music background and I was just kind of learning on the job and I had this like I don't feel like I'm naturally a creative person and there were some people who would just like dream up like the most amazing stuff. So I spent a lot of my early career working on like really structured ways to be creative. Actually, even in music, that was something that I worked on was, you know, if you've got a commission and you have three weeks to do it, you don't feel the inspiration, you've still got to write the music.

12:49Blaine Billingsley: And, and there are techniques to just get material out of you. And I've always kind of relied on that. I like this insight that I've cobbled together from a few sources over the years of like creativity is you touched on one piece of it remembering what worked and what didn't and why in the past just sort of like what you've sponged up over the years it's also kind of forgetting your principles sometimes or forgetting your requirements and sort of saying like I know we've been saying this can't change but what if it did change like what would we get and then the other piece of it is just kind of arbitrarily throwing new directions at you like take out two pieces.

13:31Blaine Billingsley: You know, one of my favorite design leaders would always tell us, make it 30 % simpler. And it didn't really matter what you did. You just had to take a third of it out. And there's all these different techniques and methods that people do to sort of like, bring that out. What I think is really cool is all of the ideas on earth are available to an LLM. And all it really takes is a little bit of patience. I think a lot of times, even in our demos, we try to show these like one shot approaches of like, make me this thing and it'll be perfect. But that's actually not how you find the best thing you sort of like poke and, well, what if we do this?

14:06Blaine Billingsley: And what if we say this thing that was important isn't anymore and play around? And what makes ChatGPT work really useful is you can have it cover a much wider range of techniques while you're at lunch, you know, so a lot of times I'll do a lot of my own ideation. and I've got to go to a meeting and I'll ask it to do some structured exercises for me and it'll generate a bunch of things on my behalf. And I can come back and be like, oh, wow, that was a total, totally not useful. And that one or two things is really cool and inspiring. And so I think just having a little bit of taking that stage seriously of like giving it the space to think about how it wants to be creative first and then relying on like generating lots of things goes a really long way so one of the things i would normally do like at

14:56Randy Silver: the in during a design sprint or in a kickoff might be the crazy eights exercise where you're generating each things in eight minutes eight designs in eight minutes and you've got you know it's a huge concern you've got no time and each one's got to be different so you're getting really funky by the end of it and you're getting a room for people to do it and it's just opening things up what kind of thing might you do in this case where you've got you know i mean you you're in the privileged case where you've got infinite tokens so true what what kind of thing are you doing to say hey i want to explore and not get locked down at an early stage totally yeah i'm glad you

15:35Blaine Billingsley: actually brought up the like the sprint techniques because you know those are sort of designed for like a whole team of people on a really short time frame to cover as much ground as possible. And those are, I think that's like a good framing for this stuff. Like the crazy eights exercise, it's really hard to come up with eight ideas that are coherent in eight minutes. It's really hard to come up with like 80 ideas in eight minutes, but an LLM can come up with 80 ideas in eight minutes. And you know, the whole concept of that crazy eights exercise is that it's all about volume. It's like, it's not really about finding like the perfect idea and being really precious.

16:14Blaine Billingsley: It's just like out of the insane volume that you'll get, like there'll be some nugget in there that's going to stick. And so I don't know if you've ever heard that, like, you know, the best way to make a pot is you can either try to make the best pot or make a thousand pots and a thousand pots approach always wins. That's how I see, that's like the superpower of being able to use a chat GPT or an LLM for creative exercises, they can just cover way more ground. And there's going to be like terrible ideas. There's going to be a few that you're like, okay, that's a little silly, but it sparks something in you and you can, you can iterate from there.

16:50Blaine Billingsley: And that's really useful. The other thing I always like to talk about is like re-imagining these processes or like at the end of that exercise of sprinting, everybody has to vote, you know, You get three votes and you put them on your favorite idea because it's really hard to make those ideas. Even just to mock them up can take a long time. And with an LLM, you don't have to vote at least as much. You can skip that or radically speed up that sticky note to low fidelity to hi-fi prototype process. You can get really rich representations of your sticky notes while you're getting a bite. And that's pretty cool to come back from lunch and have like 80 ideas mocked up for you.

17:34Randy Silver: It's great. But one of the other key things you said in there was everyone gets a vote. And one of the interesting things is, you know, we're now getting to a point where I'm seeing in startups and scale ups a lot more one or two person teams. And everyone is just me kind of thing. Yeah. You said earlier it was this new work on this. It was mostly you and an engineer, at least in the beginning. So what is that like? What does that do to you when you're in the development stage? Do you, I mean, it's the two of you and any number of agents, but it's not the same as a room full of people, is it?

18:10Blaine Billingsley: Totally. Yeah, I mean, it's like, you know, having like the most helpful coworker as an agent is awesome. But the other key ingredient in those like, you know, we'll keep on the sprint train for a minute. The other key ingredient is like all the friction, right? Of like, oh, I don't like that. I like this other idea. And why do you like that other idea? And that's really helpful to have. And you can kind of, you know, you can get that to some degree with an LLM, but there's nothing like, you know, real humans with real passions and real experience talking things out. And you are right that they're like, we're so much more empowered than ever with these tools to do stuff on our own.

18:52Blaine Billingsley: Like I, I submit way bigger pull requests than I've ever been able to directly to production. I don't have to ask anybody, you know, there there's, um, I mean, I'm empowered to do a lot more than I ever have. Other people are empowered to mock up all their own ideas and it, it does run the risk of you operating in your own little world. And especially in the design sense, like the, the fun part is like, is spitballing with people and like, Oh, what do you think about, you know, like. For me, I like just seeing somebody else be like, oh, yeah, that's it. That's what I was talking about, you know, and you can lose that.

19:29Blaine Billingsley: But I think, you know, the other side of that coin is everyone else is coming to those group meetings with a lot more fleshed out concepts or they've done a lot more sort of background side quests. it's actually scientifically proven that like the best group creative thinking is going away on your own and then coming back and reflecting you know it avoids group think especially if you have a good process for sharing once you do come back it really levels the playing field so you do have to kind of make sure you don't lose those rituals that bring you together to share in our case since we were so small it was really easy because we're just two two people chit-chatting every day and sharing each other stuff.

20:15Blaine Billingsley: But in a way it's, you know, for me, at least I can't speak for anybody else. It's fun to be like, Oh, look at all this stuff I made, you know, like I tried this and that didn't work. And I thought this was cool. And then, you know, my, my coworker will say like, Oh, I think I can get that going. Like, let me, let me, let me, let me try if you can get it to work. And I just feel like we're coming with a lot more like material to the talk instead of like doing planning artifacts. You know, there's, I haven't made like a planning deck since I joined open AI. I feel like we focus a lot more on the present moment because the kind of fear of time has gone away a little bit.

20:52Blaine Billingsley: Like we know we can whip stuff up fast. And we're just talking about what we're making actually in more real time. So I think as long, yeah, you do have to watch out for it though. I mean, I think it is easy to like be less collaborative if you don't keep those rituals to to share and show and tell you know one of my favorite uh things images i've seen recently is you know we've got the classic iterative development thing of you know there

21:19Randy Silver: was the waterfall of you do each part of a car and you assemble the car at the end right the agile of doing the skateboard the scooter the tricycle whatever and then there was uh the the llm version of starting with homer simpson's supercar and then going through an act of refinement um That's funny. You can do totally wacky at the beginning, and it's the cost of imagining is so low. Totally. So if it's you and an engineer on this team, the two of you, and you're coming together, and as you said, you're submitting poll requests, they're able to iterate on design with their comments. What is your job anymore?

22:00Randy Silver: What's the difference between your job and their job and someone else on the team's job, potentially?

22:05Blaine Billingsley: Oh, yeah. This is like the we talk. This is like the lunch talk every day of like, what are we like? What is this? And like, is this what it's is this what it's supposed to be like? Like, is this good? I candidly, I mean, I often I'm like, I'm either like, you know, a few months ahead of the curve, or I'm just like way off in left field doing a totally different job than I'm supposed to. and yeah we talk about this a lot internally of like I'm either like the worst engineer on the team or you know we're all kind of project managers and it's a very strange thing what I like about it as like as a manager I always liked kind of an equalized ritual so like it wasn't like everyone rotated taking notes everyone rotated being the lead on a meeting and I kind of like the idea that, you know, we're, we're, we all have some specialty, maybe in our experience, and we can help in one place or another, more or less, you know, like, I can't get a pull request to pass if it's failing a bunch of CI tests as quickly as my engineer friend.

23:11Blaine Billingsley: And also, if we really need something dialed in, I can do it in about 10 minutes, and it would take him a long, a longer time to do it. So we still have specialties, but I like that it's a lot less specialized. I think a lot of times at jobs kind of at their worst, the fact that I know like how to physically do something in Figma means no one else that if you don't know how to do that, then you can't share your idea in the same fidelity. And I, and I, you know, I've seen that at companies all the time. That's used as a gatekeeping mechanism of like, well, I'm doing the design, you know, or, or, you know, no matter how cool I make the prototype, if the engineer doesn't want to build it and doesn't want to do, you know, the transition, they don't have to, because they're the only one with access to the code base.

23:58Blaine Billingsley: So I kind of like that it's like this slightly more even ground. Although, right now, I will say, I don't really know, like, my specialization feels so much more vague than it ever has. And the one thing that's new for all of us, at least for many of us, is how much just evaluations of like, run this thing a bunch and see what comes out and evaluate either heuristically or through some other rubric, like, are we getting what we think is a good result? That is a huge part of my job that has never been a part of my job in the past. And it's a part of everybody's job. But it is very strange. I mean, And it's a little bit of just like, it feels like a band of just people trying things out.

24:48Blaine Billingsley: A lot less like, you know, I have this function, you have this function. It's a very different experience.

24:55Randy Silver: Let's stay micro on it for a minute, then go macro. But you just brought up evals. And, you know, it's not something I've done a ton with yet. But every time I talk about them with people, philosophically, it's test-driven design. It's behavior-driven design. Evals is essentially the same thing of how do we describe what good is. So talk a little bit about how you've learned to use evals, how you use them in this product and in this development process. What does that look like day to day for you?

25:25Blaine Billingsley: Yeah, totally. It actually was really a confounding thing for me because I'd worked on AI features in the past. At Slack, I was the director of AI. And so there's a ton of AI features there. And we had an eval process and everything. but I wasn't the one doing it. You know, it was a little bit of the more traditional breakdown of specialization there. And like Gmail, I think we launched Smart Replies in Gmail, which was like one of the first Google brain powered features. And we ran through the spreadsheet every day of training data and said what was good and bad. What's different about this is it almost feels like it's like synthetic user research in a way.

26:02Blaine Billingsley: It's like you get, depending on what you posit people are going to do, here's what the experience is going to be. And is that good or bad? The hard part is positing what people are going to do the right way. And I found myself getting really hung up at the very beginning of just, well, what are the prompts people are going to put in? And early on, I would write out some stuff we were trying. We would hill climb, as they say here at OpenAI on that and sort of get that working really well. And then we'd share it internally. okay, we think this is working really nicely. The very first thing somebody tried was like some marketing website style we had never thought about and it did really bad.

26:45Blaine Billingsley: And they're like, okay, what have you guys been doing for the last week? And so I think it really is eye-opening how limiting our understanding of like, in my mind's eye, I think I know what this is going to be used for. But literally even just sharing to like a third person inevitably opens that aperture up to be a lot wider. And so the big lesson for me in learning with evals was not being, you know, getting a set, trying to continually make other sets kind of like throwaway sets, widen up that aperture as you can. And I see online when I go on YouTube, a lot of talk about like having a golden set and using that for regression testing, you know, where you have a big long list of prompts and you make sure that they kind of don't break as you iterate.

27:30Blaine Billingsley: That's super important. but actually the, you know, once you get that, you'll build that over time. The more useful thing is like, what am I working on now? And like, what are all the different things that it could be used for? And what do I want failure to look like? What do I want success to look like? And how can I share this as quickly as possible with somebody else to see what they're going to ask it to do so I can check my assumptions? I think that's where we got tripped up early on, And just kind of, it's so hard to imagine beforehand, like all the different things people are going to try to use this plugin for or anything for.

28:08Randy Silver: Yeah, I mean, Gmail and Slack, you've got experience with this, with going out to an incredibly wide user base. And traditionally with product development, when we're trying to do something, we start with, let's thin slice this. Let's do this for a narrow group. Let's do prevent and that it works. you can't really do that with something that's wide open totally and you talked about synthetic users a moment ago what's how are you handling you know with this kind of volume this kind of infinite variety how are you using synthetic user testing versus real user testing yeah good

28:42Blaine Billingsley: question you know what this is actually really interesting because when i my first like big kid job was on gmail and we at the time we were redesigning it from scratch it was this product called inbox and um it was like redone from from the beginning and i was like just over the moon to be there and but i was also terrified i had not i had i was a really young designer at the time and i had only worked on like little little stuff and none of it was that good to be honest so um i remember asking my manager at the time like how do you how do you do this there's like you know over a billion users of gmail and like like i don't know how to think about this this is too hard And he made this really good point of like, when you go study an HCI program, you learn about like researching your users, understanding their workflows, building features for them.

29:33Blaine Billingsley: And it feels a lot more scientific in a way, because if you're building software to do invoicing or software for a very particular kind of person, you can really dial it in. It's like part of the joy of HCI is you can really understand what those people need. when you're building something like Gmail or I worked on YouTube on TV, which, you know, half the world's internet population goes to every day. You can't think in those traditional terms of HCI because the scale is so big and you have to think about it more of like general mechanics and how those mechanics fail and succeed at their extremes and, and just kind of bake in these like simple mechanics of capabilities into the product.

30:17Blaine Billingsley: And that really, it took me like years to really kind of like let that bake into my brain and understand that. But I feel that so much more here where there's no way you're going to cover the ground of what people do. And so it's not worth trying. The trick is to try enough different stuff that you're sampling gives you some perspective and to focus just as much as you focus on what good looks like, how does failure look? Like what's the worst it will do? And do you feel okay with that? You know, we used to call it on inbox, like where we declare bankruptcy. You know, we want declaring bankruptcy.

30:54Blaine Billingsley: We want to know what happens when the app declares bankruptcy and is just totally borked. We don't want that to surprise us. So that's kind of my take on it is there's no way to cover everything people are going to do at a scale this large. The best you can do is pay attention to the failures and try to sample as wide as you can and start to see with these few mechanics that we offer, like what does it do good? What does it do bad? And how can we sort of cushion the blow on the bad stuff?

31:25Randy Silver: So are you actively doing the research on this with synthetics, with real people, with production data? What does that actually look like?

31:36Blaine Billingsley: Yeah, good question. You know, So this particular work, working on this product design plugin for ChatGPT, I feel actually a little embarrassed talking about how we did because it's not how the rest of OpenAI works. And it's not like I wouldn't put this out normally as like a job interview of how we did it because it was really like experience and vibes based early on. Like I said, it was like we don't have a ton of time. We don't have a ton of access. We don't have like, we don't have a bunch of companies we can get to sign an MDA and use it. We don't have anything to use. And so it was really like, based on my experience talking with, you know, the design team here about how it would work for them now.

32:22Blaine Billingsley: What was it like at their previous companies and just doing like the best research we can with the people closest to us, which I think is really valuable. I think it's really easy to kind of get in your head about like, oh, I can't, I don't have access to like real users or it's hard for me to see what real users are doing or it's hard to get that information. But like any information is good information. And so we, you know, we looked at that. We read on the community forums, you know, on Twitter and Reddit. We just sort of like soaked up, like what are people talking about and tried stuff out.

32:57Blaine Billingsley: And if we liked it, if we thought it was good, then we would show a couple more people at work and they would, if they liked it, we'd show a few more people. And then we'd build like eval cases to cover all the stuff that it did really bad on. And then we'd start, you know, poke around at why it was bad at that and explore some more. And for example, like icons, it was like so bad at getting icons. It was like, it could do so much. It was like magical. It would build an entire app for you. But like we got stuck on like the weirdest little stuff sometimes. Yeah, so we did it really like kind of guerrilla style.

33:33Blaine Billingsley: Like if, you know, we were just two people, we didn't have a lot of access to other companies. At the very end, right before we launched J2PT Work and released the plugins, we got a couple of companies signed up to just like try it like literally a couple of days before and just like did it, you know, did it turn on basically? Did it work? And did you get something? And then after that, it was just lots of post-launch, lots of reviewing, like classifications of prompts. So we don't have access, except if people have opted in, in certain cases, to the actual prompts themselves, but we can get classifications of them.

34:11Blaine Billingsley: And so we get these like breakdowns of like what types of things people were trying to build. And then we would just sort of start making up our, you know, examples and testing and seeing how it went from there.

34:22Randy Silver: one of the things i've noticed in playing around with with various tools and and hearing from other people is you know that 40 first 80 percent of product development getting to prototype stage is so much faster and so much more efficient but unfortunately that second 80 percent of going to production is still really hard so what was that like for you and as you're playing around with this and trying to work in a new way what was that like we talked about prototyping initial stage 100 % true.

34:55Blaine Billingsley: I actually think, you know, it's really, I love that, that saying of like the second 80 % because it's just so spot on. Like in a day or a weekend, you can be like, oh my gosh, I got this. This is great. And then the next few weeks of your life are just like, absolutely grueling. We certainly had that experience. You know, one of the things about open AI that's really fascinating, I touched on a little bit earlier, that kind of blew my mind was like there's not a lot of like roadmap planning, you know, like at my other companies, we'd have these like huge cross team request processes. And every quarter, we'd like try to align priorities for the next four months.

35:33Blaine Billingsley: And so one of the things we had really optimized for ChatGPT desktop app, which is local on your machine, it can do a lot of magic, because it can control your computer, it can store things on your local device. And then we found out we're merging these two products together and bringing this harness into chat GPT on the web totally different thing and it was I they did it in like three weeks or something I mean it was the most insane timeline for like such an insane project and so then it was just a scramble of like oh we were testing all of this in one environment now we got to test it all in another environment and rewrite it and make sure it works in both places and yeah you know it's uh that's I'm almost like normalized to it because that's actually just every day of like, you're just trying to get to like making it work based on what the priorities of the day or the week are because things move really, really fast now.

36:27Randy Silver: So a lot of what I do as a consultant is working with organizations as they're scaling up or, or they're large and they have the issues around communication and collaboration and prioritization at scale. And you just touched on that. When things are moving at such rapid pace and anyone in the company can make changes and is encouraged to be working on making. How do you get that alignment? How do you maintain an organization to be focused on, you know, it may not be OKRs, it may not be roadmaps, but what are you doing to try and create some sense of alignment so that there is a cohesive whole?

37:06Blaine Billingsley: Yeah, it's an interesting question. And I don't, I've seen so many different approaches. You know, I've worked at like 10 person companies where we're all friends and it's like, we just have to kind of look at each other and we're kind of like, okay, we'll do that. We'll do what you said, you know. And, you know, I've worked at places like YouTube where they have a huge responsibility to a massive scale of people. The process of alignment is very formalized. And then here, you know, at Slack, it was very different. And Slack was maybe a little more actually how it is at OpenAI in some ways where something would become a priority and like everybody would swarm on it.

37:42Blaine Billingsley: Like, you know, from every team, everybody kind of played their part. But I like that feeling of like, okay, we know what the company's goal is. And everybody, no matter where you're connected to that goal, it might be really far away from that goal. You're all contributing to that. And I find that to be a good rallying cry, especially when the pace picks up of just like knowing really clearly, being able to say really clearly, having anyone at the company being able to say really clearly what the one or two most important things the company's doing are. and having everybody working on that together is a little chaotic it's a little rough at times but it's like all that alignment pressure falls away and it's suddenly everyone's helping each other and I'm a little idealistic in that sense but I like it when it's like okay we don't have to argue about is this important or not we're just figuring out how we all fit together and I think especially with the speed and momentum moving that's the only way to do it is say like Like we know what the rallying cry is.

38:48Blaine Billingsley: And now everybody just play your part. You know, to me, I think where companies get hung up is when there's like cross team competing priorities. And then you have to, you know, that kind of stuff ends up becoming like an internal struggle as opposed to like, well, what is the company, what does the company need to do right now? And maybe what I, my team needs to do isn't the most important thing right now. So how do I contribute to the bigger, the bigger thing? I think just developing that is the way to get rid of a lot of unnecessary strife.

39:18Randy Silver: Yeah, it's one of those things that, in my experience, culture-wise, it works really well when you're in the exponential growth phase. But once things start, it's a lot harder to maintain that attitude and things turn different. But we've talked a lot about how you've built this. We're going to try something new on the podcast because we haven't done it very much. We are recording video. We are on YouTube as well. So, Glenn, you're going to show us a bit of what you built as well, I think.

39:47Blaine Billingsley: Yeah, I'd love to. I hope it goes well. Let's see here. So I did this a little like a cooking show style, just so you don't have to sit here and watch my ChatGPT think for the whole time. But I wanted to give you a little bit of a breakdown. So I'm using the desktop app of ChatGPT, which, as I've mentioned a few times, recently merged between ChatGPT and Codex. It's all the same harness, kind of under the hood. And the desktop experience, especially for workers day-to-day, I think has a few really cool abilities because it's actually on your computer and can do things on your machine. And the thing I've been working on is this plugin, which I'll just show you quickly so you can kind of get your brain around it, this product design plugin, which you can install once you download ChatGPT.

40:40Blaine Billingsley: And I'm just going to show you a couple of the ways that I like to use it and the ways that it'll work best just to give you some inspiration, I guess. So this is probably the most boring one, but it can save your preferences for you. So at any time, you can tell it like, remember where I like to get inspiration or remember where my design system is or remember where my project, you know, source of truth is. and it'll remember that. So it'll make things quicker if you're doing the same thing over and over again. Here for this set of demos, I just had it remember where I like to get my inspiration, like Twitter or a couple of other websites.

41:17Blaine Billingsley: So that's a really nice thing about it. It just will make your life easier. Once you're using it, you can ask it to do that at any time. But one place that I always start is just kind of like looking around at what's out there. And so I'm starting a new project. I want to make a chore app for my wife and I. I just asked ChudGPT to go browse the web and find some cool ideas. And it actually opens up Chrome. It opens up like 10 tabs and it goes hunting and pecking for stuff that might be useful. And it got a few things. I asked it to get a little more from Twitter. It just random design because it focused a little more on chores specifically.

41:57Blaine Billingsley: I told it to make sure to check out the people I'm following. and in about 10 minutes, you know, it pulled a bunch of ideas, a bunch of references, and it made this fig jam for me. Another thing I should call out is this is a browser. You know, I can go to any website I want here, including tools like Stigma or Linear, and you can actually just use it right next to your thread. And it can kind of do stuff on your behalf. And there's ways to connect those even deeper if you want, but even if you don't do that, you can literally just have it like click around for you, which is really nice. Now from here, I might want to, yeah, let's see.

42:38Blaine Billingsley: I'll show you really quickly if you ask me to organize it.

42:43Randy Silver: To organize the Big Jam mood board for you. Yeah, exactly.

42:47Blaine Billingsley: And so you see it's kind of taken over this tab. We've got a new player here, this little cursor, and it'll start moving stuff around for me and doing stuff. So it added all the sticky notes. It added all the stuff. I didn't touch the Figma once. And you can do that. It's taking a little longer than I'd like. Might be an interesting time. Live demos are always. Yeah. So, yeah, you can see it's making a section for me. And this is amazing. Like, in my personal life, I have it go, like, fill out forms for me and stuff. And, you know, it just browses the web on my behalf and does all the boring stuff for me.

43:23Blaine Billingsley: It's really cool. also i highly recommend uh playing around with just what you can get it to do um just a little preview of that so then from there um you know the next step is like hey we got some inspiration come up with some ideas for me so this goes back to your earlier question um i just forked this conversation from the mood board and um and then asked it to mock up some vibes i told it to make 10 different styles to your point about how to make it be creative i gave it a thing yeah i gave it a few different things but it was like go crazy and make 10 versions and uh i got 10 versions here that i can play around with that are all very very different you know i've got this kind of like nice the ai khaki vibe um very popular um we've got i like to this this kind of web 2.0 one a lot too.

44:16Blaine Billingsley: This one's really cool. Windows 95 is a mobile app. It's something we probably don't need. But you know, you can get it to do whatever you want. And you could keep going forever. I like to think about the next phase of like, you can either go deep on an idea, or you can keep riffing and stay wide. I'll show you kind of what that looks like across the board. So if I like this web 2.0 approach, another one thing I like to do is like, well, hey, mock out the flow and this is where i actually find chad gpt starts to think about like well what is this thing gonna do you know some of these you're just getting a vibe you know like the done button here makes makes very little sense on the home screen and it has it on a lot of them but you're like getting a feel like okay i like that feel now actually mock up what the flow might feel like to do a feature um i think that's what i i asked specifically i said yeah i love the web 2.01 mock up what the next step uh what should happen when i do the next up feature and then mock up all the key screens uh of the bottom nav and so it did that for me i mean this is like mind-blowing in 10 minutes there's no way i could get this much going of like this uh home screen i get the whole flow of what it's like to mark something complete all the other key screens

45:35Randy Silver: something like this what i found is there are times when i've been working with tools and it does something like this and it just makes some weird assumptions oh totally things that are that are definitely all right so it's really helpful as a starting point and then i want to mark it up and i want to refine and i want to work with it uh and with other people but the temptation is well it's done isn't it you know just go ahead and build this and ship it how do you totally when you're working with other people how do you remind them that

46:09Blaine Billingsley: this is not production ready you know i think the best i love that question one of the best ways of doing that so like that whole idea of like you know right now we're sketching basically right it's just like a much higher fidelity sketch and like you said like there's a lot stuff in there that doesn't make a ton of sense and that's okay like that's actually part of the process the whole point of that like rapid cycle of of ideating and prototyping and rinse lathering and repeat is actually not to get stuck in one part of that cycle so i i have a little bit of a hot take maybe but i like the process of yeah build it make a prototype for me and it will not make sense when I use it and I will know immediately.

46:55Blaine Billingsley: And if I think a lot, I mean, my whole career is littered with me making prototypes that I did think make sense and nobody didn't make sense to other people.

47:03Randy Silver: It's very easy to argue about screens and it's making you have a definitive opinion. And then you try using your, Oh no, no, that's not, that's just not right.

47:13Blaine Billingsley: Exactly. So to me, the fun part, I actually think the right way to go about it is like, you know, if it doesn't make sense to you, if you, if you see it and it's gobbledygook, well, you can tell it you know i actually think this next example where instead of going through flows i was iterating and um this is such a great failure example where it put mark complete like in the middle of nowhere and it started making this a card and then it decided it was going to be full bleed and it's like you know a rather avant-garde styling choice here um and so like I know that's not right. And I should, I can just, you know, pull this up and, and I can tell it, I can comment on it and say like, you messed this up and send it and it'll fix itself.

48:00Blaine Billingsley: And so like, it's good to like work out all the stuff that just literally gobbledygook. The other feature I love is the remove feature where you can literally just like blot out things that you don't want to be there, which is really helpful. Yes, exactly. Exactly. This is basically my favorite feature is being able to like remove stuff really quickly. But from there, like let it, you know, put it, put it into a prototype and, and see what you get. It's really easy to get a working prototype rather quickly. This was another example. If you just ask it to prototype a mobile app, you know, it'll make it here in React for you, but it puts it in a nice screen for you.

48:40Blaine Billingsley: It'll do, This is actually not a great example, but it'll do like carousels. It does all the dragging for you. And you can actually play with it. And then you can share this through sites, which is another great chat GPT feature and share the link to your coworkers and say, what do you think? Does it make sense? Is it ugly? What do you like? What do you not like? And to me, the whole point is like make a ton of throwaway stuff. And so I think it's good to just go all the way through it. Let it not make sense. And I find that people get that more when there's more volume. You know, when you only get one prototype in a monthly review, you're like, you can be precious about it on either end or you can hate it and not like any of it.

49:20Blaine Billingsley: But when you know you're getting something every 10 or 15 minutes, you know, it's a lot easier to be like, yeah, that's pretty. I like this part. I'm a lot more relaxed about it because I know it's easy to make another iteration.

49:31Randy Silver: So, you know, I don't think, well, the way we work on a day-to-day basis, obviously, it's changed a huge amount. The tooling is fundamentally different. But from a philosophical perspective, this feels like you're just doing a lot of design sprints. Is that essentially what it is?

49:52Blaine Billingsley: I love putting it that way. And I think you're actually very right. I mean, there's a lot out there in the internet community about how like the, you know, the role of design is changing forever and it's all different. I completely agree with you. Like the principles underneath our process have always been like, get the closest analogy of the working idea as quickly as you can, you know, whether it's paper or sketches or sticky notes, like anything. How do you get there quick so that you can feel it and you can share it. And it's the same idea. It's the same idea of like getting a lot of ideas, sharing them, hearing what people have to say about them, not being precious with your ideas.

50:34Blaine Billingsley: It's all the same stuff we've been doing for a long time. We just have new ways of doing it.

50:40Randy Silver: So I think we were kind of running short on time, unfortunately, but I want to ask you, you were talking about the team that built this, which was essentially you and an engineer working together on this. We're part of Mind the Product. I'd be remiss in not noticing there was no product manager. It didn't mention as part of it. But frankly, at a senior level, the difference between a product lead, a design lead, a dev lead, you all just have slightly different biases and experiences, but you're all trying to do pretty much the same job. And it sounds like where you guys were on this. So I'm curious, so if I was to join OpenAI tomorrow as a member of the product slash design team, or if somebody else was to come in tomorrow as a designer, what kind of advice would you give us on day one about how do we work in this world?

51:31Randy Silver: What do we need to unlearn? What do we need to learn?

51:35Blaine Billingsley: Yeah, I love that question. And I'm a jazz musician, so pardon the jazz analogy. But in jazz, like everyone plays a different instrument and it doesn't really matter what your instrument is. What matters is like, where are you at that level and what role does it play in the band? Are you a soloist? Are you an accompanist? You know, you need an accompanist. You need a soloist. You need a low end, but that could be a bassoon. It could be a bass. It could be whatever. And I like thinking about teams in that way instead of thinking, oh, like, that's your job. That's not my job. I like the feeling of being on a team where we're all responsible for an output.

52:18Blaine Billingsley: And we all have different specialties. We all play maybe slightly different instruments, but we all have to know the tunes. We all have to be able to count. You know, there's like basic aspects of working together that we need. And it's actually just about finding the people and strengths that fit together well. Like I'm really bad at time management. and I'm really bad at like, like, you know, updating, updating other stakeholders. If I get kind of in my zone, I just need somebody who's good at that. And, you know, sometimes that's the product manager. Sometimes it's the engineer. Sometimes it's important for me to have to do it and I have to like figure out how to make it happen.

52:55Blaine Billingsley: But I think the best thing to do is like, don't worry so much about the title or the role and figure out what, where you fit into the spectrum on your team and how do you contribute to that team and good things are going to happen and and that's the best way I can I can put it it's just uh the the titles don't make sense anymore or things are shifting we're in a moment of transition and uh just focus on who you're working with and what are you doing and how you can contribute and and yeah good things will happen for you let me follow

53:31Randy Silver: up on the jazz analogy because i'm i'm a terrible musician uh but i'm a recovering music journalist from many many years ago oh very cool one of the things about jazz is you know to get good at doing that avant-garde the free form the collaborative thing you have to be really solid at the basics you can't so true and it's not actually all that different with other types of music i remember talking to a friend about the bc boys when they came back uh after many years as a hip-hop band and started playing instruments again and going as a punk band and the observation i made was you know you can get away with a lot by being enthusiastic on a guitar and bass but if you don't have a good drummer that's gonna sound terrible so true so in terms of you know coming to these tools which allow you to a lot of people going to vanguard and freeform because you can just do so much so quickly, but how important is it to have all the experience?

54:29And I don't want to say it's judgment or taste, but there's something there that you have to have gone

54:35Randy Silver: through and know how to use these tools. They're just tools. You still have to know how to use them. Totally.

54:40Blaine Billingsley: Totally. Yeah, I think it is. I mean, I hope that experience buys you something, you know, because that's all I got at this point. But I, you know, also there's, I mean, the young designers that I work with today are just like on a whole other ball game of like, they don't have any of you with all that experience also comes like a bunch of baggage and presumptions. And it's really refreshing, like some of the best work I've seen in the last few years is from like really junior designers. So they don't have the experience. But they've got some juice that's like like a whole new thing. And I love that.

55:22Blaine Billingsley: And so to me, it's another part of like, just don't be, you're not missing anything. You just need to do stuff. I think you just need to go make stuff and go get your hands dirty to make, to start making a thousand pots and like, find what is exciting to you. And if something's not exciting to you, don't do that stuff. You know, like the stuff that gets me excited about little interaction patterns, like just go make it i'll just go make that stuff on the side because it's fun but that's you're building that muscle of what you find cool and what you think is is good taste or whatever so you know yeah it does take it takes a lot to get there but it it it's not like years and years uh it's a forever journey but you can start playing with the band really really fast you know yeah and and just to

56:12Randy Silver: go back to jazz one more time you and r &b uh miles davis and james brown would bring in people like bootsy collins when they're teenagers and exactly the combination of the two things that

56:23Blaine Billingsley: worked really really well exactly i mean all the the heavies they were young young young and that you know uh it's a it's you do need dedication but it's not like uh you need decades of time um there's a there's a lot of it's an exciting time because it's it's very easy to explore now and that's really what it all comes down to.

56:42Randy Silver: Blaine, this has been fantastic. Thank you so much for taking the time out. We really appreciate it.

56:47Blaine Billingsley: My pleasure. Randy, it was absolutely a blast and I hope we get to talk soon. Thank you so much. Thank you.

56:55Randy Silver: The product experience hosts are me, Lily Smith, host by night and chief product officer by day. And me, Randy Silver, also host by night. And I spend my days working with product and leadership teams, helping their teams to do amazing work. Luron Pratt is our producer and Luke Smith is our editor