Soft openings aren't just for restaurants

0:00Welcome to Rework, a podcast by 37signals, about the better way to work and run your business. I'm Kimberly Rhodes from the 37 Signals team with Jason Freed and David Heinemeier Hansen, co-founders of 37 Signals. We are launching a new product very soon, Fizzy. We've been talking about it for a while. And this is the first time, at least since I've been here, that we have used outside people outside the company to help us as beta testers. So I thought we would talk a little bit about that and the process that we're using with that. So I'll, Jason, let you jump in. I There are too many beta testers is probably a problem and not enough is not helpful.

0:39So kind of talk me through how many people we're bringing in to test this product. We set up a list. So we have a couple thousand people on the list so far, and they will all get an invite before we make the public launch announcement. But initially, we're going to be inviting 5, 10, 15, 25 kind of handpicked people who we know. And this is basically just to like make sure there's no major blind spots where we didn't totally miss something. Not like in the product itself necessarily, like not like a feature we're missing or something, but like something like flat out broken that we just were so close to it, we just can't see it.

1:12And sometimes it's like around login even and sign up and onboarding and all this stuff you have to deal with, but you're not close enough to it to see it really. That's kind of why we have a small handful of people ahead of the public launch. That said, again, we will invite all these other people before the public launch just for like, thank you for signing up on the mailing list and being interested. It's not like we're saying there's no bugs in the product and we're looking for bugs to help us find them. Like we're fixing stuff all day long. There's a long list of things that aren't going to get fixed for V1.

1:41But it's just good to have people walking through and running through and playing with it and seeing it. And it's good for us to know other people are doing this because you have to start thinking about things like data loss starts to matter when you have customers. This is like their beta account is going to be their actual account. So they're going to sign up and that's their account. So now we got to be really careful. And, you know, we're already careful, but then we have to be extra careful and think about uptime and all the other things that really matter when it comes to being stewards of people's information and their accounts.

2:10And so this is a big reason why I think we get beta testers in early. We have done this before. We did it with Basecamp 4. Again, it's not like we need help finding issues. It is, though, that we all have blind spots and it's just good to have people walk through the garden, basically, to see if there's any weeds. And I'm curious if these people are folks that are already familiar with our other products, so they kind of know our system, how things typically look, or are these complete outsiders? The handpicked group will be mostly people who are familiar with us. See, the people have been begging us to get in.

2:43Basecamp customers, we have a Basecamp community. We have set up 1 ,000 plus people in there. There's some real hardcore fans there. We'll invite some of them. But the people on the list, like the email signup list where there's over 2 ,000, like, I don't know. That could be anybody. It could be anybody. Probably many people who are somewhat familiar with us to get on a list like this, but they're not maybe the super, super fans like the first 15, 20 people are. I think the magic with this kind of early access is it brings forward that moment of truth that is shipping. Because nothing drives that final decision of what goes into product, what gets cut, and what absolutely needs to be fixed, like knowing that someone else is going to see this.

3:23So it's almost a little bit of a cheat code for finding the kind of issues that you'd find like on the night of launch. Like it'd be nice if we could find it a little early. And also if we did miss something major, as Jason says, that did not show up through our somewhat golden path trip through the product. We're going to find it when 10 people are using it, 50 people are using it, not when thousands of people are using it. And I think that's slow rollout of essentially a kind of done product. As Jason said, this isn't a focus group. This isn't like, oh, come in and tell us what you think about the product and we'll totally reorganize it and make it something else if you give us just the right magic words of feedback.

4:03That's not what this is. B1, with any product we make, is always for us first. so it's sort of kind of you're here as a as an early premiere you're here for the what do they call it in theater like the opening night before the opening the rehearsal dinner or something else like that that's what you're here for right like you're not actually getting a stake to rearrange everything but it's still really valuable to do it and i think it also just takes a little bit of the natural anxiety there is around like, all right, let's go live. If you know you've let in a handful of people or maybe even a few hundred with Hay, for example, we were quite hesitant of just opening it up to everyone because email is such a high criticality product, both in terms of data security around availability and then also the features and so forth that We actually had quite a long beta process, if you will.

5:01We rolled it out in batches. It became more of a public thing at that point, but it was still staggered. And this is a tried and true pattern for lots of products. When they roll out at kind of a material stake, you just say, well, let's make sure that everything is all right. Let's make sure that all the bolts are tightened. And what you often find is that even the kind of quite rigorous testing that we do internally, we even have two people full time on QA, they'll prod it in much better ways than we often do. There's still going to be something you just don't find when you're looking at your own product and you've been looking at it for months.

5:41And sometimes that just shows up in technical exceptions, like the product simply broken and we get alerts about that. And other times it'll show up and like, oh, yeah, I hadn't even thought about when you sign up for the first time, you only see one account, you don't see three, stuff like that. It's easy to miss when you're in the thick of the development mode. The other thing I was going to add is the way I tend to think about it is like you want to keep your house tidy. But if you're going to have guests over, you just kind of do a once over. And it's like, oh, yeah, let's just make sure things like this isn't left out and this isn't left out.

6:14And that's kind of what beta testing is about. Like other people are going to be coming into our house right now. And we want to make sure it's we present ourselves well. We've cleaned the place up. Things work as you'd expect. And there's just an extra pressure on you. That's a good pressure to have when other people come in your front door. And if you were able to look over our shoulders of the past week, we were supposed to get these invites out like a week ago and we still haven't gotten them out. I think they're going to go out today, the initial batch. And it's just a reminder that, wait a second, we're not quite ready.

6:42Wait a second. We're not quite ready. And you actually don't have that feeling until you have to let someone in the front door. Truly. And so it just focuses your efforts in a really important way right at the end. It's better to do it now than to let 10 ,000 people in the door and realize that you really screwed something up badly. So you want to get your mistakes out of the way when not as many people are looking, but enough people are looking that they're going to pick up on them. And then when you're ready to let the world in, you're fully confident in what you're about to release and ready to go.

7:09And this is what I really love about this method of rolling things out. It's a forcing function for priorities. When the guests are about to arrive, you can't remodel the whole house. But you can go like, oh, do you know what? This path leading up to the front door, not quite as I wanted. That's what we should focus on next. Oh, now that's sorted. Ed, do you know what? The lock is a little jiggly when you're trying to get indoor. We've got to focus that. So it allows you to kind of disregard the very long list we have of things we want to do or things we ought to do and replace them with what must be done before strangers have an opportunity to look at this product.

7:52And I love that kind of forcing function because it takes all the sort of sophistication away from a lot of the minutia of the product planning of what should it be high priority, low priority, medium priority, should be red, green, whatever. All these ways of trying to slice up a very long list of issues you could work on, you can disregard all of it and just go like, what needs to be done before I won't be embarrassed when someone else shows up and has to use this for the first time? I love that house analogy. I'm curious about the process of inviting beta testers in. How much information are you giving them about the product and how it works before they actually launch in?

8:35Is it just like, here's the front door, y 'all figure it out? Or is it like, let me give you a tour of the house so that you're fully understanding where everything is? I would say that in an ideal world, I'd like to be completely hands-off because that's what everyone else is going to have. But there's some things that aren't quite done yet prior to launch. Like, for example, we're still finessing the onboarding experience. And so right now, we're missing a few of the onboarding things I'd like to have. And those things are going to explain how the product works. So what I've done is I've recorded a, I think it's maybe six minute video.

9:05I'm calling it an orientation video. It's like a first person walkthrough of the product that I'm sending along with the email. The email invite is like a handwritten thing. Hey, here's this link to sign up. By the way, just check out this video. It's a few minutes long. Just give you a sense of basically, again, orientation. Here's roughly how it works. Here's roughly where things are. This is basically what you're going to experience just so people are ready. So they don't get super tripped up on things that like I know they're going to get tripped up on because we haven't explained in the product yet.

9:32So I want to avoid that kind of feedback. But it, again, forces us to get into this onboarding thing because ideally you shouldn't want to have to tell people how to use something. You should put something in front of them and the thing should explain how to use it or it should be obvious or intuitive or whatever it should be. A little bit of handholding is fine in some cases, of course, but the product should do that itself. You can't expect to be in front of everybody. But again, early on, I'm including an orientation video along with the invite. And then my last maybe question is, tell me about feedback from beta testers.

10:05Is there a formal process that you're using to capture that feedback? Or is it just like people are just going to email you? What are you thinking about that? The first batch of people are going to be invited to a campfire instance, a chat that we've set up. But that only works for the first few handful of people, really, for this kind of thing. And that's kind of all we need anyway. So this way they can come and ask questions, post bugs, whatever they want to do. We also are going to have a fizzy board set up for them to report actually formalized bugs using the product itself. But if they have questions or are wondering about something, then they can use the chat room to get a hold of us.

10:44Initially, I was just going to have them email me directly, but then I can't really share that with the rest of the teams easily. So I think the campfire situation or Discord or whatever you want to use is a good format for that. One other thing that's a benefit of using those kind of collaborative ways of getting feedback is that you do develop a little bit of a community around people about early access. I've seen that time and again, when we've done it with open source, and you set something up, the first people who show up to try something new, have a shared experience that both connects them to the other people who show up at the same time to experience the same thing, and also helps shape kind of the form of the feedback.

11:22Sometimes people are like, oh, I noticed this thing. I don't know if it's important. And then you hear from three other people who will chime in. Oh, yeah, actually, I saw that too. They wouldn't have raised their voice if they had just been alone. They've just thought like, well, I just got to email Jason off on my own and I wouldn't be connected to anyone else for trying. So in those early groups, I really like that it's a shared experience. And I think you actually get better feedback from people when you do it. But it also, it doesn't scale to infinity. You are going to have to cut it off at some point.

11:52But I'll also say, I mean, it probably scales broader, longer than what we've done in the past. When I look at the kind of groups now we have in the chat rooms for Umachi, where we literally have 20 ,000 people. No, it does not have the same atmosphere as it did when we were just 50 people in there or 200. But there's still absolutely a value to cultivating those kind of communities, which is sort of what we've done with the Basecamp community. It's not a beta test anymore, but they are often the first group of customers we go to when we do want some specific feedback on a given feature. We want to give people a preview on some change to the product where we aren't quite sure of how it's going to land.

12:36Having those kinds of limited space relationships with customers, I think, is a really nice thing to do. Okay, perfect. And then this is my last question. We're beta testing with Fizzy. Do you imagine, you know, we're working on Basecamp 5. we've talked about some other future products. Do you imagine that being an area to beta test as well with outsiders? Or is this a unique situation? No, I do. And by the way, I don't even like the word beta test because beta historically means like unfinished software. And all software is unfinished anyway. This is just like early adopters. It's like if people who want to be in early, who are eager to check something out before it's fully released, who are people who typically are going to want to give you some more feedback than maybe the general public.

13:20So that's how I've always thought about it. We don't launch public betas of products. Like we don't do it that way. I just think it's really about just, you know, a handful of people early who are sort of privileged to be in there. They want to be in there. They're pumped to be in there. Please come in and help us. That's kind of how it feels to me. Yeah. And do you imagine that with Basecamp 5? Yeah, I think so, for sure. I mean, I shouldn't say for sure of anything, but it seems like it'll make sense. I mean, Basecamp 5 is going to be a morphing of four into five and we'll be able to have these two running simultaneously for a while because we're actually on Basecamp 5 internally now.

13:54So we'll be able to maybe invite a few accounts over. That doesn't work right now. It's only staff, but we can definitely make that change later and maybe include a few people. I will be sharing a lot. I've already shared some previews with the Basecamp community and I'll be sharing a lot more with them. And then at a certain point, like maybe six weeks ahead of launch-ish, we'll be sharing a lot with the whole Basecamp customer base because this is going to affect their accounts. And so we'll be doing a lot of sharing there, primarily video, probably live walkthroughs and all sorts of other things like that.

14:23Okay. Well, with that, we're going to wrap it up. Rework is a production of 37 Signals. You can find show notes and transcripts on our website at 37signals.com slash podcast. Full video episodes are on YouTube. And if you have a question for Jason or David about a better way to work and run your business, leave us a video question. You can do that at 37signals.com slash podcast question, or you can always email at rework at 37signals.com.