Steve Ruiz: The TLDR of the Collaborative Whiteboard tldraw

30:00Like every infinite canvas has like hundreds and hundreds of these like micro features that are each individually like really crazy complex. And they just keep on going and going and going. And every single one of those just I was able to turn into like a really good Twitter thread and like in GIFs and, you know, show how it works. of like, I don't know, what happens when you're rotating like a group of shapes and like, you know, why are they wobbling around and all this and get into the math. And I was, again, doing this very publicly. And this attracted the attention of Figma, of Adobe, of Miro, of pretty much everyone in this space.

30:38Well, this was that weird thing that came after Globs that I hadn't given a name to yet. And I eventually did give it a name. I used that old TealDraw domain that I had back from Telestrator. I was like, all right, I'll just call it TealDraw easy six-letter domain. Wonderful. I didn't even make that whole TealDR connection until months later. But, you know, that's what I went with. And I got to a point where, like, you know, I was working on this. I was really enjoying it. I wanted to finish it. I was either going to take a job working on the Canvas team at Figma or I was going to take a job on the internal prototyping team on the user experience department of Adobe.

31:21Both amazing jobs, dream jobs. We were just negotiating like, you know, between the two. And so I was like, you know what, I'm going to quit play. I'm going to take a couple months off. I have some contracting, you know, money saved up. These guys, I'll let the negotiations go and I'm just going to work on this Teal Draw thing for a couple months and kind of get it out of my system before starting another job. Because it was pretty intense at this point and taking up a lot of my mental time. I didn't want to start a new job while still working on this. Ended up going with Adobe and signing a contract with them in the summer of 2021.

32:02To work there. Yeah, to go work for Adobe. But I said, hey, I don't want to start until January, you know, beginning of January. I wanted like a couple months off to, at this time, it would be like six months off to finish this Teal Draw thing. And they said, okay. And so I went full time on this. And I think this was like the end of July. Immediately picked up a corporate sponsor who paid me like$75 ,000 to have early access to the source code, which was interesting. I was like, huh, why would this very large, extremely large company with a bunch of developers not want to build this from scratch and want to build on top of my thing instead.

32:44They were integrating a whiteboard into a video chat application. And, you know, talking to them, it turned out that, like, well, this type of application is really hard, as I was learning. It's really complex. But it's also, like, incredibly subjective and kind of, like, just like our arrows are the perfect freehand thing. Like, there's no right way to build it. and there's no obvious way to solve some of these problems. They were just like, well, what feels good? What feels right? And what I didn't realize, but while I was implementing all this stuff, I was constantly making those types of subjective calls.

33:23But with a background in design, background in interaction design, the big fuzzy background in visual art and sort of somehow everything added up to a bunch of, I'm not sure you could say like right decisions, but the type of decisions that some technical teams would have agonized over. And I was just banging them out left and right, you know, all day. And so it was not only like the, like, yes, it did the thing that it needed to do. And yes, the code worked and everything. But like it was solving problems that not everyone would have been able to see as a problem. and doing so in a way that would have been, for most people, choosing between two identical tasting wines or something like that.

34:10You need someone with taste, and I had the taste. You had the taste, yeah. So that was good. I went full-time on it. I started working on it. During that time, I got a lot of interest from other teams for the same reason, wanting to build on top of this and wanting to have early access to it. ended up securing like another major corporate sponsor. This time it was, I was like, yeah, how about$120 ,000 for a sponsorship? Like you get nothing, I get money. I'll talk to you like once every two weeks. And they said, sure. So cool. Basically like a whole bunch of people were throwing money at me essentially to like stay unemployed and stay working on this thing because they wanted to build on top of it.

34:52They wanted to incorporate it. They wanted to evaluate it. they wanted to just compare against it because there really wasn't any good open source tech even to learn from in this area. Whiteboards are hard, but they're also carefully guarded, proprietary, competitive advantage-like tech. And I was just building it. I'm like, yeah, I'm going to open source this thing. Anyone can take this and run with it. and so that's eventually what I did end of November I launched the open source all the code I launched tldraw.com it was free you could go there you could collaborate and it got super popular I had a ton of inbound from I mean I had a ton of users using it and it was top of Hacker News it was top of GitHub trending it was there was a ton of activity around the repository people were really into this.

35:47But then, yeah, I had a ton of inbound from other companies being like, this is great. We're going to start building with this. Or can you build these three features so that we can do this? Or like, hey, we've already started building with it. Or hey, two weeks later, like, hey, we've just shipped this to production. And it was just wild, like so much attention, so much inbound. And I didn't know what to do with it necessarily. It was much more successful than I thought it would be. I had a lot of offers to sort of acquihire me in order to basically bring that into a company and for me to lead a team kind of building their whiteboard or something.

36:32And I also started getting interest from investors as well to say like, you know, this doesn't really happen like as often as you think it does. there could be actually like, this is more pull than most startups get. Have you considered? And so after talking with a lot of different people and kind of going fairly deep with a couple of different avenues, I ended up deciding, okay, you know what, I am going to raise capital. I am going to do a startup around this. Number one thing was like what people were asking for was way more than I could deliver alone. and the types of companies that were asking for it made me think that actually it was, this could be, this could be something that, that got big.

37:17And my other big, you know, thought, and one of the things that I had like kind of been talking about, not 100 % seriously, I hadn't really taken it as seriously as I may have, maybe could have. But essentially that like this type of application, like the whiteboard, felt like it was going to be much more common in the future than it was at the time. That's not to say that whiteboarding apps were going to get more popular, like things like Miro or Framer or whatever. It was more like a collaborative whiteboard is going to be a feature of many, many, many apps that don't have it currently. Right, it's not going to be a product.

38:02Because it's just going to get – It's going to be a feature. Exactly. It's going to get – Like there was a time with like Trello where like a Kanban board was a Trello board. And like if someone, you know, if you were dragging cards between columns, like that was on Trello. Now every app has that interaction. It's just called a Kanban board. It's not called a Trello board. And that it just became like a commodity. It just became like a common feature, a type of a thing. and I had a feeling that like one collaborative whiteboard probably should already be that and I didn't really understand why they weren't until I started working on TealDraw and I was like oh that's why because it's impossible to build these things like it's incredibly hard to build these and that if if I made it easy for people to do that then they would.

38:50So when you go to raise the money then and do this full time, is the goal, like you said, there were a lot of things that were being asked of you. So you wanted to hire and you wanted to be able to deliver on them. But was it also maybe to go back to this point around, you know, this realization you previously had that, you know, maybe open source should be commercialized in some way. Was it also to go and kind of like search for a business model? Yeah. I mean, it was, I think by the end of the race, I had a decently compelling picture of like, here's what I would do. Here's what I want to build.

39:31But I think the pitch that I probably could have made was like, look, I don't know. Like, I think that there is more pull for this idea that like people want to build these types of products, but it's not worth it to build it from scratch because it's very difficult and it's expensive and it's whatever. But if we can get them almost like all that undifferentiated stuff solved and sell them that, then this thing would become much, much more popular. That we could create this sort of category of applications which are not solely dedicated to being collaborative whiteboards, but which include collaborative whiteboards or whiteboarding-like things inside of their products.

40:17And that could be open source. It could be not open source. It could be a service. It could be a hosted service. I had no idea how to like, you know, what that right way of positioning it or selling it or anything like that. I think where the future is going to be is collaboration is much more popular, is that the whiteboard is not seen as a, or the canvas is not seen purely as a graphical thing, but as a productivity space, that people want websites that just sort of feel like Photoshop or feel like Migma or Figma. And also that like Figma's popularity was getting so big at the time that like every product owner is going to grow up on this software.

41:00People are just going to want things that feel this way. And then around that, I forget exactly when the sale was announced. I think it was after we started the company. But that only accelerated the thing. It's just like, wow, this is going to be like... People realize just how much users love working with this type of interface. We just want to be there in the future ready to sell this thing. um but i imagine one you probably never could have predicted what gpt for vision could do for you no um or maybe you could i don't know uh tell us a little bit about what happened there i mean that's that must be a fascinating new direction for the company for sure um so i'll compress the year and a half that happened between then basically we just started building it i put wonderful team here in London.

41:53We kind of started from scratch again, solved, you know, basically building. And I had to call Adobe and say, hey, I'm not going to start on Monday. That also happened. How'd they take that? Well, you know, kind of the same way that a play had taken it, which is like, you know, well, I'm happier that you're not going to go work for a competitor. Right. Instead, you're going to go work for yourself. And yeah, it was very supportive. That's good to hear. Yeah. We had by the middle end of this year, I mean, we had a new whiteboard that was incredibly good, that was getting popular, that people had already hacked and built, not hacked, but just sort of built with and built upon to build a bunch of production apps, even though it was in like alpha releases, including some larger companies and more successful startups.

42:46So that was the path that we were kind of going down, getting ready to do or kind of go to market and all that. And then suddenly GPT-4 with Vision came out. And previously, the only thing that you could do with GPT was essentially use TealDraw as the output. You could say, hey, chat GPT. And we had played around with this. We never really shared it because it was not super impressive. But you could say, hey, this is my API of my Canvas application. You know, draw a Christmas tree, you know, and it would go like. You could do this before GBD4Vision. Yeah, yeah. Is that like. How are you doing that?

43:33Well, you know, it's like it has a kind of a concept of like what a triangle looks like, you know. Okay. And if you say, like, here's a function called create triangle, and here's my coordinate system or whatever, it can make three triangles that are on top of each other that are slightly bigger. And in that case, what was happening was the input was text, you know, into the AI model, and the output was text back that we were using to draw inside of TealDraw. But essentially, TealDraw, the canvas was the output place. It wasn't the input place. The input place was still text. What ChatGPT4 with Vision did was it let us use the canvas as the input rather than, what is this Christmas?

44:26It'll draw me a Christmas tree or something. You'd say like, hey, what is this thing that I just drew? Describe this picture to me. And it could. and then you could say, okay, well, you know, how many ornaments are on this Christmas tree? And it could count them all. In the case of Make Real, it was like, here's a bad drawing, a wireframe of a website. Hey, GPT, you are a web developer. You've just gotten this drawing from your designer. It's your job to make an operant, like a working prototype. Send me back the HTML that I would need in order to make like a single page working prototype. And it worked, which is amazing.

45:04Wow. So the first version of this was from a guy named Sawyer Hood, who was actually a designer at or is a designer at Figma. And literally like hours after GPT-4 with Vision was announced, they produced like a prototype using TealDry that did exactly this. You know, you drew your thing. It sent a screenshot of that to chat GPT-4 with a prompt that said, whatever your developer and blah, blah, blah, giving back the HTML. and then it put that HTML that came back into like an iframe. Iframe is like a website inside of a website. And then you could click on things and you could, you know, if you did your drawing right and said, you know, here's a number, make this button, you know, make the number go up or something, then it could.

45:52The fun part was that you could also like, yeah, you could draw the thing you wanted to see, but you could also just like annotate it. You could do arrows and say like, in the same way that you would mark up a drawing that you were going to send to a real developer, should you be limited to sending only paper instructions. Yeah. You know, you'd be like, you know, this is what I want to see. I want to like this, this, and then draw an arrow and like do a little bit of notes on the side and have like, you know, multiple figures that describe an interaction and all that. You could send that and it would come back and it would work.

46:23So we saw this and we were like, oh my God, you could make this like so much better. And so we did, we forked it and we started adapting the work that Sawyer had done into something else. The first thing we did is because, well, because our canvas in TealDraw is a normal website, like it's not a, most whiteboard canvases are like essentially images that are being repainted like every frame, like a video game or something, right? It's like a custom rendered environment using WebGL or WebGPU or any of these other technologies that allow for high-performance rendering in the browser. That's what Figma does, et cetera.

47:03So in Figma, you can do a lot, but they had to build it all. You can't use CSS inside of Figma's canvas because it's not a website. It's not HTML. The browser isn't rendering it. Figma's rendering it. But because with TealDraw, our canvas was a normal website, still was in HTML land, we could put that iframe right back on the canvas. So what would happen is like you draw your little app and you say, hey, make this thing real and it sends it off to chatGPT4 with vision. Like here's a picture. Here's this prompt about you being a developer. Give me back the HTML. We get back the HTML and now what we do is we create a shape on the canvas which has that iframe, which has that website like inside of it and we put the HTML there.

47:51And now, I mean, you can still rotate it and resize it, move it around, whatever. but it's like back on the canvas. You've never left the canvas. And the real breakthrough for us was that now that it was back on the canvas, you could also draw on top of that and say, you know, hey, this button that is green, make this red or cross out one of the elements or something like that. And then you would select the website along with all that annotation and use that as a new input. So you'd click that make real button again. And now we would send it a picture of the website along with its previous HTML, its previous output, and the screenshot that we've taken of like you marking that thing up and use that as a new prompt to say like, hey.

48:39It's fascinating. Your designer had notes. Of text. Yeah, yeah. Yeah, it's like the prompt this time would be like, hey, your designers have just come back to you with some notes on your previous output. Please incorporate these annotations and stuff into a new output, a new HTML file. And then we would get that back. And what this allowed was, and you could just repeat this. You could just keep going. You could iterate and iterate and iterate. And it was mind-blowing because this was like a workflow that I had never experienced, no one had ever experienced in anything, this ability to iteratively add complexity to and shape, like deeply shape the behaviors of a little tiny prototype using nothing but arrows and lines and text.

49:35But you could also use screenshots from other applications and say make this look like this, or like the notation, like crossing things out and another typesetter type of markup. And it worked. And people were sharing this stuff like crazy. Of course, this was going mad on Twitter. Yeah, no, I saw it. We were doing Kanye West numbers in terms of views. It was just going everywhere. And like, yeah, I'm sorry to anyone if I just took over your timeline for a couple of weeks in November. But it was like legitimately something new. And the things that people were able to make with their kids or with their classrooms or just themselves, like adding.

50:24What was fun is that they weren't just sharing the output, although you could. It gives you a link. You could open it in Replit or open it in CodeSandbox or just share that link and actually use it. It's like permanently there. But what people were sharing, what weren't like the final result, they were sharing like the journey. They were sharing all the steps, the iterations, the markup that they had written, this, you know, like that became the story of like, here's what I did, but also here's how I did it. Which is something that you just don't get. You don't get that with code because code doesn't work that way.

51:00You don't see all those previous iterations of your code at the same time. You don't get that with design. And sometimes you kind of get that with design, but like then you don't really have a working final result. And it was just it was incredibly magical. People are still using it a ton today. And what's what's even better is that because for us, this was like this was like a demo. This was an experiment. We just tried to stay on top of the popularity as much as we could. But, you know, we never charged for this. In fact, we couldn't even have charged at the time. the GPT-4 Vision APIs were so new that you essentially had to, you still do, you have to enter your own developer API key in order to use it.

51:45So you're using your, which is bad for a website to ask you to put an API key. Don't ever do this. But you can do this with TealDraw, with MakeReal. Yeah, you were just being charged directly from OpenAI for using your own API key. When I eventually did talk to OpenAI, they were like, yeah, how is this so popular? Because the rate limits are really tight. You can only make 100 of these requests per day per API key. And they said, are you doing account sharding? Which is where a developer would open up a whole bunch of different accounts and then just sort of route a request randomly to one of the five so that now you have five times as many open.

52:26And I'm like, well, no, we're not doing that, but we're sort of doing that in that. We're asking for API keys. Or end number of users. Yeah, we have n number of accounts that each user has created. We had to do a ton of QA or questions and answers, community work around how do you create API keys and all that stuff and blah, blah, blah. Yeah, that's not an easy thing to educate on for people that aren't developers. No, it was not an end user product. But what's cool is that people are now making actual production end user products around that same pattern using TLDraw as like the engine, the input device for working with AIs and generative AIs.

53:08And that's really like, that's the place where I think that, I think we would much rather be is essentially the, like providing that visual surface for working with these types of models. Because, yeah, you know, it's not text. Like text is, it's easy to write a text box. It's really, really hard to write a canvas. and it just so happens that we, you know, we had a canvas that was really, really, really ready to go right at the right time. Like I said a couple of times, what we ended up building with Teal Draw, like if I hadn't already built it, I would start building it because it so clearly fits this world in which, you know, you have to be able to not only like compose images, but like iterate and work with agents like in a longer term way on a canvas.

53:55And yeah, I was incredibly fortunate as a startup founder for history to make a hard turn into your trajectory. Got lucky there. What's so fascinating to me about this is in many ways it's come full circle back to the early days when you were learning Framer and getting deep into prototyping. TL Draw, like you said, through this hard turn in history, in a sense, has turned into a prototyping tool. But actually it sounds like you have a vision that it could be much more than that. This could be a platform whereby people actually are developing real applications and deploying them beyond just prototypes.

54:36Is that the aspiration now? That that is what Make Real is going to be? So I don't know. It's a good question. I think with TealDraw, there's always been a tension between are we a company providing a developer tool? Are we Mapbox? Or are we Google Maps? This is the open source question from earlier. Yeah, are we providing a tool to other businesses to build upon? Mapbox I use as an example all the time. If you have a food delivery app and they have a map in it. It's Mapbox. It's Mapbox. It's like no one builds maps. Maps are insanely hard to build. I wouldn't build maps ever. But people like it, and people do it, and they sell it, and they license it essentially to other companies to use inside of their products.

55:27And that has been the direction that TealDraw has always been down. But along the way, there's been a couple of times where we're like, you know, this is kind of getting pretty popular as like an end user app. Maybe we should just do that. And so we've had to re-decide multiple times that like, you know, what kind of company are we, what kind of product are we building? and it always just seems like the more interesting place to be is on the sort of map box side of things. Is that like, we could create a good prototyping tool that uses a canvas in order to be the input device and be the iteration method for building websites or products.

56:15But, you know, like, you could also use that same canvas to build, to do backend stuff, you know, to edit and create databases. You could also do it to, you know, take screenshots of applications and mobile applications and say like, hey, actually move this button over here, you know, and essentially like the pattern that we've found with Make Real, I think is an incredibly generic pattern that could to be applied to a thousand, 10 ,000 different use cases. Like it could be one good product, but I think if we did go that direction, someone else would just make that generic thing because it's just totally impractical that all 10 ,000 of those companies, before they get to make the amazing AI controlled HVAC, you draw your heating and ventilation system and we'll convert it into the actual schema or whatever.

57:17Before you get to do that, you have to figure out how to resize a rectangle when it's rotated. You know, it's like no one wants to do that. They just want to start at the point where all that stuff, undifferentiated stuff is done and just go from there. And yeah, I think we'll probably continue to experiment with MakeReal. We're certainly still continuing to experiment with Chachipatian in a couple of different ways. but my guess is that the function of make real will still be to inspire other developers to build products and buy teal draw in order to do it but to that it's more like juicing the imagination of other developers and proving that wow this really connects with people like imagine if we use this inside of our educational app Imagine if we use this inside of our org chart thing.

58:15If you could just cross out the guy that you just fired. And it's true. It rewrites the whole org chart. Yeah, exactly. And if you can draw it and communicate it in a way that another person would understand what you want to get out of it, either GPT will already understand what you want to get out of it and create the same result, or it's like six months, you know, and then it will. So I think just having like a really, really, really good flexible, extendable web-based canvas is like an incredibly well-timed product. If I hadn't made it, I would be super envious of whoever had made it. And yeah, so we are in the business of demos and we're in the business of a kind of an infrastructure product, essentially, a UI infrastructure product.

59:08and people are already building really great stuff on it that use the same make real pattern. So it's exciting to see. You couldn't have predicted this, it sounds like, but maybe if you were to look ahead, what other things might be coming down the pike in terms of AI that could impact TL Draw and your path moving forward? I think the biggest thing will be kind of AI collaboration. Okay. Essentially, the same experience that you have collaborating on a whiteboard with someone else, another human. If you can imagine that, the other cursor on the screen was being driven by a model instead. And that in the same way that you can have like a kind of a, basically the ChatGPT moment for Canvas is coming.

1:00:04Like a real-time. I know Figlet has experimented. A real-time thought partner. Yeah, yeah. That you're not only doing things like kind of an AI-based autocomplete. I start drawing a graph, and they're like, I think I know what the rest of this looks like. Let me suggest it for you. But to have a sort of a visual conversation about like, okay, here I'm planning a trip. Here are a couple maps, and here's where I'm going to stay, and here's where else I want to stay. Here's my road trip route. you know, pull in some suggestions for places we want to stay along the way. You know, how would you change this route?

1:00:40Or like, here, I'm going to start drawing a layout for my apartment, you know, and I want you as the cursor to sort of be making suggestions, to be modifying things, to essentially have that conversation with me. It is, I think the canvas as the place where multimodal prompting and interaction and iteration happens seems pretty natural to me. the real question is can we get the speed of that communication and the multimodal aspects of it like fast enough and correct enough and teach it to use the product in a capable enough way that it really does feel like you're collaborating with someone else even if that someone else happens to also know the location of every Burger King between New York and Chicago.

1:01:34go. You could totally see that happening. I think it's a great prediction because I mean, today, if you think about ChatGPT, it's more or less what you're saying, but an asynchronous experience. But if the thing could move fast enough that it could be synchronous, the type of workflow that you're talking about would seem totally possible. And that would be awesome. Yeah. And if we jumped on a mirror board or a till draw for that matter, right now, Now what we would be doing is we'd be using text, we would be using visuals and screenshots and drawing things. We'd also be using our voices to communicate.

1:02:10It's like everything at once. And I think that's why people like collaborative whiteboard so much is that it's fast and it's just like stimulus dump of information that you have multiple channels going at the same time. So you're able to communicate really quickly. You even have this notion of presence and closeness and proximity and all that stuff. and yeah, much, much more complicated than just a chat back and forth, but I think that's going to be a direction that people want to go. If you really want to maximize the type of information that you're able to exchange with a model or with an agent, it's going to be on the canvas, and my hope is that we continue to kind of stay on top of things so that we are the place where that experimentation happens and where those products happen, and that we're able to get all those people started without having them learning about, you know, how to intersect an ellipse with a second ellipse when those rotation, that's terrible.

1:03:09All this stuff that is undifferentiated will be taken care of. You just start at the, uh, start at the end and go from there. Well, I, I, for one, I'm very excited for this prediction to come true. Um, and, uh, yeah, we, we, we've covered so much, uh, and it's been, it's been fascinating to hear this story? Is there anything you feel like we haven't touched on that we should? I don't know. I mean, there's the, uh, the normal startup stuff. Uh, we're based in London where, uh, we just raised, I mean, we had our, have our seed round, uh, which was led by, by Lux and Amplify and a whole bunch of really great angels.

1:03:47Um, and we just did right before we got super popular with the Make Real thing. We had just closed an extension, which was another 2 million. Thank you. That was led almost entirely by Amplify. Great group of partners there. We're based in London. Unusually, we have a big office or at least an office that we all use. Seven out of the eight are here in London, so we're We're building that type of company. And yeah, if you're in London, if you're into this type of thing, definitely give me a shout. You know, we have more desks than we need. We work here for the day. Where can people do that? How can they get in touch with you?

1:04:33The best way to get in touch with us is either on Twitter. We're very active on Twitter or x.com slash tealdraw. We also have a big Discord where you can join if you have questions about the product or if you want to use it. Again, we license it kind of in the way that I described before. Or if it's non-commercial use, if you're experimenting, it's free, go nuts. If you are using it in a commercial product, then I'll set you up with a license and it's pretty good terms and all that. But yeah, that's the direction that we're going with it. And we're small, like we're eight people. And yeah, it's an interesting time.

1:05:12Interesting company. Fascinating. Steve, thank you so much for your time. It's been so, so incredible to hear the story and we really appreciate all the time you've given to us. Thanks so much. Thank you. Thank you so much for listening to Generative Now. If you liked what you heard, please do us a favor and rate and review the episode on places like Spotify and Apple Podcast. It really does help. And if you want to learn more, you can follow Lightspeed at LightspeedVP on YouTube, Twitter, slash X, LinkedIn, or anywhere else on social media. Generative Now is produced by Lightspeed in partnership with Pod People.

1:05:49I am Michael Magnano, and we will be back next week with another awesome conversation. Thanks so much.