Replit CEO Amjad Masad on Empowering the Next Billion Software Creators - Ep. 201

0:10Hello, and welcome to the NVIDIA AI Podcast. I'm your host, Noah Kravitz. When the hype around generative AI exploded at the beginning of this year, some folks, myself included, had visions of sitting down at our desks, speaking a voice command along the lines of, hey computer, write an app that, you know, does everything I need to do today, and sitting back sipping coffee while the machinery did its magic. I can only speak for myself, but it hasn't totally worked out that way, at least not just yet. My hopes were renewed, however, when I saw a video today's guest posted to Twitter in late March.

0:46Amjad Massad is CEO of Replit, a San Francisco-based startup building AI-powered tools to bring the next billion software creators online. Replit, who's part of NVIDIA's Inception program, offers a platform that includes Ghostwriter, an AI-powered pair programmer, a real-time multiplayer editor for collaboration, and browser-based build, test, and deployment of code. The company is also working on Make Me an App functionality for its mobile app, which you should really see for yourself in the aforementioned Twitter video. And oh yeah, Replit also recently announced their open-source complete code model and a Series B fundraising extension that value the company at just over$1 billion.

1:27That's a lot of news for a year that's not quite half done as we record this. And no doubt I've missed a few things. which is fine because Amjad is here to tell us all about Replit and the future of AI-powered software. Amjad Massad, welcome and thanks so much for joining the NVIDIA AI podcast. Thank you for having me. If you would, why don't you tell the audience a little bit about how Replit came to be and what the company is all about. The company is about fundamentally reducing the friction between an idea and a software product. You know, we live in this magical world where, you know, someone can have an idea and sit in front of the computer, maybe hours and days later, they have something that did not exist before.

2:12And we think that is fundamentally a superpower. But currently, the superpower is a little bit cumbersome, maybe not accessible to a lot of people. I sort of faced the problem myself back in college when I was doing computer science. I didn't have a laptop at the time and setting up the development environment for every programming language I was learning in school was really difficult and it was boring and was hard. And it really ate up a lot of our school time that we should have otherwise been programming. So I had this idea of I should be able to open a browser tab and start coding. At the time, I was using Google Docs and other browser-based software like that.

2:54And to my surprise, nobody had built that. So that was the year, maybe 2008. So naively, I started working on that. I thought, how hard could it be? Well, I got something working pretty quickly where you can just, it's a text box. You can evaluate some JavaScript code because the browser already knew how to do JavaScript. A lot of my friends really liked it. And I continued working on it for a while. And then I wanted to add more languages. And this is where I noticed it was going to be really, really hard. And so I started writing interpreters and JavaScript for Python, other languages I wanted to learn.

3:29And it turns out that's going to take many years, if not decades, to finish. And so I put the project down, kind of banged my head against it for a few years. Finally had a breakthrough. At the time, there was this trend of compiling different languages to JavaScript. So we were the first, me and my friends, to compile Python to JavaScript. And we put out a demo, and it went super viral. I remember at the time, the inventor of JavaScript kind of tweeted about it, Brandon Eich. And that was really the highlight of my life up to the point. Sure. And so I thought, okay, it'd be great to sort of build a business around this.

4:13I know there's going to be a lot of use cases for it. At the time, I was back in Amman, Jordan, where I grew up. And venture capital was not really a thing around the 2010s. And so I decided instead to open source everything I built. And a bunch of companies started around it, specifically in the US at the time. Maybe you remember, but there was this rise of the MOOCs era to 2011, something like that. I remember getting an email from Peter Norvig, who was working at Udacity at the time. And he wanted to integrate our software, which was pretty cool. But the thing that piqued my interest was the startup in New York making teaching code a lot easier.

4:55And it was Code Academy. So I joined them as the first employee, as a founding engineer. They got me an O1 visa. I came to the US. And that sort of kicked off my career working at the intersection of developer tools and coding. but it wasn't until 2016 that we actually started the business, Replit being the business. And we were in this incubation period for a long time because, again, we were at an intersection of two things that didn't do particularly well for venture capitalists, namely sort of hobbyist software education and developer tools. It wasn't until GitHub's acquisition that this category was sort of validated.

5:39And so in 2018, we sort of raised money and we were off to the races. And yeah, so that's sort of the brief. And now, you know, you introduced Replit and the intro very well, but now you could really do anything. You can go from an idea and very little knowledge and coding into a startup, perhaps, in a few days, all on Replit. So a couple of things I want to ask you about, and obviously the importance of AI when AI kind of started becoming a bigger part of Replit's stack, if you will, but, you know, business and technology -wise. But before that, maybe, and I'll say for the audience, but myself, you know, I can read and write a tiny bit of code, but I'm not a developer.

6:27So maybe you could explain a little bit about the problem of setting up a development environment and what you were trying to solve initially, you know, trying to just be able to open up a tab in a browser and start coding. What's that process like? How has it changed? And how has Repli kind of, you know, made things easier? for developers. Anyone who's been using software in the 90s or 2000s knows that typically, desktop software is the way to use most apps. So you would go to the store, buy a CD or floppy or whatever era you're in, and then go home, install that software and use it. Later on, as the internet became more powerful, people started downloading and buying software off the internet and downloading it.

7:21And then as the browser got more powerful, we started putting these software applications in the browser. Now, coding never really took that trajectory. It actually is stuck in the maybe 60s and 70s technology. So the way most coders still today are in the terminal, which is literally 60s built technologies. It brings me back to the VAX machines in high school. Yeah. It's called a terminal. And the reason it's called a terminal is because it's supposed to connect to a mainframe. So developers today are using mainframe style technologies. And generally, I found it super surprising that the tools, the people who are building the tools for everyone have their tools really stuck in a really long gone era.

8:21And so the trouble with it is that software is largely fragmented. It is not fully integrated. There's a lot of different packages. There's a lot of different versions. Software engineers tend to be adiosyncratic in the kind of tools that they want to use. And everyone integrates their development environments in their own way. And it's sort of like, you know, becomes this thing where you have to download the software according to your taste and integrate it according to your taste. And then God forbid you want to share that software. Now you expect the recipient of that code for them to be able to run it to have exactly the same configuration as your computer.

9:03And now we get anyone who's worked at a software company knows this idea of like it works on my machine, right? It works on your machine because it has the state that exactly instantiated that code. And by the way, that becomes a combinatorial explosion of the various versions of the packages, the compiler, the interpreter, all of the stuff that you're using. And so it is a really tough problem. And some of it is self-inflicted. Some of it is a technology problem. And so it hasn't been solved. There hasn't been a lot of incentives to solve it. The market hasn't really rewarded solving it, partly because I think it's like a sort of a cultural problem where once you go through that pain, it's sort of like hazing or now you're part of a cult and you want other people to also experience that pain.

10:03So part of Replit's strategy has been to actually target people who are not living in this sort of Stockholm syndrome scenario where they think that pain is okay. And so we've sort of grew with young people, with sort of hobbyists, with people who are typically not the professional engineer, although now professional engineers are looking over and saying, oh, how are all these people productive and happy? Maybe I want to be productive and happy. So tell us a little bit about the different features of the platform. Yeah. So, you know, essentially, you know, Google Docs has been a huge inspiration.

10:41They've done an amazing job where I can send you a link and suddenly we're looking at the same document. We're editing the same state of the document. There's this cute cursor thing that like, I know you're there and it's really fun, right? It's like software should be fun. And, you know, a lot of software today is kind of like that, you know, Figma, obviously for designers. You know, you'd be hard pressed to see any sort of industry without that sort of interaction. So that part is straightforward. You want to be able to go into a project and have the exact same state as other people in the project.

11:16So the same computers, the same underlying packages, the same file system. So we literally put people in the same cloud machine. So when you join a project, you're all in the same machine, editing the same thing, running the same thing. and it is a really delightful experience. And I still like many years after we built that initial thing, I still find it like really, it gets me really giddy about it. It comes through. People can't see the video, but you're smiling as you're saying. So you've got the collaboratory environment. You just go on and you can see your teammates where they are in the code and the document as well.

11:52And then you're also solving for that Stockholm syndrome, as you said, it works on my machine, but your environment, you're solving for that as well. Now, where, and, you know, feel free to answer this from the current time, or if you want to go back sort of in the company's history and the emergence of generative AI. Yeah. Talk about where AI comes in to do its magic. Yeah. So, you know, as someone who's worked on developer tools, essentially my entire career, one thing you notice, anyone who's worked with compilers or interpreters or what have you, one thing you'd notice is that building these sort of classical algorithms around parsers, abstract syntax trees, making sense of those trees, you tend to wonder whether deep learning could be helpful there, whether some statistical methods are helpful there.

12:50And it's sort of this question continued to bug me over time as I'm writing developer tools at Facebook or Yahoo or any of the other companies I worked at. And at some point, I just decided to research the topic. I remember reading a paper, I think came out in 2012. It was called On the Naturalness of Software. And a couple of researchers have come up with this crazy idea that code is like natural language. And by the way, in retrospect, that's not surprising because humans are users of natural language. We designed code to kind of feel natural to us. You just saved me from making what would have been a bad joke, because when you said that, I kind of thought exactly what you just so eloquently described, like, well, of course it's language.

13:42But back then, that seemed like a sort of foreign way to look at it. Well, what they found is that you can statistically model code in high precision manner. So, you know, before the current generation of natural language processing technologies, such as the transformer, which transformed kind of AI in natural language, we had classical methods such as Ngrams. So Ngrams, you can think about it as like sort of the percentage likelihood of the next token provided the previous token. And anyone who knows how transformer works, sort of transformers also kind of train on this idea of next token prediction.

14:22But in classical methods, we're not taking the entire context into consideration because it's computationally intractable. So it's called n-grams because it could be one gram, two gram, three gram, right? So depending on how much compute and storage you have. So they designed some kind of n-gram system to actually predict software. and they built an autocomplete system on top of it that when married with a more deterministic approach, such as classical sort of intelligence, it actually outperformed sort of regular intelligence, such as in any IDE. And that was eye-opening for me. It was like, okay, now it's obvious that all this sort of progress in ML is just going to get applied to AI.

15:07Right. And then there were a couple of tools coming out at the time. A lot of them were kind of primitive. So we kept watching the space, kind of playing with it every now and then. But it wasn't until GPT-2, I think 2018, 2019, that we're like, oh, this is the moment we've been waiting for. We always knew that AI was going to be a huge part of our mission. Because if your mission is to make something more accessible, automation and AI has to be part of it. because that's the most natural way to interact with human beings because that's what AI is great at, is about modeling that interaction with humans.

15:47And so we started prototyping around then, and it was like it was almost there, but it wasn't exactly there. You could do single line completions. You could do some cute stuff, but it didn't feel like an order of magnitude jump. It didn't even feel like a double-digit percentage improvement and user experience for the developer. So we're a small team. We sort of set it aside, but we had all that code for it and we had prototype a lot of things. Now, 2020, GPT-3 drops. I was like, okay, now it's actually - Now it's on. Because GPT-3 is just vastly more powerful. It understands the structure of code.

16:30It can complete entire functions. You could go to the OpenAI Playground and write code and it was like, okay, it's on. And so we started prototyping with it then. We launched a couple of things, but actually OpenAI wasn't commercially accessible at the time. GPT-3, unlike GPT-2, was the first time they actually announced that they're going to stop open sourcing stuff, OpenAI. And we're like, okay, this is amazing, but it's a closed system. how we're going to be building on top of it. It's not even commercial all the time. And so we realized that at some point we have to have our own AI and ML expertise.

17:16It's, I think, from your AI manifesto that was a blog post, but also a Twitter thread that I think was from your colleague, not on your account. Yeah. But there's a quote, if we succeed in our mission years from today, experts will look back at our growth in the AI space and quote, people who are really serious about AI should make their own platform. Yeah. Why? So that quote actually is inspired by Alan Kay. And Alan Kay told Steve Jobs famously, if you're serious about software, you have to make your own hardware. It's working out for Apple so far. The idea is like this vertical integration of the software and hardware layer nets out and superior user experiences, more reliable systems, sort of better flywheel effects of kind of the software improving the hardware and hardware improving the software.

18:10And we think the same sort of stands for AI. And actually the same analogy sort of works. If you think about the platform layer, sort of the hardware layer, semi-commoditized, it's still very hard to build and cloud systems are incredibly difficult to run. But it's possible to do that. The AI layer is the kind of layer that sort of continues to improve over time. It sits on top of a platform. It is really the point at which the user is interacting with the system. And we think that's really where the most value accrue for AI. I think there's this race now to build all these foundation models. But we actually think that really the most valuable thing is going to be the applications and really the touch points between the AI and the user.

19:02And the other thing is the data that you can collect and you can use to train your systems to get better at what the users do is incredibly valuable. So if you're using an AI through an API, that company that's providing the AI for you is never going to know the kind of use cases that your users are interested in. The reason these companies are producing these mega general models is because that's the best way to support every use case under the sun. Right. And I'm sure we're going to get into our language model in a bit, but our language model is a 3 billion parameter model. It's the smallest code model in the world or one of the smallest, yet it performs better than models 10 and 100x larger than it.

19:52And that's because we fine-tuned it on our data. And our data is highly rich data based on how people interact with our system. And so this is a few reasons of why you want to do the AI and the platform. There are a few other reasons that are actually specific to us, such as that we think the future of software development is one where AI is sort of like functioning like an actual human collaborator. and you want the AI to be able to interact with other systems and you want the AI to actually be able to, like, for example, you want the AI to be able to spend money on your behalf or spend compute on your behalf.

20:38You know, what we're calling our vision for artificial developer intelligence is the idea that you're sitting in Replit and the IDE with a co-worker that's completely virtual and artificial. But you can ask it to do a high-level task, such as produce an entire application or entire feature. And the AI has to go out and figure out how much compute it needs to spend on this, whether it needs to buy other tools from other services, or be able to even hire other people or hire other AIs. And so we think all these pieces need to be available in order to build the best user experience there. How much of that can replicate new math?

21:20We built a lot of it already. So at Replit, we actually have sort of a native currency in the platform. It is not a crypto. It is merely a sort of a centralized ledger that we have. And we built a market on top of it. We call it bounties. So if you don't know how to code, you have an idea, you can go to bounties and post a bounty to get other people to do that code. But that was mostly experimental. What we really wanted is that sort of currency that allows you to buy compute from us, that allows you to buy services from other people, that allows you to buy labor, libraries. Just enabling commerce is going to be a very important way to coordinate humans and AIs.

22:04So we built that. We obviously have the software and compute substrate. We have a complete cloud system. You can run any sort of code in any language, any library. We're building all the different SKUs you need for compute. We're adding GPUs. We have deployments. We have databases. And with our Google partnership, maybe we'll get into it in a bit, we're basically going to make the entirety of the cloud accessible via Replit. So I think we have all the ingredients today. I'm speaking with Amjad Massad. Amjad is the CEO of Replit. I called you a startup at the beginning. I don't know where that timeframe for when a startup becomes a non-startup begins or ends, but you've been doing this for a while.

22:49You've been working on developer tools since well before Replit, but Replit's been around for, did you say, 2016, 2018? So it's been a minute before we got into kind of the, I hate using the term hype cycle, because if it was just hype, we wouldn't be here talking about it. But before the current explosion of You mentioned GitHub getting acquired as kind of a moment of validation for this space. And then obviously everything going on, we're talking about with GPT and LLMs and such. I want to sort of go back in the context of what you were talking about with building a whole ecosystem and getting to a point where you've got, as a developer, you have a virtual counterpart, a colleague working with you.

23:30And that colleague isn't just auto-completing a single line of code as at the beginning you were talking about, but is able to go out on your behalf and architect things and call in other resources and buy compute on your behalf. And I'm sure figure out, you know, the best sort of compute, the best bang for buck, much faster and more capably than my human brain could do it and all that sort of things. And so that rolls into, you mentioned Reppel's concept of ADI, as opposed to AI or AGI or API is something different, but it's got those letters. But artificial developer intelligence. And then kind of going back to, I mentioned this in the intro, I took it from your mission statement, I believe, but from the website, this notion of bringing the next billion software creators online and using AI to help empower, you know, a non-developer to develop things.

24:25So let's get into that a little bit, if you would. And I'm sure there are parallels between the experienced developer on record and, you know, someone like me, a non-experienced developer who just says, hey, make me an app that can, I'm just thinking back to the examples I saw, that can access my phone's webcam with cool filters, which is a really cool video that folks should go check out on your Twitter feed. Talk a little more about this idea. It sounds like kind of like AI agents, and I've explored that a little bit in some other contexts and just kind of generally the idea of going from a single prompt to a system to a system that has kind of higher order capabilities to figure out, okay, Noah wants an app.

25:06What are the different steps I need to do? What are the ones I should go do on my own? When should I check back in and give him results, that kind of thing? I don't know, maybe in the way that makes most sense from Replit's point of view, walk us a little bit through this idea of ADI and empowering, you know, any developer to be able to create more and better and faster. So, you know, one of the most surprising things about large language models and this sort of generation of deep learning is how unreasonably important code is. And code is not just important in one capacity. It is important in a lot of different capacities.

25:46One part is the fact that they are exceptionally good at generating code. And we talked about how code can be modeled statistically, and that's partly why. The other reason is actually it turns out that training on code makes LLMs powerful at every task. The open secret in the industry now is that if you want to train an LLM, you have to have like a large percentage of your data mixture be code. In fact, a lot of people have speculated that GPT 3.5 was not GPT 3 based. it was based on codex because codex turned out it was, it was doing better at certain reasoning benchmarks than, than, than vanilla GPT-3.

26:30Right. I'll give you an example, by the way, when we trained our code model, we were surprised how it was like, although it was barely trained on any natural language, it was actually performing fairly well, well on natural language reasoning tasks. For example, like there's this idea of like theory of mind theory of mind is like, you know, So, you know, Joe enters the room, puts the chocolates in the box. Jane enters the room, removes the chocolate out of the box. Where does Joe think the chocolate is? GPT-3 actually did really bad. I almost ran in that. Did not know that, you know, it did not hold state for what GPT-3 is like, as an AI model, I'm unable to eat chocolate.

27:12Leave me alone. Basically, it just gave up. Now, it turns out that code models actually do better at this. And one can speculate why. Maybe there's some philosophical reason. I was smiling when you first mentioned that, thinking about incoherent Reddit arguments I've had with people. So if you train on that, it's not going to be as good a reasoning as if you train on code. Exactly. Exactly. So code is about reasoning. It's about logic. It's about holding state over time. Theory of mind is about holding state in a variable over time. That variable happened to be named Joe as opposed to X. we could spend this entire podcast speculating as to why code makes language models smarter, but let's take it as a fact.

27:54And then, okay, that's the second reason why code is important. The third reason is how do these models interact with a larger world? You can have a language model. It can be fun. You can go to ChatGPT and spend hours writing emails or whatever. That's interesting, but it has limited utility. How can this thing be more agentic and actually act on your behalf in the real world? Go out and buy groceries for you, go out and grab information for you. Well, it turns out the best way we have to interact with the internet and other computer systems is code. Calling an API, writing a small script to kind of go out and do something in the world.

28:37So that's sort of the third reason why code is important. The fourth reason is code allows us to audit and inspect the kind of plan that agentic large language models want to execute. There was a paper actually coming out of NVIDIA recently called Voyager. They built an AI agent in Minecraft. And it wasn't the first Minecraft AI agent. There's been a lot of like RL sort of exploration of Minecraft. The reason it was exciting is it was the way it learned was by writing code and improving code over time. So it writes a piece of code, interacts with the environment, and then it tests that code and then it improves it all the time.

29:23And by the way, a human can come and look at all the code that the AI written and we can make provable assumptions about what the AI is trying to do. So that allows us to audit it. It gets into the problem of alignment and controllability and serability. So that's a really good thing. So, okay, all these reasons why code is important for our LMs. And so we think that part of the way to get into more general purpose agents is by having agents be incredibly powerful at software, be trained on code, and be part of the environment where there's like millions of programmers programming, interacting with these AI.