Scott Wu - Building Cognition - [Invest Like the Best, EP.402]

22:29Where the first big code products, I mean, GitHub Copilot is probably the biggest of these, is a text autocomplete, where it's like, here's the file so far, predict me the next line. And if it is correct, I'll just hit tab. and it obviously saved me the time to type the line. It doesn't save me the time of thinking of the line because obviously it's still me that's deciding what's correct and what's not correct, but it does save the time of typing the line. And I think what we're now seeing and what everybody is talking about now with agents, although of course it's very early, is this new paradigm where models can actually act and make decisions in the real world.

23:00There's a lot to it and it's how humans do it too. It's not like a human just produces this perfect paragraph. Here's everything that I needed to give you and then you're all done. A lot of it is going and interacting with the real world, looking up websites or something, trying out things, seeing what works or what doesn't work. In the case of code, obviously running the code and looking at the output or looking at the logs and things like that. And I think maybe the simplest way to put it is in all of these verticals, I think we're going to see this where basically, hey, having a legal Q &A bot is great.

23:30But the next step is to have a lawyer. Having something that can answer questions about math problems or something that you have is great. The next step is to have like a full teacher. And obviously in software, it's the same thing where it's like having the text autocomplete for coding does make you faster yourself as an engineer. But what if you had a whole engineer that you're working with? That's kind of the core of this agentic work. Maybe you can describe then how literally Devin is built. Behind this question, I'm also curious about taking your experience as advice for other builders who are using core foundation models as a key part of the thing that they're building and building in such a way that is complementary to those underlying foundation models and won't just get subsumed by them.

24:12Everyone asks about this. I'm sure every VC asks every founder, like, how will O1 not just do this? So tell us about how Devon is built and how you think about building with and alongside these evolving foundation models. We are 100 % laser focused on solving all the problems along the way of software engineering. And the truth is, software engineering is just very messy. My point with that is just that there's this idea of like solving these math problems in a vacuum, for example. There's a lot of research that's going towards that. But there's also this entire thread between of how do you take that IQ and turn that into something that is actually meaningfully useful for software engineers in their day-to-day work.

24:51So there's a ton of stuff that needs to be done on adapting to the capabilities for particular tooling and particular stack. And then there's also a ton of stuff that has to be done on the product side as well. I'll just give several examples of that. A simple one is, obviously, Devon has to be plugged into all your tools. Solving problems in a vacuum is great, but Devon has to work in your GitHub. It has to be working in your code base. It has to use your Confluence docs or whatever it is. That's where the information is. It has to read the bugs on Jira. And on top of that, obviously, Devon needs to learn with the code base over time.

25:20And so even a really, really, really sharp software engineer, they join your company on day one, they're probably not to be all that productive, at least compared to somebody who's been there for five years and knows the entire code base in and out. Devon needs to learn the same way of understanding, hey, here's what this file does. Here's what these functions are for. If I want to do the schema migration, here's step one, two, three to get that done and all that. And then there's obviously all of the actual step-by-step decision-making that goes into it, which is somewhat different. We were talking about fixing a bug.

25:51I think a very natural flow of fixing a bug is, okay, I got a report for bug. First thing I'm going to do is I'm going to try and reproduce the bug myself and see what's going on. I got to go run all the code locally and try to make the bug happen. If the bug does happen, I'm going to go and take a look at the logs and see what the error was or what happened exactly, what the variables were. And then from there, I'm going to go read the relevant files. Maybe you can make some edits, test things out and whatever. And then I'm going to go test it again and see if it works. And if that all works, then I'm going to add a unit test to make sure it doesn't fail again and whatever.

26:22And all this step-by-step decision-making, it's a very different problem, obviously, from text completing what would somebody really smart on the internet say. How do you use the tools available to you and make a sequence of decisions that maximizes your chance of success? I think the other question you asked was how, as a builder, to think about this in relation to, obviously, this steady stream of improving models. It's a really interesting one because a lot of other technologies that we've had have been a lot more step function. mobile phones came out. And the first question was, what are you going to do when everyone has a smartphone in their pocket?

26:54In some ways, the core of Uber or Airbnb or DoorDash or all these things. And you can kind of think about, okay, this is the technology shift. And let's think about what happens over the next coming years now that we have this shift. AI is a lot more of a very incremental thing. Here's GPT-3 and 3.5 and 4. And each little technology gets a little bit better over time. And part of what that implies, I think, is, yeah, just really thinking about What are the models not going to do by themselves over time? And a lot of the stack and a lot of the things that we care about, obviously, are around building capabilities and the product experience specifically for software.

27:29I have no doubt that the foundation models will continue to increase in IQ and increase in IQ and so on over time. But a lot of the detail of all of the tools that you work with, how you make decisions with all of that, even how you communicate with the user, what is the user interface where you're talking with Devin and giving feedback and seeing what Devin's up to and checking in and making sure nothing's wrong and all these things. There's a lot, a lot of detail, obviously, in that flow that really brings out the inherent IQ, to put it that way. We've always thought about it in that way of how do we make sure everything that we're doing is complementary with the model increases.

28:02And by the way, complementary with everything else. The hardware is getting faster all the time. The whole stack is getting more efficient everywhere. And we have our part of the stack and we have always thought about it as how do we do as well as possible in are a part of the stack specifically for software. What improvements are you hoping for from the foundation model companies that would make Devin more effective than it is today in the future? I should mention, by the way, the foundation model labs are all super, super excited about agents. And it makes sense. I think one thing which is kind of interesting, just practically, is let's say you ask GPT a question, who is the fifth president or something?

28:35You ask one question, you get one answer, and that's a single model query. And obviously there's a consumer pricing and everything, but if you're going by the API pricing, that costs one millionth of a dollar or something like that, something ridiculously low. Whereas if you're asking Devin, hey, I've got this new feature that I need you to build down. Can you build it and test it and make sure it looks good? Over the course of that, Devin's probably going to be making hundreds, if not thousands actually of decisions over the next half an hour or something while it's going and building that out.

29:02And it also just practically means it's actually just a lot more inference on GPUs. And so for that reason, And what we'll see over the next while, honestly, even the next one year, I think agents are going to become a lot more widespread. And what we'll see actually is I think the dominant usage of these GPUs, of these language models and everything will be for agents, largely because every single agent query is just so much more magnitude of underlying language model calls than a single Q &A query. And I think what happens next with that, which I think these foundation model labs are all thinking about, is, yeah, how do we build our models to be as optimized for agents?

29:37And so I think a lot of these things of working with multi-turn traces, being able to handle long context outputs, for example, like context is a big thing that people are spending a lot of time thinking about. And with an agent, again, it's like, do you have half an hour of decisions? Obviously, it helps a lot to know what's been going on in the last half hour. So some of these particular technologies that will really help with that. How do you use Devon in your own building of Devon? We use a lot of Devon, actually, when we're building Devon. It really is this new workflow, a very asynchronous workflow.

30:09And so I'll just give an example. We do all of our communications in Slack. We have a few channels. We have a crashes channel where every production crash gets reported. We have a front-end channel where we talk about any front-end feature requests or maybe like, hey, let's switch the flow around this way. Or, hey, this design is a little bit off or things like that. Front-end things that come up. We have a bugs channel where users are filing bug reports of, hey, this thing didn't work or something or this thing broke or whatever. In all of these channels, one thing we do every time something gets reported is we tag Devon.

30:39You just go through the channel. Hey, this thing's broken. At Devon. Hey, somebody requested this. At Devon. Hey, we just had a crash. At Devon. Devon's not perfect. Devon's a junior engineer and often does need a lot of guidance. And I would say probably about 50 % of the time, Devon's output on one of these is just like a pull request code that's straight out of the box, ready to merge. And the other 50 % time, obviously you can make the call of, okay, do I want to talk with Devon and coach Devin through this? Do I want to go and make some changes myself and just touch up whatever needs to be fixed?

31:09Or do I want to just go start from scratch if it's especially far off or whatever? But having this be part of our workflow, you had to report the bug anyway. And so you might as well just tag on and add Devin at the end, and then you get a first pass for free. And so a lot of workflows like this, a lot of it is just thinking about how to put it into the asynchronous workflows. A similar one, honestly, is if one of our engineers is building some big new project and they're going and implementing things or building the IDE, for example, this happens all the time as an engineer. You're going to go through the code and you're going to see all these little things that are like, man, why did we write it this way?

31:40This is so messy. Or like this thing is technically correct, but it is not the most stable way to do this. Or lots of little things. You get caught up in a lot of these little things as you're building your bigger project. It does distract and it takes away your focus. And what we do now, obviously, is you just kick off Devin from the IDE for each one of these. And it's like, hey, can you go refactor this one real quick? Hey, can you change this to be type safe? hey, order this in a different way so that there's not going to be a race condition or something. And you're doing all these little fixes and you're kicking them off async.

32:07And then you're doing your own project in the meantime. In both of these cases, a lot of it is really just figuring out these asynchronous flows. It seems like part of the answer too, in the near term, you said earlier that maybe 10 % of the engineer's time is spent on what a friend of mine calls positive engineering, building things, the creative part, 90 % might be on make sure things work more almost like a mechanic than a creative, productive engineer. So does that mean that the near term, what Devin and other systems like it are going to help us do is eat that 90 % and keep the programmer focused on the 10 %?

32:39And then what happens beyond that? If we beat the world's best programmer in two years, why do we need programmers? Will there be programmers? What will they do? So I think it's exactly right. Eating the 90 % of all the miscellaneous implementation and debugging and iteration and whatever. I think what happens then is honestly, you just get to do 10x more of the 10%. And I think the cool thing about it, we should say it all right, obviously, there's a lot of fear in AI about what's going to happen and what happens next. But I think one of the really beautiful things about software is there really is 10 times more software to go right if we just had 10 times the capacity.

33:12Over the last however many years, we've always been bottlenecked by supply. If you talk to any engineering team, I haven't met a single team who said, yeah, we built everything that we wanted to build. There's actually no software to do. Every team out there has 20 projects that they want to do. They have to pick three to take on this week because there's just so few software engineers and software just takes so long to build. I honestly think we're going to see a lot of this Jevons paradox where we're just going to be building more and more and more software as it gets easier. It's what we've seen already, but maybe to a somewhat lesser scale.

33:41Building software now versus building it in the 80s probably is about 10x easier or so. And this shift will happen a lot quicker, obviously. But over that 40 years, we've gotten a lot more programmers and we've built a lot more software. And I think we're going to see a shift that looks like that as well. I think in the long term, I think the soul of software engineering, again, is each person getting to decide what are the problems that I want to solve and what is exactly the solution that I want to build. AI is not magic. Obviously, it's not going to go and solve all your problems for you, but it will give you the power to specify exactly what the solution is that you want to have and just be able to have that happen and build that out.

34:15Do you think in general that all this means that business people need to start thinking harder about demand and distribution? If we've been supply constrained and that's about to go away, does that mean that the people that are going to win in the future are those with better demand and distribution? I think it's software, especially. We have always been capped by supply and not demand. Obviously, there are a lot of things that are more fixed in terms of the demand. I think there's a lot of interesting questions of what happens to the economy with this and a lot of interesting second order effects.

34:42The simplest way to put it, where does all the money in tech go? Where does the money in venture capital, what does it go into? And it is primarily software engineers. So when you're getting 10x more bang for the buck for every software engineer that you're bringing on, it does change the landscape a lot of how much people can build with products and how much people can do. And then I think down the line, you see a lot of pretty interesting effects with businesses. I see a few different categories of businesses broadly. If we're being blunt, I think the toughest category to be in is business that's heavily reliant on switching costs and lock it.

35:13A lot of businesses out there, software businesses, that the reason they make all the revenue that they make is because it is so painful to switch to the other products or the other technology or something like that. If Devin is just able to do all that implementation and migration, and those costs go to zero, obviously, it's going to be a lot more about having the better product. That's going to be a real shift, I think, that companies are going to deal with. At the same time, it's not a nihilist view. There are a lot of things that will be just as strong, if not stronger. Network effects, I think, will always be golden.

35:41I think data and personalization is going to be even more powerful. So a business that owns the infrastructure, for example, or a business that has all the personal relationships and the data and so on, being able to serve for each person, an individualized product that is truly optimized just for them instead of this clunky thing that kind of works for everybody and touches all the millions of people, but is not the perfect product for anybody. I think a lot of those businesses will be able to do very well. What kinds of questions are becoming most common when you spend your time talking to the senior most people at the big established technology companies?

36:14There's definitely a lot of folks asking, how do I think about my engineering headcount? And how do I work with this? I think we're very much in the paradigm where people like to say this, AI won't replace you, but someone using AI better than you might. And so a lot of it is how do you get smart? And how do you make sure you're on top of the latest technologies, and you're ready for the shift, for sure, honestly, you're going to quicker than most people imagine, and getting your whole team set for that and getting things built for that. I think there's also a special case, actually, with these technology businesses themselves, folks who are building SaaS, products or things like that, where we talk so much about agents, I actually think one of the kind of interesting product changes is that a lot of these companies are actually going to be serving agents in addition to humans.

36:54A lot of great businesses out there that are basically meant for this collaboration, coordination, a lot of interfacing. We talked about companies like Slack or GitHub or Datadog or Atlassian or things like that, where there's a lot of work and compiling that they do. And also a lot of it is about the interface with how humans interact. I think the The way I think about it is right now, those are really valuable tools for agents to be using as well. Sure, you can imagine this nihilist view where an agent is just building everything on its own from scratch. But the truth is, there's a reason that we look at logs the way that we look at logs.

37:27The reason that we set up our documents the way that we set up our documents, and it does just make it easier. It makes the problem easier of getting the context that you need. So what will happen actually is we're going to have a lot of agents who are using these products as well. So businesses that are in that space, a lot of folks are actually thinking about how do I make my product ready for my new wave of agent customers in addition to all the human customers. I was talking to Martin Casado about these declining cost curves that technology creates. And the first one that's clearly happened already in AI is like the cost of marginal content creation has started its march towards zero in all different forms of media.

38:02It seems like with you and what you're building, there's that same curve story happening in the cost of software itself. How do you build a software business in that reality? If I fast forward 10 years or something sufficiently long, why should Devin cost anything? Why won't there just be like an open source Devin that does everything for me? And what do I need to pay for software if we're entering an era of marginal cost of creating software being very close to zero? I mean, I think right now, Devin is really the only one out there in the market that's doing that. But as you're saying, I'm sure there will be lots and lots of competitors pretty soon, even.

38:39And folks that want to do similar things in code is great for the consumer. We were just talking about the 10X engineers and things like that. The truth is that every level of decision making quality and every level of skill that you can have in your software engineer actually really does have this exponential reward curve, and it makes a big difference. And one of the things that we're getting, obviously, is this ability to really, really personalize Devin to each of their customers. And so like Devin, I had a lot of our customers today. When we work with Devin, for example, it's just like Devin knows the entire code base in and out.

39:06Devin wrote half the files in the code base. And so there's all these details. And obviously, it's not just the lines of code. It's the decision making that went into it and thought process and so on that Devin has internalized and working with the entire engineering team. And it really is that trade-off where it is about here's an engineer who's maybe smart and sharp and stuff, but it's starting day one on your code base today versus here's an engineer who's been with your code base for years and years and understands every file and has built all these things and knows exactly what your engineers mean when they ask certain things.

39:36And there's a real trade off there. It comes down to these things where it's kind of like pure technology switching costs, where it's migrating all your files to some new thing or moving to some different platform. Yeah, I think that is going to get cheaper and cheaper over time. At the same time, obviously, if you have the level of personalization, if you have the network effects or whatever. It's not about the cost to switch. It's really just about you're able to provide a better product. Maybe say a bit about pricing. This is such an interesting question. It seems like someone at OpenAI put their finger in the air and decided 20 bucks was a reasonable price to charge and a whole business ecosystem has spun up and that seems to be the default pricing.

40:12You made a very different pricing choice. It's 500 bucks a month. OpenAI just released the$200 a month option. So we're marching upward, but talk me through how you thought about and think about pricing in this new world? Yeah, we're going to see more of that, obviously. I mean, I think for us, it's funny, we had a lot of questions with our launch, for example, questions of like, how much do we anthropomorphize Devin? How do we explain the structure of it and so on? But I really do think it is a pretty different paradigm where with agents, it does come down to usage for better or for worse. There's reasons for both the inputs and the outputs for that.

40:46But the inputs, like we're saying, it is just more expensive to run. That is the case. You're doing more queries and more work. There's more GPU compute that goes into each job or each task or something like that. But I think with the outputs as well, obviously, when you're starting to take on tasks that are real end-to-end tasks, it's just much cleaner and much clearer to measure the concrete impact that something like this has. I think the proxy that we essentially want to have is value-based pricing, where it's basically, we want to make a lot of the projects or the bugs or the features or whatever that you're building 10x cheaper to build.

41:20And so we want the cost of Devon to run on those to be 10x less. It's obviously hard to have a perfect proxy for that, but the proxy that we have essentially is usage. We have a unit called ACUs, agent compute units, and it's basically all the decisions that Devon makes, the box that Devon uses, the code that Devon runs, or things like that. And it roughly corresponds to basically for every hour of Devon's work that you're doing. Again, it depends on different machines, but you're paying something like$8 to$12 or something like that. Obviously, Devon can do a lot in an hour. The setup is such that we want it to be 10x cheaper than having to do those yourself.

41:51But I think that is going to be broadly the new paradigm when we start to have AIs that are doing tasks instead of answering questions. Any thoughts on this big fun, almost has become like a cocktail party style question around whether or not we've achieved AGI or not, and whether or not that's a question that you care about? I'm curious what you think about this one. I'm in camp number two on that. We already have AGI. Someone said this in October, in 2017, you ask if we have AGI, the answer is no. In 2024, you ask if we have AGI, the answer is, well, you got to define AGI, it depends on this and that.

42:24I think there is a somewhat recursive definition of AGI, obviously, that people like to say, oh, it's doing 80 % of human labor. It's like, well, if you could do that, then the humans would be doing the 20%, so it wouldn't be the 80 % anymore. There's some recursiveness, but the way I would say it practically is, I think there's going to be a tale of things where we will push the capabilities, we'll push the capabilities, And for a while, there's going to be lots of little things that are like, well, actually, humans are still better at this one thing. And then we'll solve that thing. And then people will be like, oh, well, humans actually can still do this one thing.

42:53And we obviously have a lot of pride as humans in our own species of wanting to find the things that make us unique, and so on. But I think that practically, what matters for the world, obviously, is, is the AI good enough? And is it built into products and distributed well enough that it's actually affecting the world and affecting the economy and affecting people's day-to-day lives? In some sense, that question matters more than, is it doing 95 % or 96 % or 97 % of the human's workflow? It's more a question of, are humans actually out there doing 25 times more because of the AI technology? you?

43:29And I think that actually is much more of a practical question. And in some sense, it's like we're talking about the IQs and things like that. The math problems that these base AIs can solve are already, honestly, shockingly good. And among humans would be in the upper echelons of humans, if not quite number one in the world yet. And I think the biggest distinction is actually not whether we are number one in the world or only top 0.1 % human or something like that. The distinction is a bit more of, is that level of intelligence actually used by every human out there in the world and done in a way that's actually giving them a lot of value?

44:01That's going to be the big shift that actually gets us to the massive GDP growth, massive efficiency gains, quality of life, whatever you want to call it. Investors mostly are obsessing over scaling loss because it just has such big implications for public equities, for companies building new products in the private markets. And so that's where they tend to focus. If you are training your attention on what's going on in the foundation model world. What do you spend your time wondering about, asking your friends about, learning about? Where's your eye trained in that world? Yeah, my hot take from this is I think scaling laws are a little bit of a myth.

44:35I think we have made continued progress on these models and pushed the capabilities and things like that. And there's been a lot of new technologies, obviously, that we figured out. And there's way better post-training and there's some of this stuff, obviously, that's going on, reinforcement learning and things like that. Sure, there is some return to scale, but I think a lot of what we're seeing is basically, it is also the introduction of new technologies that are making this more efficient or allowing it to grow with scale beyond that. And so people sometimes like to talk about scaling laws as if you just sit back and as long as you got the hardware for it, it's just going to get better and better and better.

45:06And I think really, obviously, we have needed more and more hardware, but it's also been innovations that have actually gotten us there. I think practically with foundation models, we always obviously want to have a pulse on what's going on and what things are going on next. We work very closely with all the foundation model labs. We do evaluation of early models with them. We do custom training of models with them, things like that. It's like we're saying, it is an incremental thing. These things will get better. And a lot of the question as a decision maker is to basically just have a very clear sense on what are the capabilities that will get better and what are either the capabilities or the product or the human experience of it that will always need to complement these for you to deliver something of value to your customers.

45:48I'm really curious to learn a bit about the formative experiences for you as a business person. Obviously, we've talked a lot about formative experiences as a technologist, building cognition and also IOI, and these other competitions for the first part of your life. I'm sure everyone's seen the video of you when you were 12 or something winning math competitions. Obviously, you've had some pretty cool formative experiences. What have those been on the business side? For example, I'm interested to hear about your experience with Lunch Club and what you were building, why you're building it, and what that experience or that chapter of your business career taught you?

46:22Yeah. Lunch Club was my first company. I dropped out of school actually to start the company. It was a great experience. First of all, it was an AI for professional networking. We made millions of meetings happen and it was super fun team and product to build. I would say one of the big things, honestly, that I learned is it sounds simple, but it really does come down to the fundamentals. I feel like the common pieces of startup wisdom that everybody talks about, in some sense, they're so obvious that it's not obvious. It's not even worth mentioning if people feel that way, where it's like, go as fast as you can.

46:51Everybody says that. Everybody knows that. Never compromise on recruiting. Stay as close to your customers as possible. Focus on building a product that people love and want. These are almost tautologically true. But the thing is, the reality is there's always another gear that you can kick into with your fundamentals. And that's how I've really thought about it now, having gone to see it. Simple examples, everybody says you can go fast. In our case, we incorporated the company in January. We were building the first versions of the product. We got it out to the first group of initial users in February.

47:23We did a launch in March. We raised a big round in April. We did a big partnership with Microsoft in May. While we were doing that, we got on a lot of enterprise customers. We grew the team. No matter how fast you're going, and every startup in the Valley, obviously, thinks they're going quite fast, it's still worth it to really push for even faster. And no matter how close you are to your customers and how deeply you understand their problems, you're probably not going to regret spending even more time with them and understanding that deeper. And no matter how high your bar is for hiring, you're probably not going to regret trying to go for folks that are even higher still.

47:55And I think that is the biggest lesson I've learned with startups. The competitive landscape, does it feel like this big, exciting greenfield, or does it feel like battle lines are being drawn and insanely competitive for every available opening and sharp elbows. Characterize what the playing field feels like right now. You all work out of this amazing house that I've been to. It's such a cool environment, which feels so startup-y. And yet the next day you're with Microsoft, the contrast here of real true startups in startup houses and raw cultures that are emerging. They're interacting like you are with the biggest players in the world of technology inside of a few months.

48:37It's just a bizarre set of circumstances. How would you describe the playing field to an outsider? I think it's still very greenfield. I mean, the technology itself is so early. We're like one or two years in, in some sense, to all of this stuff really being valuable and getting there. One thing that I think about a lot is software is just so big. We obviously know that, but there's so many facets of software and so many details and so many different things that can be done and can be built. And so one way that I think about it is each of the use cases that we're talking about, there are enormous and really, really meaningful businesses that are built and just do that use case.

49:12A lot of folks are talking about AI for data observability, for example. Datadog is a great business. It's making billions of revenue. It's something like a$50 or$60 billion market cap. It's doing very, very well and growing quickly. And as you can imagine, there's so much to do with AI for data observability in some form. Incident response is a huge thing that's only going to get bigger and bigger. And there's a lot of great businesses in the space, like PagerD is a great business. But AI is going to make these things, again, orders of magnitude bigger. It's true with testing. It's true with modernization progress.

49:40It's true with migration. A lot of these big guys like Microsoft, Amazon, Google, or anyone, the amount of effort they have to go through to migrate customers onto their platform, even today, is enormous. It's billions and billions of dollars every year. So all of these are just such enormous spaces. And there's still so many questions even about the product form factor. And as the technology gets better and better and better. The kinds of use cases that we'll be able to support as a space are only going to get bigger. What do you think the world is overestimating and underestimating right now about your whole world, about the automatability of software engineering?

50:16The speed of it is probably a big one. One thing I always think about, if we're just looking all the way across AI, for example, I think agents in general are going to really work everywhere. Code is one of the first for a lot of reasons, but I would almost argue one of the first agents, if you want to call it, that actually was self-driving cars. It's an actual AI that's making decisions in the real world and taking in feedback and iterating with that. People are familiar with the story of self-driving cars where I actually lived in Mountain View in 2014. I would see the Waymo's driving around old time.

50:44I never got a driver's license. I had just gotten started working at the time. I'm not going to need one. They're basically there. I'm sure I'll be able to use it. We did finally get there. I think there's still a lot of room to grow, obviously, but it did take a lot of time. I think the difference between self-driving and code and honestly, a lot of other applications I think we'll see over the next year is I think with self-driving, you really, really need to have it be at 99.99999 % sufficiency. And there's not a world where it's like, or there is, but it's not as big of a difference where it's like human and machine combined can do a lot more together.

51:18And I think that's people's typical example. It's like, okay, you know, there's going to be all the edge cases to figure out and all of that, which is true, by the way. But the nice thing with software, for example, is there actually is just a really great way to do human plus machine and do a lot more. That's the case with Devon. Devon is not the 99.9999 % solution. It looks a lot more like the 2014 Waymo, or honestly, probably even a little earlier than that. But the thing is, having Devon take a first pass or having Devon send you the code and you take a look and give a review or whatever it is like that, where Devon is doing the 90%.

51:48If you're doing one nine instead of six nines, it's still super, super useful. It actually does save you 90 % of the time, whereas you're not going to get into a car that is only 90 % consistent. And so I think there's going to be a real form factor difference in how soon we're going to see it actually really transform practical software, not just because of the capabilities, but also just because the product space is better set up for that. What was the lesson that Lunch Club taught you about the world of entrepreneurship? Was it something around market? Was it something around product? If you think back on that experience, what's the big takeaway and how are you applying that learning to what you're doing at Cognition?

52:28Probably the biggest one that stands out, honestly, is sometimes it's easier to solve a bigger problem than it is to solve a smaller problem, which is why folks talk about this. If you're going after something that's truly enormous and it's the thing and it's the most exciting thing. You'll be able to bring together a group of people who's all really excited to make it happen and is really passionate and dedicated to pushing the frontier in a way that actually sometimes make it easier. Our team, for example, our team is about 20 people right now. Of those 20, I think 14 of them actually have been founders themselves before.

53:03And it's an interesting one where we've all done companies and honestly, it's a really, really talented team. And a lot of these folks would get a blank check to raise money if that's what they wanted to do. and they wanted to do their own thing. But a lot of it is just how do we come together and build something that's really great? I think that's one of the big lessons for me. What has been the most difficult or stressful moments so far in the one-year cognition journey? Oh, there's been a lot of it. Let's do a few then. I love these. One of my favorites, all of these launches, you know how it is with startups, obviously.

53:36Things are so last minute. So we had the GA launch this week and then we obviously had our initial product launch back in March. And for both of those, the few days leading up to the launch were some of the most stressful, but just the most memorable experiences. And our launch in March was huge. There's so much you have to do to get into it. And obviously, there's the product itself and testing with early users and making sure everything is truly working the way that it's supposed to, getting the info ready for it, setting everything up. And then there's obviously all of these other things with filming the video, thinking about blog posts, thinking about the content, getting things out there, working with customers to get testimonials and things like that.

54:12And our March launch, actually, we were doing all that in addition to, we're doing a fundraise right before the launch as well, actually. And so we were doing that. We had candidates who were in the pipeline, pretty meaningful candidates that we were working on and figuring out and everything during that as well. And so that weekend was a very fun time. That launch was actually, it was an article that went out by Ashley Vance, Bloomberg article about our company. That was the article, obviously, that marked the launch. It was supposed to go out at 6am Tuesday. Tuesday. We asked them to push just a little bit later.

54:40We got it to be 9am Tuesday instead of 6am. We needed those extra few hours. We weren't done filming the video for the launch as of that night at 1 or 2am. We were still recording lines. And then there was all the blog posts and putting that all together. It was a crazy weekend. Yeah. Did not get very much sleep that night, basically. It was just a really fun moment to get together and really do everything together. It's one of the reasons actually, I think that us all just living in a working in a house together is great is because there's something about the shared experience. Shading the late night energy and everything.

55:10I feel like it wouldn't be quite the same at an office. And yeah, this launch as well, you know, actually we were all in Utah, which was kind of hilarious actually. The last minute decision, it's like, okay, we're going to Utah. And we got on a plane that day. We found a cabin that would fit us all, got into that and we were all just working on the launch. And it was the same story always. Even having done this, we got no wiser. And so we were still filming at 2 a.m. the night before and putting things together for the launch and getting all the content ready and getting the info ready. And obviously in this case, it's a full GA thing too.

55:39So we got to be ready to handle 10X the load or even more than that. So all these kinds of things that we have to do to get ready. And those are some of the funnest moments, honestly. I'm just extremely interested in launching things. I think there's so much rolled up into it that it's like business encompassed or something. Can you tell me the best and worst part about each of those two launches in retrospect? Honestly, the experience with the team is the best part. Just getting to build it together and do it together with the team. It really is the soul of startups. Yeah. The worst, I mean, there's scares and stuff that happen all the time.

56:11The night before, it's just like, hey, we did our load testing. This shit didn't work at all. All the info went down. There's no way we're going to be able to handle the load. We were expecting to handle this many concurrent dev and sessions per minute. And this did not work at all. And the last minute things we had to do, the onboarding flow, for example, last minute changes that we had to make. My co-founder, Steven, actually put out a post talking about how we shipped a whole thing where you could do a Slack connect with us and work directly with our team. That got started about 1am the night before the launch and shits the morning of the launch.

56:39And Devin built a lot of that because obviously we needed the extra bandwidth from Devin. In the moment, every single thing that you're optimizing for is how do we make this not incredibly embarrassing for us all. And sometimes you just have to put it out there and do your best. How do you think about something like like O1 Pro mode and people using that to do coding versus Devin? Yeah, I think this tool is great. And I think a lot of companies are thinking about code and are focused on code. And like we're saying, code is such a big thing. There are a lot of cases where, yeah, sure. It's like you want to build something, a single file of code out of the box, and you want to think through particular problems or whatever, and have that be a really great single file of code or something.

57:21Where, yeah, you can just go to O1 Pro and consult it. Or an AI stack overflow, if you will, where it's like you can paste in an error and it'll give you its best guess from what it has. Here's what I think the error is. With Devin, there's a ton of just these iterative flows that make a lot of natural sense. And so, for example, if it's not just writing a single file and that's all, but it's like, hey, I need you to work in my code base and build a new feature and plug it into all the existing stuff and test it out to make sure that it works, then obviously all of those iterative steps, those are the things that Devin is doing for you.

57:53And similarly, if it was a debugging thing like an AI Stack Overflow thing, for example. Actually, one of our first Devon moment, I want to say is we started building this as more like research type technology and just trying to understand multi-turn agents, iterative decision-making and stuff. And eventually it made sense to build into the Devon products. But one of the first things that really convinced us is we were trying to set up our database for our own work. We were setting up MongoDB and anyone who's done it knows what this flow looks like, basically, where it's just give you a set of commands, you try it and then it doesn't work.

58:22And then you read the error and you copy and paste it and you try to figure out, you Google what's going on, you find something and you run that, and then you run into a different error. You do that 10 times until the thing works. Everybody's setup is always a little bit different, and there's always these little details that will get you. In this case, we'd spend a while trying to set up MongoDB, and it wasn't working. So we just gave it to Dev, and we're like, hey, can you just please get this working locally? And then Devon did it. And that was the thing. So what we were saying earlier, it's not just a stack overflow where it just gives you the one answer of, oh, you ran into this error, oh, maybe you should try this.

58:53Like Devin sees the error and says, okay, let me see what's going on. Let's check what ports are open. Let me see what the schema of this database is. Let me see if the socket connection is actually set up. It's going and running all those commands to go look at that. And then it uses that information to decide, okay, what's the next thing I'm going to try? And then it tries that. And if it run into a new error, then it's able to debug that. The integrated autonomous flow is I think the biggest difference. How hard is it to build that? I'm not technical enough to ask the best version of this question.

59:20But if I think about part of the secret sauce being the iterative agentic nature of the whole thing, what are the hardest parts about building that capability? The obvious one is the model capability itself. The models are roughly like, here's what a smart person on the internet might say. But if you're now thinking, okay, I'm doing a step-by-step process to make decisions to maximize my chance, not now, but after a hundred steps that I'll have solved the issue, then it's obviously a pretty different thing that you're solving for. And so a lot of it goes into the capabilities. But I think beyond that, too, there's so much with the infra and with the whole experience that gets quite involved, as you can imagine.

59:56And so, for example, Devon has its own machine that it works in. And a lot of the features that you'd want to support, for example, like rolling back state of the machine or dealing with hot loading these quickly and being able to spin up a Devon instance with a new machine immediately, being able to go and navigate through all the previous decisions and go back and forth between these different things. or the ability to work with different devins in parallel or things like that. Yeah, there's a lot of infrastructure problems that go into that as well. And I think it just comes down to this thing where the software engineering is so messy.

1:00:27And so it really is just so different from doing problems in a vacuum and building all of these high-touch components with the real world, where you're actually able to do these things in the real world. You're actually able to use all your integrations. You're actually able to maintain your own machine state or work within your Git checkout or all of these things. A lot of the mess of that is the part that I think that really comes out from us just really focusing only on top of engineering. If you had to put your most sci-fi lenses on for the future of technology in the world, what are some things that you maybe talk about with your team just for fun that don't seem feasible yet, but might be feasible in the future that get you the most excited?

1:01:08My co-founder Walden has this great line, which I've always really loved, which is, we've been playing in Minecraft survival mode for so long, and now it's going to be time for Minecraft creative mode. Describe what that means for those that don't have 10 year old boys. Yeah. And so basically Minecraft is a sandbox open world game and it's meant to be a simulation of life. You can go and grow food. You can go and forage, you can mine, you can do all these different things. You can build stuff. It's kind of funny because there's two major modes. There's one, which is survival mode, which is you are actually just have to deal with the enemies that come out to you.

1:01:38You have to go build your own shelter and make sure you have enough food to stay alive and all these things. And there's obviously creativity within that things that you do, but you are also constrained by how hungry your own character is, or whether you're being attacked or something like that. Whereas Minecraft creative mode is basically a mode where you're able to put together whatever you want. It's not like I didn't find any iron ore today. And so I'm not going to be able to build this thing. It's like I want to build this thing. I want to try this new thing. I want to set up this really cool thing.

1:02:05There's folks who have set up entire computers and computer programs in Minecraft, which is insane. It's basically just focusing on the ideas and the imagination rather than the execution. It's not about, okay, now I'm going to have to go spend another two hours going and collecting gold or because I want to build this thing. It's like, I know exactly what I want to build. And now I have the freedom to just make that come to life. That's honestly, I think the big dream of AI as a whole is to be able to free up humans. The point of AI isn't to go and just solve all of our problems for us and leave us with nothing.

1:02:36I think it's to give us the power that, or the kinds of problems that we want to solve or the things that we want to build to give us the ability to actually just turn that into reality. Yeah. It's so funny to imagine. I'll use my own experience. When I visited you, you asked, I think the demo is incredibly powerful. So you asked me to come up with an application idea. I think it took eight minutes to build and we watched it do it. Watching the iterative nature of it feels like a real aha moment because it really does look like it's in hyperspeed doing something that a person would do as it builds something.

1:03:05But maybe the more profound realization I had was it built the thing that I described in natural language. But it wasn't until I saw the thing, the little stupid application that I made up, that I realized why that idea was bad. When I said the idea, it sounded good. And when I saw the thing, I knew why it was bad. And so I would say the thing that excites me is not that I can make my ideas, but that I can get better ideas, which I think is pretty underrated part of this whole thing that just like a computer program is going to be faster now, anyone with ideas is going to be better and faster because they can see the thing and see why it stinks.

1:03:49Does that resonate? That's a great point. It's very much the idea of turning 10 % of time that you get to spend on your ideas into 100 % of the time, whereas before, you got to get a team together, they're going to go and build the whole thing. And you spend a couple days or something building this whole thing. And then you get to do your next step of idea iteration. And by the way, Devin's still honestly, a lot of room to grow. And it's going to take a lot of progress, honestly, to get to this future. But to get to a point where it's basically just you get to do the ideation work, you get to iterate against that you get to do the new idea, you get to do the new idea, and so on.

1:04:17And instead of these 90 % implementation with the little 10 % chunks in between, you just really just get to spend all your focus on the ideas. I'm super excited about that. It seems like one of the defining aspects of cognition is the degree of friendship and trust between the leaders is extremely unusual. Maybe you can describe the origin of that. But when you're moving fast, trust is really valuable. And I'm just curious to hear about that side of your building experience so far. Pretty much all of us on the team have known each other for years and years and years. Steven, Wald and Russell.

1:04:49I've known these guys for a long time and the whole crew as well. I had always wondered about, okay, is it going to be awkward to work with your friends? There's lots of things that come up. And if anything, it's honestly the other way around where it's easier to have tough conversations because you know, obviously, we were friends before all this. All of this that we're doing is coming from a place of we are just trying to take care of each other and make something great together. When you both know that's true and when that undertone is always there, it's a lot easier to just call out stuff like, hey, I think this is totally wrong or we need to change this thing right now or things like that.

1:05:20What in your experience talking to investors, present company included, what do people not ask you about the product, the company, or the vision that they should ask you about? Yeah, it's a good question. I think one thing that's definitely different talking about it versus feeling it is that folks sometimes talk about or ask about really is just kind of the experience with the team. I think the day-to-day, it's getting the product out there and seeing what folks are saying about it and talking with our users and things like that. obviously. But ourselves, I think just how we do it and how we build it as a team is truly special, I would say.

1:05:56I think we have some of the smartest and most ambitious and most capable people, and yet somehow they're all such low ego people as well. I think that is truly the, that's the soul of it for me is the 20 of us in a house together trying to do something meaningful. It's a big difference, I think, talking about it versus really seeing it live. And so one of the things that we often do as we're really pushing with it is having folks at the house just come by and see what it's like. It's going to be 1am or 2am and stuff and people are still building and there's a lot of discussions and stuff that's moving quickly and everything.

1:06:31And that's probably the thing about building the company that I'm most grateful. What uncertainty interests you the most that's outside of your control? Interesting. Yeah. One of the uncertainties I actually think a lot about is broad adoption. It seems very clear to me now that the technology will get there or maybe already has gotten there in a lot of cases to be extremely impactful. And I think the rate at which we see it happening in different industries is going to be pretty fascinating. We can both think of things like medicine or law or things like that, where you can certainly imagine it will take time to do that.

1:07:06And then there's obviously just the whole world of distribution. I will call it a win where everyone in Baton Rouge, Louisiana is actually having a significantly better life because of the work that AI is doing. It's a different curve everywhere, obviously. I think we're lucky in software that developers are just kind of inherently curious and always want to try to do it. Fast, yeah. But I do think that rates of adoption, it's going to be a real question with this technology because I think the pace of progress is going to be such that technology in some places is going to be way ahead of reality.

1:07:34When we were first together, you did this insane car trick. Are you willing to do it for us? Yeah, let's do it. Tell us the origins of this trick. Yeah. So this is a card trick. So I wouldn't even call it a card trick. It's more of a card game. And so growing up, I always used to play this game called 24. It's a little math game. And I think it actually helps a lot with just getting the fluency in math. I think it's one thing to learn. Okay, here's addition, here's multiplication and whatever, which is obviously the first building block. But also a lot of it is just this creativity and this ability to reason with numbers and experiment with numbers.

1:08:07And so I'll just show what the game looks like. So the idea is you have four cards and they're four numbers. By the way, Jack is 11, Queen is 12, King is 13. So it'd be 4, 7, 8, and 12 are the four numbers that we're dealing with here. And the idea with this is you can use the four numbers in any order, in any combination. You got to add, subtract, multiply, divide any of these. And your goal is to put together an equation that uses all four of these and makes the number 24. And so here what I would do is 12 minus 8 is 4. 4 times 7 is 28. and 28 minus four is 24. And so you can see how it uses a lot of this creativity where it's like, hmm, messing around this number four, 24 is a multiple of four.

1:08:48And so if I could just make a six from the other three, then maybe I could do something. It turns out that you can't quite do that, but you can make a seven and subtract another four out, things like that, where you're playing with these ideas. And so at these math camps and programming camps, we used to play this game all the time. And eventually we got to the point where you learn all the combinations of 24. There's only about 1300 total combinations that you have to know. Not all of them are possible, but for the ones that are possible, you know them. And so what we did next was we did a thing where you take six cards and you're making 163.

1:09:15And so I'll just steal another six cards. And so in this case, for example, what I would do is 11 times 2 is 22. 22 minus 3 is 19. 6 plus 10 is 16. 16 times 9 is 144. And 144 plus 19 is 163. And so as you can imagine, it's a lot of fun. We would play this at the math camp during the team selection training for the U.S. team. basically in between contests, we would just be playing this game all the time and doing this. And eventually you get to the point where 163, you have a lot of the patterns down and you see things. And so then the next thing that we did was we do eight cards and we make the current year.

1:09:51And so now it's 2024. And so let's just go ahead and do that. So 13 minus 12 is one. This is actually relatively easy. I'm just gonna take one times one times one, and we still got a one. And 13 plus 10 is 23. 23 times 11 is 253. 253 times eight is 2024, which times one is still 2024. Incredible. And eventually you learn all the patterns with that. And so obviously the next fun thing that we have to do is we have to do a custom number. How many digits? Four digits? Yeah. Yeah. Let's do seven, five, three, two. Seven, five, three, two. Okay. So it's a little bigger. So we're going to do nine cards.

1:10:26Okay. Okay. So 11 times seven is 77. 77 minus two is 75. We got two tens here, which is a nice hundred. And so that makes it 7 ,500. And then from here, nine times four is 36. Three plus one is four. 36 minus four is 32. So 7 ,500 plus 32 is 75, 32. Pretty amazing. I don't know how you do it. What are you doing? What is the method behind calculation or is it multiple methods? A lot of it is intuition, I guess, that you build over time. With 163, for example, a lot of these little things where I know, okay, 11 times 15 is 165. So if I can make an 11 and a 15, and then I can make a two left over, then I could do 165 minus two.

1:11:08And then I'm searching for that. And so based on the numbers, I see I'm solving for something like that. And it depends a lot on the patterns that you get and everything. It's actually, I think, a great game for kids. This is my favorite game as a three-year-old kid or whatever, playing 24 with the four cards, because a lot of it really pushes the number sense. It's not just a question of, can you add two numbers, but can you really reason around that and you know what that means and be creative with the numbers that you have, which I think is the core skill that you really want to learn, not just the mechanical part.

1:11:36I think what you're building is so cool to me because effectively what you're doing downstream is unlocking the creativity of a lot more people than currently have the ability to build things through computer science and programming. And I just think that's an incredibly cool and amazing thing for the world. I hope you and others like you are successful in doing so. When I do these interviews ask everyone the same traditional closing question. What is the kindest thing that anyone's ever done for you? The number one thing that I come to throughout my life is honestly just all of the mentors that I've had and how much effort they truly put in.

1:12:10Even as a kid, as you can imagine, like training for the US team, for the international thing. It was funny because I never had a coach. I never had formal business person, but I did have a couple of people and one in particular who just really invested in me and believed in me. And he was the one who really just pushed. I was a high school kid. I had a lot of other things that I wanted to do besides just doing math all the time. And he would always push me and be like, you got to practice, you got to do this thing. Every time I would get something wrong, you'd be like, yeah, you can't make that mistake.

1:12:36Here's how we're going to do these things. And all the time that he put in is everything honestly comes from that. I feel like I'm really only the sum of all the mentorship that I've received. And I think the same is true in startups. I can think of a few people. And again, I can think of one in particular, who's really just coached me through a lot of this and guided me through a lot of these things and been the one to give me the harsh criticism and the feedback and hit me with the things that matter and push me to do better and to do more. And people often think of kindness as sacrifice. And I do think actually that often a lot of the greatest kindness is about building more together and doing more together.

1:13:11Scott, thank you so much for your time. Thank you so much, man. If you enjoyed this episode, check out joincolossus.com. There you'll find every episode of this podcast, complete with transcripts, show notes, and resources to keep learning. You can also sign up for our newsletter, Colossus Weekly, where we condense episodes to the big ideas, quotations, and more, as well as share the best content we find on the internet every week.

1:13:56Thank you.