Cognition’s Scott Wu on how Devin, the AI software engineer, will work for you

0:05Hi listeners and welcome to KnowPriors. Today we are talking to a very good human software engineer and the co-founder and CEO of Cognition, Scott Wu. The team at Cognition calls themselves an AI lab focused on reasoning. And recently, they released a demo of Devon, an AI software engineer that can increasingly handle entire tasks end-to-end, with leading results on Sweebench, a software engineering benchmark of real-world GitHub issues. The demo broke the internet, at least among tech Twitter, and so we're really excited to have Scott. Welcome, Scott. Hey, great to be here. Thanks for having me.

0:37So you have been coding and then coding competitively since you were a kid. What first got you interested? I think I always really, really liked math growing up, actually. My older brother, Neil, was the first to get me into programming. So I think I learned how to program when I was around nine years old. I just fell in love with it. I think the ability to take ideas and then make them into reality was always really exciting for me. I did a lot of math and programming competitions, you know, throughout school. And by the time I finished high school, I was I was pretty set on going into tech. So can you explain for our listeners who may some set of our listeners that are not familiar, like what IOI is?

1:19Yeah, sure. So IOI is the International Olympiad of Informatics. So it's the Olympiad of code, basically. And it's it works just like the the other Olympics. Every country sets in their own team of their best coders, and it's a competitive programming competition. And there's gold medals, silver medals, bronze medals, and so on. And the whole idea is it's very algorithmic problem solving. And so you're given tasks to solve, and you have to figure out the optimal algorithms for those, and then also implement those into code. It's an interesting topic for somebody who was world-class at IOI. What is practice like for getting better at competitive programming?

2:02yeah i used to um you know programming and math competitions used to be my entire life when i was growing up um i uh you know what you think about the shower it's what you're spending all your time on it's you know how how every every problem in life that i would run into i would frame as as an algorithms problem basically um and a lot of it is is um like a lot of other disciplines it's just you know putting a lot of effort into it and being willing to to think really analytically and, you know, be very brutal about your own shortcomings and, you know, focus on the things that you're not doing well and just continue to push and improve.

2:39Is it like a known space of like domains and algorithms one can learn or is it really like learning to solve problems algorithmically really well and it could be completely new problems or domains in every competition? Yeah, there's definitely, there definitely are standard algorithms that exist. to shortest path or something or binary search trees or things like that. And so it helps a lot to learn the fundamentals. But the whole idea is that every problem in a contest is totally unique. It's a new problem that's never appeared before. And the beauty of the contest itself is in the creative problem solving that you're doing to figure out the right algorithm.

3:21And so while the fundamentals are very helpful, a lot of it is figuring out how to use each of these pieces and, you know, reduce the problem to a shortest path problem or how you, you know, modify, you know, certain algorithms to make them work for different use cases. Yeah. I feel like there's a whole generation of people now who are basically people who grew up with the internet being their hometown in some sense, you know, where a lot of their early community or people they interact with or sort of come from that world. And I feel like that's a lot of people in the AI world today. Was that true for the IOI community as well?

3:52was it largely like online interactions and trading tips with strangers who became friends and all that kind of stuff? And then if so, how's that impacted your career or your life or, you know, working with others over time? Yeah, definitely. I mean, I grew up in Baton Rouge, Louisiana, and so there were not a lot of other people who had the same kind of interest in math and programming that I did. And so, you know, a lot of these competitions were the first time that I got to meet others who had a lot of these same interests. And, you know, the competitions are once a year and maybe there's training camps or things like that that are a couple times a year.

4:27But for the large majority of the year, we'd be talking online and, you know, we had all of our own communities where we'd talk about, you know, competition problems, but also kind of, yeah, yeah, we became very close friends throughout too. Yeah. It's an incredibly smart community of people, at least, you know, competing at the highest level. And I think you said before that, um, CP has more in common with entrepreneurship than one might see from the outside initially, because it looks so like structured as a, as a competition. But, uh, you know, one of the reasons we met was, um, you, uh, knew very well somebody from high school that I backed previously.

5:06And I found out recently that you, I think you knew like Alex Wang in middle school too. And I was like, Yeah. Yeah. Yeah. No, it was a very, very tight knit group. A lot of honestly, a lot of that group, all many, many of them became founders and a lot of them went into AI. And so, no, it's cool to see. I think there's, you know, I think there's overlaps for sure. I think they're the more subtle kind of overlaps, obviously, where it's, you know, you're not literally using shortest path or something to, you know, to build your company. But, you know, a lot of the same constructs and a lot of the same principles are there.

5:43And I think one of the really big things that I think about all the time is, you know, a lot of the really great math or programming problems are kind of about, you know, what are all these assumptions that you take for granted and which one of those might actually not be true? You know, and a lot of these really tough problems and really interesting problems are about finding this creative idea that's actually something that's, you know, you would not intuitively think of at all at first. Right. And I think startups are much the same way. You know, I think a lot of building a great startup is about, you know, finding something that the world doesn't believe is true yet, but actually is, you know.

6:21and kind of learning how to think independently, how to come at problems from the very first principles mindset. And of course, as with any discipline, how to just really push yourself to grow and continue to improve over time. I think there's a lot of commonalities. It seems like a lot of what you mentioned really impacted what you ended up doing at Cognition and working on Devon, because when you launched it, a lot of people saw behavior that could exist today that they didn't think could exist. And a lot of people were waiting for the next sort of step up in LLM or foundation model to get there.

6:53Could you tell us a little bit more about Cognition and what Devon is and how it works and how you think about it in the context of rethinking what's possible today? Yeah. So Cognition got started around November and the initial group, a lot of them actually were, you know, many of my friends from this math and programming contest community. And, you know, we had all done all those, but since then we've all actually had our own kind of journeys in AI. And so, you know, I started a machine learning company called Lunch Club that I ran for about five years. You know, others on the team, my co-founder Walden, you know, built a lot of early cursor, for example.

7:31And my other co-founder, Steven, was one of the first engineers at scale AI. And our whole founding team was kind of like that. You know, we had grown up with a love of math and programming and problem solving, but we'd also spent our last, five or 10 years at these different AI and AI info companies. For one, I think code has a very special place in our hearts, but I think for two, I do think there's a lot that can be done with code. I would even say most of the progress in the world in the last 20 or 30 years has come from software. And it's crazy when you think about how actually everyone everywhere needs more software engineering, not less, right?

8:10I mean, every company is hiring for more software engineers. is everyone has more ideas than time to build them. We really felt like there was a lot to do with accelerating the pace of code. For anyone who hasn't seen it, can you describe what Devon does today? Sure, yeah. So Devon is an AI software engineer. What I mean by that is that Devon is fully able to make all of its own decisions in the same way that a human software engineer would. And so obviously writing and editing code is a huge piece of that, but also being able to work with the command line, being able to use the browser, being able to read documentation, being able to deploy or test or debug or all these pieces, Devin does autonomously.

8:49And so, you know, you can give Devin a simple prompt of what you'd like to build and Devin will just go through and do that all end to end. And you can interact with Devin as well. You know, you can see what Devin's looking at or what Devin's working on and give feedback on them too. And so it's very much meant to mirror the experience of, you know, looking over the shoulder of another engineer to be able to see what they're doing and guide them or give feedback. What has been the reaction to Devon and Cognition online and the popular discourse? And how do you think people are interpreting the product that you launched?

9:19Yeah, I mean, it's really fun to see all the reactions to Devon. We got a lot more views than we expected. It was honestly quite something to see some of my middle school math videos back on Twitter all over again. But yeah, Yeah, no, I mean, it's great to see the reaction. And as with anything, I mean, there's obviously the healthy mix of there are some who are skeptics who are saying, oh, there's no way this could possibly work. There are some who are saying, wow, we're all going to lose all of our jobs. Why do you think like engineering doesn't disappear as a class of work in the future? We're obviously very excited about Devin.

9:54But, you know, I think if anything, there's going to be more engineers, not less. And I'll kind of give two reasons why. I think the first is there's just so much demand for engineering out there. And so much demand, honestly, even in ways that we don't always think about, right? A lot of using Excel or a lot of, you know, working with various tools is in some ways is there because there's because engineering is hard, right? I think there are so many problems that could be solved with code. There's so much more that could be built with code that I think multiplying every single developer is going to give us more developers, not less.

10:30You know, the other thing I think is that Devon is very much not the type to decide what to do, you know, and I think there is always this core part of how you decide what exactly to build or what problems to solve or, you know, how particular things work. Rather than engineering going away, I think engineer actually becomes a more pure abstraction of those things, right? I think the average software chair today might spend 20 % of their time thinking through all these really fundamental problem-solving questions and 80 % of the time writing that in code. And I think they'll be able to do 5x more and they'll be able to spend all their time doing this really creative problem solving.

11:09And that's what we're really excited about. As a really loose analogy, I think of like human calculators previously, right? Like you probably don't want people doing that work and it allows you to expand to much more capable human beings that would have done that work. And, you know, one of my core optimisms around AI is that it is actually very democratizing, right? Like we are tapping into latent demand for lots of different activities where it was just unaffordable before, where like the software doesn't match the workflows that people want or it's not at sufficient quality or it's not as sophisticated as could be for an unlimited number of things.

11:49So I'm very excited about that, actually. Yeah, definitely. That's largely the case, I think, in knowledge work everywhere. even, right? Where, you know, there's so much to do and there's so many problems out there to solve. And I think enabling every human to do more is only going to help us all accelerate faster. I think this is one of the really interesting things about the product was, you know, I feel like there are more companies showing up now, kind of showing me Devon like UIs. And before a lot of people build agents and you'd start the agent and it would go off and then 30 minutes later, or come back with something that you didn't like.

12:24And in the case of Devon, you know, because it feels like a lot of agents today are almost like really eager interns, right? There's an intent to build what you wanted to, but then sometimes it goes off the beaten path and the ability to steer it back onto what you wanted to do matters a lot. And in the context of Devon, I think it's really interesting that you had those four tabs of, you know, planning and here's what we're going to do and sort of check that off. And then the shell and the code and then the browser and showing like, where it's interacting with the world. And I think that's a really powerful paradigm.

12:55And so I'm a little bit curious about what was inspiration for that? Was it literally you thought of it as, hey, I'm peering over somebody's shoulder or was it something else that really inspired that UI, which I think is starting to catch on in other areas now? Yeah, yeah. I mean, I think one of the nice things about space, obviously, is that we get to build for ourselves in a way. And we're all obviously software engineers. We ourselves use Devon all the time in our own work, even when we're building Devon. Working through each of the pieces, basically, and figuring out what it took from us to, you know, to work well with Devin.

13:25As you kind of mentioned, I mean, if an agent is, you know, you give it the task and then it goes off and does it, then, you know, it basically only has one try to do it all and to do everything perfectly. Whereas if you have a human intern or a junior engineer or whatever it is, you know, if they're working on a project over the course of the day and, you know, you have to check in four or five times and just give a quick, you know, 30 seconds of feedback on what's going on, you know, for one, it helps them learn a lot. And for two, you know, it means that they can still, you know, give you a lot of really valuable work over that time.

14:02And obviously, it's much, much faster to be doing that than, you know, to do all of those pieces yourself. I think we worked backwards from, you know, what made sense and how we would want to interact with an AI coding teammate and thought through technically how we would be able to get those pieces to work. What is Devon today better and worse at than human software engineers? How does that change how you guys use it internally? The encyclopedic knowledge is obviously really, really useful. I think honestly with DevOps and DevSetup, I think there's a lot that it's very good at. You know, I think DevOps is just kind of hard for humans.

14:40Actually, one of of the first really exciting moments with Devin was we were trying to set up, you know, get a database spinning, get Kubernetes up and whatever else, you know, for our own purposes. And we were stuck after like an hour or something, you know, going down the rabbit hole of debugging errors and stack traces and whatever. And we just asked Devin, hey, can you set this up, please? And then let us know how you did, because we can't do this right now. And Devin actually did. And that was one of the first really exciting moments for us. I think the step-by-step flow of editing and working with things, running commands live, looking at the errors that come up, all those pieces of, hey, do I have this port open?

15:26Or maybe I need to install this package or whatever, work very, very well in an agent workflow. So that's a big one. I think data analysis is another big one, which we've seen a lot as a good use case. A lot, you had a few use cases in that bucket, actually, in particular. But that's another very end-to-end flip, right? Often you're trying to find the right data sets from the internet or pull out a CSV file. Then you're sanitizing the data. Then you're running whatever your exact analysis is. And then you're building a visualization of that analysis. And, you know, being able to just do that all from start to finish with an agent is another really good use case.

16:06I mean, I think as far as things that it's worse at, you know, Devin is very much not the one who is going to be deciding what to do, if that makes sense. And so, you know, we very much think of Devin as you provide the precise formulation of what you want built or what you want done. And Devin is the one that is doing the thoughtful execution of that. Right. And so, you know, you wouldn't be able to just go to Devin and say, hey, you know, build me a great business or something or build me, you know, something like that. Because, yeah, I mean, I think a lot of Devin's focus is understanding how to take precise ideas and really formulate them into code and do that entire, you know, testing process and debugging process and installing packages and deploying and whatever else that are involved with that.

16:54What can you tell us about how it works and what work happens in the foundation model layer and the system around it and how you guys invest in improvement? Yeah. Yeah. I mean, unfortunately, I can't get too much into the details of how it works. I think it's, you know, it's obviously our one of the areas that we spend the most effort on. But, you know, I think a lot of it comes down to understanding the interface of the problem and understanding how exactly to optimize the problem that we're solving for. Right. And I think one way that I frame it, for example, is, you know, suppose you were given an issue, you know, you were given a GitHub issue and you needed to solve it.

17:36Maybe there was some bug that you needed to fix. Right. You know, there certainly is a view where you could take the entirety of the code base and you could figure out from the bug exactly what the diff was that needed to be done and write out that diff in clean text. And you can do that, right? And so, you know, some perfect intelligence, either human or AI, in theory, would be able to do that, right? Because it would be able to figure out exactly what's going on and understand exactly what needs to be fixed, right? But I think practically for humans today, and we think also for AIs today, the cleaner path to do that involves running the code yourself, reproducing the bug, or adding debugging statements and print statements and then rerunning the code, or taking a look at the logs, or asking Stack Overflow, or all of these things.

18:24And a lot of the problem that we solve is figuring out how to get Devin to think in that way and to make decisions in that way. So a lot of this planning and evaluation is what we spend our time on. How do you think this all evolves over time? So for example, say we extrapolate out one or two years and then maybe five years. Like what, what proportion of code do you think is written by agents like Devin or, uh, what do you think changes in terms of how software engineers work in both the shorter term as well as longer term? Yeah. So, I mean, I think in the longer term, like I think on the five to 10 year horizon, I mean, I think what we call software, um, I think, um, changes a lot, you know, and, and I think, um, we think of Devin or, or the steps that we're building with Devin as kind of the next generation of human computer interfaces, right.

19:11where, you know, I think 10 years from now, people will look back and think, wow, isn't it crazy that, you know, you had to learn all these esoteric languages and you had to, you know, work through all these stack traces or whatever just to be able to communicate with your computer, right? I mean, at the end of the day, software engineering today is a discipline about being able to work with your computer and to have the computer do what you want it to do. and I think that, you know, what happens with software engineering is I think software engineers everywhere get to spend all their time on, you know, the really fun part of the problem, which is taking any problem that you're given and figuring out exactly what the form of the solution should be, right?

19:52And so all of this work of, you know, what are the cases and the edge cases and the details, what are all the flows, what are the architectures to build to solve each exact problem and, you know, taking it from that to that code implementation of exactly that, you know, is something that the devons of the world will solve, right? I think the really exciting thing is there's so much demand for software, right? And so, you know, it's crazy to think about how, you know, there are 30 million software engineers in the world, but, you know, in Agnative, it's still only 0.4 % of people, right? Giving those people the ability to do much more and also enabling so many other people to be able to work with software is something that we're really excited about.

20:34In the next one to two years, I think, you know, I think obviously there's a lot of kind of broader changes that will happen. I do think software engineers, I think very quickly, you know, who work with AIs and are kind of very AI native on this trend are going to be able to multiply themselves and do more. It's a really interesting time to be an AI. I think there's a lot of rising tides and the hardware is getting better all the time. The foundation models are getting better all the time. Obviously, all of the agent work that we do is getting better and better all the time. And so I think, yeah, I think things will move pretty quickly here.

21:13What do you think are the things that will improve agents the most going forward? Is it the underlying foundation models? Is it reasoning, forms of memory, self-play, something else? I'm just sort of curious at a high level, even putting aside what Devin is doing, but like agents in general, like what, what, what is needed in the field to push these things forward? Yeah, I honestly, my honest answer is yes to all the above. You know, I think all of those things will be great. You know, I think, um, you know, even like inference speedups are going to be very useful for, for the end experience. Um, I think, um, as you said, I mean, I think better reasoning, for example, in the base models is going to be really, really great.

21:48Um, I think, you know, figuring out some of this planning and, you know, better tool use and so on. Um, the way I'd kind of describe it is, you know, I think agents today are a combination of all of these factors. And in some sense, I think there's a bit of a race between all of them where a perfect intelligence AI, in theory, wouldn't need anything else, right? It would be able to just look at the code and tell you exactly what you can change, right? At the same time, you know, some perfect tool user, for example, perhaps might also be, you know, a great solution. And I think what we'll see is, you know, all of these are going to be a rising tide and it'll be a matter of, you know, which of these spaces see the most progress as far as how much impact they have on agents.

22:34What do you think is going to be important from a human software engineer or just like human technology person five years from now? I realize that's a really long timescale in AI, but it's certainly not like encyclopedic knowledge anymore, right? Yeah. Yeah. And I mean, I think there's a meme that, you know, the hottest new programming language is English. Right. And I mean, I think there's a lot of truth to that. But with that said, I think that, you know, the software engineering fundamentals are obviously still super, super valuable. Right. People, you know, for example, like, you know, the Internet today is something that we all kind of are able to use and kind of take for granted.

23:11But people who work with these networks, it's certainly very helpful for them to understand the details of TCP. Right. And I think similarly, I think we'll be able to communicate our ideas in English and work with all these things. But understanding the internals of how computers work and understanding logic gates and a lot of these core pieces, like these core foundations, I think will still be very useful. And so whether that's algorithms or technologies or logical reasoning or things like that, I think the role of a software engineer five or 10 years from now looks something like a mix between a technical architect and a product manager today, where a lot of what you do is you take problems that you're facing or that your business is facing or whatever, and you're really thinking about and breaking down what exactly the solution should be.

24:06How do you think about it in an even farther timeframe? Because when I, if it was five years ago, I would have told either my kids or people who have kids, you know, you should study computer science and math. 20 years from now, I'm not as certain. So I'm sort of curious how you think about the future of this field, if much or all the work, including a lot of the planning is actually done by machines at some point. Yeah. I mean, I love that. So I have to say it's a worthwhile experience, even if it doesn't end up being practically useful. But no, I mean, I think a lot of these fundamentals will stay useful for a long time.

24:45There's obviously a lot of questions that come up about, you know, super intelligence and singularity and all of this. And, you know, it's very hard to predict. I think everyone in AI, it's, you know, we've all made our own predictions and, you know, tried to make our guesses, but I think it's hard to be very high confidence. But with that said, I do think that, you know, we're going to see AI's concrete impacts on work and economy and people's lives, I think, a lot sooner than that. You know, I think the way that we think about the problem is that, you know, even with the tools that are available today and the technologies that exist today, there's so much that's possible to really impact people's lives, right?

25:29And, you know, we're still very, very early in this whole AI revolution. I mean, even ChatGPT was about a year and a half ago at this point. And there's a lot more to do and a lot more to build, you know, both on the research side and on the product side. So that's the best guest I have. I had lunch, Scott, as you know, with one of your new hires today. And he's like, it just makes sense to me that like Scott Wu is going to be like the last human software engineer slash labeler, right? And I'm like, all right, like without a better plan, I'm having my kids like be the best labeler they can of that data.