Open source AI to tackle your backlog

0:03Welcome to Practical AI, the podcast that makes artificial intelligence practical, productive, productive, and accessible to all. If you like this show, you will love The Change Log. It's news on Mondays, deep technical interviews on Wednesdays, and on Fridays, an awesome talk show for your weekend enjoyment. Find us by searching for The Change Log wherever you get your podcasts. Thanks to our partners at Fly.io. Launch your AI apps in five minutes or less. Learn how at Fly.io.

0:44Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I'm CEO at Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who is a principal AI research engineer at Lockheed Martin. How are you doing, Chris? Doing well today, Daniel. How's it going? It's going great. On the road this week, talking about AI and security in various places, which is always fun. And often, you know, things come up in those sorts of talks. One of the things I've got a lot of questions about this week, actually, is the impact of AI on coding workflows and vibe coding and all of those sorts of things.

1:31And really happy to have with us some really amazing guests today to help us talk through some of those subjects and also share what they're doing both on tooling and models. We've got Robert Brennan, who is co-founder and CEO at All Hands AI. And then we have Graham Newbig, who is co-founder and chief scientist at All Hands AI and associate professor at Carnegie Mellon. How's it going, guys? It's going good. Very good. Thanks for having us. Yeah, thanks for joining. Maybe one of you or both of you could kind of give your thoughts generally. like I say, even this week at conferences, it seems like half of the questions that I'm getting are around how AI is impacting developer workflows, how many people are really using Vibe coding tools.

2:19If you're using Vibe coding tools, what impact is that having on code quality? All of these sorts of things. So I'm wondering, from the perspective of all hands and the work that you all are doing, what does the kind of environment around these code assistant vibe coding tools look like from your perspective right now? Kind of what does that ecosystem look like? And then maybe set all hands in the context of that would be helpful. For sure. Yeah. So there's a huge variety of tooling out there right now for code generation. So it's a very hard space to navigate. There's two ways I like to bifurcate the space.

2:59One is, on the one hand, you have a lot of tools that are really meant for rapid prototyping. There's some really cool stuff happening there, stuff like Lovable, Bolt.dev, V0.dev, stuff. They tend to be very visual. You're getting quick prototypes of games or websites, things like that. Some really fun stuff happening there. And it's stuff that's enabling a whole new set of people to experiment with software development. and people who maybe like designers or product managers don't really have coding experience, maybe have like very little coding experience, they can now build whole apps, which is super cool.

3:32And then on the other end of the spectrum, you have stuff that is much more oriented towards like senior developers who are shipping production code. They're working on a code base that's gonna go and serve millions of users where you have to be a little bit more careful about what's going on. Also some really cool stuff happening on that end of the spectrum. And then the other way I like to bifurcate this space is that you have some tools that are very tactical. So stuff like GitHub Copilot, where it's inside your IDE, it's suggesting code exactly where your cursor is inside the code base. You're zeroed in on a task and the AI is just helping you move faster through that task.

4:08And then on the other end, you have these tools that are much more agendic. They're able to just take a quick human description of a problem and then go off and work for five, 10, 15 minutes while you go get a cup of coffee or work on a different problem or catch up on email. And then it comes back to you later with a solution. And OpenHand sits basically on the right end of both of those spectrums, right? We are really oriented towards senior engineers who are working on production-grade code bases, and we're really oriented towards those more agentic workflows where you're giving an agent something to work on and it can iterate forward on its own without you having to babysit it, without you having to be squinting at your computer screen trying to figure out where you should be editing.

4:50Yeah, that's super helpful. I'm wondering, this might be an interesting question, but I know, you know, Graham, we've run into each other in the past as related to human language related work, you know, in another context. I'm wondering from your perspective as kind of chief scientist, but also a researcher, as you've dug into this all hands project and work and product, what has been surprising in terms of challenges around this and maybe things that were surprising in terms of, you know, easier than what you might have thought? Any thoughts there? Yeah, it's a great question. Thinking back in hindsight, it's kind of sometimes hard to come up with surprising things because the things that were formerly surprising now seem kind of obvious.

5:42But one of the things that I actually wrote a blog post about before was right when the OpenHands project started out, we were kind of on this bandwagon of trying to create a big agentic framework that you could use with and define lots of different agents. Like you could have your debugging agent, you could have your software architect agent, you could have your browsing agent and all of these things like this. And we actually implemented a framework where you could have one agent delegate to another agent and then that agent would go off and do this task and things like this. One somewhat surprising thing is how ineffective this paradigm ended up being from two perspectives.

6:26So the first perspective is it didn't really, and this is specifically for the case of software engineering. There might be other cases where this would be useful. The first is in terms of effectiveness, we found that having a single agent that just has all of the necessary context, it has the ability to write code, use a web browser to gather information and execute code, ends up being able to do a pretty large swath of tasks without a lot of specific tooling and structuring around the problems. And then the other thing is building many, many different agents is kind of a relatively large maintenance burden if they're not very easy to define.

7:12So we've basically gone kind of full in on having like a single agent that can do many, many different things. But in order to do that, it has to have the ability to pull in, you know, whatever information it needs. So we have a framework called microagents where basically you can pull in a new prompt or a new tool or something like that for a particular task. But the underlying overall agent is a single agent that can do many different things. Quick follow up on that. Just for listeners who aren't really familiar with agentic workflows and stuff, could you talk just a moment about what that means?

7:48You know, if so, if you've been developing for a number of years in the more traditional workflows that we've all, you know, kind of all started out at and and now we're hitting this this world of agentic possibilities. Could you talk a little bit about what's different for the user from where they came from in kind of traditional development environments to what agentic development workflows are like? Yeah. So, you know, I think the sort of like step one of integrating AI into your development process was like Copilot, right? Where it's really just plugging into autocomplete, right? We're all familiar with autocomplete.

8:22We've been using it for decades. It just got a thousand times better all of a sudden, instead of just completing a class name, now it's writing like, you know, several lines of code. So that was like a huge boost to my productivity when I adopted Copilot. I was like, yeah, this is amazing. And then I was still, you know, for bigger chunks of code, I was like going to chat GPT and I was like, hey, can you write a SQL statement that'll do X, Y, and Z or, you know, things like that. And often I found myself doing this workflow where I would ask chat GPT or Claude to like generate some code. I'd paste that into my IDE, run it, get an error message back, paste the error message back into Claude or chat GPT.

8:57And I just do this loop. And at some point I was like, well, this is dumb. Like I'm just shuffling text between one half and another. and that was actually when I built my first like agent basically where I built a little CLI that would just do that loop with Anthropic in the background and that's kind of like the core of what an agent is. It's doing a full loop where you basically you give a problem to the agent and say like okay I want to write a SQL statement that does x or I want to modify my app to add this new feature. The agent writes some code, it runs the code, it sees what happens, it gets some kind of output from the real world, whether that's like the output of a command or maybe, you know, the contents of a webpage or the contents of a file puts that back into the LLM's context, and then it can take one step forward closer to its goal.

9:43And then, you know, you can, as you get kind of better and better and more accurate at taking one step closer to your goal, you can take on longer and longer range tasks. So I would say in the beginning, agents were really good for things that would take like 10 steps, you know, something really simple, like implement a new test and then make sure it passes. And now they can implement things that take hundreds of steps, which is really cool. I mean, that's the changes that we've seen over the last six to 12 months is that they're able to take on these huge tasks. So I can say, implement feature X, front end, back end, and add testing.

10:18And today's agents are able to just continue executing, stepping forward into that until it comes to a full PR where all the tests are passing and it's just kind of packaged up and ready to go. Selfishly, maybe I'll pass on a question. I was sitting around with a number of people at the conference I'm at last night, and there were some opinions. This gets to some of what you were just talking about. I mean, some of what you talked about at the beginning about this being geared towards more senior, maybe more senior developers working in existing code bases or something like that, but also what you were just talking about, about that kind of workflow.

10:52Although it was kind of the opinion around the group that I was with last night that, hey, a lot of these tools might be well suited to senior engineers because you can iterate like that and actually have a sort of smell test for what's going right and what's going wrong. But not really for less experienced developers or new developers who really don't have that ability. I'm curious to understand like your perspective on that and maybe who's sort of using this and who's using it successfully, I guess, is the question. And what does that persona look like? Yeah, I think it's important to realize that like you still need to keep all the same code quality controls in place that you did before the age of AI, if not more code quality controls.

11:41Right. Everything needs to go through code review. You need somebody who's familiar with the code base to look at the changes that are happening. I would say one of the kind of failure patterns I see with the technology is a lot of times a junior, a junior engineer or someone who doesn't really know how to code, you know, vibe codes their way to like a pretty good MVP because these agents are especially good at like greenfield stuff. Right. They can build a, you know, a to do list app all day. And then as you layer on more features over the course of like weeks or months, the code base just starts to rot a bit.

12:12like the agent adds a bunch of maybe like duplicates a whole function because it couldn't find the original function or it just keeps expanding the single function so that it's like thousands of lines of code and has all these forking paths if you don't have somebody looking at the changes that are being proposed and critiquing them and like telling the agent hey you know you added this new function but we have an existing function that does that or you know this function is getting too big please refactor it if you're not looking over its shoulder and critiquing its work, the code base will just grow into this monster and you'll have to throw it all away because it's just, it's, it's beyond repair.

12:46Well, I do, I do want to get into some of, some of the kind of unique elements of all hands and the, you know, the perspectives that you all are taking. One of the things of course, that, that strikes me right away as I, you know, it's even top of the web page when you go there is your approach to do this kind of open source. So open source approach to this kind of tool for developers. I'm wondering, both of you could speak to this, but maybe Graham, you could start in terms of, obviously, you've built various projects over time and done research and been plugged into the research community. Why, from your perspective, was it important that at least some portions of some key portions of what you're building here are open source?

13:35And, you know, what you think that might mean for these kind of tools, including all hands moving forward? Yeah. So there are a number of reasons why we decided to do this open source. The first reason is I think everybody in our, you know, community believes that this is going to be very transformative technology and it may drastically change the way we do software development going forward. And we have two options. We have an option where software development is drastically changed for us by people, by other people, or there's the option where we do it ourselves together. And we believe in the latter approach.

14:22Basically, we believe that, you know, if this is going to have a big effect on software development, software developers should be able to be, you know, be able to participate in that. And that's kind of the ideological point of view. The other point of view is we also believe that from a research perspective, open source, especially from the point of view of like agent frameworks, not necessarily the underlying foundation models, But from the point of view of agent frameworks, open source is not ever really going to be behind the closed options. And the reason why is because academia and all of these people really love this topic.

15:03They really love working on it. If we have an open framework and we can provide a platform, both from the point of view of like having a good code base that's easy to experiment with and providing resources to people who want to do experimentation on these topics. then, you know, the open source community together will be just as good as, you know, any company that is working on this in a closed manner. And so instead of reinventing the wheel, we can all invent it together and come up with something really good that's good for developers, interesting for the academic community and other stuff like that.

15:36Could you talk a little bit about how you bring developers into this process? You know, since that's kind of foundational to how you're operating, could you talk a little bit about what you're looking for, how you bring people into your community and kind of ramp them up on that? Yeah. So it's kind of interesting. Our software is a little bit complex because there's necessary complexity in order to do things like make a very strong agent, give it all the tools it needs, allow it to run in a safe manner and things like this. So, you know, if people, one thing that we try to do is we try to, if people are interested, point them in the direction of issues they could start working on.

16:19We have a unique problem, which is a lot of the easy issues that would be good for developers to learn more about the code base are just solved by the agent. So we're still working through the best way to fix that. But, you know, especially front end stuff, you know, we have a new front end capability that we'd like to have. We've had a lot of people join successfully through that. And then we've had longer term research projects where we collaborate together with, you know, people in universities and, and we've been pretty successful at doing some interesting things there, I think. Cool. Yeah.

16:50I, I'm wondering, Robert, from, from the perspective of obviously sometimes this is hard to do on an audio podcast, but if you could just give a sense, like I just logged into, to all hands, you know, not that long ago online. So I see some visuals, but if you could maybe paint the picture. So there's like the, the open source side of things, which I'm assuming means people could maybe host all hands themselves, which might be interesting for some, but you also have kind of a hosted version of that. Could you just talk us through like those options, how kind of people can access this and, you know, what they'll see, how they, how they integrate or how they connect their code into all hands to get started, that kind of getting started picture.

17:38Cool. Yeah. Yeah. So for the open source, everything runs inside of Docker. So that includes the application itself. You just run Docker run and you'll see a web interface running at localhost 3000 and you can just drop in a prompt to the agent. You can also connect it to GitHub by generating a token inside of your GitHub settings, plugging that into the UI, and then you can start to pull and push to your repositories. It's a little bit tricky running things locally because not only do we run the application in Docker, but when you start a new conversation with the agent, we want to make sure the agent's work is done in a nice sandbox way so the agent gets its own Docker container to work inside of.

18:17So there's a little bit of trickiness. We have to deal with a lot of troubleshooting, why isn't Docker behaving properly kind of stuff. So it's a little bit of a difficult application to run locally. So we actually created app.allhands.dev where you can use open hands in the cloud. This is a really just like, it's pretty much, you know, one for one in terms of the functionality with the open source. But there's a bunch of convenience features because, you know, A, we have this persistent server in the cloud and B, we can take care of all the infrastructure for running these sandboxes for the agent.

18:49So like, you know, for instance, when you start up a conversation in the cloud, Sandbox comes up within like one or two seconds rather than having to wait like 30 seconds or so for it to start up on your local machine. And we also can like connect into GitHub a little bit more seamlessly because we can have an OAuth application where you just like one click log in and, you know, we can access everything. And then the cloud feature that I love more than anything is that if you leave a comment in a poll request, like say the tests are failing, you can just say, at OpenHands, please fix the tests.

19:21And because we have this long-lived server in the cloud, that can just kick off a conversation automatically, and OpenHands will just commit back to your poll request. Those are actually the interactions I love the most, where I don't have to go into the OpenHands UI and fiddle around. I just inside of GitHub or soon inside of Slack, I just, you know, summon the agent and it just does the work for me. And I get to, you know, reap the fruits at the end there. My favorite is programming from my phone. So you log in, you log into the app and then just tell it what to do. I do that while I'm walking to work.

19:53And by the time I get to work, I have a pull request to review. So, you know, it opens up a lot of possibilities if you don't have to run it locally. Yeah, yeah. I could imagine also like just in the spur of the moment, like thinking of some great feature to add. And a lot of those things are lost, right? So if you have the ability to just, I know, some people or it's when they're running on a treadmill, or they're coming out of the shower or something, they can just like pop in and give a prompt and like have some work be done and, and then finish getting ready and get into work. I love that idea.

20:24Yeah, it's funny. I feel like I'm still getting a lot of coding done despite being the CEO of the company and being in meetings all the time. Because as I'm going into a meeting, I'll just like quickly be like, hey, do X, Y, Z, go into the meeting. And then once I'm done, it's just the codes that are waiting for me. It's funny. You guys are actually already leaping ahead and answering the question I was about to ask as I was thinking, Robert, on your first answer a moment ago. So, and that's really, you know, it's dramatically changing the workflow and the, and, you know, not only the workflow, but, you know, how and where you're coding and stuff like that.

20:58As what, I'm kind of curious. I mean, this is, it's, you know, if you've been developing for a long time, this feels a little bit magical. And as you've had users come into this new workflow, what are the kind of the mindset shifts that are either challenges or maybe most welcome on the conversely that, you know, that get people productive and useful and recognizing the utility of this and benefiting from it? Because there's a little bit of a leap from kind of, you know, where they grew up into the bold new world of this. What's that mind shift like and how do you get people through that? Yeah, it's a great question.

21:37It's actually very similar to when I started managing folks. For one, like you just have to get good at thinking like, oh, no, I should delegate this. You have to like kind of have that switch flip and like your instinct is like fire up BS code and just like start working. And you have to have that moment of like, oh, no, like this is actually a good thing for the agent to work on or for my employee to work on. Uh, there's also like a little bit of a trust thing, right? Like when I first started managing folks, I wanted to micromanage them. I wanted to like tell them exactly how to do everything.

22:07Uh, and it ended up being just more work for both of us and frustrating for them. Once I learned to like trust my employees and know that like, they might not do it exactly like, like I would do it, but like, they're going to do a good job. They might need some coaching and some direction, but building, building that trust over time is really important. And it's the same thing with the agent. You know, the agent isn't always right. You do need to, you know, I like to say trust, but verify, right. You need to read its code and like, understand what it's trying to do and where it might've misunderstood something and maybe iterate a few times through, uh, either like a code review in GitHub or by just like chatting with it inside of the application itself.

22:41But yeah, very, very similar to that management experience of like learning to kind of take your hands off the keyboard and, uh, be really clear with somebody else about communicating. These are the requirements and, uh, here's how you can improve and things like that. Yeah, Graham, I have maybe a question that I also get a lot of times. One is actually related to, Chris just asked one question that I get a lot, which is the workflow related stuff. But the other question that I get a lot related to these types of tools is, hey, I've seen people create a lot of cool demos with these sorts of tools, small projects that you can kind of like sort of regenerate if it doesn't work.

23:19But if I'm working in a large existing code base to the points that were brought up earlier, that's where most development happens. What are the technical pieces that have to be in place for you to have an agent work in a kind of larger code base or an existing project and actually have the context that's needed to do things that fit, have the context of other things that exist in the code base, but also potentially the context of, you know, maybe it's a company, you know, style or other things like that? Yeah, it's a good question. For reference, the open hands agent is the largest committer to our code base.

24:04So we're definitely in our code base is rather large and complex. So I just checked now and it had 209 commits over the past three months and the next closest contributor had 142. So it's doing pretty well. But there's a bunch of technical pieces that need to go together to make that work. The underlying language model is really important. And fortunately, a lot of the kind of core language model providers are focusing on this. We're also, you know, training language models ourselves. but the underlying language model needs to have a lot of abilities. One kind of boring, but extremely important one is the ability to edit files.

24:45So about six months ago, this was a major problem for most language models. They were not able to successfully generate a diff between what a portion of the file used to look like and what the new portion of the file would look like, or they would like add an extra line or duplicate things or stuff like this. So this, this was a major problem. Claude is very good at this right now. A lot of the other language models are kind of catching up to be good at doing this. Another thing that's kind of more on the, like, especially a big problem for large code bases is identifying which files to be modifying.

25:24And this is somewhat less of a big problem than I originally thought it would be. Like I was kind of imagining that this would be a really huge problem. But actually, language models are pretty good, even if you give them no tools to specifically like search a code base or something like that. They use finding grep and like all the other tools that a normal programmer might use and are able to kind of navigate their way around the code base. But I do think that code search and other things like this can help this. And we have some preliminary results that demonstrate that it doesn't necessarily improve your resolution accuracy, but it definitely improves your speed of resolution.