AI vs software devs

0:05Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at Fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents. So you can launch your app near your users. Learn more at Fly.io.

0:42Hello, Jared Santo here, Practical AI's producer and managing editor of all the shows here at Changelog. Daniel and Chris took this week off, but we didn't want to leave you hanging without anything to listen to. So today's episode is going to be a little different than our usual fare. AI is permeating the entire software industry, so we found ourselves talking about its impact, sometimes in practical ways, other times in less practical ways, on many of our pods. So today we are serving you a sampler platter. You'll hear a segment from this week's JS Party podcast, where me and my two co-hosts, K-Ball and Nick Neesey, discuss the recently announced Devon project, which is making waves in developer land.

1:27You'll hear a segment from a recent GoTime episode called How Long Until I Lose My Job to AI, where Johnny Borsico and his experienced panel of friends discuss using CodeGen AI to augment your dev skills instead of replacing you. And finally, you'll hear a segment from the ChangeLog where my co-host Adam Stachowiak and I talk to Jose Valim. Jose is the creator of the Elixir programming language and he's been a guest here on Practical AI in the Past talking about Elixir AI tooling. Today you'll hear us question him regarding Elixir's place in a world increasingly influenced by large language models and how he thinks about it as a language author and promoter.

2:07Hopefully there's a little something for everyone on this episode. And if it's not approaching AI from a perspective that's compelling to you, don't worry. Your regularly scheduled programming with Chris and Daniel will be back next week. Okay, first up, it's JS Party and Devin.

2:40uh there's some news that's good there's also some news that's maybe bad maybe good i don't know the thing that everybody's talking about this week at least as we record and last week as well is Devin. Devin, D-E-V-I-N, the first AI software engineer, according to the makers of Devin, which is Cognition Labs, a new company which raised a Series A led by Founders Fund, headed up by Scott Wu, who seems to be a very intelligent person, even from a young age. If you watched that video of him doing math very quickly at ages when it seems like you shouldn't know math very quickly and um they got a demo out there of this new ai software engineer so i could say more i'll stop right there you all have probably seen the demo cable and nick or oh yeah at least heard about what's going on this is a this is a new tool which can start from scratch and do some cool stuff i'll just leave it there for now we can yeah i mean talk about the details if you're excited you too can pay for the right to have a software engineer that can only fix one in seven of your tickets and spin up lots of new ways for aws to charge you money without your oversight sounds like like an intern no just kidding uh sounds nice and what are you what are you referring to or is this uh some some specific things that devin's been up to i so high level there's a couple things that i'm referring to here so one is like they're pumping up the marketing on that this is a standalone software you know why get a coding assistant get something that can go and do your software and they publish some data on it and like it does do better than the state of the art in terms of tackling a going from a github issue to okay i'm going to actually solve this implement a change and get it to happen but like the number they published i think was 13.86 percent of issues unresolved so that's about one in seven so you pointed at a list of issues and it can independently go and solve one in seven and first off to me i'm like that is not an independent software developer like that's and furthermore i find myself asking if its success rate is one in seven how do you know which one right like are the other six those it just got stuck or is it submitted something broken right because if it sets up something broken that doesn't actually solve the issue not only do you have it only actually solving one in seven but you've added load because you have to go and debug and figure out which things are broken and like like you have a whole bunch of additional load so like i think the marketing stance there is a little um over the top relative to what's being delivered the other thing and this is around i think a part of what they do is oh it can spin up resources for you right and they show this cool demo of like you pointed at this thing and it allocates a bunch of different production resources for you and the person who's handled devops in me before and now the you know engineering leader who has to sign up off on our digital ocean or aws or google cloud or whatever expenditures you might have, looks at that and is terrified by, I'm going to give an LLM, which is known for hallucination, which is, you know, these things are not, you have to design application, and I'm building applications with LLM, but you have to design around their unpredictability and their willingness to lie.

6:11And I'm going to give that raw access to spinning up resources in my cloud. Like that sounds, well, it sounds like something I would not sign up for. I'll say that. Okay. Okay. Well, let he whose success rate at issues that is greater than one in seven cast the first stone i was wondering how what nick's ratio is over there you know like one in seven sounds about the way i would do i'd pull off the easiest one first does devin know what the easy tickets look like you know because that's the skill right there i'm over here counting on my fingers trying to see if i'm within that ratio but do you know when you fail or do you just throw out broken code and you're like here you go it's it's more of a question of do i know when i succeed i guess right which is yeah same thing you think you succeeded until you find out later that actually you failed you know that's that's been my experience or you succeeded under the constraints that you put yourself under right or that was actually specified in the ticket itself but you actually failed at some other unnamed unlisted constraints that were unknown at the time but are obviously clearly there in production and so in that context you failed it's not easy it's not easy to succeed in this world well what about uh cable what if you can't you point devon at like a five dollar a month digital ocean and say you know deploy to this and like can't you cap your risk i guess on the on the devops side probably you probably can and like i do want so i'm taking a hard skeptic stance on particularly the claim that this is an AI software engineer.

7:43Like don't hire a person, use this thing. And this is their claim. So I think it's fair for you to be that harsh on them because they say, meet Devin, the world's first fully autonomous AI software engineer. That's a very bold claim. So I think it's fair that you're being that harsh. Go ahead. They're showing some cool stuff. It looks like a pretty interesting tool to put in the hands of someone who knows what they're doing and is able to validate it and is able to say, okay, go and solve this relatively well-constrained problem where I can easily validate the correctness of your output. Go at the sandbox where I know that you're not spinning up massive amounts of resources in a way that I'm going to regret.

8:24Or even go at this non-sandbox situation, but I have the knowledge to check what you did, look at the logs, and be like, yeah, that's okay. Those are really cool things that could be really valuable that could dramatically increase somebody's productivity and those are so far from being something that i would trust independently to replace a software developer that they're not even in the same country like maybe not even in the same world like these are just completely different claims yeah i think that the sensationalism of this comes from not what it can do now but what it represents and the progress that it's made when comparing to other things like whatever it was comparing that, you know, 13 % to, to other AI chat things that can do things.

9:11Uh, it's way better than all of those. It still sucks compared to a human, but it's made like monumental progress in terms of AI. And I guess the question is, does that continue? Can it get further than that? Or will it reach some kind of limit? And then the other piece of it, I think just from a marketing thing, and I'll be honest, the only thing I've seen on it really is a a fire ship video is that it's it's already doing some work on Upwork so in a way like that's a marketing claim that they it competes against real humans for jobs truth according to them I haven't confirmed but what you said is true that they say that yes so this is the struggle with it with all of the LLM world right now and all of the AI world because on the one hand you get it we we have been in a place where we're in the rapid part of an s-curve there have been some very rapid advancements in the core capabilities of these things and they are super freaking cool like really cool and also they have a lot of limitations a lot of those limitations are baked into the architecture that's being used and so you get kind of a situation where like there's a bunch of people doing really cool stuff with this and like figuring trying to figure out what it's good for but it demos way better than it does anything reliably in production because you can get a really cool outcome you know 40 of the time some situations 70 of the time and like you show that and people like oh my gosh this is going to take over the world and i would not trust a for example ai software engineer that even that they could handle 70 % of my tickets, but 30 % of the time spins up millions of dollars of costs for me.

10:54Right? Or like other things. And once again, like, I'm not trying to take away from the technology. But I don't think these hyperbolic claims actually serve anyone, except for getting attention. They get attention. Okay, great. And you're going to get a whole bunch of people who buy this thing are disappointed. If it costs them a bunch of money, they'll sue your ass off. And like, why would you do that to yourself it's somewhat similar to generative ai in the image let's just stick with like static image world where everywhere you see is impressive results and they'll be like this new like mid journey seven is off the charts amazing here's nine examples that'll blow your mind right and if you click through on that they're all going to be very impressive like those are amazing things.

11:44But then you have to stop and think, well, Mid Journey didn't create nine examples. That blew my mind. Mid Journey probably created 40, 50, maybe 500 examples. And then you, human, decided which ones were amazing. And you cherry pick those out as the examples. And like, that's a great team work, guys, right? Computers plus humans equals better results. And so there's a cherry pick and that's the i mean that's what code review on these things will be that's what happens when you tell copilot no i did not want that function right it's all as hipster brown calls it in the chat room human in the loop and that's exactly what is necessary and i think the reason why you call them hyperbolic claims cable is because they're saying it's a fully autonomous ai software engineer human out of the loop let it rip and maybe fans of the bear will like to say let it rip but those of us who aren't fans of devin are thinking let's not let it rip too much because it might just tear the whole thing down now i'm being hyperbolic nick you're not in a long do you agree with me somewhat yeah i think like yeah it's humans who are deciding what is good out of that and kind of helping to train that going forward but in a way like i was trying to think and trying to relate this to another article i saw that wasn't about devin specifically but was about like prompt engineering as a quote unquote profession being taken over already by AI because an AI can iterate and more quickly come up with a way to answer the questions that you want by appending exactly what it wants to hear at the end of a string.

13:19And I think the example that I heard from that was like, we want you to answer this question. And it, the AI is quote unquote incentivized to answer it a little bit better if you put it into a scenario that it likes. So the AI is Captain Kirk on the Enterprise and it has to answer this question to save a planet from whatever. And the question could be, what's two plus two or something like something really simple. And by putting in all of these extra prompt words that the AI is coming up with on its own, it's making better results overall. And I'm just wondering how that marries to the idea of humans being the ones who curate the good ideas that come out of it.

13:58Well, prompt engineering, I've been convinced by swix that it's a code smell yeah like we i was at first i was convinced like this is the new thing that everybody needs to learn and i think it's just the a leaky abstraction that's we're currently dealing with as humans because the tooling's not good enough so that we have to engineer the prompts i mean google's search box is prompt engineering right like knowing how to google is it's the exact same thing it's just way harder and it's like way more magical now to like tell it the magical incantations to get the best results back out and so the fact that it knows what results are better to me is not intelligence or anything it's just like we just need that to go away and i think that's i think devin's actually an example of where they've productized and hidden a lot of the innards that we've currently been exposed to in order to make the tool work better than it would for an inexperienced user to use it like they've actually turned it into a product and i think that's great i think it's one step on a long line of iterative improvements that will make it so that prompt engineering, I mean, you're just going to basically talk to it in layman's terms, and it will know how to feed itself the correct prompt, so to speak, in order to get the goodness out.

15:12But I don't know. Okay, well, back to you. Yeah, I mean, I think, so the high level on all of this AI stuff is there's really cool stuff there. We're figuring out how to use it, and the current state is clearly... intermediate however the thing i want to keep coming back to with this is like there are things that it's like okay this technology is immature and we're going to evolve around it and you know figuring out how we handle prompts and managing prompts and what's generating them and whatever like that that fits well in that bucket and there are things that are fundamental pieces of the way the technology is designed right llms machine learning models in general are statistical probabilistic they they're very different than most things you think about in software where you're trying to make something that is logical consistent like you could put a in you get b out and that is not there with these things and so you can design applications around that and there are things that you can do to to sort of pin that down to add validation that is outside of the llm and do other things and maybe devon is doing that but i think the more we start looking at these sort of you know places that require judgment places that require precision places that like if you just make some random up it can cause a lot of problems like those are not actually like there's a fundamental thing about what the technology does that means it's not necessarily going to be a good building block for that and so making hyperbolic promises about where it's going to develop that depend on it being a fundamentally different technology than what it is feel like they are setting yourself up for a lot of heartbreak what about the job market do you think it's fundamentally affected by tools like devin as they progress over the next three to five years because we're not talking about humans out of the loop i think we're all in agreement here that that's not feasible or smart at least in today's technology plateau of llms but less humans in the loop you know that seems like it's very feasible if these tools continue to iterate and even just not have revolution but evolutionary advancements from here yeah if it makes me three to five times faster do we need three to five times fewer engineers yeah i mean i think there is so this is a technology that has the potential to dramatically impact the productivity of software engineers and i think there's a couple different things around that as we think so short term that can create some disruption, right?

17:47Short term, that means that a company that had been running on, say, five engineers and might have needed to hire and expand to 15, now they don't have to expand nearly as soon and things like that. So I think there is the potential for relatively short term disruption. I will say both the history of economics broadly and software in particular is that every time we make it easier to code, we discover there are whole worlds now that we can address and build software around that we couldn't be for. So if, for example, and this actually, there's a particular example of this that I think is interesting to dive into.

18:21So one of the big economic challenges in the tech industry in the last four or five years is that we had these massive tech companies with incredibly high revenue per employee, Google, Meta, Netflix, like the Fangs mostly, right? And so they were able to set the salary bar that was super high. They were paying ridiculous amounts of money. That's a technical term. Ridiculous for software engineers. And then when we had very low interest rates and a ton of VC money flowing in to the industry, there were lots of companies whose fundamental business economics do not support that level of salary per software engineer who were nevertheless paying that amount of salary per software engineer based on VC capital.

19:10And sort of this thesis that, okay, we'll be able to scale out of this and we'll get whatever. And I think that caused a lot of distortions and problems in the field. Now, if suddenly software engineers are three to five times more productive, the range of businesses that could use software but previously could not afford to compete with the FAANGs, et cetera, of the world, there's a whole set of business models in there that become viable because it's that much cheaper to develop software. And so I could imagine this actually dramatically expanding the number of viable either software businesses or businesses that are non-tech but would like to include software or could have custom software and dramatically expanding the number of those that happen.

19:56So I think long-term, I don't think it's a negative impact on the software engineering career path. I think that what it means to be a software engineer looks a little bit different when you have different types of tooling. That has been true as long as I've been around. Just JavaScript land, right? I remember when jQuery was a revelation. Oh, my gosh. This is going to make me so much more productive. It did make me so much more productive, all these other different things. And now the level of tooling that we have there that supports our productivity, building things in the front end is astronomical.

20:30And has that taken away from the number of people writing JavaScript? Speaking of astronomical, Astro has a new database.

20:51What's up, friends? is your code getting dragged down by joins and long query times? The problem might be your database. Try simplifying the complex with graphs. A graph database lets you model data the way it looks in the real world instead of forcing it into rows and columns. Stop asking relational databases to do more than what they were made for. Graphs work well for use cases with lots of data connections like supply chain, fraud detection, real-time analytics, and generative AI. With Neo4j, you can code in your favorite programming language and against any driver. Plus, it's easy to integrate into your tech stack.

21:28People are solving some of the world's biggest problems with graphs. And now it's your turn. Visit neo4j.com slash developer to get started. Again, neo4j.com slash developer. That's neo4j.com slash developer.

21:48Up next, we have a GoTime podcast fireside chat between longtime programmers Kent Quirk, Sharon Diorio, Stephen Pyle, and host Johnny Borsico.

22:01it's one thing to have janai you know pump out snippets of code that is part of a larger whole right whereby i'm the engineer i am engineering a solution not just i'm not just a code monkey just clacking out you know syntax i'm trying to fix a problem i'm engineering a solution to a business problem now i could go you know as high level as i can i can just open up you know copilot and chat mode and say, hey, this is what I'm trying to accomplish, start spitting out files. Now, maybe today, it can build somewhat trivial apps. I've seen YouTube videos and clips and things of it spitting out entire working React apps and all these things.

22:42And that's great. And I think over time, it's going to come even better at doing those things. But I have a hard time trying to sort of correlate that or trying to replace solution building. Because to me, solutions aren't static, right? When a business comes to me and says, hey, I need you to build a solution to this problem. I build it. They take it into production. They do stuff with it and they come back and says, hey, you know what? This is great. Now I need to change it in this way, or I need to account for this exception, or I need to account for this particular use case or this specific customer where 90 % of the time it works like this way for every customer of this type, but for this It's custom of that type.

23:25But on alternate Thursdays during the full moon, it does this completely different thing. Exactly. So now what am I, like how are we supposed to treat those like entirely made up solutions that, you know, am I just feeding that back into the system and saying, hey, so now account for these, you know, alternative approaches. Is it going to be like it was when the first generated code frameworks started hitting the scene And you'd go in and there'd be all this code. It was like, yes, super fast. If you had to do an ORM, it'd ruin all the code for you, et cetera. And then you needed to change something.

24:01And all of a sudden it was like, you know, change management in some of those days was. Yeah. Or regenerate the whole thing from scratch and oh, sorry about all your customization. Yeah. So that'll be another big test that AI has not yet proven it can do. Well, so let's talk about art for a second, though, because this is, again, a similar thing. Like, everybody's real excited. Look at the images I can generate with, you know, Mid Journey or whatever. It's stolen art. Well, right. But the point is, again, it's going out and giving you the average solution. It's going out and going, here are the things that look most like what you described that somebody else has created already.

24:40And kind of going to cobble pieces of that together. Or here's an opinion formed by the loudest voices out there that I sucked up as source data. Hell yeah. But like, I sat in a meeting today where an artist went over her design, basically her design process for a big design project. Like, here's the resources I looked at. Here's the, you know, the feeling I was going for. Here are the things I considered. I looked at these typefaces. This typeface reminded me of this, you know, building architecture, which is relevant to the site. And then that artist proceeded to churn out over the course of a couple of months, 200 pieces of support art for an event.

25:25That was a brilliant design exercise by somebody deeply steeped in art and creation who then studied the event and what the event needed and integrated all that. and yes some random person could have sat down with mid-journey and said make me this stuff and it would have been much less good but people who don't know the difference would have been sure it looks fine you know i mean we've all seen that right you know my my document has 37 fonts and 12 colors but it looks fine to me um but like there's a big difference between you know something crafted and something just slapped together. And, and yeah, I guess I think that AI is going to make it easier to slap together.

26:14But for most people though, would you argue? So here's what I'm not saying. I'm not saying that these things generated by AI, like if you're a cornwasseur of a particular art, you know, you're an architect of a particular kind of application or solution or, you know, or thing, whatever it is, you can, you can critique, right. The output of gen AI as it stands today. Again, arguably, it's going to get better at what it does, right? But you can critique the output today and be like, this is subpar, right? This is not as good as what I could have come up with. But for most people, it's good enough.

26:49It depends on what they're using it for. And so again, if you're just doing something for yourself, who the hell cares? I mean, yeah, I'll slap something together out of two by fours if I'm building it for my garage. I don't care. But if I'm going to sell it, if I'm going to make a business around it, that's the part where I'm saying, I don't think the AI stuff is there. If you're just doing a hacky project for personal use, yeah, I mean, maybe you would have had to pay somebody to come in and slap that shelf together in your garage if you didn't have the skills to do it yourself. And so now there's this kind of, you know, yes, there's a few things that I couldn't do before that now I can do today for myself.

27:25Design that invitation for my kid's birthday party. Hell yeah. I can't draw, but I can use an AI. That's, there's nothing wrong with that. And yet, you know, so yeah, there's probably some, you know, the kid next door, you're not paying 20 bucks to, to do that for you. But that's now what happens in, you know, in the future as AI evolves and improves. So, you know, did we get to this uncanny valley level of like, oh, now it's not just good enough. It's like, it's like the standard. Now it's building your whole kitchen. Right. Do we need to worry about that? And how much coding out there is most of what's out, you know, how many cruds have we ever created in our lives?

28:04How many cruds are still being created every day? Yeah. Okay. So that's a problem that's largely been solved greater or lesser degree, but yeah. I mean, it's white box, right? You make white box easy to do. Yeah. And tech has always been making things that had gates around them or real or created limited availability or, and making them more available, like artisan things that used to be only certain artisans could do that. Only musical artists had a studio and an audio engineer, and now they can go and create their own tracks with one app at home. For me, that's the part that's like, oh, I can't complain.

28:45I can't be the curmudgeon complaining about this latest thing that might make something that I do more accessible to other people. I've benefited from these other things that came along. It's time to share the wealth. I don't want to. I'm really rooting against it. I don't think it's about gatekeeping. I mean, I'm not like, I feel like a big chunk of my career has been spent trying to help people learn to program. And so I'm not thinking that the reason I'm skeptical is because I don't want other people to do what I do. I think it's more because I feel like the hype and the reality are distinct.

29:27That what the reality is producing is mostly devoid of creativity. People are confusing knowing what to look up with being creative. And I think knowing what to look up is a skill and a lot of us have it. And the better engineers I think are better at it. And so yes, AI helps to ease that problem, but knowing what to even ask about, you know, like, or looking at a new solution to a problem, that's something that I think is well beyond what's, what AI is capable of now, or in the reasonably, like the LLM model, I think is fundamentally non-creative. That's my take on it. Mmm, spicy. We're not at the unpopular opinions yet.

30:18So one thing you mentioned, like the whole teaching, like y 'all remember when maybe it was during maybe first or second Obama term or something, but there was this giant push to teach everybody how to code. Right. Like it was it was everywhere. It was in the media. It was in newspapers. It was like, you know, we need to teach our young how to program. Now I'm looking at a clip from, you know, NVIDIA CEO like three or four days ago or something saying, hey, people shouldn't learn how to program. You should now let, you know, the new programming language is a human language. I'm thinking, man, you are sitting here.

30:53You stand to gain billions of bajillion dollars, right? If your wish comes to it, because you're producing, you know, chips and stuff for these things. Of course you're going to say that, right? So, but I mean, what? I'm definitely not on the don't teach people how to program camp. No, no. I'm going to take a slightly spicy take here. I don't think he's completely off. Now, in the time we've all been engineers, we've seen waves of different things that are going to come and take our jobs. You know, offshoring and, you know, code generation, now AI. And they haven't. And my theory is that the key thing that an engineer has is the ability to communicate.

31:35And even when you're supposed to be communicating to people on the other side, product, the business, whatever affectionate term you use for them, aren't always as good at that, although it should be part of their job. But having somebody who can think back and forth, there will, I think, always be a need for those people because every CEO thinks they have the answer to every question. That's what they hate to do. Right? But they really shouldn't. If they have a business that's big enough to grow, their biggest skill is finding the people and put in the right place. So if your job right now is like doing cruds for a company that can't even explain what they want, I wouldn't worry because they're not going to be able to explain what they want to AI.

32:21Right. Yeah. No, it's the thinking logic. Yeah, there's the think about, break something down into steps and think logically. Like I once did have a client very early in my career who was a pretty good business person who really wanted to automate his business. And he was able to sit down and explain it to me. Like if he had had the tools to program, he could have written his own code because he thought about it really logically. And it was just my job to basically take dictation and turn it into Pascal for him. back in the day. But that's few and far between. Quite honestly, most people who specialize in business aren't specializing in thinking logically.

33:03They specialize in thinking about people and, like you said, about communications. So does AI then make you more what you already are? If you're a logical thinker, you'll benefit. And if you're not, you still struggle? And who gets to train the agent? I don't know. I want to be in the training side. I want to be the one doing the building of the things that you use. Fascinating. That's a good question. I think at the point when it comes to a personal AI, where it's just like, it's tuned to you, your data doesn't get shared. Then it becomes like a superpower, right? It's like you're co-pilot. But how would that work if it's only got your data?

33:47It's basically replicating to a point U. There's a generic, it's trained on the universe and then specialized for you is the way that we're seeing all of this. Yeah. It's like creating your own GPT, but it's based on a larger model. Right. Yeah, exactly. So, I mean, like my company, we built, we have a query engine that looks like SQL. and then we built an AI where we train that AI, like as part of the prompt, we basically can go out and get your data and all the names of your fields and the data types of your fields and we can plug them into the AI query so that when you say, show me my slowest service, it can go, all right, what are the fields that are named according to time, duration?