AI Isn't Making Engineers 10x As Productive with Colton Voege

0:00This is an iHeart Podcast.

0:30with kids, pets, or anyone who loves an easy-to-clean, spotless sofa. With a modular design and changeable slipcovers, you can customize your sofa to fit any space and style. Whether you need a single chair, loveseat, or a luxuriously large sectional, Anabay has you covered. Visit washablesofas.com to upgrade your home. Right now, you can shop up to 60 % off store-wide with a 30-day money-back guarantee. Shop now at washablesofas.com. Add a little to your life. Offers are subject to change and certain restrictions may apply. Parking shouldn't slow you down. ParkWiz gives every driver a shortcut.

1:08Book ahead, save up to 50 % and skip the hassle of circling the block. Park smarter, park faster. ParkWiz. Download the ParkWiz app today and save every time you park. In business, they say you can have better, cheaper or faster, but you only get to pick two. What if you could have all three at the same time? That's exactly what Cohere, Thomson Reuters, and Specialized Bikes have since they upgraded to the next generation of the cloud. Oracle Cloud Infrastructure. OCI is the blazing fast platform for your infrastructure, database, application development, and AI needs, where you can run any workload in a high availability, consistently high performance environment, and spend less than you would with other clouds.

1:52How is it faster? OCI's block storage gives you more operations per second. Cheaper? OCI costs up to 50 % less for computing, 70 % less for storage, and 80 % less for networking. Better? In test after test, OCI customers report lower latency and higher bandwidth versus other clouds. This is the cloud built for AI and all your biggest workloads. Right now, with zero commitment, try OCI for free. Head to oracle.com slash strategic. That's oracle.com slash strategic. So I love Square. They make paying for stuff at a store really easy. I just use them at Court Street Grocers to buy a sandwich here in Brooklyn, New York.

2:33Their stuff's reliable, easy to use, and it just works for both businesses and their customers. Square keeps up so you don't have to slow down. Get everything you need to run and grow your business without any long-term commitments. And why wait? Right now, you can get up to$200 off Square hardware at square.com slash go slash better offline. That's S-Q-U-A-R-E dot com slash G-O slash better offline. Run your business smarter with Square. Get started today.

3:27Call Zone Media called No, AI is Not Making Engineers 10x is Powerful. Colt, thanks for joining me. Thank you for having me. So tell me a little bit about your day-to-day work. What do you do for a living? So I'm a software engineer and I work specifically in what's called web application development. So that's kind of like building rich applications that work in the web browser. So think of something like Amazon or Google Drive or something where you can do a lot of interactive features within the application within a browser. Right. So kind of like the fundament of most cloud software. Yeah, I would say that if not the majority of engineers work in web application development, then probably the plurality do right now.

4:11And it's also a large amount of how people interact with software is now in this way, though. Yeah, exactly. Yeah. And that's why most developers perhaps work in this field now. And you'd think something like that that would be perfect for AI coding environment, surely. It is definitely the most applicable place for AI coding of all of them. There's sort of a thing where people are like, oh, LLMs don't really work really well for my language, and they'll be talking about something like Rust, which is more of like a high-performance language rather than a web language. What is a high-performance language in this case?

4:53Just something that's designed to work on weak hardware or at extremely high speeds. So for example, video games aren't built in Rust. They're usually built in a language called C++, but they're built in a high-performance language because you're trying to max out as much as you can do. You're taking advantage of the high-performance compute you have. Yeah, whereas with a website like Amazon or something like that, you're really not pushing the limit in terms of interactivity. And so more modest languages are fine. Got it. So you laid it out really nicely in the piece so you're going to have to repeat a few things, I imagine.

5:34But run me through why AI coding tools can't actually make you a 10x engineer or a 10 times better engineer. Yeah, so basically, AI is really good at AI is almost shockingly good at generating code and it's almost shockingly good at generating code that runs and it can answer a lot of questions that are difficult or at least annoying is probably the better word it's really good at dealing with things that are annoying it's really good at so in coding we have this concept called boilerplate and it's just code that you have to kind of rewrite a lot. And as an engineer, ideally, you don't have to write a lot of boilerplate.

6:23Ideally, you automate things and abstract things, so you don't need to do that very often. But it's sort of like a requirement of the job. And it's really good at writing that because, you know, it's kind of like low intent, high volume code, a quantity over quality type thing. So it's really good at stuff like that. The problem is that generating code really isn't the hard part of being a software engineer. It's one of the things that really matters. And it's like, obviously, if you go to college to learn how to code, what you're going to spend most of the time doing is typing code. But it's really just like one thing that you're doing.

7:02Like in reality, you're doing a lot more thinking about like, okay, how does this work with the systems I already have? How do I avoid creating what's called tech debt? So tech debt is basically like an easy way of thinking about it is when you write code that makes writing future code harder. So an example would be, you know, you want to write like a piece of logic that, let's say, computes sales tax. If you're making an online store, you'd want to compute sales tax based on where they are. You only want to write that once. You don't want to have two different places that, you know, handle sales tax, because then if, you know, Illinois changes their sales tax rate, you have to change two places instead of one.

7:40Doesn't the very nature of code also mean that it naturally creates tech debt? You know, you can try to mitigate as much as you can, but there's even an idea like that all code is tech debt. And what LLMs, so kind of what I mentioned before, you know, LLMs are really good at producing boilerplate, but if you get to the point where boilerplate is really easy to produce and you're not constantly thinking like, okay, how do I avoid doing this in the future? you start yeah exactly you start generating more and more code and just like generating code isn't it's kind of like writing in terms of like you know writing a book like a good writer writes fewer words rather than more yeah yeah brevity is the soul of wit now it's because what i've been realizing is that there is this delta between people that actually write software and people that are excited about ai coding because i had carl brown from the internet of bugs on and it was this thing of yeah being a software engineer is not just writing 100 lines of code and then be giving the thumbs up to your boss you have to interact with various different parts of the organization early on in the pieces where you had this bit called the math and i'll link to this obviously in the episode notes where it's any software engineer has worked on actual code in an actual company knows that you have these steps and where it isn't just like i code and then i give the thumbs up and i'm done for the day you have to go through a reviewer you have to wait for them to get back to you you have to you for context switching as well we have to change different windows and do different things there's a shit ton of just intermediary work that is nothing to do with actually writing code right yeah absolutely i mean it's it really is more of you know there's parts of coding that is more science and there's parts of coding that are more art and more you know in in college it's really common if you're getting computer science degree to take communications classes because you just have to interact with a lot of people.

9:34So the standard way that a thing gets done in a software project is you have somebody called a product manager. And that person's job is to think about what the product is as it exists now and what should be in the future and what features we should build. Basically, more than anything, it's what should we build next? And then you have designers who are going to decide how that thing should look. And engineers have to be this meeting influence because they're the only ones in the process who actually know what it's like, how hard it is going to be to build something. So a frequent interaction is like, a product manager is like, oh, we should add this.

10:13And an engineer has to kind of step in and be like, no, that's borderline impossible. Or that would take six months. Right. And coding LLMs don't fix that problem. They don't. Do they actually make it easier to develop products? I think there are uses for LLMs in coding and I quite like. I'm not even denying that. I'm just like, how, like, what do those uses end up actually creating? Basically, you know, what I found is that I don't really like using LLMs for most product work, even though they're good at, you know, say like, oh, add a button here that does this. They can be good at that, especially if you have a code base that works really well with LLMs.

10:58So some programming languages, some tools, some, we call them libraries and coding, you know, they're basically like shared software. Some of those things work really well with LLMs because the LLMs are just trained on the internet. So sites like Stack Overflow and sites like LeakCode and stuff like that. And so if they have a large body of this code to work within their training data, like these tools and these languages, they tend to be better at writing them. And so, you know, I work primarily in JavaScript, as do most web application engineers, and LLMs are quite good at JavaScript. But I still don't like to use it that much because it tends to just not understand context very well.

11:41And what does that mean in practicality? So it's kind of just like, you know, understanding the existing resources that you've already built in your code base. So this is like the avoiding writing the same like sales tax thing twice, they tend to default to rewriting things. And they tend to struggle to reuse the same like style. So you know, an important thing in coding is maintaining consistent like styles of your code. So like there's there's a whole theory and practice of actual, like how your code should look like as like like how the texture like how you should avoid what prop what things you should avoid because there's there's bad patterns there's good patterns and and you know there's everything in between um i think it's very similar to like you know when image generation ai kind of came onto the scene you had a lot of people being like all right well graphic designers are done you know they're out of a job they don't need to do anything because i can generate a logo now and like it looks pretty good like i can generate a logo for my nail salon or whatever and look at it it looks great but as soon as you try to generate a second logo that looks like the first one and you know maybe has a slight modification or something like that it it's really it's really bad at it it's really bad at using the context and you're like okay now i need a graphic designer and as soon as you need um you know you end up hiring one anyway yeah exactly and so and and you know that's it's very similar to engineers you know they a lot of what you're doing is sort of working around context and avoiding uh redoing things and avoiding moving away from your existing styles and what is the the critical nature of styles is that because is that so that people looking at your stuff in the future can say okay this is what they were going for this is so that everything doesn't break.

13:35Yes, exactly. It's about consistency. It's about, so there are certain, you know, for example, JavaScript, which is the main language I work in, is almost infamous for basically allowing you as the developer to do all sorts of buck wild stuff that you should like never do coding wise. Like a lot of old patterns, a lot of recycled stuff, a lot of, it just lets you, it's an extremely varied language in the things it supports. And so what you want to do when you're writing in JavaScript is only use the good parts. You want to strategically avoid doing a bunch of bad things. And some of those bad things, there's tools that will automatically detect if you're doing them.

14:26There's nothing like sinking into luxury. At washablesofas.com, you'll find the Anabay sofa, which combines ultimate comfort and design at an affordable price. And get this, it's the only sofa that's fully machine washable from top to bottom, starting at only$699. The stain-resistant performance fabric slipcovers and cloud-like frame duvet can go straight into your wash. Perfect for anyone with kids, pets, or anyone who loves an easy-to-clean, spotless sofa. With a modular design and changeable slipcovers, you can customize your sofa to fit any space and style. Whether you need a single chair, loveseat, or a luxuriously large sectional, Anabay has you covered.

15:07Visit washablesofas.com to upgrade your home. Right now, you can shop up to 60 % off store-wide with a 30-day money-back guarantee. Shop now at washablesofas.com. Add a little... to your life. Offers are subject to change, and certain restrictions may apply. So I've shopped with Quints before they were an advertiser and after they became one, and then again before I had to record this ad, I really like them. My green over shirt in particular looks great. I use it like a jacket. It's breathable and comfortable and hangs in my body nicely. I get a lot of compliments. I liked it so much. I got it in all the different colors, along with one of their corduroy ones, which I think I pull off.

15:45And really that's the only person that matters. I also really love their linen shirts too. They're comfortable, they're breathable and they look nice. Get a lot of compliments there too. I have a few of them, love their rust colored ones as well. And in general, I really like quints. The shirts fit nicely and the rest of their clothes do too. They ship quickly, they look good, they're high quality and they partner directly with ethical factories and skip the middleman. So you get top tier fabrics and craftsmanship at half the price of similar brands. And I'm probably going to buy more from them very, very soon.

16:13Keep it classic and cool this fall with long lasting staples from Quince. Go to quince.com slash better for free shipping on your order and 365 day returns. That's Q-U-I-N-C-E dot com slash better. Free shipping and 365 day returns. Quince.com slash better. Parking shouldn't slow you down. ParkWiz gives every driver a shortcut. Book ahead, save up to 50 % and skip the hassle of circling the block. Park smarter, park faster. ParkWiz. Download the ParkWiz app today and save every time you park. So today's show is brought to you with support from Square, who I really love. They offer quick payment processing, really easy to use terminals.

16:54Everything spins up really quickly. Every store I use them in, I don't really think about the experience, which is exactly what I want from any kind of tech. The loyalty programs work really easily. They're over at Court Street Grocers, S &P, Van Leeuwen Ice Cream. They're just a very easy to use and straightforward piece of tech. I can't recommend them enough. I've heard great things from businesses using them. And as a customer, never upset to see them in a store. Square keeps up so you don't have to slow down. Get everything you need to run and grow your business without any long-term commitments.

17:24And why wait? Right now, you can get up to$200 off Square hardware at square.com slash go slash better offline. That's S-Q-U-A-R-E dot com slash G-O slash better offline. Run your business smarter with Square. Get started today. What are these bad? Is it just the job? What's so special about Java that it allows you to do that? Yeah, so JavaScript, which is technically different from Java. Yeah, no problem. Basically, the reason JavaScript has all these bad parts is because it was created in 10 days as a toy language at Mosaic, which was the precursor to Firefox, the first browser that really gained traction and set the internet.

18:13I'm old enough to have used Mosaic somehow. Yeah, so JavaScript was created in 10 days almost just as a thing to test, just something to try. Like, okay, what if we put a coding language into the browser that people could just ship with their websites? And in response to that, other companies said, okay, well, we need to support JavaScript as well. And so when Internet Explorer came out, they shipped their own version of JavaScript, which was slightly different. And so you have this engineer that was kind of built on these very hacked together principles that was then rebuilt with slightly different principles.

18:51And then what you have now is 30 years later, sort of the conglomeration of all those things into one language. And it's improved dramatically in that time. But the nature of coding languages is there's a lot of backwards compatibility, like a video game where, you know, you want to be able to build a computer that can run a game from 1991. So you want the browser to be able to still run code largely that was written in 1995 or whatever. So a lot of that stuff still exists. And what they've done is introduce new patterns that can be better. And so there's a sort of an equal amount of like avoiding that.

19:28And then also just consistency. So there are times where there's two good ways to write ways to do something, but you always want to do it only in one way so that anytime someone reads it, they see that pattern and they say, okay, I know exactly what this does. Right. And this fundamentally feels like something large language models are bad at because they don't know anything and they don't create anything unique either. They just repeat. Yeah, exactly. I mean, they are statistical inference models. And so they are very good at generating what they think should come next based on probabilities.

20:08And when you're training a large language model, you can try to push it in one way, push it in another way to say, like, no, don't do that, don't do that. But, like, trying to do that on the broad spectrum of all things code is basically impossible. And so you're going to get things that default to the way that they were done on the internet. And so much, like I said, JavaScript is 30 years old. It's extremely, it's gone through a lot of dark times in terms of patterns. And so on the internet, there are just fast swaths of terrible JavaScript. Oh, God. So it's just these models are trained on bad code as well as good code.

20:43Yeah, exactly. And, you know, I'm sure they're trying very hard when they're training the models to filter some of this stuff out, but like trying to do a broad filter on hundreds of thousands, millions even of web pages of code. And how old was it again? Like decades? I mean, JavaScript's been around for 30 years, as have most programming languages that are in use right now. Right, but this particular one seems particularly chaotic in how it's sprawled. Yes, JavaScript is a particularly authority language, yeah. So what you're describing, I'm not trying to put words in your mouth, is that this stuff doesn't do the stuff that everyone's saying.

21:22It's not replacing engineers. It doesn't even seem like it could replace engineers. It's just not. It doesn't do. In fact, maybe it's fair to say that it doesn't do software engineering. That's almost the perfect way to put it. It does coding, but it doesn't do software engineering. And software engineering is kind of this broader practice of like everything that comes together around coding. So, you know, some people really integrate LLMs deeply into their everyday work and they do similar work to what I do. So, you know, there are people who are primarily having like LLMs write their code and they can be good engineers.

22:03But they're intervening pretty constantly from what I understand. and they're having to sort of redirect it, make sure it stays on patterns and all that stuff. And there's just kind of, this is the reason that I don't really use LLMs that much is there's just a constant tension between the lack of context they have and what you want them to do and constantly reviewing the code and making sure it's up to standards when I know I could just write it and it'll take about as much time as it would take prompting and re-prompting. It's just, I get better code that way. And that's what I care about most.

22:41So do you think it's kind of a mirage almost, the productivity benefits? I think it would vary a lot. I think yes, broadly. I think that there are productivity benefits, but like these like huge outsides, like, oh my gosh, I'm galaxy brained right now. I'm doing so much. I think that that is largely a mirage based on like extrapolation of like small wins. I talk about this a little bit in my article that you know there are one thing that I use LLMs for a lot and not a lot sometimes but that they are really good at is you know sometimes you're writing code and you're like I need to write a thing that I will only run once and I will throw in the trash or I need to write a thing and it uses this tool that like I don't have the time to learn but like I'd really like to have this tool like I'd really like to just like use this once and so you can you can vibe code something and not really understand what the code does you can run it once and not you know validate the output and make sure it works fine and you know i've saved in that time you know it might have taken me five hours to like learn how to use this tool properly and like learn how to use it with good standards it could even have taken me more and instead i spent 20 minutes you know wouldn't that be dangerous because you don't know how it works exactly exactly why i like to only use it for these like one time like low you know so for example i i wrote some code uh the other week and i was basically i basically refactored some existing code so i adjusted how it worked a little bit and i realized there was a way that i could break some existing code with it but i didn't have an easy way of checking across the entire code base so we have tests in our code base that would you know catch issues but there's always some code that isn't quite up to par with tests.

24:35And so I had this idea that a really simple language parser could go through and make sure that this code was right. Something that is only like 30 lines of code, but language parsing is really complicated, and the tools to do it are, there are a lot of them, and they're very well supported, because language parsing is a huge deal. But it would take me a lot of time to learn that. So I just vibe-coded a little tool. I said, hey, find me every case where I use this function, and I call it like this in this entire code base. Or actually, I said, write a script that will find me that. And I looked at the code.

25:14I was like, yeah, that looks right. That looks right. That looks right. I ran it. It said there was no issues in my code base. So I intentionally created an issue just to make sure it worked. And it worked. It showed it. And so then I got rid of the intentional issue. And I was like, OK, this is probably good. And I pushed my code, and it turned out to be good. But that's, you know. That also seems low stakes. Exactly. And it's what I would say wouldn't scale. So if I really needed to start doing like language parsing constantly, like I was doing it daily at my job, I simply would have to learn.

25:45Like you said, you know, how do you know that there aren't any issues? I would have to learn the tools. Right. So the time that I saved was by avoiding learning for this one thing. But eventually, like, if you're going to make something your full-time job, you have to learn it because you can't fully trust the output. Because otherwise it isn't your job. Like, you were just kind of mimicking. Yeah. And, you know, you have to, you know, the LLMs make mistakes in occasionally extremely catastrophic ways. There's a thing called slop squatting. Have you heard of slop squatting? No, but please tell me.

26:23I love this term so much already. yeah so basically um you might you might have heard of like domain squatting so this is where like i think i know what this is and i'm very excited to hear more yeah so like you know this was a thing where like in 19 you know 91 you're like okay i think the internet's gonna be big so i'm gonna grab nike.com right i'm just gonna hold it so that's squatting a domain so slap squatting is basically where you um the the lms you know they're statistical inference machines they They don't actually understand what they're doing. And so sometimes they will import software.

27:04They will... Is this when it looks on GitHub for something that doesn't exist? Yes. And it will just add code to your project that's like, import this thing. And it will install it. And it'll say, okay, this is the library that you want to use. And it's not right. It's either misspelled or it's, you know, for example, you would be looking for a library called left-pad, and it would import something called left-pad with no dash. And so what people realized is that they can, you know, when the LLMs, they're statistical machines, and so they frequently, they do the same thing a lot. They make the same mistakes a lot.

27:45So you could grab that library, left pad with no dash, and put code in there that works, that does the thing that the library is supposed to do, but also retrieves all of the secrets that are in your environment or looks for crypto wallets or production database passwords and stuff like that. And if you're someone that can't read code or can't read that kind of code, you would have no idea this is happening. You have no idea. And even if you're somebody who does read code really well, if you look at that, you said it's imported left-hand out, you're like, that sounds right. That looks right. Unless you are really familiar with this library.

28:28And even if you are familiar, you might just gloss right over it. So it's really, really, really dangerous. And this is the type of thing that could, you know, like maybe if the LLM is making you even twice as productive, you know that doesn't mean much if you know there's a chance you could destroy your entire company with a catastrophic security breach you know yeah leak all your database to this this hacker um and so yeah not good yeah exactly and is that becoming more prevalent i haven't heard much of it like happening in the wild but it's it's just one of those things that is is bound to happen because again these are just just statistical models they don't they don't have the ability to really reason about the actual nature of the things that they're doing.

29:17They can try. They can make a sub-LLM call and ask the other LLM, does this look right? But then you're burning more and more tokens. And also at some point you are trusting the statistical model to measure a statistical model's ability to do its job. Yes, exactly. And it kind of devolves. Anybody who's used an LLM for coding knows that the deeper you go into a single prompt, like the more back and forth, the larger the context window, the more garbage it gets. And so like, as you have things like working off of other LLM input, which is effectively what you're doing in a large context window, you know, it's just what the LLM is sort of reprocessing the text that it generated and that you've added to it, it steadily gets worse the later in the context window.

30:05So basically, all of these sort of mitigations are, you know, they've made surprising progress with the way that things like this don't happen as just raw hallucinations of like, I think this library exists. They happen a lot less now than they used to, but they're just always a risk. And they're also always going to be there. Yeah, pretty much. It's not really something you could, unless we invent new maths. Yeah, I mean, at least with the way we approach AI right now, which is based purely on language as tokens, and it can't really fundamentally understand things outside of word probabilities.