Adventures in babysitting coding agents (Friends)

33:15Let me talk a little bit actually. Can I talk a little bit about how this is affecting Teams and Enterprise? You guys mind? You said it's going to start at Enterprise? I imagine it starts with individual business startup people. But you're saying it's going to start at Enterprise. Well, I guess there's both going on right now. So it's starting in both. Yeah, because obviously we're seeing, you know, VCs tell us that vibe coding is writing most of the code in startups and so on. But that's engineers. Like, I'm not seeing, like, my neighbor's vibe coding yet. That would be the sign to me that it's really spilled out there.

33:48But I am seeing my neighbor PM's vibe coding. So in that sense, it is actually starting an enterprise ahead. The idea of non-engineers using it to code. All right. To do something real. So like UX designers use it to actually fix the UI instead of putting in a ticket to make the engineer do it. And once you work with it, that's Daniel, our UX designer at Sourcegraph. He's doing that. He's badass. Why would you ever want to work with a UX designer that isn't going and fixing the UI instead of tugging on your shirt sleeve? Right. And it gives them agency. He's much happier with his job because he doesn't have to wait on us.

34:24right and we're happier because we don't have to go and uh you know implement things for him that's that feel like right they they ought to be trivial and now they are they are and so this happens with product managers too right they they can go and get stuff done without waiting on engineers and um and so that sounds like oh gosh we need fewer engineers but ultimately they need a human to be accountable for that software and they're going to want an engineer to review it and that's why we we always come full circle to everybody's going to need a lot of engineers. And I think that engineers will become a gig economy inside of enterprises.

34:58And I, and I don't think it's just engineers. I think that all special skills and specialties from finance to product management to design, all of them are going to become like a gig economy, the way this, this thing plays out. Okay. With everybody vibe coding. Cause, cause let me tell you something, Jeff Bezos predicted all of this 25 years ago. Okay. The guy was so far ahead of his time and I had no idea. I worked for him and we didn't know. So his two pizza teams are making a sort of resurgence right now. Have you heard of his two pizza teams, Amazon? Yeah, recently. This is like your team should be the size where you could feed them all with two pizzas.

35:32You can feed them with two pizzas. But more than that, the team is cross-functional. It consists of a bunch of experts from different domains. It has one person that's a customer service from the CS department to represent the customer. There's one from product. And there's one engineer and there's one finance, whatever. And they all work together, one supply chain. Whatever the problem is that you're trying to solve, and they get an objective function. They get a fitness function that they have to define and Bezos had to approve that was going to measure. They had to drive it up and to the right.

36:06And the team that I ran was customer contact reduction. and uh it was really interesting because we had the sort of autonomy and we had the sort of agency and the authority to go and make the changes that we needed to make to the to the company okay that's really hard to pull off and if you don't have a jeff bezos there to to like pull out and say well if you don't do it jeff's going to come right you can't do that at most companies you know it's really hard to get teams to cross-functionally coordinate that all changes with vibe coding okay now product manager you can put a team together that's a two pizza team and you don't even need an engineer on it because the engineer can be the ai all of them now have access you're all junior level specialists in all specialties now all humans are now junior level specialists because all specialties are available to the ai now you're not a senior level one you're not able to tell whether the ai is bullshitting you or not that takes a lot more work and time and effort but but you can get to a basic like you know how how often is an engineer needed some help from a designer or needed some help from a business owner or from a product manager and had to wait a day or had to wait a while.

37:10And it wasn't a problem specific to their project. It was just a general, I have a product manager question, that kind of thing. So what you do is you use the AI to get all of the stuff done that you can get done cross-functionally. And then you go to each dimension. Security. We already do this today. Security is a great example. They are a consultant organization in every company fundamentally. Everybody becomes like security. You with me? And people go around at the end of their project, just like at Google. Launch engineers were a specialty at Google. And when you were getting ready to launch a new service, you'd go to a launch coordination engineer and you'd be like, yo, LCE, help me.

37:50You'd schedule time with them and they'd sit down and walk you through the checklists and the playbooks and all that and pre-flights and make sure that you were doing it right. Imagine all software development working this way in enterprise from now on. And every team is this two-pizza team who is empowered to do whatever the hell they want. They're able to move independently from the rest of the organization. They're decoupled. The blocking is minimal. They can even speak to each other better, guys. An engineer and a PM historically have always been kind of dogs barking at each other because they're kind of set up at odds.

38:20And also, they don't really speak the same language. There's the old jokes about how, you know, end of day means, you know, Friday means, you know, morning to a PM and evening to an engineer, you know, in a scheduling terms, things like that. All of that stuff gets smoothed over and kind of goes away when the PM can kind of sort of query the code base themselves. And they can get engineering answers to engineering questions and they can even prototype and do engineering exploration by themselves. And then when they come to the engineer, they are so much better prepared to have a conversation, a high bandwidth conversation with the engineer.

38:53Same goes the other direction. Engineers don't have to fumble around when they're trying to talk to a UX person or a finance person. Everybody is getting smarter here. Everybody's getting leveled up. But we're all getting more important for each other because in the end, a human has to be accountable for auditing and reviewing all of the work that AIs do. You see? Yeah, yeah. So to me, it's this massive gig economy opening up. It is so cool. I'm so excited for it. I'm still not getting this gig economy part. How does that translate into gig economy? I need a product manager, man. We only need a product manager for one week on this project.

39:29But let's get one for that week. So you reserve one, right? A human. You don't want an AI. You're already using an AI. You want a human to come and look at it. That kind of thing. You see what I'm saying? Yeah, I see what you mean now. Interesting. So how will that change enterprise then if those folks are all floaters, essentially? Can they go from – is there no enterprise for them? Are they – do they sit above all the enterprises and they just gig for all the enterprises? I think what happens is – so my friend Brendan Hopper, he's a CTO of technology at Commonwealth Bank Australia, which is their central bank.

40:02He thinks about this a lot. He characterizes this big centrifuge, a centrifuge that's stratifying people. It's been doing it for all of civilization, but it's super fast right now. And the stratification right now, the people that are getting drawn up to the top in enterprise by this inexorable force are the ones who are good at AI and good at multiple AIs, good at the cognitive overhead of managing multiple work streams at once, and good at dealing with other human beings who are also managing AIs. those become the most important skills. And it doesn't matter if you're a junior or senior or what your credentials are or your degree, whatever.

40:39All that matters is can you use AI effectively? Because some people can, and they're making you look bad if you're not. I'm trying to think of an analogy because I feel like this is burgeoning for us at this moment. And obviously that's why we're having this conversation. But in a year or so, maybe five years from now, like it's just the way. Today, it's managing AIs. And tomorrow or the future, in the close future, it's just how it works. And I'm thinking like maybe side roads versus like highways. You can get to that side of town with all the back roads, and then you put a toll road in. And it sucks because you got to pay the toll.

41:16But it's the way now. That's how you go there. Because who would take the 25-minute route when you can go the five-minute route? I don't know if that's a one-to-one, but it's like there's a new way now, and it's just the way. Yeah, that's fair. That's a good analogy. We actually likened it to, yeah, you've got to walk 40 miles across the desert. Do you start walking yourself, or do you just wait for somebody to come along with a car? Because you know it's coming, like if you need to be able to get the AI. But yeah, we had a really good one, an analogy for this, which is computer graphics. I was in computer graphics in the 90s.

41:53And I was looking at the work of people who were in computer graphics in the 70s and 80s. And there was a lot of excitement back then. And if you got too excited about what we were doing at the time, which was rendering polygons, kind of starting to do scenes with lighting and different angles, really early shit, all static, right? if you got too caught up in that and you're like i like rendering polygons you were screwed because all that got moved into the hardware like the next year right nvidia started doing it in in in you know in their chips and uh and and gradually all the graphic stuff started moving down into hardware and you couldn't get too attached to any of it it just was changing it just that's the like you say adam it's that's the way it was there wasn't any fighting it it just was happening and and what you had to do is you had to change and so what what the what they taught in school and what they did at work and what they interviewed for all changed in graphics over the last 30 years to where now it's completely unrecognizable kids can make mods for skyrim using these toolkits unreal engine and all the dirty the the concerns on your mind as a graphics programmer today are so infinitely higher up the abstraction ladder and more interesting by the way than rendering polygons okay that i i'm mad at everybody who's holding on to coding right now because you have so little imagination all right and that's probably why you're holding on to it you're scared because you're going to need some imagination because the people who will excel in the new in the new economy where everybody can build is the ones who have taste the ones who have imagination and the ones who can make stuff that other people like and that's scary to a lot of people who all they wanted to do is build stuff that other people came up with I saw that similarly to the state of the job market for engineers is that there's a lot of and I empathize, obviously, with anyone in scenarios that are just challenging because of not having a job or not having an opportunity or having to deal with.

43:52I've submitted my resume, I've got nothing back, et cetera. I get that. But this person's perspective was that now engineers or this market that's sort of saturated with talent and not enough placement is essentially now you actually have to network. Now you actually have to know what you're doing. Now you actually have to have taste. and so while you can sort of just skate by with some skills because you can produce something but now you actually have to involve yourself with and visionary with other humans and collaborate in unique ways and you actually have to have a network or care about your your fellow human beings around you like that's what networking is like i care about you not because you can get me what i want but because i care about your life and i ask you about those things i truly care and as a result of that we now have a closer bond so you give me opportunities you would not give to somebody else because you like me or I like you or you care about where I'm going.

44:47That people actually have to, I don't want to say it like this, but they actually have to work beyond their skill set. They actually have to work in other bounds and boundaries, whereas before it just wasn't as required and you can sort of skate by. You know what's funny is all other knowledge work, basically all knowledge work, right? Everybody from doctors, lawyers, FBI agents, school teachers, therapists, dentists, all knowledge workers, dozens and dozens of occupations I could name, have continuing education requirements. You have to continue learning to keep your job. Software engineers have never been measured by that yardstick.

45:26We have to stay good at our job in order to keep our job, which is actually weaker. You can find jobs where you don't have to learn anything. And that's what a lot of engineers do. They get a specialty. They may be really smart working on really hard problems, but they find like a comfortable sort of sandbox in some company that needs their domain expertise. It can be anything. They can be an SRE. They can be a whatever, but they're kind of like isolated into this thing and they never have to learn anything outside of their domain or the sandbox like ever again. Learning is optional, right? They're good at their job, but they're not learning.

45:57Like they don't have to stretch out of their comfort zone. Every other profession does have to, right? pilots have to train on new airlines or whatever uh you know new new plane models you know when the old one gets retired you know and we sometimes have to like learn new tech but it's kind of at our own pace and stuff and so now software engineers that that you know have been dodging that requirement for all these decades are finally faced with this notion that that companies are going to expect them to learn how to do this workflow and i tell you man we we you know we talked to a director i was telling you we talked to a director of productivity at this big company who told us that the engineers there are starting to do this and they're starting to make the engineers that aren't doing it look bad.

46:35Now, this is becoming a serious problem. He says that, okay, let's just arbitrarily, I don't know what his exact numbers are, but let's just say arbitrarily 10 % of your engineers start, they pick up agenda coding and they become five to 10 times as productive. Now, what does that mean five to 10 times as productive? Well, the director told us, okay, was that they're submitting, you know, double digits more prs per time unit per unit time than their colleagues who aren't using agentic coding now the ai submitted prs uh they get turned back more often but the ones that are making it through are dwarfing the work of the people who are doing it by hand and performance review time is coming okay and this is a serious serious problem because there's such a discrepancy, it's such a disparity in productivity between the people who have picked up, you know, Cloud Code, Codex, Sourcegraph AMP, right?

47:31And given up their IDE and all that other old bullshit they used to do. It's so big that they're going to be embarrassed at performance time, right? What are they going to do? So they're literally sitting down and starting to have HR legal discussions about whether they need to get rid of all of the engineers, which right now are a majority in this big company who are refused to switch over to agenda coding because it's clear that it has proven itself to be better and they're refusing to do it because of all the things that you and i have just talked about it requires you to come out of your comfort zone it requires you to enter and learn things new and it requires you to interact with other people and your job role is going to change and all those things are like just they're just uh unwelcome news to a lot of people.

48:13And it's right. And the sad thing is, think about my graphics analogy again. Would you rather be rendering polygons or building Skyrim mods? My God, the answer is so obvious. We're all going to be so much more productive. We're all going to be, and your engineering skills are all going to be incredibly valuable. Trained engineers will be able to do more with AI than non-trained engineers at all times, always. Right. So just like we're headed into an incredible new world. Incredible. Stop digging your heels in is my advice to people. resistance is futile and it's not only futile it's stupid you're holding yourself back you're going to have a lot more fun in the new world once you get over the hump i don't disagree with that generally i mean there's some purists out there and it's hard because it's such an art and it's so subjective to to have this blanket view of it but i think what i think about i suppose is what is the point right if you're a software engineer or somebody who's called themselves a software engineer, a developer, whatever you want to call yourself, you have this skill set and you produce results.

49:15And that's the point is to produce some results. But if you sort of resist this scenario, what is the point of the work you do? It's to solve a problem. And it's to capture, it's to solve the problem of somebody who at some point, you mentioned Jeff Bezos. He's probably one of the most famous visionaries in our time, right? He's so famous, have done so much and was ahead of his time in some respects and in many respects in terms of like two pizza teams or predicting ai or whatever i heard a thing recently that zuckerberg predicted ai as well but who didn't or no it wasn't zuckerberg it was uh chris wastroth actually it was like a github universe like almost a decade ago where he was talking about one day ai will do x and like yeah here we are you know but i feel like what is the point of the work we do as a human race, as this becomes the way five years from now, 20, you know, 2028, which is three years from now.

50:13And if we're, if the AI is 16 times smarter or better or faster, whatever that, whatever the extrapolation is to, to go to the betterness, what is the point of our work? What is the point of our work? Is it to do the work or is it to get the result from the work? Well, that was a rhetorical question, right? So, well, it's, it's more than rhetorical. It's more like, it's a question that everybody is going to have to wrestle with. Yeah, exactly. Well, I'm over here wrestling with it. Let me provide a perspective that's slightly different from your guys'. I am not resistant to change. I'm a lifelong learner and I'm perfectly happy with being results-oriented.

50:51I'm not someone who identifies closely with the code that I write or anything like that. That being said, I'm having a hard time extracting the value from these tools that you are, Steve. and other people. I also see a lot of the results of the early vibe coding demos and stuff and the programming horror on subreddit on the subreddit for, you know, all the horror that's happening. And I'm just not a person who likes to just roll the dice and I'm getting snake eyes more often than I like. And so I'm, I'm with you and I want this future. And I believe that there's people living there, but you said, get over the hump.

51:31I haven't been able to get over the hump. I'm I'm using, I'm using AIs while I code in order to not have to Google in order to get answers, you know, pasting my errors in there. Like the whole chat bot thing, I'm with it. I've asked certain, I've done single agent, like, you know, refactor this function for me. Like I'm doing that level thing, but I'm not where you are. So how, how do we, people like me, how do I get over the hump? How do I get there? Well, uh, yeah, I really wish I could say, get my book and read it because it actually Actually, we literally like took what you just said and it's such a common refrain.

52:09Everybody wants it to work, but they're struggling with it, right? I mean, it's not like they're all just rejecting it. A lot of people just haven't figured out how to make it work yet. Right. So it feels like it's wasting my time more than anything else. And I'm like, I'll just write it myself, you know? Yeah, yeah, yeah. And so we put a lot of effort in this book, you know, into sort of like walking you through at a very, it's very conversational, the book. There's no code in it. And there's no pictures, screenshots of tools or anything like that. Did you vibe write it? No. I mean, we started with that in mind, but I hated Claude's writing so much that I insisted rewrite every single, every - I was going to say, you have way too much voice in your writing.

52:52I don't think you could allow an LLM to write for you. Claude's writing makes me physically ill now. I'm allergic to it. I can actually, we should have, you know how they have like those Hemingway, you know, what do you call the fake Hemingway contests, you know, or whatever, or fake GRRM, right? We should have a fake Claude contest or who can write most like an LLM because they're really obvious dead giveaways. So yeah, no, we didn't write, we didn't vibe code the book. It's all our voice. And, um, uh, you know, I think people are going to actually crave that after a while. Oh, I think so too.

53:24I think there's going to be a market for human written things for sure. Yeah. Like bespoke, handcrafted, artisanal. Right. Just if you're going to be a human, be as human as you can. And people are going to want that, you know, I think especially with pros. Yeah. Well, maybe, or maybe there'll be a market for a, for a bespoke artisanal code. I don't want code written by an AI. This was actually suggested by a friend of mine over in Krakow. It was her idea. But, but yeah, who knows, right? The world can go in wacky places. For sure. We don't predict. But it's really, we're in a weird spot, right?

54:00In such a weird spot. Because, man, the answer to your question is, it's hard. It's like it. Is it worth waiting, though? Is it worth waiting? That's where I get is like, can I wait six months and the tools will catch up with everything? It'll be easier. Oh, dude. Hey. Depends on who you are. Can't respond to that. The proverbial. Yeah, there's a clip right there. Just Steve Yeggy sound. I like it. Yeah, I mean, everyone's different, but okay, let's not say Jared Santo. Let's say like I'm a mid-level engineer at an insurance company who writes Java 9 to 5, and I got a backlog of JIRA tickets. You know, I'm a typical software engineer.

54:44Do I wait? Do I dive in? Am I vibe coding my free time? Am I working? Do I have agents working underneath my desk, you know, at work? Okay. So first of all, you don't do anything that your work doesn't let you do. Okay. You should only worry about whether you need to be using agents if you see that other people at your company are using agents above board, getting PRs in, and starting to work that way. As soon as that starts to happen, you're in trouble. Okay. Right? But there's a lot of hurdles for a lot of companies before anybody will get to that point. All right? Yeah. The company that told me the story was more advanced and what they are is more of like a, you know, right, a harbinger of what's to come for other companies.

55:33But it's probably six to 12 months before we get to that same, call it six months before we get to where every company has a few people who are vibe coding with agents and all of a sudden performance review starts to get awkward. right because the delta in performance so six months so in the meantime what i would do is recognize that they're kind of too raw to use right now for real work unless you really want to be kind of out there like me uh you know or some of the early adopters like simon willison right you know don't be like us you know be be conservative uh but uh but learn this stuff because there will come a time sooner than you think when your company is going to expect that You know how to use it.

56:15And the thing is, you're not going to learn it overnight. So start practicing now in your hobby time, your spare time. Here's what you should do. Anything you ever thought that you wanted to do, but it was just a little out of reach, just a little too much of a pain in the ass, just a little, little on ROI, right? All those little projects you thought about doing, doesn't matter what it is, okay? Have the AIs do those. Do all of them. spin up four consoles or has four terminals and four source graph amps and just be like yo you solve this you solve this you solve this you solve this okay there's an art to it and you will you will discover it yourself if you're just pushing on it you don't even have to read a book you'll figure it out for yourself there's no math there's no science it's an art you have to learn how to the first thing you'll learn is never talk to them always talk to the plan and then copy it out of the plan to them or make them read the plan.

57:13Never talk to an agent directly. There's all these rules that you are going to learn, okay, the hard way, but you have to start now because it's going to take you six months before you feel really comfortable with them. And I'm talking about daily use. And I mean, don't wait for it to prove itself to you. Force it to get the shit done. Hold high standards, hold it to a high bar, send it back to the drawing board a hundred times if you have to, but make it work. All right. That's how you get good at this. And and your company starts making everybody do this a year from now. A year from now. It's coming.

57:45Dun, dun, dun. It's coming. Well, friends, it's time to build the future of multi-agent software. And you can do so with Agency. That's A-G-N-T-C-Y. The Agency is an open source collective building the internet of agents. It's a collaboration layer where AI agents can discover, connect, and work across frameworks. For developers, this means standardized agent discovery tools, seamless protocols for interagent communication, and modular components to compose and scale multi-agent workflows. You can join Crew, LineChain, Lambda Index, BrowserBase, Cisco, and dozens more. The agency is dropping code, specs, and services with no strings attached.

58:29Build with other engineers who care about high-quality multi-agent software. Visit agency.org. That's A-G-N-T-C-Y dot org. And add your support once again, agency.org. A-G-N-T-C-Y dot org.

58:50Okay, is that the tactical advice you'd give? Because we've been talking somewhat theory pie in the sky. like if we was to say get tactical how do you go today besides yeah trying around like literally where do you go who do you look at where's the tip where's the information coming from how do you literally get started with agents and babysitting i mean if you're just trying to like get started started i mean you know follow simon willison follow gene kim you know follow source graph uh we have lots of good uh you know pointers and tips uh i i i've began starting to you know record my workflow and try to walk people through, you know, how I do it, where you have to keep in mind.

59:27I'm going to try to get some way to get some chapters of our book out for people to look at, because there's some really useful tips that we might be able to, you know what I mean? But right now, man, it's so raw. It's so new, you know, that to some extent, you just kind of have to try it. But tactically, it's as easy as this, man. NPMI-G, you know, what's the shortest one? at openai.co.exe. and you're a vibe coder now okay because you're literally you're you're you're you're you're codex and now you're inside or better yet you know like let's use source codex is a little on the crashy side for me and maybe it's gotten a little better recently but the reason i use amp is not actually because i work for source graph i was using cloud code happily until amp got better for me if that makes any sense and the reason i switched over to amp was that it just like you want them to be out of your way you don't want them to be in your face you kind of want the agent should just be invisible you want to be doing work and not right and claude is is really paranoid and really like uh kind of right kind of finicky and like claude's always kind of in my way because they're so worried about i don't know security or whatever sometimes claude will just be like i'm sorry i can't write your file system you know whatever weird codex crashes it's amp doesn't have any of those problems so i like amp but amp is really better maybe for enterprise customers?

1:00:51I don't know. I'm not really sure. You can try it. There's, right? You can try SourceGraph AMP. Any of the three, though, I have every morning when I sit down to work, the first thing I type is update agents. It's a script I wrote that runs NPM install on all three of them, CloudCode, Codex, and SourceGraph AMP. You really want to have at least two of those, ideally Codex and one of the other two, because they use different models, right? chat gpt versus you know clod right because when you're vibe coding with agents sometimes your agent will get stuck and a different model will blast through the problem and it's completely random like some models do better with some other problems right yeah so so tactically yeah that's how you get started take your take your favorite pet project make you know make clod code do it and uh and if when it doesn't inevitably you'll be like what is this this is garbage keep making the problem smaller, right.

1:01:45And smaller. And, you know, until you've got it doing one little thing at a time for you and build your way up to where your project's done just by talking to it and sharing a plan with it, man, that's, that's, that's going to, how you're going to, that's how you tiptoe into the new world. When you said, talk to the plan, don't talk to the agents, talk to the plan. Yeah. Can you describe exactly what that means? Well, sure. The agents have limited context windows that fill up as you're working with them or as they're working. And then they have to come back and it's all 50 first dates. They completely forget who you are.

1:02:19And all you've got is they watch a little video at the beginning of the session saying what happened. Right. And so the problem with them getting amnesia all the time is that you have to have persistent memory somewhere of what was going on. And so your number one goal is to get all that persistence into files that you two can both read, which typically is marked down because it's just plain English, plain text. And so what you do is you always have it update the plan because the plan is always gradually shifting as you knock things off or you discover things, right? But each agent has its own plan.

1:02:54Each work stream has its own plan. And the agents love to write plans. You can tell them to write plans and they'll write plans so they're blue in the face. And so you have to tell them to clean up their plans too. You have to say, this plan overrides all the other plans, so get rid of them or merge them or whatever. There's like a lot of your time in VibeCoding is actually spent planning. And I mean literally like working on the plans with the AI or sitting down and dictating. You just dictate it, right? You should talk to the AI, not type. The problem with that is that the dictation is not good today at picking up when you're talking about directory names and software names and stuff.

1:03:29Jargon, yeah. It's a mixture of the two, but often you can just dictate to the AI what the problem space is that you're trying to solve and then make it come up with a plan from that big mass of words. And now the plan will expose instantly. If you glance over the plan, you'll see, oh, it was intending to do something wrong. It was intending to do something that I wouldn't have been happy with. So the plan is also a contract. Yeah. And so, so yeah, the plan is super important and, and, and also they crash or whatever, they get sidetracked. And so if you spend a lot of time typing into one of these things, unless you're using one that has persistent history, they don't all, you're, you're, you just lost all that work.

1:04:06Your computer crashed. So just type everything in your Markdown file so you can retrieve it later, or you can take the same prompt and put it into a different agent. That's what I'm going to ask you. The plan is portable, right? It's totally portable. It also lets you work across machines. I can go upstairs and keep working. So keep it in source control. In fact, my workflow is four agents right now because cognitively, I have not been able to get myself past being able to keep four full at once, mostly because I don't code full time. But Emacs is in the middle. That's the control panel. And I think that's kind of a vision of what the future is going to look like because you're going to have a lot of agents working, a dashboard, and then you're going to have some control panel that has the plans and the status and some way to communicate with supervisor agents.

1:04:50We talked about this a little earlier before. Remember we started talking about how supervisors can do a lot of babysitting for you? And it's because of all this stuff I just told you that you have to do as a vibe coder with coding agents is you have to do a lot of stuff before with planning, and you have to do a lot of stuff after they do the work with verification. Okay? You verify it. They verify it. You rewrite the test. You run the test. You make sure they ran the test. all these things, not necessarily in that order, but pre and post work, all that stuff is really super important. And a lot of it is mechanical.

1:05:24A lot of it is repetitive. A lot of it is pattern matching. A lot of it can be done by an agent. And so you're like, damn, I could have these five agents, right? Because what is my test one doing? My test work stream is the simplest of all my work streams and the most productive. And it can write 10 ,000 to 12 ,000 lines, 15 ,000 lines of code a day, good tested code in that one work stream, because all it's doing is taking my half million line code base, it's 30 years old, and writing tests for it, new tests. Okay, no new code, just new tests. So it's very low risk. Yeah? Yeah. And so I can let that thing jam.

1:06:00In fact, I could, if I could find a way to isolate them a little better, I'm sure I will at some point, I could have many of them jamming. And the instructions are always the same. Are the tests testing all the functionality? Go double check. Are the tests hacked? We could talk about reward hacking all day, but unfortunately, just be aware that these things cheat. Cloud4 does it to 67 % less. Now it cheats 67 % less, but they still cheat. They were trained on a reward function. They were not trained not to hack that reward function. And so they will say all the tests pass, but they deleted your tests.

1:06:33And so technically they're correct, but they actually passed away. That's hilarious. Yeah. These are all things we talk about in the book. They're all things you're going to have to learn as a vibe coder. It's just the kind of facts of life, the birds and the bees of working with LLMs. Yeah. Can't you put that in like a rules document, like never delete all the tests or something? Sorry, that just reminded me of Dave Barry. Never stick your finger in that part of the doggy. Oh my gosh. It's like raising toddlers, right? Right. It's like, yes, you can put it in the rules file, but they'll ignore it because they never ignore the rules file.

1:07:08That's the other rule. Never ignore the rules file. Yeah. The problem is if you get too aggressive and greedy, you will get greedy working with these things. You'll be like, yeah, I can do more and more and more. And you'll get greedy and give them too much. And then they'll start ignoring your rules file. Okay. Because what happens is about once the context window actually studies have shown there's some initial research seems to show that they start getting confused. as early as 3 ,000 tokens in. But once you've got 50K or 100K tokens in that window, 200K window, they're starting to have to juggle a lot of stuff.

1:07:41And then all of it starts to look important to them. And so the rules file, it's more of a guidelines file. Sounds like a real human. Yeah, they are in a lot of ways. And that's actually a real problem because they're not a real human. And you're going to expect them to act like a real, you're going to get into a groove where you think you're working with a real human and then they'll make a terrible mistake and do something really weird. And you can't fire them. You can't fire them. You know, you almost want that though. Right. Like, don't you want, I mean, there's times, maybe it's not exactly a one-to-one with my children, but there's times when I'm surprised by my don't do that.

1:08:20And then they go do something and then something glorious happens as a result of like curiosity and exploration. Right. Like, isn't, isn't that something that's like kind of like a good thing in a way to like break the rules and explore when it works out well if you are neuroplastic and a lifelong learner and adaptable and all that stuff yeah yeah if you're in your comfort zone you haven't changed in five ten years and you really don't want to learn anything new then i'm really sad i'm really sad for you genuinely heart heartbroken because that's going away and even as even though we're saying there's this big theme park and it's wonderful like what if you're an introvert I'm an introvert.

1:08:58Like, believe it or not, I'm not getting energized from this talk. I'm going to have to go like sit in a dark room somewhere. You know, it's costing me energy. I don't really like you. This is hard. No, no, it's great, right? I love hanging out with people and chilling and stuff, but it drains me. Yeah. And if you're that kind of person and I'm telling you that you're going to have to go and work with a bunch of teams in your new role, you're probably going, well, what the hell, man? Well, okay, but I promise you, you don't have to work with humans. any more than you already did. There you go.

1:09:30You're working with AIs. You can boss them around. And then working with them is a lot like working with human teams. You're going to have to have some manager skills, like merging their work and keeping them from colliding and keeping them on track and stuff. But it's not the same as managing a human, and it's not that yucky. Because managing humans can be kind of icky. Because their personal lives can blur into work and all that stuff. None of that is necessary or happens when you're dealing with AI. So don't be scared that you're not going to be able to still be an engineer first and foremost.

1:10:02You will. You'll be an engineer and you'll be faster. You'll just be working differently. And that's the part that saddens me is that there's no way around it. Jeans out of the bottle. Pandora's box. You mentioned AMP a bit ago. And I'm curious, who wrote the copy for ampcode.com? I don't know. I haven't looked at code.com. Is it bad? Is it bad? Should I claim this or not? It just says everything will change. The heading at the very top changes. It's like a manifesto in a way. I mean, I can read it to you, but it's kind of like it's just talking about the models you're in for more for the tools and tokens.

1:10:41We as humans hold them back and make them ask before they can change a file. We got to give them the tools and the tokens and everything changes. What we use what we use them for, how we use them, how many we run at the same time, how they talk to each other, how they talk to you, what they even are. It's all going to change. This is all on ampcode.com. AMP is embracing it, our way of keeping up, question mark, shipping. We add and remove every day we're building for where these models are going. If that means AMP will look differently, completely different three months, so be it. If you want long-term support and the same UI in 2032.

1:11:18That kind of just goes on from there. But anyways, it's, it seems like a manifesto. I was just curious if you played a role in, in writing that because that's what a source graph amp, it points to is ampcode.com. No, I, I wasn't involved in that. But I get where I get where they're coming from. We, we are trying so many different ways to do the messaging. Like this conversation I'm having with you is an attempt at the messaging, the same messaging, right? Which is the world's moving in this direction. You can, the manifesto can work. Everybody's going to like learn it differently. Everybody gets hit differently by these things, right?

1:11:56What works for you might work, not work for somebody else. And we're not really sure what it is that's going to, we've had a lot of trouble. The skeptics are like really out in force, right? I mean, there's a lot of like still really, really severe skepticism of this stuff. And look, Look, hey, when I, you know, when I talked to Dario, his vision of the future was that he shared with me was a little bit bleaker than what he shares with the public typically. And he, you know, he and Jason Clinton, his CISO, both make what I would say pretty what people consider kind of out there predictions about how, right, there will be badged AI employees next year and that there will be, you know, you know, we'll have switched to using AI for all coding, you know, by the end of next year, and so on.

1:12:46It was still all fairly rosy compared to what he shared with me personally, okay? He's very worried. He's worried about society, okay, because of the stuff we've been talking about. Society doesn't like to change, and we're talking about millions of people having to change in the tech industry, and there's going to be fallout from that, yeah? And he's worried. He characterized society is the proverbial, you know, the classic immovable object. And tech is the unstoppable force. And when they collide next year, okay, he calls 2026 the end game, casually, without even any hint drama. It's going to be a mess.

1:13:26It's going to be a mess. And we're already starting to see it. The skeptics, the people who are like, sending a PowerPoint presentation to their managers saying, we need to stop using AI at this company. We heard that from one Fortune 100 company. We've got like, there are people are resisting hard, they're pushing back, they're finding any excuse they can to say, well, maybe for you, but not for me. And that's, I think, the origin of this messaging on AMP code is, look, we're trying every possible way to message this, and people are just like, they're not listening. Yeah. The way it ends, I fumble my words, but what it says at the end is kind of interesting and important.

1:14:06It says, if you want long-term support and the same UI in 2032, if you want to spend a maximum of$20 per month, AMP is not for you. If you want to find out where all this is going, come with us. And then it says, read the manual. And I haven't read the manual yet, but it goes on to ampcode.com slash manual. So, I mean, AMP is fundamental. We're an enterprise company, right? Sourcegraph is. So, AMP is built on the Sourcegraph stack. It's a coding agent, but it's got all the SOC2 compliance and all the way to FedRAMP, and it's got all of the enterprise security controls and auditing and admin controls, and it just goes on and on and on and on.

1:14:47Sourcegraph's been around for 11 plus years, and they say it takes 10 years to make an overnight success. Yes, if you're an enterprise, you know, you're going to want an enterprise grade coding assistant. If you're at home, right, use Klein, man. It's going to bust you. It will break you. You will go bankrupt if you try to do what I just described, generate 10 ,000, 12 ,000 lines of test code a day. Really, it'll cost you$100 ,000 a year of tokens. Okay, you can't do it. I can't do it. It's not sustainable or feasible to work the way I've been talking about at home. You can do it at work if your company's paying for it.

1:15:24Use whatever budget they gave you, right? That's going to be one of the big gating factors in this stuff taking off is the inference costs. Well, that's what I remember thinking when I first told Adam, hey, let's get Steve back on the show because I read your post in March about the rise of the junior developer or the revenge. Revenge. The revenge of. And I read that post and I was like, oh, there's lots of good thoughts here that are, you know, future looking. And you made somewhat black and white predictions, even with timings of like when this is going to happen. And you went through some of the math and I was thinking like, dang, this is expensive.

1:15:57Yeah, that's that's the one. I mean, like, it's just it's ridiculous. It's beyond expensive. Now, we are in the early, early, early, early days where, you know, like electricity and steam were no doubt very expensive to get going when they first came out. and the power that we're harnessing is, you know, of that order of magnitude, change of civilization. So, yeah, it's not surprising. But, yeah, you know, AMP is an enterprise product, and so use that at work. And then use, I would say, Klein or RueCode, or maybe there's one or two others that are open source that I've been dying to play with on my Mac M4 Mini that I just got because I understand that the M4 Mini can just run some of these big llama models.

1:16:41Really excited and maybe deep seek. So I don't know, right? There's going to be, look, look, check this out. This is why I decided I was going to write this book. You are never going to be able to afford the frontier models, right? At home, you know, or probably even at work. Frontier models are for the people with deep pockets. You're going to be able to - You mean to run them. To use them, to pay, to be accustomed. Right, to run them, yeah. To get the inference from them, yeah. Yeah, right. Unless you're at Google or Microsoft or something. Instead, you're going to be paying for cheaper models that can do the same job, but they take longer because they're dumber.

1:17:17All right? They're not as smart. So a perfect example. Opus is really expensive. I believe Claude Sonnet 3.7 is probably a lot cheaper. I haven't looked at the exact numbers. But if you want to save some money, you'll go back to Sonnet for any problem that you can. Right? So this game of finding the cheapest model is going to start to get really fun as soon as Lama or DeepSeek or one of these open models is as good as Claude Sonnet 3.7 at coding because they're getting better too. And at some point, when they're as good as Claude Sonnet, everybody will have unlimited open access to at least one agent at a time, one model running.

1:17:58Okay. Or, or actually maybe, maybe, maybe, maybe somebody clever can actually have that one model serve multiple agents on your, on your box. If they're not, you know, CPU bound, they're probably IO bound, right. Running buildings and stuff. So yeah, there's a future. I can see a future and I'm lied by future. I mean, Merry Christmas Santa's coming future this year where engineers will be able to do what I'm doing now for free for, for the cost of a, of a computer with a GPU, right. Which they already have. Everybody with a gaming computer is suddenly like, I think. So that's how the cost problem gets solved.

1:18:32And it's all predicated on the notion that all of the models are gradually getting smarter, well, exponentially getting smarter. And so we will be able to get by with cheaper models. Why? Sonnet 3.7 or a model of that cognitive power, you can give it the problem. And what they do is they brute force their way to the answer by burning tokens, right? Right. And so they'll find their way there unless you've given a problem that's just too big to fit in the context window, which is easy to do. But as long as you give them a problem of the right size, they'll find their way there eventually. They're not that smart, so it might take them longer and it'll cost you more tokens than it would have cost, say, Cloud4.

1:19:11But the tokens are so much cheaper that overall you save money. You see what I'm saying? So like, remember, remember, we always used to talk about hallucinations. When's the last time you talked about a hallucination on your podcast? Well, we still joke about it all the time because of that conversation, but I don't know if we've had like a real one of late. It's not an issue anymore. At least, I mean, if you're using agents, it's not an issue anymore because they, of course they hallucinate, but then they say, oh, that was a hallucination. They detect their own hallucination and they, they, they fix it.

1:19:38Right. So that problem, just that class of problems just kind of went away. They don't keep making the same mistake. They kind of like, yeah, I've experienced that where even if it's like a math thing, I'm like that, that's off. Oh yeah, you're right. Let me fix that. Sorry about that. And then it's done. it's not like this i went down the wrong direction for so long i'm lost and upset that the thing just took me the wrong direction right it's a it's a bit more casually fixed and not an issue well the move there now is now that they have tools that they the llm actually calls a python program that does some math for it so it doesn't actually have to do its own math it can just write the program to do the math or call a program right and actually get predictable results from a stochastic parrot, but.

1:20:20Yeah, it's getting a lot better. Well, okay. So tool use is the, is the, you just, that was an example of tool use, right? Yeah. Them using tools is like, you know, the biggest game changer since they came out. Because right, you know, if you've seen the, like the IntelliJ MCP server, you know, they, they, they'll be able to operate via MCP. They'll be able to operate any program, any application that has a platform interface, right because it other otherwise they're limited to using you know crude operating system level primitives to try to click mouses and stuff right mice and keyboards um or puppeteer so so now for yeah puppeteer type things so now everybody who's smart enough to come up with like a rest api or some sort of way in grpc or something into their application so that you can you can manipulate it programmatically which you know if they if they got the platform message most of them do it's going to be able to run your music software it's going to be able to run your whatever, all of your software and that's just incredible right because now it's not just a coding agent, it's an assistant a true assistant that can assist you with all kinds of tasks that you have every day as a developer just like I started this show with it was a new kind of task that I gave I gave Claude this morning right which was, or last night which was go find out why this is slow right which was just man that was cool right now now you can tell it go go use final cut pro to eliminate all of the ums and ahs from my right and if there's a way for it to manipulate the thing it can go do that for me right so i think we're moving in quickly in a direction where we're all going to be paying hundreds of dollars a month for these these operator type you know agents because it's going to save us more than that in terms of time right and money yeah well don't let this hit the household like how can how can like let's just i mean obviously this is a software podcast but like how do households everyday households change they want to be more efficient they want to have more fun they want to go on more vacations they want to enjoy their lives and spend more time together how does how does this impact a household or is that not even worth talking about uh do you ever read there will come soft rains by ray bradbury all right well everybody who has laughed really hard and dismayed.

1:22:42Okay. Alrighty. I'll take your word for it. What's the laugh? Tell me. What's the joke? It's a story. It's not obvious from the beginning of the story. It's a story about a house that's intelligent and it's doing things for its owners. They're not there. Okay. Don't plot. Yeah, you ruin the plot. It's a twist. There's a twist. There's a more twist. Another Ray Bradbury's greatest greatest hit came up recently the veldt it's some song or something oh no it wasn't that recently it was dead mouse yeah ray bradbury he's a he's a good author you know you gotta go back and read some of his stuff wait what was the first ray bradbury story i said it was well you're talking about the house yeah house um so uh well if you ask apple syria intelligence is here now did you hear that senior vp got fired the senior vp of apple intelligence got fired or sir of syria i think because uh somehow the sales and marketing teams did that thing that they're not supposed to do where they they go and sell something that engineers haven't built yet oh yeah and uh apple you would think would know better by now uh and so yeah heads rolled they forgot because right because they were like it's going to be in your house and you're going to be able to do it all all the stuff we just talked about they put out commercials they were promising it for like basically right around yeah they did oh man they're still recovering from that mess man Siri intelligence.

1:24:06So yeah, so not this year, apparently. And I know the Alexa team's struggling with this too, right? I'm sure they have a mandate to get LLM-based Alexa out there because Alexa is stupid. And they've talked about it. It's already in like whatever, open beta or something. But right. I mean, like, you know, to what extent can you tolerate, you know, an Alexa that could potentially, you know, teach you how to make meth or something. I don't know. Something bad, Right. Well, especially at that scale. I mean, when you have failure modes in small scale, you know, a 1 % failure rate's not the end of the world, but when you're in every house in America and around the world already and they're all talking to Alexa, you know, that 1 % hits like millions of people.

1:24:49So. Right. Right. A lot of large numbers holds them back, you know. Yeah. So, and we've seen some situations, right. Where like, I think there was a case where an LLM talked somebody into suicide. Yep. Right. You don't want Alexa doing that. Right. So that's why actually having this stuff in your house, like in that sense, is still a ways out. Like it probably at least two years would be my guess. Yeah. Well, my kids talk to Alexa all the time. And one of my the solaces I have with it is how simple and basic it is and just stays that way. Because if it goes beyond, you know, all of a sudden it's like much better.

1:25:27And now I'm actually as a father more worried and want to be more involved in those conversations because who knows, you know? So it is a higher risk factor there. Well, this is another one of those things. It's not related to coding, but it's another one of those things that worries Dario about tech pushing society harder than it wants to be pushed, right? Sure. At some point, AI is going to start making its way into our lives in ways that some people don't like. Yeah. We've already seen it with the memory stuff, right? Some people like that it has memory of you and some people don't. Right. And it's already bifurcated into these two big crowds of they want anonymous transactional interactions versus they want a best friend.

1:26:02Well, that's why I like the, it's a mode now, right? And in both ChatGPT, I think Grok has it as well, where it's like, forget me mode, or I don't know. It's like an incognito tab, basically. You can have it remember you, because it's actually very useful that it knows certain things. Like, for instance, your schedule, when it's trying to give you advice on things, like to know that you do this every day at this time. Yeah. But then there's also times where you're like, I just want that anonymous transactional answer to this thing. And I don't want you to add this to my personal profile, you know, because most of the time it's because it's completely like a non sequitur, you know, like some of the stuff that you look up, I ask a thing or I'm like this, please don't put this on my profile because I'm asking for somebody else or, you know, they're just completely free from context.

1:26:49Amazon has that problem, right? You buy one gift for your niece or nephew, and all of a sudden they're showing you kids stuff forever. And it's like, no. YouTube has a problem too. You know, like I have a, I got a mechanical failure on my kid's four-wheeler. And so I'm trying to figure out how to fix it. And all of a sudden it thinks I'm a four-wheeler enthusiast. And it's like, no, I just wanted to, I just wanted to fix this problem. Now it's fixed. I don't want to see another four-wheeler video ever again. So that's a hard problem. You know what? That's interesting enough. if you should get somebody on this show to talk about like, yeah, what's going to happen when, yeah, I just, just the general problem of, of how this stuff is going to interact with, with our kids and with us.

1:27:30Yeah. That's the concern. I think that's the concern that you alluded to earlier is the, that's why I asked the household question. Cause I know it's not, we're excited about the step change we can do in our day jobs or in our visionary missions, how we want to frame it, you know, that we can now go so fast and command agents or babysit, depending upon your perspective, is that how does household or society get impacted? And literally last night, my son asked about, I forget what he said it was called, but he said there's an AI that he wants that doesn't tell you the math problems. He was like selling me.

1:28:12He's like, dad, it won't tell me how to do one plus one or whatever the multiplication is. It won't tell me what the answer is, but it'll be my friend. It'll be it'll be something I can talk to. And I'm just like, how do I answer this? How do I respond to this? I'm like, listen, I don't know yet. Let's look at that. But at some point, we're going to have to have this conversation with our loved ones, old or young about AI, about what it truly is, how to leverage it, how. And I think at this point, it's just sort of a guess what it really is and how we'll use it. There's trepidation in my heart when it comes to how it will impact my kids.

1:28:53But at the same time, I think I can I can keep them safe to some degree. But at some point, the steamroller of life will bypass dad and I can no longer be the guardrails of my son's ability to have access to this tool who the heck knows you know so it's it's like one of those things where you sort of get to this position where you know we're using it in great ways in our in our careers and we're seeing tremendous results but then it's how does that impact our households and then that's that's the fabric of society like that's where neighborhoods are born is like my household your household boom friends and neighbors you know well i never had kids i would not want to be a kid today it sounds really tough good luck good luck good luck I actually think I would be excited to be a kid right now I mean it's like the best of times and the worst of times it is it really is I mean I would like it I would I would uh if I could be born today and be I don't know 10 years old 12 years old right in this moment I'd be kind of cool oh you're gonna be born at the age of 10 this is a nice you know i'm trying to say like if i was born in the in this latest era and i was now and i was 10 or 12 i you know the experienced somewhat wisdom-filled person that i am or at least i feel i am i come kind of hopeful about a 12 year old's life in the future of this world i think there's a lot of cool stuff that's going to happen that we just can't see because we're held back by the bounds of the past well you asked what's the point jared yes what's the point of all our work?

1:30:32And the point is everything gets better. Just like when I look back to when I was a kid, like in the seventies, you know, it was really crummy and everything was just really crummy and boring. Right. And now everything's really bright and shiny and fun. This today is going to look like the seventies in about 10 years. Gosh, it's so wild to think about that. Like I watch movies or older movies that have like older cars, eighties cars, even not even seventies cars. You're like, what a weird era no one was on phones you had to go to the pay phone to call somebody if you were not at home and if you were at home you had a 30 foot phone cord because that phone went the whole house even upstairs with this long cord what a different era you know really it was just almost yesterday and el caminos man what a crazy time el caminos right who thought of that well are they making a comeback or something so i don't know just saying going back to the 70s and 80s and just think about some of the cars they're just wild oh they're real popular for a while all right well things are getting better um steve let's check back in obviously the book will be out sometime this fall it sounds like september maybe you don't know you're not in control of that but uh we'll help share that around when it ships i can give you a link and you can pre-order it on amazon oh there you go i think that'll get you some early content and we're working on it that's all well and good we'll link that up in the show notes but then let's check back in i mean six months from now i mean you're saying it's going to be radically different in six uh let's bring you back on the pot and talk about how different it is maybe six months eight months away let's say november 19th when are you doing november 19th you're gonna book it right now let's just in time for christmas you know it's just a few weeks before you know right before Thanksgiving.

1:32:19You need to say we're going to have some Christmas presents. So looking forward to it. It'll wrap the year up for us. There you go. Yeah. Interesting. I wonder if we'll be on cloud five by then. I bet we will. We'll see. We'll see. We might be broke by then. Who knows? Cloud six. All right, guys, it's been a pleasure. Yep. As always. Thanks, Steve. Absolutely. Thanks, Steve. Bye friends. Bye friends. Bye. Have you heard? We are doing a live show in Denver, Colorado at the end of July. Saturday, July 26th at the Oriental Theater in Denver, to be exact. Join us there for the entire weekend if you want.

1:32:56We'll be meeting up at a local pub Friday night, recording live on stage on Saturday morning, hiking Red Rocks Saturday afternoon, and who knows what else. It's going to be a blast. Get all the details at changelog.com slash live. And join us if you can. Seriously, do it. Seriously, do it, do it. Thanks again to our partners at Fly.io. to our sponsors of this episode, Retool. Go to retool.com slash agents. Heroku, heroku.com slash changelogpodcast and outshift by Cisco at agency.org. That's A-G-N-T-C-Y dot org. Next week on the pod, news on Monday, Richard Feldman tells me all about the ROC programming language on Wednesday and Justin Searles is back on Friends to help us digest all the WWDC announcements on Friday.

1:33:44Have a great weekend. send this episode to your friends who might dig it and let's talk face to face in denver