Inference is the new 401k matching and what we’re learning from AI-related outages

0:04Ben:So Ben, did you hear that meta acquired multbook?

0:09Andrew:Oh yeah. I mean, I've heard it too many times at this point. It seems like everyone wants to talk about it. What's Mark Zuckerberg's like obsession with monetizing personal relationships with AI? Like what's up with that? Like it just feels like this is another step in that journey. I don't know. I mean, what do you mean?

0:27Ben:It's the perfect thing to add to his repertoire. Another website for users pretending to be people. I think that it's another notch in his belt for sure. But actually, I think it's really interesting to think that it would be something that is hungry for an acquisition. And nothing in my mind really struck me as like, oh, I want to buy a moltbook.

0:46Andrew:Yeah, I'm like, what insane experiment can I create that some company will come along and buy?

0:53Ben:Well, I do think there's an interesting telling there about maybe this being like a wink to where things are going and like the new consumer economy, because obviously you have these advertisers in their ad marketplace making the bet that in the future they're going to be selling to your agents more than they're going to be selling to you. So you might as well start owning the places where those agents go. And it could be a really interesting kind of transformation of the Internet, just depending on if it's artificial readers end up outnumbering its real ones.

1:21Andrew:kind of makes me wonder like what happens can you just make agents that create enough value on their own that they become like an acquirable target but yeah i'm just like reflecting on like how chaotic like this like open claw like has made this the ai space you know like like its inventor went to open ai now we have like the social network that was spawned out of it that's gone on to meta but then like you also just have like companies like anthropic over here that are just like quietly building co-work and like using all the same conventions, you know, but that are letting like smart people like you and me just like solve our own problems

1:58Charlie Guo:with them, you know, rather than having to like get a product out of it or something. But it's also a difference in the organizations.

2:05Ben:Anthropics somewhere, I think they built co-work in what, like 10, 12 days or something. Meta super intelligence lab. That's a place where a bunch of smart people and interesting ideas go never to be heard from again. So far, it's really just the Willy Wonka factory of AI. And so I just don't know what this means even for a notebook, but definitely something to watch.

2:26Andrew:Yeah.

2:26Charlie Guo:Well, anyways, welcome to the Friday Deploy. I'm your host, Ben Lloyd Pearson. And I'm your host, Andrew Ziegler. Here's this week's news.

2:33Andrew:Getting paid in GPU time, a playbook for harness engineering, a spate of outages, including incidents tied to the use of AI tools. and are you falling behind if you're not running agents every minute of the hour? Andrew, let's just start right at the top of that and talk about these people who are getting paid in AI compute. What's the story?

2:55Ben:Yes, so OpenAI's Greg Brockman said inference compute is increasingly driving software productivity but also becoming part of compensation packages. And one submission on levels.fyi even listed a co-pilot subscription as a benefit. So it's really interesting to think that AI usage could represent upwards of even 20 % of total engineering compensation based on some VCs kind of estimates on this news.

3:20Andrew:Yeah, you know, I think both of us can kind of relate to this. You know, we've had discussions about how, like, quickly just raw token costs can become like one of our biggest budget needs. And it can just like suddenly ramp up like overnight once we find a new workflow that we're going to use. I mean, the moment that I started like doing agent orchestration, you know, I immediately felt this like anxiety around my token usage limits. Like, you know, I didn't really like being in this place where I had to like temper my goals and expectations around what I could think, what I thought I could fit into like a five hour token session.

3:57Andrew:There was a number in here that I saw that there was an estimate that token costs could reach a level of 20 % of an employee's salary. I think just looking at the horizon that exists today, if the trajectory continues, I could see that being a reality that you actually would be consuming so many tokens that it would be a significant portion of the cost of hiring someone. So yeah, I don't know. What do you think, Andrew? Like, especially around this,

4:27Charlie Guo:like it being a part of compensation packages. As an engineer and someone who consumes a lot of tokens,

4:35Ben:actually, I think that this whole idea is totally unacceptable. And it flies in the face of what I think the idea that the inference is unlocking for companies is supposed to be doing. You know, we've talked extensively on the show about the value transfer of being a 10X, 100X engineer. and using all of that time and inference costs to create benefits that ultimately get captured by your employer. That's the structure in which we work inside. And so this is like telling somebody working in the dot-com boom that they're going to get paid in bandwidth. You know, if you want me to buy more tokens or want me to be more on the cutting edge of whatever technologies we're using, then make that infinitely available to me as part of my job.

5:19Ben:isn't part of the narrative that these teams are supposed to be getting smaller because one person's doing five, 20 persons people's work. So why wouldn't they have five or 20 persons people's budget? It doesn't make sense to me to tie it into a compensation package. I think it's extremely accessible to get access to a near infinite amount of tokens without breaking the bank. And frankly, a lot of engineers are already tapping into both sides of this with their work consumption and their personal consumption anyway. So the idea of trying to make it part of the compensation doesn't make sense. As much as it is, you should definitely roll it into the cost of the employee.

5:58Ben:But that's kind of the agentic workforce we're stepping into. Maybe that employee is two or three times more expensive or something. And a large part of that's going to their tokens. But you're getting 5x the output, 10x the output. And so for me, I think it's fascinating. It made me actually think of the song where he's singing like, oh, you load 16 tons and what do you get in another day older and deeper in debt? But it's like in this case, you just get what a co-pilot subscription? Like, no, it's unacceptable to me.

6:28Andrew:yeah yeah uh yeah you know i think that's a that's a good way to frame it like it really is something that you know i i often compared like in the early days a co-pilot subscription to like a grammar checking software subscription like we use grammarly for example and it's like we don't really like uh talk about whether or not it's something i should have like we don't really debate it you know it's just i i walk in the door at any job and it's something that i'm given because we recognize that it's something that makes me more productive to have access to that. So the story really just highlights just how in demand all of these services really are becoming at this point.

7:09Andrew:And not only that, but how much value we're extracting from them now when you figure out how to do agent orchestration. So yeah, it's an interesting conversation nonetheless. us for sure all right so moving into our next story so this story covers the emerging harness engineering playbook and i really like this article because i think it illustrates a lot of great examples of what we're describing of these teams that need high token costs or high token consumption because of the thing the types of things that they're doing uh so this this article comes from friend of show charlie gow he walks through a whole bunch of situations of of engineering teams using agentic AI to accomplish some pretty amazing things.

7:50Andrew:Like at OpenAI, there's a team that built a 1 million line product, internal product with over a few months with just three engineers. You know, Stripe is doing tons of agentic PRs that are being produced every single week. So Andrew, I'm curious what you think about this article after reading it.

8:08Ben:This article is a really great nuanced breakdown of the challenges that are facing engineers right now, because we've talked a lot on the show and everyone's heard this saying a lot of times of engineers have to become managers, right? And engineers need to think in the level of how their manager would typically assign and figure out what's going to have the highest impact of work, right? And that kind of transformation, how you think about your work is one journey. But then there's another journey happening here as well, is once you've decided what needs to happen and you're acting upon it, it's more than just telling your agents to do it, like you know exactly what to do.

8:44Ben:You also have to curate the right environment and give them the right tools and set the right expectations so that they can not only get to where they need to go eventually, but do it quickly, safely and on task. Because as we've talked about, like over 70 % of agentic coding time is typically spent refactoring past agents work. And so the idea of having proper alignment and guardrails from the very beginning is so important. And that's what this article dives into. The idea of harness engineering being that you have to manage the work environment just as much as you're managing the work that the agents are doing.

9:19Ben:And acknowledging that it's not a step one, step two process. They feed into each other. You switch between them and they enrich each other. And there's actually as much as there is to learn and reflect on from engineering managers and product leaders and how they think about and delegate technical work. There's also a lot to think about just in general group practices, leadership practices. If you've ever been in front of a group and doing a presentation, leading a team in any capacity, you know that you have to put the right tools in your people's hands over them to get the job done. And this is what's now happening on a micro level, not just on the engineer's level, but inside all of their individual projects.

9:58Ben:And this article dives into some of the nuances of that skill of building that harness. Yeah.

10:05Andrew:And specifically, some of the practices that stood out to me is, you know, first of all, like planning is the new coding. And, you know, I've been keeping this phrase in my head, be intentional. Like I keep thinking that at all times as I'm building stuff, because I found that's that's the way that I ensure that the agents that I'm operating are moving in the direction that I want them to move. there's also advice on documentation being your new system of record you know everything needs to be documented and stored and you know i think historical context we're learning is just as important as like knowledge context you know you need to build that the the historical reference of all the decisions that have been made why they were made and how it impacts things moving forward You need that chain of decisions.

10:52Andrew:But most of all, there's some, you know, the advice that I think we all need to just really, really keep at the center of this is just avoid the AI slop. Like you should have higher standards for AI generated code than you do for human generated code. And I think that's one of the missing nuances behind when you hear all these executives out there that are like, we have 70 % of our code being generated by AI. And they don't always mention the part where they also include substantially more testing for that code than they've ever required in the past. So it's just naturally generating far larger volumes of code than what they would otherwise.

11:29Andrew:So, yeah, there's a ton of great advice in this. And yeah, definitely encourage everyone listening to go read the article. Yeah. All right. Well, let's let's let's move on to a topic that we seem to be coming back to over and over here on Dev Interrupted. And I kind of feel like it's going to keep the story is going to keep going for a little while. This comes from Gary Marcus about these recent Amazon outages that have happened. These types of things probably are already happening all the time, but it really only makes the news when it's a company like AWS that it happens to. So yeah, Andrew, what's your take on this article?

12:05Charlie Guo:Yeah, I think that's an important distinction.

12:07Ben:The call out is that this kind of process and incident, in reality, this growing pain is happening everywhere. It's just when you're Amazon, it's inescapable when it affects you and your customers. all of that said i think that this is like a really interesting sign of like how much the growing pains of developing those new skills around understanding what needs to be done and then also creating the right environment for it to be done uh as folks build those dual skills you kind of get this awkward one step forward one step back you know lurching kind of motion and i think that's what you're seeing is like from this there's obviously going to be learnings that will produce new guardrails, better environments, more strict security, a deeper understanding also of how the tools that are getting put on your team are being utilized and adopted.

13:01Ben:I really think that's like a calling card here is like, if you have incidents like this that are either visible or you think are happening under the surface, it's really important to understand how that does tie to your organization's AI velocity and what tools you're using. And having a scope of, you know, if there's a problem that's happening because of like an agentic process, being able to implement guardrails to prevent it is something that's so achievable now in a way that actually was fundamentally harder when you were trying to implement these guardrails with humans, human engineers. And so I think there's a lot of interesting ground to explore in being able to understand like how AI adoption is tied around these kinds of incidents and then actually be able to create the best working environment for it moving forward.

13:50Ben:It's going to be an uncomfortable process because we're effectively reinventing the SDLC.

13:55Andrew:Yeah, yeah. I think just last week, I mentioned the, you know, the three steps forward, two steps back with AI, which is technology progression in general. And I've seen this as I've tried to adopt all the latest tooling. there's basic capabilities that you would expect to have or want to have that just aren't there yet because the tooling is still like very immature you know even down to just like giving basic permissions like allowing ai to to perform certain actions on certain parts of your your code or or your workspace versus other parts of that workspace the tooling to do that isn't isn't always in place without, you know, again, being very intentional about putting it in place in the first place.

14:38Andrew:So, yeah, there's a lot of just, I feel like fundamental capabilities that we're all relearning at this point. And it's even more risky when you're doing this in the hands of people who don't have a lot of experience. If you have, you know, more junior engineers or something like that, or just less structure around your engineering process, maybe it's just your your engineering team as a whole is less mature uh you know that those are the types of things

15:05Ben:that open up these these risks so at the end of our lineup here i want to end it on a fun one this is a blog post from george hot saying uh you know every minute you aren't running 69 agents you're falling behind but it's really just kidding it's a nod nod to the reality of the kind of toxic race that we all feel like we're running and sometimes the idea that if you're not running 100 agents right now, then you're falling behind or you're not changing the game or you're going to be replaced. You know, it's definitely a time of like very breakneck velocity, right? And at times it can feel like there's no relief from it because you have to keep running.

15:43Ben:And I think that this is like a nod to like acknowledging that reality that we're all running this relay, figuring out where it goes. But he also dives into a little bit about if you know, if you are running that race, if you are waking up every day and doing your best to figure out where it's all going to go, then you're already in such a great position. What did you think of this one, Ben?

16:02Andrew:You know, I, like you said, I think it's just something that is just some, the type of thing that we should always be thinking about just for our own mental health, for the health of our teams, for our companies. It is very tempting. Like it's intoxicating to work with AI and see how quickly it can help you move. And it's, it's really fun and really great, but it's also very taxing, there definitely is a lot of pressure to adopt, some of which is reasonable, but I think also a lot of which is often unreasonable. So I think it's really important that, you know, again, I'm going to keep saying this for, for, I don't know how many weeks, but for many more weeks ahead, I think, you know, be intentional.

16:40Andrew:Like if you're going to, to accelerate yourself with AI, make sure there's a clear intention behind it and you know exactly what you want to achieve with that and what to what to do with the outcomes from from what you're doing and i think that's the best way to just you know adopt a ai in a healthy way that that helps you move faster without like just scattering you to the wind with all these

17:03Charlie Guo:different priorities and i think a strategy for for doing that is being mindful about how you

17:09Ben:are building your skills and what you're working towards and ultimately what you should be thinking about is using these newfound abilities and multipliers to remove complexity from your job and your life, not to add it, because that's only going to make that breakneck pace more untenable and more unmanageable. But there's actually such a bigger opportunity to use that to pair away things that before we took for granted, that before we used as crutches, that were our own human harnesses in a world before AI and actually reinvent ourselves in a simpler way so that you can be more impactful in very specific ways.