Empathetic leadership for tech overlords, a good backlog completes itself, and who’s agent is this, anyways?

0:04Kelly Vaughn:So Kelly, have you heard the news? I'm sure it's impossible to avoid it, that Fable has come back into our world.

0:11Andrew Zigler:Fable is back in our lives again. Yes, it was a bit of a, I say it was a bit of whiplash, but like I didn't actually get a chance to use it that much before. And I haven't really gotten a lot of time to actually use it this time around. But I can confirm that Fable does recognize that there are three R's in strawberry.

0:30Kelly Vaughn:Wow, incredible. So you were able to get to the bottom of it with the latest model.

0:34Andrew Zigler:Yes, I cannot wait for like my analysis of like how I've used the different models. And it's like, you spend how much on this query?

0:42Kelly Vaughn:It's like the strawberry benchmark is really where you go to first. That's incredible. And, you know, I think it's funny that you comment on like having a chance to try it again. I remember when it came out and then it got taken away. Remember, I was in your DMs on LinkedIn and I said, I was like, Kelly, did you try Fable? And then you said no with a little sad emoji, I think. So sad. Yeah. Yeah. You're not alone. There were a lot of folks who I DMed on LinkedIn. I was like, did you try it? And a lot of people were like, no, like I didn't. It was actually staggering to see the number of folks who had like they didn't get a chance to.

1:17Kelly Vaughn:So to have it come back and they get to try it is really great. We do only get this one limited week of it being on our subscription tier pricing. So what actually I think this talks about more than ever, and we'll dive into in a moment, is the importance of choosing the right model for the task. Because I think Fable is going to be one of the first of many of these flagship models getting really, really expensive for highly specialized stuff. So what do we need to be learning from it? And that's what we're going to be talking about today. And to those listening, welcome to the Friday Deploy brought to you by Linear B.

1:48Kelly Vaughn:I'm your host, Andrew Ziegler. And Ben is out this week enjoying the great outdoors, whatever that is. But don't worry, because it does feel nice to go and touch grass. But, you know, here on Dev Interrupted, we touch keyboards and we chat with friends. And so don't worry. We have a past guest and, in fact, your favorite engineering manager's favorite engineering manager returning to the show, Kelly Vaughn. Kelly, it's amazing, as always, to have you here. I'm so excited to run through this week's news with you.

2:16Andrew Zigler:I am so excited to be back again. You know, I love doing this. I love hanging out with you and I love talking whatever tech news is on the plate for the week.

2:24Kelly Vaughn:Exactly. And it's good to gossip about it out here in the open instead of in our DMs on LinkedIn. So let's get to the bottom of it. Because this week there's been a lot of interesting stuff going on. Obviously, Anthropics control over a fable and releasing it to the market has changed. And we're going to talk about what that means. We're also going to talk about things around agent identity and how you should be thinking about that in combinations with your workflows. We're going to be diving into a data science benchmark from past guest Brian Bischoff. You might remember the data science America's Next Top Modeler that I participated in back in October.

2:54Kelly Vaughn:We got a post retro on that that's really fascinating. We're going to learn some stuff. And we're going to talk a little more on the interpersonal side because, as we all know, creating software is a deeply communicative sport, right? It requires a lot of communication, team building, collaboration, and understanding the perspectives of others and how you're working together. So we're going to be talking about why arguing isn't necessarily the best way to get about proving your point and avoiding some villain bond leadership based on some advice from our favorite charity majors as well. So, Kelly, I'm really excited to dive into some of the things today.

3:31Kelly Vaughn:But, you know, I just wanted to start by revisiting just for a moment the fable getting brought back out into the world. Because, one, I think this represents a really interesting change in how models are going to be released moving forward. I think you're going to see more interference from organizations and entities in the models coming to market, at least in the West, which is starting to have more restrictions around the capabilities and concerns around things like internal cybersecurity and stuff. But also, too, on a practitioner side, having the model come out and then I tried it. You know, we talked about this extensively in our new segment.

4:07Kelly Vaughn:I use like all of my tokens, writing the dragon, the fable, basically. And I couldn't stop talking about how helpful it was. But the fable has come back. It actually has a lot of model routing capabilities underneath. It'll actually send things like coding requests, in many cases, off to an opus model underneath. It'll use things like Sonnet, the new Sonnet 5 that came out along with Fable 5 to do a lot of things like writing. So it kind of begs the question of like, what is Fable? And what is Fable doing? And I think it has to do with like this organizational intelligence of how work gets done.

4:40Kelly Vaughn:And it represents the new kinds of models that we might have moving forward. And that these models might get really expensive. But, you know, Kelly, what were your thoughts on this thing coming back to market?

4:49Andrew Zigler:Yeah, I think we're going to be seeing a lot more of this. I think the big splash of this model is not going to be something we continue to see in the future. I think we're going to see even more interference. And it's a weird time to be in this phase. I would say it is probably predictable that we would have ended up here to begin with because we fear things that we don't understand as humans. And we understand that we don't really understand what we're doing here yet. This is a very fun exploratory period, but as these models get smarter, we don't know what we don't know yet. So I do understand that, you know, I don't love the national security piece to only release it to certain very special customers that aren't Zapier, unfortunately.

5:41Andrew Zigler:Otherwise, I would love to be in on early. But I intend to see more and more of this. But one thing I do kind of want to touch back on is the choice of model. The model routing, I think, is actually a really smart thing to do here because I think we also have a tendency of overestimating how difficult something might be. And so we're going to immediately default to like, oh, I need to use the smartest model for this, the most expensive model for this. And if I'm foreshadowing for anything, I apologize in advance here. But I think it's important that, you know, we all want to kind of play with the new stuff.

6:16Andrew Zigler:But if we default to using whatever's newest, we're going to run out of money really fast.

6:22Kelly Vaughn:Oh, yeah. I think that's really great advice. The whole idea of being aware of what model you pick up and which ones you're using for what task is only going to become more critical as these models, I think, become more specialized. Up until now, we've been seeing these like step level changes and foundation level capabilities. but and you're still going to see those you know at an obviously slower rate moving forward and kelly i think you're really smart to call out that you're probably not going to see the big splash of releases anymore because now i think the name of the game is going to be how can i get the intelligence out the door day by day how do i how do i uh i guess like you know continuously deliver the intelligence and not ship these big models that way you would sidestep this whole oh, now there's a curtain falling on security and interference.

7:07Kelly Vaughn:And I think that might be a strategy we see where you just now see a latent increase in capabilities. But on top of that, I think you're also going to see them split into highly specialized domain-specific models where you have that really smart flagship model or you have a open-source way alternative. There's a lot of them now that are open-source. Apache 2 that you can build a business on top of. And you take them and you fine-tune them on your domain expertise on synthetic and real data from something you're infinitely like knowledgeable about and you build a highly specialized intelligence for handling things in that domain, that becomes like the new frontier.

7:44Kelly Vaughn:Because if you trained that, then it's like, yes, sure, over here, maybe the step level capabilities keep going up, but none of them will ever be as good at that specific domain task as this one that was trained off of this one. And then if we ever want to move it along with it, then as open source gets better, we train along with it because the data set is modular. So, like, these become the new strategies I think people turn to. I think Fable is going to be an event that causes folks to really wake up to that reality. Speaking of waking up to the reality, our producer, Adam, who's not here today, he was on the ground this week at AI Engineers World Fair, recording some really cool stuff, capturing some things on the scene, but also hearing from Human Layer co-founder Dex Horthy, who's also a past guest on Dev Interrupted.

8:30Kelly Vaughn:And Dex, you know, he made a lot of news and folks were talking about his presentation about how, you know, teams are over-relying on poorly trained models and they're having a fast throughput of code that maybe goes through some facsimile of a review and delivery process. But then, you know, days or weeks or months later when there's incidents, when there's outages, when there's critical flaws in the software, being unable to turn back to the work that was done and understand the legacy of how you got there. What were the decisions? What were the artifacts? What was the code and the tests? And, you know, that level of auditability, observability is different for everybody.

9:07Kelly Vaughn:But, you know, he's really calling out here the importance of putting the human back in that loop of being cognitively aware of what is happening at stages of this process. That way it doesn't take months for these AI-generated flaws to surface in your product and what you're delivering to customers. And really, he was attacking to the idea that, oh, just optimizing your harness is everything, that just getting the loop is everything. You know, getting the loop and getting really good skills is one element of being a really effective agentic worker. But sharing context and aligning with others and their agents is actually the differentiator for teams that are delivering durable software.

9:45Andrew Zigler:Yeah, I love this topic because it's something that I firmly believe as well. I am a strong proponent of multi-threaded agentic engineering. I think there's a lot more that we can do at the same time, but that doesn't mean that every task is one size fits all. I have seen product teams ship a whole new product in, let's say, four weeks, four months, whatever. And what they're capable of doing now is amazing. But I've also seen what happens when those incidents arise. And we're like, okay, we're now using AI to diagnose a problem that AI wrote and AI reviewed and AI shipped. We are the humans in this conversation.

10:26Andrew Zigler:And there's a reason why I'm also a big fan of making sure engineers stay employed. Engineers serve a very important role in engineering. We are the ones who do hold the context and nuance and really being able to understand the systems that we're building is what keeps us in engineering. And so I love this whole take. And I, of course, was not at the presentation, but I'm bummed that I did not get to see it in real time.

10:58Kelly Vaughn:I know. I'm excited to dig into more. Everything from Dex, I think he's really spot on with how folks should be thinking about their engineering practice. So more to come always from Dex. I recommend folks follow him on LinkedIn if you're not already. And our next topic is getting a little more granular around some of the problems of having agents roaming around where all your code is in the first place. And this was a really interesting article that dove into different ways that teams are giving their agents identity. The idea of the identity being borrowed from the human user or being something distinct and token-like to the workflow or being something cohesive and durable cross-sessions like an agent identity.

11:39Kelly Vaughn:These are all different ways that folks are tackling the idea of how do I understand who did what and under whose authority and with whose permissions, which I think is one of the biggest dangers right now lurking with AI inside of large enterprise organizations, especially those that are still figuring out how do we share these durable AI practices across the team. And also, too, what I love about this article, we'll share it because it gets technical in terms of the level of ways that you'd go about instrumenting and actually applying these three different things. But critically, at the end, it does a really great job at comparing when would you pick up one over the other?

12:17Kelly Vaughn:We know these three exist. Why do some orgs fall into bucket three where they have to build a very complex agent identity system where everything has an ID that goes back to an agent? is like, for me, it makes me think of like, oh, yes, because like huge, large, sprawling enterprises, there were times where they all had glowing eyes for building microservices or like, let's just be on Kubernetes. And it's like, why are we on Kubernetes? We have 500 users. So it's like, and so like these kinds of, that's what it makes me think of when I look at the agent identity thing of, it's like, is this an over-engineering?

12:51Kelly Vaughn:Is this wanting to hit, is this, you know, wanting to hit the hammer with an arrow or whatever the metaphor is, I feel like that's kind of what it starts to feel like. You know, Kelly, what do you think about how, even within your own team, you have a lot of folks and work that happens underneath you and the permissions and scopes that they have are all varied. How do you think about it as an engineering manager?

13:12Andrew Zigler:Yeah, I mean, the governance piece of this is critical. And, you know, we've, at Zapier, this has always been a conversation. You think about automation's running. Automations have applications, like integrations connected to it. Who owns that ZAP versus who owns that integration? It could be different. And the governance piece is always a question that has to be answered. And so I'm just seeing the same thing in a different flavor. When I'm thinking about attribution for who is working on what and who is responsible for this, if everything goes through an agent's ID and there's no actual human tied to that, that becomes, you know, worrisome for me because I don't know who to talk to.

13:56Andrew Zigler:And like my, like the other EMs that I work with are seeing the same thing. You know, given these three, like, as you were saying, the possibility of like over-engineering this, I completely agree. One of the things that kind of stuck out for me is this or with this was that it very much feels like it should be like a step letter in a way or like you should you should work through the rungs and when you start feeling pains explore the next option as opposed to immediately being like well you know one day we're going to you know need this this massive massive thing so we're going to go ahead and build it now good luck like having to support that in the near term like we all have a tendency of over engineering i know it's exciting to think through like the the wide the broader widespread like agent identities but like it's okay to start small yes small yeah

14:42Kelly Vaughn:Yeah, you should scope it. And by scoping it, they say, you know, step one, start by borrowing the user's credentials and their permissions and be scoped about it and have auditing and logs, which you should already have for every agent to call anyways. And so, yeah. And so it's like we're just talking about baseline instrumentation to achieve the identity. And then the rest of this becomes things that we're already familiar with, like API tokens, which at this point, everyone just assumes are compromised the moment that they're generated. and so really it's like when you dive into this article you'll see this uh thing about like spiffy inspire and like all these different systems for setting it up and i will i will uh say that like my engineering brain was like oh fascinating problem and i could definitely hear the cogs turning but that was the danger for me is i was like i could spend a long time trying to build this answer um and i don't think anyone has the real answer yet but the doing treating it as rungs in the latter.

15:36Kelly Vaughn:It's a really smart strategy. So you don't overextend, I think, in the meantime.

15:41Andrew Zigler:Yeah, totally. Totally.

15:42Kelly Vaughn:So our next story is a recap from a hackathon that happened last year. This is America's Next Top Modeler. Yes, it's an incredible name. This is hosted by Brian Bischoff. He's the head of AI at Theory Ventures and friend of the show. Most recently was with him at AI Council where I presented. He was also a guest on the show. And he had this hackathon, America's Next Top Modeler. And he really was kind of like this crazed villain running this hackathon. And for those of you who've been listening since last year, you might remember the premise was that he brought 150 engineers together and one real analyst at like an analyst company to be given like a huge dump of enterprise data that was made up for a shipping company.

16:25Kelly Vaughn:Had all these products. It was like SQL tables and system logs and like literally like 800 ,000 PDFs. There was even a physical dusty binder that represented a binder sitting in a warehouse somewhere for the shipping company. Like he went through all means of being like information is messy and it lives in a bunch of places. And it's not wasn't ever created with the intention of an AI built using and doing stuff with it. He challenged us to as, you know, nascent data scientists, AI engineers to build a system or a practice that would be able to answer this system, this set of eval questions. So we got like an eval system.

16:59Kelly Vaughn:And really what this article is, is it's a benchmarking and diving deep on how that process unfolded from the person who designed it. But I personally found this very fascinating because I spent six hours maddeningly turning my brain over in cursor trying to get the right results for the different eval questions. And there were folks that were doing all sorts of different brute force methods of trying to actually get the right answer, including someone who was running like 30 Claude Code sessions in parallel. And at one point, his computer froze. And so it was just everyone was doing something.

17:34Kelly Vaughn:And it was really, really, really fascinating. So this article dives in on some of the things that stand out for data scientists that are pitfalls in AI. Like data science is often a matter of sequencing together operations and changes and data to get your final transformation that you're looking for. And each of those are very susceptible to having the wrong parameter or the wrong kind of keyword and suddenly your end result is wrong. And so it can become much messier and harder for the agent to backstep and understand it. So they even created this really cool grid where commonalities of agents and their approaches would break down when trying to do these joins and sequences.

18:11Kelly Vaughn:Really interesting data deep dive for anyone who has tried to throw an agent at data. And I'm wondering, Kelly, is that you? Have you ever tried to use an agent to roll over lots of data?

18:21Andrew Zigler:To some degree. I mean, with the amount of data that we have in Databricks, for sure, I've tried to parse through a lot and try to create a story out of that with medium success. Mostly because, and I think this is important for like one of the takeaways from this is like, what big goal are you actually translating into something to be really precise and verifiable and step by step? And that is an interesting takeaway for like the sake of this study, like this hackathon. But this is a takeaway that should apply across the board. You know, it's the same thing like throwing a very vague like Jira ticket at an agent and saying, do the thing.

19:02Andrew Zigler:If it's vague. Wish casting. Exactly. Yes. And so I think this is an interesting takeaway. Like this, I'm glad you participated in this. amazing name for sure um i think the takeaways are just they're so uh applicable across all different facets of engineering beyond just eie bells yeah i even think of the of the andrew who

19:26Kelly Vaughn:was sitting in the chair in october doing that hackathon and what his engineering practice looked like and i don't even recognize that person so it also speaks to how fast because i'm like oh wow if i did i read i'm reading this article and i'm like oh wow if i could do this again today it'd be a piece of cake. And I know I'm still wrong, but it's just crazy how much that cognitive stuff has grown since then. Andrew and I, we just wrapped up this really great session on token maxing. And we had a lot of fun, you know, running through it. We got a lot of feedback, a lot of questions from the audience.