Your engineers need an AI control plane, not more tools | Guild.ai’s James Everingham

0:04James Everingham:Hello to our listeners and welcome back to Dev Interrupted. Right now, a lot of engineering leaders are stuck in the same loop. They roll out AI tools and they see promising demos, but when they go to measure productivity, the numbers get fuzzy and the organization quietly drifts back into business as usual. Sound familiar? James Everingham, however, has lived the opposite story. After nearly a decade at Meta, including leading Instagram engineering, he was pulled back to help run DevInfra, a thousand-person org responsible for the internal developer experience of 40 ,000 engineers and technologists.

0:43James Everingham:There he discovered that the real gains come from when you stop thinking about AI as an authoring tool and start treating it as sentient fabric across your entire SDLC. And that's what we're going to talk about today because internally that work became DevMate, an agent platform that went viral inside of Meta and grew to the point where its agents were submitting 50 % of all diffs. And now James is taking those lessons to the rest of the world as the CEO of Guild.ai, where he is building enterprise infrastructure for AI agents so that every engineering org, not just Metascale companies, can centralize and orchestrate and safely scale their agent workflows.

1:24James Everingham:This is something we've been talking about quite a bit on Dev Interrupted, so we're really excited to dig in. And James, welcome to Dev Interrupted. Thank you so much for having me on. I'm excited to be here today. Great. Well, let's go ahead and jump into talking about that mental transition, the one I kind of opened with here, about moving from the idea of AI being an editor, a helper, to being part of the fabric that you work within. You know, when you were leading Dev Infra, you realized pretty quickly that like, you know, autocomplete and just finishing your idea of what to implement is not where the biggest wins are.

1:58James Everingham:You went for these larger leverage opportunities within the infrastructure beyond the editor. What was that realization like and what led you to go there?

2:07Andrew:Yeah, sure. I think, you know, when we first started looking, you know, we were on the same path as everyone was trying to accelerate developer productivity using AI. We started with the same tactics that we saw, such as, I would say, the in-authoring experiences. You see the cursors and the co-pilots out there, which are auto-complete. These have evolved to much more than that, but that's where we all started. We had an advantage there where we owned all of the internal tooling. We had to build our own editors. We had to build our own source control because the repository was so large. It wasn't that we were an NIH company.

2:51Andrew:It's just that infrastructure so vast and large, these tools just wouldn't work. So the advantage of that, though, was that we got to measure and experiment quickly with a very closed economy of developers. So we could learn pretty quickly. We did learn that, like, hey, these tools like Cursor, which we basically came pretty close to feature parity with internally with our internal product called Code Compose. you would get a certain level of productivity more with the junior engineers and actually, surprisingly, some of the very senior engineers. But the bulk of the engineers in the middle weren't actually, it wasn't moving anything meaningful.

3:28Andrew:And that's a whole conversation itself is like, what is moving something meaningful actually mean, which we put a lot of thought into as well. So we experimented and eventually we started trying some different things. And one thing that we did, we had this realization that, hey, you know, AI is a very powerful tool, but if you actually move some of this new agentic behavior closer to your source control system as part of the infrastructure, you get a lot more impact and you can do a lot more interesting things. You know, source control is like the canonical source of truth, like your teams, your tools, everything connect to it.

4:07Andrew:There's your code, your history. And if you can intercept some of those tool connections and invoke agents and you could start doing some super interesting things.

4:19James Everingham:You mentioned this opportunity you have that because you owned all these different parts of the stack and you had everything custom built. you get these really amazing opportunities to slice and dice and examine. And that's like a privilege and an ability that like not a lot of orgs have to like the level of granularity that they get. And so I think this oftentimes muddles the process of trying to experiment and measure the impact. Like what would you say to somebody who is working within an org and doesn't have that level of detail in slicing their whole top to bottom system? Like where should they start to try to zoom in first and look at what they can get their hands on?

5:01Andrew:So that's a good question because like it's, it's, I think that's one of the things that the industry is also struggling with right now is like, where do you even start? Like, you know, a lot of the companies that we talk to now outside of Meta, that's the first thing that we come across is like, Hey, you know, we're all getting pressure to, to do more, to develop faster and things like that. But it's not even clear where to start. So what we found, the formula that we found inside Meta that really helped kick things off was that we had a centralized place where when I talked about this platform, which was DevMate that connected to the infra more, it was also a centralized place where engineers could go and actually see what agents were there, what was working, what wasn't, and they could build on them.

5:51Andrew:They could view them. They could dig in. They could branch them. They could create their own space on those. And that was a great starting place. You know, as engineers, you know, speaking for myself, I was a better, like, thief than coder. Like, I would go look for something that would do kind of what I want. And then, like, I would try to get that and wrangle it into something that would work for my specific thing. And I would learn along the way. And if it didn't ultimately work, I'd have enough knowledge to go build something new. And I think that's not, I don't think I'm that unique. I think a lot of engineers work that way.

6:26Andrew:And so when we had this centralized surface, it was a clear starting place. So, you know, for companies that are outside meta is like one, you know, where do you go? Where do you go find this clear starting place? You know, where do you go and point at things that are working outside? And, you know, we're getting better. It's like becoming more clear what works in the industry now. you're seeing a lot of more uh referenceable uh pieces of technology they're actually doing real things so i think this will become less of a problem over time but right now you know it's it's not clear where to start and it's not even clear what you know the holy grail of metrics is developer productivity like you know right um is it lines of code i don't think it is you know some of the most productive developers delete more code than they create, right?

7:18So is it feature

7:21Andrew:velocity? You know, what is productivity in your company? I think another place to start is figuring out and getting pretty detailed on what that actually means in your company.

7:32James Everingham:You really hit on some really amazing details here because focusing in on the first idea that like you have to understand what matters to your company and your organization before you even go out and hunt these things. And the idea that also of, you know, always being a better like beef than like a original creator, I totally like that resonates with any kind of engineer, we don't wake up and we're like, Oh, I want to rewrite grip today, like we go and find the tools that help us get the outcomes that we're looking for with software. So, you know, I'm always looking to leverage and learn from other people that are playing within the same space.

8:07James Everingham:So what y 'all did was really smart, you created this, like a corral, basically, where all these agents and these workflows and these experiences can go. Folks can go and learn and experiment, fork and then standardize. That's kind of like you get to that core of that like idea of a fabric where the agentic work is happening in your infra. So, you know, as part of that too, I bet that you got a really great glimpse into emergent kind of solutions and agentic things that you probably definitely didn't have on your radar or didn't think about. But because an engineer within your org was empowered and had this place to share, that stuff got elevated.

8:45James Everingham:Like, what were some of those that stood out to you?

8:47Andrew:Yeah, you know, you hit a very important point is like, you know, one of the things that was interesting in DevInfra is like, we would get stuck in thinking like, what are the best things that we can build for developers? And what we found when we pushed the tools to the developers themselves to solve their own problems, the best things were coming from the teams outside of DevInfra. And, you know, so I think like the measure of a successful initiative or a successful platform is when the ideas are coming from your customers and, you know, and they're able to build on it. So a few examples of that were one, we had something called a risk score, and it was using an LLM in a rather novel way.

9:32Andrew:and this was open sourced actually last year, where it would measure the amount of risk that a diff would have when you push it to production to crashing the system. And of course, like many companies, we freeze at Meta, we would freeze the code base like December 9th through January during the holiday because that is where the most traffic is a very important time for the company. This allowed us to start removing that code freeze. And the way that these, This happened wasn't by saying, go build diff risk score and do this. It was a challenge to the company saying, how do we eliminate code freeze?

10:11Andrew:And then the engineers came and built these tools to start doing it. So that's one example. Other ones that just sort of organically popped up that I thought were amazing was this incredible onboarding agent that an engineer built. And, you know, once again, you know, I have a friend that I keep re-quoting saying, you know, building great software is an act of empathy. Um, and this was no exception to that. It was, it was an engineer that was trying to familiarize himself with a code base to be productive. And he built an agent to go help him do that. This agent, I'll explain it a little bit. Like back at Instagram, which you mentioned, if you brought a new engineer in and you said, Hey, I want you to work on these specific camera filters.

10:54Andrew:That engineer might take a week just to figure out where those files even are. and it's a complicated code base and process. The onboarding agent sort of made the source control base sentient in a way. So you could ask it questions. You could say, I went to work on these specific camera filters. And it would say, well, that's these files. Would you like a system diagram? And it would draw you a system diagram. Would you like to know about the history of this and why things are done? And you could have a conversation with the source base And this onboarding agent would effectively make the source base onboard the engineers directly.

11:36Andrew:And we had measured something called TTFD or time to first diff. And that was like, how long does it take to get an engineer in and get them to submitting their first diff? And this agent alone was decreasing that pretty significantly. And at scale, that's a lot of manures reclaimed in a large workforce. So those were just a couple. Hundreds of these popped up, things that I had no idea, like would never have imagined. It was really fun to watch.

12:05James Everingham:that onboarding agent is a really powerful unlock because what that represents actually is like a huge amount of downstream leverage that you were able to unlock as the person implementing that kind of infrastructure by creating a space where other people could create infrastructure and save themselves time someone else then created a tool a system that then saved everybody a ton of time and that was a leverage beyond your initial leverage and that's what happens when you create these spaces where folks can solve the hardest problems within your engineering work. And I think like something like time to first diff is really critical for huge code bases.

12:43James Everingham:And I know a lot of our listeners who work on a huge code bases really identify with that pain, especially when it comes to bringing in folks who are trying to have instant impact. I think that like in this world, you're describing this as the ideal, right? Where, like you said, your customers, your stakeholders are coming to you with things they want to do and they're building it on your platform. But, you know, sometimes that script is not necessarily what is happening or unfolding within an org. Sometimes you get more top-down mandates and things from leaders within an organization about how AI will be used or what our policy around it is, or a top-down goal that maybe doesn't really tie that well to like technical throughput for engineers.

13:25James Everingham:So, you know, what do you think about that approach? How would you advise somebody navigating that as an engineering leader to try to really unlock value?

13:35Andrew:Yeah, sure. I, you know, I think that I'll make, you know, I've, I've been not quiet about this. I do get a little, a little frustrated when I see CEOs and leaders saying, mandating their teams use AI for productivity. I think the best developer tools, like you have to earn the usage. Now, like developers are smart. They're, you know, they want to do good work, but they're also, you know, they have a set of tools that they've become very proficient with over time and they take some convincing. Right now, the way to do that, in my opinion, isn't to mandate like, hey, you all need to use these specific tools.

14:15Andrew:I think what we ended up doing that really helped was to put business challenges out, one, to your organization. That forces them to think differently. Like, for example, another one that we put out there in DevInfra is, can you make self-healing fabric? Can you make it so that if the system does crash, you can identify it automatically? You can write the code to fix it. You can write a test case to prove that it fixes it. You can run it through that loop until it finds it, land it responsibly, and minimize the impact radius of that outage. When you start putting these types of challenges out, like there's not many other tools that you can use.

14:59Andrew:You're sort of forced to use AI to do that. So that's just one part of the equation. Now, AI is very good at doing simple things. It's a little more challenging to get it to do very complex algorithmic things, right? So getting a product team is a lot easier to prove utility with these tools than the bowels of a VM or languages or a kernel team. Those are very hard things to do. But you can still put challenges out there for those types of teams. Can you optimize compiler output with an LLM? Can you change the spec of a language to actually make more, an LLM can more accurately build the right code?

15:46Andrew:Things like that. Once you get that, the next part of it is to get some senior engineers who have had success with it, and they should be your evangelists.

16:00Andrew:Luckily at Metal, we had these amazing internal workplaces where everyone could communicate openly to the company. And we had senior engineers that were like, look what we just did. We did this with this agent. And the other senior engineers then would be able to see that and they would want to actually try something or it would spur something where they're thinking. And that organic earned usage is what really ended up moving the dial, right? Like they're going to use the tools if they can help them. But, you know, the early tools, they're getting better, don't help every engineer. Some of them slow them down.

16:38Andrew:And so the mandates are a little misguided, if you ask. Yeah.

16:44James Everingham:And that's really good advice for how to level set it because you take that top-down kind of mandate and you find a way to turn it into a collaborative space where the organic gains can surface. Like if you have a top-down mandate, okay, then use that to establish a recurring learning and sharing meeting and then use that to incubate these best ideas that maybe in an ideal world, like with the one you were describing at Meta, like you would, it would be evolving and happening in like this share kind of agentic mesh or like a corral of different agents doing different stuff, you know? so within that too i think once you get to that point and maybe you're in a position of having to prove a value back in like a top down kind of way once you have that kind of muscle built then you can start exploring things like um like you said like getting rid of your code freeze is a great example i'm sure there's like a lot of bandage or like back of the room back of the office kind issues that like any team could think up and tackle.

17:45James Everingham:Like, are there any other like big ones that come to mind for you that you'd recommend people like every org has this debt, go tackle this debt with AI and prove your worth?

17:54Andrew:Yeah, I think that look, testing is one that everyone does is test cases. And, you know, you can have 100 % test coverage code wise, but humans actually will end up using the products in different ways where different code paths get exercised. And you can never find all of the problems. But one challenge we put out there is, can you build a testing harness that would simulate how a human would actually use the software? And that's something really good that an LLM can do. It can mimic behavior. So instead of saying, go and automated test suite, test this image, test this filter, test this, you can basically give it a script and say, you're a user, you're walking down the street, you take photos of this, you wonder where there's restaurants.

18:44Andrew:So you look those up and you can describe a user scenario and then, you know, have an LLM go replay that into your software. And so like that was another example. But we had a lot of big challenges like eliminate code freeze, self-healing fabric. Can we expand who contributes to the feature set beyond engineers? Can we go directly to product, to designers, to everyone in the company, people outside the company? What are these big challenges? And it's fun. Here's the great opportunity to do with your org that we really just... we might've gone overboard at some point with it, but look, you know, we get to write science fiction stories about like how development that's going to work in the future and like, get your team to do this.

19:32Andrew:Go like, go write a science fiction story about like what development is going to be like in five years and go make that happen. Um, and that's where a lot of these ideas came from, you know, it was people just dreaming and, uh, thinking about like where their pain is specifically and coming up with these novel ideas.

19:50James Everingham:I think that's a fun way to reframe it. It's definitely how I go into building things these days. You go in with all this baggage and expectations about how an app is supposed to be built, how it's supposed to be delivered or onboarded, what the experience is like. And we do. We get to live in this, like you described, like a science fiction world where you can throw that away and you can rewrite what it can be. None of us have the language or the knowledge to know what these kinds of tools and apps will evolve into. Everyone has their take on what's going to happen to AppSolentic in the world of AI, right?

20:18James Everingham:And so like in this world, like what you said is you're going out to solve a pain point, a problem. Like we become engineers because we want to have like a lot of impact. We want to create technology that changes the world for the better. And I loved what you said earlier about this quote that you say about building great software is an act of empathy. Like I really relate with that. I think it speaks to the core of what we all do as engineers about using technology to address a problem and helping other people. Putting yourself in other people's shoes is like really fundamental part of engineering.

20:54James Everingham:And I think that like AI, even in this case, can be a huge leverage that engineers can use to unlock.

21:00Andrew:Yeah, absolutely. I mean, that is one of the things we've always tried to do, or I've always tried to do in my companies and organizations, is put the software between, you know, make sure my team is somehow using the software. Even if it's not relevant to them, figure out a way to force them into the software. If they're not using it every day, it's just not going to get better as fast as if they were. So like, always find a way to get your developers to feel the pain that your customers will have.