How to turn your 1000x engineer into a 10x everyone | LinkedIn’s Karthik Ramgopal

0:04Joining me today is Karthik Ramgopal, Distinguished Engineer at LinkedIn. And Karthik, LinkedIn is obviously becoming the, or has been, the primary platform for the professional world to understand how the world around us is changing. It's a place where we can share ideas and collaborate. And behind the scenes is an engine that keeps it all humming. It's defining even the next generation of professional tools and professional people and what they expect out of the networks and communities that we build at work. And at DevInterrupted, we've been talking a lot on the show about an effect called the agentic halo.

0:45This is a transformative effect where an AI-powered leader or individual uplevels and upskills those around them. Instead of being a singular 1 ,000x engineer, they turn everyone around them into 100x engineers. And those types of leaders are really redefining the future of software. And I know that you've been at the center of this at LinkedIn and their agentic platform. And so today we're going to dive into how you've been exploring these concepts in a massive production environment and what it's been for you to be like to scale that effect beyond just raw compute. Also addressing it with other parts of the AI stack, including the people part, including the technology parts that sometimes get neglected when all of the other buzzwords are flying around.

1:37So as you can tell, Karthik, there's so much spinning through my head, ready to talk with you about. And it's so great to have you here on Dev Interrupted. Well, I would say that, you know, like, thanks for the opportunity. And let's jump right in. I'm excited to talk to you, too, about all of this. Great. Well, let's start at the top. How about we learn a little bit about what has been happening at LinkedIn with its agentic platform for its engineers that are building and servicing and developing the tool that the platform that we all use? When you've been shifting into not only being an agentic engineer and having agentic engineers around you, but also now providing AI features through the LinkedIn service to your customers and users, what have been some of the fundamental architectural shifts that you've made in your day-to-day?

2:25Okay. So I think the word agent platform is overloaded. We effectively have, I would say, two incarnations of it. A lot of technology is shared, but the usage is obviously different. The first is the agent platform for internal productivity for engineers, but not just for engineers, for a variety of other job functions as well. Product, design, legal, marketing, business operations, a bunch of people use this stuff, internal productivity, right? AI is pretty universal in that way. It's like a rising tide which lifts all the boats up. It's the same way for AI. It's not just engineers, it's everyone else.

3:03So there's that. And then you have the agent platform, which we use as the framework for building a bunch of our production agents, either on the consumer side or on the enterprise side to serve our members and customers. Again, as I said before, a lot of the technology is shared. some of the core building blocks here are stuff around basic stuff, right? Prompt management. How do you do inference? How do you abstract away common orchestration operations when like building agents? How do you do context management? Again, we started off very simple. We had chain-based systems. We have graph-based systems and we have more harnesses right now.

3:45But one of the things we have realized through this entire journey is that a very important thing for an AI system, which is very different from a traditional software-based system, is agency and personalization. And personalization, especially over the long term, requires investment into a form of memory, right? So that it remembers, it gets sort of like better over time and truly understands you. So we've also been making a lot of investments into our cognitive memories stack. Right. Right. So I love how you frame the problem, the opportunity even for teams that are working with these tools that it's not something that's locked within an engineering world.

4:26What it does is that it allows you to bring deeper action, insights, and leverage to all kinds of roles within your organization. And in doing so, bring them closer to the problem space that y 'all are all working in. It's not just the engineer's responsibility to be customer obsessed and be really close to the conversations that their customers understand. You know, we try so hard as organizations to bridge those two very often like very gapped kind of parts of the org. But it's also like everyone else's responsibility to understand how they can have a role in this. What are the new skills they need to develop?

5:04How can they share their best practices with folks around them? So you had to, you saw this as an opportunity of if I can create this internal ecosystem where people can come together and share, then I trust that the coming together and sharing will happen. And then it did. It was kind of like, you know, if you build it, they will come kind of deal. And how did you assess that problem and even start? Because I know a lot of leaders, they maybe see, they get hungry for like, I wish I had that ecosystem pin, that innovation thing. How did you start to tackle that problem at an org so big? So I think that the first thing to understand is in order to reap the benefits out of AI, everybody needs to be able to use it to a reasonable level.

5:47Otherwise, you end up creating bottlenecks. It's almost like you have an acceleration in one place because you have built a freeway with N lanes, right? And after that, the freeway suddenly ends and you get onto a dirt road and everybody slows down. The freeway doesn't help you, right? Same thing. If you just accelerate the engineering function, and even within engineering, a lot of people make the mistake of acceleration being limited only to coding it's way more than coding it's how you write your specs including design specs how do you write your docs like engineering docs design docs collaboration docs whatever it's also stuff around operations how do you deal with alerts how do you deal with deployments how do you deal with production issues even just looking at engineers you also have product management you have design you may have questions for legal, right?

6:36So everybody needs to experience the acceleration so that you can actually see a business acceleration. Otherwise, if you only accelerate the engineers, everybody else slows down or is operating at the same pace. It results in frustration as opposed to acceleration. So you need to start with that mindset of, hey, how do we accelerate everyone? The second thing is that there is a lot of hype, right? A lot of different tools are coming in every day. It's like a tool A, tool B, tool C, etc. I think the real change required though is a an attitude change and a mindset change. It is sort of, you know, like less about the tools and it's more about how you use these tools to accelerate your productivity.

7:19A very important thing to note here is that whatever tool you buy, most likely it's not going to know about the internals of your organization, right? Every organization has a bunch of proprietary stuff. Sometimes it's proprietary data, proprietary business practices, how people do things, et cetera. Sometimes it's also proprietary tools and workflows, which people use, which require more of software integrations. At the end of the day, this boils down into a giant context management problem. So we at LinkedIn have invested a lot in recognizing that even as the tools advance and they will advance, we will continue to get limited by the context we provide them.

8:02Unless you provide them context, it's just useless, right? So unequivalent is you hire the smartest engineer on the planet, let's say, and if they aren't able to access your systems, useless, right? You aren't harnessing them. It's in the same way for AI. And as I said, it's pretty universal for other job functions. Then the question comes, hey, you do invest in this context curation, context sort of like management experience. But how do you do it in a way such that as the tools advance, you don't have to keep reinventing this, which is why we've invested a lot in exposing this stuff via open standards like MCP, the model context protocol, for example.

8:42So that, hey, as tools evolve and as they start supporting MCP, you just sort of get this for free. We've also seen other evolutions. For example, a lot of folks are doing CLIs right now. There's been this entire thing around progressive disclosure and the skill.md standard. So as these standards evolve, we are also evolving alongside these standards. But it's very important that we stick to the open standards for exposing our proprietary context to the tools so that as the tools evolve, we keep up and we aren't left behind. Exactly. It's about making the smart bets on the durable primitives and where those primitives are being built and managed.

9:22I think that's exactly right on the head. And I know a lot of engineering leaders are in the position or maybe even like from last year, right? We're all coming into this year with like a huge baggage of budget of all of these tools that we bought over the last year. Just kind of like, oh, maybe this will work. Maybe this will work. Maybe we'll try this, this, this. And so people have built up a lot. And I think that that's going to ultimately come to a reckoning. And the teams that just bought those tools and didn't build durable learning practices, didn't figure out how do we actually interface with this tool?

9:54What do I have to bring to this tool every time? That like context engine that you're describing here. And so now that they're going to be at the end of this road, they didn't get, you know, literally now they're on a dirt road or whatever the case is. And they're not getting the benefits that they were expecting. and maybe the tool churns or they go to a different tool, the problem just repeats itself because they're not building the durable internal ways of managing how they interact with that. And so then that becomes the responsibility of the org to create this kind of open standard, open platform that's managed by standards that are going to be durable over time.

10:35And that's a bet both for you as the organization and your engineers and keeping the lights on, maintaining whatever context engine you build. But that's also a smart bet for the tools you'll buy because chances are they're going to conform to the spec that you're conforming to because they know what you're investing in. So now your procurement process is easier. It's a win-win. Our procurement process, our security and validation processes, and also, honestly, evals. see a lot of AI products people try it almost as a like vibe check oh this looks good but like vibes aren't reality so unless you have a robust set of evals you really don't know how well is it going to perform in the real world so generally when we build these integrations it's not just exposing the data in an agent friendly way in a secure compliant way those things are very important yes but it's also building the evals because at the end of the day, you're dealing with a non-deterministic system.

11:39And unless you have the evals, you don't have the guarantees that, hey, this is actually worth it. And I do second your point on build versus buy. These are decisions we are continuously making here. Hey, do I build this? Do I take some open source project and use it? Or do I buy some product? And also for the open source and the buy part, do I use it as is? or do I build some adaptation, like layer on the top to adapt it to my internal systems and internal processes? So these are decisions we are continuously making. Space is moving very fast and sticking to standards helps with all of this. Exactly.

12:17It just helps you make smart investments, but also be able to move quickly and benefit from the best and the greatest. As soon as it hits the market, you're already ready and positioned to take advantage of it. You don't have to have an entire, you know, onboarding cycle, a quarter to figure out how do we use this tool again? And in this world, you start to get a lot of benefits on top of benefits. It's like this is where the exponential gains of being able to work with these tools in a fluent way really starts to pay off. What is, you think, like maybe like what people would consider like an old school engineering discipline that maybe they think AI is replacing it, but now in this new world is more important than ever?

12:57And you maybe think they're underestimating or something. Strong knowledge of systems fundamentals for engineers. A lot of engineers treat AI as this magical like black box. And they expect that even if they do not have system fundamentals, they go prompt the AI and the AI can do like magic for them. The AI is great at doing things, but it's also great at fooling you. right so unless you have strong system fundamentals you cannot make the right judgment calls on is what the ai doing correct or not and sometimes even if it is okay correct is it the most elegant way to solve problems or not so those system fundamentals ironically are even more important because ai is like super smart so you've got to be smarter than it right you cannot keep up with its throughput because you're human and it's sort of a machine, but you have to keep up with the nuance, the intelligence, the smartness, the judgment quality that is solely on you.

13:58And even more so for more junior engineers who are entering the workforce right now, this is incredibly important. Otherwise, it's going to result in skill atrophy. Yeah. If somebody was in a position of, I need to build these system fundamentals for a new system I don't understand or for systems I'm still trying to understand, how do you think they could use AI as a learning or an upskilling tool instead of a magic black box? So the first is, this is an incredibly important point which you made, using AI as a tutor and not just AI as a doer. Jensen Huang, I believe the NVIDIA CEO said this a while back that everyone has a teacher in their pocket right now or something like that, And he was alluding to this, using AI as a tutor.

14:46Asking AI the right questions is important. The first is the attitude to ask it these questions. This is important as a practice and important as a validation step itself instead of blindly trusting what it produced to ensure quality of results. The second is how do you know how to ask it the right questions? There is an art to asking it questions. right? What parts do you want clarity on? How do you get the AI to provide citations or like references, which you can go and independently validate to confirm that it did not hallucinate? And also when you're starting off with things, again, this is more so for early career engineers.

15:27So sometimes it helps to just take a step back, maybe go to a whiteboard or do it on a piece of, pen and paper to write this out automatically and ask, hey, can you explain? Have I understood this right? As what they explained really making sense. The other interesting technique is you can ask the AI to test you. You can literally ask the AI to test you, right? It's almost like you have exams at school. You can tell the AI, hey, ask me questions about this topic. And you give it answers. And after that, it essentially judges you. and it says, hey, did you get it right or wrong? And if you got it wrong or right, you can again ask clarifications, questions for more nuance.

16:11It's like a highly personalized tutor. Again, you got to spend the cycles though to invest in this. I know there is a lot of push for productivity everywhere, but I keep telling people that learning cannot be sacrificed at the altar of productivity. Durable learning is what really keeps us in the game long-term. So we have to invest the cycles in ensuring that we learn things and just don't keep mindlessly producing things in a cycle without understanding what we are really doing. Exactly. It's like constantly investing in your education is investing in your productivity. Because at the rate in which these tools change and the ways we have to use them evolve, if you aren't actively learning and investigating and asking questions and maybe wondering if you should throw away that practice that you did two weeks ago now, then you're going to end up with a lot of baggage, misunderstandings, misconceptions that are going to be able to do that.

17:13add up over time and ultimately hamper your ability to get the most out of the new way of work. And I loved how you framed that it has to come from being curious and asking the right questions, slowing down and asking, you know, what do I not know and trying to get a better glimpse at things, but then also making it more deterministic in a sense of like, you know, quiz me, like actually help me prove to myself that I understand this. Give me the citations that go read further. And that just comes from being deeply curious, which I know most engineers, you know, everyone who listens to this show by definition is deeply curious.

17:57And so it's a practice that's already natural to many, and it's just that they have to operationalize it, They need to accept that that's not a passive thing that I could do on the weekends. It's actually something I need to make a part of my regular, even daily practice because it's the way I work. And so if maybe for someone listening to this, if maybe they really relate with how you're thinking about solving this context problem, this memory problem internally, that way you can interoperate with the future. If they were trying to maybe express this and start this at their own org, where do you think that's best started?

18:39How do you get stakeholders? How do you prove the value? How did you get early investment in this idea? are so a lot of early investment in this idea essentially came from providing like negative examples which is here is where things break down without the external like without the internal context or just using the external tools i almost call this as a poking the balloon moment right because otherwise there is so much hype the hype is like a giant gas balloon and you're like yeah just use it but you got to basically i mean like poke it to show that no in some cases it doesn't work so you start with the negative examples but you cannot continue on the trail of like negativity because at that point you're not being okay constructive so then you try to build prototypes or showcases to show that hey here i did this context injection and it actually became like better so then the question gets asked hey is it useful for everyone to do this again and again and again?

19:40Or do we build a repeatable set of durable primitives, abstractions, platforms around this? And then the conversation becomes, who does it? Is a central team doing it? Is it going to scale? Or should this actually be the decentralized responsibility of any team? So this was the journey at LinkedIn. And right now, it's almost like all of our platform teams have two kinds of users. You have the human users and you have the agentic users. And the primitives you build for them are very different actually because the access modalities are different. But it's great that it's become decentralized because I, as the owner of a platform team, understand my system the best.

20:24So when I evolve it, when I change it, I ensure that I do it in a way that my users, in this case, both the agent users and the human users also evolved in tandem, right? Otherwise, you have a lot of context awareness problems for the humans building and maintaining these systems, maintenance overheads, et cetera, simply doesn't scale. So this is the journey. Yeah, it's really critical to understand that you have to have that same shared layer that becomes that fluency layer, that which through stuff passes, you and the agents. And you're right that the agents are going to ingest the information way differently and probably use a whole bunch of esoteric CLI commands to pull out stuff in JSON formats that we're not going to read.

21:12And we might want very nice web dashboards or integration into Asana or whatever our favorite tool is to interop with it our way. But the key is to figure out how do I connect all of those dots so that all of that information can flow back and forth with as little friction as possible, with as much cohesion. And I loved how you gave the recipe for starting. It's so simple. You find the bad examples. Find the really awful ones. Find the ones that are causing problems at the highest stakes. And then you literally make some evals for them. What would make them better? What's the problem I'm trying to solve?

21:48And then you go throw together some prototypes. You throw some agents at it. And then you show that to people. You find an owner for that problem. Whose life, whose job is that going to make easier and better? Who owns the outcome of what this fixes? And then you rinse and repeat. And then by doing that, you build those durable internal practices and mindsets. Now suddenly you get people raising their hand. Karthik, over here. We got like all these problems that's coming out of our AI system that we have just been hitting like a whack-a-mole for months. Like, how do we build a system to survive that?

22:22And part of this, too, and I want to switch to talking about memory, because you've hinted at that a few times as being a really big part of how you solve that. And I think most people stop at just context. Like, oh, I can just go get the context. I can structure it. Now we're fluent back and forth. But where does memory come into play? How do you think about memory in terms of how an organization like yours interacts with the tools? See, at the end of the day, everything is about context management, right? Memory is primarily helping with context management beyond a session. Okay, that is the way to think about it.

22:58So when we look at the memory, we decompose it into different forms. You have the working memory, which is essentially a raw conversation history, which is tied to a session. Now, the reason you need it is because sometimes these sessions can go very, very long. way beyond the context window of a model. So you need a way to retrieve the right parts from it, right? Sometimes you do some very naive things like, hey, I just get most recent N in terms of history. Sometimes that's not good enough. You need a semantic lookup in order to retrieve the most relevant portions, etc. Sometimes you use smaller models because you're trying to save on cost and smaller models have smaller context windows.

23:41So having this kind of memory system augmented will actually help you reduce cost and reduce latency because you don't need a larger model with a giant context window because this external layer is actually managing the context for you. this is the relatively easy part though because it's mostly systems engineering except maybe a little bit of you know embedding based retrieval for that semantic like retrieval stuff which is not so much systems as a little bit of science as well the harder part where the memory comes though is with what i call as more longer term like memory where you are inferring things based on what users are doing either in your agentic application outside, etc.

24:26And you're collecting various sort of signals. We decompose this into three different kinds of memories. We have procedural memories, which is where we try to understand what are people doing? Right? And how do we learn what people do so that the agent can start automating those things as opposed to having humans do it. Or let's say if the agent is doing a few things and the human is not happy about how those things are being done and they want to give feedback, great. All of those switch into procedural memory. Again, it's important to remember these things, right? Otherwise, it kind of gets irritating.

24:58I told you, here is how I like it. You keep repeating the same things again. Not a pleasant experience. So that is procedural memory. Then we have stuff around episodic memory, where across sessions or even within a session, we try to capture episodes which describe specific intents or specific jobs you're trying to accomplish and try to understand preferences associated with that episode. and then you have the long-term memory which is hey there are just few things which are your preferences regardless of the task or job you're doing just as a person how do i infer these preferences how do i store them again a lot of this long-term like memory processing happens asynchronously some of it happens offline some of it happens near line because processing it online is actually very expensive it's a lot of data and it's very expensive to sort of crunch and munch in a short period of time.

25:52Some of it has to happen online, just given the nature of the product, but a lot of it happens offline and near line. And it all gets updated asynchronously into our memory agent. Now you may think, oh my God, memory itself is an agent? Yes, because when you're given an abstract query, you need to know where to look in the memory, how to reconcile potential conflicts, and how to return the right answer. So we've built a bunch of systems infrastructure and AI infrastructure for this and exposed it as an agent. Again, using standard protocols, which all of our other agents can talk to, to almost use this as a sub-agent to enrich their own context and improve the user experience, both for our external products and our internal products.

26:40Now, how these kind of like memories are actually structured is going to be very dependent based on the use case, right? We have hosted agents running in the cloud, which needs access to distributed memory stores with consistency primitives, etc. You have your local agents running on the machine, which actually would do very well with file system based memories because it's just on your own like machine, etc. So again, the architecture, the system fundamentals slightly differ. the trade-offs you make slightly differ based on your needs, but the overall principles are the same. Wow. That was a really cool glance at how your org has built around that huge problem.

27:24Really, what this does is it bridges that last little bit of the gap in the context between it just being like, oh, yes, it kind of gets us. Oh, yes, it's like an intern in our org. oh, yes, it is like aligned, but doesn't know every little bit of what we do to actually covering that last bit of a gap of like, this is a skilled and this is a skilled member of our organization. This is somebody that can be trusted with this kind of scope. And it unlocks a new kind of way of working with the tools in a reliable and repeatable manner. And so, you know, because that is such a big unlock, you invested in being able to understand the different layers of what memory is.

28:05And you walked through some of them and these are just like really amazing concepts to understand of like, you know, even coming back to how we think about when we provide context for an agent, you might have like the system prompt, you have the assistant prompt, you have the skills that you give it, you have the tools that you provide. And all of this is like creating some sort of like, you know, bit of a mash of some kind of like memory process or like best practices for that tool. But the reality is, is that like there's a lot of nuance to how those things are layered. The same thing is true for when you're going to approach memory.

28:38You have to understand how do I cut the edges of these episodes and define them? What even matters for us as an episode? And those are a lot of like internal alignment conversations. is a lot of build it and figure it out. It's a lot of understanding the nuance of how did I get that result and having really good evals to sniff out all of the weird edge cases you're introducing in this new memory system. But the biggest thing that I hear that you said that really stuck with me is kind of how then it gets exposed as an agent. You know, this is actually a kind of way of working with AI and agentic memory that I myself use in my own personal practice.