951: Context Engineering, Multiplayer AI and Effective Search, with Dropbox’s Josh Clemm

23:52That was really the beginning. We called everything prompt engineering. You're fat fingering in both your query, but you might be providing a lot of extra information that these LLMs might need. And that's very powerful overall. What we're trying to now put in the context window for these LLMs is just continuing to grow. If you want to be able to support more complex queries where you're doing tool calling, maybe you're retrieving from remote servers, Maybe you're making a right action later. That all kind of starts to fill in the context window. And at some point you get really bad quality degradation the longer it gets.

24:35You might see these new frontier labs, you know, these models get released and it's like, okay, we have a 1 million context window. Gemini was really the first one, but all the new ones. Okay, now we're at 2 million. We've got 2 million tokens we can handle in our context window. The problem is if you were to use all of that, you would see significant degradation in quality. They end up filling up very quickly, almost more like 100K or 200K. And there's some really interesting benchmarks out there. One I've always referenced is this benchmark called NoLima. And it's very much like this more complicated sort of needle in the haystack type situation where you give it a very, very big doc and then you're trying to find some information and trying to extract it.

25:25And the longer the doc, the worse. You see that curve kind of drop down. More likely, because a lot of us aren't necessarily loading the Harry Potter series into an LM and then asking questions. More likely, we are asking an LM a question and then we have a follow-up question and then we have another follow-up question. And there's more papers on that sort of multi-turn conversation where these LLMs just, they get lost and the accuracy degrades quite a bit. On our side with Dash, we were seeing the same thing. Anytime we were adding all these tools and it would go off and retrieve some information, the data would come back.

26:09Our accuracy just dropped off considerably. We ended up blogging about this. We talked about sort of our approach and how we're solving it. And it's very much, instead of using multiple tools, we're going with just one tool. We're going with sort of a super tool in a way to access that index I talked about earlier that has all the content. You're seeing other companies do the same thing. Cloudflare, Anthropic, they've been blogging recently about the same topic. and their solutions is often to have the LM write code to invoke those tools or pick a tool among a selection of tools. There's a few different solutions here that you can do, but it's all about trying to tighten up that context window.

27:02It's like this sweet spot where if you give it too much, you're going to have bad results. And if you don't give it enough, The LM is going to have to just fall back to whatever data it was trained on, which frankly, in the workplace, it's not going to know anything about your work. So you have to kind of pack it with the exact data that you need and nothing else.

27:22Jon Krohn:How can our listeners, if they want to be great context engineers, and you've called online context engineering the high status job in AI right now. So we might have lots of listeners thinking, oh, I might like to, you know, up my up my pay or my value to my organization by being a great context engineer. What is what is the secret to being a successful context engineer? Well, you know, I kind of think back to more in the early software days, some of even my experience. And anytime I was writing software, there was always limits. You might have memory limits, CPU limits. If I'm working in embedded computing, you only have so many bytes to work with.

28:00Some of the past projects I was working on writing software for UAVs, like low-end computers, or working in very lossy networking environments. And so you're always thinking very deeply about what data am I shipping with my product to run on these low-end computers? or what data, how do I reduce the amount of data where I might have to send over the network? So folks who kind of understand that almost like constraints breed creativity, you have to be very creative when you have these constraints. They're going to do very well. You know, just think about like a payload you're sending with the Mars rover.

28:44You got to send that thing off. It's going across space for five, six months. It shows up. Does it have everything it needs? You sure hope so. So people who kind of understand that piece and are very good at planning are going to be quite good overall in this area. A few other disciplines that make sense. A lot of people who have worked with search. So search is in effect this. This is what Google's been trying to do for decades now is just taking the world's information and serving it up in these little snippets, these 10 blue links. So if you've got that background or have an interest in getting that background, you're obviously going to be able to then pull out the most relevant information from a very wide corpus.

29:31You know, search has sort of been, it used to be the hero. And it's still the hero. It's just more of an unsung hero in the world of AI. Part of that is retrieval. Part of that is like the recommender systems, almost like more classic machine learning. And the last, I think, area that matters for really good context engineering is understanding the outputs, evals, understanding whatever goes in, I need to then be able to verify what comes out. And somebody who's very good at tests, defining things up front, connecting it with the inputs, you're going to thrive in this field.

30:10Jon Krohn:Excellent. Thank you for those tips. And I love how you were describing search as the unsung hero of success in AI. And so on that note, let's talk about RAG, Retrieval Augmented Generation. So you've argued online that RAG isn't dead, which implies that a lot of people think it is. And so tell us about how it's evolved into richer forms of context engineering. Yeah. Yeah. So the reason a lot of people will say RAG is dead is really how RAG originated. This is the retrieval augmented generation. There was a paper many years ago. And the technique that they described was using almost like generating different vector embeddings from your data and storing it in a vector database.

31:00And then, of course, retrieving that later, passing it along with the prompt, and you're off and running. The reason vectors were chosen, it ties really nicely with the underlying architecture for large language models. Everything is very kind of token and embedding based. And a lot of people really went way in and decided to create vector databases and store all their content in vector databases. and again there's nothing wrong with that because at the end of the day you still have to retrieve something to be able to add an additional information to these llms how it's evolved i would say a couple things one vector uh retrieval is great at kind of like meaning based I mentioned before, creatives might want to search for old car and vintage cars will show up.

31:58But a lot of people are recognizing that these more keyword search approaches using techniques like BM25 can be very, very effective. And so hybrid retrieval has become much more important overall to getting your content for these LLMs. But the other kind of pattern that's emerged is more around agentic retrieval. And the idea here is, think about the world of, you have the data layer, you have all this data, and then you have almost like the application that'll go reach and go fetch that data. In the old world, you often just had one chance. I'm going to make one retrieval call. Whatever I get back, I'm going to use.

32:41And that was really kind of the initial architecture for RAG. Again, hybrid retrieval, better. But if you want to handle very complex queries where based on the data you get back, you may take a very different path. That's where you want these agents to not just fetch once, but have multiple tries to go fetch content. And that might be from a database. That might be from a third party, an API, MCP call. That might be actually taken in action. And then based on that, it may decide to do a retrieval. So you have a bit of a data layer and then the sort of almost agentic layer. And I think it's okay to kind of think about both because you do want those multiple tries.

33:27But you also want to organize your data and index your data in a way where it can be explored. And you're seeing this quite often, again, kind of going back to coding agents. cloud code, cursor, these are extremely popular tools. They need access to your code base. And the code base isn't as simple just going and looking up some chunk of code. You need to almost explore the code base. You need to understand folder structures. You need to understand obviously parts of the code itself. You need to find exact function names. So you You kind of like have this hybrid architecture emerging where, yes, there's some amount of indexing your code base, but these coding agents have an opportunity to almost explore that index and then pull back all of these different bits of context to get you the best outcome.

34:25Jon Krohn:Nice. I love that explanation. Thank you for the deep dive into the evolution of RAG in search. and now tying what you were talking about right at the end there, you brought back your love of, if I dare call it vibe coding, of building easily with tools like CloudCode. And let's now tie that into the search conversation that we've been having recently because you wrote an AI search app with a hundred lines of code and open sourced it. What did you learn from that experience? And why did you do that? so i'll go back a little bit of history i'll give you a little bit more of a side app i work on um i'm a big fan of sports i like football and i like fantasy football and i like fantasy football for the reasons you'd expect i love the the stats part of it and i love to write my own almost programs or apps to help me do a better job in in my fantasy football leagues And one of the features I always wanted to build was more of almost a season preview for a particular player.

35:34You see these in a lot of publications. Here's what to expect this year with this particular player. And I was thinking, like, how do I do that? I could do sort of Mad Libs style. But I started to explore a lot of language models. And, you know, my time at Uber Eats, we were starting to do some work with conversational AI. So I was familiar with the technology. And this was back maybe 2021, 2022. And I was looking at a lot of more open AI models, GPT-3. Okay, there's something here that is very, very impressive. I can create the content. Of course, I had to go fetch fresh stats for the upcoming season.

36:15That's where you want retrieval augmented generation. And I was trying to kind of put that together. And it wasn't super easy. I ended up getting a version of it working, but I learned a lot about prompting. And I then stumbled upon a product called Perplexity, which I was a big fan of. And they really did a phenomenal job, kind of almost pioneering this really intuitive product interface where they're collecting all the search content and able to present in a really easy way, effectively answers. And so when I saw that, I was like, you know what? I want to kind of go back to some of the work I was doing.

36:53I want to maybe bring in some of the newer models, think about different techniques that I had sort of observed from that product and ended up building my own search app, very similar to that, kind of connecting those two things, the fantasy football season outlooks with trying to get my own like sort of search app going. And because, you know, I learned a lot, I figured let me put that out there and open source some of it so that others who may come along the way will learn, you know, different ways of prompting, different ways of adding grounded facts through citations, really trying to kind of push the state of the art with retrieval augmented generation.

37:34And then a little bit of a preview of what you could do further. So that's sort of like really starting to just understand how these models worked, how I could get the most out of them. And frankly, you know, it's a lot of fun. And a lot of those techniques are still going strong, which is pretty cool to see.

37:51Jon Krohn:Nice. And then it must make it helpful for things like Dash at Dropbox, having that real experience of making something work. It must be easier to then talk to engineers on your team and help them brainstorm on how to be getting through some of the things that they're struggling with. Absolutely. Yeah. I mean, we want to be that answers engine for the workplace. and it's essential. You got to get these things right. You've got to be able to reduce the hallucinations. You have to be able to present real user facts and the better way, the better output where you can have citations so you can almost prove your work, that is going to be important for any product, especially with these AI products.

38:34Jon Krohn:Nice, nice, nice. Makes a lot of sense. You've actually had more than just your fantasy football side project get onto our radar here at the Super Data Science Podcast. You also, it looks like you might have the number one earthquake app on Android called Earthquake Alert and something called Yaddle.ai, which is an LLM powered search engine. And so, yeah, tell us a bit about those projects as well and how doing these side projects helps keep you sharp and grounded. Yeah. Why you think they're an effective part of your leadership style? Yeah. Well, so Yaddle is that 100 lines sort of answers. I gotcha.

39:16That's the fancier version. I continue to work on it on the side. And it's just, again, it's a very fun, almost playground for me to do some of the real-time search and try these different models that come out. I have a model switcher. I use some amount of query classification to figure out, in certain cases, which model to route to. It's a fun project for sure. Same thing with the Earthquake app. This goes back way back when I was getting my master's and I was taking an entrepreneurship class and we started to learn about these emerging platforms with mobile, with the iPhone and with Android.

40:01And so I was like, hey, you know, let me just sort of play around with this to see what I can build. It turns out when you're an early independent developer and you put an app out there, you end up getting a nice flywheel effect. And so, yeah, over the years, anyone that's been looking for earthquakes, earthquake information would spot my app. And, you know, I like earthquakes. I grew up in the Bay Area. I've had to experience a lot of earthquakes in my life and just had sort of a fascination with that. It was really a learning opportunity overall. So kind of connecting your second question, I do feel very strongly that leaders, engineering leaders, they really need to find the sweet spot with how technical they are.

40:49You obviously can't be too hands on because then you have no time to think about strategy, people development, org design, things like that. But you also can't be not technical and come across like, oh, you're up in the ivory tower. You're just saying, hey, we got to go do this and not have that backing to say, here's why or here's why I agree. So I do think leaders finding that kind of technical balance is really important. That's an important part of my own philosophy, who I look for in good leaders at work. And a lot of times you may not have a lot of time during the day, so you can do things on the side.

41:31I also feel very strongly AI is changing things considerably. These tools are very, very new. And it's very hard to say, hey, go use more AI. We talked about that earlier with work slop. That's sort of why we're in the work slop situation. You almost need to use them yourselves. You need to embrace them. You need to understand how they work. and frankly, their limitations. And then you're just so much more willing and able to provide crisp and concrete advice on, yes, this is an AI tool we want to deploy. Here's why. I thought about it. Here's some gotchas. Here's how we're going to address that.

42:16And so I do think AI changes things, which is why just staying much more technical, much more sort of on top of the trends, is essential in today's modern leadership.

42:28Jon Krohn:Nice. Speaking of your leadership, something that I haven't mentioned yet on air is your tremendous background. So you've managed and scaled large teams from, based on our research, over 150 engineers at LinkedIn, over 400 at Uber. And you did that driven by a belief in a people first approach, which I hear coming through in your answers already. So tell us what's the most challenging part of maintaining this people first approach once a team surpasses, I don't know if you know Dunbar's number. Yeah. Yeah. Yeah. So, uh, Robin Dunbar, he's an Oxford university researcher and he studied lots of different communities all over the world.

43:10Jon Krohn:And basically there's this number of, it varies somewhat, but it's around 150. It's kind of the maximum number of direct relationships that you can have in a community. And that LinkedIn number is right there at Dunbar's number. And then the Uber number of 400 people, that's well over 150. So how do you maintain that people first approach once you can't know everybody's name? Yeah. And at LinkedIn, that was more of a total number, but absolutely. It's very important to think about when you're growing teams, there's sort of growing teams and then there's scaling teams and the things that you were doing or almost the spirit behind the different processes or cultural pieces, norms that you've defined, you want to figure out how to kind of replicate that in a way where you're just not taking on more work yourself because that just won't work.

44:00You know, maybe in the old day, you're doing that kind of sit down team meeting where you're doing open discussions as the team grows. Maybe those turn into small group AMA sessions and you can do multiple of those. As it grows, now you're trying to meet with maybe smaller groups or your skips. And at the same time, being very deliberate with the leaders that you have to try to kind of replicate that all the way down throughout the organization. Maybe it's a version of a regular all hands. That's something we actually love doing. I started doing this back at Uber during the pandemic where everybody's now working remotely.

44:42You're kind of losing the sense of self, sense of team identity. And I got a really great suggestion from a manager of my team where, hey, he's like, hey, why don't we start sort of a very short and sweet weekly check in where they can kind of hear from from you on priorities. We can maybe share wins together. We can kind of talk about any shifts that we may want to do. And I said, yeah, let's give it a shot. We ended up doing that for almost three years. And I still continue to do that today at Dropbox. and it's just like a fun ritual to bring everybody together. That's only one way of doing it.

45:21You sort of have to think through all these different aspects of how to kind of scale culture. And the other thing I like to really kind of emphasize is I like to really obsess about the inputs of teams or team productivity. I think a lot of times we're looking at the outputs. We're seeing how many PRs are they putting up? How many features have we shipped? And a lot of times the solutions are on those inputs. It's, do we have clear goals? Is our strategy sound? Do I have the org structured correctly to match that? Do I have all the right leaders in place? How do I think about team structure? Do I have good seniority mix so that we have mentors along the way?

46:09How's our operational rigor? If you have outages all the time or a lot of bugs all the time, teams are constantly context switching back, not being able to really move forward. You think about tooling, you think about a lot of those different aspects. I think that's one way to make sure that you can still drive in a very people first way, but also make sure it's scaling overall.

46:35Jon Krohn:Yeah, related to what you were just saying, an interesting piece that we pulled out is that you often describe your role not as building a product, but creating conditions for great engineering to happen. There's a tweet where you shared a meme about thinking like a farmer. So the farmer doesn't yell at the plants because they aren't growing. Instead, the farmer focuses on preparing the soil, irrigating, fertilizing. And I love that our researcher, Serge Massese, he's a data scientist at Syngenta. the agricultural company. And so he went into a lot of detail on his question on this farming one.

47:11Jon Krohn:And so using some inspiration from Serge here, what's one quote unquote soil problem that we have in tech companies today? What are we getting wrong about the soil for engineering culture? And how can we improve it? How can we farm better products? Yeah, so I don't know when I came across that picture, but it's, I'm sure many of you have maybe seen it. It's just sort of a screenshot from some sort of conference, Think Like a Farmer. And it really kind of touches on the aspects I was just mentioning. You're obsessing about the inputs. You're really making sure, okay, am I planting correctly? Am I thinking about seasons?

47:53Am I building resilience in my organization? And I do think that's probably one of the bigger challenges here. The world of AI, we're moving very quickly. There's all these new innovations, new frontier models coming out. They're constantly leapfrogging one another. And there's just always these questions around, do we bring this in? Do we adopt it? Do we not? What's our competitors doing? And I think that kind of whiplash does affect a lot of teams very negatively. They end up losing a lot of morale. They maybe are working long hours, but not me. moving in the right direction or making meaningful progress.

48:29And so I think just kind of building that kind of more of a regular culture of resilience is really important. Being very upfront with your team saying, look, like things are going to be tricky or yeah, there's going to be these different competitors coming in. Almost emphasizing, here's what we think about it. Here's our strengths. Here's how we're okay. just like that kind of constant communication to recognize the current environment we're in because it is messy and it just feels overwhelming at times but create that right culture and I think the teams that understand that, embrace that they're going to continue to thrive they're going to kind of get through the noise they're going to be much more focused on their goal and teams like that are going to win

49:15Jon Krohn:Great, I've got one last kind of technical leadership question for you before we start wrapping up the episode this is another tweet of yours where you called out an anti-pattern where big organizations become so risk averse that they outsource every decision to an A-B test. How should leaders distinguish between decisions that need data and decisions that just need your gut instinct to get right? It's quite problematic, especially with bigger companies, because there's some amount of risk aversion. You may be operating in a position of strength. You might be a market leader and you almost don't want to lose that.

49:55And you kind of open yourselves up for other startups or other competitors to disrupt you. And it kind of just creeps in. It's a bit of a problem overall. And it's sort of a problem where the more scale you have, the more data you have. And so it's very clear, oh, I just have to look at the data. It'll tell me what I can do. And And I see some analogies with AI tools. I do think you're seeing a world where people who are starting to embrace AI for almost like all their decision making, they're losing a little bit of their own sense of here's what I think might work or some conviction. And so I think those analogies are very, very important.

50:39And it's essential to try to stay grounded as much, to try to maybe talk to your customers more, get more of the qualitative data to offset some of the quantitative data you might be seeing. And kind of just bringing that together and make much more well-informed decisions. I remember a long time ago at, I think it was Uber Eats. We had a smaller competitor that was moving very quickly and shipping a lot of features. And we were always like, whoa, what are they doing? What are they seeing? And later we found out they didn't actually have an A-B testing framework even set up. This was very early days for them.

51:22And it was almost like you're envious of that. because it's like, wow, that requires you to be very convicted. If you're going to ship something, you're going to ship it. You're going to move quickly and you're going to learn from it. And if it doesn't work, sure, you can pull it back. But a lot of times you bring in those extra data sources. It'll draw out decisions. It'll really kind of create some amount of lethargy in your organization. You've got to fight that as much as possible. You got to sort of stay sharp, stay convicted, stay human, and be very intentional with your decisions.

51:56Jon Krohn:Excellent soundbite to round off your technical responses there, Josh. Thank you so much. So yeah, before I let my guests go, I let you know just before we started recording that I always ask my guests for a book recommendation. It sounded like you might be reading something interesting right now. I am reading Masters of Doom. It is the story of the two Johns, John Carmack and Romero and they are the co-founders of doom the game the video game doom oh right yeah yeah yeah that was such an iconic thing for me as a kid growing up that I remember I grew up in downtown Toronto and so I take the subway to school and when I was very small sometimes my grandmother would come and take me and she always wanted to spoil me and so she took me to a bookstore and I had her buy me the Doom strategy guide, even though we did not have a computer.

52:53Jon Krohn:And I just kind of like, you know, memorized gameplay and the guns and the demons. And yeah, it was a really iconic thing for me in the early nineties. Yeah. It's been a fun read. It gives you that kind of glimpse of the early days of video game development. And yeah, You're back in the days when the only computers were at universities. They were these big mainframes. And then it started to transition to the personal computer, the different Apple, Apple II, Commodore 64. And a lot of that was like, some of that was a part of my childhood, a little bit on the later side. But it's nice to sort of get that nostalgia hit.

53:36And it's just sort of fun to see the creators of something and their backstory. All this stuff comes from somewhere.

53:44Jon Krohn:Yeah, exactly. All right, Masters of Doom. There's something for our early 3D gaming lovers. And yeah, before I let you go, Josh, I've cited tons of great posts, LinkedIn posts you've made, tweets. Where are the best places for people to be following you after the episode? Absolutely. Definitely check me out on LinkedIn and I'm on X Twitter. I like to bring in a lot of my observations in AI, but I do like to connect it to the world of maybe large-scale software development that dominated my first kind of decade, couple decades of work. because I am spotting a lot of similarities around what agentic architectures compared to just microservices.

54:40There's a lot of overlap and I think there's a lot of lessons that we can learn from one another. The ML practitioners, the backend engineers, I think there's a lot of really nice synergy there. So that's going to be a lot of the content I put out. And of course, I'm bringing up a lot about our innovations on Dash, what we're doing at Dropbox, how we're thinking about machine learning, how we're thinking about different optimizations, context engineering that works.

55:03Jon Krohn:Love that. Yeah. Our listeners, I'm sure, got a great taste of your brilliant ideas related to AI and engineering leadership, product development, and I'm sure that you will have a whole bunch more followers after this episode as well. Thanks so much, Josh, for joining us. And yeah, maybe we can catch up again in the future and hear how Dash and other AI initiatives at Dropbox are coming along. Yeah, thanks, John. It's been a pleasure. love talking about Dropbox, love talking about Dash, and love talking about some of those, you know, stories from the past.

55:40Jon Krohn:Plenty to learn from the rich experience of Josh Clem. In today's episode, he covered how enterprise search is broken because workers now have 12 to 20 different search bars across their apps. Dropbox Dash aims to solve this by creating a universal search layer that ingests content from all your work tools. He talked about how multiplayer AI means AI systems that understand not just you but your entire team, their projects, and how everything connects, enabling collaborative intelligence rather than single-player chatbot interactions. He talked about how context rot occurs when you stuff too much information into an LLM's context window.

56:14Jon Krohn:The key is context engineering in only the needed data and nothing else. And we also talked about how over-reliance on A-B testing creates organizational lethargy. As always, you can get all the show notes, including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for Josh Klem's social media profiles, as well as my own, at superdatascience.com slash 951. Thanks to everyone on the Super Data Science podcast team, our podcast manager Sonja Breivich, media editor Mario Pombo, partnerships manager Natalie Jaisky, researcher Serge Massis, writer Dr.

56:48Jon Krohn:Zahra Karchet, and our founder Kirill Aramengo. Thanks to all of them for producing another excellent episode for us today, for enabling that team to create this free super data science podcast for you. We are deeply grateful to our sponsors. You can support the show by checking out our sponsors links in the show notes. And if you'd ever like to sponsor the show yourself, you can head to johnkrone.com slash podcast to learn how to do that. Otherwise, otherwise share this episode with folks who would benefit from it review on whatever podcasting platform you use or youtube i think that's helpful for driving more engagement and viewers on the show subscribe if you're not a subscriber but most importantly just keep on tuning in i'm so grateful to have you listening and hope i can continue to make episodes you'd love for years and years to come till next time keep on rocking it out there and i'm looking forward to enjoying another round of the super data science podcast with you very soon Thank you.