The best model for your team? You haven’t invented it yet. | Ai2’s Tim Dettmers

0:04Tim, I wanted to ask, just kind of like before we start diving into today, like I'm curious from a personal perspective, what is it like to sit in this kind of like swivel chair between academic and industry? Like what kind of perspective like has that been for you? And like, you know, what makes you most excited about being in that in that place? Yeah, I mean, sort of industry and academia, the divide feels quite extreme. that feels to be also sort of one divide sort of resources. I think there's also the divide in perspectives. And then it's also, if you look at locations like most of the AI happens in the Bay Area, and it's just if you're in a Bay Area, you're like in this bubble, everything's super exciting, everything goes super fast and that sort of thing.

0:48And some of it is true, but some of it is not. And so as an academic, you could sort of lean back and sort of take it in and sort of make up your own life. And with that, you can actually carve out bits where you say, I can be competitive in this field, even if they have all the resources, but for pursuing sort of particular ideas that they might not be able to pursue. And so that is sort of an interesting space. But the other sort of is just this trade-off between working with the most resources and being sort of a cock on the wheel or being sort of resource staff, but trying to make most of it.

1:29And, but you have all the freedom and you're working on big pieces that, you know, for bringing this out, that would be sort of my doing instead of being like a tiny piece in a big team. I love that framing that you prefer working in this almost like resource strapped environment where you're able to fully explore the realm of thought. and then not only explore it, but then figure out the methodologies and get there. And whereas in this other environment where it's like resourceful and there's like so much that you can be pulling on and with each other, but the incentives are different. That's like, it's a capitalism-minded machine.

2:06What you're trying to drive is a production and an output. And so like you said, the perspectives are so different and you're after the quest of like, what is best? What are the foundational elements of this that we all need to understand? And that's like the really important gritty work that maybe it's sometimes folks that are more industry minded, like don't want to pay much attention to. Right. And so it can be like a little difficult there. Yeah. Maybe also a small story there that might be sort of quite interesting. And that is doing my PhD, I was doing like a sort of long term internship at Meta.

2:41They had a lot of GPU resources. And it's like the dream of the PhD students have like hundreds of GPUs. So I had these hundreds of techniques where I just run experiments and experiments and experiments. And I was making sort of progress, but not as much progress as I wanted to. Then at some point, it was time to go back to the University of Washington. I had much less resources. But now each experiment needs to be very carefully chosen, very carefully analyzed. And then I actually made some discoveries that I wouldn't have made if I just looked at experimental results. Now I dove deep into the data, found some curious things, and I was actually more productive.

3:15So more resources doesn't necessarily mean better results. You can make more insights with less resources if you dive deeper. In fact, when I think you have to be resourceful, that's when you're pushed to make those really ingenious kinds of discoveries. And it challenges all of us to strip away complexity instead of adding it on. It's a really great takeaway. And I'm really excited to explore all of this in our conversation. because, folks, today my guest on the show is Tim Detmers, a research scientist at AI2 and an assistant professor at Carnegie Mellon who has built a career about finding the signal and the noise of high-performance computing.

3:52And like we just talked about, while larger companies and industries might have huge resources that cook up massive foundation models that can solve wide-domain industry problems, Tim and his small team recently built SARA, a state-of-the-art coding agent through what he calls a hot plate and a frying pan, like the complete opposite of being so full of those resources and working with a strappy team and strappy GPUs. And we're talking about the tactical engineering behind that today, the automation muscles that make it possible, but also the groundbreaking research that we've covered from Tim here on the show about how this type of breakthrough can allow more teams to return to and embrace specialized models of their own and not necessarily be beholden to foundational off-the-shelf tools.

4:42So we have a big conversation to dive into today. It's very academic-minded, super excited for it. And Tim, welcome to Dev Interrupted. Yeah, thank you so much for having me. Of course. So I want to start at the top with just the analogy I gave of a hot plate and a frying pan from your recent report about how you and your team created Sarah. You know, for our audience, maybe we'll start at the top and you can walk us through a bit of what Sarah is and what it is able to, you think, unlock for engineering teams using AI coding tools. Yeah, yeah. So the big problem here is there are like awesome coding agents out there.

5:22Can we reproduce in them sort of academia? And so if you look at these coding agents, it seemingly means endless of resources. Like if you look at good coding agents, usually only the big companies can produce them. And so the question is, can we produce them sort of with less resources in academics at Angla or at AI2, the Institute for AI? We're sort of a mix between sort of industry and academia. And the main goal that we have is those open source models, those open source systems, bring all the information out there so people can replicate things. And so the question was, can we replicate coding agent performance with a few resources?

6:03then basically give it to everyone. That was sort of the main goal. And yeah, if you look at sort of industry, they have like these sort of industrial kitchens that are just lots and lots of people and lots and lots of sort of machines, GPUs, and everything ties together and sort of big reinforcement learning infrastructure. And we didn't have that. We had just a couple of people sort of working on this project. We started out with 32 GPUs. So that's the comparison between like a hot plate or a frying pan that we had against the industrial kitchen that sort of these big companies have. And so, yeah, that basically pushed us in this resourceful domain to try sort of as efficiently as possible, get good performance.

6:50We made certain trade-offs, supervised fine-tuning instead of reinforced on burning. We were very sort of careful with developing a synthetic data generation procedure that is highly efficient, very cost efficient. And then again, yeah, if you put it together with the right sort of pieces, we got extremely good results. We could replicate small, basically, close source models like Mistral Small or QAN3 Coder. And yeah, so that was quite successful despite the resource bottleneck because that forces to be more exhaustive. So to understand more clearly, the idea is that you were able to get almost a comparable kind of performance in like a specialized or a pre-existing or what we call like a brownfield code base, right?

7:35Using a tool that's trained on specialized, like synthetic data from that code base. And this is a realm that is familiar with folks who are working with these tools and have explored things like fine tuning and have looked at how do I create these large data sets And, you know, part of your research is about the methodology that goes into creating that data set and how you could actually use something that's resourceful and low compute in order to generate that kind of data. And so I kind of want to, before we kind of talk a little bit more about that, I want to ask how you think this challenges the assumptions that engineers and engineering leaders have had up until this point.

8:16That, you know, oh, if I want really great top-of-the-line competitive performance on my engineering team and their output, I have to use the open AIs and the anthropics of the world. How does that challenge that assumption? Yeah, yeah. I mean, the main assumption is basically you need what the big labs have. What the big labs have is they have lots and lots of sort of environments. And here, an environment is basically a playground for an agent for a particular problem where they can do all kinds of things. And then what you want to have is sort of large-scale reinforcement learning infrastructure where you reward the agent if it does something right in the environment.

8:55And because you don't have real training data, You don't have human data. For example, human programmers program. It has very little data out of that. You generate synthetic data. So you look at how the agent does it, put it in a data set, and train a new agent that learns all of the data. And so the assumption was like, that's what you need. Everybody was trying to do it. But yeah, what we found is you can be very precise in terms of how you generate the synthetic data and basically how long do you interact with the environment. and with that, you can sort of scale very easily. And that sort of challenges that you need all this infrastructure, you need all the GPUs, you need all this complexity.

9:38So we broke it down to the most simplest component. And even in the beginning, we made it simple, but then we made it simpler and simpler and simpler. Then we did it also more efficient. I love that you call out that you make it simpler and simpler and simpler. That's something that we call out a lot on the show about how a lot of times unlocking that velocity, but also that just like raw performance from an agent or from an LLM can be achieved by taking away as much as possible and creating really efficient closed systems that both you and the model can perfectly understand at any given point.

10:16But it's still like doing some kind of knowledge work, some kind of transformation, right? And if you kind of create that shared thinking space, you can get some really effective kind of outputs. That's right. Yeah. I also want to know about this type of tool that a team might explore creating. Let's say they have a pre-existing code base and maybe it's very specialized. I think a large problem that some engineers experience is that the ability to adopt AI is not equally distributed across engineers and the domains in which they work. Like front-end engineers and folks who work with TypeScript or whatnot, they're able to use these tools in a much, much, much faster way than somebody working on a very deep embedded language, embedded systems and stuff.

11:04So does this type of approach allow those teams to actually start unlocking games within their own code bases? Yeah, so that was one of the main points that we made also with the Sarah paper. You can quickly specialize on your private data. This is a very big advantage for open source. Like if you're OpenAI and Anthropic, you will not pick like random data from some company and put it in your training set. But if you're the company, you can take an open-weight model and now try to train on your data. And I think it's sort of common knowledge. Frontier models work well, but if you work with less and less and less common data, they work worse and worse and worse.

11:45And if you're very sort of specialized data in your company, code bases, but it can also be code bases related to documents, These setups are very common. A lot of companies say, like, frontier models don't work for us. And so what you want is basically specialize a coding agent based on the company data that you have. And then it can often be actually more effective than frontier sort of model. We had the beginnings and sort of Sarah was the beginning of that. And so what we developed is basically a procedure that makes as little assumptions of the new data that you have. A lot of methods previously, they were applied, for example, software tests to see that the synthetic data that you generate from a data set is correct.

12:29Like you make some changes in the code, you know that the code changes are correct. We threw out this assumption. So that means we can take any code base and generate synthetic data. And we don't make the assumption that the generated synthetic data code is correct. But if you do it just in the right way, that data is actually as good as sort of data that you verify. And so what that means is you can just take a code base, a private code base for your company, generate lots and lots of signed data quickly because you don't have this assumption. You don't need to carefully verify, which is very expensive.

13:09And then you train on that data and you get very good performance. And so we show we can get better performance in many closed source models that are of the same size when you just$700 to basically train such a model. So if you're a company, it's pretty efficient to do this. And right now, these are the beginnings. You want to push it to very, very large models, like 350 billion or so. I think that will become very interesting. And you can actually exceed sort of frontier performance by training on this private data. I think there's this biggest advantage of open source and open-weight models.

13:47Yeah, I agree. This is an opportunity, I think, for the open source community, the open-weights community to be able to kind of prove the efficacy of this research and this kind of approach. Because the way you just broke it down, and I kind of just want to zoom in on it for a second. You say that the synthetic data that's generated on the code base or the thing to be trained on, it doesn't have to be correct in this case. And that's the assumption that we're throwing away from inference training and fine-tuning from before is that we wanted to verify, which is computationally expensive, and it takes a long time then to assemble the data.

14:23But if you just assume that it doesn't have to be correct, that's where soft verified in Sarah comes from, then it allows you to go a lot faster while still achieving comparable results. And this is actually something we see in like across all knowledge working domains. I think of any kind of training data set that goes into any kind of specialized knowledge working tool that I've worked with, that I've seen, that's been constructed. A lot of the conversations and pairings go in are not perfect. And there's not always a perfect deterministic way to resolve a conversation or a query. So they exist already in this nebulous kind of like gray area realm.

15:04So if you embrace that same kind of idea for something that we see as more deterministic, it actually unlocks an ability to get progress from these tools. That's right, yeah. And it's just very cheap, and you don't have any further assumptions. If you want to have correctness, you often need software tests, but you can only generate as much data as you have software tests. And for some parts of the code, you might not have good software tests. and now with this you don't need you throw away all this you just generate data and it's actually very common finding even in reinforcement learning where you just try to do the correct thing people a lot of people say like hey if you don't if you use some of the incorrect data actually works so model is better and how to think about it is the model not necessarily needs to learn what is correct but it needs to learn how to map an instruction that is related to a code base to basically the steps required to translate this instruction into an outcome.

16:06And the exact outcome is maybe as important as the entire process. If you have a weaker model, the process is actually more important because the model has difficulty modeling the exact outcome. But that's how you make quick progress. In the end, you might need reinforcement learning, but in the beginning, it's just a much more efficient tool. So help the model basically learn this process, mapping instruction to the process of how doing a task, a coding task. Just to maybe package up the idea of Sarah and what folks, our listeners, can do with this kind of insight. What would be your call to action and your advice to an engineering leader, to somebody who is working with AI engineering teams that are using AI coding tools regularly on code bases and to produce output for the business?

16:56there's been a lot of experimentation and adoption around new tools, new workflows. If this was something that a team wanted to seriously explore, how would you recommend they maybe even measure or understand the impact of that? Do you have advice for them in that situation? Yeah, so that's actually a big gap in academia that you have this gap of you can evaluate on data sets, but almost certainly big models have been trained on this data set. So even if you have small open-weight models, they're probably trained on the second step. So in a company, you really have private data. And if you evaluate on that, you get sort of this real gap.

17:33And so for that, you need to create the evaluation benchmarks. And the models that we have there, we're getting rapidly specialized. They're not quite there yet. But as a sort of engineering leader, what I would sort of recommend is pay attention. This will move very quickly. And I think very quickly, we actually will have models that are better than the frontier models. because they're specialized to your data. And so there will be this transition point. And I think as an engineering leader, you should be aware when this transition point comes because then you want to quickly switch. Everything moves very fast and you can move faster if you basically transition exactly at this transition point.

18:13It's really great advice. It's about understanding how it's moving and making the switch at the most opportunistic, economic, it makes the most sense to jump that bridge. And thanks to the foundational research that you and your team have done, now there's a roadmap for teams to be able to achieve that. Zooming out a little bit though, Tim, I want to talk a bit more about your own experiences as an academic and somebody who does research with AI and around these types of tools and applying AI to software engineering. And behind that, there's obviously a lot of, you know, agentic workflows that you've picked up, adopted, explored, especially probably in the last year.

18:59I'd love to learn more from like your recent writings on your blog, but also how you approach this as someone who's more academically and research minded. Yeah, yeah. I mean, I wrote this sort of blog post about using agents that we left behind a couple of weeks ago. I feel since then it also changed dramatically. Everything is just going so fast. I mean, for me, it almost feels now. Now it's sort of we are at a point where productivity can be measured in tokens. The more tokens you generate, the more productive you are. It's not true for all jobs, but it's approximate. And I use agents both for research, for software projects, then sort of as a professor for certain tasks.

19:47And yeah, sort of week by week, you improve, you generate more tokens. And right now it's exponential. My doubling time is around 10 days. And so it just keeps growing. I'm just like crazy. So yeah, I think we're at a point in time where how to use agents doing that well from you, the most important skill that you can learn. And I think part of that skill as well is understanding when to build an automation, when to turn to an LLM or an agent to solve a regular recurring problem to get those compounding gains. Like what is your math on that to decide like how to approach like automating things that keep you busy?

20:28Yeah, yeah. It's actually sort of this double-edged sword. And there are like two perspectives. And I would sort of, so I'm German and I work in the automation industry in Germany. There they have like a very sort of straightforward calculation of basically, what is your return on investment? And then there's, I would say, the more sort of scrappy approach that has been quite common in China is just try things and see if they're better and learn things and sort of improve them all the time. And so the German perspective that I also basically learned is make this calculus of, you try to estimate how much more productive you are.

21:08So you say like, I do this task once a day, takes me 10 minutes. And then you just think about how long does it take me to automate? And then not only this automation step, but every day you probably are frustrated with your solution. You need to improve it. You need to think about it. If you add up all this time, will it be more than basically the time that you save? So if you need, instead of 10 minutes, 9 minutes, you save one minute a day. But if you develop this thing for like 10 hours, it might not be very sort of, and the payoffs can vary late. But that perspective can also be deceiving because it doesn't account for learning rates.

21:49So when you learn to work with the task, then the next task that you do, you might be able to automate it more quickly. And then the next step more quickly than that. And so that's sort of this more Chinese perspective is try some things, learn some things. At some point, you're so good at automating things that you can automate things very, very quickly and very, very efficiently. And so sort of rational calculus in terms of cost and payoff can be helpful, good perspective, but one shouldn't be fooled sort of in the long term because it can be also a bad choice. I think these are sort of the trade-offs that give you a good framework, how to think about problems in terms of automation.

22:33And if you're someone who's able to maybe achieve that and get that 10x, 100x kind of output that's compounding, or perhaps you're a team leader or manager and you have somebody on your team who is unlocking that but others aren't, how do you maybe advise them to help distribute those gains, educate and share that ability to others, but then also use these compounding gains they're doing to compound onto others? Do you have strategies for being able to talk about that? Yeah. I mean, it's sort of quite interesting. I work with my students and try to make them more productive with agents. Often we have discussions about what each other is doing and how we can benefit from each other.

23:15So very similar to what you described, sort of this problem of doing better ourselves, but then also sharing that knowledge, sharing infrastructure and so forth. And one learning is, for my staff personally, is if you build certain tools that you use your agents, you can sort of build on top of them. At some point, your productivity gains stack. But then if you share those tools, it might not quite work for others because they work in a different manner. So I don't know how to solve it. Maybe you standardize certain tools and people sort of share and at some point, merge in a particular way of doing things.

23:51Maybe that's also working with engineering, like the workflow, how you work GitHub, and post-request and whatnot, it's standardized, basically. And so I think it will evolve. But I think the most important thing is just talking to each other, learning from each other, and just being sort of also humble in a sense that certain approaches that might not look great, you might get great payoffs. And so just being open-minded, try a lot of things, and just talk about it together. I think for now, that's the best strategy. Standardization might make sense, but probably not now. Do you see strategies, differences between people that are in the software engineering industry versus those that are in academia and what they choose to automate and how they tackle it?

24:45Like what are some, being in that swivel chair, like we talked about in the beginning, like what is it, what are the perspectives and what do you think each side could learn from each other? Yeah, yeah. So it's actually quite interesting. The first things that I automated were sort of related to my job as a professor. And these were actually the hardest tasks to automate because it's very different from engineering, like agents are really good at engineering. But then the next step was for me, basically, research, which was more engineering-heavy, systems research. And at the end, it was just pure sort of engineering.

25:19And so if I look at those, you learn different skills if you do certain things. So for my job as a professor, it was like, for example, creating proposals, doing literature research, sort of figuring out how pieces of research is published fits to certain projects for certain students, brainstorming ideas, then just administrative stuff, like, I don't know, receipts, putting them together and that sort of thing. And so that was sort of the first sort of professor level stuff. Then the next sort of stuff is sort of research, for example. And then the tools that are already built for literature research, they immediately become relevant to research because now it's just a couple of extra steps to figure out where are the gaps in the literature.

26:08The agency can figure out itself. The agency can sort of brainstorm ideas very quickly. And then you can go down certain ideas very quickly. You buy pipelines where they automatically launch jobs on the cluster and can sort of just tolerously sort of do things. And then at the end, it was sort of, I was moving to this engineering domain, and that is the simplest domain. And in the beginning, I had trouble launching jobs in parallel, sort of in academic settings. Like, how many chances have been paralyzed? And a software setting is very easy. Like, we have an issue here, and a bug fix here, and we're going to add a new feature here, and another feature that's independent.

26:46We just launch parallel jobs, you know, at the same time. And academia for academic tasks was sort of more difficult. But what I later learned was you can also do it in all kinds of tasks. And it looks a little bit different. You don't do it entirely in parallel. And how you can best think about it, and I think this is a very useful perspective, is if you want to be most productive, what you should do is, as an agent, operate as long as possible autonomously and reduce the time they just brought getting instructions. So if you do that, you can basically intentionally launch sort of problems and then come back basically to the beginning and the engineer will just finish and you can use instructions.

27:30So it's almost this parallel but sequential sort of loop. And that works well in any scenario. So now I do a lot of parallel things in this way in sort of like demo domain. And so it feels like now these ideas are merging and everything becomes the same. It's not only engineering, and it's now everywhere. And that's what it fears. Yeah, exactly. It's like the ability to apply that technique applies for everything now. Like the orchestrator pattern that you just described is something that engineers have a lot of luck and even like a lot of experience with a few months in at this point. Yeah, really since like November, top of the year is really when you started to see the orchestrator pattern start to take off.