The tool design tricks behind Benchling's AI agents | Nick Larus-Stone

20:36Nick Larus-Stone:You mentioned that you guys use a bunch of different models before. When someone kicks off a run in BenchLink AI, how many different models might it hit under that run? Depends what it's doing. I'd say like upper limit is probably seven or eight. I'd say most of the runs are maybe only one to two. We typically have like one main driver model for our kind of core harness, but it'll certainly call out to usually at least one other model. And then depending on the complexity of the task, maybe several other different models. I feel like a big part of Bio is running experiments and learning from that.

21:17Nick Larus-Stone:Do you also think of these agents as learning from what they're seeing their users do? Or are you thinking of memory for your agents in any way? We're, I'd say, still early here. One of the things we've recently released was this ability to have agent skills within the platform. And we think this is actually really powerful, primitive for scientists. Of course, we've added the ability for agents to be able to create and update those skills. So I would say that's like maybe the first, maybe simplest layer of memory is just, hey, you did a task. The next time I need to do this task, I want you to be able to do it in a similar way and hopefully do it much faster and with higher fidelity and better.

21:58Nick Larus-Stone:And are those skills, like will the user say that exactly what you said? Like, hey, remember this? Or is the agent kind of like, you know, seeing that the user did it three times in the background and doing thinking to itself that it should create its own skill? Yeah, so the agent actually has the ability to create its own skill. It's been tuned and we've prompted it such that it's pretty biased towards a user instantiating, though it could do that. We are also in the process of adding the ability for agents to run in the background. There, I think you could start to do a little bit more of what you're talking about of, hey, I've looked through your last 10 chats.

22:32I think, you know, I should create these three skills. We've been a little conservative there because we don't want to create a bunch of skills that are not helpful. So it has the capabilities, but it is kind of prompted to be pretty responsive to what a user wants.

22:46Nick Larus-Stone:And are these skills right now at a user level, at an org level, at some, the user can choose whether to share it? How do you guys think about that? We have a concept of personal skills so users can create them for themselves. And then, you know, since we have been around 14 years, we've built a lot of permissioning into our platform. So we have the ability for users to then share those skills with other users or teams organizationally. And then we also actually have, as Benchling, have been building skills and sharing them with kind of all of our users. so that one, we think they're kind of general purpose and useful.

23:23And we've done a bunch of work to, you know, learn from how people are using our agents. And two, they provide kind of an example for people who are maybe less familiar with skills of like, what should a skill look like? What could go into a skill? So yeah, we think the organizational piece doesn't seem to have been solved broadly as like a ecosystem, I would say, as part of having a platform where people are already sharing things and working with teammates. we think it's important to build that directly in.

23:51Nick Larus-Stone:You mentioned background agents and potentially using those for memory. Are you also thinking of using them for other things? We think of two types of background agents. One is kind of like schedule-based. You know, every week, give me a report because I have a discovery meeting. I think that's super useful. Scientists spend, again, a lot of time, you know, documenting things, building PowerPoints, building kind of like documents to share with others. If you can get a first draft out every week, that's going to be very powerful. The second one is event-based. So based on something that happens within the platform, can you trigger an agent to run?

24:26And I think that, again, is one of the advantages we have for being around for a lot of years. People can do a lot of things in bench link. And so we have a very rich kind of interface where lots of different things are happening. So you could imagine, for instance, if you're a scientist, you're running something in the lab, You have an instrument in your lab that actually takes a measurement. We've built over several years the ability to like take data from that machine, bring it to the cloud, right, and kind of automatically register it within BenchLink, register those results. You can imagine that automatically triggering an agent to do analysis.

25:04So by the time you get back from your lab to your laptop, you don't just have the kind of clean data. You actually have the analysis of like, hey, here's what happened in your experiment. And here's what I think you should do next. I think I'm kind of maybe most excited about some of those use cases where, you know, you as a scientist are already doing kind of your normal work and the agents really are in the background seamless, but just like getting you to the next stage that much faster.

25:30Nick Larus-Stone:You've mentioned a lot of really cool and advanced features that are like reminiscent of a lot of things in coding agents. Your users are scientists. Like, how are you adapting? Like, are they are they do they understand all these things? How are you thinking about the UX? Like, do they understand what skills are? Do they know how to think about skills? Do they know how to think about events, background agents? How are you thinking about the UX for these for these agents? Yeah, we spend a lot of time on user education, I would say. I think, you know, sometimes being in Silicon Valley, right, you get caught up in, you know, everything that's going on and you don't realize that, like, not everyone has maybe the time or the inclination to experiment with all the latest and greatest that's coming out.

26:11And so we spend a lot of time working with our users, showing them what they can do, listening to their problems, making sure we're building the right things there. So take skills, for example. When I go and talk to scientists, I'd say it's maybe like 50-50 of whether or not they've heard of skills at all. Right. So there's some people who like, yeah, I've heard of skills or maybe kind of my, you know, company has some enterprise coding agent or chat client and there's some concept of skills there. But rarely have they spent a lot of time building skills. And then there's, you know, half of people who just like have never kind of touched skills at all.

26:48And so it's a matter of us saying like, hey, this is this thing, right, this way of working. Luckily, I think there's actually like a good parallel in science, which is an SOP, a standard operating procedure. I'll sometimes describe skills as like SOPs for agents, right? You aren't guaranteed that a human following an SOP will do the right thing. But, you know, you write the SOPs with the goal of them kind of doing exactly those steps. And, you know, similar idea, I think, for skills. So we spend a lot of our time doing that translation, I would say. So do you guys have a forward deployed engineering team?

27:21Is that basically what this is? We might not call it that, but Benchling actually from the beginning has had a large service organization because science is really complex and it's really varied. So no two companies are really doing exactly the same science. And so even when we deploy kind of core Benchling, we spend a lot of time with the customer customizing it to their needs. And so we're, I'd say, like pretty familiar with doing that. And I think in the AI era, we've quickly seen that it's not just give people AI and they'll know what to do and they'll immediately be more productive. Sometimes that happens, but oftentimes it's a matter of, you know, deeply understanding their workflows, their bottlenecks and kind of showing how that can map into the tools that we're building.

28:06Nick Larus-Stone:It seems like a lot of the architecture is similar in a lot of ways to coding agents and how that's kind of like brought into this new domain. Have there been cases where you've seen something in a coding agent or coding harness and been like, that is wrong. That won't work for this domain. Like other things might, like skills, sure, great. Let's take those. But this thing, that doesn't transfer. Some of the rules or spec-based development kind of testing-driven doesn't work because you don't have a way to verify in silicon, right? So sometimes it feels like coding agents are really good when you can give them a defined end condition.

28:42Maybe like this goal setting idea, the Ralph loop that has been more popular recently. Your goal is like make a drug that is safe and efficacious in humans.

28:53Nick Larus-Stone:Unfortunately, you can't verify that in silicon, right? And so like that doesn't map super well to what we're doing here. I think there's certain tasks that maybe are more kind of, as you're saying, verifiable in silicon. But in general, I'd say like those types of patterns just don't map well because the speed of your loop is, again, like you physically have to go do the experiment in real life that's going to take days, weeks, months. I saw a tweet the other day. You might have retweeted it or something about how coding is the first domain that a lot of the model labs are kind of like building in.

29:27Nick Larus-Stone:And it seems like bio might be the second. If you know that tweet, like why do you think that might be the case? Two answers. One is those of us who work in the life sciences are kind of blessed with working on problems that are really important. Like human health is kind of like universally important to everyone. Food security, right? Like there's kind of like many positive externalities. And I think the model labs, you know, if they're building this kind of general intelligence, this super intelligence, you're going to have to like apply it to something that is really worthwhile. And I think like at the end of the day, to some extent, it all comes back to life sciences and kind of like, you know, improving human health.

30:14So I think there's that aspect of it. I think there's also an aspect of a lot of researchers come out of the life sciences. So like Dario did his postdoc in neuroscience. So like there is, I think, some kind of just like inherent interest in familiarity with that domain. So I think it's a combination of both of those things.

30:33Nick Larus-Stone:One of the things that you said before this to me that I thought was interesting was that working in life sciences is kind of similar to like working with agents in some way, like the mental aspect of it. Could you expand on that? Yeah, I think LLMs are funny, right? So, you know, in the history of software, we've been building kind of deterministic software, right? And I think there's a certain mindset that people who are good at writing code have. And we call ourselves software engineers, right? And so there's this engineering mindset that is very productive when you're writing code. People have been trying to engineer biology for a long time.

31:07That's actually how I first got interested in it. Biology just doesn't work that way. We are not good enough at understanding biology to make it do exactly what we want. And we spend a lot of time building assays to understand what's happening in the underlying biology. and we probe it and we perturb it. And when I see some of the research that's coming out of model labs around interpretability, it feels like they're actually doing biology research in some ways, right? They're kind of like poking the system that they don't fully understand and watching how it responds. And this is how biology works, right?

31:43Like scientists have been doing this in labs for over 100 years. And for me, it's kind of funny to think about actually sometimes the people who are maybe best at, you know, understanding LLMs or using LLMs are much more similar to scientists and potentially biologists than they would be to software engineers.

32:05Nick Larus-Stone:It's like this black box that you're kind of like probing and see what happens and then using that as experiments. Exactly, and many advantages in working in LLMs, right? Like in theory, we understand it all the way down, even though kind of in practice, there's still some lack of understanding. We understand biology far less, right? Like it's a much harder system. It takes much longer to do experiments on. But if you look at, you know, some of these large training runs, right, they take weeks or months, right? And then you only see what happens at the end, right? Like it's it feels more like biology than traditional software engineering ever has.

32:38Nick Larus-Stone:What's the profile of builder that you found most successful in building agents on your teams at BenchLink? Yeah, I think it's really product minded engineers. At the end of the day, right, like we're trying to solve problems for our users who are scientists and it just requires someone who's, you know, one, willing to talk to and listen to scientists about their problems and kind of deeply understand that. And then two, someone who's willing to kind of like work with LLMs, really deeply understand where they're strong, where they're weak, and then kind of merge those two things together. So the people who are best at that don't necessarily have a science background.

33:16I actually don't think you need a science background to be effective here. But I would say it's more of this like product engineer

33:21Nick Larus-Stone:archetype. When will agents discover a novel cure for a disease? That's a loaded question. In the biotech industry, there's lots of discussions about what is an AI discovered drug. I would say right now, I think we're in an era of agents making it cheaper, faster to discover drugs. We think actually agents will be able to speed up the time from initial discovery to bringing a drug to patients by 2x. How? Why? There's a lot of white space when you're doing drug discovery. So take the example I talked about earlier where maybe you're running an experiment in the lab and your results are on that machine.

34:09Maybe historically, like you would actually take a flash drive, plug it into the computer, right? You'd copy the file over. You'd bring that back to your laptop, you'd email it to a statistician or a bioinformatician for them to do the analysis. They might take a day or two, email you back. And maybe that's actually like, sounds like it's only a day or two, but maybe like, you know, the next time you can get into your sequencing core, that window has already passed, right? And so like now you're actually a week behind. And so like a little thing like having the data ready for you to analyze and knowing what experiment to do next when you get back to your laptop doesn't just save a day, it actually saves a week.

34:46And we think there's like lots of those little things that just add up over time. So that's one aspect. The second aspect is I think actually like agents can make you better at doing science and like do fewer experiments over time. So one example that we've actually seen some of our customers do is using Benchling AI to help them with DOE, design of experiments. So taking a more structured approach to designing their kind of next set of experiments rather than maybe just tweaking kind of like one experimental parameter at a time. You do like a factorial approach of, OK, I'm going to change all of these parameters, but I'm going to do it in a structured way that I get statistical power.

35:26And so I can run 12 experiments instead of running six experiments, then changing something else, running another six, running another six. And I ran 24 in the end. Right. So I think between those two things, we'll get some benefits to your original question. When will like agents discover a new drug? I think we're still years out from that just because, you know, I think the underlying technology of the models are like not good enough at understanding biology right now. and like kind of our kind of overall knowledge base of biology is weak enough that they'll be helpful, but you're not going to have like a solely agent discovered drug in the sense of find me a cure to cancer for quite some time.

36:10Nick Larus-Stone:And so is the blocker that the models aren't good enough or we're still building kind of like the systems around the models to let them run experiments? Because I imagine even if I like if I ask a human, find me a cure for cancer, like they're going to run a bunch of experiments and that's the system around. And so like, how would you, yeah, how would you break it down between the model versus the system? Yeah, that's a great question. Both. We see a lot of cases where the models lack some understanding of both the physical world and kind of the underlying science. So maybe one example of this is, I'll go back to automation here.

36:46If you're a scientist in a lab and you've designed this experiment, typically the way you do this is you then go run the experiment yourself, right? You'll kind of pipette, you'll move liquids around kind of by hand. There's a whole class of technology called lab automation where you have robots do this for you. But the translation from an experiment that you do by hand to lab automation is really hard. There's a whole career called lab automation engineers who their job is to basically do that. And there's a lot of complexity in that. So in theory, that is actually just a code writing task, right?

37:21Like you could just program these robots. In practice, understanding that this specific robot has kind of this tolerance to liquids and actually, you know, the liquid class that it needs is going to be slightly different than what you did by hand. It's very hard. And I don't think that the models are very good at that because there's not a lot of public training data on this. That's kind of like a class where we don't think LLMs are that good. Overall, kind of like experimental design, I think they're still getting better. The models need to get better. The second piece is like, yeah, like even if the models get better, I find it very unlikely that they'll just be able to think really hard and give us kind of an answer.

37:59They're still going to have to kind of interface with the real world and generate data because we haven't generated enough data to understand all of the science yet. And so I think that actually, unfortunately, will move slower. It's very hard to build kind of robust, multi-purpose lab automation. I think you're going to have humans and agents working kind of in concert to push that forward for quite some time.

38:21Nick Larus-Stone:As you think about that future where models and agents work with humans to do longer and longer horizon things, what is the right UX for that end up looking like? Yeah, I think this is, again, kind of shaped by our opinion as a 14-year-old company where scientists have been doing science for the last 14 years. Sometimes what is good for a human is good for an agent. Sometimes it's not. But one example of that is kind of a core part of our product is the lab notebook. So scientists record what they're going to do and then kind of fill in what they actually do in the lab. And that actually turns out to be a reasonable abstraction for a scientific agent as well, right?

39:06You want it to have a surface where it can say, like, here's what I'm going to do, have other people look at it. And then, hey, here's what I actually did. And we built out a whole review process on that. And so I think sometimes actually like those concepts do translate, which is nice. Sometimes they don't. And this is where I think, you know, having agents that can generate code, we've seen, you know, agents be able to build kind of dashboards, HTML dashboards kind of like in the app and have custom interfaces, be useful for certain things. But at the end of the day, you don't want that for everything.

39:39You don't want your agent kind of vibe coding your structure prediction viewer or your plasmid editor. We think it's going to be some combination of, hey, there are things that are useful for your humans to record. Some of those will work for agents. Sometimes you'll want something more vibe coded that an agent can interface with. But in my experience, most humans don't want something that changes on them day to day, week to week. And so having at least some continuity is quite important.

40:08Nick Larus-Stone:One of the things that we've added into LinkSmithFleet, which is our no-code thing, is the concept of like an agent inbox, where you've got agents running the background, doing things, but then you still want the human to approve certain actions or certain steps. Have you guys thought about like a similar like notification center or things like that? Yeah, we're actively doing that right now because we think, again, both these background agents will be really important, but still having that like human in the loop and ability to approve things and monitor what the agents are doing is crucial, especially in our kind of like both like regulated and non-regulated domains.

40:43Nick Larus-Stone:One of the things I feel like we've seen in software is that everyone's becoming like a manager or reviewer of these agents. And on our software engineering team, we've seen that some engineers are getting like review fatigue. They don't want to be reviewing code all the time. They want to actually be building an engineering. Have you seen the same thing happen with scientists or do you think the same thing will happen where like, hey, I'm now a manager. I have to review all these experiments. I don't get to think about what experiments to run. And that takes some of the fun out of it. I don't think we've seen that yet, in part because I think the agents that are really good for scientists are still behind the coding agents that are really good for software engineers.

41:16We sometimes think of it as like roughly a year behind. So if you can imagine we're a year ago in the software industry, I'd say that's actually kind of a good analogy of where these scientific agents are. But the other thing I would say is actually, I think most scientists really like the part of their job where they're thinking about hard scientific problems. You know, they're thinking about what experiments to run in that design and like whether or not this is the right experiment rather than the kind of like ticky tack, like I need to fill in the data from this PDF or I need to like write some Python code, right?

41:53Like there's like a different part of the job that people get enjoyment from. And so I think that actually lends itself much better to being more of a reviewer here, kind of a senior scientist, right, where you're giving feedback to your RAs. You can kind of think of the agents like that today. So I don't think we've actually seen that type of fatigue yet.

42:14Nick Larus-Stone:BenchLam AI sounds like it's doing a lot of things. How do you guys think about charging for the work that it's done? Yeah, so we have a usage-based model. so we charge based on the amount of kind of usage of Benchling AI and then we give a monthly allotment of free credits to each customer based on the number of seats that they have with us. We think this is actually like a good model for a lot of these scientific organizations because you might need to like spike up on like your credit usage but like maybe most of the time you have a baseline level that you can kind of, you know, buy credits for, use free credits for.

42:55But then let's say you're writing an IND, you're going to use a lot of credits to write all the reports, right, and get that done. So you can kind of like spike up rather than having kind of any sort of like flat fee or seat-based model.

43:06Nick Larus-Stone:Have you seen that people care about the underlying models at all? Like, will they say, hey, I need to use Opus for this task because I'm spiking on a really hard task and this is like data entry on something that I think is easier. so let's use a cheaper model for that. We sometimes get that question. We don't actually expose the models. I would say like we see that as part of our job is to make sure you don't have to think about that where we will do the work on like, hey, these types of tasks, we think this model is sufficiently intelligent enough and we'll build evals around that and we'll monitor production traffic to make sure that that's true.

43:41But I think like unlike software engineers where they're a little more willing to experiment and they, you know, follow every model release and kind of have this intuition of what kind of different models are good at. We don't think our users should have to care about any of that. Like that is actually our job to make sure that it works.

43:57Nick Larus-Stone:And how much of Benchlane AI is like a single agent versus like under the hood, it's actually like one agent here, one agent here, a different, maybe just like an LLM call here, but it's all packaged up as Benchlane AI. We initially launched kind of multiple different agents and have been combining them into a single interface because there was at least some confusion with users about like which agent should I use for which task. So we think there's a lot of value in having like one interface that a user can go to. Under the hood, we are still mostly one agent. We do have sub-agents, right, for kind of certain tasks, not even sub-agents, but like sub-workflows, which might be kind of a series of LLM calls.

44:38We've been doing a lot of work to kind of unify it as much as possible. though there are some tasks again in some of these like harder scientific tasks where we've been doing more experimentation of okay maybe actually a different harness is the right solution here and so like there may need to be kind of like multiple agents under the hood.

44:58Nick Larus-Stone:Do you have any hot takes on like what the future of harnesses looks like here or like what's the most like are you experimenting with things that you haven't seen anywhere else because it is so different from coding or things like that? I don't know if there's anything that hasn't ever been done. I'd say we're often inspired by seeing what other people do and then just trying to translate it to our domain. I think actually the kind of like overall driving force behind this is we think our tests are often intelligence limited, which is not true in other domains. A lot of this we think, you know, smarter models will actually get better at science.

45:35And so to some extent, it's a question of like, how can you spend more tokens fast enough? but also like reasonably enough in order to get to better answers. So I would say like a lot of the current thinking and experimentation is around like, how can you spend a lot more tokens because we think it's worth it for some of these harder scientific tests?

46:00Nick Larus-Stone:And does that mainly look like kicking multiple model families off in parallel as we discussed earlier? Yeah, that's kind of where we are today. But I think there's more creative things that we could do here. you know some combination of background agents in that or you know potentially like entirely different model harnesses that we haven't even thought of yet so for example you could imagine building a harness to to really focus on like novelty here so some people say like lms can't do anything novel right their next token predictors i think people just like haven't experimented enough with this they haven't cared enough about kind of novelty right like a lot of software engineering tasks are well scoped, right?

46:42But in our case, actually being able to like make connections between different parts of kind of your research data as an organization and come up with new ideas is really valuable. And so you could imagine kind of building something, some combination of this kind of novel harness where you're, you know, spawning a bunch of different ideas and kind of like cross comparing them, kind of combine that with some of the background agent work that we're doing, where you're just kind of like looking at kind of new science that is getting done as an organization to come up with novel ideas. And then actually, you know, as a scientist, your day changes from, hey, here's the, you know, list of tasks that I have to do to here are some ideas that are kind of proactively suggested to me by my system, because it understands the totality of my organization's science.

47:31I'd say like, we talk a lot about building an AI scientist as kind of the overall goal here. We think a lot of the interesting work there is actually like getting it to work in the lab, right? Some of these like harder physical challenges. But I think part of it also is being able to kind of understand the totality of your organization science and kind of proactively suggest ideas

47:53Nick Larus-Stone:to you as well. And is that the main part that's like intelligence kind of like limited, as you said earlier? Yeah, I don't think like data entry is necessarily intelligence limited, but some of the understanding of the physical world and the kind of what should you do next, right, is kind of the core question here. I think a lot of that is actually intelligence limited. Bio is a very specific domain. And if you guys are intelligence limited, have you have you guys considered kind of like training your own models or post training as a way of putting more of this like very specific intelligence into the model weights themselves?

48:24So our customers are very security and privacy conscious. So we would not kind of like cross train like across customers because leaking any data between customers is big no no. So anything that could lead to that is something we wouldn't do. In terms of like kind of fine tuning open source models or anything like that, I don't think we've seen a lot of success with that. There have been a number of people who have, you know, tried fine tuning models on this domain. In general, they don't appear to be much better than kind of state-of-the-art frontier models. And so then, you know, I think you lose out on a lot of the other model capabilities, and that's often not worth it.

49:06It's not to say we won't ever do this, but right now I think that there is benefit in just having the smartest model that there possibly is. The other aspect of this is like science is made up of like a thousand little tasks, right? And so if you get something that's really good at kind of a specific task, that is only one of the other 999 things you need to do. So I could imagine like having kind of a core frontier model as the main driver and then kind of farming out to task specific models. But it's not something we've seen a need for yet. It doesn't mean it won't happen.

49:38Nick Larus-Stone:I feel like we've started to see more of it recently, but I feel like the main difference is people are doing it for cost reasons because they're not intelligence constrained. So they've got something working and now the volume is going up and up and up and they're just eating all these costs. And so they're trying to drive costs down. But you guys are actually interestingly in a completely different boat, where at least for some tasks, you're intelligence constrained. Yeah. And I think honestly, like it's not a solved problem on like how to make these models smarter at biology because it's not a very verifiable domain.

50:05Right. And so like, you know, there's a lot of work on doing RL for specific bio problems. And I think we will see the models get smarter there. It's not quite as straightforward as some other tasks that you could imagine kind of RLing a model for.

50:18Nick Larus-Stone:Thanks for listening to Max Agency. If you liked this episode, leave a review and subscribe. Send feedback or questions to maxagency at langchain.dev. We want to hear from you.