The Agent Harness: Building Secure Sandboxes for Autonomous AI Workloads

32:14And one is awareness. Do they even know you exist? Like, if that doesn't happen, there's no way they can pick you. Two is preference, which is, you know, the pricing, the brand, the person, the different features, whatever, like it's preference. And the third thing is like, is there a deterministic thing that you offer that no one else has? So the easy example for the third one is like, do you have FedRAMP, right? Do you have that certificate? If you have like, I can only, the customer can only use you and no one else. but the two above are quite interesting, which is like how to make sure that everyone knows who you are.

32:49And then you work on the, um, then you work on the preference. And so for me, the all, I don't, it's not all I think about a large part of what I think about is if all things were equal. So if our product is equal to everyone else's product, what is the differentiator to that? And so like one, can more people know about you than others? Like is the branding there? Is the, um, the, um, the experience there, experience is a nuance, which some people don't think about, some people do think about, but it's like, do you prefer the feel of this product versus that product? And these are all things that have nothing to do with the actual technical capabilities of the product.

33:29And so my co-founder, Vedran, our CTO, his job is mostly to make sure the product is better than anything else in the market. And my mandate is to make sure that even if our product is equal, that we can supersede that. And so that's how I think about the go-to-market in general. Do you think of customer support and customer service as... All of that. It's all go-to-market. All that is go-to-market. And we've seen this, we've chatted about this, where we've been getting users and customers just because we are so good at that. And so all of it is an experience. So if you think of any experience as a human, you go to like store, restaurant, whatever, it is the entire experience.

34:12It is from your, the way, what is a brand? It's the perception of that brand itself. It's just a perception. And the perception is like, if you go into whatever store, pick your brand you want, the smell, the music, the people, the smile, that whatever you get, all that together is the perception of that brand. And so if you think of that through the lens of a product, which might sound counterintuitive, I don't know, or non-obvious to people. Like I think about that altogether. Let's say we were selling sneakers, right? Like we don't sell sneakers, but the, the, the sneaker, which we won't, man, we won't say other brands, but if you do, you can always pivot to GPU.

34:52Yeah, exactly. Like if you go to the store here in New York, it's like a beautiful store. The people are very nice. Every, everything's aesthetically pleasing and you just like enjoy that entire experience. And it's a good sneaker. Right. And so you have to have that all together. And so that's how I think about this as well, which is, you know, you have to have a good product, you have to have all these things, but all these other things have to co-alide with that. Now, the risk is, and we've seen this with a bunch of startups, like there's no sustenance. So the product is not good and you have everything else.

35:21And that is sort of, that's when you get into trouble. But if the product is good or at least as good, if it's better, it's amazing. And you have all these things. For me, my belief is that that is a key driver to continue growth. great uh you mentioned twitter and x uh a minute ago to which extent is that part of the whole uh the whole it's hard to measure yeah like i've worked in bigger companies and if someone if i was working for my former company they're like how do we measure that i have no idea they can measure that but what i can say is that i was never active on twitter very i have an account since 2009 but never very active until like this holiday season where there was like this semi-viral tweet that went out.

36:04What did you say again? So the tweet was, if you're taking a break these holidays, you're NGMI. You're not going to make it. Yeah. Which I honestly believed. Yeah. Like the tweet was basically, I wasn't even thinking about, there's like a typo in the tweet. Like it's not, people are like, you're rage baiting. It's like, no, I was, my thought is we are a part of this like super cycle right now. And the super cycle does not last forever. And so if you're going to pause by the super cycle, you are seeding market. Like that is what you're doing, right? And also it's like, it's very much towards, you know, founders and executives and tech leaders.

36:39It's less about the individual person that can't have an impact on all. In that, like people have to take their breaks. And so people took that in all different ways. They're like two poops, two poops. Like you should die, you're a capitalist, you're whatever, all these things. And the reason why it became so popular, like it was kind of popular, which was some person said, dude you never i've never heard of you or whatever and then i said basically that's why yeah to keep working and that was perfect perfect and now again not i was like instant reply on that and so basically not talk too much about that is that from then i had noticed that people even if they like your tweets or don't or are completely opposed or not has no bearing on i should say it has that positive bearing on their thoughts of your company.

37:32So we have found that people that even are completely opposed to the posts end up being customers, assuming that they need the product line, not everyone is there. So it was quite interesting to me that, oh, just because someone doesn't agree does not mean that they won't like that because it's all generally awareness. and so that has been something that I've spent a bunch of time trying to catch up to you on the Twitterverse. That's super interesting for any AI builder or AI founder. So the PLG motion ultimately to create inbound is a combination of all the things. So we talk about conferences and meetups.

38:15Twitter is important. Is there anything else that people should know? We talk about customer service. You mentioned no salespeople as of now. We have no salespeople now. Yeah. So I think like the, how do you experience the product? Right. So like once you've seen the product, how do you experience the product? It's like your door to the product, like the website, the login, the whatever. We can fix a lot of these things to be very clear. I'm not saying we're the best at least, but there's that. And then what are the feature sets are in there? Can I get things easily? Our SDKs are really, really, really good.

38:45Like people really like, like the ergonomics of them. So it's like really good. That part is great. And then the thing that, and this is even public on Twitter, all our case studies that we've done, and we outsource the case studies to third parties, so we're not part of this, is that we reply very, very fast. And so this is a very public thing, and it's something just core to who I am and how I learned to work. Because one of the, let's call it first real jobs I had was a system admin. So my job was like to fix the printer and computer and whatever. And so one thing that I learned, and I try to talk to my entire team is like, one thing you have to do is like, one is the first response very fast.

39:24Just the first response very fast. Like people are then calm. They know they have transferred their problem to someone else and someone has acknowledged that. And so when they do that, they feel rested, right? And then you just have to promise that you will solve or get back to them at X amount of time, whatever that is. And they were calm till that moment. And the thing that you have to do is one, either solve the problem until the given time or call them, message them, whatever, and track them before that time and state a new time. That is the solution to support. That is it. There's nothing more than that.

40:00There's obviously, there's like key things if you're actually down and don't work and the person can't work, that's a completely different thing. But every other non-critical problem and you have the most happy customers in the world because they don't have to think about their problem anymore. You think about their problem. And so you can like keep that on until you get it done. But the key part is if I said that something's going to be fixed or deployed or done or whatever in two days, a day and a half later, if it's not going to be in two days, a day and a half later, I said, hey, this is going to be delayed two more days.

40:32And they're fine. You thought about it before they thought about it. And it's not magical. I mean, it's a magical experience, but it's not very, it's not a complex thing. but I found that people don't understand that intuitively. And so that is what I learned back in the day. And that is what we do in the company today. And that's why we have very, very happy users and customers. Okay, fascinating. Now let's switch back to the more technical stuff. So we talked about the concept of sandboxes. Just walk us through what a sandbox is technically. Yeah, there's a lot of things. There's a lot of things there we can talk about.

41:08The way I think about Sandbox is the ergonomics of consumption of compute. And so, because there's a lot of different layers on that. I basically put it into three different layers, which is the infrastructure, the primitive, and the tooling. Those three things together. And what I mean by infrastructure is like, does it spin up very, very fast? So like we spin up in 60 milliseconds. Can you spin up a lot of them at once? So the rate of creating them. And so we can spin up 50 ,000 in 70 seconds. So a little less than a minute and a half. And then once they're up, how many can you keep running?

41:48Like we can have, we have customers that have billions a day. So those are like the infrastructure parts that are non-trivial. Why does it matter how quickly you can initialize a new sandbox and how many you can run? Again, it depends on the user and the use case. So if you're like a long running background agent, again, everyone prefers it to be fast. Like no one wants to wait. Like you don't want to wait for a reply. Everyone wants to be fast to be very clear. So the faster, the better. But generally there's an actual reason why you want it very, very, very fast. And that is especially if you so for I was gonna say for a background agent, a background agent might work for like 10 minutes or an hour or whatever.

42:23So the incremental millisecond might not matter. but I still believe that from a user perspective, even a second of waiting is kind of uncomfortable. You don't want that. And so it's 60 or 90, maybe less so, but there you want that one, two seconds for sure under that. But the more interesting part where that is really, really important is for the researchers, where when you're doing reinforcement learning, you basically have a lot of GPUs and the GPU is more expensive than the CPU. So the vast majority of sandboxes are CPU boxes, to be clear. So they're the computers that we all work on. There might be a graphics card in there, but basically it's the RAM, the CPU, and the hard disk that's in there.

43:02And they are cheaper or less expensive and easier to get, at least for now, we'll see for how long that lasts, than GPUs. Which means you want your GPUs always to be at maximum utilization and the CPUs can then idle if need to be idled, but you don't want the GPUs to idle. And so what that means is you want to make sure that the CPU machines, the sandboxes, spin up so fast because you don't spin them in a training run you don't spin them all up you spin them on like depending on how many you have it's like a thousand at once they do the task then do the next one the next one next one and so between turning off and on the cpus you want that time to be as short as it can so that the gpu utilization does not go down right and so that's why that's very very very important and obviously the number that you have concurrent so depends on so on the background agents, let's say just for the sake of people know lovable.

43:56So the amount of users that they have is astonishing. And so each user for every task, and they can have multiple tasks, multiple agents, they need a sandbox. And so imagine, I don't know their user number in the millions, whatever. So you have to have millions of sandboxes up and running. On the RL side, you can have people, smaller labs that will have concurrent 5 ,000 or 10 ,000, But we have a request today for 5 million concurrent looks. So 5 million at one point in time, right? So being able to handle those types of things is part of that infrastructure segment. The other two are the primitive and the tooling.

44:31And so the primitive is essentially, let's call it the computer itself, the isolation, the VM, the container, the micro VM, the isolate, the whatever. We can get into details what those are, but there's different shapes of them. And also what feature set do these things have, right? And so for the most part, a VM that you would find in, you know, AWS or whatnot is usually much slower to start, much harder to get those spikes. But also when you get a certain size, they're usually that size forever and forever. And then we just give one example, whereas in Daytona, you can define a size, the CPU, the RAM, the disk, and then while it's running, you can resize that.

45:15So if an agent gets to, you know, use the entire memory or 100 % CPU, the sandbox would, or VM would die. In our case, we can expand that. And there's other things. It's not, for us, it's not just a Linux CPU box. It can be a Windows, a Mac, an Android. It can be, you know, they can have a GPU in there. It can not have a GPU depending on what you need. And so that's the primitive. And the last thing is the tooling. The tooling is, the tooling is either there to support the agent, to be better at its job, or to be guardrails on the agent to stop it from doing dumb things. stuff basically so you can think of it we have a bunch of tools so headless you know terminal and file operations and other things that enable it to use less tokens get the job done faster but also guardrails like secrets manager and a firewall and all these other things and so the combination of those three the infrastructure primitive and tooling essentially creates what we call a sandbox so what's ultimately what's the difference between a sandbox or a container VMware, VM, micro VM.

46:16All the things, all the joy. So most sandbox providers are firecracker VM, micro VMs, most. Let me remind people what firecracker is. Firecracker, it's an isolation privilege. So it is essentially a type of VM, a virtual machine, that's very stripped down. It was made by the AWS team. It's used inside of Lambda. And so the idea, which are like functions, and so the idea was to have something very very fast that can spin up and stateless that that was the entire time and they were very very ephemeral and so they were used for you know when your website has a large if it's black friday a bunch of people hitting your website you can spin up a lot of these very very fast like handle the load and enable all these humans to to look at the website and buy whatever they were buying right and so that technology is there and it's a very stripped down version of a full on VM.

47:14So it has less features there, but the things that it does, it does very, very, very, very well. So you can spin them up fast. It does that because it can sort of lazy load things into memory. You can do point in time snapshots. You can do all these nice things. The things that I can't do is it can't run a, let's call it sandbox now. It can't run a sandbox with a GPU. It just doesn't work. So you can't have a firecracker. So if you want a GPU, you have to do something which is a cloud hypervisor or a keymu spelled Q-E-M-U, which are two different types. Keymu is almost a full VM. So it has all these different things inside of there.

47:49So you can't do that. But let's, for example, if you need to run, we were solving this for one customer earlier today is they need a machine that also has an Android device inside of it. And so the only way we could solve that was inside of one of these key moves. Like it can't work in a firecracker, it can't work in a container, it can't work in anything else. And so there's different types of there. These are like the micro VMs of the world. There's also containers. And so container, most people know Docker, which is there. Daytona originally started running Docker containers. We do have them.

48:18The problem with Docker is that they're much less secure than a VM. The thing that we do in Daytona is we harden that with Sysbox, which is VM-like isolation that wraps around that. It's really good for handling density of these machines. It's very fast. It allows us to run Docker and Docker. There's a bunch of features that it's very, very good for. And so there's differences there. There's also like isolates, which we've talked about, or like other app containers, which is abstractions of these containers, just you're running a single app. And so these different things all exist in the world and they can all be theoretically sandboxes with these other two things that are there.

49:06And our original idea is to get back the original conversation, most sandbox providers started as firecrackers. But as we start seeing that agents have different needs in different use cases, they are going to need all these different shapes consumed through one interface. And so we at Daytona now support containers and all the micro VMs, depending on your use case. The user doesn't know the same ergonomics. It's like, oh, I need a Windows, it'll spin up this one. I need a Linux, I'll spin up that one. And so we will continue to add these things inside of the same infrastructure. Sometimes it'll be slightly faster or slower, but the concurrency will be there.

49:47All the tooling will be there. And so everything that you would come to know and like about a Daytona will be there, but it'll be different sizes and speeds and configurations depending on your use case. Why are snapshots and orcs important? There's a lot of reasons. Let's get into the commercial one, which is probably the one that people most think about, which is pausing a sandbox. So if you are an app layer company and you have 10 million users and your 10 million users spin up, let's just say 10 million sandboxes, just to make the math easier, these things cost, right? Because they're running, they're using CPU, RAM, and hard disk the entire time.

50:26The most expensive thing is the CPU. Second, the RAM, disk is almost free. It's very inexpensive. And the user sends the agent to do something. the agent does something and now it's waiting for a reply it could be waiting for the human or it could be waiting from a service depending on what it's trying to do you ideally don't want that sandbox to run idle because you are now regardless if you're a customer Daytona or running your own like there is a cost to running these things and so what you want to do is pause that and wait for a reply either from a service or for a human and then resume and the feeling and experience is that it never turned off so you feel like it never turned off but it actually did turn off.

51:08And so that one is from a compute management perspective, if not from a cost perspective is probably the first reason. The second thing is you can enable your agent to take multiple paths. So your agent can say, oh, at this point in time, I'll take a snapshot and then I will either continue and be able to roll back to this point, like a point in time, or at this point in time, I will replicate the sandbox and have two sandboxes and I can try two at the same I mean, obviously it can be more, but right. But those are the reasons why you would have that there. You mentioned performance and speed.

51:43And in connection with that, you said that you had to rebuild your own scheduler. Yeah. So what is it? First of all, what does a scheduler do? And then how did you go about it? Every cloud that exists today, NeoCloud, Hyperscale, or whatever, they are all built on servers, right? Like metal machines. And like historically way back in the day, we actually stacked these data centers, these data centers, me and my co-founder a long, long time ago, but they're all just like servers, like CPU, RAM, disk, they're computers, basically. And on top of these computers, there is a software stack that everyone has built for their own reasons.

52:19So AWS has their software stack. Cloudflare has theirs, which is very different. Everyone has their own on top of that. And so basically what you're trying to do when you think of these machines, these servers, basically what we do is we cut them up into small little machines and then give you that sandbox. And so you have these small, on these big servers, you have these small little machines that can run for a minute, a second, three hours, whatever, and you don't have to worry about this. And so basically the scheduler or the orchestrator is the one that basically says, oh, after you send me a request, I send you to this server and turn on that sandbox, a little computer there, and then I turn it off or I kill it or I snapshot it.

53:01It's the management of all these things there. And most of our other companies in the space basically have off the shelf schedulers. So it can be like Kubernetes or Nomad or whatever. And when we decided to build Daytona, the sandbox product, we inadvertently with very naivete were like, oh, this stuff doesn't work because we had worked with all we had built our own. Then we have used Kubernetes. We're like, we've seen all these different things and we knew what was good and was bad or what was use case or not. none of these were made for these like super fast, stateful, long running machines.

53:35And so we're like, we have to do this again. We have to build it again because the way we thought about it, which was not known at the time is, because at the time sandboxes were very ephemeral, similar to a Lambda function. And we're like, no, why would your sandbox be ephemeral by default? Like your laptop, you don't want it to die. Like you want it to work until it's done and it has to be very fast. It has all these things. and so my co-founder mostly built this sort of initial version of our scheduler and that scheduler of ours is the basis of everything that we do and it gives us a lot of things like outside of just like the performance it also enables us to do things like have four different isolation provide like we are the only company right now that you as a user don't know but like we have depending on your use case we will spin up any one of these firecracker cloud hypervisor whatever to get the job done.

54:27So whatever is needed, we'll get there. The other thing, the benefit is we built this, we have co-location providers, so we have like bare metal machines. So you can think of us as our own cloud, but we're more and more akin to now, like the NeoClouds, like the base tens and like the fireworks, because for us, and we've got to this question yesterday because we have a large request of number of CPUs, is like, we can't get enough. And so how do we solve that problem? Well, we've already solved it. where our scheduler can be attached to any CPU machine, any server that we think is good enough, and it becomes part of our cloud, essentially.

55:05And so why I say Base 10 and Fireworks is that both of them run on more than a dozen compute providers, depending on what you're going to call them. And so we can do something similar because we've created this in such a fashion. So that has given us, I believe that has given us sort of an advantage there. and is there a fundamental technical difference between ephemeral and long running so if you have a an agent that runs for 24 hours how does that translate in terms of sandbox requirements the reason most sandbox environments do not run forever or even it is because it's a technical problem if you think about servers underneath these servers also have to be managed right and maintained and so if your sandbox can run forever that means that you can never reset you can never reboot the underlying server you can't update it you can't patch it you can't do all these things without turning off all the sandboxes and so the way you solve that the easiest way to solve that is having sandboxes that have a termination time it's like they will only last an hour 24 it doesn't matter pick your time and then if you decide that you have to do something with the underlying machine, you just flag that machine as non-schedulable.

56:22And so at some point in time, there's nothing else on that machine. You can do whatever you can fix it. You can like reboot it. You can do whatever you want. Easy peasy done. Right. And so because historically, most of the workloads were fomeral, you didn't have to try to solve that problem because you didn't care. Like most workloads, like a Lambda function usually runs what? Five minutes, 10 minutes, like whatever. It's not a problem, right? You never had that restraint or constraint. now that you do, to have something that can run forever, you have to be able to live migrate the sandboxes itself between the machines so that you can reboot and manage these machines.

56:58And I understand that most people, it's interesting, even technical consumers are like, oh, all computers elastic. Well, yes, your consumption of it from us or from AWS is elastic, but someone actually has to like screw together a server, turn it on, boot it, make sure it works, right? And have enough of that there. so that is the difference from from our perspective and why most start off ephemeral is just it's an easier way to do it we believe and we've seen this i think two and a half percent of our revenue now comes from or two and a half percent of our sandboxes run longer than 24 hours but it's a large percent it's like 20 percent of revenue so like it's not an insignificant part to be there just because it can run for a much, much, much longer time.

57:44On the user perspective, just to finish on that, sometimes users actually want it to be ephemeral. One is they don't want any data retention. So they want it to run and die. Like whatever was run inside of that, they got the data they wanted and they want that data dead, dead. And so they flag it with us as ephemeral, which means that we won't force it to die. But when they kill it, it's no longer it goes into the either. It does not exist anymore. Listening to this whole technical deep dive part of the conversation, the thought crosses my mind that a lot of people seem to think that they can create their own sandbox and it is not that hard.

58:22But again, listening to all of this, I'm like, good luck. We've seen this multiple times. And I'm sure you've seen this across other companies as well, where, you know, everyone's like, oh, I can spin up a firecracker on a machine. Yes, you can spin up one, but spin up a million is a problem. Adding all these features is a problem. Having throughputs, different use cases, they're all different problems. And what we already see is like companies that we're like, oh, we'll do ourselves, come back three months later, six months later, eight months later. It's like, oh, actually, actually now we can't do X and Y.

58:52And so there's a bunch of things on technical side, which we can like address. And I think we can finish with that is that the thing that Daytona does is we, if you think of most sandboxes, this is a performance thing, Most sandbox and most VMs actually, they use the CPU RAM from the server, and then they use the hard disk from a network drive. You do that for a bunch of reasons. Your life is so much easier if you manage that. Why is it easier? Well, it's easier because if a server dies, then all the data is on a shared drive, and I can reboot that server really fast without maybe even the user noticing.

59:33And there's no data loss. maybe something in the memory, but nothing there is lost. So it's very, very easy to manage. Whereas what we do is we actually use the CPU, RAM, and hard disk of the machine, which means we have so much more overhead of if that machine dies, how can we reboot it back? You have to do backups that users don't see. You have to do all these things that happen there. But what that means is that the IOPS, IOPS, the speed with which you can move data to the hard disk in which the CPU and RAM interacts with. From a network drive, just to give people numbers, is in the hundreds of thousands.

1:00:15If it's a local drive, it's in the tens of millions. So it doesn't matter if you know what the numbers are. It's just extraordinarily large. It's very, very different. And so you have use cases that no one really cares about that speed, which is fine. We have new customers that have a use case. They're like, it has to be that fast. Right. And so when you get to the point, it's like, oh, I can use this, but wait, now I need this performance. Oh, now I can't do that. Right. And so to the complexity of sandboxes, if when people are starting out, their use cases are like, oh, I just need to run code.

1:00:49You can run that on anything. It's very easy to do, except scale. and then when you get to scale and then you need performance because now, oh, agents can do a lot of things, then it starts like breaking that sort of mold. And security. Yeah, because we didn't even touch the security part, but yeah. Great. Zooming out as we close this conversation, you mentioned something that caught my attention. So we have been in a very well-documented GPU shortage, but it seems that we might be heading towards the CPU shortage. Yeah, it might be happening. So one of your former guests and was at our conference, Dylan Patel, like sent me an analysis.

1:01:27They wrote the report. And I believe it's somewhere like October. We have no more CPUs. And people hadn't thought about it. Like it's all GPUs, all GPU intensive, then memory. But basically now, now that agents need all these computers, like it is very much needed. now that RL is the way that we've gotten most of the progress in models over the last 18, 20 months, you need all the CPU machines to do this. Now that the GPUs are more powerful, they can do in parallel more of these CPUs, CPU boxes. And then you see this, there was a tweet that I put a while back and is like, you should definitely invest, not financial advice, you should invest in Intel and look at their stock price, right?

1:02:11And so like, there's definitely, need for that. And it's something that people didn't understand or intuitively that you would need there. So it's something that we think about quite a bit is like, can we, alluding to also how our product is created, how do we make sure that we can match the demand of customers for the CPU that they have? Because I don't know it goes to the extreme to where GPUs are, because that is very, very, very extreme. but it is quite high probability that there will be shortages of CPUs going forward. How do you think all this agent stack evolves? Does it feel like we have all the core pieces together or do you think like in two years the overall architecture may look very different?

1:03:00So we talked about the stack there and things that have to be done. The identity part does need to be solved. Again, it's basically solved, but not solved. So there's a lot of things to do there. I'm actually also wary, wary. I think about people think that the models that we have, the technology for the models that we have is the end state. And I really, really don't think that is true. Like I think, and that would probably be the most surprising thing is like we have a new type of model that is just better or different than what we have now. that either one needs less compute or doesn't use GPUs or doesn't, like, I don't know.

1:03:43There's things that we've seen where, you know, we've seen things with wetware where people can have like fake human brains and it learns how to play Doom. You have the, what was it? The fly brain that was replicated inside of a computer that then works like a fly. Like, can you replicate a human brain? And I say this not to replicate myself, but can you make a generic human brain? And then it's like an, it's an AI that is essentially equal to a human AI. And so what does that mean for like all technology? I mean, I'm saying, I'm now saying extremities for people not getting me wrong. I don't think that happens tomorrow or like that I'm completely crazy, but like, I don't think that this is the end all of the state.

1:04:24The question is, does that happen sooner or later? Or do we keep our transformers, the vast majority of intelligence going forward? And if it is, then it continues probably going directionally where it does. But is there something that happens that changes that fundamentally, something that I think about? Because then all our bets, yours and ours, are very, very different, right? Okay. Well, Ivan, that was fabulous. Thank you so much for spending time with us today. Thank you for having me. We're great. Hi, it's Matt Turk again. Thanks for listening to this episode of the Matt Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from.

1:05:09This really helps us build a podcast and get great guests. Thanks and see you at the next episode.