Scaling Agent Workloads at Vercel

24:36It is extremely well documented. So the agents do a really good job figuring this out. When you start prompting for this type of behavior, it will learn that this is basically the way to do it inside of EVE. That's super interesting. Before we go deeper on the tech side, something you mentioned. So you talked about going enterprise and then maybe dialing back to personal use and things like that. I'm going to direct a question over to Char. Who are you seeing as the key user audiences for this? and what is the sort of rollout and usage look like? Yeah, as Barbara said, we did initially target enterprise, given that that was where most of the requests came from.

25:13And we have the infrastructure, so it was kind of a no-brainer. And also, we were scratching our own itch to an extent. So we actually use EVE internally for all of our internal agents. And now we have hundreds of them running on EVE. So that was the initial target audience. But very soon after we launched, we saw thousands of our users actually adopting it. And those are not necessarily just within the enterprise segment. You can see them around our hobby users or pro users building. So the range of use cases we've seen go all the way from business agents, what we call what enterprise is launching, where you automate a business function down to a hobbyist building a iMessage multiplayer agent on a weekend.

25:58And we want to enable basically all of those use cases and let you extend the framework to your needs. And so I love that you started scratching your own edge, right? I think especially one of the advantages working in a developer tooling company is like you can really do a lot of that. I'm curious, you know, if you were to look at the distribution of internal agents that you're building, how much of those are engineering related versus other functions within the business? It definitely started within engineering. And engineering is the function that actually spun up a lot of these agents initially.

26:33But as we saw the adoption over time, it became go to market, move everything they had on EVE. Then marketing is working on automating content creation, socials and so on on EVE and so on the rest of the team. So you see billing that help EVE was an example. So you see examples of product teams that give the product documentation to the agents to answer questions both on the docs and internally and sometimes share those with customers directly on Slack. And so the distribution is still heavy on the engineering side where those are the main users and builders of the agents. But I think you're seeing that distribution kind of get more neutralized as it gets easier to spun up agents and build them as we get to self-evolution and so on.

27:22Yeah, that makes sense. The advantage of the approach you guys are talking about in terms of like, hey, we make this really easy for coding agents to work with is increasingly coding agents are available to everyone. right? Like it's not engineering specific. Interesting. So I guess a question, are there any patterns that have been emerging in terms of how folks end up adopting this? And particularly once again, thinking outside of the engineering context, like I immediately am like, oh yeah, I'm used to working with agents, I'm doing things and all of that. But as you start to see go-to-market adoption, as you start to see marketing adoption, are they approaching this differently?

28:01Or do their agents look very similar to the engineering developed agents? Yeah, there are three types I've seen, I would say. The most common one is the conversational agents where the agent is channeled through Slack, mostly. We are a Slack company, so most of the communication happens in Slack. That was the initial use cases. So like DZero, for example, which was the initial use case, did primarily work on Slack. The second one was kind of background agents. And these are mostly the agent's job is to run some sort of a report and also send it to Slack or send it through email to internal folks.

28:40And the last one, which GTM, for example, is a big user of this, is actually applications. So actually build an agent, but your front-facing is a web app that you're working with. And that web app is actually powered by an agent, and you also have a chat interface with the agents on the web app. So those are kind of the three different types we've seen that come out of it. But yeah. So that makes me want to, I'm going to send this back maybe to Andrew then. So in the case where I'm building a web app of some sort, but it's got an agent powered backend, how does that interact with Eve? Because you mentioned this is directory structure focused.

29:17Do I embed these directory structures inside my web app directory structure or are these independent repos running in different ways? How does this integrate with the rest of the development lifecycle? Yeah, so we wanted a pretty opinionated story with Next.js. So Next has the pages directory and then more recently the app directory. And so we knew the agent directory had to just work. So there's first-party support for that type of setup where the agent literally lives next to your Next.js app. But it does not have to be that way. And I think in many cases, there's still lots of teams that are deploying a separate front end from a separate back end.

29:52And Eve can work absolutely just like that, where you deploy your Eve agent as its own project, and then you just connect to it over HTTP. And that's with that default. We give you the Eve channel, which is basically our HTTP protocol, our stream protocol. So it has things like sending messages and canceling and things like that. And then ultimately, we give you back this durable stream, which you can basically use to power any UI that you can think of. And so yeah, it really is just an API. So the way we think about it internally is EVE is a back-end framework. It is not a front-end framework.

30:28And so yeah, it works with your front-ends the way any of your other back-ends would. Okay. On the application side, we've actually seen some really cool examples of our users shipping web apps with EVE. Two of the examples we saw, one of them was an open-source CRM. So what happened was the user goes in, adds a lead, adds a name and email, and the background agent goes in and researches the lead and completes the whole thing behind the scene. And the second one was, I believe, an AI music maker where you kind of go in and add the tempo, but then the agent kind of collaborates with you on the site and completes the music.

31:05So you kind of are seeing these agentic native applications that are being built. and the Eve agent really is like the back end brain behind the scene. Yeah, no, I think it's really interesting. And this is one of the spaces I feel like is under explored yet. Like a lot of generative UI interfaces are like, let's throw a chat bot in this thing or let's asynchronously generate a report. But I think there's a tremendous untapped opportunity for almost collaborative interfaces. Like what does collaborative document editing look like when you have an agent involved? Right now, you can sort of do that with Claude and their artifacts.

31:45You could try to do it in Google Docs with Gemini, but I wouldn't recommend it. So I think there's a really rich space here for easy to interact with agentic pieces. So I'm curious. So, Andrew, you said basically you're exposing this durable stream and now you can do whatever with it. Is there any sort of primitive that is missing there? I feel like that's pretty low level, I guess, in terms of interacting, which is fine for first generation. But I wonder if there are also opportunities there in terms of what are the abstractions that make sense in interacting with an agent inside an app context.

32:23Yeah, it's interesting. And I think the Vercel Lab team is doing really interesting stuff here, I think, with tools like JSON Render, and taking these lower level protocols and producing UIs out of them. And Eve, the stream protocol that we have is very low level. You were literally getting delta chunks of reasoning and things like that. So it is a lot to work with. The TypeScript client makes this stuff much easier. So we have really nice APIs for iterating over the stream and strongly typed phases and events and things like that. And so you can definitely work a little bit higher up. But you're right, not much.

32:58It's still a fairly low-level API. So yeah, I'd love to see a lot more done here. I do think Eve is a lot more than just interacting with a chatbot, like you said. We have cool features where when you send messages, you can actually specify an output schema, for example. So this is really interesting because now, if you know your UI takes some shape of data, you can agentically generate that thing. So this lets you do really completely different experiences than just asking a question and getting some English response back, right? Oh, interesting. So question on that immediately. Do you include built-in validation of that schema and serialization of it?

33:35Or is this just... Yes. Okay. Right. So what does that look like? Tell me more. Yeah. So we take the output schema, and we basically plumb that all the way through down to the... It was built on a tool loop agent from AISDK. So basically what we end up doing is we actually produce a tool called submit result, and that validates your schema. And it's retribable, of course. So a lot of times these agents will call a tool like that, and they produce the wrong thing, and then we can hint back to the agent basically saying, oh, you missed this required property, and it'll do it again. So it's not guaranteed to get your schema, but especially with these frontier models and the way they are today, it's quite good.

34:14Yeah, they're good at that. Now, the challenge with tool calls is that you lose your streaming once you go to tool calls, right? So now you've suddenly got to wait for that whole block to do it. No, not exactly. So the stream will produce all the reasoning in between, right? Because it's just the final step. Instead of getting a message, right? It's some English message out. You're just getting that final tool call, but you get everything in between. So, you know, if you're using some high reasoning effort, and if you want to, you can inspect all of that, even though you're waiting for some sort of output, you could still see kind of everything, everything in between.

34:47Totally. No, no, no. I guess what I was saying is, yeah, you're streaming up until that tool call, but the tool call introduces a synchronous block, essentially, where you're not getting pieces of the tool, like, you're not getting a partial schema results along the way, which influences the UI you want to build, right? Right now you have to have like a waiter or something like that rather than streaming out the pieces of it as you go. But super interesting. Let's talk a little bit more infrastructure, getting back to this like cloud scale thing. So you mentioned a little bit of like, hey, if I got a thousand requests, suddenly I can just like spin up a thousand different processes to answer this.

35:21Very interesting. That now introduces questions around durability, resumability, all those different pieces. Like how does Eve handle that? Yeah, so I mean, it really started where we knew we wanted to build EVE on top of workflow, which is basically our version of Temporal, I think a lot of people are familiar with. And so the way we modeled this very early on was a single workflow is a single EVE session. And today it's actually two workflows, but you can think of it as one. And so the partitioning is basically every session is completely separate from every other. And there's ways to share data between them.

36:01Or I should say we provide guides on how you might want to share data between them if you bring your own storage. But that is the partitioning. And so a thousand sessions versus a single session, you shouldn't have to think about that. The cloud should do that work for you. But yes, as soon as that turn starts, you are initiating a workflow. And then we do some clever things. So we run on serverless functions, which historically have 15-minute timeouts. These agents obviously can run for a lot longer than 15 minutes. And so we can do some really clever things here where every time we invoke a tool, we can move over to another function.

36:35And it just continues the workflow. Because it's another synchronous break. Yeah, no, this makes sense. Exactly. And in the future, we might decide to, if we have a lot of time allotment left, maybe we run multiple tool calls on the same function. But today, we actually moved to a new function for every single step. And so as long as workflow gets faster and continues to be performant, you don't notice this, and the agents can effectively run indefinitely. That's cool. What about things like durable memory and stuff like that? So one of the things that's interesting working with something like an OpenClaw or a Hermes or one of these things is that they have as a part of their built-in loops some amount of, oh, I'm going to remember this.

37:19I will write my own file that I can reference, or I will iterate on this skill that you maybe gave me a first version of, but I can actually modify my own prompts and sources based on your feedback. Are there mechanisms for that type of learning agent within EVE? Yeah, this is probably the biggest thing I'm working on right now. So all of memory is open NPRs, which you can go inspect if you want to right now. But yeah, so I'd say all of this is, and this actually goes back to what I was saying before, where we started enterprise. And funny enough, these types of things aren't necessarily your enterprise use case, but they are very personal agent use case.

37:57And so now we're starting to double back and build up these features for the personal case. And self-evolution is a really, really important part of that for all the reasons that you just said. The idea is you just deploy the out-of-the-box agent and it just starts getting better as it works with you. And there's different forms of self-evolution. And I think it's also different. Eve is going to do it a little bit differently than maybe something like Hermes, which can literally just update its code in place and then restart itself effectively. So on Eve, we're thinking about self-evolution in a few different ways.

38:29Memory is obviously one of them. And we are doing a first-class folder called Memory. And you will be able to define different memory slots. And so you can actually, you would be able to have, let's say you wanted like workspace memory for your Slack channel. So this is like literally global memory that's shared across all your users. But then you could define an additional slot that is just memory for yourself. And so now the agent actually has multiple memory banks that it can pull and save to, depending on what scope you define. And so it's kind of like a mix of, we'll give you a really easy personal memory provider, but we think it has the primitives for much more advanced enterprise memory use cases.

39:08So hopefully it threads the needle on both sides there. But yeah, so memory is one form. And then the other one is obviously modifying yourself. So can you add a new connection to some MCP server or write new tools, produce new skills, things like that. Skills maybe is a special case or it might actually be closer to memory. But for other things, you literally have to write code to go make it happen in EVE. And for now, we're going to lean pretty heavily into Git. Our customers are using Git. And so what we want to do here is open PRs. And so we will do a coding sub-agent that you can define, and that is basically your self-evolution.

39:46And so we'll kind of do all the hard work of when you deploy this thing, it'll have a sandbox for you with your code base. We'll have the right prompts so it knows when it has to invoke the coding agent to change something. But the goal is to effectively just open PRs against itself. And so that's kind of our model of the VPS world where they just get to change their files. Yeah, no, that's honestly a really nice, clean model because it allows you to define whatever level of oversight you want on those. Exactly. You want to let it self-merge? All right, you can build those automations. You want it to have heavy human review on everything?

40:21You can configure that the same way you would anything else. So yeah, that's really nice. Yeah. So you used a term slots, and this is, I think, the second or third time you used it in this conversation. It may not be familiar to everyone. Can we define what you mean when you say, oh, we're going to define a few slots for this? Oh, yeah, I use it specifically in memory. So yeah, when I generally say slots, I'm talking about basically our top-level directories, right? So skills, tools, sandbox channels. So memory is going to be, if I was talking to my coding agent, I would call it an authored slot.

40:51This is how I would refer to it. So it will be a top-level directory. And then inside of memory, we're basically giving you, it's effectively kind of like memory-like drawers almost, or shelves that have different drawers inside of them. And so one drawer might be global, one drawer might be user-based, one drawer might be channel plus user. Maybe you want memory just in specific Slack channels. So it'll give you the control over how you want to scope those memories. And this lets Eve do interesting things where we can control. So the classic example is you almost never want a tool call that takes in a user ID, for example.

41:26You don't want the agent to guess or think it knows the user ID. And so with these memory slots, do is they're lowering tools that are bound to those things. So it cannot guess the wrong slot, right? Yeah, yeah. You want all of those permission and other related things to be deterministic under the layer of what the agent does. It just asks for a thing and you say, oh, that is scoped to this user. We're going to include that in the actual functional call. Yeah, exactly. Yeah. And so we went back and forth on whether we wanted memory to be first class or not. And the more you look at cases like that and you realize that you can start grabbing memories from the wrong people and this and that.

42:02When we see problems like that, it kind of hints to us that, yes, this does belong as a first-class thing because we can provide the right guardrails and the nice APIs around them and things like that. Quick detail question on memory. How do you expose that to the agent? Is it through a tool call? Is it embedded in the context all the time? What does that look like? We took a stance here where we have this concept called provider and the provider can give whatever tools it wants. And so Eve actually makes, we have no opinion on the tools that should exist when it comes to memory. And so the model is actually, we determinously invoke three functions, recall, save, and this tools function.

42:46And the provider, it's entirely up to it on what it wants to do there. And so the provider that will ship in the framework is a file provider. And this one is very, very simple. Recall, it returns the file in full as context. Save, it completely ignores. So there's no implementation of save. And the tools it provides is basically add memory and forget memory. And so the agent can add a single memory, in which case we will resave the file to blob storage. And then forget memory, same thing. We remove the memory and resave it. And so that's the file memory implementation. And so just to make sure I understand then, this is a deterministic thing.

43:27It's going to be called and the result would be whatever gets returned by recall is embedded in the context of the agent from the start and it has access to these tools. So in this version, the contents of the file are going to be in every agentic prompt and it has access to save or forget particular memories within it. Exactly. That's like the most basic version. This is very similar, I think, to like Hermes has like a user.md file, I believe. And so this is kind of our version of that. But the APIs we're providing let you go way beyond this. And so we really wanted to support something like SuperMemory, where all of the recall and saving is done on their service.

44:05So Eve has no opinions on those things. And so the SuperMemory provider actually looks quite different, where they might provide no tools. They don't want the model calling save memory or forget memory, whatever. They don't want to do that because what they want to do is, we'll call save at deterministic times. they actually take that message history and they will go run some agentic process and figure out what to save and so it doesn't want to expose tools right because they're doing that in their own service and then their recall can do something much more interesting than like the file case they can actually look at the inbound message and say let me recall agentically based on this inbound message and so it's a completely different recall interesting so recall is called in every turn of the loop so you could actually change what your sort of embedded prompt looks like based on nice okay exactly yeah and so we call recall there's a few phases that we call it in obviously turn start is is the main one but we also call it after compaction for example because we can't guarantee that compaction keeps you know your memories around so we'll call recall again after compaction and someone like super memory they might actually look at the compacted prompt and say oh no we're good, it has what we need, or no, we have to re-inject.

45:17So they can do really sophisticated things, and Eve doesn't care. In that world, sorry, I'm going way down in the guts here, but are these each turn recall, is that being injected as a new message at the end of the stream, or are you actually editing the system prompt? So we give you the choice. The default is append, so we don't want to bust the prompt cache if we don't have to. So we use the role user for memories, but you can override this, and you can do a role system. And so for the file memory provider maybe role system actually does make sense. But the downside of that is every add or forget does bust the system prompt.

45:52Yeah, that makes sense. Awesome. That's super fun. Thank you for geeking out with me on that. Yeah, it's all open in PRs. So hopefully it launches soon. We'll see. Yeah, no, I actually, I was going and looking. You've got a lot of PRs open. So actually, that's kind of an interesting question. So this is a Vercel project. You all are open sourcing it, it looks like. What's the contribution model? What's the ongoing management ownership model look like? Yeah, so we have a bit of a funny problem right now where we want to accept outside contributions. But one of the problems we actually have is because our test suite is hitting models and is actually deploying projects to Vercel.

46:37So it's using a bunch of secrets, basically. So we actually have this problem where we can't easily test outside contribution PRs. So we actually have to carefully pull them into a sandbox, make sure that they're not doing anything malicious, and then we have to basically recommit them under our accounts with a co-author, and then we can actually run the suite. So yeah, this is definitely a bummer, and we're trying to figure out a better process here. It hasn't been the easiest thing. But yes, we want the outside contributions, and we've been trying to take in as many as we can. We obviously have EVE agents internally that are actually analyzing these PRs daily to look for the ones that we should be pulling in.

47:13But yeah, the more contributions, the better. And the EVE team was really two people full-time from March until about maybe mid-July, I want to say. And we just brought on four more people that have been doing a great job. So yeah, it's about six people full-time on it now, engineering-wise. Well, we're getting closer to the end of our time. I have one more question I'm going to put to each of you. and then also anything that we haven't talked about that you think would be important, we can bring that up. But the question for each of you in turn, and maybe we can start with you, Andrew, and then Shara, you can close us out, is like, what's the vision for where this is going?

47:50What do you see coming down the road, the next three, six? I don't know if we can project out nine months in today's AI age, but what's coming down? What are you excited about? Yeah, for me, I mean, one of the very early principles behind EVE was we're going to bet on the models getting smarter. And the way we designed the framework was really meant to take advantage of this. I think a lot of other agent frameworks that we had seen, they scaled vertically, where they just appeared to get more and more code. And eventually it kind of doesn't look like an agent anymore. You have a lot of determinism.

48:23It's almost like these workflow style APIs. And what we really wanted with Eve was to go horizontal and basically just say, we are going to bet on the models making the right decisions. This is also generally where the folder structure and things like that came from, where we just want to provide more things for the model and let it make the right decision. And so Eve is going to stick to that principle. It's clear that the model intelligence is not slowing down anytime soon, and we think Eve is going to take advantage of that in a great way. Awesome. What about you, Shar? For me, I think the way we're thinking about it is that company building is agent building.

49:03So the idea is that Eve is more fundamental than your certificate of incorporation. So your agent predates your websites, your domain name, and even incorporating the company. So you start with the agent and the agent incrementally builds the software factory for you, including your complete software development lifecycle. And your job is then to actually fine tune that factory. And we believe that Eve is positioned to basically become that brain of your company. moving forward. That's kind of where we're heading. Well, that makes me think immediately, what's the hosting story outside of Vercel?

49:40I mean, obviously, it's great that you guys can manage all of this and all this sort of thing. But if I'm building my company on top of this, I want to know that I can move it where I need to move it whenever that might happen. So how does one host an EVE-based agent framework or set of agents? Yeah, so I mean, this is actually very important to us. And we do have customers internally that are running EVE on Kubernetes, on their own hardware even. And part of this is taking advantage of what is called a world inside of Workflow. And this is part of our end-to-end test suite where we test different worlds, like a Postgres world, a local world, and things like that.

50:15And so as long as you can define a world for your infrastructure, and many of them exist, like there's worlds for tons of other providers out there, that is really all you need to host EVE yourself. And so, yeah, there's a lot more work that we're going to be doing here to make that experience much better. And the key with things like memory and self-evolution and these other types of things is there has to be adapter contracts. We can only do so much in the framework without bringing in external things. Dynamic scheduling, for example, that will be first class on Purcell, but you will need to bring in something.

50:51But the key is just making sure that those APIs and those adapters are thought of early on in the process and not some afterthought after it already works on Purcell. I love it. That's a pretty good cut unless you guys have something else you want to talk about. No, nothing for me. That was great. Yeah, that was fun.