⚡️ Ship AI recap: Agents, Workflows, and Python — w/ Vercel CTO Malte Ubl

0:02All right. We are here in the remote studio. Thanks again to FDot.ink for learning us this space with Malta Ubo, who is CTO of Vercel. Welcome. Hey, how's it going? Glad to be here. Did I get it right, Ubo? I've actually never pronounced it out loud until just now. That was completely perfect. It rhymes with Google. Ah, okay. So perfect that you worked on Search at Google and AMP and Wiz, which I think still people don't know enough about Wiz. It is no longer a secret, but yeah, you can't use it unless you work at Google, in which case you probably know what it is. Otherwise, there's no reason to really know.

0:34Anyway, suffice to say that you are responsible for a lot of the web as it is today. So thank you for spending some time with us. You're also obviously now building the next web, as we say, with Vercel. And we can cover framework-defined infrastructure. I think you probably saw I have a lot of interest in self-provisioning runtimes. We can cover v0. But here, really, this part is recorded right after you did ShipAI, which we're trying to sort of recap, right, for the general Latent Space audience who may not be watching Vercel as closely as I do or you do. So basically, just generally, what I guess is your message to the broader AI engineer audience on what Vercel is doing with AI?

1:16Yeah, I think the super high level view is that what we're really trying to do is like we're like the biggest fan of the AI engineering movement. And we are also fans of, you know, we're not just like going super hard on hype on hype and the big ideas and, and talking about things, but like being very concrete about like, you know, agents are very exciting and you can actually build them. Right. And so like, I think our entire conference was about both making that easier. Right. And, and discovering the right abstractions as we're kind of figuring out what the, what people actually want to do.

1:49Right. Which is emerging as we speak, then the way Vercel always does these things is by building things ourselves. Right. And so that is both in terms of products, so agents that are products that you can purchase from Vercel and stuff that we do basically in our back office to make our own operations more efficient. And so this kind of building of apps lets us ground kind of what we do in that reality and then kind of extract the abstractions that we feel are really helpful to then put that on the road. And I think the probably most talked about thing that we shipped at the conference was our new workflow development kit, which really, really is just a way to make writing like workflows, like very idiomatic as something that just becomes kind of first class and something you do every day.

2:41You think about it, you know, write 15 design docs just because you want to one of them. It's just something you do literally every day. I think like since you your audience also more generally like in the thing probably like listening to what people talk about I think that's a lot of talk about our virtual development kit but also like more generally like what is what are workflows what are agents how are they related do you use one or the other I would actually love to talk about that as well but like obviously like in our in our conference basically introduced just like what we hope is by far the easiest way to make your you know make your agents something that is easily embeddable into complex workflows and to make those workflows durable, zoomable, and so forth.

3:19Yeah, I mean, as listeners might know, I have a long history of workflows at Temporal. And I think what's weird is a lot of people are discovering this for the first time. You don't really learn about this in CS classes. You don't really learn about this in bootcamps or anything like that because it's not really a unit of compute and storage that is taught. It's kind of like emergent from Uber and Stripe and everyone else. I don't know if there's a version of this at Google. I mean, there is, there's a version of this at every single company that has been doing anything in computer since 1950.

3:53Yeah. Like, but what's not necessarily the case is that it has been abstracted in any way, right? But like when I, when I run a, you know, bank transaction system in 1975, then I invent this, right? Maybe I'm in pure batch processing world and I kind of avoided it. But, but the reality is that I built this, right? And so either I use something very productized, which obviously Temporal innovated on that being a thing, or I use something that's ad hoc. Right. So I go and say, well, I need some kind of queue that tracks the work. And I need a database to store the state at any given point. And I don't know, maybe I write a con job that makes sure that the stuff on the queue doesn't get stuck.

4:35Right. And so I think what's extremely common in essentially every transaction processing system that has ever created is that there either is an explicit abstraction for workflows in it, or someone made one ad hoc, because otherwise the thing just doesn't work. Yeah, yeah, totally. So the headline thing for people who maybe haven't dived into workflows enough is that you can sort of wait and resume code. So it's as though the serverless function is kind of indefinitely long running. Like literally, you can run an infinite loop inside of your serverless code. And that breaks a lot of people's mental model if they don't really understand that the code pauses and resumes and you can wait multiple days and it doesn't matter it doesn't cost anything actually i don't know if it doesn't cost anything maybe i don't know if you charge no it literally does not cost anything um so yeah you can you can run compute for infinite time you can then also whenever like one of these steps fails automatically retry them stuff like that that makes things more reliable yeah and so like it it has a lot of parallels to long-running orchestration problems for agents.

5:37If you want to do human in the loop as well, it's a simple task of waiting for, what's the API that you guys have? It's not, I want to say signals, but you have something like resolve webhook or something. Yeah, I think it's a little bit, it's similar to a signal, but like the idea is that you basically, you make a webhook, like an ephemeral one, which is just a URL that you can ping. And so the realistic flow would be, you reach that step where you want human approval. You know, let's say you write, You get the webhook URL and you write it to some database. And let's say the user now, they lock into their computer.

6:10Two hours later, they have a queue of things they need to approve. That's from the database. They click on one. They say approve. Now what just happens is that that system now calls that webhook. And then from the perspective of the workflow that you originally implemented, you were now able to await that webhook. And now it resolves and you can just proceed with the program. Yeah, it's very elegant. I would say that it eliminates some complexity that we introduced at Temporal, and that's probably for the better. And the other thing I think, just obviously as someone who is in this space a lot and I've seen all the solutions, you made it open source, which is another above and beyond thing.

6:45You could have made it proprietary, but you didn't. Yeah, the way we think about Versal, I don't want to go too deep into that tangent, but I think about open source as having essentially three business models. The first one is Red Hat, where you just sell support. You think it's open source. The second one is open core, where you are the only one that gets to monetize it, but everyone else gets to run it if they want. And Vercel maybe has not invented this, but certainly kind of is the most successful edit model where you say, okay, I have this software library and it's truly open source. Everyone can run it.

7:15It comes with like adapters for every place on the planet. And that makes it really popular. And then we get a piece of the pie. And so our strategy is to grow the pie, while what we actually see is that our pie piece is relatively constant size in proportion to the pie, right? And so we can drive the open source project. And so that's why, like, you know, I don't want to say we're like, you know, we're in it for the business model. But like, I think it's a business model that has more winners than the alternatives. I think also something that if people are seriously evaluating for production workloads, they care about because, and I ran into this in Temporal.

7:49These are going to be extremely valuable workloads that you're going to put on workflows. And so you want to set, you want some ownership, you want some auditability. In practice, like who's going to actually run it themselves? Probably not, but you want the option. You want to check the checkbox. 100%. Yeah. And I actually do think people will run it themselves and that's great. Awesome. So that's workflows. By the way, I think the most disgusting is the use of directives. You know, use cache, use no memo, use Nemo, use whatever. So fun. So fun. I don't know if you have a take on directives in general.

8:19I don't, to be honest, I don't feel super strongly. I do find, in particular, inside of the workflow DevCube, I find the use pretty elegant. I could imagine other ways of doing it. I think we did post a blog post about all the alternatives we considered. Because there are some that you think about after five minutes and after two hours of thinking about it, you realize, yeah, maybe this isn't such a good way to do it. But there could be other ways of doing it. I think we're working with CC39 to bring decorators into more places, which would kind of make this literally the same. The same thing would happen.

8:54You move the line above the function instead of below the function. So it's not a big difference, but it would become, you know, for example, you could make it part, like TypeScript be aware of it without a TypeScript plugin, which you already provide, right? And so literally I was vibe coding this thing on Sunday. And I said, okay, Claude, you have no idea what news workflow is because it came out on Thursday. But here's the docs. and, by the way, you know, put it on my side. And, yeah, so I did it. Installed a TypeScript plugin for me. So I had, like, the perfect DX. And it just worked from scratch.

9:23So that, yeah, was great. Okay, awesome. So we can come back to a workflow anytime you want. But I just wanted to keep moving on, like, all the stuff you announced. We should probably also just touch on the ISDK. I know you're not, like, as closely involved to that team. But obviously, one of the most successful open source projects. I mean, obviously, Versailles is very good at frameworks. But I think it was not a given that AISDK would be a winner because of Langchain, because of Mastra, because of everyone else trying to get that spot. Except that you guys have the perfect package name. So I think that helps a lot.

9:56I actually don't. I'm not sure how much that helps, but it's great. Like there's a fun background from what people thought AI was for 10 years ago. But yeah, I think, you know, we announced version 6 beta. And I think the big, I mean, it's not really news because these things are open source and you can follow them very closely. Right. But what it does introduce as a stable feature, because it's already as kind of experimental in AISDK 5, is a direct agent abstraction, which so far wasn't there. Right. People would build agents with AISDK, but they would have to do it in a bit more Baybones fashion.

10:30What I do want to mention is actually because you mentioned it's very successful, which is true. And I think the reason why it's successful is because we could strain ourselves to be humble about what we know our users might want to do. The example I would like to give, like if when you build like a new web framework in 2025, you know exactly what people are going to do. Like it's such a well-explored space. As the person doing it probably has done it three, four, five times in their life and failed and learned from that and tried the other things, right? It's so mature. Like it's the most mature thing.

11:09Even 10 years ago, that was also true, right? That's why Next.js is so good. Because when Guillermo started building it, he knew exactly what to do. He knew exactly what the app would be. Almost nothing has changed. So the AI app space is the absolute opposite. Like we know absolutely nothing and we still know absolutely nothing. Things are emerging, but we're so early. And so if you put a very thick abstraction, then it's probably going to be the wrong abstraction. So you have to be humble and say, OK, I need to stay low level so that this can be flexibly used as trends emerge. Right. And so that's why we didn't have to rewrite AISDK when everyone went from writing chatbots to writing agents, because we stayed at a level where that, you know, almost looked the same.

11:55Just stuff people wouldn't on top. And I think that's why it's successful because it doesn't, you know, we didn't say, okay, we know what the apps are going to look like. And we're going to do this like Hollywood principle. Don't call us, we call you style framework where like you just have to fill in the blanks. It's super structured. You'll be happy, right? Like we didn't do that, even though that was like so in our DNA, like we did have to like really restrain ourselves. But like, that's why it's successful because it's so low level, right? And so that's why, you know, on the other hand, you know, that's why we don't have an agent abstraction yet.

12:23like every other competing library like leads with that right yeah like opening eyes uh sdk day one it was exactly master etc and you know i mean i'm not saying it's bad but and obviously that's that's more accessible right now you know you have to understand an agent's tool with a loop what do i do i use the stream text function and gives it give it tools okay added we added all kinds of like control already in the sdk version 5 where you can prepare the step you can select the tools on every loop. You can do all these things in a pretty advanced fashion. Half of those other frameworks are built on top of ASDK anyway, right?

12:58So that it forms the basis. And so what we're doing now is we're bringing what is emerging as the patterns that people built over and over again as abstraction into the library as the users just kind of are solidifying. Yeah, I have interviewed enough agent framework builders and Big Lab model people that I actually find that I can push back on you. uh okay go ahead it's it's really interesting because i think uh you are saying basically you're saying we will be at the jQuery where they kind of at the jQuery era right like we are where we don't know what we want yet we're building all these like uh tools to make our the smallest possible things easier uh and then it composes up and we're just starting to emerge of agents i would say that the big lab people are obviously the opposite but they're not they're coming at it not from like a dx point of view they are they're very big model pills they they want everything to go through the model.

13:48The reason they want the Hollywood principle of we'll call you is because they want the model to control the tool calls, the reasoning, what have you. And I feel like there is a, there's a mentality. Obviously you can have frameworks that do both, but there's a mentality in the big labs. If you work to big labs that you always want to give the wheel to the model. And then for you guys as framework developers and people who are software builders it's more comfortable to build like the the smallest possible thing instead of like the sort of agi thing if that makes sense yeah yeah i actually don't think about it in in in those dimensions i think the like because i 100 agree that like people i think have to be willing to let go and let the tool sorry let the model kind of take control right and to to get emergent behavior and certainly on coding agents that works incredibly pretty well.

14:40But that I'm totally on point with, right? And that's, yeah, SDK does this very well today. Like all the agents that I've personally built work like this. That's not the same. But the other thing is like, how do I now embed this intern application, right? Like the model apps couldn't care less because they're not really building applications. And so that's something that a company like Vercel thinks about a lot. Like, again, what does the developer actually want to express and how do we let them do it? And so I think one of the key things that people wanted and still want is they want streaming because these models are slow.

15:15And so suddenly this almost obscure sub-genre of programming where people are like, you know, it's five milliseconds. I'm just going to not ship it, right? Suddenly it's 30 seconds and it becomes absolutely important. And so we give people the tools to build streaming applications in a way that feels intuitive, right? And I think that unlocked a lot of value there because that was like genuinely hard and we made it easy. And so those are kind of the things that we're looking for, but that, you know, are not obvious when you're kind of mostly concerned about the AI part. Fair enough. And I think like the design space has more dimensions than what I tried to simplify it down to.

15:51Just one more thing on AISDK and then we can move on to like the other agent stuff. And obviously you guys announced so much. It's so hard to cover. So Vercel is like a house of frameworks, right? You have so many framework authors, all of them legends in their own right. What's one, I guess, philosophy that you're also applying from all your years and all your people who work on frameworks, right, that is informing you? I have one, and feel free to counter propose, which is what Sebastian Markboga, who is obviously the tech leader of React, is used to say, which is have a small API surface area.

16:24I feel like that has maybe been like not as important or like there are other overwhelming priorities. But I just want to get a sense of like, you know, what governing principles really resonate with you. Yeah, I mean, Seb and I are talking about this a lot. But I think I'm often representing the kind of enterprise side where it's like, no, but I actually want to just control this. Give me one API for this. Just one more, bro. So I want to be in control and I want to be able to configure it and I want to define the defaults the way I see it. But it's good to have tension around these things. I think the thing that coming down from Guillermo is just the absolute founding principle of ourself is that we never give you an abstraction that we haven't used ourselves.

17:10Like dogfooding is ultimately the thing, right? Right. Like that, like AISDK was extracted from V0 and then we build it and we kind of diverged a little bit. And then we took on the substantial work to bring back V0 actually fully hosted on AISDK. And then we learned back and we made sure that migration isn't too hard, which the users appreciate as well. And so and so forth. Right. And so there's this constant feedback loop where if you don't have that, which is like this sounds like so obvious. right but the the reality is that framework builders are usually not application builders yeah and so they build ivory towers that when they're hyper geniuses or they get lucky they happen to be good but like but but if you want to do this in a reproducible fashion with a high hit rate then the only thing you can do is you you have to you have to try that stuff out yourself and like that's what we do every day i i really like that principle obviously a good idea it's just very hard to practice in real life, obviously, because when you're a maintainer of a framework, a lot of bugs come to you and, you know, they pile up and you have to spend some time working on framework level issues.

18:19Happy to move on to Vercel Agent and maybe the Agent on Every Desk program, which, you know, I think you're kind of also championing. So like, yeah, let's talk about the use cases that you guys use internal agents within Vercel and what emerged. Yeah, let's structure this two ways, because I do think there's a difference between kind of the agents that we're building internally versus the stuff that's, you know, essentially a Vercel product, right? Yeah. And which you can use today. I thought they were the same thing. No, they're not the same thing. It's actually, I think that's quite important.

18:53Like we also, we're distinguishing between like agent as a service, right? And so the Vercel agent, that's what it is, right? It's ultimately an agent as a service product similar to, you know, codecs in the cloud or the cursor agent. Like not as in like it's the same product, right? But as in like these are things where you go somewhere and you say, I would like to use this agent. And then, I don't know, maybe you give them a credit card and it works, right? Which is different from the stuff that we run internally. But let's talk about the Vercel agent for a second. Like we've been basically, I think our strategy overall is to have an agent that helps you build applications on Vercel.

19:28This is, you know, there is some overlap with coding agents, but like I think the thing that's unique about the Vercel situation is that because we have your runtime data, we see your error logs, we know where the preview deployments are, that they always will exist. We know how to start the dev server. We already have the secrets. So there can be like a quite integrated solution for something that otherwise can be quite hard, right? If you ever onboarded, for example, I mean, you have onboarded Devon a few times, but you probably have been in that situation where it's like, feels like onboarding a junior employee, right?

20:01And so some of these things, like if you're within the Rysel ecosystem, become much more simple. And so in that world, we've been shipping away on different things, right? A while ago, shipped a code review agent, which I think is really good and well integrated. And the thing that we announced last week is our broadly DevOps agent, which is actually tied to our anomaly detection system. So whenever there's an anomaly that we detect on your production site, it kicks off the agent and the agent does an investigation of what's going on. From a technical point of view, what this agent has, it has several tools.

20:36It can make any observability query against your project. So as a query builder, it can execute the queries. It has a way to read logs, obviously with queries as well. So what's really magical is that it's just very good at this. Like by the time you click on the anomaly, it will like almost all the time just very precisely tell you what happened. It shows you all the graphs I looked at. It just like, I don't know, man, it's just so much easier than doing it yourself. Like it takes away certainly minutes of work. But I think what I'm actually very excited about, and I think this is an overall pattern that we see with agents, is that there in many situations is this, what we in search call recall precision problem.

21:21And that also happened with anomaly detection. With anomaly detection, you have to tune it, right? And you either tune it to be very aggressive and then it fires and in the worst case, it pages you in the middle of the night and nothing was wrong, right? Just a team in Asia sent a newsletter. And so the traffic went up, right? Or you tune it not like aggressive enough. And so you miss events. and with an agent, you can just say, okay, I'm actually going to have this tuned very aggressively and I'm not waking anyone up. I'm telling the agent and the agent can take two minutes to run. I'm actually fine with that because no one would have reacted in that amount of time in a very reliable fashion.

21:57And now it can actually look at a time series, can look what happened. It can look at the IP addresses that are making the request. It can look at the type of error messages, right? And it can make it like a call whether to escalate to on-call and wake someone up. or to say, okay, this is completely fine for someone maybe to take a look next day, which is, I think, the perfect decision for agents to make. And so that's something I'm very excited about, that you have this essentially co-worker that has no sleeping problems in the loop and they get woken up instead of you. Yeah, I think the dream of AISRE has been a long time coming.

22:34And I'm actually on the record at the start of this year, I made a podcast saying, like, oh, I don't think anyone's going to do AISRE. So I'm very excited. I haven't tried it out personally. I have seen you tweet about it. And I think, yes, obviously that is the goal. That is the dream. We should make Brian Johnson happy and have good sleep. But, you know, we're not exactly there yet. And I think, like, the question is really fold, which is time series analysis is not exactly within a distribution for language models. There have been a lot of people doing, like, time series models. There's a deep field of anomaly detection, which is basically what you're doing.

23:09and you know there's a question about like is this a solved problem or you know like how much can we trust it uh and then i think the other one is like aligning the the human preferences right like sometimes like i don't know until i've seen a few examples of like oh yeah this one you should wake me up the other one you should not and then pretty much like when i solve the problem you know it goes away so like the next problem you're always fighting the last war in in sre yeah i give you i give you a bunch of things you can take i think that the the like you have to try the product. It works really well.

23:39So we don't do the anomaly detection in the LLM, right? The anomaly detection is a separate part for cell products. It's a completely separate part. It's a pipeline that works in our time series database, right? And launched independently of this. And so, but once you have this, so our experience is that, you know, if you give these agent a tool that does queries, it's really good actually at digging into individual parts of the time series. And the other thing it has access to is logs. And logs are actually, they're just text, right? So if you say, if you see from the time series, like what happened, can I somehow figure out like what this is?

24:15And then you head over to logs and do a deeper dive. Now you're kind of more in the world where the model is comfortable. One thing that we don't do today, but we'll do in the future, is that we also give the model X, that particular agent access to your source code so that it can, first of all, figure out, Like, what does the error message mean, right? And every time you can actually make a PR to just fix it. But every so often that'll be possible. And so it would then also do that. Which is why I think like people like Datadog and Sentry are trying to do that, obviously, because they're observability platforms, but they never own the code.

24:53And so like they're always limited in what they can do, you know, like 100%. But also, like, I mean, what I want to qualify where I'm so happy about how it works, it's like, you know, it's a small part of the overall problem, right? Like, we're actually not here to, like, build the AI SRE that replaces that job function. Like, that's another part of what I feel pretty passionate about, that it's at this moment, they're, like, agents are both extraordinarily effective and still very ineffective. And you have to find the right problems. And then when you find the right problems, they are super magical.