From Coder to Manager: Navigating the Shift to Agentic Engineering with Notion Co-Founder Simon Last

12:06Simon Last:We still do reviews on all the pull requests. And I would say, and, you know, all the pull requests are now written by agents. They're often, like, larger and more complex. That's, like, the worst part. But the better part is that they're often, like, a much better tested, and we can demand sort of a much better testing for the things that merit it. I never produce a PR that hasn't been fully intent tested anymore. And so it's like you can get to a pretty high degree of confidence that it works. But it requires like you're not just vibe coding by saying the thing you want. You're sort of thinking carefully about like what is the change I'm trying to make and how can it be verified and how can it be deployed safely.

12:47Simon Last:And then enlisting the agent to help you with that process.

12:49Sarah Guo:When you think about where you said the general assistant like doesn't quite exist yet, But what do you imagine Notion's agents being able to do over the next year or two that are still unblocked? They're still blocked by either capability or your harness work.

13:06Simon Last:We struggled for a few years to build an agent. And it always sort of works, but then it wasn't that useful. Largely, it was too early. So we tried to build an agent, I would say, actually three or four times. and then we finally launched it last fall, so like last August, September. So the Fused Notion AI now, it's like the full agent that has accessed everything in Notion pretty much. So that totally works. I would say like a lot of the original vision that we had totally works now and it's like fully shipped. Last August or September, we shipped our personal agent. So it's pretty much every user in Notion has an agent and basically it has access to all the things that the user has access to.

13:53Simon Last:So, you know, it can create a database for you. It can update things, create documents. It can search the web, do research. And then the second big thing that we just launched last week actually was custom agents. So you can basically, you can create a new custom agent, give it a name. And unlike the personal agent, by default it doesn't have access to anything. So you have to grant it access. But then once you do, it can actually run autonomously in the background. So, for example, you can give it access to its own database to file tasks, let's say, and then you can attach it to a Slack channel, and then it will start responding to people in Slack and filing tasks.

14:26Simon Last:That's one use case. Another one is maybe you could give it access to a database of weekly reports and then let it search the web or search your workspace. So it's sort of a custom agent sort of represents some work or job, some knowledge work tasks that you want to be done autonomously. One thing I'm really excited about this going forward is we want it to be extremely good at sort of bootstrapping its own capabilities, basically from an initial kernel, allowing it to basically bootstrap itself to do anything, right? So even, for example, maybe building an integration that we don't support yet, deploying that, and then using it.

15:00Sarah Guo:So you imagine that Notion agents are actually the broader definition of agent where, like, writing code is a tool it's got access to.

15:08Simon Last:I think it's pretty key. Yeah, I think of coding agents as, like, the kernel of AGI. AGI will be a coding agent. and code is just a really, really useful primitive for representing deterministic logic. The thing that's really exciting about it, replying it to a knowledge work agent, is that it can bootstrap a capability. So like I said, if integration doesn't exist, it can build it. If it needs to connect itself to a new data source, it can do that.

15:39Sarah Guo:Given you have a notion, is at scale but is operating in a landscape of productivity and platform players that are at even more scale, right? Many of these will end up with their own agents. Lots of people from the labs to the Microsoft world are trying to integrate other data sources. You have this, like, cross-attempt to integrate and index. Like, how do you think that plays out? Like, what do you imagine that Notion agents are best at or what they have the right to go do?

16:06Simon Last:If you look at the landscape, like, I would sort of say there's the labs and then there's maybe the software platforms and then there's maybe like infrastructure. In terms of the labs, you know, we see ourselves as kind of like the Switzerland for models. We think and our customers, they don't want to be locked into a certain labs model. They're always releasing new versions. Any given month, one is better than the other. So we want to be a place where basically you can easily get access to all the best models at any time and you can easily switch around.

16:39Sarah Guo:Do you think open source plays into that as well?

16:42Simon Last:Yeah, yeah, absolutely. I think the open source models are actually getting really good. There's like four different Chinese models now that are quite good. We actually just released one of them in our agent last week, and we're going to do all four for sure. They're actually quite good, and they're way cheaper than the Frontier models. So I think there's a lot of use cases where you'd want that, and we want to give that as an option. In terms of like the other, you know, So we think of our role as sort of taking all the best models that we can, creating really high-quality, state-of-the-art agent implementations where people can easily and conveniently get access to them, and then making sort of a collaborative workspace that is really good for humans and for the agents to coordinate on.

17:27Simon Last:I think it's something that's very needed in the world, and we're just trying to do it in a really tasteful, well-executed way.

17:34Sarah Guo:You were describing you need the index to make the agents good. You give the agents access to the tools that we humans have in Notion. How do you think about the structure of Notion and where it's useful or even not useful or relevant for agents, like blocks and databases and such?

17:53Simon Last:It's all still pretty useful, extremely useful. There's been a challenge to sort of, you know, we want to make it really convenient for the agent. I think that's a new thing that didn't exist. In the past, it was convenient for humans, and then we also made APIs convenient for humans writing code. Here's our API. So we essentially have a new customer, which is the agent. At first, that was definitely a problem. So, for example, our API uses this crazy JSON format for blocks that by default is crazy verbose and horrible for the agent. But we basically took on that challenge and designed just really convenient APIs for the agent.

18:36Simon Last:We created sort of a markdown dialect that looks like the default normal markdown, but it's sort of enhanced with all the notion blocks. And the models are really good at it. It works really well. So that's how it reads and writes the pages. And then for databases, we use SQLite. So basically it's just the speak in SQLite, which also works really well. So the default thing did not work really well, but then we just took that on as an engineering challenge. And I would say now we have extremely convenient APIs that the agents are really naturally good at.

19:11Sarah Guo:How did you understand or figure out what would make the API better for agents?

19:18Simon Last:That's a good question. Yeah, I would say it's a combination of just trying things. It's very empirical. So we're just playing around and like noticing, oh, it's not very good at that. Oh, that's way too many tokens. How can we make this smaller? And then a little bit of just like first principles thinking of like, you know, what is it the models are being trained on? And what's in their prior? What do they know? And what do we think it would naturally be good at? And like how does the agent loop work? And like what would be the convenient, efficient pattern for accessing these things? And so and then just, you know, a lot of playing around.

19:51Sarah Guo:I hear user research where the user is actually agent and then ongoing eval.

19:58Simon Last:Yeah. I mean, you just chat with it. The user's always there. It's ready to talk to you.

20:03Sarah Guo:Yeah. Actually, that is wonderful where you have infinite access to it.

20:06Simon Last:You have infinite access to it, yeah. And you can script and scale the access as well.

20:10Sarah Guo:I assume you have. Actually, I know you do because you walked in. You're like, hey, I need to get access to Wi-Fi. I need power. We can't block the agents while we're doing this. What do you have running right now? Tell me about your setup.

20:20Simon Last:I'm working on a new prototype, and so I have a couple agents. I'm working on that. And then, yeah, my setup these days is just either Cloud Code or Codex. I like the CLI tools. They're super simple and work pretty well. I'm pretty comfortable in the CLI.

20:39Sarah Guo:You don't need my generated game to teach CLI commands.

20:42Simon Last:It's a very cool idea. I would say, yeah, my whole goal these days is essentially to just have as many running as possible and to run them all the time. And, you know, so for example, like every night before I go to bed, I'm like, okay. Let's go, guys. Yeah, basically what I have to do is make sure that I've given it enough stuff that by the time I wake up in the morning, it will still not be done. And so I've maximized. That's victory. Yeah, that's victory. So yeah, like I've done that, I would say last five nights pretty well. My personal record is that I've had a coding agent running for I think it was 13 days straight without stopping and just just basically working

21:23Sarah Guo:through like tasks well well prompted yes I admit to having woken up in the middle of the night at

21:29Simon Last:least multiple times this week I'm just being like are you still going yeah I know yeah it's it it's kind of nerve-wracking I I always like there's always like I'll check it one last time before bed and just really make sure that it's still spinning what about on the notion agents like

21:42Sarah Guo:Do you have a workflow there that is core to daily work?

21:46Simon Last:Yeah, I mean, I use our personal agent all the time. So it has all the context about our company and everything that's going on. So, for example, last night I was asking you about how the custom agent launch was going and what the signals we're getting from it. We're super useful for that. And then I have many custom agents that are running. My personal favorite is I have an email triage agent. so it has access to all of my work and personal emails and it just wakes up every day and just archives all the stuff I don't need to see. I trained it over time to learn my preferences.

22:22Sarah Guo:Do you actually label data for it?

22:23Simon Last:It's pretty easy to do this actually. So all you have to do is you make the agent and then you give it access to your email and then you can make a blank page. It's like it's memory and you let it edit that page. And then you just say, okay, now go look at my emails and then interview me. Ask me which things, you know. So sort of it will like propose things that it thinks it should archive. And then you can kind of correct it. And then we'll use that to essentially generate like a list of rules about like what it thinks are correct or not. And so for the first couple of days, I was sort of like correcting it on things.

22:54Simon Last:After a couple of weeks or so, I dropped the approval entirely. And it just automatically archives all the things I need to see now. Wow.

23:01Sarah Guo:It's a lot of trust.

23:02Simon Last:It completely solved my email problems. Because for me, like I don't use email that much for work stuff. Like it's mostly in Slack. 95 % of the personal emails and work emails that I get, I don't need to see at all. And so it's just a waste of time. And so it completely solved that.

23:18Sarah Guo:So now I need my inbox. It's like only stuff I need to see.

23:21Simon Last:I've got lots of custom agents running. There's another one that I built that triages all internal feedback and bugs. So we have a Slack channel where basically people just post random like product feedback and bugs. In the past, it would sort of sometimes get answered but then sometimes like haphazardly get ignored just because, you know, there's so many teams working things. So its entire job is just to route it to the right place. And it uses a similar sort of like memory pattern where it sort of learns on the fly where it's supposed to file bugs. And then over time, it's built up like hundreds of rules that it just sort of like learned over time.

24:04Simon Last:So, for example, there's a bug about the mobile app and there's the route to the mobile team and then a file a task in their database.

24:10Sarah Guo:Do you look at that, like the generated and updated memory? Because it's legible to you to say, like, did that make sense to me?

24:18Simon Last:I think I did it at first. But then sort of once you trust it's kind of working, you kind of ignore it. And then if it ever breaks, I'll go fix it. It'll break every now and then. But the benefit of not reading your email is here.

24:33Sarah Guo:Yeah, just not Rita. So yeah.

24:35Simon Last:Yeah. I mean, generally, I would say, yeah, the general pattern I follow is sort of I build it as a prototype. I have it in sort of like an approval mode where I'm sort of, you know, watching it closely. And then but then after it runs a bunch of times, you kind of trust that it's working.

24:50Sarah Guo:Is there anything you do internally at Notion to make sure non-technical teams have the intuition for how to build agents or how to like express that productivity too?

25:01Simon Last:Yeah, that's a great question. I mean, we do sort of workshops and hackathons pretty frequently. So like, for example, like a month ago, I did a hackathon with the people team and sort of got them. The people team has been amazing. They're actually one of the highest adopters of custom agents. Cool. You know, they do all these kind of workflows in like Slack and Notion, kind of like manual work like that. And, yeah, I would say, yeah, like people are super excited to try it and sort of like maybe just need like a little bit of a push in terms of intuition and like getting them started. But then, honestly, I've been super impressed.

25:35Simon Last:Like I think the concept is like kind of intuitive, sort of like once you get past sort of a little bit of the technical barrier of like what is a prompt and like what is the agent and how does it get triggered and woken up and like how does that even work? But then once you sort of get past that, I think it's actually a very human-like interface.

25:54Sarah Guo:Yeah. Maybe the biggest barrier is actually just getting people to try and assuming it's going to work at all. Right?

25:59Simon Last:Yeah.

26:00Sarah Guo:You and Ivan originally met on the internet, Tools for Thought community. It feels like, you know, the tools we have for thinking are very different now. Has your, like, core conception of Notion changed over the last few years because of all the AI stuff? Like what thinking does the tool do for you? Should agents do for you? What do you get to do?

26:22Simon Last:Yeah, I mean it's, I would say, changed quite a lot. I mean, broadly speaking, before AI, our goal was to create the best tool for humans to directly perform their work. And then now the goal is to create the best tool for humans to manage agents to do the work for them.

26:44Sarah Guo:That's a big shift.

26:45Simon Last:That's a pretty big shift. It's pretty fundamental. But it turns out that you need most of the same primitives. You actually, all the primitives that we built are actually still extremely useful. It's more that we just needed some new primitives, like representing what is an agent and how does it interact with your pages and databases. But you still need the same primitives. You still need a document. It's an unstructured way to write stuff. Agents love to write markdown documents, so it's still very relevant. and you still need a database. You still need structured data. If you're working with your swarm of like 100 background coding agents, you don't want to have 100 chat threads.

27:24Simon Last:You want a Kanban board. It's the same as before.

27:26Sarah Guo:Makes sense. You still need the coordination structure. What is one thing that, just because you're ahead on this stuff and then trying to figure out how to bring Notion and then users along with you, what is something that's really changed about how you personally like build even in the last six months?

27:46Simon Last:I mean, it's completely changed. I haven't written code since like last summer. I don't type code anymore. Yeah, it's completely shifted. I mean, we went from humans type all the code to like we're still typing, but we like tab complete to sort of like we talk to the agent and it sort of does little tasks for us, but we are still in the outer loop. and then now it's more like I design an end-to-end task that involves making some change and end-to-end verifying it. And then I'm just the outer verifier sort of like double-checking at the very end that it's correct and if it's going off the rails, kind of like monitoring it.

28:26Simon Last:So it's a complete shift. I'm now like the agent manager instead of the coder.

28:33Sarah Guo:Amazing. Well, thanks, Simon. This has been a super great discussion about how we're all going to become agent managers

28:38Simon Last:and hopefully in Notion. Cool, yeah.

28:44Sarah Guo:Find us on Twitter at NoPriorsPod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.