How I run autonomous coding agents from my phone with OpenAI Symphony + Linear | Alessio Fanelli (Kernel Labs)

0:00This is my favorite positive outcome of AI, which is small business creation. Just the ability to like intersect the human world in a way that has been historically very inefficient has been a quality of life improvement for me.

0:12Alessio Fanelli:You know, my dad, their business, they deliver fish to restaurants. They got like this freezer with the frozen stuff and like somebody's going out there with like the pen and paper every morning, kind of like writing down what's there. Sometimes they're like, oh my God, we're missing like three tunas or like we're missing a box of shrimp. All of that work now can easily be automated. even just with the Meta glasses. And you have another use case, which is the use case that my nine-year-old wants to see. So let's do our Pokemon card by AI use case. So I use Codex for two things. The first one is like getting the PSA certificates to keep track of a specific number for each crate.

0:47Alessio Fanelli:Then the next thing I'm working on is when you go to like all these trade shows, people are coming to you, they're selling you cars and you gotta price them in real time. That whole process is super inefficient because people are like searching each car manually, like on eBay or like DCG Player, getting the number. You can actually use AI to save clock time for real people by doing these things autonomously.

1:09Welcome back to How I AI. I'm Clara Vo, product leader and AI obsessive here on a mission to help you build better with these new tools. Today, I'm speaking with Alessio Finelli, founder of Kernel Labs and co-host of the Latent Space podcast. He's going to show us how he uses OpenAI's Symphony plus linear to automate all his engineering tasks and how he has Codex Goal shopping for very expensive Pokemon cards. Let's get to it.

1:36Alessio Fanelli:Quick word from today's sponsor, Firecrawl. If you're building with AI agents, you've probably hit the same wall. Your agent needs data from the web, but the right pages are difficult to find, buried in JavaScript, or blocked behind logins. Firecrawl is a web data API that lets agents search, scrape, and interact with the web at scale and get that clean structured data they can actually use. Over a million developers, including myself, build on it. It's open source and it's free to start. Stop fighting the web for data and start powering your AI agents and apps with Firecrawl at firecrawl.dev.

2:17Use code HOWIAI to get 10 ,000 free credits today. I'm excited about what you're going to show us because I think we have heard a lot of people talk about orchestrating many agents autonomously across a project. We actually haven't seen many people do it. I still see a lot of prompting, even if you're prompting into a loop or a goal or something that spawns sub-agents, people are really still human in the loop. And so I'd love for you to tell us how you came to this point of doing more autonomous management of your agentic tasks.

2:52Alessio Fanelli:I started this podcast called Latent Space three and a half years ago, and my co-host, Wix, he had built this thing called Small Engineer at the time, which was kind of like the first autonomous coding thing. Over time, it's always been a cool demo, but I feel like the models were not quite as good to really do longer running tasks. They definitely changed, you know, end of last year, and I think everybody kind of feels the same. And what really clicked for me was like starting to move away from being an agent prompter to kind of be an agent manager. And that has kind of taken a lot of different ways.

3:23Alessio Fanelli:So the first thing that everybody tried was kind of like the Kanban board. You would kind of put all these things in there, move them back and forth. What I found is that it was hard to get two, three, four turns through that. Like it was easy to get to the task and kick it off, but then it was hard to intervene on it. And also like having it local just didn't quite work. So the big thing for me was moving away from kind of like local runtime to having it in a VPS in the cloud and then having different channels to talk it to. So you can kind of like text the agents, you can use linear to talk to them, you can prompt them directly in the shell.

4:01Alessio Fanelli:And this is also like something I guess like in the last month, you know, Codex also added, Codex Mobile, Cloud is also adding kind of like the mobile management, but I'll kind of run you through what I do and then maybe people get some inspiration for me great and i will really benefit from this because i'm staring at my four mac minis over here so i'm still running locally um and i just come downstairs and like kick them alive every now and then right so i'm you maybe you'll convince me to move all this to the cloud let's see yeah so um i try to use some more fun examples right in like the meta one so i own a car store in san carlos called merleon games um and so one of my interests is trading cards.

4:43Alessio Fanelli:So my setup is I have this thing called ZOO. ZOO is basically like an agent plus a VPS. So this machine, for example, is like, you know, 32 giga RAM per course. I have all my coding agents pre-logged in here. And you can also use some of the open source models if you want. And what I have here is kind of like the sourcing thing where you can basically use it as your own server. on this, I have the OpenAI Symfony setup. So Symfony is basically, I mean, you can kind of look it up for a better description, but it's kind of like a loop for turning issues into coding runtime and then having kind of like linear as a source of truth for it.

5:25Alessio Fanelli:So what I have on my linear, I basically have all these different projects. So this one is PowerBuyer, for example. And I work on it sometimes in TrueSymphony, Sometimes I work on it through codecs directly or clock code directly. And if you go into any of these things, basically what you see is you have the original task. So this is what I wrote as the initial spec, which is pretty simple. Then I'll basically move it from here to to-do. So this tells Symfony it needs to work on it. What Symfony does, it creates a codecs workpad. So the agent kind of makes a plan on how to implement it. And it has a plan, it has an acceptance criteria, different validations.

6:05Alessio Fanelli:and Symfony has a file called workflow.md where you basically explain what it should do for these. This will kind of go to work and then eventually move it to human review. So what you can do here is review it on GitHub and you can add, let me open the PR and show you. You can add all these different comments, you know, I guess now the result is like too long, blah, blah, blah. And then you just move it to rework. So once you move it to rework and then we'll kick one off while we record this. It creates a rework checklist. So it goes through all the comments and it's like, okay, these are all the things that went wrong, kind of addresses them.

6:46Alessio Fanelli:It tells you how to address them line by line, moves it back to rework, from rework to done once it gets merged. And that's kind of like the flow. I don't really look at the traces one by one. I just kind of direct the agent to work on it. So whenever I'm, even if you're outside, right? Like I might be on my phone and I'm looking at something here. Let's say here we're like, hey, this is kind of like noisy. What I can do is like I can create a new task as I clean up bringing stable. Let's remove the spread column. It's too noisy. Let's also make the set name clickable. So I look at other cards.

7:34Alessio Fanelli:And here I'll simply put it in to do, create issue. And then each of these symphonies has its own dashboard on it. So this one is TCG by our buyer. These are previous tasks that are run. So one of the things I'm also trying to do is try and figure out how much is software going to cost to build. So I think people understand the idea of like the agents fried it. But sometimes it's hard ahead of time to know how many tokens it's going to take. And so it's hard to price and understand what it's actually worth doing. So as you can see, most of these tasks are kind of like, you know, 15, 30, 60. But then this one is like 221 million tokens.

8:13And so you can kind of go back here and be like, okay, this task was how to make it deployable in Vercel.

8:20Alessio Fanelli:So, you know, this whole thing was just not working. It was originally built. It's kind of like a local thing. so it had to like you know revert the storage kind of like you know change of the request to render blah blah blah so this is like quite a big task so it kind of makes sense that it costs a lot of tokens um and so from here you can kind of start to think about how in the future can I make these more efficient by either adding more checks or adding better descriptions or better tooling so the task we just created is it just kind of went live so here you can kind of see you know obviously there's usually like you know four or five of these in different projects that are running so i don't really want to see the whole thing but i just want to glance and i'm sure i can make this ui a little prettier maybe once they give us fable five back um that will be good enough um but so this is working right and so in a little bit this is going to go from from in progress to like human review and once it goes to human review then we can kind of look at the Vercel preview and we can make comments on the code and on the front end and kind of move it back.

9:24Alessio Fanelli:But I could be doing this here. I could be doing this on my phone. I could be doing this anywhere, really. And to kind of put it to the extreme, I had let me see if I can find it. I created this project called PyQ, which was basically like putting your repo plus the Py agent in a VPS and then anybody on the internet could send you, I think I but it's in a different project could send you like a coding task to your product so it's almost like in the future you know and I think some people now have this idea like a request for prompt instead of like request for for requests everything is just how do you transfer context between people so before we move on to maybe another workflow what you just showed me was look you can just create a linear project for any one of your code bases you can integrate that with Zo and with Symfony.

10:19And then all you're doing is really tasking linear as sort of like your state machine for all the work that needs to happen in your code base. You can manage that on linear from your phone. You can manage that from your desktop, from the web. And you don't really have to worry about the framework of how that task gets, you know, broken down, how it gets implemented, even how your comments get reviewed. That's all set up. And I just wanted to share for people symphony um is something that open ai open sourced as sort of a framework for autonomous runs so it's it's just a very opinionated way to do this work and it basically does what you just showed it monitors a linear board spins up agents when it gets assigned something um and then you know you can you can land it in a pr and it gets marked as done how simple was it for you to like actually set up Symfony?

11:10Because I think people look at these things and they're like, okay, that makes sense. But what do I actually do with this GitHub repository? And I know they have these two options here, which is like basically tell your coding agent to build it for you, or there's this reference implementation. How did you actually implement Symfony?

11:26Alessio Fanelli:Yeah, I took the Elixir implementation that really like the core things to change are like the workflow.md and the main folder that kind of like explains how to do it and then build the UI. So the Symfony itself doesn't have a visual UI for it. And I also don't think it has the same, it doesn't have by default the ledger for token usage. So it's only like a state monitor. It doesn't actually look at like, you know, how much have you spent per task and kind of like all these different things. But yeah, I think like overall the harness, you know, it's pretty straightforward. I think like the reality is like, how do you build tools for it to be more effective?

12:10Alessio Fanelli:And that's kind of like one of the main things also at Kernel Labs we've been working on. So we built this other thing called Glimpse, which is kind of like a Playwright extension that coding agents can use to take screenshots, to do visual diffs between screenshots, take videos. And so it's almost like, yeah, how do you let these runs kind of go longer and longer? So it's less about the orchestration itself and like the tools you give it to keep going. versus coming back to you with the human review. And I think that's also why it's so important to like keep track of how many tokens and how much time it takes, because it's usually like, directionally it explains you how many issues it ran into.

12:50Alessio Fanelli:You know, so if you expect something, you should start to have at some point, some idea of like how many tokens you think this will take. You know, is this like a 10 million token task? Is this like a hundred million token task? And if the reality is very far away from your expectations, There's probably something in the tooling layer that you can do to improve. So that's really where most of the kind of like value comes from for people. Yeah. One of the lessons I want people to take away from this is I get asked all the time, like, Claire, how do we build our own agent orchestration platform?

13:23And like they send me these like giant, very complicated documents and workflow diagrams. And, you know, pointing them to just like Symfony's spec MD, which just describes how the system is supposed to work. It's in natural language, and it just is very prescriptive about what the primitives are of this workflow and what to store and record and how to move things forward in sort of the software development lifecycle. It's very long. It's very detailed, but it's literally just a markdown file. And I think people kind of over-engineer at first what these things can be. And ultimately, the power of LLMs, especially these newer models, is you just give them a spec for how they will work.

14:09And they will, you know, they will lock to that spec when executing whatever you hand it.

14:16Alessio Fanelli:Yeah, I think everybody just wants to have the magic skills file that does everything for them on like the magic MD that solves their business problems. I think the reality is like now more than ever, like small sentences of like very, a lot of weight, you know? Like for example, I built this Work3 manager before the coding agents themselves had it. And in every AgentsMD, I was like, you have to use the Work3 manager. And now sometimes I forget I had in some projects and then I start a task and it's like reinstalled. I'm like, you don't need it anymore. But like, because I had that line, every time it's not doing that.

14:58Alessio Fanelli:And I think a lot of folks have been talking about purging your markdown files now every few months. I think that's something that obviously makes sense. I think the models themselves also have this like tendency to like add rather than remove. So if you're like, hey, you don't need to always use the Workthrough Manager. Instead of removing that line, it's going to add a line to say that you don't have to use it all the time though, you know? And now you're kind of getting more and more confused. So for example, the skill.md, it's not super descriptive on what to do, but it's like, hey, this is where you put the symphony.

15:36Alessio Fanelli:This is how you should architect it. So every symphony instance has kind of like its own name. It's got the repo that you're working from. It's got the workspaces for each task. It's got logs, which include the token usage. And it's got the state of this run. and then these are the things that you need. So if you don't have them, ask me. Otherwise, don't ask for stuff. These are the exact commands. These are the flags. But I'm not really telling it what to do and what to use. I'm just saying these are like things that you got to keep in mind and then you kind of let the model work. See, this is like a good example.

16:14Alessio Fanelli:I think it just added this today when I was adding a new project. It's like this one is already registered here and it's like this should not be in this file like it should look up every time it should just search every time what's already there you know those are like all examples of like if you just let the models kind of do their own things that you just end up with this like very um weird things okay so red diff your skills and red diff your markdown files and get some stuff yeah i feel like the create skill thing that the codex app for example added i think it's like a great idea, but I think for a lot of people, it just puts them in a lot of trouble because it's like they're not very descriptive in the skill itself.

16:56Alessio Fanelli:And then the model is like very focused on following the skill. And so they're actually doing themselves a disservice a lot of time. This episode is brought to you by Jira Product Discovery. AI has made individual PMs incredibly productive, but multiplayer mode is where it still breaks, getting everyone aligned on what should actually get built. Decisions live in a markdown file from last week. The roadmap's a spreadsheet no one's looking at. Jira Product Discovery is where teams actually decide what to build, capture ideas, prioritize them as a team, and share a living roadmap everyone works from.

17:33It's powered by Atlassian's Teamwork Graph so it can pull in customer feedback, what your team shipped, plus your goals, and suggest what to build next. And when a decision is made, you can hand it off straight to Jira so a developer or even an agent can pick it up and start building. Teams at Canva, Deliveroo, and Toast already use Jira product discovery. Join more than 25 ,000 teams at Atlassian.com slash how IAI. Start building the right things together. Do you feel like you're getting a lot more done on this orchestrator? Is it just like better ergonomics and that you can manage it on the go, what's the benefit you feel of using something like Symfony?

18:17Alessio Fanelli:I think it's definitely, I don't know about the getting more done. I think, you know, in the limit, you get more done. But in practice, it's like you can't really like stay on top of like 100 new things a day. The thing that's really helpful is like having the full S3 of one task in one place. So because it has the original spec, it has the first work pad, it has the rework work pad. Every time you're like, how did things go wrong? You can kind of pinpoint where that was. And then you can use that to inform, you know, your agents MD in the future, like the symphony workflow.mp. Versus if you're just using codecs, it's kind of, it's really hard to search through conversations, you know?

18:56Alessio Fanelli:And so everything is like, how do you shape the context? You know, like symphony is just a way to shape the context. It's not giving you any new capability that you wouldn't have by using the coding agents directly. It's just helping you wield it. I love that. So, okay, you have shown us how you're doing sort of this orchestration of agents, or at least like workflow management of agents. Again, like, let's make this much more accessible for people. This is just a workflow to manage your agents, especially around coding. And you have another use case, which this one is the use case that I wanted to see.

19:31This other one is the use case that my nine-year-old wants to see. So let's do our Pokemon card by AI use case.

19:40Alessio Fanelli:People think there are a lot of startups. There are way more Pokemon cars than there are ever going to be startups to follow. So one thing, the reason why I built this power buyer thing is for us to kind of keep track of, you know, inventory we want to buy and things like that. Yep. So I use Codex for two things. The first one is like getting the PSA certificates to keep track of like, you know, basically Pokemon cars, you can get them graded by PSA. and then there's kind of like a specific number for each crate and that's available through an API but you need one certificate number here to start from there's obviously no way to just download that so what I have is like you know I have codex pursue goal fill out the certificate number for every card that costs more than a thousand dollars so I give it browser access for example here and it's just you know going on on the internet and it's looking at things and It's like downloading the images, extracting the number from it.

20:40Alessio Fanelli:And then what you can then do is like, you know, use the eBay PSA premium. Let's find some underpriced cards. Okay, so is that a skill? Yes. The skill is basically telling you how to figure out which cards are worth looking for. And then it just says how to bash them, you know. So just do five per batch. You don't want to get like, you know, captured by eBay or stuff like that. And then there's also things, for example, there's different grading companies. So you might have, you know, a PSA 10 is the same as, like a PSA 9 is the same as like a BGS or CGC 10. And so all these different rules, you might want to have in there.

21:30Alessio Fanelli:And so here just, you know, through the API is looking up from our PowerBuy Air software, the cards that we want to look at. And then it's using the in-app browser for eBay. Let's see if I can open it here. So now it's going here and it's like looking at just searching. And these are all like, you know, $10 ,000,$20 ,000,$50 ,000 cards. Like you don't really do this for$5. And so now it kind of goes in here and, you know, you'll see it then. tell us, you know, this card is underpriced or this card is, you know, worth buying right now. And yeah, this is like, you know, a good example of like, you can build anything, but then at some point you got to have some business outcome or like some way to make money with the software that you built.

22:21Alessio Fanelli:And trading cards is actually a great example of like the more money you can put to work, the more money you can make because it's like an inventory-based business. So you can only make as much as you can sell. And so having this to help us automate looking for some of these like higher value cars, for example, is like super useful. And then the next thing, you know, I'm working on is, and this is, I guess, getting in the weeds of the business, but when you go to like all these trade shows, people might see them on kind of like Instagram reels or whatnot, where people are coming to you, they're selling you cars and you got to price them in real time.

22:56Alessio Fanelli:the whole process is like super inefficient because people are like searching each car manually like on eBay or like TCG player, getting the number, blah, blah, blah. And so you're actually losing a lot of money. So when people talk about, you know, AI response time, I think for these long running tasks, it's actually not that useful, but you can actually use AI to save clock time for real people by doing these things autonomously. So yeah, it's just been fun to try and apply it outside of like, hey, I'm building the tool for you to build the tool, to build the tool so that hopefully somebody down the line does something that is worth for someone to use.

23:38Alessio Fanelli:So yeah, for example, we got this Amplon right here. Let's see, I guess it broke the lane. See, even it happens to the best of them. This is something actually, it's funny. people talk about software automation but Codex has this kind of like the dollar sign it's like tied to skills so sometimes when you're like linking with the dollar on the thing it pre-fills it to a skill so right now right now it's looking for a skill instead of a URL you see like TCG Power by ear HTTPS so that's a good that's a good example of like you know even these small things in software are sometimes broken yeah but yeah just this is overall one trend where it's like there are a lot of businesses that are based on kind of like highly heterogeneous data that have been impossible to scale with software because before you have kind of like something as malleable as an LLM that can go through these things it's really hard to use even like text or image classification for these things and so yeah I think you're gonna see a lot more of of these businesses like i think the same i i forgot who it was but the same thing is happening with like vintage clothing for example um some people are doing something similar for for desktop um because again you see it all the time right oh my god i went to goodwill and i saw this i don't know product back that was like in the goodwill thing and um this has always been kind of like a mismatch between like human bandwidth and like the information that is coming from them?

25:14This is my favorite one in that I think it enables what a very positive outcome of AI, which is small business creation. And I think, you know, this business of clearly trading cards are a huge business, but no, what you're able to create this bigger business because you have the leverage of AI. And this is something that a human would have to manually do. And just the limits of time space and human cognitive capacity means you probably unable to capture as much of this business as you are today. I also think I love this use case because it shows where AI helps you intersect the physical world in a really effective way.