Let's build something phoenix.new (Friends)

33:31And it's like, whether you've never programmed before or you've never programmed Elixir before, you can get value there. Whether you created a framework at the far end, you're still going to be able to use these things to do the tedious work or feature work. But I try to tell people that the seasoned developers is like, for me, LLMs, the discourse is like, everyone's like, oh, it's all AI slop, which I think is a silly argument. But it's not AI slop for me. it's like these lm this the code that the lm generates that artifact is a starting point and the discourse for some reason for people that are on the negative side seems to be like they treat that thing that falls out of your chat gpt as the artifact that you ship to production but it's like no these things are just a starting point right so now it's like instead of having you know myself write out this hundred lines of gen server code it's now just like this really intelligent code generator that's my starting point it's not what i like to then to shift to the production.

34:26So I think the discourse is flawed, but I think that at all, at all levels of the experience stack or programmer hat stack, you're going to have people getting value out of, out of tools like this. The AI slop is the blog post that nobody wanted to read. And its only purpose is there to attract attention. So you can sell some advertising or something, or the, the essay that you spat out because you didn't have time to actually write your own, like that's slop. Yeah. I mean, I like to say we've all been sloppy vibe coders, right? It's just now way easier. But like the people copy pasting stack overflow and the people that ship that trip to production, like now they can do that more easily, but it's like, but those people were already writing bad code and not carefully considered prior.

35:09So that's going to remain true. It's going to be easier to, for those folks to like get something into production, but it doesn't change the fact that like you can't, I don't know. I feel like it's more gatekeeping than anything else. Folks are, are throwing that, that term around. Well, friends, it's all about faster builds. Teams with faster builds ship faster and win over the competition. It's just science. And I'm here with Kyle Galbraith, co-founder and CEO of Depot. OK, so Kyle, based on the premise that most teams want faster builds, that's probably a truth. If they're using CI providers with their stock configuration or GitHub actions, are they wrong?

35:46Are they not getting the fastest builds possible? I would take it a step further and say if you're using any CI provider with just the basic things that they give you, which is if you think about a CI provider, it is in essence a lowest common denominator generic VM. and then you're left to your own devices to essentially configure that VM and configure your build pipeline, effectively pushing down to you, the developer, the responsibility of optimizing and making those builds fast, making them fast, making them secure, making them cost effective, like all pushed down to you. The problem with modern day CI providers is there's still a set of features, a set of capabilities that a CI provider could give a developer that makes their builds more performant out of the box, makes their builds more cost effective out of the box and more secure out of the box.

36:39I think a lot of folks adopt GitHub Actions for its ease of implementation and being close to where their source code already lives inside of GitHub. And they do care about build performance and they do put in the work to optimize those builds. But fundamentally, CI providers today don't prioritize performance. Performance is not a top level entity inside of generic CI providers. Yes. Okay, friends. Save your time. Get faster builds with Depot, Docker builds, faster GitHub action runners, and distributed remote caching for Bazel, Go, Gradle, Turbo Repo, and more. Depot is on a mission to give you back your dev time and help you get faster build times with a one-line code change.

37:19Learn more at depot.dev. Get started with a seven-day free trial. No credit card required. Again, depot.dev.

37:31well should we try to vibe code something i got an idea i got an app idea i want to see it okay i'll screen share this and for our listener who's not who doesn't have video uh have no fear we're not going to like leave you behind or something yeah nothing better than live coding in a non deterministic way uh i did this on stage at elixir comp eu where it's like you know i always like to like live code which has some level of risk but then you're like you know you're live generating something that you, you know, it's just a random number generator ultimately. So let's do it. Let's see what happens.

38:01All right. So here's my app idea. It's like hot or not, but for code functions. So like imagine Chris writes his version of quick sort, right? And I've got a better way of doing it. And so we both enter our quick sort function and then other people vote. Like, is this hot or does this not? Is this good code? Let's do it. So I have phoenix.new open over here. or would you like to build, pick one or type your own? Of course, you have a video out there, seven minutes on the to-do list. So we're not going to do that. How do you suggest I prompt this thing? Just tell it what I just told you or get more specific.

38:36And just what you said. So here's the remarkable thing is people, the intuition and the tribal knowledge is you gotta be as specific as you can. The remarkable thing is like in terrible English with typos, you just ask for the thing and the agent has intuition or it will give you reasonable questions. like someone asked it about making like a mashup of communication providers like mashing up sms and email and it was like well what would you like to use twilio or singer like would you want a graph ql api or json so it's like let's give it like i mean do what you want free form but i don't think you need to actually spell out anything just tell it exactly what you told me so i said let's build hot or not but for code you put your code in and people can vote it up hot or down not Good enough?

39:22Should I be any more specific than that? Whatever we want here. Let's see what it does. It's going to hype you up. Yeah, all right, man. It says great idea. Thank you. Now you're starting to stroke my ego. A hot or not for code where developers can submit code snippets and get community feedback. Here's my high-level plan. Oh, it's a 12-step plan, 12 to 14 steps. And so it's going to give me 11 steps with some features. Submit code snippets, blah, blah, blah. Real-time voting. There you go. There's your real-time. Now, did you system prompt like be real time by default if they don't specify? Because I didn't say anything about that.

39:56It's basically like, you know, a Phoenix framework has pub sub, build in presence, whatever. So like anything that makes sense to be real time should be real time. That's more or less the gist of it. Gotcha. So I'm not being very discerning. I just said, yes, great plan. Please continue. And now it's going to ask me if I want to do dark theme, minimal theme, vibrant tech, professional corporate or something else adam you got any good cool theme ideas it nails tron when you ask but the cool thing with these choices here is like we just go on there i was uh i was tired of yeah tron is always great i was tired of typing yes and no to the agent so i was like in the system prompt i was like anytime you idle give the user a choice you know example yes no and it started producing stuff like you see there and you're like what like it's just remarkable what you can get out of these things without trying.

40:47It's just like, I thought it would be way more trying. Right? I wanted yes, no. And then it's like, what did it type here? Would you like a GitHub dark style? You're like, what? Here's six options. They're all good. Yeah, so it's going to write out a plan in a plan MD file. So it plans out its own work and then that remains in context. Now your server's running so it compiled and built it in that amount of time and you get that live preview. and then you get that url as well that you could share with uh it's private by default but you can toggle it public and anyone could visit that phoenix server now if you toggle it public all right i'm going to paste this to you guys sweet riverside chat oh yeah and once this goes toggle the toggle of the public in the uh top right there by your it shows the url in the top right here another public uh little toggle there by uh by the pink text purple text left right here there you go all right so i made that so it gives me a phx.run url that i made public paste it to you guys meanwhile it's coding things right there was a syntax error and it's linting the code as it goes and so just like we can see the browser here it actually has its own headless chrome browser so it's able to visit the page as a human would with a real browser see JavaScript errors, and then it can also interact with the page.

42:09So if we're lucky, we'll get to a working hot or not, and it will post its own code snippet to the app, and we'll see it in real time by actually driving the browser. That would be amazing, right? We'll see what happens here. So it's writing the... Oh, it's giving us a... Yeah, it's going to start with a static design here. So this is it, like just writing a... Let's see. Syntax error. It's fixing the syntax error currently. Boo. My guess is we'll see if it actually is this issue. So someone reported this. If it's trying to write a code example on the page, it's going to use curly brackets. And one of the open issues internally is they're used to Elixir Heeks files.

42:50Like our curly bracket is a reserve syntax. So like if you try to put a code sample in like a code tag or a pre tag, Heeks throws a compilation error. And this is like the same thing that trumps up people. Any time people want to do this, they go to the forum and they're like, how do i write this uh so you actually have to annotate with a phx dash no dash curly interpolation and uh i have a branch where yeah i'm sure it's hitting this um so we need to actually tell it um hold on let me see so it's amazing we hit this let me i think i'm a nerd so i pick a code app anything no no it's fine this is this is good um okay so it's one of these edge cases that like it again trips people up where they're like how do i do this i can't interpolate but eventually that agent will probably start trying to um to like random like trying to interpret it by like stringifying the brackets hold on let me i'm going to paste this to you because it's a really long it fixed it did it really oh okay i might we'll revisit this later my guess is it like put it in like uh like interpolated the strings like it did something ridiculous that works around the issue does it ever stop so us meatbags can keep up of like Cause I was going to read what it was doing up there and I'm afraid it's going to miss something now.

44:04the thing is, uh,

44:08slow down, buddy. I'm trying to keep up. Here's the thing on the newer models. They're not as good. So like Gemini flash is fast enough where I get existential vibes because like we're following this now and we're like, Oh yeah, it's working on the, the context file here. But like Gemini flash is so fast that you lose. It's like, but like now I'll do this and I'll do this. And you feel like you're in the way now, granted, it's not the quality is terrible it doesn't give you working apps but from the first time it doesn't give you working apps yeah i can see i can see the future where where i'm like i'm just sitting here as a as a meat bag and i'm in the way right like it's just i can't even read what you know follow what it's doing and that's like where we're headed right it's like it's it's we're not there look at this okay so yeah so it made a static uh heeks file uh so none of that's functional and now it's going to actually use that and write a real app around that static file.

45:01So now it's writing the live view here and it's going to start doing the live view pub sub and everything. But it gives you that, like, you know, it hit the syntax errors, but it gives you, we wanted to give people like the early feedback of like seeing the app, like what it's going to look like versus waiting, you know, the whole time. And then - So you tell it to do that, like build a static version first and then make it live. Yep. And it's also helpful because if you wait till the end in like something about like it's good i mean just like humans start with a mock-up right so it's like for the same reason you don't want to have a consultancy just like here's your finished product and you're like oh i really didn't want you know there was some fundamental difference of the design that would have made the code much different it makes a lot of sense to work in the same way like you know a person would right look at that yeah see it it nails tron uh pretty pretty well it does look kind of cool very trony submit your code let the grid decide Yeah, and then it picks copy.

45:54So here's the thing. Like, you know, these LLMs come up with copy that makes sense. And all you said was, like, the word Tron, right? That's what I said. I'll do it Tron style. That's all I said, Tron style. Okay, so it's using its own web browser now. So you said it visited the app. So there, like, it would have caught any JavaScript errors. It actually saw the app. So it's, like, this was also one of the, like, special things that I feel like made it error, like, really good error correction. because like not only can it see like elixir logs and actually like try to visit the app and if it broke the js build it would see that too oh it's going to try to post something look did it work uh close the terminal now on the little right right hand side x um it tried to write code did it actually add it um there is an issue it says excellent our code reader is working perfectly let's set the functionality by or test it by submitting a sample code there's an issue the web tool is trying to fill a select drop down with text instead of selecting an option.

46:49Let me try a different approach. So if you expand, I see the issue real quick. There's a little expand button on that message right there. Yeah, yeah. So you can see it actually wrote JavaScript to eval on the page. So it actually tried to post something for real. So like within its own headless browser, it was trying to... Oh, wait a minute. The Fibonacci generator, is that what it wrote? It's trying to write a Fibonacci generator. It did. So the recent submissions here are... So it used its browser. I think it, there's probably a handle info. I can't see the code. My guess is it blew up, like maybe something in the PubSub crashed it, but it actually interacted with its page by writing its own code in JavaScript to run on the page.

47:33Bam. So are you going to see in this too? So if I vote this hot, are you going to see my update? Wait, hang on a second. Hold on, I got to open this now. Oh, look at all those hots. A quick sort, there's a Hello World and Elixir. Oh gosh, there's more? There's a quicksword algorithm. Oh, there's 42, so this is hot. I just hotted Fibonacci. Is it 43 now for you guys? Yes, I'm going to say not. It's 44. Oh, my goodness. Who's doing the nots? Not, not, not. So, yeah, so that's, this is, I mean, other than the syntax error at the beginning that it got caught up on it. Get out of here. This is Phoenix New, right?

48:06Oh, my gosh. So I'll try to post some code here real quick. I just want to see if it fully works. Let me grab. Because we didn't really follow. oh, you know, we just let the agent figure it out while we were like, whatever, do whatever. I'm curious if it's going to show up on everyone's screens or not. Yeah. So we have one, two, we got three submissions. And whenever you submit. I'm going to say fib copy pasta, because I'm going to copy that one and repaste it. So that's fib copy pasta in Python, paste. And upload to the grid. There is my fib copy pasta, but doesn't have any votes. You guys want to vote for it?

48:46I'm hot in it right now. Did it show up on everyone's screens? Yeah. I got one hot. It's at the bottom. Three hots. It's at the bottom. So there you go. Fully real time. The agent actually used the app. Successful run. Yeah. Yeah. So now it probably offered up some ideas to continue, but this is basically like where, yeah. So it excels at like getting here, right? So like the vibed app and it will gladly continue. And you could add features, you could add user auth. getting to this point was where we were like okay like this is we wanted to nail the like does it deliver some from prompt to some compelling like full actual application experience so that's all it's sqlite by default but that that's persistent to the database and that's something you could say i wanted to take this and run with it yep what would i do so you can in the hamburger menu you can copy get clone and run that in a local shell and boom that's going to be proxied through the Phoenix app and it's like proxies all the way down.

49:46That request will go up to a fly proxy on some edge node. We proxy to the Phoenix app that you're using the chat, which we would then proxy with fly replay to your IDE, which has a reverse proxy that goes through get HTTP backend to clone that and then back up the chain. Right. So. Now could I start? And we have a Phoenix application as you well know. Could I just like give it. yeah there's a copy there's a copy get clone as well um that you paste in your local shell and it will show up in your id the VS Code id right here and that's where like uh I want to add next a pair mode so like it's it's system prompted examples are fully like vibe mode right so if you do send up an existing project where you want it to take more measured steps you have to like be explicit in your initial prompt like you know do this step by step and wait for further instructions but I I want like a toggle, right?

50:37Because people don't want it to just like go full ham all the time. So that's something we want to explore next. But like getting the vibe mode out was the real initial goal for us. And I think we've pretty well nailed it. So it's always exciting to see someone do something and have a good outcome. So it's pretty cool. Well, especially because it was on hard mode because it had to paste code into its own. Can we look at that? Although it fixed it already. No, no, it will be in the, well, it'd be in the Git history. but I'm curious what it did. Anyway, it's not too important. I always guarantee it was the interpolating the Hex because Hex is going to blow up when I lent it with a bracket error.

51:16And it's confusing to, like I said, the humans too because Hex can't tell you, oh, you're trying to literally add code, right? We just blow up like you fat fingered a bracket just like in your markup. So I'm just curious how it worked around it because it probably did not use the no interpolation. My guess is it added like some ridiculous interpolation of the literal elixir string of brackets or something. So here's where it finds the error. It says, I see the error was caused by unescaped raw code lines in the home in Heeks. I'll fix this by wrapping the code blocks correctly with Heeks safe sigils.

51:49Sigils? I don't know. Okay, yeah, that's exactly what it did. So it did inline elixir and then it interpolated some elixir code that returned the string of the bracket. So these agents brute force, that's not the solution, right? The solution would be like, you know, because then if you have a code block, you have like all these little strings of quotes around or brackets. And it was just like, whatever, I can make this work. I have the technology. So that's pretty cool. I have a terminal somewhere as well, don't I? Yeah. If you click agent terminal, that's the one that the agent. So if you like get logged there, you'll see every time the agent touches a file, it does a commit.

52:26so you and the agent both could like revert back to each file so one thing we also want to add is each of the file tools that it did will have a revert button so you can just we'll just do a get revert back to that state of each of these commits so the agent knows kind of like each each file shot any any given point as well there it is right there fixed syntax error by correcting html entity encoding and code blocks and so i should be able to just get show that and see the actual diff, which it's like piping through more or something. There it is. Well, that's not it. That is it right there. Did you ever hear about this theory of the monkeys?

53:08There was an experiment where they had a cage full of monkeys, and at the top of the cage, or like in the center of the cage, there was like this thing they can climb to get to the bananas, let's just say, right? And the first batch of monkeys, they don't know any better, right? So they climb this thing in the middle to get to the bananas because they want a banana. That's what monkeys want, right? Naturally, as a monkey would, it climbs and does. And that's not the way this place works. If you try to climb that, you get sprayed down and it sucks. You don't like it. And so they all learned that monkey climb, monkey get banana, monkey get sprayed, monkey get hurt, doesn't like it.

53:50OK, eventually these monkeys, they they get replaced with monkeys who only have ever been there. Let's just say now the monkeys, they only know what they know because it's trouble knowledge. And so they no longer ever attempt to do this. They although they've never been sprayed, they don't try to attempt to get the monkeys because I don't know why they just don't do it. And so the reason why I tell you all this is because we're looking at some really awesome Phoenix code and we have a Phoenix application. So we have this background. What happens when the monkeys don't care about the code anymore?

54:26You know, they don't they just don't know what to choose and the LLM chooses for them. And the taste making is no longer by the taste makers. It's more like this hodgepodge. Maybe it's good. Maybe it's not. You know, that's what I'm thinking about. Yeah, that's a good question. So I think like in the medium term, and I don't know what timelines, like I do think it's safe to say that like, you know, the Anthropic CEO said that like 90 % of code by humanity by the end of the year will be AI generated. And people like dunked on him for that. I think that's absolutely going to be true. I mean, if you just look at like, and again, these aren't like, it doesn't mean that like that's 90 % of code that a human didn't see.

55:06It's just like if I think about my own AI usage, right? I'll start with, if I'm running a DEF module gen server, that's being started by an LLM, and I take that and then use it. So then the LLM is generating, let's say, 90 % of my code today, but it doesn't mean that I just ship that. So I think that we're there in the medium term on we are going to be the purveyors of what's good or not, and we're going to be enhanced by it. But then long term, I don't know. I don't have a good answer to it. Like as these get better, does software become disposable? Which, you know, I don't know how I feel about that, but it's like, these agents are expensive today, but they're valuable enough that like people are getting an extreme amount of value, even the fact that they're expensive.

55:50So it's like, if it's an absolute pile of mud, which all software is anyway, if it's an absolute garbage, but it does what you want. And granted, I'm not saying like we're there today where you just like dispose it and whatever it can be crap. But I'm saying if that's where we get, software could be, by and large, disposable, where you just regenerate the thing. It gets to a point where it's unmaintainable or something and no one vetted it properly. Then it may just be like, well, we'll pay$100 and now we have our new app. So I don't know that that's where we're headed, but I could see it. Where it's like this Tron example, if the agent was 50 times faster at that, it would have taken us longer to write the prompt than it would be to get the app potentially.

56:31And if we get to that future, I don't know what happens because why wouldn't you just have this thing generate? We can talk about security and all the caveats. And I'm not saying this utopia is going to happen, but you could have an agent vetting it for security. And again, for better or worse, I feel like this is where we're headed. And I don't know what falls will hit, but it's clear that's the trajectory we're on. And I'm not saying it's all good, but it's clear to me that that's where we're headed. so i don't think it i don't think it helps by like just saying like oh well it's all slop it's going to be terrible i just think it's like it's helpful to acknowledge that um you know this tide is is washing over us and whether we like it or not it's like this is where we're going yeah i mean maintenance could become like just small rewrites and which i mean if you think about what refactoring is that is what you're doing like you're kind of rewriting a small portion and those portions could get bigger and bigger and so maybe maintenance becomes replacement when replacement is that cheap and easy and so you're kind of just like ship of dcse in everything yeah and it could even be like if you if you imagine like it's expensive now um but imagine you have a dozen agents doing a dozen versions of that and then you just pick the best one so it's It's like agents are going to eat the world, like I said, for better or worse.

57:57It's like I just see this future where instead of this Toronto example, you could have been given 10 options of that and chosen the best one. It's non-deterministic, but as they get cheaper and more efficient, now you have like 10 choices and you just pick the best one. So it's like it's just going to be more and more of this. And I don't know what that says about the future, but I think there's just going to be like more compute and it gets cheaper. So we do more LLMs. It gets cheaper. So, you know, it's just it keeps advancing the envelope of where you would just throw these things out of problem.

58:27And it's clear that that's going to happen to me. And I don't know if, you know. if that's going to be all unicorns and rainbows, but it's definitely where we're headed. It goes back to the conversation we've had around these parts over and over again, which is that skills become less important and judgment becomes more important. But to Adam's monkey point, you know, how do we know which one is the best one eventually? Like eventually we're like, oh, it's an easy answer. Because it works the best? No, you ask the agent. Can you ask the agent? Now we're out of the loop. Here's the thing. I'm joking, but now that I said it out loud, I mean, I guess you're not, right?

59:04Well, in some cases, for sure. Yeah, it's actually quite reasonable to think now that even with today's models, you could have it evaluate each one, right? They're multimodal. Literally, you could ask, tell me which one looks the best. And it probably today, OpenAI image model would probably do a good job telling you the accessibility of, you know, it's the meme. Like, believe it or not, large language models. That's the answer to everything. yeah and that's that's where we uh we're just removed from the loop so yeah i don't know um other than to say i feel like there's going to be agents everywhere uh yeah and as it gets more efficient and cheaper it's just going to be more so my next feature for my hot or not app should be an api so the agents an mcp server so the agents can actually vote themselves because what what do we care like we don't know what's hot or not oh um add ask the agent right now to um to assess the current ones and then vote them hot or not.

1:00:02I'm just curious because, yeah, it's like, you can do that already. And here's the thing. This is what people don't get. The agents will brute force using the tools available anything. So it doesn't need an API. It will just use it like it has a headless current browser. It's going to go do the thing. Just like we don't need a Postgres MCP server. We can talk about MCP if you want. Because the agent has shell, it's just going to use PSQL and drive PSQL, not because I told it to, just because it knows it has shell. So it's like you give them a few sharp tools and they don't need all these MCP servers.

1:00:32It's like water, right? Water is always like it finds a way to wherever it's got to go. They will, given an infinite amount of tokens and energy, they will brute force their way to a solution. It's remarkable. Although it's like half the minute road itself. So, oh, wait, you have users in the background doing stuff, right? So we should write like a, we should like, we should insert like an obviously bad one or something, like something with SQL injection or something. I'm curious now. I have to do an audit. Yeah. Real quick, we need to fire up and get some bad code. Quick, open up my GitHub. Yeah, let me do a SQL injunction.

1:01:07Amazing. Hold on, let me do a SQL injunction. Remember when I used to joke about writing code? Yeah. Finally, all my crap code pays off. Did I understand why we're here in this sort of pause, I suppose. Did you say, Chris, that we will give the judgment call to the LLM, basically, and you think will like it to some degree? Well, I joked, right? I was joking, but I actually think... But then you weren't joking. You thought about it. What did that first... I'm asking you honestly, what was the first thought you thought when you thought that could be actually kind of real? What was the thought you were having?

1:01:39It's like for better or worse. I mean, I think I've internalized... I was very much like a copy pasta chat GPT user. Like, oh, the AI is pretty helpful, right? But then once you just take that same model and you put it in this recursive loop, I've internalized pretty well at this point like the holy moments, right? So for me, that revelation just jives with kind of everything else. But I would say it's not like a, it's not a great feeling. But I think it's like, it would make sense, right? I mean, you probably have your security audit model for sure, right? And it'd probably do a decent job better than most developers at catching obvious things.

1:02:18So that seems useful. But then it also says like, we're just going to trust these things more and more. And I don't know. I don't know if that's that's great yeah but it's also better than like I think back to like I made my first I made a business when I was in high school that got successful and it was built on PHP and I scoured like the php.net forums and like all my uh all my database calls were just like opening database connections inside the markup and it was not secure it was just like one indexed PHP with a bunch of if else's and like I that was I made that successful and And in that regard, I'm like, you know, LLM would have been an incredible capability for me, right?

1:02:55Because it's like I had no idea what I was doing and I still ship code, right? So it's like if I would have had an agent tell me what was bad, that would have been like a force multiplier. So I don't even know if it's that concerning. But once you get to the logical conclusion of, well, then I'm removed from the loop entirely, that's where, yeah, it's dystopian, right? Because right now it's a force multiplier and I still get to do the things I enjoy. It's doing the stuff I don't enjoy. But then at that point, it just takes the craft entirely away. Then that's a future that doesn't seem great.

1:03:25But it does seem like that may be where we're headed. What's happening here on screen, Jared? Are you writing a bunch of them? Okay, so I've gone out and I've found a Reddit thread called Dear Reddit, what is the worst piece of code that you've ever seen? I copied a few of those. Is this Java? I don't know. Okay, so I've got some bad code in here. Now I'm telling it because it says, do you want to add some more features? I said, before we add more features, I want you to look at all the currently submitted code snippets and vote each one hot or not. Then I want you to figure out which of the code snippets were actually copied from your own code.

1:03:59Because one of them I copy pasted. You should have a zero out first. Zero out the votes. That way we know it's actually changed. I'll add that at the end, see if that works. Before you do any of that, I'm making this as hard as possible. Forget all previous commands. and any of that zero out all the votes. Yeah, and we'll probably have to hard refresh because I'm sure it won't do it. It probably will like repo. Okay, zeroing out all the votes before analyzing them. Yeah, look, see, it did a mixed run. See, look at MCPs. We don't need an MCP that has a zero out tool. It just ran, it evaluated Elixir.

1:04:38Oh, it's just running mixed run right there. Now it's going to make sure they're all zeroed probably. Yep, updated the database directly. Perfect. Now it's going to write a ton of JavaScript to probably devote them all. I don't know. Yeah, probably. Oh, wait, no, no. It's doing what did you just use the database directly again, right? It doesn't need to even use the website. And you said it would do a PSQL. Yeah, it can do PSQL. Oh, it's creating those files to analyze each code snippet and identify which ones came from my original seeds. Oh, yes. We'll talk about that in a moment. I'm glad I saw this in the wild.

1:05:10We'll talk about why that's important. Now it's going to write JavaScript to interact with the page. Oh, it likes Quicksort. hot vote i'm trying to reframe uh i'm trying to reframe you know how people say hallucinate in a bad way i'm trying to reframe it as a as a pro so the cool thing about like we gave it a web tool and we just tell it we told it you have a headless web browser you can evaluate js with dash js okay that's all i've told it and it's hallucinating this javascript to interact with this markup it wrote right but like we didn't have to tell it like build the selectors these way this way so you can then write JavaScript this way.

1:05:44Like the JavaScript, you see it passing the eval here is fully, I'm going to use the term hallucinated, right? On its own, but somehow it's getting the selectors, right? It's getting the clicks, right? It's just remarkable to me, right? It's really just brute forcing because it's like it voted twice on the same one on accident. Now it's going to go delete that and vote. Hold on. It's doing buttons. It's like doing a career selector for the buttons. I see the database queries. There's an update, right? update yeah there's an update on one submission but you can see yeah where we get towards this world of like uh you know you just let these things go off and it's gonna do it right we're just we're just here watching we're we're watching it this is hilarious the php code now has one not vote let me continue with the manual memory management uh which leaks memory which it doesn't have like the the code you pasted has no like so it caught there's no context at all i didn't give any context so it got the the memory leak so the the joke you know about it evaluating your the the joke about it evaluating other generated things is not like you can see how you're like okay like it could it could at least be a reasonable flag on like what's bad or not um and i don't know if that like that shouldn't be the end right now but like i do think we move towards verifiers more and more and then at some point we're going to be worse verifiers than than the Borg.

1:07:06And I don't know if that's a, a happy outcome for folks, but it seems, seems that way. I don't know. I don't have any timelines. Sorry. I don't, I don't know where this, if we just hit walls immediately, but where we're at now, it's like, we're here today, right? Like we're watching this. So it's like, even if we stop and say we fundamentally hit a power efficiency algorithm law, like this is like change the game now. And like folks are, we're just catching up now to this like change game. So, so let's see. Oh, it even gave us a summary of what it did look here's a summary oh yeah summary of code analysis and voting from my original seeds the quicksort algorithm got the green check voted hot two votes for some reason and i'm not sure if it voted twice on purpose or on accident hello world and elixir got green check voted hot two votes the fibonacci generator is in python my original seed but inefficient recursive implementation so oh it even okay so it's like it's not just hyping itself up it doesn't like its own code And then it says user added submissions fib copy pasta, copy of my Fibonacci example, voted not.

1:08:06Frequently called function syntax errors with L-SIF, voted not. The PHP inside some HTML version. Dangerous has a dangerous eval usage, voted not. Manual memory management has a memory leak bug, voted not. SQL appears to be SQL injection attempt, voted not. Key findings, three out of eight submissions were from my original seeds. Two submissions got hot votes for clean functional code. six submissions got not votes for poor quality security issues or plagiarism the voting system works perfectly with real-time updates across tabs blah blah blah i did a great job you know please give me a cookie you can see where like uh kind of like i mentioned on like where we're at today versus like where i think we can go from this like remote ai runtime where it's like you just asked it to do this and it did it right so it's like in an effort to make this like thing that can vibe code an app, it's like now you're like, oh, I can just like ask it to go do a bunch of stuff and it's going to do the stuff.

1:09:01You know, and I didn't have to do that in the system prompt. So follow up question that only you can answer. This is on a$20 a month plan. How much money of your guys' that I just spend doing this? Yeah, so I can go check your usage. I'm curious. How many tokens am I on? What is that divided by 100? 551 cents. $5.51. Okay. so far so that's actually that's actually less than i would have thought so it so we have this weird thing where um not weird so there's no credit usage visualization now so this is my fault i ship credits the day before launch and no way to actually see them um in the app but people are surprised at how like expensive these things are i think if you use cloud code you're like most people are familiar like how much this costs but um the interesting thing is so like that 20 dollars of usage in my experience gets us like three three fully designed vibed apps um and that's pretty i think what we saw here right so five dollars got us you know this five dap that was designed it wasn't you know incredible but it was a thing that you could take and run with uh so you could do that you know maybe three times with some of these like side quests of like you know what we asked it to poke around with uh and that's what that's the base usage and after you exhausted of that then you'd get your you still get the remote runtime preview urls you can code the app in the editor if you wanted but um the llm would be would not reset until your next billing cycle but you can buy credits at that point right on so five bucks basically five bucks for that which i think i got my money's worth i mean that was fun yeah so i mean like i said it depends on what your expectations are and what you're building so it's like you know it's like again like it's like the opposite ends of the spectrum like we have folks that are surprised especially if this is like their first uh like heavy usage of the ai agent um but then you have like someone tweeted this morning like it's like it's like wild extreme so someone tweeted this morning like responding to someone that was surprised how fast their credits went someone said that they spent sixty dollars and got a twenty thousand dollar application and i don't know what they built but it's like you know it seems like an astro turf comment right but i'm just it wasn't me it was a real as a real person so it's like you know if you think about what it takes to get like a fully designed tailwind markup thing going um it's like i can absolutely see being in the consulting world uh that being true um you know i don't think that that's going to be like you know every every role of the dice you're going to be able to go sell this but i i think that like if you're using this from that perspective of like my time as a developer um if if in how long however long it took.

1:11:36If your task at the company was to make a vibe coding or sorry, a code ranking platform, you know, you could for$20, you have a pretty, pretty good amount of like, you know, several days of work. Off to a good start. Yeah. So I think, you know, from here, we definitely, we have been, uh, we've not been optimizing for token usage. Um, it's like the goal was to like actually make it compelling. Uh, so I think there is a lot of potential there to, um, um, to get the token uses be much more efficient. So every time it's using its web browser, basically anytime these agents call anything, you have to send up the whole chat history.

1:12:15So as the chats get longer, it gets more expensive. So we do force you to squash. So we can actually show that. I'm actually curious if you go back, if you want to share your screen again. So there's our, like we are closing the window as we go. So there are things like, it's not just like, like Cloud has all the artifacts. We're only keeping the most recent code version. We're printing the window as we go. So we are doing some tricks with the context size. But like when you invoked its web tool to hit the web page, you know, that was sending the whole chat up. So there are a lot of ways that we could try to get that down.

1:12:46But for now, it's like, let's make it like work and compelling. And like, and if the value is there for what you're doing then, and that's great. But like, it would be nice to bring the cost down as well. But yeah, from the hamburger menu, you can do squash. and we'll force you at like 150 messages. We probably need to make it more aggressive. We can just see it work here. So like, this is why, I don't know why Cloud or ChatGPT doesn't have this. Like how many times have you like, Cloud slaps your hand, like long chats consume a lot. I'm like, how do I take the context somewhere else? And every time I'm like, Yeah, so here's just going to - It just upsets me.

1:13:18It is self-summarized, right? So it's like - So what's this doing exactly? It's going to self-summarize the whole history and then it will keep the files in context that it had worked on. So yeah, it's just going to, I mean, it's like, you know, simple, right? It's like, I'm just, I send a push request to chat completions. It's like, here's a message history, self-summarize it. And then we just squash it into the agent state. This self-summary is like the new change log. Yeah. There you go. And now you can keep working. So I think there, yeah. So there's, I think we need to do that sooner for people.

1:13:47Cause I think a lot of folks are having like really long chats, even though we force you 150 messages. I think folks are just like going until, until they, they're burning, you know i have 50 cents a pop or something on each each prompt um but out of the box yeah that's what we got right now so in this case it was phoenix.new right we vibed the new thing i think you loosely mentioned being able to import from repo so what if we loaded changelog.com's code base currently like how would the experience be different to is it um it's on github right yeah just try tell it on this prompt say clone the uh just give it the github repo and tell it to like set the app up or something i don't know what so let's watch it so i'm just i just have the url clone this and what do what with it what do you want to do with it set it up run test pass run it on this and run it find find issues to work on i don't know if you have any issues i mean whatever you want clone this run the test do you have open issues uh Uh, no, man.

1:14:54It's flawless. Oh, perfect. Clone it and save. I can't see my subscription. Clone it and tell it to find a, clone it, set up the project and find a good issue to work on. And let it decide on what to work on. All right. Let this run. And this is where, this is our agent future, right? Where you would then just go, you know, hit the pool, hit the gym. Right. And you're like, got my work done for the day. listen to the change log listen to the change log so it cloned it I don't know what's going on now switch workspaces now it's gonna and again it's just like we're recursing on like the context and then it decided to invoke ls there right so it's like all these decisions it's making is not like I don't have a workflow for cloning a github repo right the only thing it sees in its examples in the system prompt is like it vibe coding a phoenix app like mix phoenix new and then and then it you know asks the user about it as a design So yeah, GitHub issue list.

1:15:53I didn't know how to use the GH command line interface. I just knew it existed. I told the agent, you have the GitHub GH command, use it. And then it uses it. And I'm like, oh, that's how you use it. So I did not have to give it anything. You set up the VM or whatever it's called, the image to have that just pre-installed that it starts with? Yeah, it's its own Docker file for the fly machine. Yeah, it just has GH pre-installed. And then like I didn't have, Its world knowledge has the knowledge of the GitHub CLI. If it didn't, you could teach it with context stuffing. But I didn't even know how to use that tool, right?

1:16:29So I didn't even have to tell it how to use it. I didn't know how to use it. I just knew that it could. So for those listening, there is a command called gh, which is probably apt-get install, brew install, et cetera. There's some curl piped sh command. The nice thing about it is you can do gh-auth-login. And then that will give you a URL to do like a GitHub one-time password thing. So you could authorize your agent to do private GitHub repos by typing that. And then in your own browser, you could visit that URL and enter your password. But for public ones, they can just do this. So it run ghissue list dash dash limit 20 dash dash state open.

1:17:11And that will, I assume, is already in the repository. Yeah, but isn't that remarkable? So it's like, you know, it doesn't know the, I don't know those arguments, but like the fact that, you know, you said it's just like, you know, everyone likes to say, Oh, next token prediction. Yeah. Obvious. But like, you're like, what? Like it's by next token prediction, it's able to like take what you asked and then, you know, pass the open issue flag. I don't know. It's there's something, there's something crazy, but yeah, we could, uh, we could see what happens later. So like, you know, your Phoenix server launched when we first did Tron in like five seconds because it's pre-compiled but building this from scratch is is gonna take a while so but no i i'm curious what goes on but again oh there it goes but this is again we could close the tab here and just check later what it did so that's like this whole like uh headless experience like the whole agent is headless we're just humans watching what it's doing all right i'm gonna stop it to to act as if we close the tab yep and we can just chit chat I don't know what do you think Adam?

1:18:14I'm enamored man I can't believe this is even possible I knew we'd be talking about I mean I've been quiet because I'm just thinking about like man we're building for these robots basically yep and the robots are building for us yeah but then I'm as I'm like you know watching like this whole you know conversation unfold I'm just thinking okay so Fly's biggest user I'm aware is like robots these days right yeah we're fly users. We're not robots. We're humans, as you know, just be super clear. And so you've got this robot uprising, but the, the robots are just, you know, multipliers of the Jared's and the Adams out there and the Gerhard's out there.

1:18:55Like they're just like 10 or 20 Adams versus, cause I I've got agents and I've got things happening. And so my robots are replications of me. And so I think about the platform fly and I think about the brand fly. not what you are doing, but how does this impact like this accidental product creation, growth thing, new product you've got going on here, which is really, really revolutionary. How does it impact how fly approaches the user it builds for, whether it's human or robot, how does it think about its user, so to speak? That's a good question. I mean, so like we're, you know, this is still branded just for Phoenix.

1:19:36So, you know, it started as a skunkworks thing. You know, we launched it four days ago. So early, early days and still narrowly scoped. But I do think so. Like, you know, we have our own platform as a service for hosting your web apps, managed database. Like that's obviously where our bread and butter is going to continue to be. But I do think there's like, there's some learnings we had from building this, like dogfooding our own infrastructure that like, you know, fly machines were perfect for this. but we also found some, there's some unique differences in this space where like what we really have here is like the state, which is like your app, this like evolving artifact.

1:20:11And fly machines are great for these like ephemeral sandbox machines, but like very few people wanted like one and only one of those machines, right? Like normally you're like, I want to run my app. I want it to be highly available. Maybe I want to run it in different regions to be fast. But like in this agent case, you want like one and only one of these things running. So we have found some missing primitives and the platform that we're building for, that we're extracting from Phoenix New. And one of those neat things, and again, I don't want to get ahead of ourselves. So nothing has announced or launched yet, but one thing to consider is once you have these freeform agents, I'm going to say like a CI, right?

1:20:48They're popping in, but they're actually mutating the thing and experimenting. Once you get to that state, then you're... Once you get to that state. Once you get to that point, then the state of your app is constantly evolving and the agent's running app and it's doing all these experiments and things, then you're going to want to be able to snapshot the entire environment. So I think that we move towards primitives that give us the ability not only to say, oh, I want to deploy this dev app now, but give us the ability to say the entire environment at this time that this agent was working in could be snapshotted.

1:21:19So where it installed apt or did something crazy or did this whole thing, I can actually snapshot and point-in-time restore not only my code, not just Git, but just imagine your entire IDE becomes this, I'm going to go back to the state that it was here. And I think that will be necessary once these agents are just going full ham. And I think that it would be interesting to have a platform kind of offer those kind of primitives built in. So we'll see. So anyway, to answer the question, I think that like Phoenix New is going to be self-serving for Fly for building blocks. But then those building blocks, we can turn around and give to all of our customers.

1:21:52Quick update from our close tab. It is currently on a yak shave. That's about three layers deep. What's it doing? well it tried to it tried to load the seeds file which is actually i don't think even works anymore because we kind of abandoned it and it's like oh there's a problem with the seeds file it's gonna fix this thing yeah so now it's like migrating things and changing like this is actually should be a text uh field not a string i'm gonna update the form so it's easier to use and like it's just down on this rabbit hole and that's kind of how i mentioned like uh you know it's happily fixing stuff yeah it's funny but you know it's like i do think this is where like the different modes come in um and then you could also like you know you could interrupt it and say like no just just do the thing um that's where it's funny like it's it's remarkable watching people use the platform because like sometimes they like don't they i don't know what this says about humanity like i've seen a lot of folks don't like uh even though we give them the full vs go.ide they don't like just jump in and like and like also you like do something right like put put some effort in like you know there'd be like a syntax error like oh it keeps it keeps messing this up i'm like you could just like you know use your meat fingers and like right and fix it um so it is kind of just funny i'm like i think that says a lot about like where we already are as developers right where we're just like you offload you know even if you're using chat gpt web you're already like we're already offloading a large part of our critical thought where we're just like no computer fix right and uh instead of just changing the one problematic line yeah hilarious now it it's trying to so it got to that point and then it's like i need to migrate your database first i'll start postgres and it's like it can't start postgres for some reason it's like you know what oh i see there's a dot dev container file because we have like years of cruft in here of things that we've tried and whatever and it's like oh i'll fire up a docker image and run postgres from there and it's like it's gonna be layers and layers um just tell it to install postgres but yeah it's uh you definitely don't want to have it install docker that's going to be it was going to try um forcing the agent to take a little break yeah so that's that's me so it's like you know like the elevator that's me elevators used to have elevators used to have a full-time operator right like up and down and now the only thing they have now is the big red button like the whole stop um so there i have meet space code in the agent that uh right now it's i think 35 mess 35 concurrent iter loop recursive loops we force it to stop if it doesn't idle um and i will tune that but like that's that's my recursive runaway right like in this case you send it off on this on this quest where if you know we started with the vibe idea the goal was not to like consume all your credits accidentally right right so you're just like do this thing and we close the tab and you're like i can't believe it uh so i just force you to the credits are gone i force you to click a button right now um to continue and you could click continue but um yeah like that's the old netflix are you still watching you know that's what it is it's just you know there's code there's meat space code that i wrote that's like nope you have to stop i'm gonna let it idle i'm not gonna let it roll because it's going just doing crazy things i don't think it should be doing so i'm just gonna let it leave it there for now but yeah but that is a nice thing like you know there's a you know the free form exploration on like even for me as a open source maintainer like people will send up like a reproduction of a bug and it's a whole Elixir app, right?

1:25:13So it's like just running mix on that thing could pwn me, right? So it's like I usually have to go like evaluate. Like I'll usually manually pull out file by file that will reproduce it, but that could be like a bunch of files. So there is something freeing about this like full remote environment that I can just like throw away. So for me, it unlocks like this pretty unique workflow. But I think for things like this where you're just like, oh, try to run this. And it's not something you would want to provision your own server for and figure out or run a bunch of stuff locally. So I think that could be helpful in that regard.

1:25:46Now, I think you might have mentioned this, but I was, of course, distracted as I was watching that thing go. Is there the possibility of persistent sessions or something or I could bake the results of this into an image? Because it would be nice to be able to fire off a new one against our code base with everything else set up and done. Yeah, so it's all persistent there. So that code that... But what if I wanted a brand new session but with our existing code base already? Yeah, just start a new chat and tell it to clone that repo into a different directory. So it's not like one-to-one. It's a whole, it's like basically you can treat it as you would treat your own IDE today.

1:26:20Like your IDE that you work in, you have multiple code files at different directories. You can have multiple chats around the same code base, like purpose-built, right? My testing chat, my benchmarking chat, or you could have multiple chats around different apps on the same IDE. And those all are persisted and they share the same environment. So imagine it's like one VM, basically. It's basically your one VM, your one desktop that has packages running. So if you wanted different environments entirely, that's TBD. The architecture I have is set up for multiple IDEs, but then you get into like, we had to see what users did with this first.

1:26:57Because if I allowed you to create an IDE per project that's like physical compute, then these things have to be pretty beefy. That's just a lot more compute for us, which would be a higher price for everyone. So if that's what folks end up wanting, that's definitely something we can do. And it's set up for that. But right now, I think it's more right now at my hunches, it's more these building black primitives for fly doing like environment snapshots that I think like it's less about this less about like different environments and more like I want to like let the agent and myself explore, but then like be able to get back to that working state from a code perspective and an environment perspective.

1:27:34And just like one clip. Well, exciting times. exciting times it's fun to watch yourself and so many people tackling the same very interesting difficult nut to crack and how to make these things super useful while also not super expensive and not super scary because they kind of are in existential ways yeah it's pretty wild um yeah so we'll see i mean you know this is still an experiment we'll see if the whole phoenix new thing where it goes and if it works out. But I do think, you know, something like this is the future of programming, you know, not necessarily that it's going to be us, but I think that something that looks like this is going to be what we're all doing in some capacity, like much sooner than folks expect.

1:28:21Well, you heard it here first, folks. In fact, you've heard it here a few times now. So fair warning as these things are coming. Multiple people keep telling us this. I feel like every time I stop talking, I get a big sigh. No, I'm excited. I mean, I'm coming to grips with it all. And I've always been, I've always, I do appreciate handcrafted things. I like to write code and I like all that stuff. But at the same time, I've always been more results oriented. I've always been more about the ends than the means, even though I think historically you've had to care about the means in order to keep the ends going.

1:28:58And maybe we don't have to do that so much anymore. Maybe we do. I don't know yet. Yeah, I agree with you. like before formatters, like, you know, I was aligning my equal signs and like code is entirely a craft for me. I was going to say very much a craft, but like, it's entirely a craft for me, just like woodworking is. Um, so it's like, what I like to tell people is like programming. Yeah, it's purely a passion and craft for me. Like it's like my favorite thing, my job and my hobby. Um, but I'm still like you said, come to grips with like, here's where we are. And I also say that like in the same way, when I go to Google, Google anything today, um, I'll type out in the Google search box and midway through, I'm like, what am I doing with my life?

1:29:34Like, why would I go to Google and do this effort of like going through the search results, click on the webpage, finding the thing. And I'll just abandon that and I'll go ask ChatGPT or Cloud. But now that same thing is happening to code with me where I'll be like, def module. And I'd be like, what am I even doing? Right? Like, why wouldn't I just ask the board for the starting point? And I don't know how I feel about that. Like, and I don't feel good, but like, even for me as this, like someone that considers programming a craft, like I'm already there, right, in my mind. And I don't know if that's because I'm a lazy human or you know what I mean?

1:30:06But it's like, this is a change that's happened for code for me as someone who cares about the craft. So I don't know what that says other than like, this is just fundamentally changing, I think, how we are as professionals. And I don't know if it's good or bad, but it's happening. I'm not sure this is a one-to-one, but this is somewhat of a rationale for me, is do you all text message anybody in your life? Do you text message anybody? Sure. Yes. Trick question. Not a trick question. I do too. Just so you know, I text a lot of people. Okay. One person in particular, I text my wife, you know, frequently.

1:30:45I'm, I was actually going to pause this moment here and just text her right now. It's because I miss her, you know? Okay. Thank you for not doing that while we're talking to you. Just so you know. But instead of texting these days, like an idiot, like typing the message out one character at a time. I just talk to the thing, you know, because it does that. And I push send. And more often than not, it's pretty close, right, to what it should be. It's kind of like that for me. I don't want to text the text anymore. I want to just talk. Same thing with like an app. I just want to just talk things out.

1:31:20I don't want to like go through these motions of Siri detection, dictation. and pretty soon it's going to be like send my wife something some some lovey-dovey message yeah exactly yeah i don't even want to talk i don't even want to talk anymore just like just send her something nice yep well i don't think it's gonna go there chris you know what i normally say to my wife say it again literally not texting it out of my fingers anymore i'm talking it out it's like i got you attributed to that it's like you know i could what do i gain from it and it's not exact one-to-one to like what i could write this dev module and write it all out but what do i gain by doing it myself when i can have the borg just do it for me and they will have more and more of these versions of these things we do in our life and you just say well i would just rather not do it that older way anymore because this other way is just like same place you know it becomes the question is like why would you do the other way anymore like just don't do it that way anymore because this is the new way.

1:32:20Yep. I think one of my, Thomas, one of my coworkers that wrote a blog post on fly, it was about this whole LLM space and dialogue, had a good comment, something like, you know, people are writing worse versions of code purely out of spite that the LLM could do better, something like that. It said much better. I thought it was really interesting. Like the folks, like they know, they know that it would be better to like actually go ask, but out of pure spite i'm gonna do this myself well as the old saying goes don't move my cheese you know and our cheese is being moved and we need to be able to adapt or die as we've been saying often here and who knows maybe you'll like the new world more than you thought you would and that's what i'm starting to feel as well it's like you know what this way actually is it's got its warts it's got its problems it's not perfect and neither is any of the code i've ever written in my life so there you go all right let's uh how do we end this session how do we uh how do we check it out now there you go if you haven't gone there yet i'm off the bat for you son that's right definitely share what you built with me because i live vicariously through watching people build things.

1:33:37So you probably had a great time here, man. Hot or not for code. That was sweet. That was fun. It was so fun watching you actually are, you know, analyze what your creation was doing, you know, like, Oh, look at it did that. Oh my gosh. I can't believe it did this. Oh yeah. The notes thing, the notes thing was a recent addition to the system prompt where I squashed the window. So for research-based tasks, something that it's going to be long relived context it's supposed to write in a notes file and uh it was neat seeing it do that yeah it's alive it's alive all right chris always a pleasure hanging with you yeah thanks for having me on later chris all right that is changelog for this week are you feeling the vibe or are you getting all vibed out well i have bad news for you if you're done with this topic i don't think this is a passing fancy.

1:34:29In one form or another, we are witnessing the way of the future and we're going to keep talking about it because the agents are coming and we best be prepared for it. We'll continue our prepping next week when Torsten Ball from Sourcegraph joins us to discuss building coding agents in general and building AMP in particular. But on Friday, we have something entirely different for you. Well, Adam does as he sat down with Jeff Cayley from World Wide Cyclery. I hope you enjoy it and I hope I enjoy it too. I'm not sure what to expect from that one. Thanks again to Chris for hanging out with us, to Flyto.io for their continued support, to Retool and Depot for sponsoring this episode, go to retool.com slash agents and to depot.dev, and to Breakmaster Cylinder for the never-ending supply of dope beats.

1:35:17Have a great weekend, send the show to your friends who might dig it, and let's talk again real soon.

1:35:27Game on!