Programming with LLMs (Interview)

27:19We have moved from AIs that are new college graduates to, in effect, AIs that are now among the best developers in your code base. And that difference is a profound one for software engineering in particular. You know, if you're writing a new application from scratch, you want a web page that'll play tic-tac-toe, piece of cake to crank that out. But if you're looking at, you know, a tens of millions of line code base, like many of our customers, Lemonade is one of them. I mean, 10 million line mono repo, as they move engineers inside and around that code base and hire new engineers, just the workload on senior developers to mentor people into areas of the code base they're not familiar with is hugely painful.

28:01An AI that knows the answer and is available seven by 24, you don't have to interrupt anybody and can help coach you through whatever you're trying to work on is hugely empowering to an engineer working in unfamiliar code. Very cool. Well, friends, Augment Code is developer AI that uses deep understanding of your large code base and how you build software to deliver personalized code suggestions and insights. A good next step is to go to augmentcode.com. That's A-U-G-M-E-N-T-C-O-D-E.com. Request a free trial, contact sales, or if you're an open source project, Augment is free to you to use. Learn more at augmentcode.com.

28:43That's A-U-G-M-E-N-T-C-O-D-E.com. Augmentcode.com.

28:55So you've been using those in your day-to-day programming work for the last year. And back in early January, you wrote this post, how I program with LLMs, which I found to be refreshingly practical and straightforward your findings you said you've been actively trying these things I feel like I've been passively trying them not really trying to optimize my setup but just like you know like a Neanderthal kind of poking at a computer box you know like oh does this work no okay for the last couple of years so I do use these things but I don't think as effectively as most or at least some. And I loved your finding that, of course, you're building something as a result of it.

29:39But can you take us on that journey over the last year or so where you started with LLMs and what you found in your day-to-day programming? Yeah, I don't think your experience is unusual, actually. I think almost everyone has your experience. And for most software, I am in the same category. I try things at a very surface level when they're newish and see if there's any really obvious way they help me. And if they don't, I put them aside and come back later. Great example of that is the Git version control system. It was 10 years before I really sat down and used it. I was using other version control systems.

30:16After 10 years, I was like, okay, this thing's probably going to stick around. I guess I'll get over its user interface. Fine. I was reluctant, but I got there in the end. LLMs really struck me as fascinating. I made this active decision to not do that with them, set out on a process of trying to actively use them, which has involved learning just a really appalling amount, honestly. It's very reasonable that most engineers haven't done really significant things with LLMs yet because it's too much cognitive load. If you're writing computer programs, you're trying to solve a problem. You only have so much of your brain available for the tools you use for solving problems because you have to fit the problem in there as well and the solution you're building.

31:03And that should be most of what you're thinking about. The tools should take up as little space as possible. And right now, to use LLMs effectively, you need to know too much about them. And that was my big takeaway 11 months ago, which is why I started working on tools with some friends to try and figure this out. Because there has to be a way to make this easier. And my main conclusion from all of that is there's an enormous amount of traditional engineering to do in front of LLMs to get there. So the first really effective thing I saw from LLMs is the same thing I think most engineers saw, which was GitHub Copilot, which is a code completion.

31:44So actually, GitHub Copilot has taken on new meanings. It's more than that now, right? Yeah, it's an umbrella brand that means all sorts of products. and I honestly haven't even used most of those products at this point. The original product is a code completion system built into Visual Studio Code where as you type, it suggests a completion for the line or a few lines beyond that of where you are, which is building on a very well-established paradigm for programming editors. Visual Studio 6.0 did this 25 years ago with IntelliSense for completing methods in C++. This is not a new idea. And around the same time, we had eTags for Emacs, or C-Tags, I should say, which gave us similar things in the Unix world.

32:41And so this is sort of extending that idea by bringing out some of the knowledge of a large language model in the process of completing. And I'm really enamored with the entire model. Like, you know, CoPilot's original experience when it came out was magical. Like, it was just like, there was nothing like this before. It was. Really, I think, jump-started a lot of interest in the space from people who hadn't been working in it, which was almost all of us. And from my perspective, the thing that really struck me was, wow, this works really well. And wow, it makes really obvious, silly mistakes.

33:16It was both sides of this. It would suggest things that just don't compile in ways that are really obvious to anyone who takes a moment to read it. And it would also make really impressive cognitive leaps where it would suggest things that, yes, that is the direction I was heading. And it would have taken me several minutes to explain it to someone and it got there immediately. And so I spent quite a lot of time working on code completion systems with the goal of improving them by focusing on a particular programming language. And we've made some good progress there. we actually hope to demonstrate some of that publicly soon, like in the next few weeks, probably in this sketch.dev thing that we've been building.

33:55We'll integrate it so that people can see it and give it a try. But so those models are interesting because they're not the LLM experience that most users have. Like when everyone talks about AI today, they talk about ChatGPT or Claude or these chat-based systems. And the thing I really, really like about the original co-pilot code completion model is it's not a chat system. It's a different user interface experience for the knowledge in an LLM. And that's really a lot of fun. And in fact, the technology is a little bit different too. There's a concept in the model space called fill in the middle, where a model is taught a few extra tokens that don't exist in a standard chat model.

34:37With fill in the middle, which is a lot of fun, a model is taught a few extra tokens. and it's called a prefix token, a suffix token, and a middle token. And what you do is you feed in as a prompt to the model the file you're in. And everything, all the characters before where the cursor is get fed right after a prefix token. So you feed in prefix, all the characters of the file, then you feed in suffix, and you feed in all the tokens after the cursor. And then you feed in the middle token and then you feed in whatever goes into the middle to complete it. And that's the prompt structure for one of these models.

35:24And then the model keeps completing the thing that you fed in. It writes the next characters. And you train the model by taking existing code out there. There's a few papers on how these models are trained because Meta published one of these models and Google published one of these models under the Gemma brand. There's a few others out there. There's one from Quinn and some other derived ones. And you take existing code files, you choose some section of code, you mark everything before it as the prefix, everything after it as the suffix, and you fill in everything after it as the middle. And that's your training data.

36:03You generate a lot of that by taking existing code and breaking it up into these files randomly. by randomly inserting a cursor, then you've taught a model how to use these extra characters and how to complete them. And so it's not a chat model at all. It's sort of a sequence-to-sequence model. It's a ton of fun. And the advantage of these systems is they're very fast compared to chat models. And that's the key to the whole code completion product experience is you want your code completion within a couple hundred milliseconds of you typing a character. whereas if you actually time Claude or you time one of the open AI models, they're very slow.

36:44They take a good minute to give you a result and there's a lot of UI tricks in hiding that minute. They move the text around on the screen then they stream it in. It's very clever. Because you're reading it word by word as it comes out but it's basically stalling. You're like, come on, just give me the answer already. That's right. Exactly. you can really feel it with the new reasoning models 01 these things because there's this pause at the beginning it hurts. The thinking phase I'm like come on I don't and it tells you what it's thinking which is cool but it's like think faster I don't care what you're thinking give me the answer I think it's actually kind of cool when you see that reasoning come up I mean you get to see like I feel like this is the closest we've glimpsed into the future that we've ever been able to by watching the reasoning in real time you see the act of reasoning that it's happening.

37:36It explains the reasoning. Users ask me this, so I'm going to think about that. Okay, I've thought about that, which causes this. And it's like this step process, and it reminds me of how I think too. So I'm like, that's pretty dang cool. But it's also a great trick. I agree. Yeah, it is a ton of fun to watch. I agree. And it is a lot of insight into how the models work too. Because the insides of the models are a large number of floating point numbers. holding intermediate state. And it's very hard to get insight into those. But the words, you can read them. You can make some sense of them. So code completion is, I think, extremely useful to programmers.

38:12It varies a lot depending on what you're writing and how experienced models are with it and just how sort of out on the edge of programming you are. If you're really out in the weeds, the models can get less useful. I used a model for writing a big chunk of AVX assembly a few months ago. And the model was both very good at it and very bad at it simultaneously. And it was very different from the typical asking a model to help with programming experience. It would constantly get the order of operations wrong or overcomplicate things or misunderstand. And it was a very different experience than typical programming.

38:57What model was this? How did you find it? I used all of them for that. Okay. And this is what I meant by I'm spending a lot of time actively exploring the space. Yeah. I'm putting far too much work into exploring the model space as I do work. It makes sense that there are specific models that are good for autocomplete versus search versus chat. but have you found the correct one for each particular subtask or what's your advice there is it like use them all or just stick with this you'll be good i can't advise people to use them all you know that's a bunch of them yeah it's uh and this i think is the big problem and you know you mentioned that most programmers are probably using this as far as we can tell not one fifth of programmers are using these tools today.

39:46How can you tell that? Through surveys. A couple of people have done surveys of programmers and it seems to come back that most people are not using these tools yet. Wow. Which is both shocking to me because they're so useful and also makes a lot of sense because it's a lot of work figuring out how to use them. I have an analogy that I'd like to share. If you're a runner, you probably wear running shoes, right? You're probably not going to run barefoot. I think it's like admitting to running barefoot. Like you wouldn't do that. You would run a marathon with rocks on the road and debris and things like that barefoot versus running shoes designed to aid you in the process of running to make it more speedy, comfortable, agile etc I feel like that's where we're at like I've I've changed my tune let's just say because I feel like it's not going to go away and to hear that one fifth I'm I haven't dug into these surveys but that's surprising one fifth is using it and it seems like I guess then the the four of the five are saying no or for for the time for the moment denying it I mean do either of you disagree with that analogy is it way off or is it jared i saw you kind of like shake your head a little bit what's your thoughts on that now i mean i don't think that these tools have come to the place that the running shoe has i also think there's probably plenty of world-class runners who run shoeless and would never run with a shoe on because that's for fools but i'm not going to go there um well would you run a new york city marathon with no shoes i wouldn't be i wouldn't be so foolish as to try a new ski marathon.

41:27Okay. You have to admit, though, that having the shoes on is probably better for you than worse for you. Well, at what point has the shoe proven itself to be useless? Because these tools routinely prove themselves to not just be wrong, but dramatically wrong in ways that if you follow them, you will be like Michael Scott, who drives directly into the pond. No, no, no, no, no. Look, it means go up to the right, bear right over the bridge and hook up with 307. Make a right turn. Maybe it's a shortcut, Dwight. It said go to the right. It can't mean that. There's a lake there. I think it knows where it is going.

42:01This is the lake. The machine knows. This is the lake. Stop yelling at me. No, it's up there. There's no room here. Oh, yes. Because his voice assistant or his GPS tells him to keep going straight. And he just keeps going straight. I concur on that one, too. I concur. So I can see why you could get frustrated and throw up your hands and say, I'm going to come back to this in a year or two years. But I'm going to let all the frontiers people like David figure all this stuff out. In the meantime, I got code to write. I can see a lot of people saying that. I'm not. I could see myself saying that I haven't because I am curious and I don't want to fall behind.

42:37But I'm still don't feel like this is a must have for everybody today. But there are moments where I'm like, that was amazing. So for sure. I'm actively trying to fit it in into everything I do is I guess my perspective I'm actively if it's home lab it's that if it's contracts agreements proposals if it's thinking if it's exploration if it's coding if it's you pick it's your if it's kind of thing I'm trying to fit it in and I'm just so I'm sitting down on my bench I got my socks on and I'm trying to put the shoe on let's just say you know if to kind of extend my analogy you're gonna wear it yeah I believe that, you know, I'm going to put this shoe on, I'm going to wear it for every scenario that makes sense because I can tell you I move faster.

43:21I think differently when I'm in those modes. Are they wrong? Do I always check it? Of course, but I know that it's coming for almost everything we do. Every task we do that's productive, coding, thinking, writing, whatever, it's coming for it in a positive way. I mean, I totally agree that it is coming for it. I also think it's very early days and a great reason to not learn this technology today is that it's changing so fast yeah and that you can spend a very long time figuring out how to make it work and then that can all all of that accumulated skill can be sort of made useless tomorrow right by some new product if you remember stable diffusion first dropped probably two years ago now yeah and we were enamored with prompt engineering and what was that artist name that you had always if you added it to your stable diffusion prompt it automatically get awesome and then he got mad because everyone's using him to like make better pictures like that whole technology you know that magical incantation is just completely moot at this point like this probably it's easier now to get better pictures without being such a wizard and whatever name you're invoking in the past is just that name doesn't do what it did on the last version of stable diffusion just as one instance like prompt engineering has changed dramatically.

44:40And anybody who's ignoring it all and just listening to us talk about it on the changelog and just like staying with their regular life, like they've saved themselves a lot of time than those of us who dove in and decided they were going to memorize all the magic words. Yeah, absolutely. Like a year ago, a common technique with open models that existed was to offer them money to solve problems. You start every prompt by saying, I'll give you$200 if you do this. And it greatly improved outcomes. Yeah. Or let's take this step by step. Like that phrase was one of those magical things that made it better.

45:12All of those techniques are gone now. Yeah. Like if you try bribing a model, it doesn't help. There was a great, there was a great example I saw of that where someone would kept saying, I'll give you$200 if you do this. And they did it in a single prompt several times. And they got to the nth case and it said, but you haven't paid me for the previous ones yet. Listen, we had a deal. No means no. That's great. No money means no. Yeah, there you go. All right. They're very funny models. So I spent a long time believing, and I actually, I still believe this in the long term, that chat is currently our primary user interface on the models, and it's not the best interface for most things.

45:53The way to get the most value out of models today when you program is to have conversations with models about what you're writing. and that's, I think, it's quite the mode shift to do that. It's quite taxing to do that and it feels like a user interface problem that hasn't been solved yet. And so I've been working a lot with Josh Bleacher Snyder on these things and we have been sort of, we spent a long time looking for how can we avoid the chat paradigm and make use of models. That's why code completion initially was so interesting because it's an example of using models without chat and it's very effective.

46:34So we spent a long time exploring this to give you another example of something we built in this space because we've just been trying to build things to see what's actually useful. We built something called Merd, merd.ai, which I think we put up a few weeks ago and it does merge commits for you. So if you try and push a git commit or do a rebase and you get a merge conflict. You can actually use LLMs to generate sophisticated merge commits for you. It turns out that's a much harder problem than it looks. You would think you just paste in all of the files to the prompt and you ask it to generate the correct files for you.

47:17Even the frontier models are all really bad at this. You almost never get a good merge commit out of them. But with a whole stack of really mundane engineering out the front, mundane's not the right word because a lot of it's actually really very sophisticated. But it doesn't involve the LLM itself. It's about carefully constructing the prompt. Traditional. Traditional is a much better word. You can actually get very good merge commits out of it. And that user experience seems much better for programmers to me that you could imagine that being integrated into your workflows to the point where you send a PR, there's a merge conflict, it proposes a fix right on the PR for you.

47:56And in fact, we attempted a version of that where there's a little Git bot that you can at mention on a PR and it sort of generates another PR based on it that fixes the merge conflict for you. And that sort of experience doesn't require the chat interface to be exposed to the programmer to make use of the intelligence in the model. and that is where I dream of developer tools getting so that everyone can use them without having to learn a lot about them. You shouldn't have to learn all the tricks for convincing a model to write a merge commit for you. There should be a button, or not even a button.

48:34It should just do it when GitHub says there's a merge conflict. And so it works pretty well. We've seen it generate some very sophisticated merge commits for us. We'd love to see more people give it a try and let us know what the state of that is. But so just because that is such a hard state to get to, we built Sketch, which exposes the traditional chat interface in the process of writing code. Because we're just not, we don't think the models are at a point yet where we can completely get away from chat being part of the developer's workflow. So at what level of granularity is Sketch working at?

49:17And do you imagine it moving up eventually, wherever it is? Because the panacea, right, the silver bullet is what some folks are trying to do with Devin, for instance, where it's like you describe at a very high level a system and it goes and builds that system. V0 from Vercel is another one that's doing these things. And they're very much at, in my opinion, the prototype slash demo level of quality, not the production level of quality and their output. And it seems like they're very difficult in my in my limited experience with these things. They're very difficult to actually mold or what do you do?

50:03I'm losing a word here. Sculpt. I don't know, like a sculpture to actually like sculpt what they come out with and change it into something that you actually would write or like. but those are like the very high level of like well it should have a contact form that submits to this thing but maybe you're looking down more where the where i use them currently which is like yo write me a function that does this particular thing you know and at that level it seems a lot easier to even chat to if i have to i would rather not chat to it but spit out code that i could copy paste and modify versus being like i'm going to throw this away and rewrite it right yes i I think that lines up really well with the way Josh and I think about these things.

50:47Where today, if you open up a model, a cloud provider's frontier model, or a local DeepSeq or even Alarma 70B, you can ask it to write a Python script that does something. It could be a Python script to go to the GitHub API and grab some data and present it neatly for you. And it will do a great job. You know, these great models can basically do this in a single shot where you write a sentence and the outcome is a Python script that solves a problem. And like that's that's an astonishing technical achievement. I really it's amazing how quickly I've got used to that as a thing. Yeah, you're not even impressing me right now.

51:25You know, it can do that. We all know. It is amazing. Exactly. Like five year old, five, me five years ago, if you told me that I would struggle to believe it. and yet now I just take it for granted. And so that works. We've got that. We've got a thing that can write really basic Python scripts for us. Similarly, these systems, at least the frontier models, are good at writing a small React component for you. You can give almost any of them, you need more than a single sentence, you need just a few sentences to structure the React component, but out comes some HTML and some JavaScript in the React syntax, the JSX syntax or the TSX syntax.

52:05And it's pretty close. It might need some tweaking. You might have some back and forths to get there, but you can get about that out of it. And clearly models are going to improve. There's no evidence to suggest we're at the limit here as the models keep improving every month at this rate. And part of what we're interested in Sketch is getting beyond helping you write a function, which I also use today. I get Frontier models to write functions for me to sort of how can we sort of climb the complexity ladder there. And so the point we chose is a point that is comfortable for us and what is helpful for us is the Go package.

52:45How can we get a model to help us build a Go package to solve a problem? And there's an implicit assumption here in that the shape of Go packages looks slightly different at the end of this. Packages are a little bit smaller and you have a few more of them than you would in a sort of traditional Go program you wrote by hand. But I don't think that is necessarily a bad thing. Honestly, in my own programming, as a Go program, I tend to write larger packages because there's a lot of extra work involved in me breaking it into smaller packages. And there's often this thought process going on in my mind of like, oh, in the future, this would be more maintainable as more packages.

53:25But it's more work for me to get there today. So I'll combine it all now and maybe refactor it another day. And switching to trying to have LLMs write significant chunks of packages for you makes you do that work up front. That's not necessarily a bad thing. It's perhaps more the way we'd like our code to end up. And so Sketch is about taking an LLM and plugging a lot of the tooling for Go into the process of using the LLM to help it. So an example is I asked it the other day to write some middleware to Brotley compress HTTP responses under certain circumstances because Chrome can handle Brotley encoding.

54:11It's very efficient. It's not in the standard library, or at least it wasn't the last time I looked. And the first thing it did was it included a third-party package that Andy had written that has a Brotley encoder in it. And so Sketch, Go gets that in the background in a little container as you're working and has a little Go mod there that modifies so that as you're editing the code, you get all the code completions from that module, just like you would in a programming environment. And more importantly, we can take that information and feed it into the model as it's working. If we run the Go build system as part of it, and if a build error appears, we can take the build error, feed it into the model.

54:54It's like, here's the error, and we can let it ask questions about the third-party package it included, which helps with some of the classic problems you see when you ask Claude to write you some Go code, where it includes a package and then makes up a method in there that doesn't exist that you really wish existed because it would solve your problem. And so this sort of automated tool feedback is doing a lot of the work I have to do manually when I use a frontier model. And so I'm trying to cut out some of those intermediate steps where I said, that doesn't exist, could you do it this way? anything like that you can automate saves me time.

55:27It means I have to chat less. And so that's the goal, is to slightly climb the complexity ladder in the piece of software we get out of a frontier model and to chat less in the process. Are you achieving that by having a system prompt or are you actually fine-tuning? Like, how are you as the sketch.dev creators taking a foundation model and doing something to get here? Today, it is almost entirely prompt-driven. Uh, there's actually more than one model in use under the hood, uh, as we try different things. For example, we use a different model for solving the problem of if we want to go get a package, what module do we get to do that?

56:08Uh, which sounds like a mechanical process, but it actually isn't. There's a couple of steps there. Uh, so a model helps out with that. There's very different sorts of prompts you use for trying to come up with the name of a sketch than there are for, uh, answering questions. But at the moment, it's entirely prompt-driven in the sense that a large context window and a lot of careful context construction can handle this, can improve things. And that can include a lot of tool use. Tool use is a very fun feature of models where you can instruct. So to back up and give you a sense of how the models work, an LLM generates the next token based on all the tokens that come before it.

56:55When you're in chat mode and you're chatting with a model, you can at any point stop and have the model generate the next token. It could be part of the thing you're asking it or its response. That meta information about who is talking is sort of built into just a stream of tokens. So similarly, you can You can define a tool that a model can call. you can say here's a function that you can call and it will have a result and the model can output the specialized token that says call as function, give it a name write some parameters and then instead of the model generating the next token you pause the stream you the caller go and run some code you go and run that function call that it defined paste the result of that function call in as the next set of tokens and then ask the model to generate the token after it So that technique is a great way to have automated feedback into the model.

57:53So a classic example is a weather function. And so you define a function which says current weather. Then you can ask the model, hey, what's the weather? And the model can say call function current weather. Your software that's printing out the tokens pauses, calls current weather, says sunny. You paste Sunny in there, and then the model generates the next set of tokens, which is the chat response saying, oh, it's currently Sunny. And that's the sort of easy way to plug external systems into a model. This is going on under the hood of the user interfaces you use onto Frontier models. So this is happening in chat, GPT, and Claw, and all these systems.

58:40Sometimes they show it to you happening, which is how you know. You see it less now, But about six months ago, you could see in the GPT-4 model, you would ask it questions, and it would generate Python programs and run them, and then use the output of the Python program in its answer. I had a really fun one where I asked it, how many transistors fit on the head of a pin? And it started producing an answer, and it said, well, transistors are about this big, pins are about this big. And so I guess, and a magic little emoji appeared, that this means this many transits that's fit on the head of a pin, some very large number.

59:19And if you click on the OG, it shows you the Python program it generated to do the arithmetic. It executed that as a function call, came back with the result. And that saved it the trouble of trying to do the arithmetic itself, which LLMs notoriously struggle with doing arithmetic. This is a great thing to outsource to a program. And so - It's a funny workaround because if you're a calculator for words, You're not necessarily a calculator for numbers. Yeah. And when you can't do those reliably, then you could just write a program that does it and returns the same thing every time. Yes. They're very good at writing programs to do the arithmetic and very bad at doing the arithmetic.

59:55So it's a great compromise. The thing we do with Sketch is try to give the underlying model access to information about the environment it's writing code in using function calls. So a lot of our work is not fine-tuning the model. It's about letting it ask questions about not just the standard library, but the other libraries it's trying to use so that it can get better answers. It can look up the go doc for a method if it thinks it wants to call it. Use that as part of its decision-making process about the code it generates. Can you describe let it ask? I mean, you've said a couple of times that I've been curious about this.

1:00:32When you say let it ask, what does that mean? Like decompress that compressed definition. So at the beginning in your system prompt or something like your system prompt, depends on the API on exactly how the model works, you say there is a function call, which is get method docs. And it has a parameter, which is name of method. And in the middle of, and then you can ask a, you can construct a question to an LLM that says, generate a program that does this with the system prompt, which explains that there's a tool call there. And so as your LLM is generating that program, it can pause, make a system call, make a function, a tool call that says, get me the docs for this.

1:01:18And so the LLM decides that it wants to know something about that method call. And then you go and run a program, which gets the result, gets the documentation for that method from the actual source of truth. You paste it into the prompt, and then the LLM continues writing the program, using that documentation as now part of its prompt. And so this is the model driving the questions about what it wants to know about. And just blocks and waits for that to come back. Yes. Effectively. Yeah. Yeah, so it's like an embed. If you step back to running Llama CPP yourself, or something like this, you can oversimplify one of these models as every time you want to generate a token, you hand the entire history of the conversation you've had, or whatever the text is before it, to the GPU to build the state of the model.

1:02:21And then it generates the next token. that actually generates a probability value for every token in its token set. And then the CPU picks the next token, attaches it to the full set of tokens, and then does that whole process again of sending over the entire conversation and then generating the next token. And so if you think about that sort of, that very long, big, giant for loop around the outside of every time there's a new token, the token is chosen from the set of probabilities that comes back, is added to the set, and then a new set of probabilities is generated for the next token. You can imagine in the middle of that for loop having some very traditional code in there that inserts a stack of tokens that wasn't actually decided by the LLM, but then become part of the history that the LLM is generating the next token from.

1:03:14And so that's how those embeds work. You can effectively have the LLM communicate with the outside world in the middle there. By it driving that or you don't even have to have it drive it. You could have software outside the LM that looks at the token set as it's appeared and then insert more tokens for it. So this is all the fun stuff you can do by running these models yourself. Yeah, I know. It's so fun.

1:03:42Well, friends, I'm here with Samar Abbas, co-founder and CEO of Temporal. Temporal is the platform developers use to build invincible applications. but what exactly is Temporal? Samar, how do you describe what Temporal does? I would say to explain Temporal is one of the hardest challenges of my life. It's a developer platform and it's a paradigm shift. I've been doing this technology for almost like 15 years. The way I typically describe it, imagine like all of us when we were writing documents in the 90s, I used to use Microsoft Word. I love the entire experience and everything, but still the thing that I hated the most is how many documents or how many edits I have lost because I forgot to save or like something bad happened and I lost my document.

1:04:27You get in the habit when you are writing up a document back in the 90s to do control S, literally every sentence you write. But in the 2000s, Google Doc doesn't even have a save button. So I believe software developers are still living in the 90s era where majority of the code they are writing is they have some state which needs to live beyond multiple request response majority of the development is load that state apply an event and then take some actions and store it back 80 percent of the software development is this constant load and save so that's exactly what temporal does what it gives you a platform where you write a function and during the execution of a function of failure happens we will resurrect that function on a different host and continue executing where you left off without you as a developer writing a single line of code for it okay if you're ready to leave the 90s and build like it's 2025 and you're ready to learn why companies like netflix doordash and stripe trust temporal as their secure scalable way to build invincible applications go to temporal.io once again temporal.io you can try their cloud for free or get started with open source it all starts at temporal.io.

1:05:42Is Go particularly well suited for this kind of tooling because of the nature of the language or is it just your favorite? Yeah, that's a really good question. The best programming language for LLMs today is Python. And I believe that is a historical artifact of the fact that all of the researchers working on generative models work in Python. And so they spend the most time testing it with Python and judging a model's results by Python output. There was a great example of this in one of the open benchmarks I looked at. And I believe this has all been corrected since then. This is all about a year old.

1:06:27There was a multi-language benchmark that tested how good a model is across multiple languages. And I opened up the source set for it and looked at some of the Go code, because I'm a Go programmer. And it had been machine translated from Python so that all of the variable names in this Go code used underscores instead of camel case. And the models were getting a certain percentage success rate generating these results. so Josh went through actually and made these more idiomatic in the Go style of using camel case and putting everything in the right place and the model gave much better results on this benchmark and so that's an example of where languages beyond the basic ones that the developers of the models care about are not being paid as much attention to as what you would like and things are getting a lot better there The models are much more sophisticated.

1:07:29The teams building them are much larger. They care about a larger set of languages. And so I don't think it's all as Python-centric as it used to be. But that is still very much the first and most important of the languages. As for how well Go works, it seems to work pretty well. Models are good at it by our benchmarks. Like we said, if we took the benchmarks and made them more Go-like, the models actually got better results. They have a real tendency to understand the language. we think it's a pretty good fit. There are definitely times and models struggle, but it's a garbage-collected language, which helps, because in just the same way the garbage collection reduces the cognitive load for programmers as they're writing programs, it reduces the load on the LLM in just the same way.

1:08:16They don't have to track the state of memory and when to free it, so they have a bit more thinking time to worry about solving your problem. So in that way, it's a good language. It's not too syntax heavy, but it's also, it doesn't have ambiguities that humans struggle with. Yeah, it seems to work well. Pretty small. Yeah. There aren't a lot of, I haven't seen much research into what is the best language for an LLM. It does seem like an eminently testable thing. Like there's some interesting, in fact, it may end up influencing programming language design in a sense of imagine you are building a new programming language and you develop a training set that's automatically generated based on translating some existing programs into your language and you train models for it.

1:09:08You could imagine tweaking the syntax of your new language, regenerating the training set, and then seeing if your benchmarks improve or not. You can imagine driving readability of programming languages based on your ability to train an LLM to write this language. And so, you know, there's lots of really fun things that will happen long term that, you know, I don't think anyone started on work like that yet. Right. So the level that you all are working at with Sketch, with Go in particular, is the prompting you're doing and the contexting and everything else that you're building, is it at a layer of abstraction where you could replace Go relatively easily with insert general programming language or is it like well that would be a new product that we would build like how hard is that yeah it's a good question it's all of the techniques we're applying a general uh but they are all very uh each each technique requires a lot of go specific implementation so in much the same way that like a lot of the techniques inside a language server uh for a programming language these are the systems inside vs code for generating information about programming languages The techniques are general for what methods are available on this object.

1:10:22Very similar in Go as they would be in Java, for example. But the specifics of implementing them for both languages are radically different. And I think it's a lot like that for Sketch. The tricks we're using for Sketch are very Go-specific. And if we wanted to build one for Ruby, we would have to build something very, very different. So yes, I consider it very much a Go product right now. And I really like that focus that that gives us. Because Go is a big enough problem on its own, let alone all of programming. Yeah, yeah. I'm just asking that because I wonder how valuable and important tooling like this would be for each language community to either provide or fund or hope that somebody builds.

1:11:05because if the LLM related tooling for Go because of Sketch just hypothetically becomes orders of magnitude more useful than just talking to ChatGPT about my Elixir code, for instance, well, that's a real advantage for Go and the Go community. I mean, it's great for productivity for Gophers. And going back to maybe the original question about should Tailscale have its own little chat bot built into it, does each community need to take up this mantle and say, we need better tooling? Or is it like VS Code should just do it for everybody? I mean, that's a really good question. Good job, dude. Yeah.

1:11:48So to, you know, I very much admire VS Code. I use it, which, you know, I don't actually have to admire a program to use it. That's better than admiring is using, I think. Yeah, that's right. But actually, I do both. I both admire it and use it. But to look at the inside of VS Code, which I've been doing a bunch of recently, VS Code didn't actually solve language servers for all programming languages. They built JavaScript and TypeScript, JSON, and I think they maintained the C Sharp plugin. They started the Go plugin, I think, and then it got taken over by the Go team at Google who now maintain the Go support in VS Code.

1:12:35I don't think the Microsoft team built the Ruby support in VS Code. I don't know who did the Python implementation, but a lot of the machinery in VS Code is actually community maintained for these various programming languages. And so I'm not sure there is another option than imagining a world where each of these communities supports the tooling in some form. I don't know if each programming language needs to go out and build their own Sketch. Maybe there is some generalizable intermediate layer, some equivalent of a language server that can be written to feed underlying models. Given our, we're just starting to explore this space.

1:13:18Sketch is very new. We basically started it sometime near the end of November. So there's not much to it yet. Yeah. Yeah, but so far what we've found is it's far more than the sort of language server environment that you get with VS Code. More machinery is needed to really give the LLM all the tooling it needs. The language server is very useful. We actually use the Go language server in Sketch. Go, please, is a big part of our infrastructure. It's really wonderful software. But there's far more to it than that, to the point where we need to maintain an entire Linux VM to support the tooling behind feeding the model.

1:14:02So what each community needs to provide, I think that's the research in progress. It's figuring that out. It's an interesting question and one I think will be open for a while. I do not want to see a world where Python continues to proliferate merely because of its previous position. I do see with tooling like Devon and Bolt and v0, these are very front-end JavaScript-y companies that are producing these things, which is fine. But it's like, if you are just going to go use that, it's going to produce for you a React and Next.js front-end with a Prisma-based backend. And like, it's all very much like these are the tools it does and that's all well and good, but that's going to proliferate more and more of that one thing where I'd love to see a diversity where it's like, yeah, where's the, is there a specific thing for rails people?

1:14:58Is there one for people who like, uh, zig moving outside of the world of web development, but you know what I'm saying? And I think your answer might be right, which is like, well, every community is going to have to provide some sort of like greasing of the skids for whatever editor is popular or used in order to make their tooling work well inside of these LLM-based helpers. Beyond just being like, ChatGPT knows about you, which is kind of like what people are at right now is like, does ChatGPT know about me? It's the new, am I Google-able? It's the new SEO at this point. I've heard people talk about that.

1:15:41A startup founder, who I won't know him, mentioned that they were busy retooling their product so that the foundation models under things like V0 and Bolt would be more likely to NPM include their package to solve a problem. That's super smart to do that right now. I agree. Did they divulge any of the how? What are the mechanical steps that you do that? I was actually really happy. that they said that their plan was to make it really easy to NPM IA package and not require a separate signup flow to actually get started. Oh, that's nice. Yeah, I thought it was wonderful. Like their solution to make their product more chat GPT-able, I guess you might say, is just make their product better.

1:16:27Which, you know, if that's... How avant-garde of them. Yeah. I'm sure one day we'll end up in the search engine optimization world of Frontier models, but today... It's definitely going to be some black magic for sale. You know, here's how you really do it. Yeah. I don't see why a Frontier model couldn't run an ad auction for deciding what fine-tuning set to bring in. I had, again, to talk about experiences, I was using one of the voice models and talking to it as I was walking down the street and I asked it some question about WD-40 because I had a squeaky door. And I think I described in my question WD-40 as a lubricant.

1:17:11And it turns out I just didn't understand that it's not a lubricant. It's a solvent. And the purpose of it is to remove grease. It took me years to realize that. I think someone finally told me because I've been using it as a lube all these years. Yeah. Oh my gosh. Why do you got to keep reapplying it? You know, it's not very good lube. Well, I just had your experience, but it was an LLM that told me. Oh, hilarious. And it mentioned in passing, it's like, yeah, you could also, you know, you could use WD-40 and then use a lubricant like, and then it listed some brand name. The moment I heard the brand name, I was like, oh, I see a frontier model could run an ad auction on fine tuning which brand name to inject there.

1:17:50And that would be a really... Yeah. It wouldn't require doing it into the pre-training months ahead of time. You could do that sort of on an hour by hour basis. so that world is coming and then once there's a world of ads there's a world of seo and all the rest of it well the more paramount they become and adam you can probably speak to this because you're injecting it in every aspect of your life like with the if it if the answer includes a product right there like you're just gonna be like all right i gotta get that sometimes you don't even realize kleenex is a product you think that that's a category but no that's a product yeah absolutely hard to tell honestly kleenex is an easy one for me because we don't have kleenex in australia where i'm from it so i came here everyone started calling tissues kleenex and it's uh it was a bit of a bit of a surprise to me it's like coke or coca-cola something like that you know yeah right exactly yeah uh you know i don't know if i've gotten some hallucinations let's just say on products and even limited information of what's the true good option when it comes to product search.

1:18:55I haven't done a ton of it, mainly on like the motherboard search. Like I want to do something that has the option for either an AMD Ryzen or Threadripper with more of a workstation enterprise class CPU and I want to like maximize some PCIe lanes. So I'm just trying to like figure out what's out there. I'd prefer the chat interface to find things versus the Google interface, which is search by nature to find things. But thus far, it hasn't been super fruitful. I think eventually it'd be cool, but it's not there yet. I imagine in the YouTube video of this, a little Intel Xeon banner will appear just as you say.

1:19:34That's right. Yeah, on the ad. So yeah, exactly. So I'm a fan of Intel Xeons too. I got the Intel Xeon 4210. Well, now it's really popping up. There you go. Bing, bing, bing. It's like in Silicon Valley when Dinesh was talking out loud and they had the, I think they had AI in the VR and it was, yeah, it was doing some cool stuff. It was pulling up ads real time. It's cool. It was cool. But yeah, Intel's cool. AMB is cool. But PCIe lanes are even cooler, you know? Give me the max. Buy 16, you know? David, maybe we close with this. For those who aren't gophers out there, of course, brand new, you know, hot off the press, still in development.

1:20:15Three months old, sketch.dev. Check it out if you're into Go and those kinds of things. But let's imagine you're just a Ruby programmer out there and you came across your blog post about what you've been doing. These three methods of working with AIs, you have autocomplete, you've got chat, you've got search. Where should folks get started if they haven't yet? First of all, is today the day? Like, is it worth it now or should I wait? And then secondly, if I am going to dive in and I just want to use it in my local environment to like just code better today, What would you suggest? Yeah, good question, especially for non-gophers.

1:20:52I would suggest trying out the code completion engines because they take a little bit of getting used to, but not a lot. And depending, if you're writing the sorts of programs they're good at, they're extremely helpful. They save a lot of typing. And it turns out, I was surprised to learn this, but what I learned from code completion engines is a lot of my programming is fundamentally typing limited. There's only so much my hands can do every day. and they're extremely helpful there the state of code completion engines is they're pretty good at all languages with a caveat that they're probably not very good at COBOL or Fortran but all the general languages especially like Ruby, I'd expect them to be decent at I suspect the world of code completion engines will get better at specific languages as people go deeper on the technology It's a thing I continue to work on.

1:21:47And so I feel confident that it can be improved. The other place that I think most programmers could get value today, if they're not a Go programmer, is writing small isolated pieces of code in a chat interface. So you could try out a chat GPT or a Claude, or if you really want to have some fun, run a local model and ask it to solve problems. like try Lama CPP, try Ollama, try these various local products, grab one of the really fun models. It's especially easy to try on a Mac with a unified memory. If you're on a PC, you might have to find a model that fits in your GPU. But it's a ton of fun.

1:22:31And use it to say, like, write me a Ruby function that takes these parameters and produces this result. And I suspect the model will give you a pretty good result. So those are the places I would start because those require the least amount of learning how to hold the model correctly. And you'll get the most benefit quickly. Love it. Good answer. I'm just wondering out loud and feel free not to know. But when it comes to prompting, I know we're past the age of magical incantations. But as you guys have been building out a product, which is basically sophisticated prompting, are there guides that are useful or they're like i remember finding a site i can't remember right now there's like people are just sharing their system prompts for certain things they do like maybe there's like a ruby prompting guide which makes it a little bit easier to get quality results out faster does either you guys know yeah i've seen people write guides like that i would say the guides i've read are now out of date uh like we were saying earlier they are guides go out of date.

1:23:38The thing I find most useful is to think of the model I'm talking to as someone who just joined the company. Sometimes I think of them as an intern, though every now and again, the models produce much better code than I can. But interns have done that too. That happens. uh you know it's uh so and then as you're writing the question for it imagine it like here's a you know i'm talking to a smart person who knows nothing about what i'm doing and they need some background and that gets me really far with the current frontier models and so that would be my general piece of advice that i think applies to any programming i agree with that too the random just give me X is you'll get a result, but you'll have to massage it further.

1:24:30You have to, it'll ask you more. I will often give it context. Like you've said, this intern, the smart person that's new to context. They don't have the background awareness that, uh, that you want somebody to have. And I'll often like give it a lot of that, have a particular request, but then also say, is there anything else I can give you or any other information you need to give me a successful way, just some version of like be successful with our, our goal. And it's strange even talking like that too, as I even say it out loud, like our goal as if it's, you know, human and Jared, you know where we're standing with this, but Jared and I have some history.

1:25:09So I've been very kind, please. And thank you. He's very nice. He talks to, we're on the same team and stuff. If it's, if it gives me a great result, I say fantastic. You know, I'm like, you know, why would I have high fives? Why would I be any different? You offer it money. no I haven't done that yet I gotta try that I gotta try that honestly but I will ask it like will do you could you be more successful is there anything else I can give you any more information I can give you to get us to our goal you know and I found that that's it like it's context it's a full circumference of the problem set as much as you can that makes sense and then you will have a more fruitful interaction.

1:25:47I will also say that I'm more exclusively using chat GPT. And so the O1 model, while it's expensive, let's just say, I don't have the expensive plan. I still don't feel like I can be the person that spends 200 bucks a month on this thing. I would much rather buy a GPU than spend 200 bucks a month. Somehow that math makes more sense to me. But then like, Oh, one's been pretty successful with thinking and iterating and, you know, being more precision, whereas four Oh was a bit more brute, but that's just my personal take. I think you're, I think you might be onto something with being nice to models.

1:26:27I caught myself being pretty curt with models a few months back. And you're discussing this a lot with Josh, he mentioned the conclusion we came to was one of the challenges of not being nice to models is it sort of trains you to not be nice to people. You're using all of the same tools and so it might just be good for you to be nice to models. I mean, I just don't, if it's humanistic even, you know, similar to a human, why not just be kind? You know, why not? I hate to break it to you, Adam, but this is not similar to a human. it's the iteration is it is it certainly is if i were collaborating if that was a human over there giving me the answers back it would be very human volleyball iterative if if it was right i get that it's not but i'm also like like david why not i think i'm i think i'm somewhere between your two positions uh because i i do think it's just a machine and it's oh yeah i don't think it's human Don't let me think that.

1:27:31Well, you kind of just said that you think it is. Did I? I mean, is that what I interpreted it? I just meant that if it, maybe what I mean by that, it's just to be more clear is kind of keen off what David said, which is like being kind. Just why not? I don't know. I'm not, I'm like overly like, thank you so much. You're amazing. Because it takes time. I think it's just, I'm a kind person. Were you nice to Google when it got you a result in the search engine? Like you just, well, this, it is not a problem. where there's a, an ebb and a flow or a back and a forth, you know, it's just simply returned an answer.

1:28:06Yeah. How are we doing? You know, ask you at the end, how are we doing? You know, I'm not, that being said, I'm not like, thank you very much. I'm just, I'm just antagonizing them at this point. Yeah, I know you are. You're really digging into me, but I, I can catch myself saying like, that's awesome or great job. Or I agree with that. Things, isms like that, like you would say to another person. if that makes you feel more like a nice human then you should just keep on doing that but i don't think it's doing anything for the computer i don't think it is either it's actually costing resources from the world listen to you i mean i think you're right it doesn't help the computer uh i say please and thank you to the models now so that i remember to say please and thank you to humans that's that's it it's it's purely you know i don't want to get into the habit you're training yourself exactly it's all about training that's fair self-training yeah i don't feel like i have to.

1:28:56I feel like it's just a natural atomism. How I do things. How I operate. Be yourself. It's who I am in my core. I'm a kind person. Well, David, thanks so much for joining us, man. Thanks for sharing all your knowledge. You've learned a lot and I've learned a lot from you, so we appreciate your time. This was a ton of fun. Thanks for having me.

1:29:16Okay, so hopefully hearing David's experience has helped you on your programming with LLM's journey. I'm sure you have thoughts on the matter, let us hear them in the comments. Yes, there is a Zulip topic dedicated to this episode, and I'm sure there's lots of insightful things being posted by ChangeLog community members right after they listen. There's a link in your show notes so you can see what's up and join for zero dollars at changelog.com slash community. Let's give one more shout out to our sponsors of this awesome conversation. Thank you to fly.io, to retool, retool.com slash changelog, to Temporal, find them at Temporal.io, and to Augment Code, head to AugmentCode.com to get started.

1:29:58And thanks, of course, to our beat freak in residence, the one, the only, Breakmaster Cylinder. Yeah, I like beats. Oh, and we have a little bonus for our favorite kind of listener. That's a Changelog++ listener, of course. Stay tuned, friends. We go even one layer deeper on what a potential tailscale AI might look like. If you aren't a Plus Plus member, head to changelog.com slash plus plus today. Right now even. You're free right now, aren't you? Okay, that's it. This one's done, but we'll talk to you again on Change Login, friends, on Friday. Bye, y 'all.

1:30:55Thank you.

1:31:19Game on!