Prompting the future

0:05Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at Fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents. So you can launch your app near your users. Learn more at Fly.io.

0:42Welcome to another episode of Practical AI. This is Daniel Whitenack. I am founder and CEO at Prediction Guard, and I am joined as always by my co-host, Chris Benson, who is a principal AI research engineer at Lockheed Martin. How are you doing, Chris? I'm doing fine. How's it going today, Daniel. It's going great. I'm pretty excited to prompt our guest today and hear what he has to say. We're joined today by the prompt master, Jared Zonerike, who is founder at PromptLayer. How are you doing, Jared? I am doing well. Excited for this. We're excited to have you. It seems like maybe from my perspective, there was kind of the release of all of this generative AI stuff.

1:30And then there was this realization that there's kind of a new skill needed around this thing called prompt engineering. And then it seems like some people have kind of like, I don't know if they've moved past the term or they've tried to develop other terms. I see other terms being developed around like AI engineering and such and as related to generative AI. So could you just give us a sense of like from your perspective who you know is obviously building things for prompt engineering maybe to start out kind of what is prompt engineering and from your perspective how have you seen it develop as a skill over the past year since people have been thinking a lot about prompting generative models yeah the last year since the word was invented or maybe a little more than a year ago.

2:22But yeah, no, I think it's a really good question. I think this question of what does prompt engineering even mean is also a good question. And I think I'll tell you how we think about it. So honestly, so for PromptLayer, we consider ourselves a prompt engineering platform. So we've really steered into this term that is kind of overloaded. We've embraced it for sure. And it was half by accident. and I would say half realizing it was kind of beneficial to us because the fact that it has no definition means anyone who's kind of getting into LLMs and getting into prompt engineering says, oh, there's a prompt engineering platform.

3:01Maybe I need that. And they don't even necessarily know what it is. So it kind of helps us a little bit there. I guess prompt engineering, I first started hearing that term, I'd say GPT-3 days, kind of a little bit before chat GPT, maybe GPT-2, back when everyone was just using the OpenAI Playground. And you'd start to hear a little bit about prompt engineering and stuff like that. And it was kind of cool. But that was, I guess, the days before it clicked for everyone. Maybe you can call it that. Days before ChatTPD came out and before people really realized how much potential this technology has.

3:37So that was when I think prompt engineering first became a word and then kind of got even more, you know, scale AI, I think is famous for maybe the first one to publicly hire someone with a role prompt engineer. And from there, we called their platform a prompt engineering platform. At the beginning, people brought a lot of people to us who had no idea what prompt engineering is. And kind of the definition we've started to roll with is, as a company, we consider prompt engineering to be the tuning of the inputs to the LLM. So the prompt is the main input, but it also includes what model are you using?

4:16What are your temperature? What are your other hyperparameters? But the whole process of prompt engineering to us is what goes in and what comes out. And that specifically is, and I can talk a little bit more about this if it's interesting, but specifically a little bit different to the MLOps definition or the standard machine learning definitions of hyperparameter tuning and standard traditional ML. And specifically, we call ourselves a prompt engineering platform and not an LLM ops platform for that reason, because I do think there is slightly a difference. So I don't know if I answered the question fully, but those are my thoughts on prompt engineering.

4:59As we all dived into it, like you talked about, you know, the open AI playground originally that I think everybody kind of dips their toe into first. at least before the chat GPT release days. With that, one of the things I discovered as other models started coming out was some of the skills I was developing for prompting did not always translate as I expected across to other models. Have you seen one of those things where every model had its own kind of variations of what seemed to work from a productivity and output standpoint? Any thoughts around that? Like, how do you guys see that with all these different models coming out and little variations across them in terms of how they respond.

5:38Yeah, that's for sure a good observation. I think it's become clear that each model, I mean, each model is made a little bit differently. So you have to talk to it a little bit differently. I think these differences are going to maybe get a little less significant over the future. You know, when I guess ChattyBD came out and the big thing was if you were nicer to the model, you're going to get a better answer maybe because stack overflow questions that are nicer get better answers or something like that. And I think now you have people talking about, oh, if my grandma's going to die, you need to answer this, or I'm going to tip you$100 if you answer this.

6:18These are all, I think, tricks that work today. They're not going to last forever. These are like little things people figure out. But the part you mentioned of talking to different models differently, I think is not going away. A lot of these models are made very differently. We think it's pretty conclusive now that we're not going to live in a world with an open AI monopoly of language models. I think just a week or two ago, I saw a good tweet. Like, now we have Mistral, Claude, GPT-4, and a few others that are all really good. And they're all made somewhat differently, and there are intricacies.

6:53And I think our philosophy and what we talk to our users about regarding prompt engineering is think about it as a black box. It's almost helpful to be kind of a little bit naive and stupid here and not try to understand how an LLM works and just try to track the inputs to the outputs, if that makes sense. Have you found any difference from people that are coming maybe from a deep sort of like data science background where they like are overanalyzing everything and then other people who are coming maybe from a not technical background, but there may be domain experts and they're getting into developing these prompts?

7:29Have you seen different struggles on each side of that spectrum? Because you have this very interesting mix of people that are trying to be prompt engineers, quote unquote, some of which I've seen are very much just non-technical domain experts who are really good at even psychology and writing narrative instructions, being articulate. And then you kind of have this other side, which is the data science side, and they're really into modeling and wanting to analyze all of the outputs and that sort of thing. So have you seen different struggles on both sides of that in terms of being effective prompt engineers?

8:13I think you put it well that there's kind of these two groups coming at LMs. Traditionally, I mean, this is what makes LMs so cool to me, is that traditionally ML, traditional machine learning standard, like math and machine learning, kind of need a PhD. Maybe you don't need it, but a PhD is very helpful. It's intense how you're building these models, doing a lot by hand still. You're doing a lot of this tweaking by hand and traditional ML. And then OpenAI came out with this amazing API. And I've done a lot with DevRel in the past and hackathons and stuff like that and helping companies make APIs.

8:50And in my opinion, OpenAI's API, it's like the best docs I've ever read in my life. It's so simple. You just give it text and you get text out. And like you said, this kind of opened up this completely new technology that's just so much better than everything else to non-technical people. You don't need a PhD to be able to understand how to communicate. And I think it brings about this new skill set, which is prompt engineering, which, in my opinion, is kind of a mixture of communication and like being able to write succinctly, but also being able to think algorithmically. So I don't know the exact word for thinking algorithmically.

9:28I've heard Stephen Wolframs use that word. I like that way to talk about it, but kind of just a scientific method. Do you know how to think in terms of creating a hypothesis, trying it out, tracking it? Are you strategic about this? And I think it's the same challenge on both sides. I think some people try to overcomplicate it if you're coming from an ML background and you try to understand why a certain token gives you an output. And I almost think these things are getting more complicated, not less. You kind of just need to take the naive approach and say, hey, I'm just going to talk to it. I'm going to try to get the output I want.

10:00I'm going to keep trying stuff till it works. As someone who's used the APIs a lot, and you're talking about OpenAI being so good, so many folks listening to this may have only used things like the normal chat interfaces on each of the models when they've gone and tried them out or paid for a subscription for the top end and stuff and have never touched the APIs at all. Could you take a moment and just talk about what an API experience is like to someone who has done a lot? I'm kind of taking advantage of you as an expert in the area. Share that a little bit with listeners just so they kind of get the other side of that because not everybody, probably the vast majority of them don't.

10:39I'll explain in that just to sidetrack myself for a sentence. I think the fact that you use the word expert is so funny because it's such a new field. It's almost amazing. I tell everybody, you could become an expert in this thing very easily. Like nobody really knows what's going on. You kind of just need to study for a week and you're an expert, which is a very unique place to be. But anyway, it makes it fun to be able to dive into something and get as deep as the leaders in the field. So, yeah, 100 percent. And just to be on the cutting edge and know that nobody really knows what they're doing here.

11:10Some people do, but there's very few of them. So regarding the APIs, ChatTBT, the way I think about it and the way I usually explain it is it's basically a very thin wrapper on top of the API. Every time you talk to ChatGPT, behind the scenes, it's using this LLM technology. It's sending your message and a little bit of a preamble before your message. And the preamble is basically, we can call it the prompt. And it's basically saying, hey, you're an AI assistant. Make sure to be helpful to the user. Maybe don't be controversial. You can use a calculator if you need, stuff like that. And then giving the user messages.

11:47And that's all it is. And the process of prompt engineering is how do you tweak that preamble and how do you get it to respond in the way you want to respond by telling it what to do. And when I talk about the API, I'm talking about the things OpenAI has exposed to let you build your own chat QPT and let you build your own products on top of it. And I just think OpenAI has done, if you want to get started and you haven't touched these APIs today, the best, best thing to do is just go to OpenAI's docs and just read the getting started tutorial. It's really well done. And I think as someone who's running a developer tools company, I'll tell you how hard it is to write good docs.

12:27I think our docs, from my perspective, I think they should be much better. Other people say they're great, but there's always room for improvement and it's just a very hard thing to do. So that's where you should go. From your perspective, over the last year, as people have gotten into doing this practice of prompt engineering, have you generally seen people that are engaging with your platform be sort of more informed coming in? Like they've done that experimentation with ChatGPT or the APIs and they are like using words like few shot, whatever, blah, blah, blah. And like this stuff, or is it still kind of like people coming in that say, I know I think I need to have a prompt engineering platform, but I'm kind of don't know where to start.

13:18Have you seen that shift even over in this last year? I would say we're still very early in this. And there's a few leaders, but everything's up for grabs in terms of AI products. I don't buy the whole notion of it's only the incumbents who are going to be able to use AI. There's so much. The AI products people are making today are just scratching the surface. Having said that, I have seen a bit of a change since we launched our product. We launched our product January of last year. So January of 2023. I guess that was a few months after Jack Tipiti came out. But these things, I think we were the first prompt engineering platform.

13:54Maybe there's some argument there, but we were one of the first at least. And when we launched our product, we had a lot of indie hackers and individual hobbyists using our platform. That was the whole community back then. That lasted for a little. Then we kind of moved on to AI-first startups. So like one or two-person startups. These are like the really cutting-edge ones. We used one of them. It's a great company to actually refactor our whole code base into TypeScript. A lot of really cool stuff. And then from there, I would say starting in the fall, and I've heard this from a few other founders in the space, it felt like there was starting to become a real shift where real companies, and when I say real, I mean maybe companies that actually make money, started actually getting serious about AI and getting serious about LLMs.

14:43And I think we're still seeing that maturation continuing where these real teams are building AI products that they care about and not just like Twitter demos are one thing. Promlayer is interesting for a Twitter demo, but Promlayer becomes really useful for a team that is serious about building their product and has multiple stakeholders and wants to collaborate. And so I am seeing that this shift is still happening. More and more companies are getting serious about their LLM products and getting value and revenue from them. but we're still at the very, very beginning of the curve. So a lot more to come.

15:19I think that was a great sort of intro into prompt engineering and the state of prompt engineering. I'm wondering if you could help us maybe understand, yes, it's good if people get hands-on with these models, kind of gain some intuition about how they behave and different ways that you prompt them. What is it about this discipline of prompt engineering that needs sort of systematic ways of managing your prompting methodologies? And how is that different to or the same as different sorts of engineering in the past? You know, we've always had version control and that sort of thing. What's kind of unique and not unique about this discipline of prompt engineering in terms of how you need to approach it systematically?

16:07Just starting from first principles, there's one fundamental thing that's changed. And it's that we're now building upon a probabilistic technology. So we're now building on a technology that sometimes gives us some answer, sometimes gives us another answer, and is trending to be more confusing on why it gives one answer and not less. And confusing, I mean, yes, theoretically, it is deterministic. You can really dive into the weights and maybe figure it out. But nobody's really practically going to do that. It's virtually too hard for 99.9 % of use cases with models. So we're at a place where you're working with a black box now.

16:46And yes, some servers, some architectures become black boxes because of bad code. But that's a different type of black. That's not a real black box, but we're building technology. That's the code I write. Yeah, me too. That's why I'm going to get banned from our repo soon. But yeah, so you're building technology on this black box and you need to think about it differently. And I think this is a big philosophy we have at PromptLayer. I think we have the philosophy of we built a lot of great stuff in traditional software and traditional machine learning and had a lot of great learnings. Git is a fantastic tool.

17:23Version control is important. Access controls are important. Test driven development is important. But do we necessarily want to take one-to-one of everything? Not really. I think the biggest difference between LLM-based development and building AI applications versus building standard software is who are the stakeholders, like we were talking about earlier. Now you can have subject matter experts who are not necessarily software engineers, but rather these prompt engineers, these AI whisperers, these people who are able to talk to the black box and able to communicate with the AI, we could call it in a little sci-fi sense.

18:01But we have this new stakeholder in the process of software engineering who is not going to jump into the code and not going to jump into Git. And that's why, at least for PromptLayer, we've taken a very first principles ground up approach where we're saying, hey, what can we learn from normal software and how people work together on software and do version control and collaborate? And how can we take that into LLMs and bring in new stakeholders and bring in new collaborators and let people actually build on this black box technology in a systematic way? So hopefully that makes sense. It does.

18:37I'm curious. You said some things there that really kind of piqued my interest that you kind of were contrasting it with deterministic programming that we've all kind of grown up with. And now we're in this new age and we have these non-deterministic things and you can give it the same prompt and it may or may not on any given day give you the same answer back. So how has that kind of fundamentally changed software development in general? And I'm encompassing all the things when I say software development, both the AI and the systems around it to feed it, because when you're dealing with that potentially unexpected return and that non-deterministic black box that you're talking about, how do people handle that when they're trying to devise and say, hey, I want to use a model in the thing that I'm building.

19:23How do they change in the way they think about that? I don't know how people think, but I can tell you a symptom of how they think, which is we work with a lot of LLM teams, of course, because they're using our platform. And we talk to them. We're always trying to talk to people, trying to figure out what to improve. And one of the big pieces we've come to with the prompt layer is that the iteration cycle of prompt engineering is different than software engineering. So what I mean by that is your code deploys and continuous integration and that whole sort of thing is happening at a different cadence, almost always in mature software than prompt engineering because of a lot of reasons.

20:04Maybe you're updating the prompt frequently. Maybe, again, different stakeholders are updating the prompt. That's why we encourage people to have their prompts in a CMS or we call it a prompt registry. You can put it in a Postgres database, something like that. But you don't want to block your prompt engineering cycle on edge deployments. And that's, I think, the symptom of this new thought pattern of how you're building, let's call it black box software, and how you're kind of thinking through these problems where they are different problem sets. Also, you're not solving the same type of software problems.

20:39you're solving that's called language problems or things that can employ this new type of software and so it's not just a different like way to build but it's also a different way to think about it and then we can also get into like how do you code now how that's different and how uh i think uh maybe this is maybe a more contract i don't i don't i don't like to predict things but uh because i I think it's kind of a fool's errand. But if I had to predict one thing, I think as a lazy programmer, there's something really nice about just having a little block of LM do something for me. So for example, parsing strings or parsing chunks of text and reordering it.

21:22You can do that deterministically and it'll probably be better. But there's probably a world where models are going to get cheap and quick enough where maybe it's worth the engineering time not to build it well and just outsource it to AI. But maybe that's a whole different tangent. Yeah. And maybe you could get into a little bit of the implications of these prompts in a registry, right? Because you could have like these sort of random tasks that I don't know if you all saw the Devon thing that's been going around where it's like a junior engineer agent that can, you know, write scripts and interact with software.

22:04documentation. And today I was thinking about that similar to what you were saying, Jared. It's like, hey, well, I could go and take all of these strings and go and interact with the API to translate them into another language, which is what the task was that I was doing. Or I could just like say, hey, write a script that does this and do it for me. So there's those random things. But then there's another type of prompt, which as soon as you're starting to expose a system to end users, then small changes in the prompt, like on the backend, could produce very different changes in the behavior to your actual end users and cause actual problems.

22:47So could you talk a little bit about maybe the implications of changes in your prompts and best practices that you've found around... I know we're talking a lot about prompt versioning and registries around those prompts. And I know that PromptLayer is thinking about more than that around evaluation and other things. But yeah, maybe before we go to those other things, could you talk about what you've seen in terms of how people are managing these different prompts, some that have very low risk in terms of changing them very often, and maybe some that have actually a large risk in terms of maybe small changes in the prompt?

23:23Yes, I think there's a life cycle here for how you care about your prompt. And I think every prompt in any mature product. If you have a product, an AI product that's making your company$10 million a year or something like that, you're going to care about any change to it. And I think there's that cycle. So let's say that's the end stage. At the beginning, you probably are just going to ship any prompt. You're just going to write a prompt. It's going to kind of work. You're going to try it once or twice. Say, all right, good enough. Let's get the MVP out. In that case, maybe the prompt is in your code.

23:58Maybe don't care so much about what you were just saying about like updating, having a breaking change. All right, whatever. I got to get it out. Let me get five people using it. All right. So you've done that. What's next? You probably now have like five different prompts on your system. They're scattered everywhere. I was just talking to a founder who was visiting us, our office today, who was actually had this exact same story. And then he moved all his prompts to a text file. Or actually, in this case, it was just like a.ts file on his system. So now you have your prompts in one place. That's the next step, but still in your code base, it's still linked to EngDeploys.

24:35And like you said, you still have no way of knowing if you push a new one, what happened? Like, is it breaking 20 % of our use cases? Not great, but I would call this stage, I like this word for it, like vibe-based prompt engineering. So this is the vibe-based prompt engineering stage where you're kind of writing a prompt, you're testing it in the playground, you're doing it once or twice, and you're just judging Like you're just looking, okay, yeah, it's pretty good. This lasts a little bit. The time when this is no longer good enough is usually when your product's either getting to a greater level of maturity, maybe you're rolling it out to GA, or you're adding more stakeholders to the team.

25:14Maybe you're adding a PM, maybe you're adding a content writer or a subject matter, like we were talking about earlier, a psychologist or a lawyer or something like that. And you need more people involved. So now you have a non-technical person writing your prompts who isn't really capable of building out the whole dev workflow to test it out. You really want to make sure it doesn't break everything. So there's a few things I've seen people, a few strategies I've seen people employ here. One is, of course, traditional software. Let's borrow some stuff, A-B testing. Let's release it for some people.

25:48if we're monitoring user feedback, maybe if users are giving us a thumbs up, thumbs down, we should be able to see it pretty quickly. Also having prod, staging, dev staging, different like we call them release labels, a lot of words for them. Let's call that category slow releases. So that's one way. Then there's two other big ways of solving it. Second way, regression tests. So again, another concept borrowed from software engineering. Let's find cases where it's failing and see if we succeed. in this test case. Third, which is also kind of like regression tests, I guess is backtesting. Let's just run it on old examples and see if it changes.