The last mile of AI app development

0:06Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at Fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

0:43Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist building a tool called Prediction Guard. I'm joined as always by my co-host, Chris Benson, who's a tech strategist at Lockheed Martin. How are you doing, Chris? Doing very well. We're still in 2023, the most exciting year in AI history. It is, you know. it's hard to keep up, but it's also sometimes hard to understand like what of these cool demos and models and integrations are like actually production ready and how are people actually taking these things into production. And we're really happy to have with us today, Travis Fisher, who is a founder and CEO at a stealth AI startup and is focused a hundred percent on that delivering products with AI.

1:33So we're happy to have you here. Welcome, Travis. Thank you, guys. It's a pleasure to be here. Looking forward to the conversation. Yeah. Well, on Twitter, you posted this diagram, which I think maybe you have penned right now, which is how to use large language models effectively. And it's sort of like a start simple to complex scale. And I found that really great. And I've actually shared that diagram with a number of people and various Slack channels and all. Awesome. This is how you should be thinking. How did you, maybe not specifically with that figure, which we can talk about, but like, how did you get into thinking about how to use large language models effectively to like actually how to build products with these models?

2:17Yeah, it's a great, great question. So I'll give you kind of my quick, what I've been up to in the last six months, which is going to answer some of this stuff. I'm a huge fan of open source. when ChatGPT launched on November 30th, 48 hours later, I released the ChatGPT NPM package, which was using unofficial API. And it just allowed like thousands of developers to go and start building with this cool new thing that, you know, like the GPT series and LLs have been along for a while before that, but it was kind of this step function in terms of just their mainstream adoption and really just caught people's attention.

2:51You know, after that, I released the ChatGPT Twitter bot, which now has about 123 ,000 followers. I run a group called ChatGPT Hackers.dev that has about 9 ,500 AI developers. Just a whole spectrum of people, right? We have like researchers in there and then we have like prompt engineering script kitties. And because, I mean, my background is computer science and I do have some formal education in machine learning, but it's not like I'm an AI expert like my idea means, right? Like, and what has really captured me really over the past year or so has just been the rate of progress and trying to wrap my head around it and understand it.

3:27And because it's been moving so quickly, I've been optimizing for my rate of learning. And I personally learn best by building out in public and building open source and just sharing what I learn as I go. So, you know, I think there's a lot of like complexity and terms in AI, as you guys know. And to some degree, even just having a mental model of like, what are the different approaches that you could start with or how to approach solving a problem is already like a difficult starting place, right? And I know for the how to use LLMs effectively, my inspiration there was Andre Kaprati recently, maybe a month or two ago, tweeted something about, you know, all these big companies are interested in using AI.

4:06They're aware that they should be using it to some extent. And so they're like, well, we need to hire a team of ML engineers and get on this, right? And the huge unlock now is with these foundational models, like for most problems, you don't need to do that. And, you know, yeah, there's the production side, the practical side, and I'm sure we'll get into that. But like in terms of where to start and starting simple and actually validating for your business use case that you can actually solve it with AI, that you understand the problem domain enough that you have, whether it's a training data or that you're actually solving a real customer problem, like starting as simple as possible with hosted foundational models.

4:42a lot of times is a great way to get started and just to validate quickly. You know, as you kind of inevitably find points where your workflow breaks down or where you're not getting a quality or the cost or some hard constraints like security, privacy, you know, of your data, there's kind of this ladder of complexity I like to look at. And, you know, you start with just prompt engineering up at the top. And then it's about like, well, how much can I reduce hallucinations or add domain-specific context into my prompt by, you know doing um information retrieval and then you know at some point you're like well a single prompt isn't doing it so maybe i add some iterative process to that where i use another language model to uh there's all these techniques for doing kind of multi-step prompting but you can do all of that with a hosted model and you can get like 95 of the way there for a lot of problems and domains these days in a way that was previously like locked behind you know proprietary data providers and you had to have so many resources to be able to do that so it's really this like democratizing point in the industry at the applied AI level that we're at right now.

5:46And I think from the conversations I've been having with folks who are a lot of them, you know, like full stack TypeScript devs for building applications, right? And they want to use AI, they know it's cool. They don't know how to get started or they're like, oh, I need to learn Python. I need to train these custom models and stuff. And like all of that is super important and it comes into play at a time or for particular types of problems. But the majority of people for getting started, like start simple is the main takeaway from that. I think that's a great insight. And I think that's one of the places where so many people go wrong is jumping into too much complexity.

6:21They don't find a simple need and potentially even don't look for things that work just as fine that are not AI in that way. So I love the go simple and build from their philosophy. I think that's incredibly practical. I get the sense that a lot of this sort of like chaining and like bringing models together, doing the information retrieval, it's sort of like almost like a hacking culture around this language model prompting, which is really cool. And like that can go so, so far. Maybe there's like you say, there's like privacy or domain specific concerns with like enterprise use cases. But in your like you mentioned the community that you've kind of built up and you're part of on Discord.

7:06What are some of the things that have maybe like surprised you that you've seen that? Hey, I didn't even think that maybe this was possible with just this layer of like using a hosted model, using pre-training, using retrieval, whatever it is. What are some of those things that you've seen that kind of surprise you or maybe like help develop your thinking around this topic? I have a few examples in one story. Examples would be folks who are taking these models and like applying it to their personal finances. Right. And there's one guy in our discord who's like an ex-hedge fund guy. And he created a very basic agent that uses a large language.

7:51I think he's probably using GPT-4 to take this unstructured information from his bank's website about his expenses and extract structured information about that. And then he can graph it and whatever. So I think there's a lot of hacking going on around this stuff. It is very, very early. Another story of something that surprised me, and this is just a fun story. But when I released the kind of unofficial API wrapper for ChatGPT, we kind of had this cat and mouse game going back and forth with OpenAI for a while. because apparently there was kind of a group within OpenAI that was like, oh, this is amazing.

8:23Look what the open source community is doing. They're building all this cool stuff. And then there was another group that was like, well, we're going to have the official API eventually. We want to control this stuff, right? So there was kind of this back and forth, right? And at one point, our community found a public model, but it wasn't like publicly disclosed. It was security through obscurity, but it was a fine-tuned chat model that ChatGPT was actually using at one point. and all of the open source projects started to use this thing. And there were tens of thousands of actual real consumers at the end, you know, who were building on top of this.

8:55And of course, opening, I knew that we were doing this. I talked with one of our security engineers about this after the fact. And they, instead of like what you would expect, just shutting it off, instead they switched it out with what they call cat GPT. And just all of a sudden, one day in our discord, we started getting hundreds of messages from users saying, i think i got hacked i'm seeing all these meows like in response to a thing right so it goes to show you know the moral here and i ended up hearing from the open ai uh engineer that they were watching our discord taking screenshots and laughing their asses up um you know at this happening but it goes to show that like one the kind of level of um there's no switching costs to these things right it's like text in text out fairly basic and there's like entire new venues of like vulnerabilities of like swapping it out with a you know a cat or something like what does security look like in this world when um it just thought it was an interesting kind of anecdote probably that vulnerability of like all of a sudden getting meows like that is a possibility but i'm wondering like as you've spent a lot of time with these models you've also So like you're building products on top of these models, like from your perspective, taking an LLM integration to that sort of last mile of like integrated into a product, supporting users, what are the things that should be on either developers or data scientists minds as they think about like taking the step from like demo to like product integration, I guess would be the question.

10:36So I like to say that absolutely everything in engineering is about trade-offs and it's about really thoroughly understanding trade-offs and then being able to effectively communicate those trade-offs and the pros and cons and everything. And it really boils down to those two things like over and over. So let's talk about some of the trade-offs that are most important to using language models in practice. You have the most obvious one, which is quality. Like, can I use these language models to actually perform the task that I want? you know you have uh oftentimes secondary but equally important trade-offs like how much does this cost to run in production uh what is the latency for my use case for the end users how consistent and reliable is it like can i have actual uh is my use case fault tolerant which is a great initial question because we're kind of moving from a world of like very deterministic human driving the program to a world where the more control you give to the language models and their reasoning abilities, this is getting more into the agentic side of things, the more that it becomes slightly non-deterministic or very non-deterministic.

11:37And so the ability to have guardrails around these things, the ability to have consistency and predictability is extremely important. And one of the first questions that you should ask yourself, are you thinking about like integrating with LLMs is for your particular use case, for your job to be done, for your customers, to what extent, you know, do you need a hundred percent reliability versus like 99 % reliability? And that may sound like a little bit for certain domains of problems. It's everything. Right. And so that's one fundamental question. There are techniques and we can talk about them.

12:09I'm sure you guys are very aware as well of like going from that 99%, getting close, adding extra nines of reliability. And, you know, that's also a very active area of research where folks are actively figuring out ways to increase the reliability of these models. But the fundamental trade-offs are you know quality cost latency and reliability and using a hosted model it's going to be great for quickly like with minimal resources and validating your use case for a lot of those types of trade-offs it may make more sense than uh to use a local model and there's kind of been this cambrian explosion of open source large language models and other you know specialized machine learning models and we're going to continue to see that proliferate I like to think of the open source kind of state of the art as, you know, six to 12 months behind the proprietary versions.

12:56We'll see if that holds. But, you know, it's kind of because there's like zero switching costs with these models, because there's just so much competition, the price is going to keep going down over time. We're going to see the open source side of these models continue to get more powerful. and so for a lot of use cases where you're dealing with well maybe i need ultra low latency on device or maybe you know cost is a factor and i need to be running in my own data centers or maybe you know you need to after a certain point once you validate your use case you want to fine tune and distill the model down and have a really locked in like a checkpointed this is like completely unit tested this is you know evaluated version of things and i think we're at the stage right now where there's so much like hype and so many people building ai applications and demos and that's great right just getting it out there proliferating through open source through twitter whatever it is right this is awesome but the version of that last mile and the productionization concerns really need to dive deep on all of these kind of fundamental trade-offs that i'm talking about in the hosted models versus local models and and you know fine-tuning distillation they all become really important very quickly

14:13so uh before the break as we were talking about these different characteristics that kind of affect applied ai and affect deployment i was really taken by the fact that so many of them are not really ai specific you know if you could almost argue that applied ai and in so many ways is about software. It's about the systems. It's now about cloud. It's about all these other things blended together to produce solutions that are productive in the world and have value for people and organizations. We talked about unit testing and stuff like that. What is your thinking around kind of the integration of all those things?

14:51Because the model itself, to your point about hype, still kind of gets all the attention and the amazing things. And it is amazing what we're doing. but to make this stuff work in life, there's all these other concerns that there's so many cool things in 2023 happening on the model side that the other 99 % to make it real kind of gets, when you're working with people around understanding how all this fits together so that they can do that, how do you frame that so that their attention gets on the right thing, their budgets are properly allocated to attend to all the things? I've seen organizations really struggle with that because they go into it with hype, focusing on just the model and building skill sets and budgets around the model.

15:37And then they try to figure out the whole thing with clouds and deployment and things afterwards. And they have a hard time. How do you navigate that given the hype cycle that we're operating in? My first piece of advice would be that for your particular use case, your job to be done, whatever business use case you're solving, and to keep in mind that AI, like all software is a tool and it may be a really shiny tool it may be a tool that is evolving very quickly in front of us right now it's a very powerful tool but it is a tool to solve you know a business use case and a problem for humans so rooting you know the framing in that i think is very important the second thing i'll say is a lot of ai right now and especially the stuff that gets a light shined on it and in the open because the application layer is so new and there's so much low-ganging fruit, you know, as you said, like we need to have more emphasis on the engineering rigor under the hood.

16:31And so one practical piece of advice there is to really focus on an evaluation set for your particular use case. And you might have existing data, you might have existing kind of input-output pairs for your particular example. You might have, you might not have that, but like starting from there and working backwards of like, this is what the end user is going to see. and then working backwards from that to think about, well, how can I use language models or other expert-focused machine learning models to solve that, I think is very important because that also gives you a grounded North Star. So much of the prompt engineering and tuning of these models is based around, well, I think this is going to work better or I eyeball it on this one example and it seems to work for this, right?

17:16But really applying some fundamental engineering rigor at that level where you have an evaluation set that you can track, that you can improve over time, that you can, and not just tracking the quality of these models, but, you know, tracking the other trade-offs in terms of pricing, latency, recall, like there's a whole slew of trade-offs that can matter depending on your particular, you know, use case. And then the other piece of practical advice I would say is the kind of diagram of this ladder of complexity that I was referring to before. Like every time you take a step down that ladder of complexity from using a hosted model, just using prompting, and then going to, you know, some type of information retrieval embedded in the context to having a multiple chains of prompts to going down to fine tuning, you know, a hosted model or fine tuning a local model that the very, very bottom is building your own model, right?

18:06Like every time you take a step down that ladder of complexity, it adds engineering complexity. It's going to make your solution more complex to maintain. And so really like having a good handle on how you can start simple and only move down, you know when you need to or when you hit a constraint like okay this is great and i have a working solution with a hosted api but now i need to worry about the price because i'm going to production and you know the unit economics like maybe at that point then you think about well now i have this great solution and i can auto generate an eval set for myself and you know have a bunch of inputs and outputs and fine-tune a model that is hyper distilled and efficient and focused that's great don't start there for most use cases right the one other thing i would say at the practical level is where language models tend to break down or lack reliability is oftentimes when you're trying to give too much to the model to do it once.

18:55And so breaking the problem down into sub problems that are a lot more focused is one of the most practical. Like I just find myself telling people over and over again, it's like, okay, that's awesome. Break your problem up into some problems. And you know, how to do that is a whole arc forward itself. And maybe someday in the near future, language models will do that for us. I don't know, right? That's getting into the more sensational side of things. But as a general principle, breaking your problem up into sub problems, thinking about how you can articulate your problem as succinctly as possible in a way that is native to the language models is a really key practice.

19:31I love how you talked about like evaluation, forming your evaluation set, getting some ground truth, also breaking up your problem, maybe having an evaluation set for each of those sub problems would be a good idea. I think there's this general perception that large language models are this kind of unique thing. These chat interfaces are this kind of unique thing. Like we can't, how do you like evaluate that in the way, like, I think what people have in their mind is, oh, if I'm doing sentiment analysis, it's either positive or negative or neutral. And I can like calculate an accuracy, for example, whereas they might struggle to think about, okay, well, there's this output from this language model.

20:14It's seems coherent and fluent. Like, how do I evaluate this? And so I think there's maybe a bit of confusion around the evaluation side. Can you share any tips or thoughts in terms of what you found to be useful in your own work in terms of evaluation sets and like how you think about how good the output of a language model is? My first thought would be the less that it's about me thinking about how good it is and the more that it can be objective like using some constant way of evaluating it the better there's one project that i i really like recently uh by lance martin it's called auto evaluator i don't know if you guys have seen it but it's specifically for the uh domain of qa so question answering and he recently partnered with lang chain to create a hosted version of it but the way that i think about this is a little abstract and it's really like starting from your job to be done oftentimes sentiment analysis isn't the job to be done It's like a piece of a job to be done, right?

21:16So again, it's like breaking up the problem and understanding how to think about and structure those problems as whether it's an expert model that just does sentiment analysis, or it's using a large language model that like it can do sentiment analysis. It's really good at that, but it's also like it can do a whole bunch of other things as well. So the more focused your task is, the more clearly articulated your task is, and the more structured of the output that you have at the individual LLM call level, the better and the easier it is to create reliability around these things and to actually test them with more traditional software engineering practices like writing unit tests or integration tests.

21:53One thing that I'm actively working on right now for the TypeScript community is a way to invoke large language models and have structured guards on them. I know like Prediction Guard, GuardWills, there's a few projects that are doing this. But really then having that actually be typed in TypeScript so you can make an LLM call like it's a function, but get some JSON that has these fields, that has these types and have it kind of, you know, there are techniques that you can do under the hood to self-heal if the JSON isn't properly formed or, you know, maybe you want to generate some types of code and you want to validate that's a correct AST or something.

22:26Like there's techniques that you can do to constrain the output of the language model for your particular task. But in my view, these techniques are constantly shifting, kind of the best practice in the state of the art there. And so I think libraries like Laneshain and the open source framework that I'm currently working on, you know, I think will do a lot to help developers to abstract out some of the complexity of just viewing this as a general purpose tool. And again, it's like you start simple. One of the great things about language models is they can do just about anything. That's also one of the downsides, right?

22:56Like when it's so unconstrained, how do you even approach the problem? So having best practices, having like examples, constraining the problem. And really it's like the ability to have a unit test or an assertion in a traditional programming language at the large language model call level, where it's like, I assert that the output should be valid JSON, or I assert that the output should conform and be valid typescript syntax. And if not, like actually self-reflect on that and put it back into the large language model and regenerate it. All of those things, I think, are foundational primitives at the large language model level that will allow developers who want to build real reliable applications to do so more reliable because they can focus on their domain specific, you know, business logic or aspects that are away from a lot of this kind of implementation details that are also constantly shifting under our feet.

23:46Right. Fantastic explanation. You keep talking over and over again about how things are shifting and the evolution of the engineering around it. that puts a burden on these hackers and developers that are trying to go out and implement these things at this point, because this year has just been, you know, phenomenal progress, but that makes it really hard for mere humans, uh, here to kind of track that and keep up with it. So you kind of talked about some of the concepts right there, but if you're thinking about like, you're about to turn to somebody who's a hacker and they're looking for that guidance, what are some really, you know, not necessarily comprehensive, but hey, go do one, two, three or ABC.

24:28And that will help you kind of keep leveling up. Like what are some of the things that you're telling people these days to say, if you want to keep up this year with what's all with this insane progress and LLMs and, um, and all the different model types that we're seeing the progress of, what are you going to do in a practical side as a hacker to manage that? Do you have any tips that you can kind of take us through about absolutely one it's super noisy there's so much happening we're in the middle of this exponential wave and you know i think a lot of people are like oh i felt well i want to be on this wave right but where do i start and there's just just so much noise which is great on the one hand but on the practical side like how do you give advice where do you start so you know there's a couple levels to this um i have talked uh given some talks to kind of like a chat to be deeper beginners type crowd and really it's like my main advice is one just use it, just go and try it, right?

25:19That's simple. But two, more importantly, like the next time you have an actual problem that you think like maybe I could use a JGVT or a language model for, actually try using it to solve your own problem. Because what that does is it starts to build up this muscle in your brain around thinking about using these new type of tools to solve problems. And it's really a different type of tool. It's like exercise. You need to start exercising that muscle early and often. And there's a lot of noise. There's a lot of different AI tools. the side of things which i'm confident will be just as relevant you know a year from now a couple years from now is building up that muscle to think about using uh how to actually use ai to solve your own particular problems it's one thing to talk about like uh hypotheticals and general like cases of problems where these tools excel but it's another thing entirely to start building your own personal you know muscle totally agree with that i have you know whether it's a personal problem and you want to just go, you know, talk to the chat to PC or you have a problem at work and you're like, well, I think I could use a language model, a hosted API to solve this or something like, starting simple, starting from your own problems will start to build up that muscle and, you know, you'll naturally learn it and take it from there.

26:45So, Travis, you've mentioned a couple of times, like TypeScript, Node, this community that like you're a part of. And I think there's probably a lot of Python people listening to this show, maybe data scientists, practitioners. To me, it almost seems like there's like two communities. There's like all of these data scientists trying to figure out like, oh, large language models, generative AI has sort of broken my intuition around like how. what I need to be doing. Like, do I need to be training models now? Like, how do I solve this problem now? I was training models last year. Do I still need to be doing that?

27:21So there's like that side of things. And then there's like this really vibrant community of front end developers and other developers that are building even like people that are maybe like low code, no code people building really cool products around this technology. You seem to be sort of like exposed to a lot of those things, how do you see these communities developing over time? And like, how have you seen the, maybe the TypeScript, the Node, JavaScript type crowd kind of rise up to meet these technologies, maybe in a way that the sort of traditional data scientist crowd has not or has differently, I guess.

28:03To some degree, a lot of the JavaScript TypeScript world is like jealous of the Python world, right? Because all the cool new AI stuff is like Python first or using this machine learning framework or whatever. And this is where like hosted APIs, whether it's Replicate or Hugging Face, so you got a hat on, right? Hugging Face or OpenAI or Cohere. Like all these hosted models are a massive unlock for application developers. And TypeScript is the largest programming language in the world. Python is, it's big. It's the largest by far in the data science in machine learning world for sure. And so, you know, there's this dynamic between the two where I think we're seeing at the application layer, folks who are good at building full stack apps that can easily plug in to hosted models and things who really push the envelope in terms of like unlocking people's imaginations, building a good UX around these things.

28:55That's so important, like to make it more approachable for people and to really like show people what a lot of the machine learning people have known about for a long time. right but like you need both sides of that equation so one of the projects i did a couple months ago was i ported scikit-learn to typescript and it's not like a full port you know it's like auto generates all of the typescript classes like 260 classes and then under the hood it creates a subprocess a python subprocess and then marshals and does the interprocess communication between them but it works extremely well and you can call and do k-means and pca and just all these fundamental things that the python you know machine learning world takes for granted there There are versions of that that exist in the NPM ecosystem.

29:35It's just they're all over the place in terms of quality, in terms of like there's so many just fundamental aspects of machine learning that the TypeScript world is missing out on. And one of the primary drivers behind kind of what I'm working on, and I'm happy to share, I'm building a reliable TypeScript open source framework for building reliable agents. Very cool. Thank you. I view agents as this new, like if large language models are CPUs, right. And kind of this new compute paradigm, there are these reasoning engines. Like, yeah, they're great at generating text, but the real emergent property, the real game changing property of them is reasoning.

30:11If they're kind of the new reasoning engines and you have like that, that's your CPU layer. And then you have like a storage layer. That's all these vector databases and kind of overhyped on that side of things. On top of that, you have like, how do you actually run programs? And that's, you know, agents. And I view it as like, there's kind of a spectrum of like traditional programming that might happen to use a large language model. And then on the other end, you have like full self-driving agents that are making decisions and creating tasks and just fully autonomous. Right. And I'm excited to kind of focus on somewhere in the middle and focus on more reliable like use cases that we can actually build reliably today.