AI Engineering Pitfalls with Chip Huyen - #715

0:00If you have a question, finding the answer, if the answer exists, is a lot easier analysis with AI. But I do think what is hard is like coming up with the right questions. So that made me realize it's like, okay, as a writer, as a technical communicator, that I need to be able to like think through things and come up with like, ask the right questions.

0:33All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host, Sam Sherrington, and today I'm joined by Chip Nguyen. Chip is an independent AI researcher. Before we get going, be sure to hit that subscribe button wherever you're listening to today's show. Chip, welcome to the podcast. Hello, Sam. Hi, everyone. Thank you so much for having me here. Oh, by the way, I'm like an independent researcher in general. AI is just like one of the things I'm interested in. Awesome. Awesome. Well, it is great to connect. It's been a little while. I think a couple of years since we last spoke.

1:10And obviously, a lot has happened in the world of AI since then. Maybe you can catch us up on what you've been up to. I think you were just starting Claypot at the time when we spoke. You were getting into real time and some other things. What's been new for you for the past couple of years? Yeah. Yeah. So we started Claypot AI and then we sold the company last year and then stayed with the acquirer for a while. It's actually like for things to settle. And this year, and after that's why I'm being unemployed for this year and I'm pretty really unhappily sorry pretty happily unemployed not like i guess it's better than like unhappily employed uh but it's been pretty good um yeah and i yes i think this is here i feel like it's been the last few years have been pretty intense especially uh with like running a startup and then with all this craziness about AI.

2:09So this year I just decided, let's just try to pursue my interests and see why they pick me and try not to have a lot of deadlines and expectations and goals. Wow, I do sound a bit lazy, don't I? Not at all. And I don't think anyone that follows you would call you lazy. You just had a book come out from that book. You just posted a really interesting post about agents and we'll be digging into that topic. Maybe we can start, though, with independent researcher. Tell me, do you structure that in any way? You were specific to call out that it's not AI. What are some of the other things that you're researching?

2:52I think AI as a tool, and it's definitely a very fascinating tool. It's really hard to avoid AI nowadays. It's just like I have this test, like if I go to a party, how long until somebody brings up AI? This is like in the first five minutes. Like it's really impossible to escape it. So definitely AI has to be a part of it. But I'm interested in like problem solving. So there are several problems I'm interested in. And I just want to see like how to solve these problems. And if AI is part of the solution, it's great. But if not, I'm very happy. I would like to be very happy to look into other directions too.

3:30What are some of those key problems that you're interested in? um so so one thing is just like i think that's the way people read and write and learn is going to be so different so when i was writing the book uh my book i read a lot of um papers like a lot like i think my uh my book has about reference alone like about like over a thousand links and of them is like 400 somethings are like archive papers so that as only the papers i referenced in the book and I read a lot more that were not referenced so and it's a process of reading these papers I realized as a way I read has changed so for example in the past right I would like read uh so abstract and read the I read the I read the conclusions discussions and then I skimmed through the paper but now it's just like I felt myself like with AI I become a lot like the time it takes for me to read a paper like it's a lot less like I could read the abstract still but then most of it I would throw into ChatGPT or Cloud, and then I start asking questions.

4:32So one thing I usually ask is, especially for me, I understand a new technology better in comparison to what is the old technologies. So if there's a new technique coming out, I would like, okay, but how is it different? How is this improvement? And why does it work compared to the last techniques? So I would put two papers or three papers into ChatGPT and ask to explain to me how is that an improvement? like why from the previous ones and i was doing it i was just like okay if i don't read things like end-to-end anymore then why do i write things from end-to-end you know like why do i write the whole thing people like that so it does it does make me think about like what kind of writing is harder to automate and then i realized something is that like first um if you have of questions finding the answer if the answer exists it's a lot easier nowadays with ai like yeah first of all if you ask it to explain like what what is x you ask chachibidi it can come up with like a thousand of different definitions like you can explain to you as if you were five you can explain to you as if you were like i don't know a dancer or like i don't know it's really good at coming up with like explanations or definition for you but i do think what is hard is like coming up with the right questions.

5:48So that made me realize it's like, okay, as a writer, as a technical communicator, that I need to be able to like think through things and come up with like, ask the right questions. So I'm less pressured in like asking, like trying to like define things or explain things. I mean, like I still want to do a good job at it, but I mean, I'm a lot more concerned with like thinking through things, like structure things, asking the right questions. So that is one of the interests you asked me. But yeah, I do. I'm interested in learning, education, maybe information consumption in general. That makes me think a little bit about this joke that we've heard or seen come up a lot recently, which is people spending a lot of time using ChatGPT to help them take their little blurb and expand it into something formal.

6:44and then people on the other side are taking that formal thing and expanding it back into little blurbs or bullets. How do you see that evolving communication? I do think that it will change a lot now in the way we communicate or the way we interact with each other. So the way of communications... So one example is just like, you know, search engine speak. It's just like people have started learning how to type. gets gets a like ask just engineer google so so he gets the right thing and perhaps start learning to like talk like that with like chat gpd one is it's very interesting so i have i have a friend who who has like a quarantine a covid baby right so the so the kids like spend the first like two years being very isolated because of parents uh a nanny and alexa and she when she was free like i noticed that she talked a little bit like with people the way she could talk with alexa she'll be like mom get me this dad get me this right it's just like so so i felt like we can see that so so i have no doubt that's like we interact with mobile ai wouldn't change the way like in san francisco have you seen that people like have already been studying uh self-driving cars um it's because like you you you know you know that like self-driving would always stop for you to cross the street so when you see several cars it was like okay i'm just gonna cross now i don't i don't have to wait for the for the for the crosswalk right or like sometimes i was just like let's annoy i think i'm not saying i did it i'm not saying some of my friends did it i just like to see certain regards like let's let's annoy the center in god like given that there's no one inside but i feel like um i think with definitely change uh one thing i'm actually really curious about is that um um i've seen like a few companies that uh provide um ai companionship and I think I was very so I have I was able to see some of that data and it's a little bit crazy like how much time people spend on like chatting with AI like it's really insane if you go first of all I think you can pre-see some of it if you go to like some subreddit of like character AI or like some others and it's just like people spend a lot of time talking with AI and it just made me feel like let's imagine like you are younger today right you were like and and then i maybe like not maybe you know in the teens or like twins like pretty young and you you spend a lot of time talking with this ai and it was so used to like ai like just accommodating you like just saying exactly what you want to hear like always be there for you then how was that experience compared to talking to humans like why would i bother to talk to humans who i cannot program who i cannot like just say things and don't want to hear.

9:35So it just made me feel like, hmm, how is this going to be in the future? But anyway, I'm not sure. I don't remember what our question was. I just went to Iraq. Yeah, no, you know, we were talking about the question was how you think our interaction with AI and more specifically, I was asking about our use of AI tools would change communication, and kind of reflecting on that loop that you used in reviewing the papers, how that has shifted and how your writing has shifted.

10:13Did you, to what degree beyond using the chat GPT and Claw to summarize papers, what other AI tools did you use in the writing of the book? Oof, I use a lot of it. Not a lot of tools. As I say, I use some tools a lot of times. So I actually, like, did, I actually use Jachipiti and Cloud. So, and also use, like, something like, okay, I don't watch, I feel like it's weird mentioning those names because it sounds like promotions. I do use a few other tools, just, like, heavy editing, like, catching, like, you know, just, like, general writing assistant. And I just, like, recently, like, just exported Jachipiti.

10:55and like during the so i found out like during the period of writing the book i i had about like 3 000 conversations with judge pd so a lot of them is not just like summarize paper comparing paper but also like uh do the analysis um so yeah so i just trying to understand or like help me like create a graph or like just brainstorm like ways of like how to explain the concept some validating ideas um i use cloth a lot because one of the interests i have right now is write a math novel which i really want to do but i felt like there's pretty no market for it uh but but i just like like math i think math is very beautiful and people keep asking me like hey is there like any like so so i did competitive math for a while when i was a child in 10 years and I think it's like people keep asking me hey is competitive math useful I do study like number theory and stuff I do use it nowadays and I feel like a lot of it probably not but I do think it's like mathematical thinking it's like really really important and it's just once you write like a novel to like just like how beautiful math is and I did you like cloud and chat GPT just like to like help me like think through the character development is that realistic like is that probable and like coming up with like examples because sometimes you know it's like it's pretty they're both pretty good at coming up with examples and in the process I found out it's like it's pretty useful for like writing a book on AI as well because I learned a lot about like what they are good at and what they are not good at and also like just in general just helping understanding have giving more ideas so it gave me a bunch of ideas about different benchmarking and design like how to evaluate different capabilities of these different models because for one example right like creep writing um i was talking to someone at like um anthropic and that person told me it's like oh claude is really good at creep writing and i'm just like how do you know do you have like internal benchmark for it and this person was like no people on twitter told me so and i'm like hmm even if people were really good at it really smart right and apparently like this kind of thing is like quite important to them i still don't have benchmarks to measure things like that then how like it must be like really hard and that's made me think about like how we can design like benchmarks to evaluate things like script writing storytelling um yeah so i have been like thinking about like a different uh approaches to like benchmark design so that's one of the things i want to pursue in my like leisure time or like unemployed period.

13:38Nice, nice. Well, let's jump into the book a little bit. The title of the book is AI Engineering. The book is mostly about generative AI as opposed to, you know, other traditional machine learning, for example. Maybe start by why do you call it AI engineering as opposed to something around generative AI or, you know, any number of things that you cover in the book, what does AI engineering mean to you? Yeah, no, that's a good question. And I did agonize about it a lot, about the name, quite a bit. So I think it's like, first it's like, should we stimulate machine engineering? Because machine engineering has been around for a while.

14:23And I do think that's why there's a lot of overlap. And a lot of the foundational things that we know from AI engineering can be applied. AI, like, foundation models do come with a lot of new things that I do think is, like, we need to talk about those new things. So we do need something to separate AI engineering, like this new thing from, like, traditional ML engineering. And there were a lot of terms flying around. Like, some of them were talking, like, LM ops, some of them were like AI ops, right? And I was like, I didn't really want to use the word that end with ops because I do think it's, like, a lot, there's just a lot less ops ratio.

15:00like operations about this right it's a lot more if everything's behind the service yeah yeah yeah um and i think there's a lot more creativity like uh more problems it's like it's a lot more i'm not saying operations not fun i'm just saying this can be a lot more fun um so so i did ask a bunch of people so at that time i i knew quite a few people who are already building like really cool applications with jet of ai and i asked them like so so how would you call what you're doing and they were like engineering so after you got like 20 people saying that i'm saying okay i'm just gonna go with what people call it you know i'm not gonna pick my own you know so interesting interesting interesting um and one of the topics that you cover in the book is agents you know before we dig into some of the way you're thinking about agents nowadays i'm curious about your meta thinking about agents and that is um you know i see a lot of debate raging on the the social medias about whether agents are an are a thing or whether agents are just kind of rebranded llms like llms plus tools is you know now we're calling that agents um how you know assuming you kind of have thought about that a bit how do you think about that um and do you think kind of referring back to your comment earlier about asking the right question do you think that's an interesting question or is it you know just kind of throw away social media fodder and there are more interesting questions to ask about this and what are those questions i think it's it's an interesting question and i really want to go into this but can we just i just want to end the thought on like the engineering versus ML engineering and then we go into because I feel like we're going to be spending a lot of time talking about it.

16:54So yeah, this is on the note of like engineering and ML engineering. So I don't think they are like mutually exclusive. So I do think it's like in the vast majority of Jive AI system I have seen, it's just not like a Jive AI or traditional ML, a classical model. It's like it's both. So for example, you build like a customer support chatbot, right? Like so you have a request from like a customer before sending the request to like the ai model um you might want to have like a intent classification to see what the request is about maybe there's something like easy you can answer with an faq yes or router so that you can build with like a classical machining model or like as a scholar right after you get a response back from a from ai you may want to say hey is that safe does it contain pii right does it reveal anything you and say, or like whether it's good enough or not.

17:48So you can also build those scores with classical ML. So I do think it's like, it's very easy. They do go hand in hand. So in a lot of organizations, I see that they have the same team for engineering and engineering. And in some organizations, they separate them out. But I do think it's like for anyone who wants to build like solutions to real world problems, maybe like complex problems. it's very likely that that's a person we need both like traditional ML engineering engineering and I think like the terminologies I don't think we do I don't yeah I don't think we need to hang get hung up on the terminologies because even the same role title or different companies can mean very different thing even the same company has the same title can I do very different things yeah so I think just to close out that thought and okay next we talk about agents right And in contrast, while we don't need to get caught up in terminology about ML engineering versus AI engineering, it sounds like you do think the terminology around agents is interesting.

18:57So, do you know if I had this feeling, it's like you like something, but just hate the way it's called?

19:05yeah so so like i just traveled to vietnam recently but uh with my friend and he was like i realized this dish like it's like very tasty like over the charcoal vietnamese snack but then people try to sell the tourists and call it like vietnamese pizza i was like he was like why would you call it something so tasty vietnamese pizza like totally not that so um so so anyway so it's about the naming uh but um but yeah so um i think a lot of people don't like the word agent and a lot of people is like oh like you said it's like what is agent is a new thing or just like lm plus tools um for me um i try not so i try to avoid debate on on naming because i feel like So one thing I really hate about being a writer, I love writing, but one thing I really hate to write is definitions.

20:01Because I feel like no matter how I define something, someone's going to get upset. Because it will never be good enough. So I did have a section on agents in my book and it went with agents because that's what a lot of people call it. So I know a lot of people don't like it, but it's only something that people call it. And agent by itself is not a new term. And I have a lot of friends who just really hate the term. And there are already so many things with the name agent. Like I have reinforcement learning agents. You have user agents. You have a lot of different kind of agents, right? What is this agent?

20:39So I did go through a lot of reading AI books, some classic books on AI and see how they define agent. And I think definitions I quite like is that agent is anything that can interact, that can perceive the environment and interact with the environment. So that means... Yes. Yeah. So it's a great book, by the way. Which is written mid-90s. Yeah. To underscore the idea that this is not a new concept. Yeah. No, no, definitely. So an agent itself, right? So it has to perceive the environment. So it means that agent has to operate in an environment. So first of all, a coding agent, it operates inside, maybe coding maybe like in VS code or like in terminal, right?

21:28Or like if it's Charged GPT is an agent that operates maybe on the internet, right? It can like browse the internet. It can also like answer a question that you interact with it through the internet. So, and then to like interact with the environment, you will need like be able to perform a set of actions. So a lot of things I've interacted with nowadays are only agents. Like Charged GPT is certainly an agent, right? The action they can do is like execute code, browsing web, user calculator, like even generate image or retrieve image. So there's a lot of things JARP can do already in similar as Cloud.

22:03So agent is already here. But the capability of an agent, a lot of it is not what people hope for yet. Maybe people think it's like, oh, agent is like, it sounds cool and all. like you want to have something autonomous, I can have you do a lot of things. But maybe the promise is like the actual capability is not quite where people want it to be yet. But the question is, it's like how far or how big is the gap? And I'm pretty bullish on the improving capability. So I do think that the capability of an AI power agent depends on shooting. One is a set of tools you're going to get access to. And the second is ability to leverage those tools to solve problems.

22:50And the second aspect, we can go in planning, right? Like given a task from a user, how does the patient think? How does the agent think about using those tools to solve the task? May come up with a sequence of actions, like a roadmap to get the goal and maybe determine whether the goal has been accomplished, right? So it's a tool. So I think a lot of agents, like what we use today, LM, we think of them not as agents because they have a limited set of tools. But they have tools, right? But I do think the more tools we give AI models, the more powerful they're going to become. However, using tools well requires a lot of capability.

23:37It's a lot easier to use two or three tools well. it's a lot harder to use 3 ,000 tools well. So I have some particular benchmark, and I want to see how well a model can use a different number of tools. And in my personal, not peer review, I just want to put it out there, it's a personal benchmark. I feel like AI model fails very, very quickly with the number of tools increases. So the planning capability of AI has to improve. people to use their tools well and I solve the task well. Can I ask you a question about this? Your non-peer-reviewed research study on tool use? Have you looked at, like it strikes me that an interesting factor would be like the semantic distance between the tools.

24:32Like if there are a bunch of tools that are closely related to another Or, you know, inches to millimeters, millimeters to inches, inches to feet, that kind of thing. And then like a recipe tool, like maybe there's more confusion around the things that are closely related and, you know, less confusion around things that are different. Or maybe it's the opposite. I don't know. I'm wondering if you have any thoughts on that based on what you've played around with so far. I think it's a very good research question. so in my benchmark a lot of tools are like not quite similar there's a range of different tools but I do think this is a school like a very subset of papers I've read so it's like this go into like what kind of not as a kind of tool like there's a higher abstraction a higher level of abstraction than that so let's say it's like okay let's say like you write code, right?

25:34Write code and it's an abstraction. And then lower, you're going to write Python code, write C code, or writing codes, right? So when you ask AI to generate a plan, you can either ask to generate a very high-level plan, a very granular plan, right? It's just like planning quarterly planning, you can do monthly planning or weekly planning. So I think it's a different study on how good the generated plans can be like the different level of like randomity and also there's a question of like um you can ask some model to generate plan using the function name directly or should you generate something like in natural language and then have a translator from this natural language into the into the into the function names so so i do think there's like there's a lot of studies around that and i felt like a lot of assignments around agent is great uh but i still like feel like a lot of the excitement nowadays is about like hey how to build an agent very quickly right like hey this is framework to build agents in like five minutes and i'm always like okay build an agent and then what you're like but what do we use that agent for right so so i feel like i think like maybe we can have some of that energy directed toward like understanding like how do we properly structure these agents or like use that you know like yeah understanding how it uses tool like how to, so planning language or like just how to get the agent to do things like more correctly.

27:04It's going back to this, the initial question around agents versus LLMs and tools, or, you know, does agents equal LLMs and tools? I think that, you know, it is true that a lot of that is kind of semantics and not all that interesting, but an interesting potential aspect of that is if, you know, you want to explain to someone who knows LLMs and knows tools, you know, what opportunities there are in thinking about some broader universe of agents. And if that requires, you know, new skills or new things to learn or, you know, new approaches. How do you think about it from that perspective? Meaning, you know, if you've got someone who, you know, knows LLMs and tools pretty well, and it's coming to you and says, you know, is there anything that I need to learn beyond that to work with agents or to take advantage of, you know, agentic systems?

28:10you know what are the things that you talk about so i think the question of like whether agents is just m plus tool is the same as the questions whether a car is just a battery plus wheels you know like yeah you can have like on this component but is that enough for you to like want to buy it so there are a lot of other things about it first one has to like first like package into a product that people want to buy or like have it like be with your own problem like efficiently. So first we have an LMP can execute a lot of tools. And imagine you just keep on calling execute, like calling maybe open a million web pages and send out like 10 ,000 API columns.

28:54It can do that very well. But then what problems will that LMP plus tool can solve for you? How easy is that for the users to be able to specify a task and then get back the result? How can that be integrated seamlessly into the workflow? and then how do you as an Asian developer get all of this feedback from users and continue to improve the system? So I do think that is a big question. So definitely, like I say, LM is the brand. So I do all the planning and reflections and the tools. It's just how it can execute and perform actions to accomplish a task. But there's a whole ecosystem around this.

29:36You have to do system design. You have to think about and maybe supplement it with memory, think about how to evaluate the agent, different failure modes, like put guardrails around. Because the more tools the agent has access to, the more dangerous in a way it can be. Anyone says it's dangerous because it can scare a lot of people. But this thing is, we just need a higher level of security. If you have an LAM, just can I say read email versus an LAM that can send email. or if we have an LIM that can like beat you, like tell you the bank balance versus an LIM that can like transfer money out of your account.

30:18So like this is more action you can give the LIMs or more guardrails. And the more we can think through like how to catch the failure, make sure that things don't go wrong. And I do think that's a lot of work around that. And it's way more than just like LIM plus tools. When you see folks approaching, you know, getting started with agents or even kind of this broader picture of AI engineering, like what are the things that get people caught up? What are the mistakes that you see people making kind of over and over again? Yeah, that's an interesting question. I feel I have a lot of thoughts about it.

30:58They have to be careful not to make it sound like I think everyone's an idiot. Yeah.

31:06So I do think that engineering as a field, engineering field, is still pretty new. And with anything new, we're still learning how to do things right. We're still coming up with understand best practices and be around the right level of traction. So I clearly think it's a lot of like making mistakes is inevitable, right? And that's how we learn. So there's certain mistakes or common pitfalls, I see. so first of all one thing is that a lot of time people start way too complex too early right like first of all you see some new framework coming out let's try it out let's build an agent right let's multi-agent yeah and it's very easy to get caught up in this like the layers instead of like thinking about what you actually want to do so first of all before like using a framework that's like how abstracts away on the API columns, like provide with like some pre-built prompts.

32:07Maybe just try out, like call the model directly, write their own prompts, you know, and just try to see how things go. And because like, actually during the process, during research for my book, I actually went into the code bases, a lot of like popular frameworks, and I saw a bunch of them built in prompts, and a lot of them have typos. because we need to understand that framework developers can make mistakes as well. And a lot of times using the right tools can help, but using the tool too early can introduce unnecessary and necessary

32:49complexities and hideaway mistakes. So I think it's good to just understand how things work underneath the hood really, really well before introducing different levels of like tools to make it easier um is that might change the future though okay i do think like over time if we have like much trust in frameworks or like we have best practices and then we can trust us like these frameworks and are reliable and do what expected then i think it's like it's okay to start more more complex because like nobody nowadays actually they write assembly code anymore right we we do go like absolute level attraction abstraction over time Another example is a lot of people when they do Rack, Retrieval Augmented Generations, people start with vector search, using very complex databases and embedded embeddings and stuff.

33:37Whereas you can do something a lot similar, just like Turnbay, Retrieval, or something very old school system like BM25. It's actually really, really hard to beat. another thing is just like that I get a lot of like debate about is like how much AI knowledge do you need to start building applications so in a way like with engineering there is a lot less ML like before if you wanted to build an AI application you would need to build some models yourself but nowadays you just access this build model right and it's an incredible thing because now like anyone Like, it's, like, a really low entry barrier for people to, like, access AI and use AI for their needs.

34:23And I do think that you can do a lot, like, starting out with it. But I do think it's, like, to use it well or, like, to, like, avoid headaches. Then some understanding of how AI works and neighborhood can help you go a long way. So, like, first, like, it can help you explain some, like, weird behaviors of AI. For example, understanding how AI sample their responses can help you maybe understand more about why hallucinations happen or why the AI changes its responses. Or it can help you come up with strategy to improve the model responses by changing different ways of the model sample responses.

35:07Or maybe you can learn from the past. so for example like hallucination again there's like a sub-fill of NLBs that can study something like why the models output things that are not consistent with what's given to it for example like the question like hey I wanted to I actually like summarize this book why the summary not consistent with the book but it comes up with things that's not in the book right so that's filled with natural language inference like textual entailment. So in textual entailment, you have a hypothesis, right? And they have a statement. And you want to determine whether this statement is consistent or can be derived from the hypothesis.

35:54First of all, it has a hypothesis of like, all fruits are tasty. And if you say like, apple is tasty, then you can say, okay, this statement is consistent with this, because apple is a fruit. But this saying is like married like apple, then it's not. You can't really derive. It's mutual. It's not related to this. So it's very useful in the context of LOM. So, for example, you give it some documents about your company policies, and the answer is like, can it be derived from this document? Is this consistent, or is this contradictory, or is this something you cannot sufficiently derive? from it.