BONUS : How LangChain’s Abstractions Help—and Hinder—AI Innovation

0:00Building those agentic tools is not the hard part. The hard part is the prompt engineering and a prompt tuning. So for LangChain to come in and build all of these abstractions on a relatively simple set of problems actually caused more of a hassle for individuals who are looking to take their EOCs into production. Welcome to the Talking AI Podcast, where we talk AI with both experts in the field and early adopters. I'm your host, Matt Page, and we're here to demystify AI for you so you can get some value from it. Let's talk some AI.

0:33Today, I'm joined by our new CTO, Omar Shanti, and we're going to try something new. So AI is moving so insanely fast, it's extremely difficult to decipher from what is hype, what is real. So to help with that, we're going to have Omar on as a reoccurring guest to do these short kind of micro episodes. We're going to talk about specific trends, news, tools, and AI to really just help you cut through the hype as told from the perspective of a CTO in AI. Think of this like Sesame Street, I'll be Big Bird. Omar, you're going to be the count because you're super smart and you're like the mathematician here in the room.

1:10There you go. Got the counting going for those watching on YouTube. And I'll play the role of asking any and all dumb questions so you don't have to. That way, we can all make sense of everything happening in AI together. But Omar, welcome to the first of many Talking AI episodes. Thank you, Matt. It's great to be here. I love the level of conversation that we've been having, and hopefully this mini-series will be a good intervention on par with the rest of it. Yeah, this is going to be exciting. We've been chatting a lot offline, obviously, a lot of cool topics that are going to be coming up in future episodes.

1:42But for today, and our goal is to try to keep these in under 15 minutes. I'm not super confident, Omar, we're going to do it on the first one, but we're going to try. So these are nice kind of bite-sized things you can consume quickly. But for today's topic, we're going to talk about abstractions. Are they a good thing? Are they a bad thing? Or does it depend really? And first off, Omar, give us some context. What are abstractions for some of our non-techie audience so they understand and how do they add value? Absolutely. The root of the word abstraction comes from lacking specificity. But that's not quite what's going on here.

2:21It's less about lacking specificity as much as adopting a certain way of thinking about things that focuses on how things connect, the interoperability. So sometimes it's good to remove some detail when you're analyzing a lot of information, right? Not everyone needs to know, for example, the inner workings of a machine in order to know how to interact with that machine. And that process of removing some of this inner detail so that you can more completely understand how to work with the system, that's called abstraction. In programming, it's got a slightly more specific meaning. It's one of sort of the pillars of object-oriented programming, and it carries weight as a central foundational pillar in the dominant paradigm that many people program in.

3:08and historically we've thought of abstraction as a an inherent virtue and i think what we end up seeing is abstraction is an inherent virtue if done right and to know just the right amount of abstraction you need to do it helps to understand what that interface looks like how people are going to be interacting with this black box if abstraction is designed to simplify that interaction, we better know quite well what that interaction entails. And oftentimes in emerging tech spaces, people lead with abstractions without really understanding the usage patterns. And in a way that's taking the user off of center stage and putting the technology on center stage.

3:50And that's really not how you build human-centered products. So at the core, it's making things easier for folks. That's the merit and kind of the value behind it. Give me some examples of current abstractions versus back in the day where you're doing everything, say, in the terminal and coding way back in the day. Give some examples of abstractions that you think have added a ton of value that you want to use on a regular basis just to hammer that home in terms of what it is. Absolutely. I'll start off talking about the the programmatic field, but then we'll talk about life in general because our life is filled with abstractions.

4:30So in the programmatic field, one common form of abstraction is a library or a framework. When you do things repeatedly, you never really want to repeat yourself as a programmer because that just introduces multiple sources of truth and makes it a maintenance nightmare and so on. So one of kind of the fundamental principles is do not repeat yourself dry. So people talk about coding with dry practices. Quick break in the pod. If you're listening to this podcast, chances are you've been thinking about how to actually use AI inside your business. And that's exactly why we built the AI Opportunity Finder.

5:05It's a free tool that helps you uncover high impact, tailored AI use cases based on your business, your goals, your pain points, and your industry. No fluff, no generic use cases, just real ideas that fit your business and the rank by ROI potential. It takes about three minutes to run and it's like having your own personal AI strategist for free. If you want to try it for free, check out the link in the show notes or go to hatchworks.com backslash AI dash opportunity dash finder. So when you're coding with these dry practices or dry principles, a lot of the times you're going to take functionality that is reusable, package it up, and then reuse that package of software.

5:45And to interact with that package of software, you're going to have what's called a contract, which is here's what it takes in and here's what it takes out. The inner details, how that function works, is totally abstracted away from you. You just need to call that function. So at the fundamental level, any function is an abstraction. Now, functions get packaged into libraries. Libraries sometimes get packaged into software as a service offerings. And all of these are abstractions built on top of abstractions. The good ones are the ones that really tap into and index a specific usage pattern. The bad ones are the ones that force you to adopt specific usage patterns that are alien to you.

6:23To bring this to the real world, one common abstraction that you might see is... Actually, I'm sorry, Matt. I'll get back to that thought. Go ahead. Okay, yeah. But no, abstractions on abstractions. So these things can stack on each other. And I think too, here's the meaty part here. with everything going in in AI right now, there's several companies moving to create these abstraction layers and tools to make it easier to work with AI. And one super popular one that's been out there has been LangChain. We've used it in some of the solutions we're building. They're doing a lot of great stuff there.

7:03But you have some folks finding some issues with it, Octomind, an end-to-end AI testing tool, came out and wrote a piece saying why we no longer use LangeChain for building AI agents. It's like powerful words for all the folks that are the LangeChain train. But I think like from the CTO perspective, what's behind that? LangeChain is ultimately, as you described, an abstraction tool. It's abstracting away some of the things you need to do to make these AI agents and orchestration work. Where is it falling down and why do you think that is? It's interesting. The title is some pretty choice words, but the hacker news and the Reddit threads have got some equally choice words.

7:58what we saw over here is an example where octomind started a conversation that all of a sudden you saw dozens of people from different companies from different walks in life contributing and so octomind in a way it was almost this the emperor's got no clothes on moment yeah and lang chain was revealed for abstractions that sort of didn't serve us so let's talk about this With Langchain, ultimately, the promise was for a cross-functional, cross-model library that makes it very easy to build LLM or Gen.AI-powered applications. And specifically, Langchain's implemented this specific paradigm called the React framework to allow you to build agents, which we now take for granted agents, but at a time where this new thing, oh wow an LLM that can take action an LLM that can use a calculator.

8:58So Langchain popularized this and it did so by relying heavily on abstractions and packaging up code in certain opinionated ways that served developers at the start but ceased serving developers in a lot of use cases. People quickly found that Langchain would help get them out of the gate quickly. So it will get them 80 % of the way. But that final 20 % is almost insurmountable. And so let me be very clear what I mean by that. Building an agentive application requires things like orchestration, data indexing and processing if you want to do RAG coupled with vector storage and retrieval, usage of tools, and so on and so forth.

9:40What people are calling out these days is that the promise of Langchain of making all of that workflow easier is actually not that strong of a value proposition. Building those agentic tools is not the hard part. The hard part is the prompt engineering and the prompt tuning. For Langchain to come in and build all of these abstractions on a relatively simple set of problems actually caused more of a hassle for individuals who are looking to take their EOCs into production. So as an example of Langchain making the hard things easy, but then also making the easy things hard. And that led to a lot of people reassessing their toolkit when going to production.

10:21You're almost throwing the baby out with the bathwater in some cases. It was one of the, I think, Y Combinator or Reddit threads you shared. This is actually an intern kind of building a rag-based solution. And they mentioned they were hitting just a standard use case, but they had to go through five layers of abstraction just to change a minute detail. And I think that's where abstractions can sometimes go wrong is when you don't have that clear understanding of what's happening below those different layers. That's where, to your point, it can do a lot of great things, but if it's also putting up these hindrances along the way, what's the net of it?

11:01And curious your take, does Langchain still have its valuable use cases and whatnot within the broader AI space. And I think it's all new and emergent too, is the other piece of it. It certainly does. And we really need to hold the developers that line chain with the grace. At the time, they were trying to position themselves as best as they could. And positioning themselves as best as they could meant adopting abstractions in a way that allows them to scale up with the number of models that people might want to connect to, as well as the number of usage patterns that people might want to connect to too.

11:36And they were playing in such an emerging space that picking out those abstractions were fairly difficult. From the software developers, sort of software engineer or software architect's perspective, you've got these set of principles called the solid principles. And the idea is that a system should only depend on systems that are more stable than it. So if I want a system to be stable and I am building a dependency against something that's really volatile, then I can only expect to be as stable as the volatile system that I'm depending on. It operates similarly with abstractions, where abstractions depend on usage patterns.

12:16And if you are building abstractions without those usage patterns having been fossilized and formalized, you might end up building the wrong ones. And what we've seen is Langchain was that initial spark, followed by a couple of big players, such as Lama Index. But then that led to this proliferation of tools, almost like this fan-out effect, where now you have tools for all sorts of workflows within LLMs. So I think in the whole, Langchain was really valuable for showing us the art of what can be done. I think their abstractions in their initial release were a little bit not so much grounded in what the user and usage patterns were involved as much as forecasting what the space could be like.

12:57And some of those predictions went a bit awry. But the LangChain team has actively been working on LangGraph and other solutions around observability within that LangChain suite. That definitely is a redeeming factor and that make people do want to adopt LangChain. And so all in all, it's got room to play. It was a sort of forerunner of the space. And even if Langchain as a tool is getting deprecated and other tools such as Langgraph and the variety of other libraries are coming into the fore, it has definitely done more good than harm in its lifecycle. You mentioned the observability. Again, I'll ask any and all dumb questions.

13:33Go into what you mean by that. Is it in terms of having observability of how the solution is working, the abstractions in the different layers so it's easier to understand? define that for us? Yeah, observability is, as it sounds, observability is the process of gathering observations, the process of being able to see what exactly is happening across a process. And when working with agentive processes in general, a lot of the thinking is done by the LLM. The LLM ought to be able to expose sort of its thought process, right, its reasoning steps and so on. Additionally, the prompts that are sent to the LLM to generate those, and then the context that is embedded into those.

14:14All of this is basically a trail of actions and thoughts that must be rendered observable and legible. And that is tough to do at scale. Now, imagine a workflow where you're hitting one LLM to generate a text prompt to then pass into another text-to-image LLM to generate an image, which you're then passing into a third base model to then perform some action, such as convert that image into ASCII characters or something like that. It could be very difficult to understand in even a three-step workflow, what the interface looks like. And so it helps to have an observability tool at that orchestration layer, tell you exactly what each LLM is doing at each step of the way and what that inner weaving looks like.

15:02That helps you pinpoint where you ought to intervene, how you can optimize, how you can achieve better performance. Yeah. And when I knew we were going to talk on this topic, I went down a chat GPT rabbit hole, just looking at when abstractions go wrong. I think the key one we've been hitting on today is premature abstractions. This space is very new and novel and things are hitting a little bit early. I think over abstraction, we hit on that one. Leaky abstraction, exposing high complexity things throughout. And it just sometimes makes it difficult to work with. I think the last piece I'm curious to hit on, you mentioned orchestration a couple of times and like actually invoking an LLM and using an LLM isn't necessarily the difficult part in code.

15:49But when you're starting to orchestrate these agents together, how they work, the prompts that they're following, all these different things, that's where some of this complexity starts to come into play. But curious from your perspective, having built these solutions, using these solutions, what's your take on orchestration today, maybe where it's going in terms of the AI landscape? Orchestration can sometimes just be thought of as, I think orchestration has historically been slept on a little bit. It's increasingly coming to the forefront now. But ultimately, as a concept, it's fairly limited to the developer's repertoire.

16:30are, like only the devs are talking about orchestration. It's something that a lot of people take for granted. I ought to do this, then I ought to do this. If this thing fails, then I ought to do this other thing. When building sort of production grade systems, the way that you orchestrate has to guarantee things such as fault tolerance, efficiency. And so by fault tolerance, if there is an error, are you able to salvage it? Or are you able to take an appropriate action to report that error? It ought to guarantee things ranging from cost and scalability down to like accuracy and speed. So orchestration as a layer is, or as a sort of a set of tools or a set of processes rather, is definitely something that's crucial.

17:12And when building LLMs, it's no different. However, I think the question is, exactly what amount of orchestration do you need? And that'll depend on your use case. If your use case is fairly straightforward and fairly linear without much parallel processing, without much looping or recursion, then you might be able to just adopt a sort of standard orchestration of a object-oriented sort of imperative programming language. In other words, you don't necessarily need an orchestration tool. If all I need to do is hit an OpenAI endpoint and then take the response and then pass that into another endpoint, I can get away with a try-catch, a for loop maybe, and just some good logging.

17:55So I can do that in Python. However, if I wanted to scale that across a network of multiple machines, that's when having a really solid orchestration tool comes in. And this, I think, is a nice bridging point into machine learning and big data processing in general, where orchestration is like a first-class citizen. When you train a model or when you use that model to make batch predictions, a lot of the times you need to orchestrate that work. So you're building a direct, a cyclical graph. Sorry, a directed, a cyclical graph, a DAG, in other words. And that DAG is responsible for making sure that everything follows step by step in the sort of the targeted workflow and that any failures are captured and get redone.

18:37So the short of it is your orchestration ought to scale with the complexity of your task. Yeah. Yeah. It's funny how those first principles just always still apply. There's like this art and science to it all that still happens here. So I think we've hit right around the 15-minute mark, maybe a little bit over. We did better than I thought we would. Omar, any parting words and Langchainer? Is it still going to be around? Do you still see the use and the value in it in the future? Really, I think it's center your people and center your processes and the technology will follow. If you understand that your product is going to be highly complex, then you might consider using a tool that scales up with complexity.

19:22If you know that you're going to have a pilot tool and you're going to use it internally and try to do it more for learning, then Langtrain might be a fantastic tool. So all it is really is the requirements vary from context to context. Be very clear on what your functional and what your non-functional requirements are. Be very clear on what you anticipate the product to evolve into. And obviously, you need to exercise a very healthy grain of salt there. And then also keep in mind your people and what their skill sets are. And with those three things taken together, you can figure out the non-functional requirements that will then guide you to the correct tool.

20:02Great. And that, talking AI listeners, is the nuanced kind of perspective you'll get from a CTO that's been building these solutions along the way. Omar, thanks for joining us. Excited to do more of these. Let us know what you think about them. Let us know what topics you want to hear from Omar on. We already got some other ones teed up that we want to dig into. And if you want to connect with Omar, find him on LinkedIn, any other areas to find you. Omar is LinkedIn the best spot for folks to catch up with you. LinkedIn and the local cafe with a cappuccino. There you go. He takes cappuccino bribes as well.

20:42Great, Omar. Appreciate the time. And we will talk to you on the next Talking AI episode. Thank you. Goodbye. Thanks for listening to the Talking AI Podcast. If you enjoyed the show, give us a follow or subscribe on your favorite podcast platform. And don't forget to leave us a review. We love those. For more info on Talking AI, visit TalkingAIPodcast.com. The single biggest mistake we see companies make with AI is they don't properly train their teams. We see it all the time. Companies roll out AI tools and expect people to just figure it out. but using AI effectively requires a totally different mindset and skillset.

21:22And that's exactly why we built training for every level of your org, from AI training for teams and executives to training engineering teams on our generative driven development methodology. Or if you've already identified your AI use cases and want to just prioritize where to start, we offer an AI roadmap and ROI workshop to help you build a clear plan. It's all about going from we should use AI to actually driving real value with it. Head over to hatchworks.com to learn more. Thank you.