Context Engineering: Beyond Simple Prompting to LLM Architecture

0:00Welcome to the Deep Dive. Today we're plunging into a really fascinating shift happening right at the cutting edge of AI development. We've all heard plenty about prompt engineering. Oh, yeah, definitely. But a prominent voice in the AI community, Andrei Karpathy, he suggests we should maybe be talking about something else entirely, context engineering. That's absolutely right. And look, this isn't just about, you know, slapping a new label on things. It's a fundamental reframing. Yeah. It's about how we actually build industrial strength, large language model applications, LLM apps. Thanks, Mark.

0:31We've dug into a whole stack of sources, academic papers, industry analyses, you name it, to explore why this distinction really matters and what it truly means for the future software. So our mission today is to unpack this evolution. We want to show you why focusing on the entire information payload and LLM processes, not just the initial prompt, is really the key. The key to building reliable, scalable, and truly valuable AI products, moving way beyond what some people might just dismiss as a chat GPT wrapper. Exactly. We'll dive into the concrete engineering challenges and the solutions that really define context engineering, helping you understand this thick layer of non-trivial software that Carpathy talks about and why mastering that layer is where the real competitive advantage lies.

1:19Okay, let's get into it then. So maybe let's start by defining our terms just to level the playing field. What does prompt engineering traditionally mean and why is it starting to feel, well, insufficient now? Right. So traditionally, prompt engineering has been seen as this art and science of crafting a single input, right? A single instruction to guide an LLM to the output you want. Think of it like giving the AI a very specific roadmap for just one trip. Okay, like a specific instruction. Yeah. We've seen techniques like zero-shot prompting. That's a direct instruction with no examples. Like just summarize this article.

1:55Simple enough. Then there's few-shot prompting where you give it one or more input-output examples. You're basically teaching the model the format or the tone you want, showing it what good looks like in context learning, basically. And of course, chain of thought or cut T prompting, which inscreened the model to break down complex reasoning into like intermediate logical steps, thinking step by step. But the popular view often frames the prompt as this relatively static thing, maybe manually crafted text designed to get one single high quality response back. And that view, honestly, it just falls short for complex multi-turn applications.

2:31It's too narrow. And that's where Carpathie's pivot comes in, right? Moving beyond just that single prompt. He really pivots hard. It's not just about making the prompt bigger. It's a totally different mindset. Context engineering, as he defines it, is the delicate art and science of filling the context window with just the right information for the next step. Just the right information. Okay. Exactly. It shifts the focus from the explicit user instruction, the prompt, to the entire context window. That's everything the model actually sees at that moment. The prompt becomes just one part of this bigger programmatically assembled information payload.

3:05So what else goes in there? Well, you've got dynamically orchestrated elements. Things like the core task descriptions and instructions, sure, but also maybe dynamically selected few-shot examples. Definitely retrieval augmented generation or ARGI pulling in documents from external knowledge bases. Tool definitions so the model knows about APIs it can use, even applications date, and crucially managed conversational history. It's now seen as a formal discipline, really, that transcends just simple prompt design. That sounds like a much clearer separation of concerns, almost like different jobs, maybe.

3:40What's this mean for, you know, the developer actually building these things? That's a great way to put it. If you compare them side by side, prompt engineering is mostly that user-facing query. What did the user type in? Context engineering. That's the whole dynamic context window, the machinery behind the curtain. Backend versus frontend, almost. Sort of, yeah. The goal for prompting is often that single-turn response. Get one good answer. Context engineering aims for multi-step, reliable goals, orchestrating a process. Your artifacts for prompting are prompt strings, templates. For context engineering, you're looking at software architecture diagrams, beta pipelines.

4:17Totally different deliverables. Right. And the techniques shift from, say, creative phrasing in the prompt to things like art egg, tool use, memory management, model dispatching. More systems thinking. Exactly. The activity moves from the UI to the application backend. And the skill set leans less towards maybe linguist writer and more towards software architect, data engineer, ML engineer. So why is making this? Right. So you can think of the context window as the LM's working memory. It's RAM, basically. It holds everything the model uses to generate the next response. And it's measured in tokens.

4:50A token is roughly, say, three quarters of a word in English. And these windows are getting huge now, right? Oh, massively. Models like Google's Gemini 1.5 Pro can handle over a million tokens. But, and this is key, this expansion creates its own set of problems. Like what? More isn't always better. Not necessarily. First, there's increased cost and latency. Bigger contexts need more compute power, so API calls get more expensive and responses get slower. And second, there's this fascinating needle-in-a-haystack problem. Research shows LLMs often struggle with long contexts. They tend to pay more attention to information at the very beginning and the very end.

5:28And stuff in the middle gets lost. It can, yeah. Critical information buried deep inside might get ignored. So just naively stuffing the context window with tons of information is often counterproductive. It creates noise. The challenge shifts from just fitting information in to carefully arranging this huge window so the critical info, the signal, is salient and doesn't get drowned out. It reinforces Carpathie's point about just the right information. This seems like where retrieval augmented generation RG comes in. It's everywhere now, but it's way more than just a simple plug-in, isn't it? Oh, absolutely.

6:02RGE is anything but simple. It's a cornerstone architectural pattern. It fundamentally connects LLMs to external, often up-to-date knowledge bases. Its main jobs are to overcome limitations like knowledge cutoffs. You know, models are only trained up to a certain date and critically to reduce hallucinations by grounding the model in facts. So it fetches info first, then generates? Basically, yes. Instead of just relying on its internal pre-trained knowledge, a RAG system retrieves relevant info snippets based on the query and then augments the LLM's context with that retrieved information. The benefits are huge.

6:37Access to current info, way fewer hallucinations, and source attribution, which is critical and feels like finance, law, healthcare. You can trace where the answer came from. That traceability is huge. It is. Plus, updating a RAG knowledge base is much cheaper and faster than fully retraining a giant LLM. Okay, it's clearly powerful. But walk us through how it actually works. What's that pipeline look like? It sounds complex. It is definitely a complex, multi-stage pipeline. You don't just flip a switch. First, there's pre-retrieval. This is heavy data engineering. You're collecting, cleaning, normalizing data, and crucially, chunking documents into smaller, semantically coherent units, maybe 100 to 300 words.

7:16You also add rich metadata for filtering later on. Because, you know, the old saying, garbage in, garbage out, is incredibly true here. Data quality is paramount. Right. Get the input right first. Then comes the retrieval phase. This draws heavily on information retrieval science. You convert those text chunks and the user query into numerical vector representations using embedding models. These vectors get stored in specialized vector databases, things like Pinecone, Weevy 8, Chroma, QDRANT. Advanced systems often use multi-stage retrieval, maybe with re-ranking models, to really improve precision, finding the best chunks.

7:51Okay, so find the relevant needles. Exactly. And finally, there's post-retrieval, the generation part. Here, the system formats the top-ranked chunks and strategically places them into the context window, along with the original prompt for the LLM to use when generating its response. And the field is evolving fast. We've gone from what people call naive RAG to advanced RAG with more pre - and post-processing, and now even modular RAG where components are more flexible. Wow. Okay, so RAG isn't an add-on. It's a serious engineering discipline. It really exemplifies that non-trivial software Karpathy mentions.

8:25This is where the hard work happens. Absolutely. It requires deep expertise. What about examples? We mentioned few-shot prompting earlier. How does that become an engineering task in this context engineering world? Yeah. In context learning, using few-shot examples is really powerful. You provide examples right there in the context window, acting like on-the-fly training. So you might show the LLM, say, three examples of how to classify customer sentiment before asking it to classify a new piece of feedback. But you said dynamic selection. Right. In production systems, you typically don't use the same static examples every time.

8:58You'd have a library of examples, and the system dynamically selects the ones that are most semantically similar to the current task or query. To provide the most relevant guidance. Exactly. But this introduces a really interesting failure mode called example bleeding. Example bleeding? What's that? It's where the model gets confused between the examples you provided and the actual current conversation or task. So imagine you gave it an example about creating an invoice for Sasha Ivanov. Later, for a completely different request, the model might mistakenly say, I've already created an invoice for Sasha Ivanov because the example bled into its working memory of the current state.

9:35Ah, okay. That could be bad. How do you prevent that? It requires careful context formatting, using clear delimiters between sections, structuring inputs, maybe using JSON, and giving very explicit instructions to the model about what's an example versus what's the current task. It's trickier than it sounds to get right consistently. Yeah, I can see that. Okay, so beyond just getting information in or showing examples, how do we actually empower LLMs to do things, to become agents that can act? Great question. That's where tools, state, and history management become critical for enabling agency and coherence.

10:10Tool use is a big one. This involves describing external APIs, like a function, to search a database, send an email, check flight prices in a structured format within the LLM's context. The LLM can then reason about when it needs to use a tool, generate a specific tool call request, and an orchestration layer outside the LLM actually executes that API call. And feeds the result back. Exactly. It feeds the results back into the context. This thought-action-observation loop is fundamental for how modern LLM agents work. Okay. And state in history. Multi-turn conversations seem tricky. They are. Managing state in history is essential for any task that takes more than one turn.

10:49But just naively appending the entire conversation, history quickly blows out the context window limit. Right. You run out of space. So you need more sophisticated strategies. Things like summarization, maybe using a smaller, faster LLM to condense older parts of the conversation to save tokens. Or using Scratchpad's external memory where the LLM can jot down notes, intermediate conclusions, plans, and then selectively re-inject relevant bits back into the context later. And even structured memory where key entities mentioned in the conversation, like account numbers or user preferences, are extracted and stored in a separate database for consistent retrieval across terms.

11:25So lots of ways to manage that limited context space effectively. When you're building an application, how do you choose the right strategy for providing knowledge to the LLM? Argy, fine-tuning, just stuffing the context window. That's a core architectural decision, and it involves trade-offs. Let's compare the main three, reg, fine-tuning, and long-context stuffing. Argy, as we discussed, offers high data freshness because it can pull real-time info. It provides high factual grounding, author of citations. The upfront cost is relatively low, but per-query cost can be higher due to the retrieval step, and latency is medium.

11:59Implementation complexity is high. Best for knowledge-intensive tasks needing up-to-date, verifiable info from large document sets. Think customer support using current knowledge-based articles. Okay. What about fine-tuning? Fine-tuning involves retraining the model on specific data. This means data freshness is low, the knowledge is frozen at the time of training, Factual grounding is medium. It can learn facts, but can still hallucinate style or facts not strongly represented in the training data. The upfront cost is high training is expensive, but per query costs and latency are low. Implementation complexity is medium.

12:36And it's sweet spot. Best for teaching the model specific skills, a unique style or tone, or adapting it to a narrow, relatively static domain where you control the knowledge corpus. Got it. And the last one, just stuffing everything into a long context window. Long context stuffing. Data freshness can be high if you load recent documents, but factual grounding is often medium to low because of that needle in a haystack problem we talked about. Upfront cost is low, but per query cost can be very high because you're processing so many tokens, and latency is also very high. Conceptual complexity is low, you just dump it in, but performance can be unreliable.

13:13When would you use that? Maybe forgetting a holistic understanding of smaller, self-contained documents where almost all the information is relevant and needs to be considered together, but it's often less efficient or reliable than RRAG for larger tasks. Understanding those tradeoffs seems absolutely crucial for architects. It really is. Choosing the right context provisioning strategy is fundamental to building an effective and efficient LLM application. Okay, let's get back to Carpathy's emerging thick layer of non-trivial software. What does this layer actually look like in practice? How are people building software around these probabilistic, sometimes unpredictable, LLMs?

13:49Right. This thick layer is really the heart of a modern LLM application architecture. It's purpose-built to manage the unique challenges of working with non-deterministic models. A huge piece of it is control flow orchestration. Most useful tasks aren't just one LLM call. They need to be decomposed into smaller steps. This layer manages the sequencing of those steps. How does that work? Does the developer code the sequence? Sometimes, yes. You can have developer-defined control flow. The application code explicitly defines the workflow, often like a graph. The LLM acts as a component within that graph, called for specific functions like summarize this or extract entities here.

14:28This is good for predictable, repeatable tasks, like a standard customer support ticket resolution flow. The code is the orchestrator. But what about more dynamic tasks? That's where you see LLM-driven control flow, which is basically how agents work. Here, the LLM itself determines the sequence of actions based on a high-level goal you give it, plus the tools and memory it has access to. It decides, okay, based on the goal, I should first search the web, then analyze the results, then maybe draft an email. It figures out the plan dynamically. This is ideal for complex, open-ended problems, like a research assistant.

15:04And there are frameworks to help build this. Oh, yeah, absolutely. Tools like LangChain, LangGraph, Lamin Index, and Microsoft's Semantic Kernel and Control Flow are specifically designed to help developers manage these complex, often stateful workflows involving LLMs. And I imagine you're not always using just one single LLM for everything within these complex flows. Exactly right. Another critical part of this thick layer is model dispatching and routing. Sophisticated applications rarely rely on a single model. They use what you could call a council of models. A council of models. I like that.

15:36Yeah. Think of it like having different specialists. You intelligently route each subtask to the best model for that specific job. What factors go into that routing decision? Several things. Capability versus cost is a big one. You might use a powerful but expensive model like GPT-4 for complex reasoning or planning steps, but route simpler tasks like data extraction or basic summarization to a much cheaper, faster model. Load and resource management is another factor. Routing requests based on current GPU availability or API queue depth to ensure responsiveness. There are tools like LMD emerging for this.

16:13And task-specific requirements. Maybe routing to a model fine-tuned specifically for legal text analysis, or a medical Q &A model, or even a multimodal model if the input includes images. That makes a lot of sense. Use the right tool for the right job, even at the model level. But this brings up that core issue again. LLMs can hallucinate. They can be wrong. How do you build reliable systems on top of something that isn't always accurate? That's the million-token question, isn't it? Yeah. And the answer lies in designing for non-determinism and fallibility from the very start. Two key strategies here are generation verification flows and defensive UX.

16:46Okay, break those down. Generation verification. This often happens in the back end. It's technical loops where, for instance, one LLM call generates a potential answer, and a second LLM call, or maybe a set of rules or a check against a knowledge base, acts as a verifier for that answer. You see this a lot in areas like math problem solving or code generation, where you can actually check if the output is correct or runs without errors. Generate, then verify. And defensive UX, that sounds like it's about the user interface. Exactly. It's a term coined, I believe, by Eugene Yon. It refers to designing the user experience in a way that acknowledges the AI might make mistakes and builds user trust gracefully, even with those potential errors.

17:26It's moving towards LLM first product design. What does that look like in practice? Key patterns include setting expectations up front, being transparent about what the system can and can't do reliably, providing attribution, showing users where information came from, like an RAG, so they can judge its credibility, and crucially, designing human-in-the-loop workflows, building UIs where users can easily review, verify, correct, and ultimately oversee the AI suggestions or actions, the user becomes a supervisor, guiding and validating. So the UI isn't just taking orders. It's facilitating collaboration and correction.

18:02Precisely. It's a partnership model. Okay. So you have orchestration, routing, verification, defensive UX. This layer is getting thick. What else holds it all together? There must be an operational backbone. You absolutely need a robust operational backbone. This includes things like guardrails. These aren't just about safety in the sense of harmful content, but also programmatic constraints on the LLM's input and output. checking for correct syntax, preventing prompt injection, ensuring factual consistency against a known source, enforcing specific output formats. Frameworks like NVIDIA's NemoGuard Rails or Microsoft's Guidance help define these rules and corrective actions.

18:37Like bumpers on a bowling leg. Yeah, keeping the ball on track. Then there's security, which is a huge topic in itself, addressing new attack surfaces like sophisticated prompt injection, data leakage. This needs input sanitization, output filtering, data masking, strict access controls. Evaluations, or EVOLs, are also non-negotiable for production systems. You need continuous evaluation pipelines using task-specific test sets to measure performance and catch regressions when you update models or prompts. How do you evaluate these things effectively? It's challenging. An emerging best practice is the LLM as a judge pattern, where you use a very powerful LLM, like GPT-4, to score the outputs of your application model based on predefined criteria.

19:19It's surprisingly effective for many tasks. Using AI to evaluate AI. Interesting. It is. And of course, underneath all this new AI specific stuff, you still need rock solid traditional software engineering practices, robust logging, monitoring, error handling, data persistence, version control for prompts and configurations. All that good stuff is still essential. So if you pull all these pieces together, the orchestration, routing, verification, guardrails, evils. What is this new architectural paradigm look like for a software architect? It sounds like a whole new world. It really is. When you list out the components of this thick layer, control flow orchestration, managing the sequence, model dispatching routing requests, generation verification UX facilitating oversight, guardrails enforcing constraints, evaluations measuring quality, you see a distinct pattern emerge.

20:09The profound shift is that this new LM native architecture has a core computational unit that is probabilistic. It's non-deterministic. That forces entirely new design principles. We're borrowing ideas from distributed systems, from data engineering, from classic software design, sure, but everything has to be adapted to manage this inherent uncertainty and potential for error. Okay, that brings us to this term you hear thrown around, often negatively, chat GPT wrapper. What exactly is that, and how does Karpathy's thick layer argument push back against that idea. Yeah, that term wrapper definitely carries a critical tone.

20:43And it captures a very real anxiety, especially in the AI startup world. The anxiety is, how do you build something valuable, something defensible, when you're building on top of these massive foundational models developed and owned by huge tech companies? Right. What makes something more than just a wrapper? Well, a stereotypical chat GPT wrapper is seen as just a thin user interface slapped on top of a basic API call to an LLM like GPT-4 or Claude. The criticisms are usually threefold. 1. Lack of defensibility. It seems easy for competitors, or even the foundational model provider itself, to replicate the functionality quickly.

21:222. Often poor unit economics. The cost of the API calls can easily exceed the revenue generated, especially if pricing is competitive. It leads to a race to the bottom on price. 3. Dependency and commoditization. Your core value is tied directly to the underlying model. If that model changes or its pricing changes, your business can be instantly disrupted. You're building on rented land, essentially. So the big question is, when does an application stop being just a wrapper and become a genuine innovation with its own value? Exactly. That's the crucial question for anyone building in this space.

21:53So how do you build a defensible moat in AI then? If owning the foundation model isn't feasible for most, where does the defensibility come from? And this, I think, connects directly back to Karpathy's point. The thick layer of context engineering is precisely where that true innovation and competitive advantage lie. The value shifts. It moves away from the raw commodity intelligence of the LLM itself and towards the sophisticated, often proprietary system you build around it to direct that intelligence effectively. Can you give an analogy? Sure. Think about successful SaaS companies. Many are built on top of standard databases like PostgreSQL or cloud infrastructure like AWS.

22:33Their value isn't that they use PostgreSQL. It's the unique business logic, the workflows, the user experience they build on top of that commodity infrastructure. Okay, so the value is in the application layer, the system. How does context engineering create those moats specifically? There are several ways context engineering builds defensibility. First, proprietary data combined with superior R-GADE systems. If you have unique, valuable data sets and you build highly optimized RJ pipelines to leverage that data, the complex engineering around cleaning, chunking, embedding, retrieving that combination is incredibly powerful and hard to replicate.

23:09Got an example. Harvey AI is often cited here. It's a legal tech startup built using OpenAI's models, but it achieved a multi-billion dollar valuation. Why? Because it applied those models in a highly specialized way to proprietary legal data and complex legal workflows through sophisticated context engineering. The value wasn't just GPT-4, it was the application of GPT-4 through their engineered system. Okay, data in our ag is one mode. What else? Second, domain-specific workflows and orchestration. Embedding deep industry knowledge into the application's architecture, building complex multi-step control flows using that orchestration layer we've talked about that are tailored specifically to a vertical market, like financial analysis, medical diagnostics, or scientific research.

23:51This codified industry expertise, the custom logic built into how the LLM is prompted, used, and verified within a specific workflow is highly defensible. It's not generic. Makes sense. What's third? Third, user feedback flywheels. Remember that defensive UX and human-in-loop design. When done well, it's not just about usability. It becomes a powerful data collection mechanism. Explicit feedback, like thumbs up down on response, and implicit feedback, like whether a user accepts, ignores, or edits an AI suggestion creates this virtuous cycle. This feedback data is gold. You can use it to continuously refine your Aureg retrieval models, improve your dynamic few-shot example selection, tune your guardrails, maybe even fine-tune smaller models.