211 - AI SERIES: 19 - You're Using AI Wrong, Here's How to Fix It with Amanda Orson

0:00I don't use ChatGPT or Claude to my thinking for me. I want it to prompt me, not the other way around. One of the really valuable things that you can do if you're inside a company is to go and maybe that's not cool to say, but like it's fantastic. It's such a good output. The two things that I start every single project with are the strategic context prompt and the anti-sircophant prompt. Who would I call if I had the ability to call anybody? And it's not just living entrepreneurs. It's dead entrepreneurs.

0:33okay Amanda Orson I am so excited to have you back you are a serial entrepreneur serial founder you've most recently founded this company called Galleon which is a marketplace for buyers and sellers in real estate that eliminates the need for a real estate agent it's a freaking phenomenal platform that's a company you've been able to scale up to about$60 ,000 a month in monthly recurring revenue by leveraging AI and not building out a massive team. We talked, I don't know, six months ago and your episode on AI and how you're utilizing AI kicked off this like curiosity driven state that I've been in, in trying to figure out AI.

1:09What we're going to talk about today is sort of building out AI in a way that doesn't just validate, but it also prompts you by asking you questions. I'm fascinated by this whole process of like, how do I get the right prompt and how do I get it to act the right way? I think that people have gone from leveraging AI to get their team more efficient, which you just said, to now shifting their focus to agentic. Is that fair? I think that is fair. I think that's a broad, broad brush. And a lot of people have a different definition of what agentic is. But yes, I think that is where we're going. I don't feel like I've gotten like a really good definition of what agentic means, because I think that people are using it in a way that it's not actually intended to be used.

1:52What does agentic mean to you? This is like the debate on what is gen AI. I think that agentic really comes down to an interaction that is not purely reactive. So it has a non-binary output, meaning like I put X in and I'm not just getting X plus something. It's actually conversation based on large context and and should be dynamic, should be contextually relevant, should be personalized. I think that you know you're interacting with something that is more agentic when the result that you get is a surprise or is net new information. Do you think that sort of this chain of thought is actually a better way to approach agentic models as opposed to the traditional LLMs?

2:37I think that it depends on what it is you're trying to do. Like, are you trying to do something for which there are finite permutations? Or are you trying to do something for which like there is an infinite number of potentially good responses? You're way smarter than me. So I'll just say what I'm actually thinking. I think that the current state of AI, which people usually use it for like chat GPT is really good at taking fairly large data sets and then spitting out some type of analysis or summary or sort of regurgitating the data. In my experience, it's not great at finding novel things, new things.

3:11I think of the agentic as it can help find the novel things. It can go out and search and sort of do some analysis and like, oh, let me validate whether or not that's a reasonable thing. So like now when I use Grok 4 or when I use Claude or Chattopiti, I see it thinking. I see the steps that are like being permutated that it's going through. That feels more like a reasoning and chain of thought as opposed to just this regurgitated synopsis of what the data says. And that's how I think of the difference between just AI LLMs and agentic models. I don't know if that's a fair framework or not, though, because I don't know that I've anybody's like, there's no consensus on what it means.

3:49So I think that the simplest explanation is, I think, as you've just alluded to, like when you think about agency, you think about something that can actually have some independent thought or has the ability to come to conclusions other than that, which you've anticipated because it has agency. I think that that is just a helpful framework for understanding whether or not something is, you know, I added x plus y and I got, or, you know, I added x, and then I put some context in and then I got this whole new novel thing. For somebody who's listening is like, okay, nerds, move on to the practical stuff.

4:24The reason that this is important is in my opinion is because think about the last time you went on chat tpt, where you've used it and said, I'm trying to make this argument to this person or this legal argument or I'm trying to do X, Y, and Z. What do you think of my idea? What do you think of my thought process? What do you think of my analysis? And most of the time, almost all the time, actually, it sides with me. Oh, that's really smart. Yep. This is why you're right. Oh, I really like that. And it's like, it makes sense to me because you would want the AI to make you feel good about what you're thinking because then you're going to go back to use the AI more.

4:57So from a product standpoint, it feels like they've sort of engineered it. I don't know if they've done it on purpose, but that would make sense to me. However, when you're making high leverage decisions, I need impartial people who can be in the room to like say, oh no, this is where you're wrong. Actually, that's a terrible argument because of X, Y, and Z. And it doesn't do that unless I explicitly prompt it. And even then there are times where it doesn't really return things in the way that I want it to be. So that's why I like to talk about this like agentic versus LLM synopsis versus actual reasoning is because when I'm trying to use it in its application, I don't want something that's just going to be summarizing and saying, yeah, good job, Nick.

5:39That was a great idea. I want something that's going to be pushing back on me and saying, eh, that's not good for X, Y, Z reasons. I don't use chat GPT or Claude do my thinking for me. I use it as a Socratic mirror. I want it to prompt me, not the other way around. So the way that I use artificial intelligence as it applies specifically to the way that I'm thinking about things for Galleon is how do I construct sort of a living set of instructions on how to be a thought partner to me to help elevate the decision making and the decision making speed, the quality decision making speed that I'm making.

6:17Yeah, that's fast. Okay. Like how do I make it a board member or an advisor that's always in my pocket. Always in my pocket that I can go to. Okay, tell me how you know the answer. That's almost exactly what I was going to say is that it is more like a board member. And I think it's really, really important, the more senior you are within an organization, or the more isolated you are, because there aren't that many people that you have the ability to, you know, have sort of this thought partnership with, particularly that are going to be up to speed on the individual pieces of what's going on in your business.

6:55So I think the easiest way to do it that is most approachable for everybody is not actually setting up a custom GPT. I recognize that that's probably the right solution for higher level things. You'll get more like a larger context window over time. But I think the simplest way is to use a project either within Claude or within ChatGPT. For people that are maybe more familiar with one than the other. What I will say is that on the whole, I think Claude does a better job of adhering to the instructions. And I think that it has, its context window in your project is much larger. You can upload a lot of files and it's size limited where inside of ChatGPT, its project windows are limited to 20.

7:40So video's cool, but I have what scientists call a face for radio. And so what's even cooler is long form audio via my podcast and my newsletter. Nickonomicspod.com. Go there for free, subscribe to my newsletter. It's one email per week, super tactical. And then go to my audio podcast. I do three to five episodes a week, depending on how curious I am. And it's stuff like this. It's all free, no sleazy sales pitches. Nickonomicspod.com. So 20 files of whatever size. So it depends on what your use case is. I personally defer to Claude because I like having a lot of information in there. And then I distill the file format to the simplest, lowest size file possible, which is almost always RTF or a text file, if I can get it.

8:21Inside that context window, I have very specific project instructions. Can you show us? Can you just like pull up what Claude looks like? Yeah, yeah, yeah. And while she's pulling that up, you know, you said a GPT versus a project. So what you can do is you can actually create a GPT that has specific sets of instructions, which is effectively like a really long prompt that will function as, oh, you're a trusted advisor and I need you to give me unbiased feedback and I need you to find these pitfalls and you're an expert in whatever, biology, yada, yada, yada. The problem with a GPT is that information can be static and is more difficult to change.

8:56But in my experience, at least, the cool thing about projects is I can go and change that project knowledge, quote unquote, fairly easily. And it automatically just updates. So what we're looking at here, this is Claude. And oh, cool. I have my own project called Next Podcast. This is the project. And in here, you can see there's a context window. This is the normal place that you would chat and ask for instructions or help or whatever it is you need. And then this is the project knowledge. And this is what I'm talking about. With Claude, you have a lot more space to put a lot more context in.

9:31And it tends to adhere to your instructions better, in my opinion, than ChatGPT. It also is better at just not throwing random emojis and dashes and horizontal rules into things, which I appreciate. So if you're a business owner, project knowledge would be something like, I have a medical billing company. One of my coders is using a specific project. Let's call it healthcare coding is the project. I could upload all the ICD-10 codes. I could upload all of the diagnosis definitions. If I own a company that does, I don't know, electrical contracting, and I have a contractor out in the field. I could have them uploaded here a specific SOP for how they actually go out and price different projects, what they take into account, what's important, et cetera, just as a reference for them so that they can come to this and be like, hey, I can't remember if we're using, I'm way out of my depth here, if we're using copper versus silicon, how do I price this?

10:25And it can give a specific answer because the knowledge set from the company is included here in the project. So that's typically where the project knowledge will sit is in that set. But there is also the instructions, right? Yep. So how do you differentiate between the instructions and the project knowledge? I would say that instructions are persistent and always on, meaning that no matter what it is that you're actually engaging with, the instructions are always the instructions. For the actual context, especially if you're in a business, I highly, highly, highly advise these two that I have added.

11:01Now, your strategic context is going to completely vary, but I have what I call a strategic context prompt. And in the strategic context prompt, it gives very specific information in the context of a startup, your cash position, your monthly burn, your critical milestones, what your team looks like, who's on your team, your revenue engine. I mean, it goes into very specific detail here. And again, this is going to vary depending on your industry, but I highly recommend having a strategic context prompt for your business or whatever project it is, you want to be able to use this kind of thought partner, second brain board advisor, make sure that you have like persistent strategic context.

11:41And this is the file that I will update most often inside of any project because the strategic context changes all the time. Your team members. You're actually naming it that you're naming it strategic context. Yep. Do you have like a format for it? Or you're like, okay, these are the five things. Okay. Of course you do. I'm happy to actually share this prompt with you. So that you can give it to the watchers on this or the listeners on this. And the second one that I do is actually a tweet. And I will credit that person with the link at the end. But it's what I call my anti-sycophant prompt. So a big part of the problem.

12:16Yeah, huge part of the problem with using NALM is that it's validating. And that's unfortunately not what you need, especially in a leadership position, particularly in the most senior leadership position. So I was saying to Nick before we actually jumped on to this video, a big part of the problem with LLMs is that it helps you go really fast, but going fast and in the wrong direction is not actually going to help. So this is clutch. This actually allows, you know, Claude and to a lesser extent, ChatGPT to like gut check you and make sure that it's not just telling you what it thinks or has been programmed to think you want to hear.

12:54So the two things that I start every single project with are the strategic context prompt and the anti-sircophant prompt, just to sort of set the table. How long is that the total prompt? Is it just that paragraph? It's just that paragraph. Wow. Okay. And it really works. And you just upload these to the project knowledge right away? Right away. Yep. And how often do you update the, sorry, the strategic context? You said how often do you update that? I update it as needed. And again, mine varies a little bit. I'm going to stop sharing a lot for a second so I can actually share my window. But I think that it's really important to keep that one as up to date as possible because making decisions based on last week's information may not be what you need for next week.

13:37It just changes so fast. It's really, really important. So this is the document that I'm actually going to use to form the project instructions. And you'll see it is not a short document, but it gives very, very specific instructions on what I want us to be focused on from a company like Optempo perspective. What are the core principles of the way that I think about things based on reading, based on understanding best principles and certainly based on experience? How do I want the team to be held accountable? And then, you know, obviously what the sort of core principles are. Again, you'll see that I do a lot of coaching within the project instructions on making sure that it is not sycophantic.

14:20It's really, really important. And it is the bias of all of the LLMs. Like you should assume that the LLM is engineered to tell you what it thinks you want to hear unless you specifically instruct it otherwise. And even then.

14:38How long did it take you to create this prompt? It's tough to say because I probably wrote it once, twice, three times and, you know, have revised it and rewritten it several more times over since then. So I'm actually going to cut and paste it and make sure that that is inside. I mean, the quality of the output is dependent entirely on the quality of your input, right? Right. It's garbage in, garbage out. But it's why thinking through this and reiterating on your initial context prompts or your initial project instructions is so valuable. Because as the quality of the output you receive improves, so does your quality of decision making or the quality of your product or the quality of your customer experience or whatever it is that you're working on.

15:28So you'll see it in real time. And for me, it's a product of the quality and the velocity. Like I can go much faster this way. And I don't second guess myself as much because I've spent so much time, so much rigor in sort of thinking about things from all aspects. And this is where I would love to actually show you. I spend a lot of time thinking about who would I call if I had the ability to call anybody. And it's not just, you know, living entrepreneurs. It's dead entrepreneurs. It's everybody that I can possibly think of. Like who has already solved a problem that is either similar to this or who would have an interesting perspective?

16:07And that's, I think, where I can really get some cool results that I don't think a lot of people are doing right now. Maybe they are. But I spend a lot of time actually going through and either transcribing or downloading the transcriptions or gathering quotes, whatever, from specific entrepreneurs. The two that I prepared here are actually, one's not an entrepreneur, but I grabbed some stuff from Chris Jenner and some stuff from Chris Voss, who wrote Never Split the Difference on negotiation. These are the kinds of things that I will commonly update into project knowledge so that I can ask specific questions as if I am prompting or asking them.

16:47So, I mean. Okay, so you've created this project. We have the two that you always put in there, the strategic context, as well as the anti sycophant prompt. You've also got the instructions for the project. Now what we're talking about is, okay, we've got the table stakes for the game. Now let's go and get the context that we want in order for this to turn into the project that we need for our business. and you're talking about going to find people who have philosophies that might give us insights or evaluate our decision making in order for us to make good decisions. Can you show your desktop?

17:25Is that okay? Because I think that was really interesting to show. I just thought that was fascinating the way you had organized stuff. Hey, I don't know if you remember this, but when we started this podcast, we entered into a social contract. I would spend time, energy, and money producing this podcast, interviewing these individuals and giving you insights into how to build, buy, start, grow your business. And you would like, subscribe, and leave me five-star review. Now, out of that, we both get to talk to really cool people and hear really cool insights. We both get a ton of value. But I just want to help you keep your word.

17:57So would you do me a favor? Will you go leave a five-star review for me on Apple or Spotify? It would really help. And if you want, even share this with a friend. This is not my typical screen. I actually normally have a lot of other things up here. But for the purposes of this podcast, I wanted to just highlight. So I will go and grab transcripts. That's my favorite source. I like getting primary sources. I think that's actually really valuable. If you can get people in their own words, I'm happy to go and grab stuff from individual books, if they've written books. I think that people tend to say more things, honestly, off the cuff when they're being interviewed.

18:30So if I can grab interview transcripts, I will grab interview transcripts. I really like to use a tool. I'll go to YouTube. I'll go to YouTube and get the transcriptions of people's conversations. Yeah, absolutely. That's actually one of what was the tool? I was going to say nada, but it doesn't matter what the tool is. There are multiple tools that will do this for you. But I think that YouTube into a tool that actually gets the transcript, if it's not already there organically under YouTube, that is the way. So that's literally this is a transcript. And I will add one of the really valuable things that you can do if you're inside a company is to go and grab your meeting transcript.

19:04Maybe that's not cool to say, but it's fantastic. It's such a good output. Like if you want to just distill what were the actionable takeaways, what were the deliverables, who said they were going to do what by when, like meeting transcripts all day long, and then upload them into your project. So is this the transcript or is it the analysis of the conversation? This is actually the transcript. This is combined analysis. I think I asked actually perplexity and ChatGPT to distill a whole bunch of information. You can see it's black. There's still a whole bunch of information into its essence for Chris Voss.

19:39For Chris Jenner, unfortunately, this transcript actually failed, but I do have the other one. I was doing this quickly. I do have the other context window. So typically for my own projects, I will have a person or a company. It doesn't matter, but I'll have like large volumes of text. So I got really into the industrialists at the top of the year and I have like reams of data on transcriptions from podcasts and you name it on how the early industrialists did what they did because they had a greenfield experience in much the same way that I think a lot of entrepreneurs do right now with AI. So I really wanted to think about how they thought about doing something that had never been done before.

20:20And that was how I sort of trained my own brain to think. Okay, so you'll take the transcripts and now you've uploaded them to Plot, just the transcripts. Correct. So when I do it, I usually will take the transcripts and I'll take multiple transcripts and then I'll upload it either to Claude. My favorite recently has been notebook LM. I love notebook LM for large written data sets. And then I'll ask it to write me like a really in-depth summary of the essence to steal your word. And then I'll take that and I'll upload that to my project knowledge because then I'm not taking as much, I guess, space for that context window.

20:56So do you do that or do you just upload the transcripts? I, as far as possible, upload transcripts. So definitely if you have a finite context window, because especially if you summarize things with chat GPT, I'm going to beat up on OpenAI here. It gives you like the 30 ,000 foot view, but it's not doing you the service of like the really super tactical takeaways. Like I just reread Essentialism by Greg McKeown over the weekend. And I'd read that book probably 15 years ago, or excuse me, 10 years ago when it first came out. And I missed some details in there. And one of the great details that if you go to Goodreads or any of the other places that have, you know, created transcripts and quotes about that book that I hadn't picked up on was he says, you know, no matter what it is you're doing, add 50%, like add 50 % of the time that you think it's going to complete or take to complete.

21:45And then he gives all the scientific and statistical analysis on why you should always do that. I completely missed it. And in no recap, did I see it? And that's the kind of stuff that you're going to miss when you get just like the 30 ,000 foot overview and ask ChatGPT to synthesize the most salient takeaways from a transcript, where if you actually upload the transcript, it'll pull it out. That's why I look, do you use Notebook LM at all? Yep. I love Notebook LM because it doesn't hallucinate and it cites everything that it says. let's come up with a scenario where we would want to get what Chris Jenner's opinion is or how Chris Voss would handle a negotiation.

22:25And bear in mind that this strategic context is not actually populated. So I didn't give it the specific information about our company, but let's think about a potential scenario for maybe a popular company right now that we would want to understand kind of what they think their feedback would be? OpenAI. Okay. What about OpenAI? If I was OpenAI, I would want to know how their users... What I'm seeing right now is like this shift from Golden Boy to Mark Zuckerberg. Like there's a lot of distrust with Mark Zuckerberg. There's a lot of people who look at Mark Zuckerberg sideways. And I feel like Sam Altman burst onto the scene as a Golden Boy and now has done a number of things that people are like, that's weird.

23:10you know like they don't they don't fully trust him especially in the tech community they're very weary of him but outside the tech community i think people are just like getting this perception of oh he's going to do anything he can to win and if that's the case then what does that mean for ai and is he going to do things that are potentially dangerous so if i was them and i was looking at the future i would want to know what is our brand perception how do people view us in the market. Okay. So the scenario that we want to feed back, and ironically we're putting this into Claude, is that I am Sam Altman.

23:48The public perception of my personal brand may be, let's call it, distracting from the growth of the professional brand. Yep. What are ways to think about repositioning or pivoting this brand? or what are ways in which, you know, I could, if, if I was talking to say Kris Jenner, what are ways in which she would think about repositioning? Yeah. Man, you've got my head spinning with uploading the transcripts. I think that's amazing. I'm definitely going to start doing that. And let's see what comes up with. I genuinely, this is all Greenfield. I have no idea. And we did not open up a lot of context here from Chris.

24:32Does it actually go to just Kris Jenner's transcripts or is it going to all the transcripts? So it's going through all, but I've specifically asked it to tap the stuff that we have uploaded from Kris. And it's actually pretty, it's pretty good. I love the column. Okay, read some of it. I can't read it. Okay, so first of all, Claude opens up as Kris here, which I'm sure Kris would not approve of. So apologies in advance. Please do not sue us, Kris Jenner. your personal brand isn't just distracting from open ai it is actively undermining the company's potential the hard truth is that you've become the story instead of open ai being the story every controversy every board drama every cryptic tweet is bandwidth stolen from your company's mission oof it's true and then it goes into some very tactical things that you should start fixing one redirect the stop the spotlight stop being the face of every ai announcement like give some of that power back to your engineers and your researchers and actual innovation that's fantastic strategic vulnerability the whole boy genius saves the world narrative which is funny because it just totally anticipated what you had said but it did not have project context on is undermining and it's played out the third point boring is profitable your twitter needs to become ruthlessly focused on open ai's mission and progress no philosophical musings no industry hot takes pure business value yeah so anti-sycophant is so good it is really good and a lot of this stuff so some of this the hard truth and here's what we're fixing immediately that's actually in the project instructions that i gave this particular project but this is like the pithiest lowest effort possible output from a completely like greenfield this wasn't even a good prompt.

26:21We just totally shipped it. But this is what I mean. Like if you actually train it. Dude, this is so cool, Amanda. Wow. Yeah. Because you're like, hey, I want to know what would Steve Jobs say to me or what would, I don't know, Andrew Carnegie or what would name your person? Exactly. So actually, on that note, I will give a shout out. David Senra at Founders Podcast has his own AI that has actually been trained on the transcripts from his own podcasts over many years now and the notes that he's taken from those individual books. So that is a great way to go and tap and seek, I would say, like new discovery knowledge that you may not already know about.